International Conference · 2023
Multi-Person Tracking Method Robust to Dynamic Viewport Changes for AR apps
Abstract
Augmented reality (AR) devices have gained a lot of attention in recent years due to their ability to enhance people’s abilities through 2D/3D spatial sensing and recognition functions. RGB cameras are most often used as sensors in this type of spatial recognition, and a particularly important task is the detection and tracking of objects and people in physical space. However, the camera positions and orientations on AR devices such as smartphones and smart glasses, frequently change due to the user wearing them on their head, leading to non-linear and complex motion in the video frames and reducing the accuracy of tracking people. To address this issue, the proposed method combines person re-identification based on deep metric learning with trajectory prediction to estimate the person’s sequential positions in 3D space around the camera. The experimental result shows 95.45% accuracy with our dataset.
Research Note
In recent years, RGB cameras and depth cameras integrated into Augmented Reality (AR) glasses and smart glasses have been utilized for recognizing people and objects in real-world environments. Various applications have been proposed to overlay cyber information onto real-world visuals or directly onto smart glasses. For instance, in the field of sports, systems have been developed to detect and identify players during games or training sessions and provide relevant player information to spectators and coaches wearing smart glasses. To realize such smart glasses applications, tracking people in captured video footage is essential.
Numerous object tracking methods have been proposed. For example, DeepSORT detects target objects in images using the object detection method YOLO and tracks them across consecutive frames in a 2D image sequence using a Kalman filter, which assumes stable motion. However, most existing person tracking methods, including DeepSORT, are designed for fixed-camera video footage. In situations where the camera position and orientation change, extended occlusions lasting several frames, unexpected movement within or outside the image frame, and other disruptions frequently occur. As a result, tracking accuracy significantly deteriorates when relying on a Kalman filter, which assumes smooth movement.
This study proposes a method to determine whether a newly detected person in a video corresponds to a previously tracked individual, thereby extending conventional multiple object tracking (MOT) methods designed for fixed-camera footage. By appropriately applying person re-identification (Re-ID) when a previously detected individual reappears, we enable continuous tracking of people in variable viewpoint videos captured by smart glasses. Furthermore, our approach estimates the last known location and disappearance time of a detected person to predict their likely position over time. By utilizing the gaze direction and position of the smart glasses, we estimate whether a person is within the current field of view, thereby suppressing unnecessary re-identification of individuals who are unlikely to be present. This enhances the accuracy of person re-identification.
発表論文
- 高橋直也, 天野辰哉, 山口弘純, & 東野輝夫. (2021). 深層距離学習を用いた AR デバイス向けの人物識別手法. マルチメディア, 分散協調とモバイルシンポジウム 2021 論文集, 2021(1), 388-394.
- 高橋直也, 天野辰哉, & 山口弘純. (2021). スマートグラスの可視領域情報を用いた不連続動画上の人物追跡. 研究報告モバイルコンピューティングと新社会システム (MBL), 2021(29), 1-8.
- Takahashi, N., Amano, T., & Yamaguchi, H. (2023, June). Multi-Person Tracking Method Robust to Dynamic Viewport Changes for AR apps. In 2023 19th International Conference on Intelligent Environments (IE) (pp. 1-4). IEEE.
Related Projects
