RetroSphere: Self Contained 3D Controller Tracking
for Augmented Reality

ACM IMWUT Volume 6 Issue 4 (2022)

*National University of Singapore, #Google LLC, ^University of Maryland, College Park

Download DepthLab Paper in pdf
Download DepthLab Paper in pdf
Download DepthLab Paper in pdf
Download DepthLab Paper in pdf

Click to download a PDF of the paper.


Advanced AR/VR headsets often have a dedicated depth sensor or multiple cameras, high processing power, and a highcapacity battery to track hands or controllers. However, these approaches are not compatible with the small form factor and limited thermal capacity of lightweight AR devices. In this paper, we present RetroSphere, a self-contained 6 degree of freedom (6DoF) controller tracker that can be integrated with almost any device. RetroSphere tracks a passive controller with just 3 retroreflective spheres using a stereo pair of mass-produced infrared blob trackers, each with its own infrared LED emitters. As the sphere is completely passive, no electronics or recharging is required. Each object tracking camera provides a tiny Arduino-compatible ESP32 microcontroller with the 2D position of the spheres. A lightweight stereo depth estimation algorithm that runs on the ESP32 performs 6DoF tracking of the passive controller. Also, RetroSphere provides an auto-calibration procedure to calibrate the stereo IR tracker setup. Our work builds upon Johnny Lee’s Wii remote hacks and aims to enable a community of researchers, designers, and makers to use 3D input in their projects with affordable off-the-shelf components. RetroSphere achieves a tracking accuracy of about 96.5% with errors as low as ∼3.5 cm over a 100 cm tracking range, validated with ground truth 3D data obtained using a LIDAR camera while consuming around 400 mW. We provide implementation details, evaluate the accuracy of our system, and demonstrate example applications, such as mobile AR drawing, 3D measurement, etc. with our Retrosphere-enabled AR glasses and smartphone prototypes.

Medium Blog

Our medium blog post describing RetroSphere can be found in this link.

Talk at ACM Ubicomp/ISWC 2023

RetroSphere 6DoF tracking hardware as well as the ESP32 firmware is available as open-source design on GitHub. Unity based desktop 3D experiences enabled with retrosphere will be released soon. Please stay tuned!

ACM Citation Format