Tracking Objects Outside the Line of Sight using Speckle Imaging
Proc. CVPR 2018
Techniques for tracking micro-motion (<10 microns) of multiple objects, around the corner, with only low-cost components
This paper presents techniques for tracking non-line-of-sight (NLOS) objects using speckle imaging. We develop a novel speckle formation and motion model where both the sensor and the source view objects only indirectly via a diffuse wall. We show that this NLOS imaging scenario is analogous to direct LOS imaging with the wall acting as a virtual, bare (lens-less) sensor. This enables tracking of a single, rigidly moving NLOS object using existing speckle-based motion estimation techniques. However, when imaging multiple NLOS objects, the speckle components due to different objects are superimposed on the virtual bare sensor image, and cannot be analyzed separately for recovering the motion of individual objects. We develop a novel clustering algorithm based on the statistical and geometrical properties of speckle images, which enables identifying the motion trajectories of multiple, independently moving NLOS objects. We demonstrate, for the first time, tracking individual trajectories of multiple objects around a corner with extreme precision (< 10 microns) using only off-the-shelf imaging components.
Proc. CVPR 2018
Proc. SIGGRAPH 2017
Finalist, WARF Innovation Awards
Proc. ICCV 2015
Tracking objects around a corner (left) is analogous to direct line of sight imaging, where the objects are illuminated directly by a diffuse source and imaged with a bare sensor (right).
When an object moves laterally, the recorded speckle pattern shifts (left); when an object moves axially, the speckle pattern expands and contracts (center); and when an object rotates about the camera viewing axis, the speckle pattern rotates (right). This relationship is linear.
Previous speckle based techniques for multi-object motion estimation compute a motion histogram where each object’s motion is represented as a peak. Since the peaks are not assigned to individual objects, motion histogram is an ambiguous representation. Different scene motions can result in the same motion histogram, which prevents tracking more than one object. In contrast, we show that it is possible to track multiple objects by developing a clustering algorithm based on the statistical properties of speckle, that assigns each peak in the motion histograms to a unique object over time.
We could separate individual speckle patterns explicitly, but we take a clustering based approach based on the statistics of speckle patterns.
Speckle images were recorded at one second intervals. Despite the large dynamic range of motion between second, minute, and hour hands, we were able to track all wristwatch hands simultaneously with high precision.
The primary speckle image encodes the object motion, and can be isolated by taking the ratio of the raw captured image and the mean image (computed by averaging all the raw captured images as the object moves).
Single and multiple object trajectories. Despite the macroscopic (cm) scale of the setup, we show that it is possible to recover complex, micron scale trajectories of multiple objects moving simultaneously outside the line of sight.