Camera that sees beyond the curve leaves the laboratory and reaches the cell phone

Your cell phone's LiDAR sensor - which captures depth, generates 3D information about the environment and adjusts the camera's focus - has just gained a function that seemed impossible outside research laboratories: seeing what is outside the field of vision, such as behind a corner, using only light reflections and algorithms.

Recently published in the journal Nature by researchers from the MIT Media Lab in the USA, a study demonstrated that the ability to image objects outside the direct line of sight (NLOS) - until now restricted to specialized equipment - can be done on ordinary cell phones.

Today, LiDAR in smartphones, as well as in wearables and robots, already measures the time of flight of light with picosecond precision (one trillionth of a second). In fact, this resolution present in common devices was the starting point for detecting light signals that bounce off walls before reaching hidden objects.

Until now, this feat was exclusive to laboratory equipment that cost around US$50,000 (around R$250,000) and required rigorous calibration. The novelty brought by MIT is that the same result was obtained with sensors already available on the market, costing less than US$100.

The good news is that the new system worked without the need for calibration or prior configuration. This means that the smartphone has the hardware (LiDAR), but that the software is still at research level (prototype), before inaugurating what the authors call plug-and-play NLOS imaging - just plug and play.

How camera shake is no longer a problem

Seeing around corners with consumer LiDAR

The main mechanism of the study is a strategy called motion-induced aperture sampling. In this method, the central idea is that the natural tremor of whoever holds the device stops being "noise" and starts functioning as a source of information.

The approach combines two ideas already known in computational photography and remote sensing: image stacking, common in night photos, and synthetic aperture radar, a satellite technique that uses movement to increase spatial resolution.

The result of this is a multi-image fusion that brings together these faint signals of light reflected from nearby walls and floors to reconstruct the shape and position of objects outside the field of view - like something hiding just around a corner.

The tests covered three different scenarios: the three-dimensional reconstruction of hidden objects, the tracking of movement - such as that of a person's hands - and the location of the camera in space based on objects outside its field of vision.

Immediate applications and some limitations for the technology

An autonomous robot could use LiDAR to detect obstacles outside its viewing angle - Siddharth Somasundaram/MIT Media Labs/Disclosure
As it is software created for already available hardware, the immediate applications are numerous. According to the researchers, autonomous vehicles could detect obstacles at blind intersections and augmented reality headsets would track the user's arms and legs even when they leave the camera's viewing angle.

The novelty would also benefit navigation robots (vacuum cleaners, logistics, delivery, and autonomous vehicles) in environments with few visual textures, such as white corridors and warehouses. With NLOS, these systems would have a more complete spatial perception of the surroundings.

Although the best results were obtained with reflective objects (such as the sticker used in the tests), the authors recognize that, on common surfaces - such as walls and clothing -, the operation occurs with a weaker signal, which still needs to be improved for everyday use.

It is worth remembering that this is a research prototype: the hardware is already in cell phones, but the algorithms need to evolve, and the LiDAR APIs of current operating systems do not support this type of application. The path exists; the deadline is still uncertain.

The end of the "original" photo? AI redefines the role of the camera on smartphones

Source: CNN