Thermal imaging innovation allows AI to see through pitch darkness like broad daylight

Researchers at Purdue University have developed a groundbreaking method called HADAR (heat-assisted detection and ranging) that enhances traditional machine vision and perception in the field of robotics and autonomy. The research was led by Zubin Jacob, the Elmore Associate Professor of Electrical and Computer Engineering, and research scientist Fanglin Bao, from the Elmore Family School of Electrical and Computer Engineering.

HADAR combines thermal physics, infrared imaging, and machine learning to enable fully passive and physics-aware machine perception. Unlike active sensors like LiDAR, radar, and sonar that emit signals, HADAR utilizes thermal imaging to collect invisible heat radiation emitted by objects in a scene. This approach allows it to sense through darkness, fog, rain, and solar glare without the drawbacks of traditional active sensors.

The method overcomes challenges faced by traditional thermal imaging, such as the loss of texture and features in the images, known as the “ghosting effect.” HADAR TeX vision, as it’s called, can vividly recover texture from the cluttered heat signal and accurately distinguish temperature, emissivity, and texture (TeX) of objects in a scene. It can see texture and depth in pitch darkness as if it were daylight, effectively enabling perception regardless of lighting conditions.

The team demonstrated the capabilities of HADAR TeX vision in an off-road nighttime scene, recovering textures and details such as water ripples, bark wrinkles, and culverts. While the current sensor used for HADAR is large and slow, the team aims to improve the hardware’s size and data collection speed to make it suitable for real-time applications like self-driving cars and robots. Currently, the sensor takes around one second to create an image, but for autonomous cars, a frame rate of 30 to 60 hertz is needed.

The initial applications of HADAR TeX vision are expected in automated vehicles and robots that interact with humans in complex environments. However, the technology has potential uses in various industries, including agriculture, defense, geosciences, healthcare, and wildlife monitoring.

To protect their innovative method, Jacob and Bao have disclosed HADAR TeX to the Purdue Innovates Office of Technology Commercialization, which has applied for a patent on the intellectual property. Industry partners interested in further developing the technology should contact Dipak Narula.

The research on HADAR was featured on the cover of the July 26 issue of the peer-reviewed journal Nature, and there is a video about HADAR available on YouTube. Additionally, Nature has released a podcast episode featuring an interview with Zubin Jacob discussing the technology.