The technology helps self-driving cars learn from their own memories

An autonomous vehicle is able tonavigate city streets and other less busy environments by recognizing pedestrians,other means and possible obstacles throughartificial intelligence. This is achieved with the help of artificial neural networks, which are trained to “see” the car’s surroundings, imitating the human visual perception system.

But unlike humans, machines using artificial neural networks have no memory of the past and are in a constant state of seeing the world for the first time – no matter how many times they’ve traveled down a given road before. This is particularly problematic in adverse weather conditions, when the car cannot safely rely on its sensors.

Researchers at Cornell Ann S.The Bowers College of Computer and Information Sciences and the College of Engineering have produced three simultaneous research papers with the goal of overcoming this limitation by giving the car the ability to create “memories” of previous experiences and use them in navigation next.

Carlos Diaz-Ruiz, a PhD student, drives the data collection machine and demonstrates some of the data collection techniques that autonomous vehicle researchers use to create their algorithms.

PhD student Yurong You is the lead author of HINDSIGHT is 20/20: Leveraging Past Transversal to Aid 3D Perception, which you presented virtually in April at ICLR 2022, the International Conference on Learning Representations. “Learning representations” includes deep learning, a type of machine learning.

“The essential question is, can we learn from repeated traversals?” said the senior author Kilian Weinberger, Cornell Bowers CIS professor of computer science. “For example, a car might mistake an odd-shaped tree for a pedestrian when its laser scanner first perceives it from a distance, but once it’s close enough, the object’s category will become clear. So the second time you pass the same tree, even in fog or snow, you’ll hope that the car has now learned to recognize it correctly.”

“In reality, you rarely drive a road for the first time,” said co-author Katie Luo, a doctoral student in the research group. “Either you or someone else has run it recently, so it seems natural to gather that experience and use it.”

Led by doctoral student Carlos Diaz-Ruiz, the group compiled a dataset by driving a car equipped with LiDAR (Light Detection and Ranging) sensors repeatedly along a 15-kilometer loop in and around Ithaca, 40 times during an 18-month period. Trips capture different environments (highway, urban, campus), weather conditions (sunny, rain, snow) and time of day.

This resulting dataset – which the group refers to as Ithaca365, and which is the subject of one of two other papers – has more than 600,000 scenes.

“It deliberately exposes one of the main challenges in self-driving cars: bad weather conditions,” said Diaz-Ruiz, a co-author of the Ithaca365 paper. “If the road is covered in snow, people can rely on memories, but without memories a neural network is at a huge disadvantage.”

HINDSIGHT is an approach that uses neural networks to calculate object descriptors as the car drives past them. It then compresses these descriptions, which the group has named SQuaSH(Spatial-Quantized Sparse History) features and stores them in a virtual map, similar to a “memory” stored in the human brain.

The next time the self-driving car traverses the same location, it can search the local SQuaSH database of each LiDAR point along the route and “remember” what it learned last time. The database is constantly updated and shared across vehicles, thus enriching the information available to perform recognition.

“This information can be added as a feature to any LiDAR-based 3D object detector;” You said. “Both the detector and the SQuaSH representation can be trained together without any additional supervision, or human annotation, which is time-consuming and labor-intensive.”

While HINDSIGHT still assumes that the artificial neural network is already trained to detect objects and augments it with the ability to create memories, MODEST (Mobile Object Detection with Transitivity and Self-Training) – subject to third edition – goes even further beyond.

Here, the authors let the machine learn the entire perception pipeline from scratch. Initially, the artificial neural network in the vehicle was not exposed to any object or road at all. Through multiple passes of the same path, he can learn which parts of the environment are stationary and which are moving objects. It slowly learns on its own what constitutes other traffic participants and what is safe to ignore.

The algorithm can then detect these objects reliably – even on roads that were not part of the initial repeated passes.

The researchers hope that both approaches can drastically reduce the cost of developing autonomous vehicles (which currently still rely heavily on expensive human annotation data) and make such vehicles more efficient at learning to navigate places. in which they are used the most.

Both Ithaca365 and MODEST will be presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2022), to be held June 19-24 in New Orleans.

Other contributors include Mark CampbellJohn A. Mellowes ’60 Professor of Mechanical Engineering in the Sibley School of Mechanical and Aerospace Engineering, assistant professors Bharat Hariharan AND Wen Sun, in computer science at Bowers CIS; former postdoctoral researcher Wei-Lun Chao, now assistant professor of computer science and engineering at Ohio State;and PhD students Cheng Perng Phoo, Xiangyu Chen and Junan Chen.

Research for all three papers was supported by grants from the National Science Foundation; Office of Naval Research; and Semiconductor Research Corporation.

Leave a Comment

Your email address will not be published.