Object Importance Estimation using Counterfactual Reasoning for Intelligent Driving

Pranay Gupta1, Abhijat Biswas1, Henny Admoni1, David Held1,
1The Robotics Institute, Carnegie Mellon University,
arXiv (Coming Soon) Code & Dataset
Interpolate start reference image.

Red boxes indicate vehicles important for making safe driving decisions by the ego vehicle.

Interpolation end reference image.

The blue car is important since its removal would cause the ego vehicle to speed up.

Interpolation end reference image.

The red vehicle is important since its changing lanes would risk a collision with the ego vehicle.

We perform counterfactual reasoning to identify important vehicles in a driving scenario. We modify the motion of vehicles, and ascribe importance based on how the modification affects the ego vehicle's driving.

Abstract

The ability to identify important objects in a complex and dynamic driving environment is essential for autonomous driving agents to make safe and efficient driving decisions. It also helps assistive driving systems decide when to alert drivers.

We tackle object importance estimation in a data-driven fashion and introduce HOIST - Human-annotated Object Importance in Simulated Traffic. HOIST contains driving scenarios with human-annotated importance labels for vehicles and pedestrians. We additionally propose a novel approach that relies on counterfactual reasoning to estimate an object's importance. We generate counterfactual scenarios by modifying the motion of objects and ascribe importance based on how the modifications affect the ego vehicle's driving.

Our approach outperforms strong baselines for the task of object importance estimation on HOIST. We also perform ablation studies to justify our design choices and show the significance of the different components of our proposed approach.

Annotation Instructions and Interface

*Click on images to zoom
Annotation Interface Template