VERDI: VLM-Embedded Reasoning
for Autonomous Driving

Abstract

While autonomous driving (AD) stacks struggle with decision making under partial observability and real-world complexity, human drivers are capable of applying commonsense reasoning to make near-optimal decisions with limited information. Recent work has attempted to leverage finetuned Vision-Language Models (VLMs) for trajectory planning at inference time to emulate human behavior, but the long inference time makes them impractical to deploy. To bridge this gap, we propose VLM-Embedded Reasoning for autonomous DrIving (VERDI), a training-time framework that distills the reasoning process and commonsense knowledge of VLMs into the AD stack. VERDI augments modular differentiable end-to-end (e2e) AD models by aligning intermediate module outputs at the perception, prediction, and planning stages with text features explaining the driving reasoning process produced by VLMs. We validate VERDI in both open-loop (NuScenes and Bench2Drive benchmarks) and closed-loop (HugSim Simulator) settings. We find that VERDI outperforms existing e2e methods that do not embed reasoning by up to 11% in distance and 11% in driving performance, while maintaining real-time inference speed.

Approach

We align language features acquired from the VLM reasoning module with driving features acquired from the e2e driving model. Specifically, we do so for each of the perception, prediction, and planning submodules to effectively embed the VLM reasoning process into the e2e model.

Aligning with the e2e Model

To obtain driving features, we introduce a learnable Progressive Feature Projector (PFP) to each of the perception, prediction, and planning submodules. The PFPs map driving features to the same latent space as the language features, and align them with a cosine similarity loss.

Experiments

Closed-Loop Experiments

Evaluation on the HugSim Closed-loop Simulator Benchmark with the HUGSIM nuScenes dataset. Metrics are averaged by difficulty levels. The best results between architectural-equivalent models (VAD, VERDI) are bolded. We further show the best results overall as underlined. Results show scene-level performance on core metrics including No Collision (NC), Drivable Area Compliance (DAC), Time-To-Collision (TTC), Comfort (COM), Route Completion (Rc), and HugSim Driving Score (HDScore). VERDI achieves the best overall HDScore and improves NC by 10% over VAD-Base.

Open-Loop Experiments

We show superior performance against our direct baseline VAD, a supervised e2e model without embedded reasoning. The video shows the multi-view camera observations on the left and the BEV view on the right. Each of the example shows our successful performance on the perception, prediction, and planning modules. We also show the VLM text response for each testing case to demonstrate that VERDI has successfully distilled VLM's reasoning and commonsense knowledge.

nuScenes evaluation results. Our method performs 10% better than the direct baseline VAD. Methods are compared according to: (1) Whether a VLM is required at inference; (2) Inference speed (FPS); (3) Trajectory accuracy, measured as the \( \ell_{2} \) distance to the expert trajectory at 1s, 2s, and 3s horizon; and (4) Whether precise historical ego-vehicle state is used in planning. In a fair comparison with methods not having privileged access to ego status, including location, VERDI achieves the best performance across all metrics.

Bench2Drive evaluation results. Average is averaged over predictions in 2 seconds under 2Hz. * denotes expert feature distillation. VERDI performs over 11% better than the direct baseline VAD and achieves the best performance among all methods that require feature distillation.