ISPACS 2026

ISPACS 2026

Vehicle Lane-Change Trajectory Pattern Analysis and Policy Learning in Highway Driving Scenarios

Offline reinforcement learning with Implicit Q-Learning on the AD4RL cut-in benchmark

Hyeonsu Kim · Yiwen Shen

Department of Software and Computer Engineering, Ajou University, Suwon, Korea

MOTIVATION

Where safety and speed collide

Overtaking slower traffic is efficient — until a neighbor suddenly cuts in.

2

PROBLEM

Why offline RL — and its trap

Offline RL

Learns only from logged driving data — no risky trial and error on real roads.

The trap

Bootstrapping with max over a′ of Q(s′, a′) queries actions the data never covered — and their values inflate.

3

CONTRIBUTIONS

What this work adds

01

Velocity-proportional reward

Rewards speed, punishes collisions hard — pushing the agent toward efficient overtaking.

02

IQL on mixed-quality data

Learns one lane-change policy from Final, Medium, Random and Human-like data together.

03

Expert-tier score

89.03 normalized score on AD4RL cut-in, measured with the benchmark’s original reward.

4

METHOD

IQL: stay inside the data

1 · VALUE

Expectile regression

V(s) ← upper expectile (τ) of Q(s, a) over logged actions

2 · CRITIC

SARSA-style target

Q(s, a) ← r + γ V(s′)

3 · ACTOR

Advantage-weighted cloning

π ← logged actions weighted by exp(β (Q − V))

No max over a′ anywhere → no out-of-distribution queries.

6

SETUP

Benchmark, data and interfaces

BENCHMARK

AD4RL · cut-in scenario

  • FLOW framework on the SUMO simulator
  • Ego must overtake slower vehicles
  • Normalized score: Random = 0, Final = 100

DATA

4 quality levels, merged

  • Final: fully trained DDPG
  • Medium: early-stopped DDPG
  • Random: untrained policy
  • Human-like: IDM + LC2013 models
  • 80 / 20 episode split for validation
  • Trained for 100,000 steps (50 epochs)

INTERFACES

1-D state in, 2-D action out

  • State: ego speed; gaps and relative speeds to front, rear and adjacent vehicles
  • Action: longitudinal acceleration + lane change
  • Lane change discretized to {−1, 0, +1} before execution, as Flow’s interface requires

7

REWARD

Velocity-proportional reward

OURS · USED FOR TRAINING

R = 5 · v if no collision
R = −1000 if collision

Speed is rewarded linearly; the heavy collision penalty keeps aggressive overtaking in check.

AD4RL ORIGINAL · USED FOR EVALUATION

Weighted sum of five terms

  • Speed relative to the desired speed
  • Benefit of a lane change
  • Safe distance to the leading vehicle
  • Safe distance to the follower during a lane change
  • Accident penalty (largest weight)

Logged data
rewards relabeled with 5v / −1000

IQL training
lane-change policy

Evaluation in Flow
AD4RL reward, normalized to Random 0 / Final 100

Trained on our reward · scored on AD4RL’s

8

Expert tier (81–100)

62.50

Mean
89.03

R0

R1

R2

R3

R4

R5

R6

R7

R8

R9

RESULTS

Expert-tier driving from mixed data

89.03

± 9.78 · normalized score over 10 runs

27.70 m/s

mean speed (± 2.21)

1,845 veh/h

mean outflow (± 125)

Round 3 (62.50, gray) dips well below the expert tier, likely a high-speed cut-in where safety and speed incentives conflict; trajectory-level analysis is future work.

9

ANALYSIS

What kept training stable

Max weight = 100

Clips advantage weights, capping the gradient into the policy.

β = 4.0

Imitates strong overtakes without chasing only the extremes or copying failures.

Batch = 512

Mixes all data qualities in each update, smoothing the critic’s Bellman error.

Actor and critic loss over 50 epochs: critic loss falls from about 23 to about 1; actor loss falls from about 0.6 and levels off near 0.3 with small oscillations

Fig. 1 · Critic loss (red) spikes in the first epochs while adapting to the new reward scale, then settles; actor loss (blue) levels off near 0.3 with mild oscillation.

10

LIMITATIONS

What this study does not yet show

LimitationNext step
No baselines or reward ablationCompare with CQL, BCQ, TD3+BC; train with AD4RL’s original reward to isolate our reward’s effect
Safety only via aggregate scoreReport collision rate, minimum time-to-collision, near-miss frequency
One scenario, 10 runsAll AD4RL scenarios; more runs with confidence intervals
Round 3 explained qualitativelyVisualize and analyze the failing rollout’s trajectory
Simulation onlySim-to-real: control latency and sensor noise
No comfort termJerk limits in the action space or reward

11

CONCLUSION

Takeaways

01

IQL learns a high-speed overtaking policy from mixed-quality offline data — no online trial and error.

02

89.03 normalized score: expert tier on AD4RL cut-in, scored with the benchmark’s own reward.

03

Weight clipping, β tuning and large batches keep training stable under a large reward scale.

NEXT

Baselines and reward ablation · direct safety metrics · more scenarios · sim-to-real

12

REFERENCES

References

[1] H. Chu et al., “Decision making for AVs: mixed curriculum RL and safety intervention,” Transp. Res. C, 2025.

[2] I. Kostrikov, A. Nair, S. Levine, “Offline RL with implicit Q-learning,” ICLR, 2022.

[3] D. Lee, C. Eom, M. Kwon, “AD4RL: Autonomous driving benchmarks for offline RL,” ICRA, 2024.

[4] E. Leurent, “An environment for autonomous driving decision-making,” highway-env, GitHub, 2018.

[5] L. Zhang et al., “Copilot4D: Unsupervised world models for AD via discrete diffusion,” ICLR, 2024.

[6] J. Philion, X. B. Peng, S. Fidler, “Trajeglish: Traffic modeling as next-token prediction,” ICLR, 2024.

[7] D. Coelho, M. Oliveira, V. Santos, “RLfOLD: RL from online demonstrations in urban AD,” AAAI, 2024.

[8] R. Li et al., “ITPNet: Towards instantaneous trajectory prediction for AD,” KDD, 2024.

[9] P. S. Chib et al., “MS-TIP: Imputation aware pedestrian trajectory prediction,” ICML, 2024.

[10] R. Yang et al., “Towards robust offline RL under diverse data corruption,” ICLR, 2024.

[11] Q. Di et al., “Pessimistic nonlinear least-squares value iteration for offline RL,” ICLR, 2024.

[12] F.-M. Luo et al., “Reward-consistent dynamics models are strongly generalizable for offline RL,” ICLR, 2024.

13