An intelligent driving model might achieve flawless metrics in offline tests. But why does it still make inaccurate judgments or unreasonable decisions at complex intersections?

In recent years, end-to-end intelligent driving systems have been continuously scaling up model size, training data, and computational resources. However, these enhancements don't automatically lead to better driving performance.

Whether a model can comprehend complex traffic dynamics and handle scenarios not fully encompassed by training data depends on what it has learned and how the R&D team validates its enhanced capabilities.

For the intelligent driving industry, the emphasis should not solely be on constructing larger models but also on understanding how model size, data investment, and training methods can genuinely translate into verifiable driving abilities on real roads.

01 Why Doesn't a Larger Model Necessarily Equate to Stronger Capabilities?

An increase in model parameters implies that the model has a greater capacity to learn more intricate features and relationships.

However, a larger model capacity doesn't ensure superior actual learning outcomes.

Factors such as the sufficiency of training data, the rationality of training objectives, and the model's ability to generalize learned patterns to new scenarios all influence its ultimate performance.

When following a vehicle on common roads, the traffic dynamics around the vehicle are relatively stable, enabling the model to learn vehicle movement patterns from numerous similar scenarios.

Image Source: Internet

At complex intersections, however, other vehicles may suddenly cut in, pedestrians may change direction, and multiple traffic participants may simultaneously impact the vehicle's decisions.

The system must consider not only the current situation but also the potential consequences of different actions.

If the training data mainly consists of common driving scenarios with insufficient coverage of complex interactions, even continuous growth in model size cannot guarantee proportional improvement in decision-making abilities in complex scenarios.

The model may perform better in familiar scenarios but struggle in situations where the training distribution is inadequate or interaction dynamics shift.

This is where the relationship between model size and driving capability can easily be misconstrued.

While a larger model capacity provides the conditions for learning complex driving patterns, it doesn't guarantee that the model has mastered them, nor that it will achieve proportional improvements across all tasks and scenarios.

In other words, while model size is significant, it is not an independent metric for gauging intelligent driving capabilities. The crucial questions are whether the model can handle complex traffic dynamics and make reasonable decisions in uncertain scenarios.

02 Why Doesn't More Data Necessarily Translate to Fewer Issues?

Data is a vital foundation for intelligent driving models, but the total volume of data doesn't directly represent the amount of effective training information.

Suppose the training data contains a large number of highway cruising and regular following segments. In that case, these data aid the model in learning everyday driving patterns but may not adequately cover scenarios such as construction detours, complex intersection interactions, or unusual obstacles.

If the continuously increasing data is primarily similar, the model's performance in common scenarios may continue to improve, but gains in under-covered scenarios may be limited.

In 2025, a study on end-to-end autonomous driving published at CVPR Workshops compared training data scales ranging from 16 to 8,192 hours and evaluated the model's open-loop metrics and closed-loop simulation performance.

Image Source: Internet

The study found that under the adopted model architecture and simulation settings, open-loop metrics improved with increasing data scale, but closed-loop driving performance plateaued relatively early.

This indicates that, under specific experimental conditions, enhancements in offline metrics don't necessarily translate into proportional gains in closed-loop performance.

In 2026, another study on end-to-end driving via imitation learning, published at ICRA, analyzed approximately 4 million driving demonstration segments covering 23 scenario categories and totaling over 30,000 hours.

The study similarly found that open-loop and closed-loop evaluation performances were not entirely consistent.

It also suggested that, under its test settings, appropriately increasing data for long-tail scenarios could improve the model's performance in those scenarios.

These studies don't imply that increasing data is valueless but rather that the effectiveness of data growth depends on what the data covers and how the model is evaluated.

Continuous accumulation of common scenarios versus targeted supplementation of challenging scenarios may yield different benefits; long-tail data is not inherently effective, and sample quality, scenario context, and training methods must still be considered.

Therefore, intelligent driving R&D necessitates establishing a closed loop from problem identification to data supplementation, model training, and effectiveness verification.

R&D teams must start with failure cases, analyze the relationships between problems and environmental understanding, behavior prediction, decision-making, planning, or vehicle execution, and then determine what data to supplement, what training objectives to adjust, and what tests to conduct to confirm improvements.

Rather than merely pursuing data volume growth, this iterative capability centered on specific problems better reflects the engineering value of the data system.

03 Why Doesn't Improved Offline Evaluation Necessarily Mean Enhanced Driving?

Another critical issue in model capability evaluation is whether enhancements in test metrics truly represent improved driving capabilities.

In open-loop evaluation, researchers assess model outputs on given recorded data.

For example, they compare the model's predicted vehicle trajectory with the reference trajectory in the recorded data to observe whether prediction errors decrease.

Such metrics are useful for comparing different models and helping R&D teams determine whether training is effective.

However, the reference trajectory is not the only reasonable driving solution.

In the same road environment, drivers may adopt different but equally reasonable trajectories.

Therefore, a model output closer to a particular reference trajectory only indicates improvement on that specific metric, not independently proving that the vehicle will drive more safely or smoothly.

Image Source: Internet

Real driving is a continuous interaction process.

After a vehicle takes an action, surrounding traffic participants may react, and the vehicle's own position and speed will change, altering the environment the system faces next.

A seemingly minor decision bias may gradually amplify through continuous interactions.

In other words, a reduction in open-loop prediction errors doesn't necessarily lead to improved closed-loop driving performance.

Open-loop evaluation primarily examines the model's output under given recorded conditions; closed-loop evaluation further examines whether the system can make reasonable decisions as the state continuously changes after the vehicle takes an action.

This is especially important for end-to-end intelligent driving models.

The model must not only output reasonable driving actions but also maintain stable driving behavior as its actions alter the subsequent environment.

Therefore, it is necessary to evaluate the model's performance in continuous interactions through closed-loop simulation, scenario-based testing, and real-vehicle verification, and to check whether improvements are stable and do not introduce new failure modes.

Closed-loop simulation is also influenced by factors such as the realism of the simulation environment, the behavioral models of traffic participants, and the coverage of test scenarios, and cannot fully replace real-vehicle verification.

Different testing methods have their limitations, and no single metric or test result can substitute for system-level evaluation.

For the intelligent driving industry, the focus should not be solely on comparing single metrics before and after model training but also on distinguishing which improvements merely bring predictions closer to reference trajectories and which truly enhance the vehicle's behavior on the road.

04 What Should Ultimately Be Used to Judge Intelligent Driving Models?

For intelligent driving models, model size, data volume, and offline metrics all hold value, but they represent different observational dimensions in the R&D process and cannot replace one another or be directly equated with final driving capabilities.

To determine whether a model has become stronger, at least three questions must be answered.

The first is: What specific problems has the model solved?

For example, can the system make more reasonable passing decisions at complex intersections? Can it promptly adjust vehicle behavior when other vehicles suddenly change lanes and maintain stability through continuous interactions?

Capability improvements should correspond to clear driving tasks, not merely remain at the level of parameter scale or computational growth.

Image Source: Internet

Another question is: Does the model's improvement exhibit stability?

If the model performs better only on some test samples but degrades in other scenarios, overall capability improvement cannot be simply assumed.

R&D teams must observe performance under different road conditions, traffic participant combinations, and environmental changes, and continuously analyze failure cases.

Finally, can the improvements be validated through closed-loop and real-vehicle testing?

Whether a single action is reasonable is only a local performance; it is equally important whether the system can control risks, reduce unnecessary abrupt braking and trajectory deviations during continuous interactions, and maintain expected behavior under different road conditions.

For scenarios not yet covered by testing, clear capability boundaries must also be established, rather than extrapolating performance from limited tests to all road environments.

This does not mean model size is unimportant.

Larger models, more comprehensive data, and higher training investments can all create conditions for capability improvement, but these investments must align with effective data, reasonable training objectives, and reliable evaluation methods.

05 Concluding Remarks

At the forefront of intelligent driving, we believe that as model size continues to grow, the industry should focus not merely on how much parameters have increased or training data has expanded but on which driving problems these investments have solved and which issues remain unimproved.

Models can keep growing larger, but size alone does not prove that driving capabilities have improved.

Only by aligning model, data, and training investments with specific driving problems and validating improvements through multi-level testing can we determine whether these investments have truly translated into enhanced driving capabilities. #AutonomousDriving #IntelligentDrivingModel #SelfDrivingModel