Have you ever encountered this situation: an intelligent driving system makes an improper detour at a construction site. Adding relevant data and adjusting the model does improve the vehicle's performance on that specific stretch of road.

However, when faced with a different construction site—featuring distinct temporary markings, varying road widths, and unique surrounding vehicle behaviors—the system may still exhibit unreasonable deceleration, hesitation, or improper detouring.

With the continuous accumulation of training data and model iterations, why do some driving issues persist?

Actually, the problem may not necessarily stem from insufficient data volume.

For an intelligent driving system, collecting data from a specific scenario doesn't guarantee that the training data has adequately covered related issues. The model achieving better results on existing data doesn't necessarily mean it has learned to handle similar scenarios stably under different conditions.

01 Why, Despite Increasing Data, Are Complex Scenarios Still Not Handled Well?

There are significant differences in the distribution of driving data on real roads.

In training data, scenarios like regular car-following and lane-keeping may dominate, while the coverage of scenarios such as construction detours, sudden crossings by non-motorized vehicles, and multi-vehicle interactions depends on collection routes, data screening, and training set composition.

Road structures, the behaviors of traffic participants, and their combinations in these scenarios also vary widely.

If newly added data primarily comes from common driving scenarios, even though the training set size expands, the model's experience in key scenarios may not increase proportionally.

Image Source: Internet

Even if the data includes a large number of construction sections, if road widths, marking conditions, and vehicle interaction patterns are highly similar, it's difficult to cover the diverse changes encountered on real roads.

More complex issues arise when multiple factors appear simultaneously.

For example, in a construction section, there may be both original lane markings and temporary markings and cones. The system needs to combine road structures, obstacle positions, and the movement states of surrounding vehicles to determine the current passable area and plan an appropriate driving trajectory. If an adjacent vehicle suddenly cuts in, the originally feasible passing path may need to be reassessed.

Just because the model has seen cones doesn't mean it can correctly understand construction areas; seeing vehicle cut-ins doesn't mean it can handle situations where construction detours and vehicle cut-ins occur simultaneously.

To measure the value of training data, one cannot solely rely on cumulative mileage, collection duration, or sample quantity. It's also necessary to analyze scenario distribution, combinations of key factors, and whether different driving behaviors are adequately covered.

Whether newly added data can address the model's original capability shortcomings is more noteworthy than simply increasing the quantity.

02 Even If Failure Data Is Collected, Why Are Issues Still Not Resolved?

When issues arise in an intelligent driving system, it's necessary to trace back the data, analyze which link the problem occurred in, and then decide how to improve the training.

Taking unreasonable deceleration in a construction section as an example, the issue may involve inaccurate identification of temporary road boundaries, or the system may have identified the passable area but not fully considered the movement trends of adjacent vehicles.

Another possibility is that the system can generate a feasible trajectory, but the speed changes or steering amplitudes in the trajectory are not reasonable enough.

Although these issues all manifest as unreasonable vehicle deceleration, the underlying causes may be completely different.

If the key factors affecting driving behavior are not identified, and similar driving records are merely continuously supplemented, the model may not necessarily improve effectively.

The quality and organization of data also affect training effectiveness.

For multi-sensor training schemes, the accuracy of time synchronization and spatial alignment of data is crucial. When using supervised learning, the reliability of labels or reference trajectories is also noteworthy.

In addition, the proportion, sampling method, and training objectives of difficult samples in the training data also influence whether the model can learn effective patterns from these samples.

Image Source: Internet

For imitation learning that relies on demonstration data, the model generally needs to learn from reference driving behaviors.

If the demonstrations are inconsistent or there are multiple reasonable driving choices in similar scenarios, the training process needs to properly handle these differences. The model learning to reproduce behaviors in the training samples doesn't mean it has mastered safe decision-making patterns in all scenarios.

Furthermore, not all issues can be resolved by supplementing data.

If the problem stems from the model structure, training objectives, planning constraints, or vehicle execution capabilities, simply increasing the sample quantity may not address the root cause.

The value of failure data lies in helping researchers locate issues and providing targeted improvement bases for subsequent training.

03 The Issue Is Resolved, But Why Does It Reappear in a Similar Scenario?

Even if the training process improves a certain type of driving behavior, the model still needs to face the challenge of transferring from known scenarios to new ones.

For instance, the model may be able to pass through one construction section, but when switching to a construction section with narrower roads, blurrier temporary markings, and denser adjacent vehicles, the original driving strategy may no longer be applicable.

The model needs to re-judge based on the current environment rather than simply reproducing previously learned behaviors. This involves the model's generalization ability.

Training data cannot exhaust all road conditions; the model needs to learn patterns from limited samples that can be applied to different scenarios.

Another easily overlooked issue is that the states the model encounters during actual driving may gradually deviate from the common states in the training data.

Taking imitation learning as an example, during training, the model learns from reference driving behaviors and their corresponding environmental data.

During actual operation, the model's output actions or trajectories affect the vehicle's subsequent movement states.

Once the vehicle deviates from the originally expected driving trajectory, subsequent input and traffic interaction conditions may change, and the states the model faces may gradually deviate from the common distribution in the training data.

Image Source: Internet

For example, if the vehicle chooses an inappropriate driving trajectory in a construction section, it may get closer to cones, reducing the space for subsequent adjustments.

At this point, the vehicle's environment has changed, and subsequent decisions may also be affected.

If the model lacks experience in handling these deviated states, early judgment errors may gradually amplify in continuous decision-making. The difference between the state distribution encountered during training and that encountered during actual operation is a key issue to be addressed in imitation learning and closed-loop driving.

However, the specific mechanisms of different intelligent driving architectures vary, and not all recurring issues can be attributed to the same cause. To determine whether the model has truly resolved a certain issue, verification at different levels is required.

Offline replay can check the model's processing results of existing data; closed-loop simulation can observe the impact of the model's decisions on subsequent states and traffic interactions, but the results are still limited by the authenticity of the simulation environment; road testing further examines the performance in real environments but is also constrained by test routes, traffic conditions, and scenario coverage.

Only by complementing these methods can a more comprehensive judgment be made on whether the model improvements are effective.

04 What Should We Really Focus on in Intelligent Driving Training?

When a certain type of driving issue recurs, the focus should not be on simply judging how much more data is needed but on trying to identify what problems exist between data coverage, training mechanisms, and the model's actual performance.

If key scenario coverage is insufficient, targeted data supplementation is needed; if there are issues with sample quality or labels, data processing needs to be improved; if the model has not learned effective driving patterns, the training objectives and methods need to be checked; if offline results have improved but closed-loop performance has not increased proportionally, further analysis of issues in continuous decision-making and traffic interactions is required.

After each optimization, it is also necessary to check whether the original issue has been improved and whether new issues have arisen in similar and nearby scenarios.

Achieving better results in a single fixed case is not sufficient to prove that the model has the ability to handle similar scenarios stably. At this point, it is also necessary to observe the performance in other key driving tasks to avoid new performance degradations caused by local optimizations.

Image Source: Internet

For intelligent driving research and development, the scale of training data remains an important foundation, but data volume cannot be directly equated with driving capabilities.

Data distribution affects which situations the model has encountered and the role different scenarios play in training; the training process affects what the model can learn; closed-loop testing and road verification help researchers determine whether these capabilities can function in actual driving.

What intelligent driving truly needs is not just continuously increasing training data but ensuring that each issue discovery can drive effective improvements and confirming the validity of these improvements through more comprehensive verification.

When the model consistently performs stably in representative similar and nearby scenarios, data accumulation truly transforms into driving capabilities. #AutonomousDriving #LargeModelTraining