Why Autonomous Perception Needs More Than Flat Object Lists

A Tesla following a horse carriage is more than a funny edge case.

When the visualization struggles to decide what it is seeing, it exposes a deeper ADAS problem: many perception systems are still trained as if the world were a flat list of object categories.

This is the problem addressed in my paper 👉 🔗 “The Role of Semantic Models in Constraining Pattern Recognition in Modern AI Systems” 👈 (Intelligent Environments 2025: Combined Workshop Proceedings).

The real world is not a flat label space

Most ADAS perception stacks are trained on relatively flat object category lists: car, pedestrian, cyclist, truck, traffic light, sign. But the real world is combinatorial, contextual, and full of rare, hybrid, and previously unseen situations.

A horse carriage is not just an unknown vehicle. It combines biological motion, vehicle-like behaviour, vulnerable-road-user dynamics, non-standard geometry, unusual speed profiles, and culturally rare road context. A flat classifier has no safe way to represent that composition if it has not been trained for it directly.

This is why so many autonomy incidents can be traced back to the same limitation: the system was trained to classify what it already knew, not to reason about what it had never seen before.

Unknowns need a fallback representation

SOTIF recognises unknowns, but mostly from a process and validation perspective. What has been missing is a direct mitigation of the categorisation problem itself: what should the system do when the observed object does not fit any known class safely?

Our proposal is a cascaded fallback system. Instead of forcing every perception result into a fixed class label, the system should be able to abstract unknowns upward into a semantic taxonomy: if it cannot identify the exact object, it should still retain safety-relevant properties such as motion, physical presence, vulnerability, controllability, and interaction risk.

The point is not to guess better. The point is to fail more safely. If the system cannot say “horse carriage”, it should still be able to say: unknown dynamic road participant, physically present, potentially vulnerable, behaviour uncertain, keep distance, reduce speed, escalate caution.

Safety fails before the decision layer

Safety does not fail only because a system makes the wrong decision. It also fails because the underlying representation of the world is too rigid for the complexity of reality. If the perception layer cannot express uncertainty and semantic abstraction, the planning layer receives a false sense of certainty.

That is why autonomy needs more than object detection. It needs semantic models that constrain pattern recognition, preserve uncertainty, and create a safety net when the system encounters novelty.

The lesson is simple: autonomy cannot be safe if perception is only trained to recognise the past. It must also know how to behave when the future presents something it has never seen before.

Read the paper: 🔗 The Role of Semantic Models in Constraining Pattern Recognition in Modern AI Systems