Tracking radar objects over time sharpens sparse-scan

Tracking radar objects over time sharpens sparse-scan classification

Tracking radar objects over time sharpens sparse-scan classification far more than swapping sequence models.

October 13, 20263 min readData Science

Multi scan radar object classification on RadarScenes

The most useful finding in this radar classification experiment is also the most humbling: the sequence model barely matters. A causal GRU, a Transformer, a state space model, and point-level self-attention all land within a 0.03 spread of macro F1 once the per-scan embeddings are fixed. The real jump comes from something far simpler, pooling 20 scans of radar returns instead of classifying each scan alone. That single change lifts macro F1 from 0.7370 to 0.8613, a gain of +0.1243. Every sophisticated temporal architecture on top of that adds roughly +0.03 at best.

This is a practical lesson for anyone building perception stacks, not just a curiosity for radar researchers. The experiment, posted by a developer working with the RadarScenes dataset, shows that a single object instance carries only about 2.9 radar points on average. That is almost nothing to classify on. Accumulating observations over a tracked object's history gives you both density and temporal dynamics, RCS fluctuation and micro-Doppler from limb motion are real signals, but they only emerge across scans. The takeaway is direct: before you invest in a fancier sequence encoder, check whether your input representation is actually capturing what the sensor sees. The bottleneck here is the point-level embedding, not the temporal reasoning layer. End-to-end fine-tuning of the frozen encoder made things slightly worse, which reinforces the point that the per-scan representation is where the ceiling lives.

The broader context matters too. As Germany backs Tesla's assisted driving rebrand for wider EU adoption and Waymo taps $5B debt round from Blackstone and PIMCO to scale robotaxi fleet, the pressure to make radar classification reliable in real traffic is rising fast. Every autonomous vehicle program depends on distinguishing a pedestrian from a large vehicle at a distance, in rain, at night. This experiment suggests that many teams may be over-indexing on architecture search when the simpler path, accumulating returns over a track, delivers most of the value. That is an empowering result for smaller teams: you do not need a massive research budget to see a meaningful accuracy gain. You need a persistent track_id and a sliding window.

One open question stands out. The fusion of order-aware GRU output with an order-invariant pooled embedding added only +0.0002 over the GRU alone, which is noise. That means temporal ordering, as encoded by these models, is not extracting additional discriminative signal from the current per-scan features. But that does not mean ordering is useless, it means the current representation is not letting the sequence models see what is there. The next step is to improve the point-level encoder, not to try a bigger GRU. Watch for work that revisits the per-scan embedding itself. That is where the next +0.12 will come from.

From Data Science

I built a radar object classifier on RadarScenes, extending a prior single-scan classifier to accumulate observations over a tracked object's history instead of classifying each scan in isolation.
A single RadarScenes object instance contains only about 2.9 radar points on average, very sparse. A single scan also can't capture temporal characteristics: RCS and micro-Doppler both vary continuously as an object moves. Pedestrians produce characteristic micro-Doppler from limb motion; different object classes show different RCS fluctuation patterns as aspect angle and scattering geometry change scan to scan. Accumulating observations gives both higher point density and provides temporal dynamics.

Tracking radar objects over time unlocks deeper classification insights

Hello all, I built a radar object classifier on RadarScenes, extending a prior single-scan classifier to accumulate observations over a tracked object's history instead of classifying each scan in isolation. A single RadarScenes object instance contains only about 2.9 radar points on average, very sparse. A single scan also can't capture temporal characteristics: RCS and micro-Doppler both vary continuously as an object moves. Pedestrians produce characteristic micro-Doppler from limb motion; different object classes show different RCS fluctuation patterns as aspect angle and scattering geometry change scan to scan. Accumulating observations gives both higher point density and provides temporal dynamics. Multi-scan baseline DeepReflecs encoder (Ulrich, Glaser & Timm, RadarConf 2021), PointNet style, per point shared weights, on single scans across car, large_vehicle, two_wheeler, pedestrian, pedestrian_group: 0.7370 macro F1. Using RadarScenes' persistent `track_id`, I build a causal, N=20, per track sliding-window buffer: - x_seq/y_seq: Global, odometry-corrected coordinates recentered per scan on the object centroid. Unlike x_cc/y_cc (car-frame coordinates that accumulate over time to form a trajectory). - Cross sensor buffer: whichever of the 4 sensors currently observe the track push to the same buffer. - Stride 1, causal: every new scan updates the buffer and produces a prediction. No future context, real time streaming compatible. - Each scan is encoded once by a frozen per scan encoder and cached - Fusion concatenates the causal GRU's hidden state (order aware) with an order-invariant pooled embedding (all N scans' points as one set, no sequence structure) through a small trained mlp head. Results Model Macro F1 Delta Single scan 0.7370 (baseline) 20 scan point pooling 0.8613 +0.1243 Causal GRU 0.8895 +0.0282 over pooling GRU + pooled embedding (fusion) 0.8897 +0.0002 over GRU, noise Pooling alone, no sequence model, no notion of scan order at all, recovers +0.1243 macro F1. The GRU adds a real but much smaller +0.0282 on top. Fusion adds nothing measurable beyond the GRU. Ablation Llarger GRUs, a Transformer, a state space model, point level self attention, all trained on the exact same frozen per scan embeddings, land inside a 0.86 to 0.89 band, a 0.03 spread. End to end fine tuning of the frozen encoder makes things slightly worse (about -0.002 to -0.003), not better. Conclusion In this setup, the largest gain comes from giving the model more observations of the same tracked object: 20-scan point pooling improves macro F1 from 0.7370 to 0.8613 without using scan order at all. Temporal modelling then provides a further, meaningful improvement. The causal GRU reaches 0.8895, adding +0.0282 over the pooled representation. So temporal ordering clearly contributes useful information; it just accounts for a smaller portion of the overall gain than observation accumulation. With the per-scan encoder frozen, the different sequence architectures tested, suggests that the quality of the per-scan representation is the bottleneck than the particular mechanism used to aggregate the sequence. Full report, every ablation, confusion matrix, coordinate frame reasoning: https://github.com/brunopinto900/radar-ml-autonomous-driving/blob/track-accumulation/final_report.md Thank you. submitted by /u/bruno_pinto90 [link] [comments]