2026 Preprint · arXiv coming soon

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation

Bo Liang1, Chen Gong1, Wei Gao2, Chenren Xu1,3,*
1School of Computer Science, Peking University   2University of Pittsburgh
3Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)
*Corresponding author
Paper Coming soon Code

Wave2Body overview

Wave2Body overview comparing direct coordinate regression with radar-to-body token translation

Wave2Body replaces direct coordinate regression with a reusable interface: a self-supervised mmWave tokenizer produces continuous radar tokens, a lightweight translator maps them into discrete body tokens, and a separate localization head recovers global position.

60-second overview

Wave2Body in motion

See how sparse radar reflections become structured body tokens, why decoupled training matters, and where the resulting gains come from in one concise visual tour.

Abstract

Millimeter-wave (mmWave) radar enables privacy-friendly human sensing, but its sparse point clouds are physical measurements of view-dependent electromagnetic reflections and only indirectly characterize body articulation. Recovering a complete 3D pose from such partial, geometry-dependent observations is therefore under-constrained. Existing methods directly regress joint coordinates from paired radar-pose data, relying on the same limited paired supervision to learn radar perception, human-body structure, and their alignment. This coupling can encourage dataset-specific shortcuts under ambiguous radar observations. We propose Wave2Body, a radar-to-body token translation framework that decouples these learning targets using a self-supervised mmWave tokenizer, a pretrained compositional body tokenizer that defines the output space, and a lightweight translator between them. Experiments on M4Human and mmBody show that Wave2Body achieves stronger cross-domain generalization than previous methods while incurring much lower computational costs for training and inference.

Why Rethink Coordinate Regression?

Radar returns depend on body geometry, orientation, motion, signal propagation, and sensor configuration. They are sparse, view-dependent reflections rather than dense observations of anatomical joints, so multiple plausible body configurations can agree with the same partial measurement.

Entangled learning targets. A conventional supervised regressor asks the same limited paired radar-pose dataset to learn sensor representations, valid human-body structure, and cross-modal alignment at once. Under ambiguous observations, this coupling can reward dataset-specific shortcuts.

A structured prediction interface. Wave2Body independently pretrains a radar-token space and a compositional body-token space, then restricts paired supervision to learning the translation and global localization.

Direct coordinate regression Radar-to-body token translation

Method

Wave2Body assigns each data source a focused role: unlabeled radar for sensor representations, pose-only motion data for human-body priors, and paired radar-pose samples for cross-modal alignment and global localization.

Three-stage Wave2Body training pipeline
Three-stage training separates source-token pretraining, target-token pretraining, and paired radar-to-body translation.

Stage 1

Self-supervised mmWave Tokenizer

A Point-MAE-style encoder learns continuous radar tokens by reconstructing masked local point-cloud patches without pose labels.

Unlabeled radar

Stage 2

Compositional Body Tokenizer

A part-structured VQ-VAE learns 24 discrete tokens across five kinematic regions from pelvis-centered pose-only data.

AMASS + radar poses

Stage 3

Translation and Localization

Frozen tokenizers expose reusable interfaces; a query-based translator predicts body embeddings while a separate head recovers the pelvis.

Paired radar-pose data

Results

Cross-Domain Generalization

Under a matched radar-frame absolute-pose protocol, Wave2Body improves the micro-average against the strongest competitor on both M4Human distribution shifts and on mmBody. Its largest gains appear under subject shift and dynamic held-out motions.

M4Human Cross-Subject

113.53102.53 mm

9.7% lower MPJPE

M4Human Cross-Action

119.81115.26 mm

3.8% lower MPJPE

mmBody Official Test

139.69128.17 mm

8.2% lower MPJPE

Best matched competitor → Wave2Body. Lower is better.

Cross-Subject per-joint MPJPE comparison between Wave2Body and WiCompass
Cross-Subject: Wave2Body improves every joint, with the largest gains around the lower body and global placement.
Cross-Action per-joint MPJPE comparison between Wave2Body and WiCompass
Cross-Action: Wave2Body improves 19 of 22 joints, most clearly around the pelvis, hips, spine, neck, and collar.

Qualitative Results

Predictions remain human-like across the error spectrum. In the hardest cases, errors mainly appear as global root shifts or ambiguous distal-limb articulation under sparse radar coverage.

Wave2Body qualitative predictions on M4Human and mmBody from lower to higher MPJPE
Gray points denote radar returns, red skeletons denote ground truth, and blue skeletons denote Wave2Body predictions.
Where the gains concentrate. Wave2Body is strongest under subject shift and dynamic held-out motions. On Cross-Action In-Place actions, it is 3.6% worse than WiCompass, reflecting a trade-off between the robustness of a discrete body prior and fine-grained coordinate precision.

Training and Inference Efficiency

Independent token interfaces substantially reduce both cold-start training compute and high-throughput inference cost against the matched WiCompass baseline.

31.00×fewer training FLOPs
89.81×fewer inference FLOPs
8.07×higher throughput
92.6%lower peak memory
Cold-start training computation comparison between Wave2Body and WiCompass
Cold-start training computation on M4Human. The two paired-data stages account for 55.2% of Wave2Body's total.

Token Alignment and Ablations

Paired target-token recall exceeds identity- and action-matched controls, supporting sample-specific translation rather than a frequent-pose shortcut. Removing either pretrained token interface consistently hurts pose accuracy.

Target body-token recall for paired samples and two controls
On M4Human Cross-Subject, paired Top-1/3/5 recall reaches 22.63/53.16/69.77%, above both deterministic controls.

Sample-Specific Translation

Token-alignment mismatch correlates with root-aligned pose error, connecting recovery in the discrete target space to downstream articulation accuracy.

Top-3 miss rate vs. errorρ = 0.742
Top-5 miss rate vs. errorρ = 0.774
Wave2Body component ablations across M4Human Cross-Subject and Cross-Action groups
Removing the body tokenizer increases MPJPE by 3.3-27.5%; replacing the pretrained mmWave tokenizer increases it by 3.7-30.7% across action groups.

BibTeX

@misc{liang2026wave2body,
  title  = {Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation},
  author = {Liang, Bo and Gong, Chen and Gao, Wei and Xu, Chenren},
  year   = {2026},
  note   = {Preprint}
}