SiMDex

Mining Similar Egocentric Videos for
Cross-Embodiment Dexterous Manipulation

Nie Lin¹ Takehiko Ohkawa¹ Sijin Chen³ Ruoshi Wen² Zhuohang Li⁴

Liqun Huang² Zhengming Zhu² Yiming Bao⁵ Yunfei Li² Minjie Cai¹

Xiao Ma² Wei Xu² Yoichi Sato¹

¹The University of Tokyo ²ByteDance Seed ³The University of Hong Kong ⁴Shanghai Jiao Tong University ⁵Tsinghua University

Paper Demo Code 🤗 Hugging Face

▼

⚡ TL;DRLike a recommender system, we recommend the human demos most relevant to the robot's task.

Demo

Abstract

Recent years have witnessed an explosive trend of scaling ego-centric human videos for robot manipulation, yet it remains unclear which data actually benefits dexterous manipulation. We present SiMDex, a Similarity-based data Mining framework that casts human data selection for VLA post-training in Dexterous manipulation as a recommendation problem. For each robot demonstration, SiMDex employs a three-layer recall–ranking–re-ranking pipeline to extract task-relevant subsets from a pool of ~32M egocentric human samples, operating in a morphology-agnostic action space that requires no changes to VLA architecture or training. Against a strong baseline trained with an equal amount of randomly sampled human data, SiMDex uses only ~1.49M mined samples (<5% of the pool) yet improves the overall success rate from 47.7% to 61.1%—showing that selective curation outperforms indiscriminate data mixing.

Method

SiMDex mines the human demonstrations most similar to each robot demonstration — from a pool of tens of millions — through a three-stage cascade of increasing precision and cost.

▶Recall — cast a wide net, cheaply. We first pull broad candidates by language and hand-pose similarity, keeping the stage lightweight enough to scale to tens of millions of samples.
▶Ranking — align the actual motion. We then score candidates by fine-grained action similarity — wrist trajectory, wrist rotation, and finger motion — so the retrieved demos move the way the target task does. This stage's output is the mined subset used to train the VLA.
▶Re-ranking — verify with vision. Finally, optical flow provides an embodiment-agnostic check, filtering out kinematic look-alikes whose real motion doesn't match — human hand or robot arm, only the motion has to agree.

Mining Human Demonstrations

Task

Sub-task

Robot Anchor

Human Target

Robot Anchor

Human Target

▼

Evaluation

We design three real-world dexterous tasks as our evaluation suite, each targeting a different capability — tool use, fine-grained finger dexterity, and multi-object generalization. Every task enforces a strict sequence: fail one stage and the rest is blocked.

▶Task I · Drill — Tool use & multi-step coordination. The robot grasps the drill, aligns and assembles it onto a fixture, then presses the trigger — three dependent steps that must all land.
▶Task II · Flick Wheel — Fine-grained finger dexterity. The robot grasps the assembly, twists the wheel with two fingers, and flicks it off with one — our hardest test of precise fingertip control.
▶Task III · Pick & Place — Generalization across shapes. Four objects of diverse geometries are scattered on the table; the robot picks and places each at its target — testing how well skills transfer across object shapes.

Rather than scoring only final success, we grade each stage separately — a fine-grained breakdown that reveals where an improvement comes from and gives a more three-dimensional picture of a robot's dexterity.

Tools Introduction

All tasks are performed on physical objects. The workspace includes interactive tools (an electric drill, a wrench), modular assembly parts (bolted bars, wheels, cubes, and plastic screws), and baskets for pick-and-place targets. Their varied geometries and tight tolerances demand the kind of fine-grained, contact-rich manipulation that low-DoF grippers cannot handle — and that makes them a faithful testbed for dexterous skill.

Teleoperation

Task

View

Experiments

Main Results

Task

Overall Success Rate

GR-Dexter

0.0%

SiMDex (Ours)

0.0%

+13.4%

Ablation Study

Overall SR · 1× Data Scale

GR-Dexter

0.0%

SiMDex (Ours)

0.0%

Rollout

Side-by-side comparison of GR-Dexter and SiMDex executing the same task. Choose a task and camera view to explore.

Task

Model

View

GR-Dexter

SiMDex Ours

View

Takeaways

▶ Selection beats scale. For dexterous VLA post-training, which human data you use matters more than how much — a well-chosen <5% of the pool beats an equal amount sampled at random, lifting overall success from 47.7% to 61.1%, with the biggest gains where robot data is scarce.
▶ Collection pays off twice. By re-mining the same egocentric corpus already used for pre-training — now task-aware — SiMDex extracts a second round of value from data you've already gathered, reducing reliance on costly robot demonstrations without collecting anything new.
▶ Relevance is atomic, not semantic. Retrieved human clips need not perform the same task as the robot — they only need to match at the atomic action level of a training window (~1s for a VLA). This is why a wildly heterogeneous human pool becomes an asset rather than noise: the same twisting or grasping motion recurs across cooking, assembly, and play, even when the activities share nothing. Human video is valuable not for doing the same thing, but for sharing transferable action atoms.
▶ For dexterity, grade the stages, not just the outcome. Success rate alone collapses a rich, multi-step skill into a single bit. Scoring each sub-stage separately — grasp, align, twist, place — reveals where a policy succeeds or breaks down, exposes gains that a pass/fail metric would hide, and gives a far more faithful measure of dexterous ability.

Still a long road ahead. SiMDex is an early step toward robots that acquire new skills by retrieving human experience — not a finished solution. Much remains to be done.

Citation

If your research involves human data selection, cross-embodiment learning, or dexterous manipulation, please consider citing SiMDex:

@article{lin2026simdex,
  title={SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation},
  author={Lin, Nie and Ohkawa, Takehiko and Chen, Sijin and Wen, Ruoshi and Li, Zhuohang and Huang, Liqun and Zhu, Zhengming and Bao, Yiming and Li, Yunfei and Cai, Minjie and Ma, Xiao and Xu, Wei and Sato, Yoichi},
  journal={arXiv preprint arXiv:2608.04196},
  year={2026}
}

The core insight of SiMDex — mining the most similar human demonstrations for learning — was first introduced in SiMHand (ICLR 2025). If your work touches on hand pose estimation, learning from human video, or similarity-based data curation, please also consider citing:

@inproceedings{lin2025simhand,
  title={Simhand: Mining similar hands for large-scale 3d hand pose pre-training},
  author={Lin, Nie and Ohkawa, Takehiko and Huang, Yifei and Zhang, Mingfang and Cai, Minjie and Li, Ming and Furuta, Ryosuke and Sato, Yoichi},
  booktitle={International Conference on Learning Representations},
  volume={2025},
  pages={35662--35680},
  year={2025}
}

Acknowledgement

We thank the Bytedance Seed team for their generous support of the dexterous manipulation platform used in this project and for validating my key ideas. We also thank all co-authors for their insightful discussions throughout the development of SiMDex.