SiMDex

Evaluation

We design three real-world dexterous tasks as our evaluation suite, each targeting a different capability — tool use, fine-grained finger dexterity, and precise grasping of multiple objects with diverse shapes. Stages are sequential except in Pick & Place, whose objects are scored independently.

• ▶Task I · Drill — Tool use & multi-step coordination. The robot grasps the drill, aligns and assembles it onto a fixture, then presses the trigger — three dependent steps that must all land.
• ▶Task II · Flick Wheel — Fine-grained finger dexterity. The robot grasps the assembly, twists the wheel with two fingers, and flicks it off with one — our hardest test of precise fingertip control.
• ▶Task III · Pick & Place — Precise grasping across shapes. Four objects of diverse geometries are scattered on the table; the robot picks and places each at its target — testing precise grasps across object shapes.

Rather than scoring only final success, we grade each stage separately — a fine-grained breakdown that reveals where an improvement comes from and gives a more three-dimensional picture of a robot's dexterity.

Tools Introduction

All tasks are performed on physical objects.

• ▶Interactive tools. An electric drill and a wrench.
• ▶Modular assembly parts. Bolted bars, wheels, cubes, and plastic screws.
• ▶Placement targets. Baskets for pick-and-place.
• ▶Why these objects. Their varied geometries and tight tolerances demand the kind of fine-grained, contact-rich manipulation that low-DoF grippers cannot handle — and that makes them a faithful testbed for dexterous skill.

Teleoperation

Experiments

Main Results

Ablation Study

Rollout

Side-by-side comparison of GR-Dexter and SiMDex executing the same task. Choose a task and camera view to explore.

Takeaways

• ▶ Selection beats scale. For dexterous VLA post-training, which human data you use matters more than how much — a well-chosen <5% of the pool beats an equal amount sampled at random, lifting overall success from 47.7% to 61.7%. SiMDex leads the baseline at every robot-data budget we test (0.25× to 2×), with the largest gain (+17.3) at 0.5×.
• ▶ Collection pays off twice. By re-mining the same egocentric corpus already used for pre-training — now task-aware — SiMDex extracts a second round of value from data you've already gathered, without collecting anything new. Selective human-data mining is a practical complement to the costly collection of more robot demonstrations.
• ▶ Relevance is atomic, not semantic. Retrieved human clips need not perform the same task as the robot — they only need to match at the atomic action level of a training window (~1s for a VLA). This is why a wildly heterogeneous human pool becomes an asset rather than noise: the same twisting or grasping motion recurs across cooking, assembly, and play, even when the activities share nothing. Human video is valuable not for doing the same thing, but for sharing transferable action atoms.
• ▶ For dexterity, grade the stages, not just the outcome. Binary success alone collapses a rich, multi-step skill into a single bit. Scoring each sub-stage separately — grasp, align, twist, place — reveals where a policy succeeds or breaks down, exposes gains that a pass/fail metric would hide, and gives a far more faithful measure of dexterous ability.

Still a long road ahead. SiMDex is an early step toward robots that acquire new skills by retrieving human experience — not a finished solution. Much remains to be done.

Citation

If your research involves human data selection, cross-embodiment learning, or dexterous manipulation, please consider citing SiMDex:

@article{lin2026simdex,
  title={SiMDex: Mining Similar Egocentric Videos for Cross-Embodiment Dexterous Manipulation},
  author={Lin, Nie and Ohkawa, Takehiko and Chen, Sijin and Wen, Ruoshi and Li, Zhuohang and Huang, Liqun and Zhu, Zhengming and Bao, Yiming and Li, Yunfei and Cai, Minjie and Ma, Xiao and Xu, Wei and Sato, Yoichi},
  journal={arXiv preprint arXiv:2608.04196},
  year={2026}
}

The core insight of SiMDex — mining the most similar human demonstrations for learning — was first introduced in SiMHand (ICLR 2025). If your work touches on hand pose estimation, learning from human video, or similarity-based data curation, please also consider citing:

@inproceedings{lin2025simhand,
  title={Simhand: Mining similar hands for large-scale 3d hand pose pre-training},
  author={Lin, Nie and Ohkawa, Takehiko and Huang, Yifei and Zhang, Mingfang and Cai, Minjie and Li, Ming and Furuta, Ryosuke and Sato, Yoichi},
  booktitle={International Conference on Learning Representations},
  volume={2025},
  pages={35662--35680},
  year={2025}
}

Acknowledgement

We thank the Bytedance Seed team for their generous support of the dexterous manipulation platform used in this project and for validating my key ideas. We also thank all co-authors for their insightful discussions throughout the development of SiMDex.