Abstract
We present a system for learning generalizable hand-object tracking controllers purely from synthetic data, without requiring any human demonstrations. Our approach makes two key contributions: (1) HOP, a Hand-Object Planner, which can synthesize diverse hand-object trajectories; and (2) HOT, a Hand-Object Tracker that bridges synthetic-to-physical transfer through reinforcement learning and interaction imitation learning, delivering a generalizable controller conditioned on target hand-object states. Our method extends to diverse object shapes and hand morphologies. Through extensive evaluations, we show that our approach enables dexterous hands to track challenging, long-horizon sequences including object re-arrangement and agile in-hand reorientation. These results represent a significant step toward scalable foundation controllers for manipulation that can learn entirely from synthetic data, breaking the data bottleneck that has long constrained progress in dexterous manipulation.
Abstract (translated)
我们提出了一种系统，该系统能够仅通过合成数据学习通用的手部-物体跟踪控制器，无需任何人类演示。我们的方法做出了两个关键贡献：（1）HOP（手部-物体规划器），可以生成多样化的手部-物体轨迹；（2）HOT（手部-物体跟踪器），它通过强化学习和交互模仿学习实现从合成数据到物理世界的转换，提供一个根据目标手部-物体状态条件的通用控制器。我们的方法适用于各种形状的对象和不同的手形结构。经过广泛的评估，我们展示了该方法使灵巧的手能够追踪包括对象重新排列和灵巧的手内再定向在内的复杂、长时间序列的能力。这些结果代表了向可以完全从合成数据中学习的大规模基础控制方案发展的重大一步，打破了长期以来限制灵巧操作进步的数据瓶颈。
URL
https://arxiv.org/abs/2512.19583
