EgoCPT Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization
1 Ant Group
2 Stanford University
EgoCPT, step by step
From human motion
to robot motion.
One pouring demonstration. Four stages.
This video could not load.
Switch stages to compare the same moment. Drag the timeline to explore.
Abstract
While expert demonstrations on real robots are difficult to collect, human ego-centric videos are easy to collect and abundant on the internet. However, learning dexterous manipulation policies from human videos remains challenging—one of the most important challenges being the difficulty of reconstructing physically grounded robot trajectories due to the large embodiment gap.
In this paper, we propose EgoCPT, a novel paradigm for learning high-quality retargeted dexterous hand trajectories directly from an ego-centric video, which can then be deployed directly on real robots. The only inputs to our method are an ego-centric RGB video and a reference hand pose trajectory, which can be easily obtained through off-the-shelf software or specialized cameras during recording. Our method is particularly robust to noisy hand poses attributed to our novel contact-aware optimization framework, which iteratively optimizes the contact points between the robot hands and objects.
Across five manipulation tasks, our method achieves overall success rates of 67.3% on noisy EgoDex hand-pose reconstructions and 92.0% on high-quality TACO reconstructions collected using motion-capture hardware, outperforming multiple baselines on both benchmarks.
EgoCPT improves hand–object reconstruction
Drag the dividers to see how EgoCPT improves hand–object reconstruction. Each pair shows the same moment from the same view.
A new grasp through contact-aware optimization
Lint rolling · EgoDex
Sweeping · EgoDex
Scooping · EgoDex
Pick & place · EgoDex
Pouring · TACO
Bowl & ladle · TACO
Container manipulation · TACO
Lint rolling · TACO
Surface brushing · TACO
Object motion is the goal.
Hand motion is the reference.
Noisy reconstruction and a change in embodiment can break hand–object contact. Instead of copying these contacts, EgoCPT searches for a robot-feasible way to realize the demonstrated object motion.
Sample contacts
Search near the demonstrated hand gesture for contacts reachable by the robot.
Refine the pose
Adjust the wrists and fingers to remove penetration and close contact gaps.
Allocate forces
Find contact forces consistent with the target object motion, then refine again when needed.
Multiple tasks. A shared formulation.
Robot hand–object rollouts from egocentric demonstrations.
Visualized in Isaac Sim at 1×
From video reconstruction to motion-capture inputs
More demonstration on TACO dataset
A selection of rollouts. Playback rates are indicated in each tile.
Video
Download video3 min 5.5 sec · 1080p
BibTeX
@misc{sun2026egocpt,
title = {EgoCPT: Learning Dexterous Manipulation from a Single
Ego-Centric Video with Contact-Aware Optimization},
author = {Sun, Zhixin and Shi, Kairui and Ren, Tianao and
Yang, Ming and Cutkosky, Mark and Xu, James Jingxi},
year = {2026}
}