NVIDIA Research introduced Joints, Camera, Action. It co-generates video and 3D motion for any skeleton from text. An input image lets the motion respond to its surroundings. A few keyframes can control the motion trajectory. It turns rig motion into video and video into rig motion. The model uses a separate motion branch for arbitrary skeletons. It connects to a pretrained video model via cross-attention layers. Joint localization in video frames recovers the camera. Training used nearly 200K video-motion pairs. Evaluations show it animates diverse rigs more faithfully than video-first pipelines.
What changed
Video generation models produced motion only as pixels, not as reusable 3D motion for arbitrary rigs.
What it unlocks
Co-generating synchronized video and 3D motion for any skeleton from text, images, or keyframes.
- ~200,000 training pairs
- 2610.28632 arXiv paper ID
What you need to act on it
- a rig in rest pose
- a text prompt or input image or keyframes
Sources