SHOC: Scalable 4D Hand-Object Capture via Spatiotemporal Priors and Physics-Aware Splatting
Abstract
Recent advancements in Embodied AI and spatial computing drive a growing demand for high-fidelity 4D hand-object interaction (HOI) digital assets. However, creating these assets automatically from view-efficient multi-view setups presents a fundamental trade-off between reconstruction fidelity and system scalability. Optimization-based methods offer high metric precision but are fragile under severe occlusions due to their reliance on off-the-shelf trackers. Conversely, robust inference-based approaches lack the photometric and physical consistency required for high-end assets.
To bridge this gap, we propose SHOC, an automated 4D HOI reconstruction engine. First, we introduce the Hand-Object Transformer to aggregate multi-view spatiotemporal cues, robustly regressing parametric states and coarse geometry to bypass the convergence bottlenecks of prior works. Second, a Physics-Aware 4D Gaussian Splatting framework performs high-fidelity photometric refinement. It enforces anatomical rigidity via a tetrahedral As-Rigid-As-Possible proxy and disentangles appearance into view-independent albedo and view-dependent specularity. This prevents environmental illumination from baking into the geometry, effectively eliminating motion-induced flickering artifacts. Validated on public benchmarks and a large-scale internal dataset, our self-improving engine achieves state-of-the-art robustness and visual quality, providing a scalable pipeline for generating artifact-free 4D digital assets.
To bridge this gap, we propose SHOC, an automated 4D HOI reconstruction engine. First, we introduce the Hand-Object Transformer to aggregate multi-view spatiotemporal cues, robustly regressing parametric states and coarse geometry to bypass the convergence bottlenecks of prior works. Second, a Physics-Aware 4D Gaussian Splatting framework performs high-fidelity photometric refinement. It enforces anatomical rigidity via a tetrahedral As-Rigid-As-Possible proxy and disentangles appearance into view-independent albedo and view-dependent specularity. This prevents environmental illumination from baking into the geometry, effectively eliminating motion-induced flickering artifacts. Validated on public benchmarks and a large-scale internal dataset, our self-improving engine achieves state-of-the-art robustness and visual quality, providing a scalable pipeline for generating artifact-free 4D digital assets.
