Zhengyang Shen
Zhengyang works on high-fidelity hand-object interaction.
Research Areas
Authored Publications
Sort By
VEPHand: View-Efficient Photometric Hand Performance Capture at Scale
Danhang "Danny" Tang
Sasa Petrovic
Timo Bolkart
Emre Aksan
Shichen Liu
Jonathan Taylor
Jérémy Riviere
Erroll Wood
Kai-Hung Chang
Vassilis Choutas
Mingsong Dou
Delio Vicini
Bowen Zhao
Deying Kong
Zhe Cao
Jinlong Yang
Jingjing Shen
arXiv preprint arXiv:2606.15966 (2026)
Preview abstract
High-fidelity 3D hand capture is fundamental to digital human creation, but achieving robust results from practical multi-view systems—which balance rich photometry with inherent geometric underspecification—remains challenging. This paper introduces an end-to-end pipeline for dynamic hand performance capture and registration designed for such conditions, utilizing a 20-camera photometric setup. We address core difficulties through two primary innovations: first, a mask-free neural reconstruction method that leverages scene parameterization and scenario-specific density regularization to robustly extract detailed hand geometry and appearance from unmasked images despite limited view overlap and potential background clutter. Second, we propose a volumetric physics-inspired registration framework that aligns these reconstructions to a personalized hand model built on a consistent tetrahedral mesh, enabling the capture of fine intrinsic deformations and ensuring plausible results under severe articulation and self-contact. We demonstrate the scalability and robustness of our automated pipeline on an extensive dataset of over 8,000 sequences, showcasing its effectiveness for single hands, intricate two-hand interactions, and detailed hand-object manipulations, significantly advancing the state-of-the-art in practical, high-quality hand capture.
View details
ESAM++: Efficient Online 3D Perception on the Edge
Qin Liu
Lavisha Aggarwal
Vikas Bahirwani
Lin Li
Aleksander Holynski
Saptarashmi Bandyopadhyay
Marc Niethammer
Ehsan Adeli
Andrea Colaco
2025
Preview abstract
Online 3D scene perception in real time is critical for robotics, AR/VR, and autonomous systems, particularly in edge computing scenarios where computational resources are limited. Recent state-of-the-art methods like EmbodiedSAM (ESAM) demonstrate the promise of online 3D perception by leveraging the 2D visual foundation model (VFM) with efficient 3D query lifting and merging. However, ESAM depends on a computationally expensive sparse 3D U-Net for point cloud feature extraction, which we identify as the primary efficiency bottleneck. In this paper, we propose a lightweight and scalable alternative for online 3D scene perception tailored to edge devices. Our method introduces a 3D Sparse FeaturePyramid Network (SFPN) that efficiently captures multi-scale geometric features from streaming 3D point clouds while significantly reducing computational over-head and model size. We evaluate our approach on four challenging segmentation benchmarks—ScanNet, ScanNet200, SceneNN, and 3RScan—demonstrating that our model achieves competitive accuracy with up to 3×faster inference and 3×small model size compared to ESAM, enabling practical deployment in real-world edge scenarios. Code and models will be released.
View details
Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images
Danhang "Danny" Tang
Franziska Müller
Jonathan Taylor
Mingsong Dou
Sasa Petrovic
Thabo Beeler
Tze Ho Elden Tse
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023), pp. 14666-14677
Preview abstract
We propose a novel transformer-based framework that reconstructs two high fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, where one typically trains a deep network to regress hand model parameters from single RGB image, we consider a more challenging problem setting where we directly regress the absolute root poses of two-hands with extended forearm at high resolution from egocentric view. As existing datasets are either infeasible for egocentric viewpoints or lack background variations, we create a large-scale synthetic dataset with diverse scenarios and collect a real dataset from multi-calibrated camera setup to verify our proposed multi-view image feature fusion strategy. To make the reconstruction physically plausible, we propose two strategies: (i) a coarse-to-fine spectral graph convolution decoder to smoothen the meshes during upsampling and (ii) an optimisation-based refinement stage at inference to prevent self-penetrations. Through extensive quantitative and qualitative evaluations, we show that our framework is able to produce realistic two-hand reconstructions and demonstrate the generalisation of synthetic-trained models to real data, as well as real-time AR/VR applications.
View details
