Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 1540 publications
On-the-Fly OVD Adaptation with FLAME: Few-shot Localization via Active Marginal-Samples Exploration
Yehonathan Refael
Amit Aides
Aviad Barzilai
Vered Silverman
Bolous Jaber
Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops (2026), pp. 886-894
Preview abstract
Open-vocabulary object detection (OVD) models offer remarkable flexibility applications by enabling object detection from arbitrary text queries. Still, the zero-shot performance of the pre-trained models is hampered by the inherent semantic ambiguity of natural language, result to low precision, leading to insufficient crucial downstream applications. For instance, in the remote sensing (RS) domain, a query for "ship" can yield varied and contextually irrelevant results. To address this, for real time applications, we propose a novel cascaded architecture that synergizes the broad capabilities of a large, pre-trained OVD model with a lightweight, few-shot classifier. Our approach utilizes the frozen weights of the zero-shot model to generate initial, high-recall object-embedding proposals, which are then refined by a compact classifier trained in real-time on a handful of user-annotated examples. The core of our contribution is an efficient one step active learning strategy for selecting the most informative samples for user annotation. Our method identifies (extremely) small amount of an uncertain candidates near the theoretical decision boundary using density estimation and then applies clustering to ensure a diverse training set. This targeted sampling enables our cascaded system to elevate performance on standard remote sensing benchmarks. Our work thus presents a practical and resource-efficient framework for adapting foundational models to specific user needs, drastically reducing annotation overhead while achieving high accuracy without costly full-model fine-tuning.
View details
VISTA: A Test-Time Self-Improving Video Generation Agent
Xuan Long Do
Hootan Nakhost
The IEEE/CVF Conference on Computer Vision and Pattern Recognition (to appear) (2026)
Preview abstract
Despite rapid advances in text-to-video (T2V) synthesis, generated video quality remains critically dependent on precise user prompts. Existing test-time optimization methods, successful in other domains, struggle with the multi-faceted nature of video. To address this, we introduce VISTA, a novel multi-agent system that autonomously refines prompts to improve video generation. VISTA operates in an iterative loop, first decomposing a user's idea into a structured temporal plan. After generation, the best video is identified through a robust pairwise tournament. This winning video is then critiqued by a trio of specialized agents focusing on visual, audio, and contextual fidelity. Finally, a reasoning agent synthesizes this feedback to introspectively rewrite and enhance the prompt for the next generation cycle. To rigorously evaluate our proposed approach, we introduce MovieGen-Bench, a new benchmark of diverse single- and multi-scene video generation tasks. Experiments show that while prior methods yield inconsistent gains, VISTA consistently improves video quality, achieving up to 60% pairwise win rate against state-of-the-art baselines. Human evaluators concur, preferring VISTA's outputs in 68% of comparisons.
View details
Gaze Target Estimation Anywhere with Concepts
Xu Cao
Houze Yang
Vipin Gunda
Inki Kim
Jim Rehg
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2026)
Preview abstract
Estimating human gaze targets in-the-wild is a formidable challenge. Existing computer vision algorithms rely on brittle, multi-stage pipelines that require explicit inputs like head bounding boxes and human pose, causing initial detection errors to cascade and lead to system failure. To overcome this, we introduce the \textbf{Promptable Gaze Target Estimation (PGE)} task, a new end-to-end, concept-driven paradigm. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject's target, which eliminates the rigid dependency on intermediate localization cues. We develop a scalable data engine to generate \textbf{Gaze-Co}, a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose \textbf{AnyGaze}, the first model designed for PGE. AnyGaze uses a Transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. AnyGaze achieves state-of-the-art performance on standard gaze target estimation benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. We will open-source the AnyGaze model and the Gaze-Co benchmark.
View details
Marginalized Bundle Adjustment: Multi-View Camera Pose from Monocular Depth Estimates
Shengjie Zhu
Xiaoming Liu
Vincent Chu
International Conference on 3D Vision (2026)
Preview abstract
Structure-from-Motion (SfM) is a classical 3D vision task for recovering camera parameters and scene geometry from multi-view images. Recent advances in deep learning enable accurate monocular depth estimation (MDE) that infers structure from a single image without depending on camera motion. But integrating MDE into SfM remains challenging. Unlike classical triangulated sparse pointclouds, MDE produces dense depthmaps with significantly higher error variance. Inspired by modern RANSAC estimators, we propose a Marginalized Bundle Adjustment (MBA) to accommodate MDE error variance with its density. With MBA, we show that MDE depthmaps are sufficiently accurate to support SoTA or competitive results in Structure-from-Motion and camera relocalization. Our benchmark demonstrates consistent remarkable results from two-view, few-frames small multiview, to thousands-frames large multiview system. Our method highlights the significant potential of MDE on multi-view 3D vision tasks.
View details
Mull-Tokens: Modality-Agnostic Latent Thinking
Arijit Ray
Chengzhi Mao
Bryan A. Plummer
Kate Saenko
Ranjay Krishna
Leonidas Guibas
Vincent Chu
IEEE/CVF Conference on Computer Vision and Pattern Recognition (Findings) (2026) (to appear)
Preview abstract
Reasoning goes beyond language; the real world requires reasoning about space, time, affordances, and much more that words alone cannot convey. Existing multimodal models exploring the potential of reasoning with images are brittle and do not scale. They rely on calling specialist tools, costly generation of images, or handcrafted reasoning data to switch between text and image thoughts. Instead, we offer a simpler alternative -- Mull-Tokens -- modality-agnostic latent tokens pre-trained to hold intermediate information in either image or text modalities to let the model think free-form towards the correct answer. We investigate best practices to train Mull-Tokens inspired by latent reasoning frameworks. We first train Mull-Tokens using supervision from interleaved text-image traces, and then fine-tune without any supervision by only using the final answers. Across four challenging spatial reasoning benchmarks involving tasks such as solving puzzles and taking different perspectives, we demonstrate that Mull-Tokens improve upon several baselines utilizing text-only reasoning or interleaved image-text reasoning, achieving a +3% average improvement and up to +16% on a puzzle solving reasoning-heavy split compared to our strongest baseline. Adding to conversations around challenges in grounding textual and visual reasoning, Mull-Tokens offers a simple solution to abstractly think in multiple modalities.
View details
Preview abstract
Feed-forward foundation models for multi-view 3-dimensional(3D) reconstruction have been trained on large-scale datasets of perspective images; when tested on wide field-of-view images, e.g., from a fisheye camera, their performance degrades. This degradation arises from changes in spatial arrangements of pixels induced by the non-linear projection model that maps 3D points onto the 2D image plane. While one may surmise that training on fisheye images would resolve this problem, there are far fewer fisheye images with ground truth than perspective images, which limits generalization. To enable inference on imagery exhibiting high radial distortion, we propose Fisheye3R, a novel adaptation framework that extends these multi-view 3D reconstruction foundation models to natively accommodate fisheye inputs without performance regression on perspective images. To address the scarcity of fisheye images and ground truth, we introduce flexible learning schemes that support self-supervised adaptation using only unlabeled perspective images and supervised adaptation without any fisheye training data. Extensive experiments across three foundation models, including VGGT, π3 , and MapAnything, demonstrate that our approach consistently improves camera pose, depth, point map, and field-of-view estimation on fisheye images.Code is available at https://github.com/android-xr/fisheye3r.
View details
Preview abstract
High-fidelity removal of eyeglasses from video is a major challenge in facial attribute editing, as the underlying facial geometry is often obscured by complex refractive distortions and view-dependent specular reflections. While large-scale generative priors have shown promise in static image inpainting, they often lack the structural constraints necessary to maintain identity, expression, and pose, leading to visible “identity drift” in both static images and dynamic sequences. In this paper, we propose a novel distillation framework that addresses the stochastic nature of generative priors. Our pipeline first extracts high-fidelity outputs from Gemini, regularizes them via dense geometric landmark constraints to preserve identity, expression, and pose, and finally applies physically-based simulation of lens optics to generate realistic paired training data. This process transfers Gemini’s photo-realistic, multi-view knowledge into a specialized restoration architecture, JFSnet (Joint Feature-Spatial network). JFSnet integrates DINOv2-based semantic features with a convolutional decoder for spatial reconstruction, leveraging equivariance constraints to improve high-frequency detail preservation and stability. Evaluations show that our approach significantly outperforms existing diffusion and GAN-based baselines, achieving the lowest FID scores and ranking highest in user preference studies across visual fidelity, identity preservation, and temporal consistency.
View details
Preview abstract
Latent Diffusion Models (LDM) face three compounding efficiency challenges in practical deployment: i) the temporal latency of iterative sampling; ii) the architectural overhead of heavy backbone parameter counts; and iii) the spatial cost of high-dimensional latent grids. While recent acceleration methods have made substantial progress on temporal distillation and architectural compression, the spatial axis is often inherited from the teacher tokenizer and treated as fixed. In this work, we recast latent resolution as an optimizable efficiency axis and introduce a unified framework that optimizes all three dimensions simultaneously. We introduce a novel strategy of Score-Compatible Tokenizer Distillation (SCTD), which leverages score-matching principles to align a spatially compact latent space with the induced distribution of a frozen, powerful teacher model, distilling the teacher's generative prior into a compressed, lower-dimensional compatible manifold. With flexibility provided by SCTD, we can surrogate a computationally heavy teacher backbone with a lightweight student architecture operating strictly within this new compressed space. Finally, we apply temporal distillation to collapse the sampling trajectory, producing a one-step generator that operates at peak efficiency. Our method yields a student generator outperforming existing single-axis acceleration methods in efficiency and throughput, while maintaining competitive generation quality. With reduced peak memory usage and latency, our method enables resource-constrained deployment and high-volume serving of high-fidelity LDM.
View details
StreamLens: An Open-Source ABR Simulation Environment for Multi-Region Buffering, Seeking, and Prefetching
Bei Zhao
Jinhui Song
Jiayu Zhu
Yih‑Chun Hu
2026
Preview abstract
ABR simulators are useful for studying ABR behavior under different network conditions. SABRE is an open-source simulator for evaluating ABR algorithms with different videos, network traces,
and ABR policies. However, SABRE has several limitations. First, it does not accurately emulate user seeks or their impact on the viewing experience. Second, it provides limited support for nonlinear buffering, where the client buffers non-contiguous video chunks. Furthermore, SABRE does not provide adequate analysis and visualization support for interpreting simulation results. This paper presents StreamLens, an advanced video simulator that supports: 1) pluggable ABR policies, including BOLA, BOLA-E, Dynamic, DynamicDash, and throughput-based policy; 2) configurable user seeks; 3) nonlinear buffering with customizable prefetching policy; 4) browser-based viewer for result visualization, including
stalls, seeks, rebuffering time, QoE, bitrate, buffer, etc. StreamLens allows developers to configure videos, network traces, and seek patterns, and to evaluate ABR performance across different setups, including ABR policies, linear versus nonlinear buffering, and user seek traces. It also provides metrics and visualizations that help developers inspect streaming behavior in a straightforward way. Several predefined scenarios are included as both evaluation examples and configuration templates, covering synthetic seek and prefetch settings, session-replay workflows, generated reports, and regression tests. The software is available publicly at
https://github.com/Slothbetty/StreamLens.
View details
Preview abstract
We present MOSAIC-GS, a novel, fully explicit, and computationally efficient approach for high-fidelity dynamic scene reconstruction from monocular videos.
Monocular reconstruction is inherently ill-posed due to the absence of sufficient multiview constraints,
making accurate recovery of object geometry and temporal coherence particularly challenging.
To address this, we leverage multiple geometric cues, such as depth, optical flow, dynamic object segmentation, and tracking trajectories,
combined with rigidity constraints to estimate preliminary 3D scene dynamics during an advanced initialization stage.
Recovering scene flow prior to the photometric optimization phase reduces reliance on motion inference from visual appearance alone, which is often ambiguous in monocular settings.
To enable compact representations, fast training, and real-time rendering while supporting non-rigid deformations, the scene is decomposed into static and dynamic components,
with dynamic trajectories represented as time-dependent Poly-Fourier curves for parameter-efficient motion encoding.
We demonstrate that MOSAIC-GS achieves substantially faster optimization and rendering compared to existing methods,
while maintaining reconstruction quality on par with state-of-the-art approaches across standard monocular dynamic scene benchmarks.
View details
CAST: Modeling Visual State Transitions for Consistent Video Retrieval
Yanqing Liu
Yingcheng Liu
Fanghong Dong
Budianto Budianto
Cihang Xie
Yan Jiao
2026
Preview abstract
As video content creation shifts towards long-form narratives, retrieving and composing short clips into coherent storylines becomes a critical challenge.
Standard retrieval formulations, however, perform context-agnostic retrieval, prioritizing local semantic alignment while neglecting procedural state and identity consistency across the narrative flow.
To address this, we introduce the task of Consistent Video Retrieval (CVR) and establish a benchmark designed to diagnose such inconsistencies via semantic hard negatives.
We propose CAST (Context-Aware State Transition), a lightweight adapter that models procedural progression as state-conditioned transitions.
Conditioned on visual history, CAST predicts a gated residual vector ($\Delta$) to selectively update the state embedding, ensuring procedural coherence while preserving identity.
Extensive experiments demonstrate that CAST significantly outperforms standard retrieval baselines on our CVR benchmark.
Furthermore, we show its potential as a plug-and-play consistency verifier, guiding black-box generation models (e.g., Veo) toward coherent video continuations within long-form narratives.
View details
Generative Manifold Distillation: Aligning Restoration Trajectories with Natural Image Prior
Mojtaba Ardakani
Arpit Bansal
Chenyang Chenyang
European Conference on Computer Vision, Springer (2026), pp. 669-687
Preview abstract
Pre-trained image restoration models often fail on out-of-distribution (OOD) real-world degradations. Adapting to these domains is challenging as real-world data lacks paired ground truth, and unsupervised methods often require unstable architectural changes. We propose Generative Manifold Distillation (GMD), which reframes domain adaptation as geometric manifold alignment. GMD operates in a strictly unpaired setting, requiring only low-quality (LQ) target observations. By leveraging the flow-matching dynamics of a frozen text-to-image foundation model, GMD projects off-manifold restorations onto the natural image manifold to generate high-quality pseudo-targets. To ensure stability, a quality-gated manifold filter rejects off-manifold samples, while source-anchored trajectory regularization prevents error accumulation. Ultimately, GMD distills a powerful generative prior into an efficient restoration network. Experiments demonstrate that GMD seamlessly adapts to new distributions using only LQ inputs, drastically improving perceptual quality with zero architectural modifications or added inference latency.
View details
VidMap: Exploiting Temporal Structure for Video-Based Structure-from-Motion
Zador Pataki
Paul-Edouard Sarlin
Marc Pollefeys
ECCV (2026)
Preview abstract
Accurately recovering the camera's calibration and metric poses for any unconstrained video would unlock large-scale training data for navigation and scene understanding. The dominant approaches to this problem are severely limited: Simultaneous Localization and Mapping (SLAM) is sensitive to initialization and transient failures due to its causal, incremental nature; it is often over-optimized for real-time operation and generally requires known camera calibration; while Structure-from-Motion (SfM) typically forgoes any image ordering, enabling optimal initialization and global optimization, but lacks robustness to visual symmetries and extreme motions. To bridge this gap, we introduce a system that combines the strong sequential constraints of SLAM with the flexibility and global optimization of offline SfM, enabling the metric reconstruction of arbitrary, long, uncalibrated videos. This system leverages recent advances in wide-baseline dense image matching, treats temporal ordering as a first-class citizen for reliable loop closure, and augments global optimization with metric monocular depth priors. As a result, thorough evaluations on diverse, challenging datasets that exhibit extreme motion and visual symmetries reveal that our approach is significantly more robust and accurate than both state-of-the-art SLAM and SfM, classical or learned, with given or unknown camera calibration. The code is publicly available at https://github.com/cvg/vidmap.
View details
Preview abstract
This paper proposes RD-LoRA, a rate-distortion (R-D) optimized low-rank adaptation (LoRA) framework for neural post-filtering in the upcoming AV2 coding standard. RD-LoRA adapts a pre-trained base neural model to diverse input content via online updating of LoRA parameters, whose quantity is governed by the matrix ranks. The updated parameters are then quantized, transmitted to the decoder, and merged with the pre-trained weights for post-filtering. While more parameters generally improve coding performance, they also increase transmission bitrate.
To balance distortion reduction against transmission cost, we propose to dynamically allocate rank budget to each layer of the base model in a closed-loop R-D manner. Specifically, it incorporates a cost-aware importance assessment to discourage parameter-heavy updates, together with an RD-rank allocator to guide pruning based on global R-D optimization. To maintain robustness when certain layers are pruned to zero rank, we further introduce a lightweight fallback modulation mechanism. Experimental results show that, when deployed on a lightweight 25 kMACs ResNet model, RD-LoRA achieves a BD-rate reduction of 2.556% over the AV2 anchor, significantly outperforming the base model with only limited decoding overhead.
View details
UniGeoCLIP: Unified Geospatial Contrastive Learning
Guillaume Astruc
Jan Hosang
Loic Landrieu
Paul-Edouard Sarlin
EarthVision Workshop at CVPR (2026)
Preview abstract
The growing availability of co-located geospatial data spanning aerial imagery, street-level views, elevation models, text, and geographic coordinates offers a unique opportunity for multimodal representation learning. We introduce UNIGEOCLIP, a massively multimodal contrastive framework to jointly align five complementary geospatial modalities in a single unified embedding space. Unlike prior approaches that fuse modalities or rely on a central pivot representation, our method performs all-to-all contrastive alignment, enabling seamless comparison, retrieval, and reasoning across arbitrary combinations of modalities. We further propose a scaled latitude-longitude encoder that improves spatial representation by capturing multi-scale geographic structure. Extensive experiments across downstream geospatial tasks demonstrate that UNIGEOCLIP consistently outperforms single-modality contrastive models and coordinate-only baselines, highlighting the benefits of holistic multimodal geospatial alignment. A reference implementation is available at https://gastruc.github.io/unigeoclip.
View details
