Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 11583 publications
When Android Apps Request Location with a Reason: A Developer Study and User Behavior Experiment
May Smith
Aman Jain
Proceedings on Privacy Enhancing Technologies (2027)
Preview abstract
Recent reports have highlighted how mobile apps share user location data with third parties, risking user privacy and platform trust. Although location data is highly sensitive, when users grant apps location access, they may not know the full extent to which it is used. We study how requiring Android apps to show a reason for location access could impact developers, users, and the platform.
We surveyed 323 Android app developers and found most supported such a requirement. The majority said it would have a positive impact on user privacy, trust for apps, and trust for Android, where impact on user trust for Android correlated most strongly with support. Many developers also said the intervention would increase the number of users granting location access. Yet their open-ended comments also revealed consistent concerns, such as apps providing dishonest reasons and platform verification.
To study the impact on user behavior, we conducted a randomized controlled experiment with 2579 US Android users. We tested how users' decisions to grant location access were impacted by app type, whether reasons were included in the requests, and the content of the reasons, including monetization. We did not find the reasons impacted users' decisions; decisions were instead driven by app type and demographics. Yet we did find the reasons could have a positive impact on user perception for the platform when the reasons did not include using data for ads. Our findings provide insights into developers' willingness to implement privacy-enhancing changes, and expose limits to improving user privacy by simply adding information to user interfaces.
View details
A 3D Scene Graphs Survey: Open Challenges and Future Directions
Dennis Rotondi
Francesco Argenziano
Sebastian Koch
Nathan Hughes
Martin Büchner
Johanna Wald
Lukas Schmid
Daniele Nardi
Abhinav Valada
Liam Paul
Luca Carlone
Kai Arras
Annual Review of Control, Robotics, and Autonomous Systems (ARCRAS), 10 (2027) (to appear)
Preview abstract
3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including mapping, task and motion planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real- world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are constructed from raw sensory observations, covering both learning-oriented and construction-oriented systems. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed works.
View details
Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting
David Autor
Tanya Rodchenko
Joshua Martin
Zanna Iscenko
Scott Strand
David Pearl
Melissa Ferere
NBER (2026)
Preview abstract
Whether AI assistance builds or erodes professional expertise is unsettled. In a pre-registered three-month randomized controlled trial, we gave 133 practicing patent lawyers at eleven U.S. intellectual property law firms access to a custom AI drafting assistant and measured both their performance while using AI and their professional judgment afterward without it. All work was scored by blinded expert patent attorneys. Paralleling findings from other white-collar domains, AI access raised the quality of work delivered on benchmark patent drafting tasks at 10 days (0.34 SD, p = 0.03) and 90 days (0.38 SD, p = 0.01), with larger gains among junior lawyers. After three months, all subjects redlined an existing patent application without AI, a core task of patent practice requiring expert judgment. Treated lawyers outperformed controls by 0.32 SD (p = 0.04), but this advantage was concentrated entirely among senior lawyers (0.45 SD, p = 0.02). Junior lawyers showed no average gain; their scores instead bifurcated, with sharply fewer mediocre scores offset by more poor and more good ones. The largest gains from AI thus accrued to the lawyers who retained the least. Foundational expertise may be a prerequisite for extracting durable skill from AI-assisted practice.
View details
Optimized Deferral for Imbalanced Settings
Anqi Mao
Proceedings of the 43rd International Conference on Machine Learning (ICML 2026)
Preview abstract
Learning algorithms can be significantly improved by routing complex or uncertain inputs to specialized experts, balancing accuracy with computational cost. This approach, known as learning to defer, is essential in domains like natural language generation, medical diagnosis, and computer vision, where an effective deferral can reduce errors at low extra resource consumption. However, the two-stage learning to defer setting, which leverages existing predictors such as a collection of LLMs or other classifiers, often faces challenges due to an expert imbalance problem. This imbalance can lead to suboptimal performance, with deferral algorithms favoring the majority expert. We present a comprehensive study of two-stage learning to defer in expert imbalance settings. We cast the deferral loss optimization as a novel cost-sensitive learning problem over the input-expert domain. We derive new margin-based loss functions and guarantees tailored to this setting, and develop novel algorithms for cost-sensitive learning. Leveraging these results, we design principled deferral algorithms, MILD (Margin-based Imbalanced Learning to Defer), specifically suited for expert imbalance settings. Extensive experiments demonstrate the effectiveness of our approach, showing clear improvements over existing baselines on both image classification and real-world Large Language Model (LLM) routing tasks.
View details
Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models
Yu Jiang
Hanwen Jiang
Vincent Chu
Brandon Y. Feng
Zhangyang Wang
Qixing Huang
IEEE/CVF Conference on Computer Vision and Pattern Recognition (2026)
Preview abstract
With the emergence of 3D foundation models, there is growing interest in fine-tuning them for downstream tasks, where LoRA is the dominant fine-tuning paradigm. As 3D datasets exhibit distinct variations in texture, geometry, camera motion, and lighting, there are interesting fundamental questions: 1) Are there LoRA subspaces associated with each type of variation? 2) Are these subspaces disentangled (i.e., orthogonal to each other)? 3) How do we compute them effectively? This paper provides answers to all these questions. We introduce a robust approach that generates synthetic datasets with controlled variations, fine-tunes a LoRA adapter on each dataset, and extracts a LoRA sub-space associated with each type of variation. We show that these subspaces are approximately disentangled. Integrating them leads to a reduced LoRA subspace that enables efficient LoRA fine-tuning with improved prediction accuracy for downstream tasks. In particular, we show that such a reduced LoRA subspace, despite being derived entirely from synthetic data, generalizes to real datasets. An ablation study validates the effectiveness of the choices in our approach.
View details
POLCA: Stochastic Generative Optimization with LLM
Xuanfei Ren
Allen Nie
Tengyang Xie
Ching-An Cheng
2026
Preview abstract
Optimizing complex systems, ranging from LLM prompts to multi-turn agents, traditionally requires labor-intensive manual iteration. We formalize this challenge as a stochastic generative optimization problem where a generative language model acts as the optimizer, guided by numerical rewards and text feedback to discover the best system. We introduce Prioritized Optimization with Local Contextual Aggregation (POLCA), a scalable framework designed to handle stochasticity in optimization -- such as noisy feedback, sampling minibatches, and stochastic system behaviors -- while effectively managing the unconstrained expansion of solution space. POLCA maintains a priority queue to manage the exploration-exploitation tradeoff, systematically tracking candidate solutions and their evaluation histories. To enhance efficiency, we integrate an ε-Net mechanism to maintain parameter diversity and an LLM Summarizer to perform meta-learning across historical trials. We theoretically prove that POLCA converges to near-optimal candidate solutions under stochasticity. We evaluate our framework on diverse benchmarks, including τ-bench, HotpotQA (agent optimization), VeriBench (code translation) and KernelBench (CUDA kernel generation). Experimental results demonstrate that POLCA achieves robust, sample and time-efficient performance, consistently outperforming state-of-the-art algorithms in both deterministic and stochastic problems. The codebase for this work is publicly available at this https URL.
View details
Gaze Target Estimation Anywhere with Concepts
Xu Cao
Houze Yang
Vipin Gunda
Inki Kim
Jim Rehg
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2026)
Preview abstract
Estimating human gaze targets in-the-wild is a formidable challenge. Existing computer vision algorithms rely on brittle, multi-stage pipelines that require explicit inputs like head bounding boxes and human pose, causing initial detection errors to cascade and lead to system failure. To overcome this, we introduce the \textbf{Promptable Gaze Target Estimation (PGE)} task, a new end-to-end, concept-driven paradigm. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject's target, which eliminates the rigid dependency on intermediate localization cues. We develop a scalable data engine to generate \textbf{Gaze-Co}, a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose \textbf{AnyGaze}, the first model designed for PGE. AnyGaze uses a Transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. AnyGaze achieves state-of-the-art performance on standard gaze target estimation benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. We will open-source the AnyGaze model and the Gaze-Co benchmark.
View details
Mobility-Embedded POIs: Learning What a Place Is and How It’s Used from Human Movement
Shushman Choudhury
Shang-Ling Hsu
Cyrus Shahabi
Forty-third International Conference on Machine Learning (2026)
Preview abstract
Recent progress in geospatial foundation models (GeoFMs) has highlighted the
importance of learning general-purpose representations for real-world locations,
particularly Points of Interest (POIs) where human activity concentrates. Yet, ex-
isting POI representations remain largely static, drawing from textual metadata
(e.g., category labels, descriptions) and spatial attributes (e.g., coordinates, neigh-
borhood context), all of which describe what a place is, but not how it is actu-
ally used. We argue that human mobility provides a complementary and dynamic
signal, capturing real-world visitation patterns that reveal how places function in
practice. To this end, we introduce Mobility Embedded POIs (ME-POIs), a
pretraining framework that learns POI representations directly from sequences of
human visits. Each visit is encoded as a contextualized embedding that captures
the POI’s static attributes as well as its temporal and sequential context, includ-
ing when the visit occurs and which visits surround it. These visit embeddings
are aligned with learnable POI embeddings via a contrastive objective, grounding
POI representations in their real-world usage patterns. To address the long tail of
sparsely visited POIs, we transfer visitation distributions from data-rich anchors
to sparse locations, leveraging multi-scale spatial proximity to capture local and
regional patterns, and functional similarity to enable transfer across semantically
related POIs. We demonstrate the utility of ME-POIs for a set of automated map
enrichment tasks, critical in geospatial intelligence. We show empirically that
by embedding visitation dynamics, ME-POIs outperform text- and location-only
baselines, proving that mobility-informed embeddings provide a stronger founda-
tion for modeling place function and change.
View details
VIP-MINGLE: A Corpus for Videoconference and In-Person Multimodal Interaction in Group Language Engagement
Abhinay K Bodi
Wenxin Deng
Junrui Huang
Venu G Kadamba
Sumanth B H Karanam
Dhiwahar A Kennady
David Poeppel
Dustin Freeman
Interspeech (2026)
Preview abstract
Group conversations are a fundamental yet complex form of social interaction central to human cognition and telecommunication technology. While understanding and facilitating these interactions has been a long-standing goal, findings are often isolated within specific in-person or videoconferencing settings due to a scarcity of datasets that bridge the two. We introduce VIP-MINGLE, a multimodal dataset comprising 59 hours of recordings (32 groups, 105 participants), featuring paired within-subject sessions in both settings. The dataset includes raw audio/video, psychometric data, processed multimodal features (e.g., diarized speech, facial expressions, transcriptions), and time-resolved human annotations. Our analysis reveals significant behavioral distribution shifts across multiple modalities between settings, reinforcing the need for a cross-setting corpus. VIP-MINGLE serves as a critical resource for developing robust models of group conversations across settings.
View details
Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All,
Chenxi Huang
Alex Mathai
Feiyang Yu
Aleksandr Nogikh
Eugene Wu
Kostis Kaffes
Junfeng Yang
Baishakhi Ray
Proceedings of the 43rd International Conference on Machine Learning (ICML) (2026)
Preview abstract
Repairing system crashes discovered by kernel fuzzers like Syzkaller is a critical yet underexplored challenge in software engineering. While recent works have introduced Large Language Model (LLM) based agents for Linux kernel crash-resolution, their evaluation benchmarks are usually static and thus, do not capture the evolving nature of the Linux kernel, and suffer from potential data contamination due to LLM knowledge cutoffs. To address the above problem, we present (i) Live-kBench, an evaluation framework for self-evolving benchmarks that continuously scrapes and evaluates agents on freshly discovered kernel bugs, and (ii) kEnv, an agent-agnostic standardized crash-resolution environment for kernel compilation, execution, and feedback. This design decouples agent workflows from heavy-weight execution, enabling fair and scalable comparison across diverse agent frameworks under identical conditions. To this end, we curate an inaugural dataset of 534 Linux kernel bugs and empirically demonstrate a significant performance gap, with agents achieving up to 25% higher equivalent patch rate on bugs fixed before the LLM knowledge cutoff. Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt (plausible patches); however only ~20% of generated patches closely match developer fixes. Additionally, exposing crash resolution feedback improves crash resolution rate by 29%. Live-kBench provides the community with an evaluation infrastructure for self-evolving benchmarks that is both time and attribute sensitive; complete with a public dashboard to track agent progress on Linux kernel bugs.
View details
Preview abstract
Multimodal large language models (LLMs) integrate and process information from multiple modalities such as text, images, audio, and video, enabling complex tasks such as audio translation and visual question answering. While powerful, this complexity introduces novel vulnerabilities to sophisticated adversarial attacks. This survey paper provides a comprehensive overview of this rapidly expanding field, systematically categorizing attacks that range from manipulations of single modalities (e.g., perturbed images or audio) to those exploiting cross-modal interactions. We overview how these attacks exploit weaknesses in model fusion, attention mechanisms, and representation learning and provided analyses on their potential for real-world consequences.
View details
Preview abstract
Large-scale software systems frequently suffer from architectural rigidity caused by monolithic designs, tightly coupled integrations, and legacy technology stacks. Backend-forFrontend (BFF) architectures are increasingly adopted to address these challenges by decoupling frontend-specific requirements from backend domain services. However, designing a BFF layer requires a series of irreversible technology decisions across compute platforms, traffic routing, programming languages, frameworks, and API protocols. These decisions directly influence system latency, scalability, operational complexity, and long-term maintainability. This paper proposes a structured, metrics-driven decision framework to guide architects through foundational technology choices when designing BFF architectures. The framework decomposes the decision space into independent sub-problems, introduces weighted evaluation criteria, and applies quantitative scoring models to enable objective trade-off analysis. The approach is validated through a representative modernization scenario, demonstrating how systematic evaluation reduces architectural risk, resolves stakeholder disagreement, and improves performance and developer efficiency. The proposed framework is generic, repeatable, and applicable to a wide range of cloudnative system modernization efforts.
View details
Preview abstract
This article presents a novel approach to automating operations tasks, particularly incident triage, by using AI agents defined entirely in Markdown. These agents orchestrate actions across various observability tools (e.g., Datadog, Splunk) and use the file system for state and communication, mimicking the Unix philosophy. The system enables parallel investigations, cross-tool validation, and structured reporting without traditional coding frameworks.
View details
The AI architecture decisions that cannot wait
AI Accelerator Institute (2026)
Preview abstract
As organizations pursue AI transformation in large environments, ambitions often collide with the practical urgency of fixed-deadline data center exits. This article argues that a non-negotiable migration date should not be viewed merely as an infrastructure constraint, but as a critical deadline for establishing AI readiness. AI systems amplify the strengths and weaknesses of the underlying architecture; fragmented data or inconsistent infrastructure will lead to unreliable AI outcomes. To build a foundation for future intelligence, architects must prioritize resilience, standardization, and governance early in the migration process. Ultimately, successful AI transformation depends less on the speed of deployment and more on foundational architectural decisions such as Infrastructure-as-Code and unified telemetry—made before the migration concludes.
View details
Preview abstract
Here’s a thought experiment. Say I wave a magic wand across a codebase and an entire class of technical debt, poof, goes away and immediately evaporates if introduced in the future. For example, maybe I make it so that dead feature flags are simply no longer a problem: they just delete themselves as soon as the engineer wills it. Or maybe large-scale migrations just migrate themselves. Maybe we magically have 100% test coverage, without an engineer lifting a finger.
What will happen to developer productivity?
Surely, developer productivity increases overall.
But will the productivity metrics that we all use as a proxy for “developer productivity” move up and to the right. Let’s explore this idea.
View details
