Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 11673 publications
A 3D Scene Graphs Survey: Open Challenges and Future Directions
Dennis Rotondi
Francesco Argenziano
Sebastian Koch
Nathan Hughes
Martin Büchner
Johanna Wald
Lukas Schmid
Daniele Nardi
Abhinav Valada
Liam Paul
Luca Carlone
Kai Arras
Annual Review of Control, Robotics, and Autonomous Systems (ARCRAS), 10 (2027) (to appear)
Preview abstract
3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including mapping, task and motion planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real- world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are constructed from raw sensory observations, covering both learning-oriented and construction-oriented systems. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed works.
View details
Preview abstract
Modern transformers have been scaled up in size, data, training-time compute, and more recently test-time compute based on Chain-of-Thought (CoT), but the consequences of trading off these parameters are unknown. We construct explicit tradeoffs between two of these aspects: the number of layers in a transformer and the number of CoT tokens. Our first result is general: with mild assumptions and a single-token prompt, for any integer k, a transformer using soft attention and SwiGLU MLP layers with L layers and n CoT tokens can be converted to a transformer with at most ckL layers and ⌈n/k⌉ CoT tokens, and vice versa, although with an increase to the size of other parameters. We then a multi-token prompt path following task, finding that additional preprocessing from the multi-token prompt changes the depth-CoT relationship in this case. We construct an attention-only network with L layers and 2^(L−2) heads that can traverse 2^(L−1) − 1 edges along a path with a single CoT token, demonstrating at least an exponential relationship between increases in depth and reductions in the number of CoT tokens needed to traverse the entire path. This relationship for the path following task holds empirically as well, as demonstrated by training networks with up to 5 layers, with the tradeoff between depth and the maximum number of edge traversals in a single CoT token exactly matching our theoretical constructions.
View details
When Android Apps Request Location with a Reason: A Developer Study and User Behavior Experiment
May Smith
Aman Jain
Proceedings on Privacy Enhancing Technologies (2027)
Preview abstract
Recent reports have highlighted how mobile apps share user location data with third parties, risking user privacy and platform trust. Although location data is highly sensitive, when users grant apps location access, they may not know the full extent to which it is used. We study how requiring Android apps to show a reason for location access could impact developers, users, and the platform.
We surveyed 323 Android app developers and found most supported such a requirement. The majority said it would have a positive impact on user privacy, trust for apps, and trust for Android, where impact on user trust for Android correlated most strongly with support. Many developers also said the intervention would increase the number of users granting location access. Yet their open-ended comments also revealed consistent concerns, such as apps providing dishonest reasons and platform verification.
To study the impact on user behavior, we conducted a randomized controlled experiment with 2579 US Android users. We tested how users' decisions to grant location access were impacted by app type, whether reasons were included in the requests, and the content of the reasons, including monetization. We did not find the reasons impacted users' decisions; decisions were instead driven by app type and demographics. Yet we did find the reasons could have a positive impact on user perception for the platform when the reasons did not include using data for ads. Our findings provide insights into developers' willingness to implement privacy-enhancing changes, and expose limits to improving user privacy by simply adding information to user interfaces.
View details
Preview abstract
We study advertising in conversational LLM platforms, where the platform gradually learns a user's preferences before deciding when and what ad to offer. We model this as a stochastic control problem in which the platform balances the value of information acquisition against the risk of user departure. Although the optimal policy that maximizes welfare admits a threshold structure, computing it exactly is infeasible without strong assumptions. We develop an approximate threshold policy based on upper and lower bounds on the continuation value and prove explicit performance guarantees that scale with uncertainty, learning dynamics, and heterogeneity of the advertisers. Furthermore, we extend this framework to revenue maximization, where advertisers hold private valuations. We establish a "virtual welfare equivalence" in this dynamic setting, demonstrating that the revenue-optimal incentive-compatible mechanism is implemented by applying the welfare-maximizing policy to advertisers' properly defined virtual valuations. This mechanism generalizes classical optimal auction theory to environments where the information structure evolves endogenously through user interaction, enabling us to derive approximately revenue-optimal mechanisms.
View details
Visual Planning: Let’s Think Only with Images
Han Zhou
Caiqi Zhang
Anna Korhonen
Chengzu Li
Yi Xu
Ivan Vulic
International Conference on Learning Representations (ICLR) (2026)
Preview abstract
Recent advancements in Large Language Models (LLMs) and their multimodal extensions (MLLMs) have significantly enhanced machine reasoning across diverse tasks. However, these models predominantly rely on language as the medium for both expressing and structuring reasoning, even when visual information is present. In this work, we argue that language may not always be the most natural or effective modality for reasoning, particularly in tasks involving spatial, geometric, or physical dynamics. Motivated by this, we propose a new paradigm, Visual Planning, which enables planning through purely visual representations, independent of textual mediation. In this paradigm, planning is executed via sequences of images that encode step-by-step inference in the visual domain, akin to how humans sketch or visualize future actions. We then introduce a novel two-stage reinforcement learning framework empowered by GRPO for post-training large vision models, resulting in substantial improvements in planning accuracy and generalization across both seen and novel scenarios, validated in representative visual navigation tasks, FrozenLake and Maze. Our results establish Visual Planning as a viable and promising alternative to language-based reasoning, opening new avenues for tasks that benefit from intuitive, image-based inference.
View details
Preview abstract
We compare two time-reversal-symmetric triangular-lattice Hubbard models at half filling: the zero-flux model and a staggered π-flux model with zero and π flux through oppositely oriented triangles. Although both reduce to the same Heisenberg model at leading order in the strong-coupling limit, their distinct charge fluctuation profiles yield different quantum paramagnetic regimes. Combining a strong-coupling expansion with infinite density matrix renormalization group (iDMRG) calculations, we find that the positive ring-exchange of the zero-flux model promotes spontaneous chirality, consistent with its notion of a chiral spin liquid at intermediate coupling. In the staggered-π-flux model, whose noninteracting limit is a Dirac semimetal, the ring exchange changes sign and disfavors chiral order and we find no resolvable scalar chirality. This renders flux-tuned ring-exchange an efficient discriminator between chiral and non-chiral quantum paramagnets at finite coupling.
View details
Profit Maximization in Bilateral Trade against a Smooth Adversary
Simone di Gregorio
Paul Duetting
Federico Fusco
Chris Schwiegelshohn
NeurIPS 2026
Preview abstract
Bilateral trade models the task of intermediating between two strategic agents, a seller and a buyer, who wish to trade a good. We study this problem from the perspective of a profit-maximizing broker within an online learning framework, where the agents' valuations are generated by a smooth adversary.
We devise a learning algorithm that guarantees a $\tilde{O}(\sqrt{T})$ regret bound, which is tight in the time horizon $T$ up to poly-logarithmic factors. This matches the minimax rate for the stochastic i.i.d.~case, and is also well separated from the adversarial setting, where sublinear-regret is unattainable. By extending the strong regret guarantees from the i.i.d.~case to the smooth adversary, we significantly broaden the scope of settings where such fast rate is achievable, while closing an important gap in the regret landscape of this fundamental economic problem.
To overcome the challenges posed by this adversary, we leverage a continuity property of smooth instances and combines this with a hierarchical net-construction of the broker's action space, which is analyzed via algorithmic chaining. We showcase the applicability of these techniques by deriving a similarly tight $\tilde{O}(\sqrt{T})$ regret bound for a related mechanism design model: the joint ads problem.
View details
Semantic Representation for Agentic Self-Play in Automated Laboratories
Yves Delley
John Coller
Zachary Cutts
Alexander Daryin
Jun Kim
Oscar Donosa
Srinidhi B Seshadri
Tanner Clary
Sandy Klemm
Petar Sirkovic
arXiv (2026)
Preview abstract
This technical report introduces a lab controller system designed as the foundational infrastructure for agentic self-driving laboratories. The core challenge in modern biological research is the immense scale of the search spaces, which are too vast for traditional experimental methods. Our solution is a lab controller that bridges the semantic gap between AI reasoning and the physical wet lab by managing a high-fidelity semantic model, or digital twin, of the laboratory workcell. This live, queryable model provides four crucial services to AI agents:
Introspection: Agents can query the lab's real-time state (e.g., reagent volumes, instrument status) to inform their planning and decision-making, grounding their logic in physical reality. Simulation: Agents can safely test experimental protocols in a time-compressed, virtual environment before physical execution. This is achieved via a discrete accounting engine that validates logistical and scientific feasibility, preventing errors, resource waste, and hardware damage. Trace Persistence: The controller records a complete, multimodal history of every experiment into a research dossier. This dataset is processed by a Wet2Dry pipeline into structured trajectories for training agents via imitation learning and offline reinforcement learning, using advanced techniques to compute rewards from noisy data. Safe Execution: The controller translates agent commands into hardware actions within a multi-layered safety architecture inspired by frameworks like Safe-SDL. This uses Verified Operating Domains (ODDs) to enforce pre-verified boundaries and Control Barrier Functions (CBFs) as real-time safety filters to prevent failures and ensure safe operation.
We have validated these services through case studies demonstrating that the controller is essential for: Robust Debugging: The detailed traces allowed us to diagnose subtle physical-world failures (metamerism) that would have corrupted an AI agent's learning process. Pre-emptive Protocol Verification: The simulation service caught critical scientific and logical errors in agent-generated code before they could cause an experiment to fail. End-to-End Automation: The controller successfully orchestrated a complex, multi-instrument workflow driven entirely by a high-level AI agent command.
Looking forward, this lab controller infrastructure enables a new paradigm of research. It allows for large-scale combinatorial assays that can explore thousands of experimental conditions simultaneously. The resulting data closes the loop on agentic ideation, allowing our AI systems to learn and improve with each experimental cycle, moving beyond simple hill-climbing to sophisticated, multi-variable optimization. By providing the tools to navigate these vast search spaces, the lab controller is a foundational step toward solving grand challenges in biology. Our work is guided by a strong commitment to safety and responsible innovation, incorporating a Human-in-the-Loop (HITL) review portal that uses the digital twin for transparent oversight and an automated biosecurity gateway with state-of-the-art, function-based screening to ensure all synthesized materials are safe by design. This report outlines the what of our system, its capabilities and strategic importance, with further technical details on the how reserved for future publications.
View details
Approaching the Privacy-Enhancing Technologies Adoption from a Requirements Engineering Perspective
Oleksandr Kosenkov
Danica Vranjanin
Abhinava Singh
Vadym Honcharenko
Volodymyr Spirin
2026
Preview abstract
In recent decades, Privacy Enhancing Technologies (PETs) have been gaining attention as a means to fulfill regulatory and user privacy requirements when processing personal data in software systems. Despite broad attention in research, support by regulators, contributions and adoption by large technological companies (like Google or Microsoft), and interest from software developers, adoption of PETs in practice is still challenging and lagging. For many years, existing research points to the same typical challenges to the adoption of PETs, such as complexity or absence of training; however, these challenges remain unresolved in practice despite the recent efforts.
In this industrial challenge paper, we take a deeper, practical requirements engineering-driven perspective on the challenges to PETs adoption across different stakeholders (namely, PETs developers and integrators, and PET adopters and users) and different disciplines (engineering, law, and business).
We suggest that the adoption of PETs should be facilitated with an in-depth understanding of engineering, business, and legal challenges to PETs, such as PETs' impact on software architecture, their business impact, and contribution to compliance. We suggest that requirements engineering research can play a crucial role in better exploring and modeling such viewpoints. In practice, requirements engineering could facilitate the coordination of the involved viewpoints to enable the adoption of existing and the development of new PETs.
View details
Preview abstract
Large language models (LLMs) are trained on web-scale corpora that exhibit steep power-law distributions, in which the distribution of knowledge is highly long-tailed, with most appearing infrequently. While scaling has improved average-case performance, persistent failures on low-frequency, domain-specific, cultural, and temporal knowledge remain poorly characterized. This paper develops a structured taxonomy and analysis of long-tail knowledge in large language models, synthesizing prior work across technical and sociotechnical perspectives.
We organize the literature along four complementary axes: how long-tail knowledge is defined, the mechanisms by which it is lost or distorted during training and inference, the technical interventions proposed to mitigate these failures, and the implications of these failures for fairness, accountability, transparency, and user trust. We further examine how existing evaluation practices obscure tail behavior and complicate accountability for rare but consequential failures. The paper concludes by identifying open challenges related to privacy, sustainability, and governance that constrain long-tail knowledge representation. Taken together, this paper provides a unifying conceptual framework for understanding how long-tail knowledge is defined, lost, evaluated, and manifested in deployed language model systems.
View details
Preview abstract
This paper provides an overview of Google's TPUs across five generations, from TPU v2 to Ironwood, highlighting their evolution as scalable, resilient, and sustainable supercomputers for AI training. It details the TPU’s stable architecture and microarchitecture, which has surprisingly easily accommodated the rapidly changing deep neural network workloads, such as the rise of Transformers. Key advancements over eight years include 10x increase in HBM capacity and bandwidth per node, a 100x increase in peak node performance, and a 3600x increase in supercomputer performance. The paper also discusses the role of optical circuit switches and built-in self test in enhancing resilience, how TPU’s carbon footprint was reduced by improving embodied carbon emissions per floating point operation and a 30x gain in performance per Watt. It concludes by identifying six features that may well characterize the successful AI accelerators of this decade.
View details
Gaze Target Estimation Anywhere with Concepts
Xu Cao
Houze Yang
Vipin Gunda
Inki Kim
Jim Rehg
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2026)
Preview abstract
Estimating human gaze targets in-the-wild is a formidable challenge. Existing computer vision algorithms rely on brittle, multi-stage pipelines that require explicit inputs like head bounding boxes and human pose, causing initial detection errors to cascade and lead to system failure. To overcome this, we introduce the \textbf{Promptable Gaze Target Estimation (PGE)} task, a new end-to-end, concept-driven paradigm. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject's target, which eliminates the rigid dependency on intermediate localization cues. We develop a scalable data engine to generate \textbf{Gaze-Co}, a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose \textbf{AnyGaze}, the first model designed for PGE. AnyGaze uses a Transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. AnyGaze achieves state-of-the-art performance on standard gaze target estimation benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. We will open-source the AnyGaze model and the Gaze-Co benchmark.
View details
OBASE: Object-Based Address-Space Engineering to Improve Memory Tiering
Kan Wu
Suli Yang
Andrea Arpaci-Dusseau
Remzi Arpaci-Dusseau
Kim Keeton
USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) (2026)
Preview abstract
Hardware and OS mechanisms for memory tiering are widely deployed, yet datacenters still overprovision DRAM. The root cause is hotness fragmentation: allocators place objects by size rather than access pattern, so hot and cold objects become interleaved within the same pages. A single hot object marks its page as active, trapping surrounding cold data in expensive DRAM. Our analysis of Google datacenter workloads shows that up to 97% of the bytes in active pages are cold and unreclaimable. We propose address-space engineering: dynamically reorganizing virtual memory so that hot objects cluster into uniformly hot pages and cold objects into uniformly cold pages. We present OBASE, a compiler-runtime system for unmanaged languages that serves as an object-aware frontend for page-aware OS backends. OBASE tracks accesses via lightweight pointer instrumentation and migrates objects at runtime using a lock-free protocol that is safe under concurrency. By reorganizing the address space, OBASE enables unmodified backends (kswapd, TMO, TPP, Memtis) to tier memory effectively. Across ten concurrent data structures, six backends, and production traces from Meta and Twitter, OBASE improves page utilization by 2–4x and reduces memory footprint by up to 70%, with only 2–5% overhead.
View details
Preview abstract
Large-scale cloud-native systems generate continuous streams of operational alerts across distributed microservice architectures. On-call engineers must manually triage these alerts by correlating signals from heterogeneous observability tools, a process that is time-consuming, cognitively demanding, and prone to error. Despite advances in monitoring and anomaly detection, incident triage remains largely manual. This paper presents a declarative, large language model (LLM)–driven multi-agent approach to automating incident triage and Service Level Objective (SLO) monitoring. The proposed design constrains agent behavior using domain-expertauthored investigation workflows, enabling deterministic execution and reproducibility while preserving operational safety. The framework integrates a unified tool execution layer for interacting with diverse observability systems and an enhanced retrieval-augmented generation (RAG) pipeline optimized for operational knowledge retrieval. The approach has been evaluated in a production cloud environment spanning multiple microservices and geographic regions. Results show reductions in high-severity incident triage time from approximately 30 minutes to under 5 minutes, alert acknowledgement latency from minutes to seconds, and service onboarding effort from weeks to days. These findings suggest that constrained multi-agent systems can substantially reduce on-call cognitive load while maintaining reliability and human oversight.
View details
Learned Task Relationships in Multi-task Models
Naijing Zhang
Bosun Adeoti
Mike Leung
Florian Raudies
Yiping Yuan
Recsys (2026)
Preview abstract
We propose a framework that learns cross-task relationships in multi-task models by estimating the joint distribution of task labels. This approach improves performance via transfer learning and enhances information extraction. Although our framework applies to any multi-task system, we demonstrate its efficacy within YouTube’s production recommendation stacks. Experiments across the Notifications, Homepage, and Watch Next surfaces show improvements in both accuracy and satisfied user engagement. Finally, we propose a workflow template to facilitate broader future implementation.
View details
