Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 11636 publications
A 3D Scene Graphs Survey: Open Challenges and Future Directions
Dennis Rotondi
Francesco Argenziano
Sebastian Koch
Nathan Hughes
Martin Büchner
Johanna Wald
Lukas Schmid
Daniele Nardi
Abhinav Valada
Liam Paul
Luca Carlone
Kai Arras
Annual Review of Control, Robotics, and Autonomous Systems (ARCRAS), 10 (2027) (to appear)
Preview abstract
3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including mapping, task and motion planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real- world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are constructed from raw sensory observations, covering both learning-oriented and construction-oriented systems. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed works.
View details
When Android Apps Request Location with a Reason: A Developer Study and User Behavior Experiment
May Smith
Aman Jain
Proceedings on Privacy Enhancing Technologies (2027)
Preview abstract
Recent reports have highlighted how mobile apps share user location data with third parties, risking user privacy and platform trust. Although location data is highly sensitive, when users grant apps location access, they may not know the full extent to which it is used. We study how requiring Android apps to show a reason for location access could impact developers, users, and the platform.
We surveyed 323 Android app developers and found most supported such a requirement. The majority said it would have a positive impact on user privacy, trust for apps, and trust for Android, where impact on user trust for Android correlated most strongly with support. Many developers also said the intervention would increase the number of users granting location access. Yet their open-ended comments also revealed consistent concerns, such as apps providing dishonest reasons and platform verification.
To study the impact on user behavior, we conducted a randomized controlled experiment with 2579 US Android users. We tested how users' decisions to grant location access were impacted by app type, whether reasons were included in the requests, and the content of the reasons, including monetization. We did not find the reasons impacted users' decisions; decisions were instead driven by app type and demographics. Yet we did find the reasons could have a positive impact on user perception for the platform when the reasons did not include using data for ads. Our findings provide insights into developers' willingness to implement privacy-enhancing changes, and expose limits to improving user privacy by simply adding information to user interfaces.
View details
Proactive Heterogeneous Interconnect Arbitration via ML-Aware Compiler Hinting
Prakash Viswalingam
2026
Preview abstract
Modern heterogeneous Systems-on-Chip (SoCs) integrate highly diverse intellectual property (IP) blocks, including latency-sensitive CPUs, Throughput-intensive GPUs, real-time streaming display/camera engines, and massive parallel Neural Processing Units (NPUs/TPUs). Arbitrating access to a shared Network-on-Chip (NoC) and memory subsystem under such diverse traffic profiles expose a critical structural bottleneck. Traditional reactive Quality of Service (QoS) and Dynamic Voltage and Frequency Scaling (DVFS) frameworks incur substantial performance and energy penalties due to phase-locked loop (PLL) lag and transient queue saturation. This paper introduces a Compiler-Assisted Interconnect Hinting (CAIH) framework that eliminates reactive latency penalties by exploiting the deterministic nature of deep learning execution graphs. We propose a lightweight Instruction Set Architecture (ISA) extension (ICH) that allows the deep learning compiler to proactively signal upcoming memory phases, operational intensity shifts, and lookahead cycles to a centralized System QoS (SQoS) arbiter via a dedicated physical sideband channel. Experimental results extracted from cycle-accurate gem5/Garnet simulations demonstrate that our proactive framework reduces average CPU memory latency by 22%, completely eliminates display buffer underflows under heavy concurrent workloads, and achieves up to a 15% reduction in LPDDR6 memory subsystem power.
View details
Nine changes needed to deliver a radical transformation in biodiversity measurement
Neil Burgess
Andy Purvis
Scott J. Goetz
William Sutherland
Anil Madhavapeddy,
Tanya Birch
PNAS Perspective (2026)
Preview abstract
Biodiversity is declining in many parts of the world. The measurement and monitoring of
biological diversity are fundamental to the assessment of the causes and consequences of
environmental changes, identification of key areas for the protection of biodiversity or
ecosystem services, determining the effectiveness of actions, and the creation of
decision-support tools critical to the maintenance of a sustainable planet. The measurement
of biodiversity is rapidly changing due to advances in citizen science, image recognition,
acoustic monitoring, environmental DNA, genomics, remote sensing and artificial
intelligence. In this perspective, we outline the exciting opportunities that these
developments offer, but also consider the challenges, especially the potential poisoning of
data by AI, lack of standardisation across methods, coverage gaps in data, concerns over
losing databases, and undervaluing of on-the-ground expertise and data-generation. Our key
recommendations are (1) ensure new technologies are calibrated with existing data; (2) use
emerging technologies to fill data gaps; (3) create living databases of trusted information to
increase reliability of data and reduce the risk of poisoning by false - or AI hallucinated -
information; (4) ensure data generation is valued; (5) ensure the respect and incorporation
of Indigenous Knowledge; (6) increase in-country capacity in the tropics; and (7) increase the
resilience of global datasets to technical and societal change. Radical new collaborations are
needed between computer scientists, engineers, molecular biologists, data scientists, field
ecologists, citizen scientists, Indigenous peoples, and local communities to create the
rigorous, resilient, accessible biodiversity information systems required to underpin policies
and practices that ensure the maintenance and restoration of ecological systems.
View details
Silicon-Level Sovereignty: Root of Trust in AI Accelerators (Digital Trust & Policy)
https://www.dotmagazine.online (2026)
Preview abstract
As artificial intelligence (AI) transitions from experimental pilot programs to mission-critical enterprise operations, traditional software-based security frameworks are proving insufficient against sophisticated infrastructure-level threats. This article introduces the concept of Silicon-Level Sovereignty, a first-principles approach to digital trust that anchors security in the physical hardware rather than the software stack.
We examine the technical architecture of Hardware Root of Trust (RoT), specifically focusing on the roles of Trusted Platform Modules (TPMs) and Secure Enclaves in modern AI accelerators such as GPUs and TPUs. By leveraging cryptographic remote attestation, organizations can move from a model of assumed software integrity to one of verifiable hardware-level proof.
The discussion provides a comparative analysis of industry-leading implementations, including NVIDIA’s Hopper architecture [1, 2], Google’s Titan-backed TPU v5p [3, 4], and Microsoft’s Azure Boost Cerberus system [5, 6], alongside the cluster-scale trust challenges presented by ultra-large systems like xAI’s Colossus [7].
The article concludes that Silicon-Level Sovereignty is no longer an optional security feature but a foundational requirement for establishing the integrity, privacy, and multi-tenant isolation necessary for high-stakes AI workloads.
View details
Preview abstract
We introduce AMS (Activation-based Model Scanner), a tool that detects modifications to safety training in language models by measuring the geometric structure of safety-relevant concepts in activation space. Safety training creates measurable separation between harmful and benign content classes; certain safety modifications collapse or rotate this structure, while others leave it intact. We validate AMS across 14 model configurations spanning 4 architecture families (Llama, Gemma, Qwen, Mistral) and four safety-modification categories (instruction-tuned, base, abliterated, uncensored fine-tunes). Leave-one-out cross-validation of thresholds achieves 71% accuracy (10/14); bootstrap 95% confidence intervals on σ point estimates have median width 3.4σ and a substantial fraction of cells cross the PASS threshold under resampling. We further measure behavioral compliance on 20 stratified JailbreakBench prompts per model and find that σ on the harmful-content concept predicts compliance with Pearson r=−0.546 ( p=0.043 ); the rank-order Spearman correlation is weaker ( ρ=−0.423 , p=0.13 ). The structural signal predicts behavior directionally but with meaningful noise. Mechanistic analysis identifies a four-class taxonomy of safety-training modifications distinguished by activation-space signature: 1) training removal collapses cluster separation (e.g., base models, Dolphin variants: 0.5– 1.4σ ); 2) weight-orthogonalization-style abliteration both collapses separation and rotates the refusal direction (Llama-3.1-abliterated: σ=3.33 , direction cos sim 0.30); 3) rotation-without-collapse abliteration preserves cluster separation while rotating the refusal direction (Gemma-2-9b-abliterated: σ=4.54 , direction cos sim 0.84); and 4) behavioral fine-tuning that preserves both magnitude and direction (DarkIdol-1.2-Uncensored: σ=5.45 , direction preserved, 97% behavioral compliance). 1) and 2) AMS’s Tier 1 σ -threshold detects classes; 3) Tier 2 direction-similarity verification detects class; and 4) Class is undetectable by activation-only probing and represents a documented failure mode of the approach. We discuss threshold calibration, limitations of single-run measurement, and the open problem of detecting behavioral-only safety modifications.
View details
Preview abstract
Coding agents are rapidly changing the landscape of software development, moving from inline com- pletion to autonomous systems that edit repositories, open pull requests, respond to issues, and run scheduled or webhook triggered routines across the development life cycle. The next generation is increasingly described as proactive and long-horizon: agents should notice relevant changes before the developer asks, connect signals across tools, decide when to interrupt, and carry preferences across sessions. Yet the field lacks a precise account of what proactivity means for software development, how it differs from autonomy, what acceptance criteria proactive long-horizon tasks should satisfy, and which metrics determine whether unsolicited agent behavior is useful rather than merely active. We argue that proactive coding agents should be evaluated by the quality and improvement of their insight policy: the policy that decides what matters next, what evidence supports it, whether to surface it, and how to adapt after feedback. We re-anchor this view in mixed initiative interaction, introduce a three level taxonomy (Reactive, Scheduled, and Situation Aware), compare contemporary coding agents against five operational criteria, and sketch an active user simulation protocol with three evaluation targets: Insight Decision Quality (IDQ), Context Grounding Score (CGS), and Learning Lift (LL).
View details
When “Correct” Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?
Corina Pasareanu
Haizhong Zheng
Beidi Chen
Ravi Mangal
Xinyu Yang
James Song
Yibo Peng
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, San Diego, California, United States (2026), 15514–15546
Preview abstract
Code agents are increasingly trusted to autonomously fix bugs on platforms such as GitHub, yet their security evaluation focuses almost exclusively on functional correctness. In this paper, we reveal a novel type of threat to real-world code agents: Functionally Correct yet Vulnerable (FCV) patches, which pass all test cases but contain vulnerable code. With our proposed FCV-Attack, which can be deliberately crafted by malicious attackers or implicitly introduced by benign developers, we show that SOTA LLMs (e.g., ChatGPT and Claude) and agent scaffolds (e.g., SWE-agent and OpenHands) are all vulnerable to this FCV threat; across 12 agent-model combinations on SWE-Bench, the attack only requires black-box access and a single query to the code agent to perform the attack. For example, for CWE-538 (information exposure vulnerability), the FCV-Attack attains an attack success rate of 40.7% on GPT-5 Mini + OpenHands. Our results reveal an important security threat overlooked by current evaluation paradigms and urge the development of security-aware defenses for code agents.
View details
Toward a test of medical AI superintelligence
Ethan Goh
David Wu
Chase Walton
Liam McCoy
Anastasia Perez
Laura Wegner
Fateme Nateghi Haredasht
Luyang Luo
Kathleen Lacar
Thomas Buckley
Austin Schoeffler
Peter Brodeur
Kameron C. Black
John Havlik
John Rumsfeld
Daniel Lopez-martinez
Paxton Maeder-York
Karan Singhal
David Gunning
Bon Ku
Haider Warraich
Shantanu Nundy
Vishnu Ravi
Arnold Milstein
Jason Hom
Kevin Schulman
Pranav Rajpurkar
Arjun Manrai
Robert Wachter, MD
Eric Topol
Eric horvitz
Adam Rodman
Jonathan Chen
Nature Medicine (2026)
Preview abstract
Researchers urgently need a rigorous, task-based framework to define and measure medical AI ‘superintelligence’, because existing benchmarks are misleading and insufficient.
View details
Preview abstract
Distributed federated SQL query engines frequently
materialize query results into files in data lakes. Optimizing the
sizes of these files (e.g., balancing their sizes) is crucial for the
efficiency of not only the materialization queries themselves, but
also subsequent queries that read these files. Existing techniques
to manage file size include specifying output targets (e.g., number
of partitions), using cardinality estimation for materialized data
volume, performing full shuffles prior to materialization to obtain
accurate statistics, and/or post (background) compactions to
merge small files. All of these solutions have limitations in
practice; they typically do not provide any strong guarantees or
can be prohibitively expensive when they do (e.g., they require
background compactions to merge small files, which requires
additional I/O and CPU cost). Most of the existing techniques
can produce many small files during materialization and are
sensitive to data skew.
In this paper, we propose a novel method that streamlines
materialization within the same query execution and provides
guarantees on the file sizes using statistics at runtime. Our
approach does not require a full shuffle before the materialization
operator in order to get accurate statistics, making it attractive
and robust in practice. To show the effectiveness of our approach,
we present production metrics from F1 Query at Google that has
been running this functionality for the majority of its production
workload over many quarters. Our approach reduces the number
of files produced by a factor of 100 or more, and as a result,
it avoids creating tens of billions of files per day, saving storage
and computation cost.
View details
Preview abstract
The Private-Use Area (PUA) is an important part of the Unicode standard. It consists of several ranges of Unicode code points with no official character assignments. The PUA is primarily used as a temporary representation mechanism for characters outside the official standard to facilitate text entry and display of orthographies that cannot be adequately represented by other means. The primary downside of PUA is that characters lose their semantics if the pairing with the corresponding display font is broken. Consequently, they cannot be faithfully displayed in the general setting. Large-scale multilingual web corpora inevitably contain PUA code points of unclear provenance. We investigate the distribution of PUA characters within large-scale datasets, using filters for determining PUA tokens of linguistic interest. We analyze the resulting distributions both across scripts and writing systems, and show that PUA-bearing tokens can signal texts from under-represented languages. We explore whether an off-the-shelf large language model (LLM) can classify PUA characters as those that constitute relevant orthographic signals vs. punctuation or other noise. While the proportion of PUA-bearing paragraphs in the original corpora are small, we identify millions of paragraphs, and we argue that such data is still important for the long tail of data-scarce orthographies. Moreover, as a primary Unicode mechanism for poorly represented writing systems, the PUA is here to stay.
View details
AI-Simulated Expert Panels for Socio-Technical Scenarios and Decision Guidance
Andrew G. Ross
Alan Ross
(2026) (to appear)
Preview abstract
Socio-technical scenarios for net-zero and other transformation pathways combine qualitative storylines with quantitative models, embedding them in plausible societal contexts for model assessment. Conventional scenario generation is resource-intensive, can be limited in internal consistency and diversity of expert and stakeholder perspectives, and is rarely stress-tested.
This paper introduces a synthetic, AI-based expert panel to address these bottlenecks. An AI model first simulates domain experts who agree on descriptors, states, and their interactions. A probabilistic Cross-Impact Balance analysis then generates internally consistent pathways, using stochastic shocks to assess robustness and pathway diversity. An AI stakeholder panel
uses multi-criteria decision analysis to select a preferred pathway; an AI expert panel translates it into model-ready quantitative inputs. Although scalable and applicable to any other country or region, the framework is applied to Germany's energy transition as a proof of concept, and offers an alternative and/or supplement to scenario generation. Furthermore, it enables Virtual AI-Led Decision Laboratories for exploratory policy stress-testing and provides an approach for rapid, structured expert elicitation and decision support in other domains.
View details
Gaze Target Estimation Anywhere with Concepts
Xu Cao
Houze Yang
Vipin Gunda
Inki Kim
Jim Rehg
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2026)
Preview abstract
Estimating human gaze targets in-the-wild is a formidable challenge. Existing computer vision algorithms rely on brittle, multi-stage pipelines that require explicit inputs like head bounding boxes and human pose, causing initial detection errors to cascade and lead to system failure. To overcome this, we introduce the \textbf{Promptable Gaze Target Estimation (PGE)} task, a new end-to-end, concept-driven paradigm. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject's target, which eliminates the rigid dependency on intermediate localization cues. We develop a scalable data engine to generate \textbf{Gaze-Co}, a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose \textbf{AnyGaze}, the first model designed for PGE. AnyGaze uses a Transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. AnyGaze achieves state-of-the-art performance on standard gaze target estimation benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. We will open-source the AnyGaze model and the Gaze-Co benchmark.
View details
Neural general circulation models for modeling precipitation
Stephan Hoyer
Dmitrii Kochkov
Janni Yuval
Ian Langmore
Science Advances (2026)
Preview abstract
Climate models struggle to accurately simulate precipitation, particularly extremes and the diurnal cycle. While hybrid models combining machine learning and physics have emerged with the premise of improving precipitation simulations, none have proven sufficiently skillful or stable enough to outperform existing models in simulating precipitation.
Here, we present the first hybrid model that is trained directly on precipitation observations. The model runs at 2.8 degrees resolution and is built on the differentiable NeuralGCM framework. This model is stable for decadal simulations and demonstrates significant improvements over existing GCMs, ERA5 reanalysis, and a Global Cloud-Resolving Model in simulating precipitation. Our approach yields reduced biases, a more realistic precipitation distribution, improved representation of extremes, and a more accurate diurnal cycle.
Furthermore, it outperforms the ECMWF ensemble for mid-range weather forecasting.
This advance paves the way for more reliable simulations of current climate and for the ability to fully utilize the abundance of existing observations to further improve GCMs.
View details
