Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 11644 publications
Preview abstract
Modern transformers have been scaled up in size, data, training-time compute, and more recently test-time compute based on Chain-of-Thought (CoT), but the consequences of trading off these parameters are unknown. We construct explicit tradeoffs between two of these aspects: the number of layers in a transformer and the number of CoT tokens. Our first result is general: with mild assumptions and a single-token prompt, for any integer k, a transformer using soft attention and SwiGLU MLP layers with L layers and n CoT tokens can be converted to a transformer with at most ckL layers and ⌈n/k⌉ CoT tokens, and vice versa, although with an increase to the size of other parameters. We then a multi-token prompt path following task, finding that additional preprocessing from the multi-token prompt changes the depth-CoT relationship in this case. We construct an attention-only network with L layers and 2^(L−2) heads that can traverse 2^(L−1) − 1 edges along a path with a single CoT token, demonstrating at least an exponential relationship between increases in depth and reductions in the number of CoT tokens needed to traverse the entire path. This relationship for the path following task holds empirically as well, as demonstrated by training networks with up to 5 layers, with the tradeoff between depth and the maximum number of edge traversals in a single CoT token exactly matching our theoretical constructions.
View details
When Android Apps Request Location with a Reason: A Developer Study and User Behavior Experiment
May Smith
Aman Jain
Proceedings on Privacy Enhancing Technologies (2027)
Preview abstract
Recent reports have highlighted how mobile apps share user location data with third parties, risking user privacy and platform trust. Although location data is highly sensitive, when users grant apps location access, they may not know the full extent to which it is used. We study how requiring Android apps to show a reason for location access could impact developers, users, and the platform.
We surveyed 323 Android app developers and found most supported such a requirement. The majority said it would have a positive impact on user privacy, trust for apps, and trust for Android, where impact on user trust for Android correlated most strongly with support. Many developers also said the intervention would increase the number of users granting location access. Yet their open-ended comments also revealed consistent concerns, such as apps providing dishonest reasons and platform verification.
To study the impact on user behavior, we conducted a randomized controlled experiment with 2579 US Android users. We tested how users' decisions to grant location access were impacted by app type, whether reasons were included in the requests, and the content of the reasons, including monetization. We did not find the reasons impacted users' decisions; decisions were instead driven by app type and demographics. Yet we did find the reasons could have a positive impact on user perception for the platform when the reasons did not include using data for ads. Our findings provide insights into developers' willingness to implement privacy-enhancing changes, and expose limits to improving user privacy by simply adding information to user interfaces.
View details
A 3D Scene Graphs Survey: Open Challenges and Future Directions
Dennis Rotondi
Francesco Argenziano
Sebastian Koch
Nathan Hughes
Martin Büchner
Johanna Wald
Lukas Schmid
Daniele Nardi
Abhinav Valada
Liam Paul
Luca Carlone
Kai Arras
Annual Review of Control, Robotics, and Autonomous Systems (ARCRAS), 10 (2027) (to appear)
Preview abstract
3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including mapping, task and motion planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real- world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are constructed from raw sensory observations, covering both learning-oriented and construction-oriented systems. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed works.
View details
AgentHands: Generating Interactive Hands Gestures for Spatially Grounded Agent Conversations in XR
Ziyi Liu
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, ACM
Preview abstract
Communicating spatial tasks via text or speech creates ``a mental mapping gap'' that limits an agent’s expressiveness. Inspired by co-speech gestures in face-to-face conversation, we propose \textsc{AgentHands}, an LLM-powered XR system that equips agents with hands to render responses clearer and more engaging. Guided by a design taxonomy distilled from a formative study (N=10), we implement a novel pipeline to generate and render a hand agent that augments conversational responses with synchronized, space-aware, and interactive hand gestures: using a meta-instruction, \textsc{AgentHands} generates verbal responses embedded with \textit{GestureEvents} aligned to specific words; each event specifies gesture type and parameters. At runtime, a parser converts events into time-stamped poses and motions, driving an animation system that renders expressive hands synchronized with speech. In a within-subjects study (N=12), \textsc{AgentHands} increased engagement and made spatially grounded conversations easier to follow compared to a speech-only baseline.
View details
Preview abstract
Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. Anticipating their behavior in any specific deployment is hard. Existing strategic-reasoning benchmarks evaluate models on fixed canonical games. These benchmarks may saturate as the frontier improves, and they do not allow evaluators to generalize with confidence from benchmark performance to the varied and messy strategic environments that actual deployments involve. We introduce GENSTRAT, which uses procedurally generated strategic environments to address these challenges. Concretely, we generate a distribution of two-player zero-sum imperfect-information card games. The generator can draw fresh games on demand, allowing for evergreen evaluation and resistance to contamination. We pair the game distribution with a capability-profile methodology that decomposes model competence across six axes (state space, temporal depth, information sensitivity, opponent modeling, risk, and brittleness). We also introduce a jaggedness measure of within-distribution smoothness that detects when a model’s advantage jumps unpredictably between strategically similar games. We sample 50 benchmark games from a 2,000-game generated pool and evaluate nine frontier and open-weight LLMs in a head-to-head tournament with over 36,000 matches. Newer frontier-tier models score higher on average. Beyond that average, models with near-identical overall strength show qualitatively different capability profiles, and two of the top three leaderboard models (gpt-5 and claude) are noticeably more locally volatile than the third (gemini-3.1-pro), despite being close in overall strength. Together, the capability profile and the jaggedness measure give a deployment-relevant diagnostic that the overall ranking alone cannot provide.
View details
Preview abstract
Config Driven User Interface (CDUI) frameworks, often referred to interchangeably as
Server Driven UI (SDUI) have fundamentally altered mobile and web application development. By shifting layout and orchestration logic from compiled client binaries to dynamic
server responses, organizations bypass slow app store review cycles and ensure cross-platform
consistency. Despite its widespread adoption by industry leaders, CDUI remains underexplored in formal software engineering literature. This paper investigates the evolution of
CDUI through a hybrid methodology, combining a Multivocal Literature Review (MLR) of
grey literature from organizations including Uber, Airbnb, Lyft, Spotify, Faire, and Zalando,
with a structural artifact analysis of open-source production frameworks (Yandex DivKit,
Zup Beagle, and Spotify Hub). We introduce a taxonomy grounded in established foundational UI components and evaluate systems across three axes: Modularity, Centralization,
and Strictness. By tracing the architectural evolution from simple remote configurations to
strongly typed, Protobuf-driven contracts, our empirical artifact analysis (evaluating 570
schema commits across three repositories) demonstrates the critical magnitude of schema
governance and the complex challenge of managing “contract fragility.” We further analyze
cross-cutting concerns such as security sandboxing, “BFF Bloat,” and native accessibility
mapping, culminating in a future research agenda focused on the formal verification of UI
configurations.
View details
StreamLens: An Open-Source ABR Simulation Environment for Multi-Region Buffering, Seeking, and Prefetching
Bei Zhao
Jinhui Song
Jiayu Zhu
Yih‑Chun Hu
2026
Preview abstract
ABR simulators are useful for studying ABR behavior under different network conditions. SABRE is an open-source simulator for evaluating ABR algorithms with different videos, network traces,
and ABR policies. However, SABRE has several limitations. First, it does not accurately emulate user seeks or their impact on the viewing experience. Second, it provides limited support for nonlinear buffering, where the client buffers non-contiguous video chunks. Furthermore, SABRE does not provide adequate analysis and visualization support for interpreting simulation results. This paper presents StreamLens, an advanced video simulator that supports: 1) pluggable ABR policies, including BOLA, BOLA-E, Dynamic, DynamicDash, and throughput-based policy; 2) configurable user seeks; 3) nonlinear buffering with customizable prefetching policy; 4) browser-based viewer for result visualization, including
stalls, seeks, rebuffering time, QoE, bitrate, buffer, etc. StreamLens allows developers to configure videos, network traces, and seek patterns, and to evaluate ABR performance across different setups, including ABR policies, linear versus nonlinear buffering, and user seek traces. It also provides metrics and visualizations that help developers inspect streaming behavior in a straightforward way. Several predefined scenarios are included as both evaluation examples and configuration templates, covering synthetic seek and prefetch settings, session-replay workflows, generated reports, and regression tests. The software is available publicly at
https://github.com/Slothbetty/StreamLens.
View details
FabScore: Fine-Grained Evaluation of Fabrications in Automated AI Research
James Xu Zhao
See-Kiong Ng
Dongfu Jiang
Bryan Hooi
Qianyun Guo
Hui Chen
Pang Wei Koh
Muhao Chen
Yiwei Wang
2026
Preview abstract
In automated AI research, scientific rigor is not merely a matter of producing coherent papers and executable code; rather, it fundamentally requires that the experimental results claimed in a paper are faithfully supported by the accompanying implementation and verifiable through actual execution logs. When this alignment breaks down, AI-generated research may contain fabrications: discrepancies between the methods described in the paper and what the code actually implements, or between the reported results and those obtained by running the code. This motivates a systematic evaluation of fabrications that systematically examines the consistency between a paper's claimed experimental outcomes and its accompanying code files. In automated AI research, scientific rigor is not merely a matter of producing coherent papers and executable code; rather, it fundamentally requires that the experimental results claimed in a paper are faithfully supported by the accompanying implementation and verifiable through actual execution logs. When this alignment breaks down, AI-generated research may contain fabrications: discrepancies between the methods described in the paper and what the code actually implements, or between the reported results and those obtained by running the code. This motivates a systematic evaluation of fabrications that systematically examines the consistency between a paper's claimed experimental outcomes and its accompanying code files. This is a placeholder. This is a placeholder. This is a placeholder.
View details
Working Together on Technologies: A Case Study of Collaboration in Aotearoa
Gabi Conlon
Greg Duncum
Jamie Towler
Carrie Jones
Tane Karamaina
Hautahi Kingi
Caroline Rainsford
Ngahiwi Apanui-Barr
2026
Preview abstract
This paper presents a case study in collaboration on TTS technologies between Te Taura Whiri and Google. The former is a Māori-led organisation whose mission is the revitalisation of te reo Māori. The latter is a global technology company which has a presence in Aotearoa but which is headquartered elsewhere. This case study focuses not on technical implementation details, but rather on the challenges in the partnership to mutual understanding and alignment, and the processes we co-developed to overcome those challenges, which led to the successful development of technology which was satisfactory to both partners.
View details
OpenClaw in the Wild: Security Analysis of Autonomous Agents
Wanlun Ma
Qing-Long Han
Xiaogang Zhu
Wei Zhou
Junwu Xiong
Peter Ren
Sheng Wen
Yang Xiang
IEEE/CAA Journal of Automatica Sinica, 13 (2026), pp. 1257 - 1273
Preview abstract
Autonomous self-hosted AI agent platforms are rapidly evolving from prompt-response assistants into persistent systems that can maintain long-lived state, invoke tools, ingest external content, and execute environment-changing actions. While this transition enables practical automation, it also introduces lifecycle security risks that cannot be fully explained by prompt-level analysis alone. In this paper, we present a security analysis of OpenClaw as a representative autonomous agent operating environment. We further frame OpenClaw as a concrete case study for broader security challenges in emerging agent ecosystems. We adopt a trust-boundary-first perspective and analyze how attacks propagate across five boundary classes: Channel-Access, Session-and-State, Tool-Execution, External-Content, and Extension Supply-Chain. Our results show that threats such as indirect prompt injection, memory poisoning, unsafe tool invocation, data exfiltration, and malicious skill abuse are not isolated anomalies; they are stage-specific manifestations of a common systems problem in which untrusted influence progressively crosses into higher-privilege contexts. Building on this analysis, we discuss defense-in-depth implications for OpenClaw deployments, including boundary-aware isolation, capability-scoped tool mediation, memory integrity controls, extension governance, and evidence-oriented operational oversight. The study provides a practical framework for evaluating and hardening long-running, tool-capable, autonomous AI agents in realistic deployment settings.
View details
Preview abstract
One of the most exciting developments in the time series space over the last few years has been the advances in probabilistic forecasting capabilities, both in terms of modeling approaches and available tools. Yet adoption of probabilistic forecasting methods has been slow amongst the wider forecasting community, and data science teams and business stakeholders are missing out on the opportunities provided by more accurate characterization of the uncertainty, including their correlations.
In this presentation, we will point out that most established point forecasting approaches can be easily augmented to produce distributional forecasts, making it very hard to distinguish between the two types of approaches in practice. The gravitation towards using a single number as a forecast due to its explainability and operability makes the concrete benefits and ROI of upgrading to probabilistic forecasting approaches ambiguous at best, thus contributing to the challenges with adopting the newer probabilistic models.
We will then argue for a reframing of the probabilistic vs. point forecasting distinction, not in terms of modeling approaches and pipeline configurations, but as a fundamental first principles based distinction based on the available data and the use case requirements. Probabilistic forecasting should not be seen as an add-on to point forecasting, nor as an opportunity for an incremental improvement on uncertainty bounds. It should be clear that certain applications mandate probabilistic forecasting because point forecasting, no matter how accurate, will lead to wrong decision making. This distinction should be analogous to how the distinction between regression and classification use cases is defined, or the distinction between clustering and dimensionality reduction use cases is defined.
We will also discuss possible criteria for this first principles based distinction between point forecasting use cases and probabilistic use cases, and argue for why such a reframing might be beneficial for practitioners and business users. This article aims to initiate a discussion on making probabilistic forecasting as a principle that is more accessible to practitioners and non-technical stakeholders; that it is not an improvement to point forecasts: it is a critical must-have for many (or most) business forecasting use cases.
View details
Neural general circulation models for modeling precipitation
Stephan Hoyer
Dmitrii Kochkov
Janni Yuval
Ian Langmore
Science Advances (2026)
Preview abstract
Climate models struggle to accurately simulate precipitation, particularly extremes and the diurnal cycle. While hybrid models combining machine learning and physics have emerged with the premise of improving precipitation simulations, none have proven sufficiently skillful or stable enough to outperform existing models in simulating precipitation.
Here, we present the first hybrid model that is trained directly on precipitation observations. The model runs at 2.8 degrees resolution and is built on the differentiable NeuralGCM framework. This model is stable for decadal simulations and demonstrates significant improvements over existing GCMs, ERA5 reanalysis, and a Global Cloud-Resolving Model in simulating precipitation. Our approach yields reduced biases, a more realistic precipitation distribution, improved representation of extremes, and a more accurate diurnal cycle.
Furthermore, it outperforms the ECMWF ensemble for mid-range weather forecasting.
This advance paves the way for more reliable simulations of current climate and for the ability to fully utilize the abundance of existing observations to further improve GCMs.
View details
Preview abstract
We consider a setting where we have a ground set ℳ together with real-valued set functions f₁, … , f_n, and the goal is to partition ℳ into two sets S₁,S₂ such that |f_i(S₁) - f_i(S₂)| is small for every i. Many results in discrepancy theory can be stated in this form with the functions f_i being additive. In this work, we initiate the study of the unstructured case where f_i is not assumed to be additive. We show that even without the additivity assumption, the upper bound remains at most O(√{n log n}).
Our result has implications on the fair allocation of indivisible goods. In particular, we show that a consensus halving up to O(√{n log n}) goods always exists for n agents with monotone utilities. Previously, only an O(n) bound was known for this setting.
View details
FUSE: Scaling Verification with Zero Labeled Data
Asher Spector
Joonhyuk Lee
Emmanuel Candès
Virginia Ma
Sarah Zhao
Regev Cohen
Yash Nair
2026
Preview abstract
Verification is rapidly emerging as a key primitive for both training and real-world deployment of large language models (LLMs). In practice, this often involves using imperfect LLM judges and reward models, as ground truth acquisition can be time-consuming and expensive. This naturally motivates asking if verification can be improved by leveraging multiple verifiers, and whether such ensembling can be done without itself requiring ground truth labels. We introduce Fully Unsupervised Score Ensembling (FUSE ), a method for ensembling verifiers that is (i) completely unsupervised and (ii) can operate on a query-conditional basis. Our main insight is that when conditional dependencies between verifiers are carefully controlled, certain spectral algorithms in the ensembling literature enjoy strong unsupervised guarantees. Despite requiring zero ground truth labels, FUSE matches or improves upon semi-supervised alternatives in test-time scaling experiments with diverse sets of generator models, verifiers, and benchmarks.
View details
Towards expert-level medical AI for real-time video consultations
Mahvish Nagda
Jihyeon Lee
Matthew Thompson
CJ Park
Tim Strother
Roma Ruparel
Teya Bergamaschi
Suhana Bedi
Meet Shah
Pavel Dubov
Toshiyuki Fukuzawa
Sam Schmidgall
Craig Schiff
Joseph Xu
Aliya Rysbek
Yana Lunts
Jan Freyberg
Rebecca Hemenway
David Racz
Carey Radebaugh
Joelle Barral
Kavi Goel
Kat Chou
James Manyika
Gregory Wayne
Yun Liu
Ethan Goh
Christina Chen
Ryutaro Tanno
arXiv (2026)
Preview abstract
Audio-visual interaction is the standard for patient-physician consultations, enabling natural communication and effective assessment of illness through non-verbal cues. While text-based AI has shown promise, it discards essential perceptual dimensions and limits patients who cannot articulate symptoms in writing. Early efforts to extend medical AI to audio-visual interaction have demonstrated feasibility, but not reached clinician-level performance. Here, we provide the first demonstration of expert-level AI in real-time clinical video consultations using AMIE (Articulate Medical Intelligence Explorer) in a video configuration. AMIE (Video) is a Gemini-based multi-agent system integrating low-latency dialogue, clinical reasoning, and real-time audio-visual perception. To guide development, we established a taxonomy and automated evaluations for clinical audio-visual cues in telehealth settings. In a randomized Objective Structured Clinical Examination (OSCE) study with 30 primary care physicians (PCPs), 15 patient actors and 100 clinical scenarios, we compared AMIE (Video), its text-only counterpart AMIE (Text), and PCPs consulting via video. Clinical evaluators rated AMIE (Video) on par or better than PCPs in history-taking, diagnosis, management, and physical observation and examination. Patient actors preferred AMIE's approach to assessing and explaining conditions, while PCPs were preferred for rapport and partnership building. In modality ablation, patient actors preferred AMIE (Video)'s interface over text chat for communicative effectiveness, convenience, and feeling understood. Limitations remain in fine anatomical precision, subtle affective nuances, and high-frequency movements. While further research is needed before real-world translation, these results mark an important milestone toward AI systems capable of augmenting care across the sensory complexity of clinical practice.
View details
