Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 11620 publications
When Android Apps Request Location with a Reason: A Developer Study and User Behavior Experiment
May Smith
Aman Jain
Proceedings on Privacy Enhancing Technologies (2027)
Preview abstract
Recent reports have highlighted how mobile apps share user location data with third parties, risking user privacy and platform trust. Although location data is highly sensitive, when users grant apps location access, they may not know the full extent to which it is used. We study how requiring Android apps to show a reason for location access could impact developers, users, and the platform.
We surveyed 323 Android app developers and found most supported such a requirement. The majority said it would have a positive impact on user privacy, trust for apps, and trust for Android, where impact on user trust for Android correlated most strongly with support. Many developers also said the intervention would increase the number of users granting location access. Yet their open-ended comments also revealed consistent concerns, such as apps providing dishonest reasons and platform verification.
To study the impact on user behavior, we conducted a randomized controlled experiment with 2579 US Android users. We tested how users' decisions to grant location access were impacted by app type, whether reasons were included in the requests, and the content of the reasons, including monetization. We did not find the reasons impacted users' decisions; decisions were instead driven by app type and demographics. Yet we did find the reasons could have a positive impact on user perception for the platform when the reasons did not include using data for ads. Our findings provide insights into developers' willingness to implement privacy-enhancing changes, and expose limits to improving user privacy by simply adding information to user interfaces.
View details
A 3D Scene Graphs Survey: Open Challenges and Future Directions
Dennis Rotondi
Francesco Argenziano
Sebastian Koch
Nathan Hughes
Martin Büchner
Johanna Wald
Lukas Schmid
Daniele Nardi
Abhinav Valada
Liam Paul
Luca Carlone
Kai Arras
Annual Review of Control, Robotics, and Autonomous Systems (ARCRAS), 10 (2027) (to appear)
Preview abstract
3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including mapping, task and motion planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real- world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are constructed from raw sensory observations, covering both learning-oriented and construction-oriented systems. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed works.
View details
Preview abstract
Block partitioning framework is a core component in any modern video coding standard, as it directly determines the block size used for predictions and transforms. Flexible block partitioning plays a crucial role in the compression efficiency of these standards. This paper provides a technical overview of the block partitioning framework in AV2 video codec, developed by Alliance for Open Media. Partitioning scheme for both coding blocks and transform blocks has been redesigned in AV2. Coding block partitioning is fully recursive with newly designed partitioning options. Also, newly introduced Semi-Decoupled Partitioning (SDP) option provides additional flexibility by allowing luma and chroma components to have decoupled coding block partition trees. On the other hand, the transform block partitioning has been redesigned to use single-level partitioning with more partitioning options. In this paper, we provide a technical overview of the block partitioning framework in AV2 and also provide tool-off test results for several block partitioning aspects.
View details
Preview abstract
Cinematography, the craft of visual storytelling through framing, lighting, and camera operation, fundamentally shapes how audiences perceive and emotionally engage with video content. While Large Vision Language Models (LVLMs) have made remarkable progress in video question answering, existing benchmarks primarily focus on identifying low-level techniques rather than understanding their storytelling impact. To address this, we introduce CinematicVQA, the first-of-its-kind benchmark for cinematic video understanding that goes beyond technique recognition to evaluate film-grammar reasoning, utilizing our introduced Cinematic Scene Graph (CSG), a structured representation that links filming techniques to their perceptual effects and narrative functions. Through comprehensive evaluation of state-of-the-art LVLMs, we reveal a striking semantic gap: models consistently perform higher on describing visual presentations than on identifying the underlying techniques. Surprisingly, Chain-of-Thought prompting fails to provide consistent gains and degrades performance for most models, suggesting that current LVLMs lack sufficient cinematic domain knowledge to benefit
from step-by-step reasoning. Fine-tuning on CINEMATICVQA-TRAIN yields consistent improvements, particularly for narrative function and multi-hop reasoning. Overall, CinematicVQA serves both as a rigorous benchmark for cinematic evaluation in LVLMs and as a practical dataset for training more film-aware video models.
View details
Preview abstract
In today's AI landscape, success depends not just on prompting large language models but on orchestrating them into intelligent systems that are scalable, compliant, and cost-effective. GenAI on Google Cloud is your hands-on guide to bridging that gap. Whether you're an ML engineer or an enterprise leader, this book offers a practical game plan for taking agentic systems from prototype to production.
Written by practitioners with deep experience in AgentOps, data engineering, and GenAI infrastructure, this guide takes you through real-world workflows from data prep and deployment to orchestration and integration. With concrete examples, field-tested frameworks, and honest insights, you'll learn how to build agentic systems that deliver measurable business value.
> Bridge the production gap that stalls 90% of vertical AI initiatives using systematic deployment frameworks
> Navigate AgentOps complexities through practical guidance on orchestration, evaluation, and responsible AI practices
> Build robust multimodal systems for text, images, and video using proven agent architectures
> Optimize for scale with strategies for cost management, performance tuning, and production monitoring
View details
Preview abstract
We revisit the distributed counting problem, where a server must continuously approximate the total number of events occurring across $k$ sites while minimizing communication. The communication complexity of this problem is known to be $\Theta(\frac{k}{\eps}\log N)$ for deterministic protocols. Huang, Yi, and Zhang (2012) showed that randomization can reduce this to $\Theta(\frac{\sqrt{k}}{\eps}\log N)$, but their analysis is restricted to the oblivious setting, where the stream of events is independent of the protocol's outputs.
Xiong, Zhu, and Huang (2023) presented a robust protocol for distributed counting that removes the oblivious assumption. However, their communication complexity is suboptimal by a polylog(k) factor and their protocol is substantially more complex than the oblivious protocol of Huang et al. (2012). This left open a natural question: could it be that the simple protocol of Huang et al. (2012) is already robust?
We resolve this question with two main contributions. First, we show that the protocol of Huang et al. (2012) is itself not robust by constructing an explicit adaptive attack that forces it to lose its accuracy. Second, we present a new, surprisingly simple, robust protocol for distributed counting that achieves the optimal communication complexity of $O(\frac{\sqrt{k}}{\eps} \log N)$. Our protocol is simpler than that of Xiong et al. (2023), perhaps even simpler than that of Huang et al. (2012), and is the first to match the optimal oblivious complexity in the adaptive setting.
View details
Sexual dimorphism in the complete connectome of the Drosophila male central nervous system
Stuart Berg
Isabella R Beckett
Marta Costa
Philipp Schlegel
Elizabeth C Marin
Aljoscha Nern
Stephan Preibisch
Wei Qiu
Shin-ya Takemura
Andrew Champion
Reed A. George
Gary Huang
William Katz
Christopher Ordish
Ken Hayworth
Eric Trautman
Vivek Jayaraman
Wyatt Korff
Geoffrey W Meissner
Sandro Romani
Jan Funke
Christopher Knecht
Stephan Saalfeld
Louis Scheffer
Scott Waddell
Gwyneth Card
Carlos Ribeiro
Michael B. Reiser
Harald Hess
Gerry Rubin
Gregory S.X.E. Jefferis
Cell (2026)
Preview abstract
Sex differences in behaviour exist across all animals, typically under strong genetic regulation. In Drosophila, fruitless/doublesex transcription factors can identify dimorphic neurons but their organisation into functional circuits remains unclear.
We present the connectome of the entire Drosophila male central nervous system. This contains 166,691 neurons spanning the brain and nerve cord, fully proofread and annotated including fruitless/doublesex expression and 11,691 types. We provide the first comprehensive comparison between male and female brain connectomes to synaptic resolution, finding 7,205 isomorphic, 114 dimorphic, 262 male-specific and 69 female-specific types.
This resource enables analysis of full sensory-to-motor circuits underlying complex behaviours and the impact of dimorphic elements. Sex-specific/dimorphic neurons are concentrated in higher brain centres while the sensory and motor periphery are largely isomorphic. Within higher centres, male-specific connections are organised into hotspots defined by male-specific neurons or arbours. Numerous circuit switches reroute sensory information to form antagonistic circuits controlling opposing behaviours.
(Full author list included with the paper.)
View details
Preview abstract
This paper demonstrates that artificial intelligence can accelerate mathematical discovery by autonomously solving an open problem in theoretical physics. We present a neuro-symbolic system, combining the Gemini Deep Think large language model with a systematic Tree Search (TS) framework and automated numerical feedback, that successfully derived novel, exact analytical solutions for the power spectrum of gravitational radiation emitted by cosmic strings. Specifically, the agent evaluated the core integral for arbitrary loop geometries, directly improving upon recent AI-assisted attempts that only yielded partial asymptotic solutions. To substantiate our methodological claims regarding AI-accelerated discovery and to ensure transparency, we detail system prompts, search constraints, and intermittent feedback loops that guided the model. The agent identified a suite of 6 different analytical methods, the most elegant of which expands the kernel in Gegenbauer polynomials to naturally absorb the integrand's singularities. The methods lead to an asymptotic result for at large that both agrees with numerical results and also connects to the continuous Feynman parameterization of Quantum Field Theory. We detail both the algorithmic methodology that enabled this discovery and the resulting mathematical derivations.
View details
An Overview of Quantization in AV2
Joe Young
Jin Zhao
In Suk Chong
Todd Nguyen
Wan-Teh Chang
Madhu Krishnan
Ryan Lei
Alican Nalci
Xiaoqing Zhu
Joel Sole
Hilmi Eglimez
Luhang Xu
Yue Yu
Dong Wang
(2026)
Preview abstract
This paper presents an overview of the quantization techniques used in the AV2 video coding standard. For scalar quantization, an improved design consolidates six lookup tables into a simplified function where the step size doubles with every 24 quant index increments. Support for user-defined quantization matrices can be signaled in the sequence header, enabling segment-level matrix selection. Moreover, advanced quantization methods are described, including a parity-hiding method which can implicitly signal the parity of the DC (index 0,0) coefficient, based on the sum of the AC coefficients, thereby reducing the number of explicitly coded symbols. Additionally, an eight-state trellis-coded quantization method is introduced as a high-performance alternative to scalar quantization. This paper covers the rationale, implementation details, and the resulting coding gain benefits of these methods.
View details
Preview abstract
We introduce AASE (Activation-based AI Safety Enforcement), a framework for post-perception safety monitoring in large language models. Unlike pre-perception approaches that analyze input or output text, AASE monitors the model's internal activation patterns—what the model "understands" rather than what text it processes or generates—enabling detection of safety-relevant states before harmful outputs are produced. The framework comprises three techniques: Activation Fingerprinting (AF) for harmful content detection, Agent Action Gating (AAG) for prompt injection defense, and Activation Policy Compliance (APC) for enterprise policy enforcement. We introduce paired contrastive training to isolate safety-relevant signals from confounding factors such as topic and style, addressing signal entanglement in polysemantic activations. Validation across 7 models from 3 architecture families shows strong class separation: Gemma-2-9B achieves AUC 1.00 with 7.2σ separation across all probes; AAG achieves AUC ≥0.88 across all models on the InjecAgent benchmark; APC achieves 0.97-1.00 AUC across three enterprise policies. Model size correlates with probe quality—Gemma-2-9B (7.2σ separation) outperforms Gemma-2-2B (4.3σ). All techniques survive INT4 quantization with minimal separation degradation. AASE is 9× faster than Llama Guard 3 (33ms vs 306ms) with higher TPR (88% vs 50%) at a tunable threshold that trades FPR for detection sensitivity, adding only 0.002ms probe overhead to existing inference.
View details
An experimental evaluation of an AI-powered interactive learning platform
Nicole Miller
Yael Haramaty
Lidan Hackmon
Lior Belinsky
Abraham Oritz Tapia
Lucy Tootill
Scott Siebert
Frontiers in Artificial Intelligence (2026) (to appear)
Preview abstract
Generative AI, which is capable of transforming static content into dynamic learning experiences, holds the potential to revolutionize student engagement in educational contexts. However, questions still remain around whether or not these tools are effective at facilitating student learning. In this research, we test the effectiveness of an AI-powered platform incorporating multiple representations and assessment through Learn Your Way, an experimental research platform that transforms textbook chapters into dynamic visual and audio representations. Through a between-subjects, mixed methods experiment with 60 US-based students, we demonstrate that students who used Learn Your Way had a more positive learning experience and had better learning outcomes compared to students learning the same content through a digital textbook. These findings indicate that AI-driven tools, capable of providing choice among interactive representations of content, constitute an effective and promising method for enhancing student learning.
View details
Preview abstract
In this work, we prove computational lower bounds against differentially private (DP) coreset. Specifically, assuming the existence of one-way functions, we show that no polynomial-time $(\epsilon, 1/n^{\omega(1)})$-DP algorithm can compute $(\alpha, \beta)$-coreset for $k$-means in the $\ell_\infty$ metric for some constant $\alpha > 1$. For the Euclidean metric, we show a similar result but only for $\alpha = 1 + \Theta\left(\frac{1}{d^2}\right)$ where $d$ is the dimension.
View details
Hadal: Centralized Label DP Training without a Trusted Party
James Choncholas
Stanislav Peceny
Amit Agarwal
Baiyu Li
2026 IEEE Symposium on Security and Privacy (SP)
Preview abstract
In the Label Differential Privacy (Label DP) setting where machine learning training data is vertically partitioned between feature and label holders, centralized Differential Privacy approaches have shown superior utility compared to local approaches in high-privacy regimes. However, these centralized approaches typically require a trusted third party to compute the gradient. We introduce POSTSCALE, a novel training protocol based on Homomorphic Encryption with a ciphertext multiplicative depth of one, and no ciphertext rotations for models with affine gradient functions and few output classes. Furthermore, we present Hadal, a dataflow-based framework for encrypted computation, and Hadal-ml, a machine learning package tailored for Label DP, implementing our POSTSCALE protocol. Our approach leverages HE to enable centralized Differential Privacy without relying on a trusted third party. By integrating with TensorFlow, our framework Hadal supports both eager and deferred execution modes, while providing HE-specific optimizations and comprehensive performance profiling capabilities. Experimental results demonstrate that our approach achieves model utility comparable to existing centralized label DP techniques, surpasses local DP, and maintains privacy guarantees without relying on a trusted party.
View details
Preview abstract
Every team building Large Language Models (LLMs) faces a core challenge: offline benchmarks show performance gains and user engagement rises post deployment, but isolating cause from effect remains difficult. Simultaneous marketing, media coverage, and seasonal demand obscure whether model updates truly drive engagement gains. This paper presents a novel causal estimation approach that leverages non uniform quality improvements across capabilities within a single model version. Because capabilities improve unevenly (e.g., strong gains in coding versus modest gains in writing), users experience varied quality depending on their task distribution. This variation in experienced quality provides causal signal for estimation. We validate this approach on synthetic data with known ground truth. The raw estimator recovers 81% to 88% of the true effect, with the remainder lost to measurement error in usage estimates. Applying an errors in variables disattenuation adjustment (using a test retest reliability ratio mean correlation of 0.861) corrects the estimate to 1.017 (bootstrapped 95% CI: 0.915 to 1.109). By contrast, naive methods fail significantly, recovering only 63% without version controls and 62% using real time rather than frozen usage patterns. Permutation tests confirm the framework distinguishes true causal effects from noise. Sensitivity analysis indicates recovery improves monotonically from 78% to 87% as pre period length increases from 3 to 7 weeks, highlighting a clear tradeoff between sample duration and estimator precision. Seed sensitivity tests further confirm stability across random draws.
View details
A probabilistic framework for learning non‐intrusive corrections to long‐time climate simulations from short‐time training data
Benedikt Barthel
Rob Carver
Fei Sha
Themistoklis Sapsis
Journal of Advances in Modeling Earth Systems (2026)
Preview abstract
Despite advances in high performance computing, accurate numerical simulations of global atmospheric dynamics remain a challenge. The resolution required to fully resolve the vast range scales as well as the strong coupling with—often not fully-understood—physics renders such simulations computationally infeasible over time horizons relevant for long-term climate risk assessment. While data-driven parameterizations have shown some promise of alleviating these obstacles, the scarcity of high-quality training data and their lack of long-term stability typically hinders their ability to capture the risk of rare extreme events. In this work we present a general strategy for training variational (probabilistic) neural network models to non-intrusively correct under-resolved long-time simulations of turbulent climate systems. The approach is based on the paradigm introduced by Barthel Sorensen et al. (2024, https://doi.org/10.1029/2023ms004122) which involves training a post-processing correction operator on under-resolved simulations nudged toward a high-fidelity reference. Our variational framework enables us to learn the dynamics of the underlying system from very little training data and thus drastically improve the extrapolation capabilities of the previous deterministic state-of-the art—even when the statistics of that training data are far from converged. We investigate and compare three recently introduced variational network architectures and illustrate the benefits of our approach on an anisotropic quasi-geostrophic flow. For this prototype model our approach is able to not only accurately capture global statistics, but also the anistropic regional variation and the statistics of multiple extreme event metrics—demonstrating significant improvement over previously introduced deterministic architectures.
View details
