Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 11583 publications
When Android Apps Request Location with a Reason: A Developer Study and User Behavior Experiment
May Smith
Aman Jain
Proceedings on Privacy Enhancing Technologies (2027)
Preview abstract
Recent reports have highlighted how mobile apps share user location data with third parties, risking user privacy and platform trust. Although location data is highly sensitive, when users grant apps location access, they may not know the full extent to which it is used. We study how requiring Android apps to show a reason for location access could impact developers, users, and the platform.
We surveyed 323 Android app developers and found most supported such a requirement. The majority said it would have a positive impact on user privacy, trust for apps, and trust for Android, where impact on user trust for Android correlated most strongly with support. Many developers also said the intervention would increase the number of users granting location access. Yet their open-ended comments also revealed consistent concerns, such as apps providing dishonest reasons and platform verification.
To study the impact on user behavior, we conducted a randomized controlled experiment with 2579 US Android users. We tested how users' decisions to grant location access were impacted by app type, whether reasons were included in the requests, and the content of the reasons, including monetization. We did not find the reasons impacted users' decisions; decisions were instead driven by app type and demographics. Yet we did find the reasons could have a positive impact on user perception for the platform when the reasons did not include using data for ads. Our findings provide insights into developers' willingness to implement privacy-enhancing changes, and expose limits to improving user privacy by simply adding information to user interfaces.
View details
A 3D Scene Graphs Survey: Open Challenges and Future Directions
Dennis Rotondi
Francesco Argenziano
Sebastian Koch
Nathan Hughes
Martin Büchner
Johanna Wald
Lukas Schmid
Daniele Nardi
Abhinav Valada
Liam Paul
Luca Carlone
Kai Arras
Annual Review of Control, Robotics, and Autonomous Systems (ARCRAS), 10 (2027) (to appear)
Preview abstract
3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including mapping, task and motion planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real- world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are constructed from raw sensory observations, covering both learning-oriented and construction-oriented systems. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed works.
View details
Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action-Spaces
Haitong Ma
Ofir Nabati
Bo Dai
Na Li
Shie Mannor
Guy Tennenholtz
Proceedings of the 43rd International Conference on Machine Learning (ICML-26), Seoul, South Korea (2026)
Preview abstract
Reinforcement learning (RL) algorithms have achieved superhuman performance
on many sequential decision-making tasks, but often struggle in domains with
large, combinatorial action spaces. To address this, we introduce a practical and
stable algorithm for training discrete diffusion models to represent policies in
such environments. We formulate a policy mirror descent algorithm that enhances
training stability by reframing policy optimization as an inference problem, which
naturally aligns with the learning objective of discrete diffusion models. Through
extensive experiments on a suite of challenging benchmark tasks, we demonstrate
that our approach achieves significant improvements over existing methods in both
performance and sample efficiency. This work opens a promising new direction
for applying discrete diffusion models in RL to tackle long-standing challenges in
large-scale combinatorial action spaces.
View details
In Search of "Weird Corners": Diagnosing the Limits of Convergent AI in Professional Creative Practice
Mark Williams
2026
Preview abstract
Generative AI assistants typically employ convergent interaction paradigms to resolve ambiguity. While effective for technical tasks, this risks premature convergence in creative domains, constraining output variance. Evaluating a convergent AI probe with expert creatives (N=9) indicates an interactional paradox: structural linearity provides "ignition" utility for early ideation, but misaligns with organic workflows, often inducing "aesthetic sanitization" that standardizes individualized nuance. Prioritizing constructive friction over default agreement, the experts requested active, lateral collaborators. In response, we reframe output convergence as a "full-stack" UI challenge, advocating for Generative frameworks that operationalize the Double Diamond via fluid role-shifting and productive tension.
View details
Nine changes needed to deliver a radical transformation in biodiversity measurement
Neil Burgess
Andy Purvis
Scott J. Goetz
William Sutherland
Anil Madhavapeddy,
Tanya Birch
PNAS Perspective (2026)
Preview abstract
Biodiversity is declining in many parts of the world. The measurement and monitoring of
biological diversity are fundamental to the assessment of the causes and consequences of
environmental changes, identification of key areas for the protection of biodiversity or
ecosystem services, determining the effectiveness of actions, and the creation of
decision-support tools critical to the maintenance of a sustainable planet. The measurement
of biodiversity is rapidly changing due to advances in citizen science, image recognition,
acoustic monitoring, environmental DNA, genomics, remote sensing and artificial
intelligence. In this perspective, we outline the exciting opportunities that these
developments offer, but also consider the challenges, especially the potential poisoning of
data by AI, lack of standardisation across methods, coverage gaps in data, concerns over
losing databases, and undervaluing of on-the-ground expertise and data-generation. Our key
recommendations are (1) ensure new technologies are calibrated with existing data; (2) use
emerging technologies to fill data gaps; (3) create living databases of trusted information to
increase reliability of data and reduce the risk of poisoning by false - or AI hallucinated -
information; (4) ensure data generation is valued; (5) ensure the respect and incorporation
of Indigenous Knowledge; (6) increase in-country capacity in the tropics; and (7) increase the
resilience of global datasets to technical and societal change. Radical new collaborations are
needed between computer scientists, engineers, molecular biologists, data scientists, field
ecologists, citizen scientists, Indigenous peoples, and local communities to create the
rigorous, resilient, accessible biodiversity information systems required to underpin policies
and practices that ensure the maintenance and restoration of ecological systems.
View details
Preview abstract
The promise of tailored agent behavior is undermined by a critical explainability challenge: it is difficult to assess how closely and consistently the agent follows user-defined rules. As Large Language Models (LLMs) transition from static assistants to autonomous agents, developers have pioneered markdown-based rule files (e.g., GEMINI.md, CIDER_AGENT.md) to steer agent behavior and mitigate a "organizational context gap" that emerges when general-purpose models lack the "organizational context" necessary for contextually relevant results. This paper presents a qualitative study of 12 Google software developers (n=12) to investigate the authoring and efficacy of these agent rules. Our findings reveal that while rules are intended as technical steering mechanisms, they function as a "Black Box" of validation, where 12/12 participants rely on anecdotal "vibe checks" due to a profound lack of formal evaluation and explainability frameworks. We identify this opacity as a systemic Attribution Gap, which prevents developers from discerning whether a successful outcome was the result of deliberate logic or "pure luck." Paradoxically, these files serve a dual role as "Living Documentation," bridging technical instruction for AI with sociotechnical onboarding for humans. We argue for a transition toward library-level governance and rigorous traceability to transform agent customization from an ad-hoc craft into a human-centered science by revealing the internal "seams" of rule interpretation.
View details
Identifying Hearing Difficulty Moments in Conversational Audio
Jack Collins
Adrian Buzea
Chris Collier
Alejandro Ballesta Rosen
Julian Maclaren
Kelly Miles
Simon Carlile
Trends in Hearing (2026)
Preview abstract
Individuals regularly experience Hearing Difficulty Moments in everyday conversation. Identifying Hearing Difficulty Moments has particular significance in the field of hearing assistive technology where timely interventions are key for real-time hearing assistance. In this article, we propose and compare machine learning solutions for the temporal detection of segments containing Hearing Difficulty Moments in conversational audio. We show that audio language models, through their multimodal reasoning capabilities, can achieve state-of-the-art results for this task, significantly outperforming a simple automatic speech recognition (ASR) hotword heuristic and a more conventional fine-tuning approach with Wav2Vec, an audio-only input architecture that is state-of-the-art for ASR.
View details
Preview abstract
While the Latin script is used informally by speakers of many languages with more complex native scripts, high quality Latin script corpora for such languages that reflect actual natural romanizations are scarce and often difficult to collect. In this work, we propose a method for mining romanized language corpora in languages for which we do not have any pre-existing samples of naturally romanized text, focusing on Tigrinya as a test case. First we examine the efficacy of learning romanizations for a language based on observed romanizations in other languages that use the same native script. We then extrinsically assess such methods by using a romanization model trained on Amharic data to bootstrap coverage of romanized Tigrinya in a language identification system. Manual evaluation by two L1 and one L2 Tigrinya speakers suggests our method extracts romanized Tigrinya text with acceptably high precision. We release code to run our mining pipeline on public web corpora, such as MADLAD-400.
View details
Preview abstract
Existing image simplification techniques often rely on Non-Photorealistic Rendering (NPR), transforming photographs into stylized sketches, cartoons, or paintings. While effective at reducing visual complexity, such approaches typically sacrifice photographic realism. In this work, we explore a complementary direction: simplifying images while preserving their photorealistic appearance.
We introduce progressive semantic image simplification, a framework that iteratively reduces scene complexity by removing and inpainting elements in a controlled manner. At each step, the resulting image remains a plausible natural photograph. Our method combines semantic understanding with generative editing, leveraging Vision-Language Models (VLMs) to identify and prioritize elements for removal, and a learned verifier to ensure photorealism and coherence throughout the process. This is implemented via an iterative \emph{Select–Remove–Verify} pipeline that produces high-quality simplification trajectories.
To improve efficiency, we further distill this process into an image-to-video generation model that directly predicts coherent simplification sequences from a single input image. Beyond generating cleaner and more focused compositions, our approach enables applications such as content-aware decluttering, semantic layer decomposition, and interactive editing.
More broadly, our work suggests that simplification through structured content removal can serve as a practical mechanism for guiding visual interpretation within the photorealistic domain, complementing traditional abstraction methods.
View details
Type-Aware Ranking of Urban Similarity from Aerial Imagery
Idan Kligvasser
Yotam Intrator
Yuval Desheh
Aviad Barzilai
Niv Efron
Ehud Rivlin
Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops (2026), pp. 821-829
Preview abstract
Estimating and ranking cross-city similarity from aerial imagery is a fundamental challenge in remote sensing and geospatial representation learning. Urban environments differ widely in road layout, marking conventions, and infrastructure design, yet standard visual representations often struggle to disentangle these meaningful structural variations from superficial appearances. In this work, we propose a type-aware contrastive learning framework that measures urban similarity by explicitly modeling distinct infrastructure elements. Leveraging open-vocabulary retrieval, we construct a globally diverse dataset of road-related features, such as intersections, crosswalks, and bus lanes, and train a type-conditioned Vision Transformer that fuses visual features with CLIP-derived semantic embeddings. Crucially, we introduce an adaptive per-type contrastive loss that dynamically emphasizes infrastructure categories with high discriminative power while down-weighting less informative types. To quantify city-level similarity, we aggregate per-type cosine similarities via a lightweight classifier to generate a global city-to-city similarity matrix. Experiments demonstrate that this type-aware approach significantly improves clustering quality and successfully generalizes to unseen cities, establishing a scalable, interpretable foundation for comparative urban analysis.
View details
Ten Insights from Other Domains That Inform Responsible AI Frameworks
Patrick Gage Kelley
Allison Woodruff
Angela McKay
Dunstan Allison-Hope
Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (2026), 104–115
Preview abstract
The rapid growth of AI systems is being accompanied by new guidelines, principles, standards, regulations, and best practices (hereafter “frameworks”) that seek to ensure the responsible design, development, deployment, and use of AI systems.
Our premise is that the substance, implementation, and evolution of these AI frameworks can be informed by the practical experience of pursuing similar desired outcomes in other relevant domains (e.g., content moderation, human rights, climate change). This will help ensure that mistakes are not repeated and more rapid progress is made.
We used a “repetition test” to generate the following ten insights from other domains. Insights passing the “repetition test” are those that experts with thousands of hours of practical experience often repeat when describing the best practices that have emerged from their domain.
AI frameworks can draw from these ten insights, rather than invent entirely new approaches.
View details
Preview abstract
Optical health sensing algorithms, such as SpO2, sleep monitoring, and metabolic health sensing, critically depend on the accurate measurement of optical emission from Light Emitting Diodes (LEDs) transmitted through user tissue and detected by a photodiode (PD). A significant challenge to the reliability of these measurements is the inherent degradation of LED optical emission intensity over time due to device aging. This degradation can confound the physiological changes being monitored. Our work quantifies the impact of LED aging on sensor signal integrity, specifically examining the Current Transfer Ratio (CTR), which is a key metric defining the ratio of received photocurrent to the LED drive current used for transmission in various health sensing algorithms. We investigate the degradation characteristics across LEDs of different wavelengths. Our findings indicate a relative CTR change due to degradation ranging from 1% to 8% within 100 hours of continuous operation which translates to approximately 3.5 to 7 years of device lifetime. Furthermore, we explore the non-linearity of this degradation and the observed initial ”overshoot” phenomenon in the CTR during aging. We discuss how understanding these dynamics could inform the development of robust specifications for different physiological sensing algorithms. Finally, we present several potential solutions to mitigate the effects of LED aging. During the product design phase, integrating a calibrating photodiode or compensating circuitry around the LED can help preemptively address degradation. In the application space, run-time calibration strategies employing two differently degraded optical paths offer a promising approach to maintain measurement accuracy.
View details
Neural general circulation models for modeling precipitation
Stephan Hoyer
Dmitrii Kochkov
Janni Yuval
Ian Langmore
Science Advances (2026)
Preview abstract
Climate models struggle to accurately simulate precipitation, particularly extremes and the diurnal cycle. While hybrid models combining machine learning and physics have emerged with the premise of improving precipitation simulations, none have proven sufficiently skillful or stable enough to outperform existing models in simulating precipitation.
Here, we present the first hybrid model that is trained directly on precipitation observations. The model runs at 2.8 degrees resolution and is built on the differentiable NeuralGCM framework. This model is stable for decadal simulations and demonstrates significant improvements over existing GCMs, ERA5 reanalysis, and a Global Cloud-Resolving Model in simulating precipitation. Our approach yields reduced biases, a more realistic precipitation distribution, improved representation of extremes, and a more accurate diurnal cycle.
Furthermore, it outperforms the ECMWF ensemble for mid-range weather forecasting.
This advance paves the way for more reliable simulations of current climate and for the ability to fully utilize the abundance of existing observations to further improve GCMs.
View details
A Dynamic Numerical Model for Real-Time Estimation of Latent Cognitive States Using Oculomotor Metrics
Diako Mardanbegi
ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 22087-22091
Preview abstract
Estimating internal cognitive states from oculomotor data is fundamentally challenging due to their context-dependency and the complex relationship between various metrics. This paper proposes a dynamic numerical framework to model a task-specific behavioral signature and monitor deviations from it in real-time. The model integrates key oculomotor and motion metrics into a differential equation, yielding a continuous score that serves as an estimate of a latent cognitive state. By using tunable, heuristic parameters, our approach offers a transparent and adaptable alternative to opaque machine learning models. The framework’s strength lies in its ability to pinpoint objective changes in behavior, providing a potential tool for interpreting events like the onset of fatigue, distraction, or cognitive load.
View details
Preview abstract
Human-Computer Interaction research and design pedagogy rely on idealized process models, such as the Double Diamond, to describe how user experiences are designed. These models assume an orderly, linear design process that, while easy to understand, fails to capture the iterative and collaborative reality of professional practice. A few qualitative studies have successfully captured this complexity -- still, they often suffer from retrospective narrative smoothing and lack systemic scale. To understand how design unfolds in real products, we analyzed historical snapshots of 102 Figma files from a multi-national technology company and investigated the true trajectories of the design process at scale. Our analysis reveals that while the established process models might be applicable, the operational details are highly non-linear. Rather than a straight line from ideation toward completion, design advances are repeatedly reset to the ideation stage as feedback is received. We argue that by treating design files as operational telemetry, the industry can move beyond abstract frameworks to build practices and collaborative tools that support the non-linear realities of modern product development.
View details
