Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 11622 publications
A 3D Scene Graphs Survey: Open Challenges and Future Directions
Dennis Rotondi
Francesco Argenziano
Sebastian Koch
Nathan Hughes
Martin Büchner
Johanna Wald
Lukas Schmid
Daniele Nardi
Abhinav Valada
Liam Paul
Luca Carlone
Kai Arras
Annual Review of Control, Robotics, and Autonomous Systems (ARCRAS), 10 (2027) (to appear)
Preview abstract
3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including mapping, task and motion planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real- world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are constructed from raw sensory observations, covering both learning-oriented and construction-oriented systems. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed works.
View details
When Android Apps Request Location with a Reason: A Developer Study and User Behavior Experiment
May Smith
Aman Jain
Proceedings on Privacy Enhancing Technologies (2027)
Preview abstract
Recent reports have highlighted how mobile apps share user location data with third parties, risking user privacy and platform trust. Although location data is highly sensitive, when users grant apps location access, they may not know the full extent to which it is used. We study how requiring Android apps to show a reason for location access could impact developers, users, and the platform.
We surveyed 323 Android app developers and found most supported such a requirement. The majority said it would have a positive impact on user privacy, trust for apps, and trust for Android, where impact on user trust for Android correlated most strongly with support. Many developers also said the intervention would increase the number of users granting location access. Yet their open-ended comments also revealed consistent concerns, such as apps providing dishonest reasons and platform verification.
To study the impact on user behavior, we conducted a randomized controlled experiment with 2579 US Android users. We tested how users' decisions to grant location access were impacted by app type, whether reasons were included in the requests, and the content of the reasons, including monetization. We did not find the reasons impacted users' decisions; decisions were instead driven by app type and demographics. Yet we did find the reasons could have a positive impact on user perception for the platform when the reasons did not include using data for ads. Our findings provide insights into developers' willingness to implement privacy-enhancing changes, and expose limits to improving user privacy by simply adding information to user interfaces.
View details
GUIDE: A Benchmark for User Context Understanding and Assistance in GUI Workflow Videos
Saelyne Yang
Jaesang Yu
Yi-Hao Peng
Kevin Qinghong Lin
Jae Won Cho
Juho Kim
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026)
Preview abstract
Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software. While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency.To move beyond automation and toward collaboration, GUI agents must understand what users are doing and why. We introduce GUIDE (GUI Understanding, Intent, and Help Decision Evaluation), a benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations that surface user intent, across 10 complex software (e.g., PowerPoint, Photoshop). GUIDE defines three tasks—(i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model’s ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art multimodal models reveal that all models struggled with the tasks, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context such as behavioral state and intent significantly improved the performance, raising help prediction by up to 50.2%. These results highlight the critical role of structured user understanding in effective assistance.Our benchmark provides a path toward GUI agents that go beyond automation to become truly user-aware collaborators.
View details
Prospective multi-pathogen disease forecasting using autonomous LLM-guided tree search
Sarah Martinson
Nicholas Reich
Zahra Shamsi
arXiv (2026)
Preview abstract
Probabilistic forecasting of infectious diseases is crucial for public health but relies on labor-intensive manual curation by expert modeling teams. This bespoke development bottlenecks scalability to granular geographic resolutions or emerging pathogens. Here, we present an autonomous system utilizing Large Language Model (LLM)-guided tree search \cite{aygun_ai_2025} to iteratively generate, evaluate, and optimize executable forecasting software. In a fully prospective, real-time evaluation during the 2025–2026 US respiratory season, the system autonomously discovered methodologically diverse models for influenza, COVID-19, and respiratory syncytial virus (RSV). Aggregating these machine-generated models yielded an ensemble that consistently matched or outperformed the gold-standard, human-curated Centers for Disease Control and Prevention (CDC) hub ensembles out-of-sample. The system successfully navigated data-scarce "cold start" scenarios for RSV. Moreover, controlled ablations revealed that optimizing log-scale distance metrics prevents reward hacking, while an automated judge-in-the-loop ensures structural fidelity to complex scientific theories. By autonomously translating epidemiological theory into accurate, transparent code, this framework breaks the modeling labor bottleneck, enabling the rapid deployment of expert-level disease forecasting at unprecedented scales.
View details
Identifying Hearing Difficulty Moments in Conversational Audio
Jack Collins
Adrian Buzea
Chris Collier
Alejandro Ballesta Rosen
Julian Maclaren
Kelly Miles
Simon Carlile
Trends in Hearing (2026)
Preview abstract
Individuals regularly experience Hearing Difficulty Moments in everyday conversation. Identifying Hearing Difficulty Moments has particular significance in the field of hearing assistive technology where timely interventions are key for real-time hearing assistance. In this article, we propose and compare machine learning solutions for the temporal detection of segments containing Hearing Difficulty Moments in conversational audio. We show that audio language models, through their multimodal reasoning capabilities, can achieve state-of-the-art results for this task, significantly outperforming a simple automatic speech recognition (ASR) hotword heuristic and a more conventional fine-tuning approach with Wav2Vec, an audio-only input architecture that is state-of-the-art for ASR.
View details
Optical Transport Optimized for AI Infrastructure - Fiber Is The New Wavelength (𝝺)
Ahsaan Yousaf
2026
Preview abstract
With ML/AI as growth drivers for WAN, highlight unique challenges and drive the industry for next gen coherent pluggable and embedded terminal optics solutions with performance and RSP tradeoffs in mind.
Together with other hyperscalers, drive the industry (system and component vendors) to develop highly space and power efficient ILA solutions, compatible with our existing infrastructure. The right technology evolution will have to consider overall TCO and operational/practical aspects such as power consumption and failure domains.
Zero Touch Networking: Highlighting importance of standard open models and APIs across IP/Optical layers across module/host vendors enabling vendor agnostic config, telemetry for autonomous operations
View details
Preview abstract
Coding agents are rapidly changing the landscape of software development, moving from inline com- pletion to autonomous systems that edit repositories, open pull requests, respond to issues, and run scheduled or webhook triggered routines across the development life cycle. The next generation is increasingly described as proactive and long-horizon: agents should notice relevant changes before the developer asks, connect signals across tools, decide when to interrupt, and carry preferences across sessions. Yet the field lacks a precise account of what proactivity means for software development, how it differs from autonomy, what acceptance criteria proactive long-horizon tasks should satisfy, and which metrics determine whether unsolicited agent behavior is useful rather than merely active. We argue that proactive coding agents should be evaluated by the quality and improvement of their insight policy: the policy that decides what matters next, what evidence supports it, whether to surface it, and how to adapt after feedback. We re-anchor this view in mixed initiative interaction, introduce a three level taxonomy (Reactive, Scheduled, and Situation Aware), compare contemporary coding agents against five operational criteria, and sketch an active user simulation protocol with three evaluation targets: Insight Decision Quality (IDQ), Context Grounding Score (CGS), and Learning Lift (LL).
View details
Preview abstract
Latent Diffusion Models (LDM) face three compounding efficiency challenges in practical deployment: i) the temporal latency of iterative sampling; ii) the architectural overhead of heavy backbone parameter counts; and iii) the spatial cost of high-dimensional latent grids. While recent acceleration methods have made substantial progress on temporal distillation and architectural compression, the spatial axis is often inherited from the teacher tokenizer and treated as fixed. In this work, we recast latent resolution as an optimizable efficiency axis and introduce a unified framework that optimizes all three dimensions simultaneously. We introduce a novel strategy of Score-Compatible Tokenizer Distillation (SCTD), which leverages score-matching principles to align a spatially compact latent space with the induced distribution of a frozen, powerful teacher model, distilling the teacher's generative prior into a compressed, lower-dimensional compatible manifold. With flexibility provided by SCTD, we can surrogate a computationally heavy teacher backbone with a lightweight student architecture operating strictly within this new compressed space. Finally, we apply temporal distillation to collapse the sampling trajectory, producing a one-step generator that operates at peak efficiency. Our method yields a student generator outperforming existing single-axis acceleration methods in efficiency and throughput, while maintaining competitive generation quality. With reduced peak memory usage and latency, our method enables resource-constrained deployment and high-volume serving of high-fidelity LDM.
View details
Preview abstract
The management of a hybrid workforce comprising human and autonomous computational agents may be challenged by the use of separate systems for human capital and software assets, which can create a governance gap. A system can provide a unified framework for managing a hybrid workforce. For example, the system may utilize a labor service mesh to analyze and route tasks to either a human intent tier or an agentic execution tier. A potential principle of the system is structural symmetry, where computational agents can be assigned digital identities and managed through a lifecycle process that may parallel human resource functions, such as onboarding, performance evaluation, and structured offboarding. This integrated approach can facilitate a unified system of record and governance model for an organization's intelligence capacity.
View details
Integrating Level-Design for Editable, Shared Worlds in Diffusion Game Engines
Po Ryan
David Zhang
Amir Hertz
Gordon Wetzstein
Neal Wadhwa
Nataniel Ruiz
2026
Preview abstract
Video world models have shown immense promise for interactive simulation and entertainment, but current systems still struggle with two important aspects of interactivity: user control over the environment for reproducible, editable experiences, and shared inference where players hold influence over a common world. To address these limitations, we introduce an explicit external memory into the system, a persistent state operating independent of the model’s context window, that is continually updated by user actions and queried throughout the generation roll-out. Unlike conventional diffusion game engines that op-
erate as next-frame predictors, our approach decomposes generation into Memory, Observation, and Dynamics modules. This design gives users direct, editable control over environment structure via an editable memory representation, and it naturally extends to real-time multiplayer rollouts with coherent viewpoints and consistent cross-player interactions.
View details
Preview abstract
Generative AI is reshaping software development, yet its psychological impact remains under-researched. During May and August 2025 we conducted reflexive thematic analysis of interviews with 12 senior engineers (≥5 years experience) recruited from Western technology hubs to explore shifts in professional identity. We identify a central transition from "coder to conductor," where AI acts as a cognitive partner. Key findings include: (1) a re-architecting of focus from implementation to strategy; (2) a shift in productivity metrics from output to impact; and (3) a dual-impact on agency, where AI empowers autonomy but threatens competence through de-skilling anxieties. These findings suggest that as implementation becomes commoditised, organisational training and career progression must prioritise architectural mastery and metacognitive oversight to ensure sustained developer motivation and system integrity.
View details
A Validated Scale Measuring Student Self-Efficacy for Programming with Generative AI
Erin Spaulding
Yekaterina Kharitonova
Paul Denny
Juho Leinonen
Lauren Elizabeth Margulieux
James Prather
Yonggao Yang
Brent N. Reeves
Jamie Benario
Ernest Holmes
2026
Preview abstract
The rise of generative artificial intelligence (GenAI) has sparked a rapid change in computing curricula and teaching approaches. GenAI coding tools can accurately complete assignments, answer test questions, and perform other tasks traditionally associated with learning programming, especially at the introductory level. Because GenAI is still so new, researchers investigating student usage of GenAI have used informal rubrics and questionnaires. To advance, the field needs validated instruments that measure student perception and use of GenAI. This paper presents the development and initial validation of an instrument to measure self-efficacy while using GenAI to learn programming. Self-efficacy is an important construct in education research because it robustly correlates with student success, across disciplines and ages, including undergraduate computing education. Computing education researchers have presented several validated self-efficacy instruments, most recently by Steinhorst et al. in 2020. Critically, this instrument was created before the rise of GenAI’s popularity in 2022. To complement this instrument, we created a GenAI scale similar in style to the Steinhorst self-efficacy instrument, consisting originally of 11 items and revised to 5 items. We report two important findings in this paper. First, we found strong support for the validity of the existing Steinhorst instrument in a new context, specifically an introductory programming course that fully integrates GenAI. Second, the new GenAI scale shows strong internal reliability, discriminant validity with items in the Steinhorst subscales, and criterion validity with students’ GenAI usage patterns. Based on statistical analysis and cognitive probing interviews, we argue for the validity of the five-item scale to measure students’ GenAI self-efficacy in the context of programming.
View details
A DV Approach to Post-Silicon: Validating SoC Low Power through Synthetic Software Scenarios
Kapish Pandey
Utkarsh Prabhat
Yash Kumar
V Sindhu
2026
Preview abstract
Lightweight execution environments like the Little Kernel (LK) are commonly deployed in post-silicon validation to assess software-hardware interactions. These bare-metal kernels, however, lack the sophisticated power management features present in full operating systems, such as the Generic Power Domain (GenPD) framework. Instead of building complex software abstractions that simulate production-grade power management drivers, this paper applies a Design Verification (DV) approach to post silicon. By discarding standard software paradigms, the introduced framework leverages fundamental bare-metal kernel primitives to intentionally engineer synthetic, non-deterministic stressors. Eliminating intermediate software layers allows us to utilize the bare-metal setup to drive chaotic, aggressive stimuli straight to the hardware execution layer.
View details
Preview abstract
This framework manages AI agents by establishing behavioral boundaries and a persistent identity. It uses a multi-layered stack, combining safety rules with brand guidelines, to shape an agent's reasoning. Features include authority decay to limit power if confidence drops and memory segmentation to prevent data tampering. Centralized oversight ensures these digital representatives remain aligned with company policies through continuous monitoring and testing.
View details
Verification-Driven Interface Convergence: Methodology for Validating Functional Interoperability And Accelerating Subsystem Design for Concurrently Developed Compute and Component IPs in Coherent Subsystem
Alisha Parvez
Preethi Ashok Kumar
Shreya Singh
Jyothi Kumari
Rohit Jindal
2026
Preview abstract
To meet aggressive time-to-market goals, modern mobile SoC architectures require the concurrent development of custom Compute-IPs and surrounding subsystem integration logic (1PIPs and 3PIPs). However, this parallel execution creates a critical verification deadlock: the subsystem cannot be validated until both the Compute-IP and volatile, in-flight 1PIPs reach physical RTL maturity. Consequently, "integration-killer" bugs—such as protocol handshaking deadlocks and clock/reset sequencing mismatches—remain hidden until late in the design cycle when RTL rework costs are prohibitive. To break this bottleneck, we present a verification-driven methodology utilizing a silicon-proven Golden Proxy, Direct-Execution Traffic Profiles, Programmable Sequencers and Automated Protocol Converters. This framework completely decouples parallel hardware dependencies, pre-pulling critical inter-IP mismatch discoveries months ahead of traditional integration milestones.
View details
