Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Sort By
1 - 15 of 11557 publications
A 3D Scene Graphs Survey: Open Challenges and Future Directions
Dennis Rotondi
Francesco Argenziano
Sebastian Koch
Nathan Hughes
Martin Büchner
Johanna Wald
Lukas Schmid
Daniele Nardi
Abhinav Valada
Liam Paul
Luca Carlone
Kai Arras
Annual Review of Control, Robotics, and Autonomous Systems (ARCRAS), 10 (2027) (to appear)
Preview abstract
3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including mapping, task and motion planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real- world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are constructed from raw sensory observations, covering both learning-oriented and construction-oriented systems. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed works.
View details
When Android Apps Request Location with a Reason: A Developer Study and User Behavior Experiment
May Smith
Aman Jain
Proceedings on Privacy Enhancing Technologies (2027)
Preview abstract
Recent reports have highlighted how mobile apps share user location data with third parties, risking user privacy and platform trust. Although location data is highly sensitive, when users grant apps location access, they may not know the full extent to which it is used. We study how requiring Android apps to show a reason for location access could impact developers, users, and the platform.
We surveyed 323 Android app developers and found most supported such a requirement. The majority said it would have a positive impact on user privacy, trust for apps, and trust for Android, where impact on user trust for Android correlated most strongly with support. Many developers also said the intervention would increase the number of users granting location access. Yet their open-ended comments also revealed consistent concerns, such as apps providing dishonest reasons and platform verification.
To study the impact on user behavior, we conducted a randomized controlled experiment with 2579 US Android users. We tested how users' decisions to grant location access were impacted by app type, whether reasons were included in the requests, and the content of the reasons, including monetization. We did not find the reasons impacted users' decisions; decisions were instead driven by app type and demographics. Yet we did find the reasons could have a positive impact on user perception for the platform when the reasons did not include using data for ads. Our findings provide insights into developers' willingness to implement privacy-enhancing changes, and expose limits to improving user privacy by simply adding information to user interfaces.
View details
Preview abstract
The exponential growth of machine learning submissions has strained the traditional peer review process, resulting in slow feedback loops for authors and an immense burden on reviewers to rigorously audit technical soundness and verify literature. To address this, we introduce ScholarPeer, a multi-agent framework designed to operationalize the rigorous auditing workflow of a senior researcher. Rather than attempting to replace human judgment, ScholarPeer serves as a co-scientist: acting as a mentor for rapid author iteration prior to submission, and as an active verification assistant that augments human reviewers. The framework structurally decouples contextualization from critique by deploying a sub-domain historian to synthesize the field's trajectory, a baseline scout to proactively hunt for omitted state-of-the-art comparisons, and a multi-aspect Q&A engine that deeply audits technical soundness-scrutinizing internal logical consistency, experimental validity, and mathematical rigor-while cross-referencing claims against top-tier academic venues. We comprehensively evaluate ScholarPeer on ~1,800 ICLR submissions spanning 2020 through 2025. Our results show that ScholarPeer achieves significant win-rates against state-of-the-art fine-tuned models and search-augmented agentic baselines.
View details
GUIDE: A Benchmark for User Context Understanding and Assistance in GUI Workflow Videos
Saelyne Yang
Jaesang Yu
Yi-Hao Peng
Kevin Qinghong Lin
Jae Won Cho
Juho Kim
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2026)
Preview abstract
Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software. While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency.To move beyond automation and toward collaboration, GUI agents must understand what users are doing and why. We introduce GUIDE (GUI Understanding, Intent, and Help Decision Evaluation), a benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations that surface user intent, across 10 complex software (e.g., PowerPoint, Photoshop). GUIDE defines three tasks—(i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model’s ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art multimodal models reveal that all models struggled with the tasks, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context such as behavioral state and intent significantly improved the performance, raising help prediction by up to 50.2%. These results highlight the critical role of structured user understanding in effective assistance.Our benchmark provides a path toward GUI agents that go beyond automation to become truly user-aware collaborators.
View details
How Tech Workers Contend with Hazards of Humanlikeness in Generative AI
Eric Corbett
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, ACM (2026), pp. 1-18
Preview abstract
Generative AI’s humanlike qualities are driving its rapid adoption in professional domains. However, this anthropomorphic appeal raises concerns from HCI and responsible AI scholars about potential hazards and harms, such as overtrust in system outputs. To investigate how technology workers navigate these humanlike qualities and anticipate emergent harms, we conducted focus groups with 30 professionals across six job functions (ML engineering, product policy, UX research and design, product management, technology writing, and communications). Our findings reveal an unsettled knowledge environment surrounding humanlike generative AI, where workers’ varying perspectives illuminate a range of potential risks for individuals, knowledge work fields, and society. We argue that workers require comprehensive support, including clearer conceptions of “humanlikeness” to effectively mitigate these risks. To aid in mitigation strategies, we provide a conceptual map articulating the identified hazards and their connection to conflated notions of “humanlikeness.”
View details
Preview abstract
We introduce a new context-enriched time series forecasting benchmark TimesX. TimesX contains a wide selection of high-quality real-world time series and diverse textual contexts from an automated generating pipeline, which helps address three main issues of existing benchmarks: (1) poor generalization due to low data volume and data being synthetic, (2) restricted forms of context, and (3) an inability to mitigate data leakage. We conduct a thorough empirical study of current multimodal solutions on TimesX. Our results suggest that most multimodal solutions that work well on existing benchmarks may fail on TimesX. In contrast, simple ensemble methods that leverage the rich textual context can outperform strong unimodal baselines and other multimodal baselines.
** Below this is what was submitted to ITP. **
We create a real world multimodal time-series forecasting benchmark that encompasses diverse domains and regions. Each time-series is annotated by various kinds of contexts like metadata, date and holiday information, dynamic events related to the time-series. This is sufficiently more advanced than other available benchmarks which rely wither on static metadata alone or synthetic examples. This forms a test bed for multimodal forecasting. We also present some baseline results showing that ensembles of publicly available LLMs and time-series foundation models can demonstrate non-trivial performance on this bechmark.
View details
"You Have Been Selected as the Winner": Characterizing User-Reported Scams on TikTok
Smirity Kaushik
Kyle Beadle
Gauri Nayak
Madelyn Rose Sanfilippo
Mainack Mondal
Yang Wang
JingJei Li
Yixin Zou
USENIX Symposium on Usable Privacy and Security (SOUPS) (2026)
Preview abstract
Short-form video platforms (SVPs) such as TikTok have grown rapidly in popularity. While online scams have been extensively studied, the extent to which they take new forms on SVPs and the discourses around them remain understudied. Using TikTok as a case study, we analyzed 150 videos in which content creators reported scam experiences and offered anti-scam advice. We focus on how TikTok users (creators, followers, and commenters) discuss scams, rather than analyzing scams. Our analysis surfaces six types of scams, including creator impersonation and account badge verification scams that target TikTok's influencer-follower ecosystem. Scammers also exploit platform-specific features (e.g., direct messaging or the "For You Page") to lure victims. In response, TikTok users share strategies to identify scammer profiles and communication cues, building community support through anti-scam advice. Based on our findings, we offer recommendations for systemizing platform support to combat scams and leveraging the influencer ecosystem to raise awareness.
View details
TCO-driven Storage Provisioning for Exascale Data Centers
Timothy Kim
Prashant Nema
Jai Menon
Gregory R. Ganger
Rashmi Vinayak
2026
Preview abstract
Recent changes in data temperatures and storage device characteristics, both mechanical disk drives (HDDs) and solidstate drives (SSDs), expand the set of deployment options for exascale storage. Until recently, exascale storage systems followed a pattern of placing most data on HDDs with smaller amounts of SSD storage used for caching and performance-critical workloads. Exascale storage provisioning and dataset placement trade-offs have now changed.
This paper describes a total cost of ownership (TCO) model that captures primary aspects of modern deployments and uses it to explore the new trade-off space. Using capacity and performance telemetry information for 43 production datasets+workloads at two large hyperscalers, we show significant changes from prior analyses of workloads and storage placement decisions across a multitude of storage device types. We also introduce a storage cluster TCO optimizer that identifies the lowest-TCO grouping and assignment of datasets to device types, exposing a number of insights that can help guide future deployments. For example, our analysis shows that the highest-density SSDs are particularly favorable for clusters with heavy AI/ML workloads but are only cost-effective at exascale when combined with high-density HDDs. Finally, we use our framework to evaluate how storage provisioning and overall TCO change as a function of key parameters like device write amplification, cluster power bounds, and the maximum number of device types allowed.
View details
A Computer Vision Problem in Flatland
Erin Connelly
Annalisa Crannell
Timothy Duff
Rekha R. Thomas
SIAM Journal on Applied Algebra and Geometry, 10 (2026), pp. 14-45
Preview abstract
When is it possible to project two sets of labeled points of equal cardinality lying in a pair of projective planes to the same image on a projective line? We give a complete answer to this question, obtaining the following results. We first show that such a pair of projections exist if and only if the two point sets are themselves images of a common point set in projective space. Moreover, we find that for generic pairs of point sets, a common projection exists if and only if their cardinality is at most seven. In these cases, we give an explicit description of the loci of projection centers that enable a common image.
View details
XMob: Accelerated Differentiable Mesoscopic Traffic Simulation in JAX
Sam Griesemer
John Burge
Shantanu Shahane
Yechen Li
Naples, Italy (2026)
Preview abstract
This paper introduces XMob, a novel differentiable traffic simulation framework built in JAX to advance traditional models like SUMO’s mesoscopic simulator. By leveraging JAX’s capabilities for vectorized, hardware-accelerated computation (GPU/TPU), XMob achieves orders-of-magnitude speedups, enabling large-scale urban network simulations and extensive counterfactual analyses. A key innovation is XMob’s inherent differentiability, facilitating direct integration with gradient-based optimization for tasks such as demand calibration and network parameter estimation, significantly outperforming black-box approaches. Furthermore, XMob can be used in Physics-Informed Machine Learning (PIML) pipelines to enhance data-driven augmentation, embedding domain principles like flow conservation and shockwave theory. This ensures physically plausible and robust predictions, even for unobserved scenarios such as lane modifications. The hybrid architecture, combining a deterministic JAX core with incremental machine learning, offers a scalable and efficient solution for modern traffic simulation and optimization challenges.
View details
Preview abstract
PURPOSE: To introduce Cardio Load (CL), a metric quantifying cardiovascular work from all activities across the day, and to investigate its distribution by age, gender, and workout profiles. CL adapts the Training Impulse (TRIMP) model by leveraging continuous heart rate and movement data from wearables, enabling minute-level intensity estimation. We also discuss the derivation of weekly target loads, intended to guide fitness maintenance.
METHODS: A retrospective analysis was conducted on 31.2 million hours of wrist-worn wearable data collected over a six-week period. The dataset comprised a 40,000-subject subset (37.9% female) of consenting Google Pixel Watch® users in the United States, aged 18 to 80 years (18-39: 41.8%, 40-59: 43.5%, 60+: 14.6%). Measured data included minute-interval heart rate averages, resting and maximum heart rates, minute-interval averaged accelerometer log energy, and manually-logged or auto-detected activity types. Cardio Load scores and target loads were calculated daily for each subject and compared across age and gender. We also compared the proportions of CL gained during workouts and incidental daily activities for these groups.
RESULTS: Overall, the study population's mean ± SD weekly CL scores were 221 ± 156 (female) and 259 ± 169 (male). Median weekly Cardio Load (CL) values exhibited consistency for individuals between 30 and 75 years of age. When analyzed in five-year age groups, the coefficient of variation (CV%) of median weekly CL values within this age range was less than 4.5%, with younger and older subjects demonstrating higher and lower median CL, respectively. The median proportion of CL accumulated during structured workouts versus incidental daily activity was 41.0% (female) and 49.0% (male) for all subjects, though this varied considerably with average weekly workout duration. CV% of weekly target load and daily target load over 6 weeks was 23.6% and 35.2% respectively.
CONCLUSION: Cardio Load provides a continuous quantification of activity load from wearables, acknowledging both structured workouts and everydayincidental activity. CL is equitably rewarded for age ranges spanning 30-75 years. Weekly target loads were found to have little measurement variability and be more consistent and, consequently, more practical for planning training and physical activity than daily targets.
View details
Preview abstract
We analyze a balls-and-bins game against an adaptive adversary that sequentially deletes bins. Starting with n balls distributed across n bins, the adversary deletes a bin in each step, forcing the algorithm to redistribute its balls to surviving bins. We prove that after n/2 rounds, uniform random redistribution yields optimal O(n) recourse and $O(\frac{\log n}{\log \log n})$ maximum load. Furthermore, we show that applying the "power of two choices"' reduces the maximum load to $O(\log \log n)$ while maintaining linear recourse.
We also consider a variation of this game where the balls from the deleted bin are partitioned evenly among d<
View details
Preview abstract
We present MOSAIC-GS, a novel, fully explicit, and computationally efficient approach for high-fidelity dynamic scene reconstruction from monocular videos.
Monocular reconstruction is inherently ill-posed due to the absence of sufficient multiview constraints,
making accurate recovery of object geometry and temporal coherence particularly challenging.
To address this, we leverage multiple geometric cues, such as depth, optical flow, dynamic object segmentation, and tracking trajectories,
combined with rigidity constraints to estimate preliminary 3D scene dynamics during an advanced initialization stage.
Recovering scene flow prior to the photometric optimization phase reduces reliance on motion inference from visual appearance alone, which is often ambiguous in monocular settings.
To enable compact representations, fast training, and real-time rendering while supporting non-rigid deformations, the scene is decomposed into static and dynamic components,
with dynamic trajectories represented as time-dependent Poly-Fourier curves for parameter-efficient motion encoding.
We demonstrate that MOSAIC-GS achieves substantially faster optimization and rendering compared to existing methods,
while maintaining reconstruction quality on par with state-of-the-art approaches across standard monocular dynamic scene benchmarks.
View details
AI-Simulated Expert Panels for Socio-Technical Scenarios and Decision Guidance
Andrew G. Ross
Alan Ross
(2026) (to appear)
Preview abstract
Socio-technical scenarios for net-zero and other transformation pathways combine qualitative storylines with quantitative models, embedding them in plausible societal contexts for model assessment. Conventional scenario generation is resource-intensive, can be limited in internal consistency and diversity of expert and stakeholder perspectives, and is rarely stress-tested.
This paper introduces a synthetic, AI-based expert panel to address these bottlenecks. An AI model first simulates domain experts who agree on descriptors, states, and their interactions. A probabilistic Cross-Impact Balance analysis then generates internally consistent pathways, using stochastic shocks to assess robustness and pathway diversity. An AI stakeholder panel
uses multi-criteria decision analysis to select a preferred pathway; an AI expert panel translates it into model-ready quantitative inputs. Although scalable and applicable to any other country or region, the framework is applied to Germany's energy transition as a proof of concept, and offers an alternative and/or supplement to scenario generation. Furthermore, it enables Virtual AI-Led Decision Laboratories for exploratory policy stress-testing and provides an approach for rapid, structured expert elicitation and decision support in other domains.
View details
Online Advertising with Spatial Interactions
Yifan Wang
Mingfei Zhao
Proceedings of the ACM Web Conference 2026, Association for Computing Machinery, New York, NY, USA, 213–224
Preview abstract
Online advertising platforms must decide how to allocate multiple ads across limited screen real estate, where each ad's effectiveness depends not only on its own placement but also on nearby ads competing for user attention. Such spatial externalities — arising from proximity, clutter, or crowding — can significantly alter welfare and revenue outcomes, yet existing auction and allocation models typically treat ad slots as independent or ordered along a single dimension. We introduce a new framework for spatial externalities in online advertising, in which the value of an ad depends on both its slot and the configuration of surrounding ads. We model ad slots as points in a metric space, and model an advertiser's value as a function of both their bid and a discount factor determined by the configuration of other displayed ads.
Within this framework, we analyze two natural models. For the Nearest-Neighbor model, where the value suppression depends only on the closest neighboring ad, we present a polynomial-time algorithm that achieves a constant approximation for the general case. We show that the allocation rule is monotone and can be implemented as a truthful mechanism. For a structured setting of 2D Euclidean space, we provide a PTAS. In contrast, for the Product-Distance model, where interference is aggregated multiplicatively across all neighbors, we establish a strong (and nearly-tight) hardness of approximation -- no polynomial-time algorithm can achieve any polynomial-factor approximation unless P=NP, via a reduction from Max-Independent-Set. Our results provide a foundation for reasoning about spatial externalities in ad allocation and for designing efficient, truthful mechanisms under such interactions.
View details
