Publications
Our teams aspire to make discoveries that impact everyone, and core to our approach is sharing our research and tools to fuel progress in the field.
Publications
Sort By
1 - 15 of 107 publications
APG: Audioplethysmography for Cardiac Monitoring in Hearables
David Pearl
Rich Howard
Longfei Shangguan
Trausti Thormundsson
MobiCom 2023: The 29th Annual International Conference On Mobile Computing And Networking (MobiCom), Association for Computing Machinery (ACM) (to appear)
Preview abstract
This paper presents Audioplethysmography (APG), a novel cardiac monitoring modality for active noise cancellation (ANC) headphones. APG sends a low intensity ultrasound probing signal using an ANC headphone's speakers and receives the echoes via the on-board feedback microphones. We observed that, as the volume of ear canals slightly changes with blood vessel deformations, the heartbeats will modulate these ultrasound echoes. We built mathematical models to analyze the underlying physics and propose a multi-tone APG signal processing pipeline to derive the heart rate and heart rate variability in both constrained and unconstrained settings. APG enables robust monitoring of cardiac activities using mass-market ANC headphones in the presence of music playback and body motion such as running.
We conducted an eight-month field study with 153 participants to evaluate APG in various conditions. Our studies conform to the (Institutional Review Board) IRB policies from our company. The presented technology, experimental design, and results have been reviewed and further improved by feedback garnered from our internal Health Team, Product Team, User Experience (UX) Team and Legal team. Our results demonstrate that APG achieves consistently high HR (3.21% median error across 153 participants in all scenarios) and HRV (2.70% median error in interbeat interval, IBI) measurement accuracy. Our UX study further shows that APG is resilient to variation in: skin tone, sub-optimal seal conditions, and ear canal size.
View details
Design Earable Sensing Systems: Perspectives and Lessons Learned from Industry
Trausti Thormundsson
Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2023 ACM International Symposium on Wearable Computers, ACM (to appear)
Preview abstract
Earables computing is an emerging research community as the industry witnesses the soaring of True Wireless Stereo (TWS) Active Noise Canceling (ANC) earbuds in the past ten years. There is an increasing trend of newly initiated earable research spanning across mobile health, user-interfaces, speech processing, and context-awareness. Head-worn devices are anticipated to be the next generation Mobile Computing and Human-Computer Interaction (HCI) platform. In this paper, we share our design experiences and lessons learned in building hearable sensing systems from the industry perspective. We also give our takes on future directions of the earable research.
View details
Preview abstract
Advanced AR/VR headsets often have a dedicated depth sensor or multiple cameras, high processing power, and a highcapacity battery to track hands or controllers. However, these approaches are not compatible with the small form factor and limited thermal capacity of lightweight AR devices. In this paper, we present RetroSphere, a self-contained 6 degree of freedom (6DoF) controller tracker that can be integrated with almost any device. RetroSphere tracks a passive controller with just 3 retroreflective spheres using a stereo pair of mass-produced infrared blob trackers, each with its own infrared LED emitters. As the sphere is completely passive, no electronics or recharging is required. Each object tracking camera provides a tiny Arduino-compatible ESP32 microcontroller with the 2D position of the spheres. A lightweight stereo depth estimation algorithm that runs on the ESP32 performs 6DoF tracking of the passive controller. Also, RetroSphere provides an auto-calibration procedure to calibrate the stereo IR tracker setup. Our work builds upon Johnny Lee’s Wii remote hacks and aims to enable a community of researchers, designers, and makers to use 3D input in their projects with affordable off-the-shelf components. RetroSphere achieves a tracking accuracy of about 96.5% with errors as low as ∼3.5 cm over a 100 cm tracking range, validated with ground truth 3D data obtained using a LIDAR camera while consuming around 400 mW. We provide implementation details, evaluate the accuracy of our system, and demonstrate example applications, such as mobile AR drawing, 3D measurement, etc. with our Retrosphere-enabled AR glass prototype.
View details
Preview abstract
Consumer electronics are increasingly using everyday materials to blend into home environments, often using LEDs or symbol displays under textile meshes. Our surveys (n=1499 and n=1501) show interest in interactive graphical displays for hidden interfaces --- however, covering such displays significantly limits brightness, material possibilities and legibility.
To overcome these limitations, we leverage parallel rendering to enable ultrabright graphics that can pass through everyday materials. We unlock expressive hidden interfaces using rectilinear graphics on low-cost, mass-produced passive-matrix OLED displays. A technical evaluation across materials, shapes and display techniques, suggests 3.6--40X brightness increase compared to more complex active-matrix OLEDs.
We present interactive prototypes that blend into wood, textile, plastic and mirrored surfaces. Survey feedback (n=1572) on our prototypes suggests that smart mirrors are particularly desirable. A lab evaluation (n=11) reinforced these findings and allowed us to also characterize performance from hands-on interaction with different content, materials and under varying lighting conditions.
View details
Preview abstract
We introduce a framework for adapting a virtual keyboard to individual user behavior by modifying a Gaussian spatial model to use personalized key center offset means and, optionally, learned covariances. Through numerous real-world studies, we determine the importance of training data quantity and weights, as well as the number of clusters into which to group keys to avoid overfitting. While past research has shown potential of this technique using artificially-simple virtual keyboards and games or fixed typing prompts, we demonstrate effectiveness using the highly-tuned Gboard app with a representative set of users and their real typing behaviors. Across a variety of top languages,we achieve small-but-significant improvements in both typing speed and decoder accuracy.
View details
Preview abstract
Over the years, the mobile devices landscape has grown to hundreds of OEMs and tens of thousands of device models. This landscape has made it difficult to develop quality mobile applications that are user-friendly, stable, and performant. To accelerate mobile development and to help developers build better performing, more stable apps, Google built a large Mobile Device Farm that allows developers to test their mobile applications. In this document we share lessons learned while building the Mobile Device Farm, including pitfalls and successes, to help the mobile development community leverage our experience and build better mobile apps. While we describe both Android and iOS, we primarily focus on the Android Open Source Project (AOSP) because it is highly diverse and dynamic. We've scaled from 10s of devices to 10s of thousands, and from hundreds of tests a day to millions.
View details
PriGen: Towards Automated Translation of Android Applications' Code to Privacy Captions
Vijayanta Jain
Sanonda Datta Gupta
Sepideh Ghanavati
Research Challenges in Information Science, Springer International Publishing (2021), pp. 142-151
Preview abstract
Mobile applications are required to give privacy notices to the users when they collect or share personal information. Creating consistent and concise privacy notices can be a challenging task for developers. Previous work has attempted to help developers create privacy notices through a questionnaire or predefined templates. In this paper, we propose a novel approach and a framework, called PriGen, that extends these prior work. PriGen uses static analysis to identify Android applications’ code segments which process personal information (i.e. permission-requiring code segments) and then leverages a Neural Machine Translation model to translate them into privacy captions. We present the initial analysis of our translation task for ~300,000 code segments.
View details
EdgeSharing: Edge Assisted Real-time Localization and Object Sharing in Urban Streets
Marco Gruteser
The 40th IEEE International Conference on Computer Communications (IEEE INFOCOM 2021). (2021)
Preview abstract
Collaborative object localization and sharing at smart intersections promises to improve situational awareness of traffic participants in key areas where hazards exist due to visual obstructions. By sharing a moving object's location between different camera-equipped devices, it effectively extends the vision of traffic participants beyond their field of view. However, accurately sharing objects between moving clients is extremely challenging due to the high accuracy requirements for localizing both the client position and positions of its detected objects. Existing approaches based on direct sharing between devices are limited by the computational resources of (potentially legacy) vehicles and bad flexibility. To address these challenges, we introduce EdgeSharing, a novel localization and object sharing system leveraging the resources of edge cloud platforms. EdgeSharing holds a real-time 3D feature map of its coverage region on the edge cloud and uses it to provide accurate localization and object sharing service to the client devices passing through this region. We further propose several optimization techniques to increase the localization accuracy, reduce the bandwidth consumption and decrease the offloading latency of the system. The result shows that the system is able to achieve a mean vehicle localization error of 0.2813-1.2717 meters, an object sharing accuracy of 82.3\%-91.44\%, and a 54.68\% object awareness increment in urban streets and intersections. In addition, the proposed optimization techniques are able to reduce 70.12\% of bandwidth consumption and reduce 40.09\% of the end-to-end latency.
View details
ASAP: Fast Mobile Application Switch via Adaptive Prepaging
Sam Son
Seung Yul Lee
Jonghyun Bae
Yunho Jin
Jinkyu Jeong
Tae Jun Ham
Jae W. Lee
USENIX Association, ., pp. 365-380
Preview abstract
With ever-increasing demands for memory capacity from a mobile application, along with a steady increase in the number of applications running concurrently, the memory capacity is becoming a scarce resource on mobile devices. When the memory pressure is high, current mobile OSes often kill application processes that have not recently been used to reclaim memory space. This leads to a long delay when the user relaunches the killed application, which degrades the user experience. Even if this mechanism is disabled to utilize a compression-based in-memory swap mechanism, relaunching the application still incurs a substantial latency penalty as it requires decompression of compressed anonymous pages and a stream of I/O accesses to retrieve file-backed pages into memory. This paper identifies the conventional demand paging as the primary source of this inefficiency and proposes ASAP, a mechanism for fast application switch via adaptive prepaging on mobile devices. Specifically, ASAP performs prepaging effectively by combining i) high-precision switch footprint estimators for both file-backed and anonymous pages, and ii) efficient implementation of the prepaging mechanism to minimize resource wastes for CPU cycles and disk bandwidth during an application switch. Our evaluation of ASAP using eight real-world applications on Google Pixel 4 demonstrates that ASAP can reduce the switch time by 22.2% on average (with 33.3% at maximum) over the vanilla Android 10.
View details
Successful Internship In Antenna And Electromagnetic Design: From A Host Perspective
Jiang Zhu
IEEE Antennas and Propagation Magazine (2021)
Preview abstract
No abstract
View details
Holography-Based Target Localization and Health Monitoring Technique using UHF Tag Array
Aline Eid
Jiang Zhu
Jimmy G. D. Hester
Manos M. Tentzeris
IEEE Intern of Things Journal (2021)
Preview abstract
Radio technologies are appealing for unobtrusive and remote monitoring of human activities. Radar based human activity recognition proves to be a success, for example, Project Soli developed by Google. However, it is expensive to scale up for multi-user environments. In this paper, we propose a solution—the HoloTag system—which circumvents the multi-channel-radar scaling problem through the use of a quasi-virtual ultra-low-cost UHF RFID array over which a holographic projection of its environment is measured and used to both localize and monitor the health of several targets. The method is first described in detail, before the image reconstruction process, employing known beamforming algorithms—Delay & Sum, and Capon—is shown and its scaling properties simulated. Then, the idiosyncrasies of the implementation of HoloTag using low-cost Off-The-Shelf hardware are explained, before its ability to simultaneously measure the breathing rates and positions of multiple real and synthetic targets with accuracies of better than 0.8 bpm and 20 cm is demonstrated.
View details
Preview abstract
Personalization methods in federated learning aim to balance the benefits of federated and local training for data availability, communication cost, and robustness to client heterogeneity. Approaches that require clients to communicate all model parameters can be undesirable due to privacy and communication constraints. Other approaches require always-available or stateful clients, impractical in large-scale cross-device settings. We introduce Federated Reconstruction, the first model-agnostic framework for partially local federated learning suitable for training and inference at scale. We motivate the framework via a connection to model-agnostic meta learning, empirically demonstrate its performance over existing approaches for collaborative filtering and next word prediction, and release an open-source library for evaluating approaches in this setting. We also describe the successful deployment of this approach at scale for federated collaborative filtering in a mobile keyboard application.
View details
Streaming keyword spotting on mobile devices
Natasha Kononenko
Niranjan Subrahmanya
Mirkó Visontai
Stella Marie Laurenzo
INTERSPEECH, ISCA, Shanghai China (2020)
Preview abstract
In this work we explore the latency and accuracy of keyword spotting (KWS) models in streaming and non-streaming modes on mobile phones. NN model conversion from non-streaming mode (model receives the whole input sequence and then returns the classification result) to streaming mode (model receives portion of the input sequence and classifies it incrementally) may require manual model rewriting. We address this by designing a Tensorflow/Keras based library which allows automatic conversion of non-streaming models to streaming ones with minimum effort. With this library we benchmark multiple KWS models in both streaming and non-streaming modes on mobile phones and demonstrate different tradeoffs between latency and accuracy. We also explore novel KWS models with multi-head attention which reduce the classification error over the state-of-art by 10% on Google speech commands data sets V2. The streaming library with all experiments is open-sourced.
View details
Wearable Antennas for Cross-Body Communication And Human Activity Recognition
Wenjing Su
Jiang Zhu
Huan Liao
Manos M. Tentzeris
Ieee Access (2020)
Preview abstract
This paper proposes a novel miniaturized wearable antenna (mosaic antenna) for cross-body communication featuring a superior performance compared to five commonly utilized wearable antenna topologies. The proposed mosaic antenna optimally excites the surface waves to enable cross-body communication yet still with a low profile and a miniaturized size. As a proof-of-concept of the unique capabilities of this antenna, an inkjet-printed flexible wearable prototype on Polytetrafluoroethylene (PTFE, also known as Teflon) has been fabricated in a highly customizable, scalable and low-cost process and has been used as an enabler for human activities recognition coupled with machine learning techniques powered by Google TensorFlow. By utilizing proof-of-concept on-body networks consisting of mosaic antennas, human activity can be detected with an accuracy 91.9% for RF signal-based recognition techniques, despite the variability among different users and environments.
View details
Modeling and Reducing Spatial Jitter caused by Asynchronous Input and Output Rates
Axel Antoine
Mathieu Nancel
Ella Ge
Navid Zolghadr
Géry Casiez
The 33rd Annual ACM Symposium on User Interface Software and Technology, ACM, New York, NY (2020), 13p (to appear)
Preview abstract
Jitter in interactive systems occurs when visual feedback is perceived as unstable or trembling even though the input signal is smooth or stationary. It can have multiple causes such as sensing noise, or feedback calculations introducing or exacerbating sensing imprecisions. Jitter can however occur even when each individual component of the pipeline works perfectly, as a result of the differences between the input frequency and the display refresh rate. This asynchronicity can introduce rapidly-shifting latencies between the rendered feedbacks and their display on screen, which can result in trembling cursors or viewports. This paper contributes a better understanding of this particular type of jitter. We first detail the problem from a mathematical standpoint, from which we develop a predictive model of jitter amplitude as a function of input and output frequencies, and a new metric to measure this spatial jitter. Using touch input data gathered in a study, we developed a simulator
to validate this model and to assess the effects of different techniques and settings with any output frequency. The most promising approach, when the time of the next display refresh is known, is to estimate (interpolate or extrapolate) the user’s position at a fixed time interval before that refresh. When input events occur at 125 Hz, as is common in touch screens, we show that an interval of 4 to 6 ms works well for a wide range of display refresh rates. This method effectively cancels most of the jitter introduced by input/output asynchronicity, while introducing minimal imprecision or latency.
View details
