Advancing personal health and wellness insights with AI

Mobile and wearable devices can provide continuous, granular, and longitudinal data on an individual’s physiological state and behaviors. Examples include step counts, raw sensor measurements such as heart rate variability, sleep duration, and more. Individuals can use these data for personal health monitoring as well as to motivate healthy behavior. This represents an exciting area in which generative AI models can be used to provide additional personalized insights and recommendations to an individual to help them reach their health goals. To do so, however, models must be able to reason about personal health data comprising complex time series and sporadic information (like workout logs), contextualize these data using relevant personal health domain knowledge, and produce personalized interpretations and recommendations grounded in an individual’s health context.

Consider a common health query, “How can I get better sleep?” Though a seemingly straightforward question, arriving at a response that is customized to the individual involves performing a series of complex analytical steps, such as: checking data availability, calculating average sleep duration, identifying sleep pattern anomalies over a period of time, contextualizing these findings within the individual's broader health, integrating knowledge of population norms of sleep, and offering tailored sleep improvement recommendations. Recently, we showed how building on Gemini models’ advanced capabilities in multimodality and long-context reasoning could enable state-of-the-art performance on a diverse set of medical tasks. However, such tasks rarely make use of complex data sourced from mobile and wearable devices relevant for personal health monitoring.

Building on the next-generation capabilities of Gemini models, we present research that highlights two complementary approaches to providing accurate personal health and wellness information with LLMs. The first paper, “Towards a Personal Health Large Language Model”, demonstrates that LLMs fine-tuned on expert analysis and self-reported outcomes are able to successfully contextualize physiological data for personal health tasks. The second paper, “Transforming Wearable Data into Personal Health Insights Using Large Language Model Agents”, emphasizes the value of code generation and agent-based workflows to accurately analyze behavioral health data through natural language queries. We believe that bringing these ideas together, to enable interactive computation and grounded reasoning over personal health data, will be critical components for developing truly personalized health assistants. With these two papers, we curate new benchmark datasets across a range of personal health tasks, which help evaluate the effectiveness of these models.

For the insights and recommendations tasks, we created 857 case studies from consenting US-based users across two personal health verticals: sleep and fitness. These case studies, designed in collaboration with domain experts, represent real-world coaching scenarios and highlight the model’s capabilities in understanding, reasoning, and coaching by interpreting time-series physiological data using textual representations. Through comprehensive evaluation of model responses, we observe that performance of both Gemini Ultra 1.0 and PH-LLM is not statistically different from expert performance in fitness. While recommendations written by experts are rated higher for sleep, the performance is close, and further fine-tuning PH-LLM significantly improves its ability to use relevant domain knowledge and personalize information when generating insights and predicting potential causal factors.

To further assess expert domain knowledge, we evaluated PH-LLM performance on multiple choice question datasets in the style of sleep medicine and fitness certification examinations via manual test takers using online portals. PH-LLM achieved 79% on sleep (N=629 questions) and 88% on fitness (N=99 questions), both of which exceed average scores from a sample of human experts (76% and 71%, respectively) as well as benchmarks for receiving continuing education credit to maintain professional licenses in those domains.

Finally, to enable PH-LLM to predict self-reported assessments of sleep quality, we trained the model on responses to validated survey questions on sleep disruption and impairment using textual and multimodal encoding representations of wearable sensor data. Shown below, we demonstrate that multimodal encoding is both necessary and sufficient to achieve performance on par with discriminative models trained solely to predict these outcomes.

Together, these results demonstrate the benefit of tuning PH-LLM to contextualize physiological data for personal health applications.

To evaluate the agent’s capabilities, we curated two datasets: one to test the agent's numerical accuracy in health queries, and the other to assess the quality of its reasoning and code in open-ended health queries through human annotations.

On the first dataset, Objective Health Insights Queries, the agent achieved 84% accuracy on a dataset of 4,000 objective personal health insights queries, demonstrating its ability to handle numerical reasoning and data analysis.

On the second dataset, Open-Ended Health Insights Queries, we assess the agent’s performance on 172 representative open-ended personal health queries across over 600 hours of human evaluation of more than 6000 model responses. Overall, the agent significantly improved performance over a non-agent code generation baseline in nine of the 14 axes of evaluation, including key aspects like domain knowledge, logic, and reasoning quality.

While the agent focuses on sleep and fitness data, its framework can be extended to analyze a broader range of health information, including medical records, nutrition data, and even user-provided journal entries. As LLMs continue to advance, agents have the potential to become increasingly sophisticated and may offer even deeper insights and more effective guidance for personal health management.