Multi-agentic framework for developing and evaluating biometric-based multimodal synthetic personas: A case study on pregnancy and postpartum users
Abstract
The use of Large language models (LLM)- based persona health agents for conversational health insights with wearable-based applications is seeing high interest and adoption. These allow users to interact with their wearable biometric data, and ask an agent about their biometrics such as sleep, activity, and heart rate.
However, there are limited pipelines for robustly evaluating these agentic frameworks that utilize multimodal - time-series and text- based data. Firstly, biometrics data presents complications around using sensitive, health information and biomarkers. This limits the availability of high-quality, open-source dataset. Additionally there are limited accompanying use-case vignettes and assessment of interactions between users of wearables and personalized health agents from diverse applications and health conditions to assess suitability and feasibility, and to determine user-based improvements to the agents.
To bridge this gap, we have developed a multi-agent evaluation framework that generates synthetic personas, utilizing intrinsic knowledge from biometrics to evaluate and improve personalized health experience using biometric data. Our work focuses on using data from a pregnancy and postpartum population that presents a period of distinct changes in biometrics, highly sensitive data, and the need for highly accurate biometric retrieval, and contextual question answering. We utilize a rich dataset, which includes biometrics and survey responses, obtained through an IRB approved study from 3,467 participants during their pregnancy and postpartum period. We develop a pipeline for generating synthetic time series datasets, implementing use-case specific personas and vignettes, evaluating interactions between LLM agents and the personas .
Overall, the goal of this work is to develop a cohort of synthetic personas, that have accompanying synthetic time series data, along with different health outcomes, and demographics that can be used to evaluate and optimize LLM-based agents for pregnancy and postpartum. Our specific contributions include:
-Develop a TimeGAN for creating synthetic time series from the real population data that accurately reflects nuances of data missingness, and population distributions, while enabling privacy preservation
- Combine synthetic time series data with LLM-based multi agent orchestrations to generate biometrically-aligned multimodal persona profiles/ vignettes.
- Develop a framework for multi turn conversation using the synthetic personas and evaluate persona quality for real-world plausibility using LLM and human raters.
- Evaluation of an array of LLMs for their performance in responding to questions from synthetic personas to demonstrate applicability for evaluating and optimizing LLM-based personal health agents.
While this work focuses on pregnancy and postpartum,the tool and methods can be applied generally to other use cases that employ biometric time series data. Outcomes from this work will provide robust tooling for synthetic persona creation to understand how LLM agents work for pregnancy and postpartum and what gaps need to be addressed.
However, there are limited pipelines for robustly evaluating these agentic frameworks that utilize multimodal - time-series and text- based data. Firstly, biometrics data presents complications around using sensitive, health information and biomarkers. This limits the availability of high-quality, open-source dataset. Additionally there are limited accompanying use-case vignettes and assessment of interactions between users of wearables and personalized health agents from diverse applications and health conditions to assess suitability and feasibility, and to determine user-based improvements to the agents.
To bridge this gap, we have developed a multi-agent evaluation framework that generates synthetic personas, utilizing intrinsic knowledge from biometrics to evaluate and improve personalized health experience using biometric data. Our work focuses on using data from a pregnancy and postpartum population that presents a period of distinct changes in biometrics, highly sensitive data, and the need for highly accurate biometric retrieval, and contextual question answering. We utilize a rich dataset, which includes biometrics and survey responses, obtained through an IRB approved study from 3,467 participants during their pregnancy and postpartum period. We develop a pipeline for generating synthetic time series datasets, implementing use-case specific personas and vignettes, evaluating interactions between LLM agents and the personas .
Overall, the goal of this work is to develop a cohort of synthetic personas, that have accompanying synthetic time series data, along with different health outcomes, and demographics that can be used to evaluate and optimize LLM-based agents for pregnancy and postpartum. Our specific contributions include:
-Develop a TimeGAN for creating synthetic time series from the real population data that accurately reflects nuances of data missingness, and population distributions, while enabling privacy preservation
- Combine synthetic time series data with LLM-based multi agent orchestrations to generate biometrically-aligned multimodal persona profiles/ vignettes.
- Develop a framework for multi turn conversation using the synthetic personas and evaluate persona quality for real-world plausibility using LLM and human raters.
- Evaluation of an array of LLMs for their performance in responding to questions from synthetic personas to demonstrate applicability for evaluating and optimizing LLM-based personal health agents.
While this work focuses on pregnancy and postpartum,the tool and methods can be applied generally to other use cases that employ biometric time series data. Outcomes from this work will provide robust tooling for synthetic persona creation to understand how LLM agents work for pregnancy and postpartum and what gaps need to be addressed.
