Responsible Health AI: Psychometric Validation of a Novel Health Information Quality Assessment Tool for GenAI
Abstract
Background: Users increasingly turning to consumer GenAI platforms for health-related information and guidance necessitates validated instruments to safeguard clinical accuracy and user trust.nThis study introduces the Health Information Quality (HIQ) tool, a psychometric instrument designed to operationalize Responsible AI and establish transparent benchmarking methods for foundational models.
Methods: Validity was assessed via a three-round modified Delphi process with 23 multi-disciplinary experts, assessing item relevance across Accuracy, Safety, and Usefulness constructs. Content validity was refined through semi-structured rater interviews. A ground-truth dataset across five clinical specialties was used to assess item performance, reliability and validity.
Results: After multi-round Delphi panel review, 19 of 41 items received consensus to comprise the pilot HIQ tool. Item analysis and internal consistency evaluation were used to optimize instrument sensitivity. Descriptive statistics and sub-construct clustering informed the identification of a parsimonious, minimally sufficient item set for the final tool.
Conclusion: This research presents the first attempt to develop a psychometrically sound framework for evaluating GenAI health outputs. A tool such as this intends to provide a structured, consistent framework for GenAI platforms sharing health information and for stakeholders evaluating its use, supporting the long-term goals of public trust and safety.
Methods: Validity was assessed via a three-round modified Delphi process with 23 multi-disciplinary experts, assessing item relevance across Accuracy, Safety, and Usefulness constructs. Content validity was refined through semi-structured rater interviews. A ground-truth dataset across five clinical specialties was used to assess item performance, reliability and validity.
Results: After multi-round Delphi panel review, 19 of 41 items received consensus to comprise the pilot HIQ tool. Item analysis and internal consistency evaluation were used to optimize instrument sensitivity. Descriptive statistics and sub-construct clustering informed the identification of a parsimonious, minimally sufficient item set for the final tool.
Conclusion: This research presents the first attempt to develop a psychometrically sound framework for evaluating GenAI health outputs. A tool such as this intends to provide a structured, consistent framework for GenAI platforms sharing health information and for stakeholders evaluating its use, supporting the long-term goals of public trust and safety.
