Measuring Cultural Intelligence of AI Systems: A Unified Framework
Abstract
As generative AI technologies are increasingly being launched across the globe, assessing their competence to operate in different cultural contexts is increasingly becoming a priority. Recent years have seen numerous and much-needed efforts on cultural benchmarking, these efforts tend to focus on specific aspects of culture. While these efforts contribute to our understanding of cultural competence, a unified and systematic evaluation approach is needed for us as a field to comprehensively assess diverse cultural dimensions at scale. In this paper, we propose a principled approach towards developing a unified framework for evaluating cultural intelligence of models. We start by developing a working definition of culture that includes identifying core domains of culture. We then introduce a broad-purpose, systematic, and extensible framework for assessing cultural intelligence of AI systems. Drawing on theoretical framing from psychometric measurement validity theory, we decouple the background concept (i.e., cultural intelligence) from its operationalization via measurement. We conceptualize cultural intelligence in terms of a set of core capabilities that span different cultural domains, before operationalizing them in terms of a set of indicators designed to reliably measure these capabilities collectively. Finally, we identify the considerations, challenges, and research pathways to meaningfully measure these indicators, specifically focusing on data collection, probing strategies, and evaluation metrics.
