Beneath the surface of consistency: Exploring cross-lingual knowledge representation sharing in llms
Abstract
The veracity of a factoid is largely independent of the language it is written in. However, language models are inconsistent in their ability to answer the same factual question across languages. This raises questions about how they represent a given fact across languages.
We explore multilingual factual knowledge through two parallel aspects: the model's ability to answer a query consistently across languages ({\it consistency}), and to represent these answers for several languages ({\it shared representation}).
We propose a methodology to measure the extent to which LLMs share factual knowledge across languages, repurposing knowledge editing methods to assess cross-lingual generalization.
We examine LLMs of different types, including monolingual, bilingual, multilingual, and language-extended models.
Our analysis on a new multilingual dataset reveals that high consistency does not necessarily imply shared representation, particularly for languages with different scripts. We find that script similarity is a dominant factor in knowledge sharing, even in monolingual models. We observe that if LLMs could fully share knowledge across languages, their accuracy in their best-performing language could benefit an increase of up to 150\% on average. These findings highlight the need for improved multilingual knowledge representation in LLMs and suggest a path for the development of more robust and consistent multilingual LLMs.
We explore multilingual factual knowledge through two parallel aspects: the model's ability to answer a query consistently across languages ({\it consistency}), and to represent these answers for several languages ({\it shared representation}).
We propose a methodology to measure the extent to which LLMs share factual knowledge across languages, repurposing knowledge editing methods to assess cross-lingual generalization.
We examine LLMs of different types, including monolingual, bilingual, multilingual, and language-extended models.
Our analysis on a new multilingual dataset reveals that high consistency does not necessarily imply shared representation, particularly for languages with different scripts. We find that script similarity is a dominant factor in knowledge sharing, even in monolingual models. We observe that if LLMs could fully share knowledge across languages, their accuracy in their best-performing language could benefit an increase of up to 150\% on average. These findings highlight the need for improved multilingual knowledge representation in LLMs and suggest a path for the development of more robust and consistent multilingual LLMs.
