Adaptable and Interpretable Neural Memory Over Symbolic Knowledge
Abstract
Past research has demonstrated that large neural language models (LMs) encode surprising amounts of factual information: however, augmenting or modifying this information requires modifying a corpus and retraining, which is computationally expensive. To
address this problem, we develop a neural LM
that includes an interpretable neuro-symbolic
KB in the form of a “fact memory”. Each
element of the fact memory is formed from
a triple of vectors, where each vector corresponds to a KB entity or relation. Our LM improves performance on knowledge-intensive
question-answering tasks, sometimes dramatically, including a 27 point increase in one setting of WebQuestionsSP over a state-of-the-art
open-book model, despite using 5% of the parameters. Most interestingly, we demonstrate
that the model can be modified, without any
re-training, by updating the fact memory
address this problem, we develop a neural LM
that includes an interpretable neuro-symbolic
KB in the form of a “fact memory”. Each
element of the fact memory is formed from
a triple of vectors, where each vector corresponds to a KB entity or relation. Our LM improves performance on knowledge-intensive
question-answering tasks, sometimes dramatically, including a 27 point increase in one setting of WebQuestionsSP over a state-of-the-art
open-book model, despite using 5% of the parameters. Most interestingly, we demonstrate
that the model can be modified, without any
re-training, by updating the fact memory
Research Areas
