Google Research

Metaphorical Inference Questions and Answers

Description

MiQA

Dataset for the paper MiQA: A Benchmark for Inference on Metaphorical Questions. MiQA assesses the capability of language models to reason with conventional metaphors. It combines the previously isolated topics of metaphor detection and commonsense reasoning into a single task that requires a model to make inferences by selecting between the literal and metaphorical register.

Data Format

The dataset is available in TSV (Tab-separated) format. The columns are: literal_premise, metaphorical_premise, literal_conclusion, metaphorical_conclusion.

Usage

Two types of MiQA questions ("implies" and "implied by") can be built using each row in the dataset.

Below are some examples:

"implies"-questions

"I see what you mean"

Which of the following statements could that imply?

(1) My eyes are working well [incorrect]

(2) I understand you [correct]

"A plan is not solid"

Which of the following statements could that imply?

(1) A hammer could break it [incorrect]

(2) We should not follow it [correct]

"My friend has a huge problem"

Which of the following statements could that imply?

(1) My friend needs space [incorrect]

(2) My friend needs a solution [correct]

"implied-by"-questions

"My eyes are working well"

is implied by which of the following?

(1) I see what you are pointing at [correct]

(2) I see what you mean [incorrect]

"A hammer could break it"

is implied by which of the following?

(1) A table is not solid [correct]

(2) A plan is not solid [incorrect]

"My friend needs space"

is implied by which of the following?

(1) My friend has a huge dog [correct]

(2) My friend has a huge problem [incorrect]