Dr Mariya Toneva | In search of model organisms for language in the human brain: Text vs speech language models

When processing the sentence “The trophy doesn’t fit into the brown suitcase because it’s too big”, we understand the meaning despite the ambiguous pronoun “it”, which may refer to either the trophy or the suitcase. How does the brain process this sentence and attribute real-world meaning to it? To address this, there are some fundamental preliminary questions to answer about what information is processed where and when in the brain, in order to understand how this information is aggregated across different locations and time points. Neuroscientists have made progress towards answering the what, where, and when questions. However, how the meaning of words is aggregated together by the brain still remains elusive. Meanwhile, recent advances in large language models (LLMs) have created computational systems that aggregate the meaning of words in specific ways in order to perform a specific linguistic task, such as predicting the upcoming word in a sentence. However, it is not clear whether these computational systems truly understand the meaning of a sentence, and whether the “how” of an LLM is the same as the “how” of the brain. In this talk, we will discuss recent work that specifically examines text vs speech LLMs as model organisms of language processing elicited by listening or reading to the same set of naturalistic stories. Overall, we present evidence that neurolinguistics can benefit from using LLMs as model organisms for how information is aggregated during language processing in the human brain, despite LLM’s differences from the human brain.

Poster