Using LSTM-based Language Models and human Eye Movements metrics to understand next-word predictions

Abstract

Modern Natural Language Processing models can achieve great results on linguistic tasks. For example, LSTM-based models can generate abstractions to make predictions about upcoming words. This ability opens a window in the cognitive neuroscience field. It is known that the probability that a reader knows a word before reading it (i.e., cloze-Predictability) impacts on the time spent on it. Nevertheless, little is known about when or how these predictions are made. Here, we trained LSTM-based models to predict future words and used their predictions to replace cloze-Predictability in statistical models from the neuroscience field. We found that the LSTM-Predictability can model eye movements with high overlap with both cloze-Predictability and the lexical frequency. Also, this performance varies depending on the training corpus. This study is a step forward in understanding how our brain performs predictions during reading.