On early detection of hallucinations in factual question answering

Abstract: While generative large language models (LLMs) show impressive abilities to create human like text, hallucinations remain a major impediment to their widespread adoption. In this work, we explore if the artifacts associated with the model can provide hints that a response will contain hallucinations. Specifically, we probe LLMs at 1) the inputs via integrated gradients based token attribution, 2) the outputs via the softmax probabilities, and 3) the internal state via the hidden layer and attention activations for signs of hallucinations on open ended question answering tasks. Our results show differences between hallucinations vs. non-hallucinations at all three levels, even when the first generated token is a formatting character, such as a new line. Specifically, we observer changes in entropy in input token attribution and output softmax probability for hallucinated tokens, revealing an “uncertain” behavior during model inference. This uncertain behavior also manifests itself in auxiliary classifier models trained on outputs and internal activations, which we use to create a hallucination detector. We further show that tokens preceding the hallucination can predict subsequent hallucinations before they occur.

Back