Abstract:
With the rapid growth in the use of generative neural network models for practical tasks,
the problem of explaining their decisions is becoming increasingly acute. As neural network-based
solutions are being introduced into medical practice, government administration, and defense, the
demands for interpretability of such systems will undoubtedly increase. In this study, we aim to
propose a method for verifying the reliability of self-explanations provided by models post factum by
comparing the attention distribution of the model during the generation of the response and its
explanation. The authors propose and develop methods for numerical evaluation of answers reliability
provided by generative pre-trained transformers. It is proposed to use the Kullback-Leibler divergence
over the attention distributions of the model during the issuance of the response and the subsequent
explanation. Additionally, it is proposed to compute the ratio of the model's attention between the
original query and the generated explanation to understand how much the self-explanation was
influenced by its own response. An algorithm for recursively computing the model's attention across
the generation steps is proposed to obtain these values. The study demonstrated the effectiveness of the
proposed methods, identifying metric values corresponding to correct and incorrect explanations and
responses. We analyzed the currently existing methods for determining the reliability of generative
model responses, noting that the overwhelming majority of them are challenging for an ordinary user to
interpret. In this regard, we proposed our own methods, testing them on the most widely used
generative models available at the time of writing. As a result, we obtained typical values for the
proposed metrics, an algorithm for their computation, and visualization.
Keywords:neural networks, metrics, language models, interpretability, GPT, LLM, XAI
Fulltext:
PDF file (529 kB)