Fine-tuning large language models for hypernym discovery task: sister terms do their part

F. Sadkovskii^ab, N. Loukachevitch^b, I. Grishin<sup>b

a</sup> RAS Institute of Linguistics
^b Lomonosov Moscow State University

Abstract: This paper addresses the problem of bias introduced by cohyponyms–nodes sharing the same parent hypernym–in training datasets for hypernym discovery tasks. While the removal of test items from training data is essential for preventing data leakage, we argue that excluding cohyponyms is equally critical. When fine-tuning a model on on a dataset composed of hyponym-hypernym pairs extracted from a taxonomic resource WordNet, puncturing only test nodes is not enough to adequately assess the quality of the model on test data. Cohyponyms act as implicit hints for identifying hypernyms, artificially enhancing the performance of model and misrepresenting its utility in real-world scenarios. We fine-tuned LLaMA-2 using the TaxoLLaMA training procedure of Moskvoretskii et al. (2024) on an extensive number of WordNet-derived subsamples of hyponym–hypernym pairs with and without their definitions. Evaluation on the SemEval-2018 dataset showed that including co-hyponyms in the training data artificially inflates performance metrics.

Key words and phrases: hypernym discovery, taxonomy enrichment, wordNet, taxoLLaMA.

UDC: 004.912

Received: 28.02.2025

Language: English