Abstract:
The rapid growth in the number of annually published scientific papers places a significant burden on the editors of scientific journals and conference organizers. In particular, quickly selecting relevant reviewers becomes difficult. Automation of this process is hampered by the lack of publicly available information on reviewers of already published articles in accordance with the requirements of double-blind peer review. In this paper we aimed to take the first steps toward developing an reviewer recommendation system. Our research focuses on Russian-language scientific articles on mathematics. At the core of our approach lies the comparison of the semantics of the target paper with those of the papers available in the external library. The most similar papers from the library are then aggregated by author, resulting in a list of potential reviewers. This list is subsequently refined through a series of filters. Additionally, we experimented with an extra step: re-ranking the most relevant candidates using a large language model (LLM). To assess the quality of recommendations, we introduced several metrics based on the Universal Decimal Classification (UDC) system, specifically UDC Jaccard similarity and UDC accuracy. The best results were achieved using the E5-multilingual and E5-mistral embedding models. Overall, we were able to achieve a quality higher than 0.88 according to UDC Accuracy@1. The introduction of the LLM-based reranking stage showed mixed results based on preliminary evaluation. While it improved precision and recall metrics at lower k values, human evaluation indicated a preference for the system configuration without reranking. At the same time, the experts’ assessments were predominantly positive: most recommendations received ratings of 4 and 5 on a five-point scale.
Key words and phrases:scientific texts, reviewer selection, large language models, text embeddings, recommendation systems.
Fulltext:
PDF file (597 kB)