Abstract:
An encoder as part of a language model is a mechanism for converting text information into an effective numerical representation which is suitable for solving a wide range of text processing tasks by means of neural network methods. This paper suggests a way of decomposing of the learning process for a language encoder. The author considers the issues of expediency of such decomposition taking into account reduction of computational costs, quality control at intermediate training stages, provision of the interpretability of the results on each stage. The quality evaluation of the encoder is given.
Key words and phrases:natural language processing, neural networks, language model, encoder, context-sensitive representations, lexical ambiguity resolution.
Fulltext:
PDF file (1157 kB)