HSE researchers are training large language models (LLMs) to understand Russian-language scientific terminology while improving their energy efficiency. The adapted model runs 2.7 times faster and requires 73% less memory than the original open model, allowing it to operate on more affordable hardware. The programme has passed state registration.
The volume of scientific and technological information, such as patents, articles, and reports, is growing rapidly every day. AI helps manage and use this enormous body of data effectively. Most LLMs currently available on the market are multilingual and trained on data in multiple languages. However, in popular generative AI chatbots such as ChatGPT, English-language data predominates, raising concerns about the emergence of an AI data monoculture.
Researchers at the HSE Institute for Statistical Studies and Economics of Knowledge further trained existing LLMs to create a tool capable of analysing Russian-language scientific texts more accurately while considering subject-specific aspects. The new model is based on the iFORA-QA data corpus, manually compiled by more than 150 ISSEK experts using analytical materials and reports in the fields of science, technology, and innovation.
After adaptation, the model demonstrated improved accuracy in answering highly technical questions related to science, technology, and innovation. Its generation speed increased by 2.7 times, while memory usage decreased by 73% compared to the original open multilingual model.
Anastasia Malashina
'Universal language models know a lot, but only at a superficial level. We need a model that understands what Russian scientists and engineers are writing about. Through our research, we were able to train the algorithm to think in domain-specific categories, recognise connections between complex concepts, and correctly interpret queries,' comments Anastasia Malashina, Chief Analyst of the project, Research Fellow and Leading Expert at the ISSEK Centre for Strategic Analytics and Big Data.
This year, the researchers will develop additional tools based on the adapted model. The first is a smart search engine designed to reduce the risk of hallucinations by ensuring that the model only produces conclusions with references to scientific sources. The second tool is a knowledge graph that will enable the identification of patterns, including hidden ones, based on the structure of the source materials. In addition, the model will gain the ability to work with incomplete and ambiguous information and to perform reasoning tasks: it will not only generate answers but also assess what information is missing, ask clarifying questions, and only then formulate a detailed response.
All these features will ultimately be integrated into a single multi-agent system capable of automatically solving complex problems.
'We are building an integrated system of intelligent agents adapted to the realities of Russian science. It will be based on a large language model and will be able to independently analyse scientific and technological information and identify hidden connections. This is a step toward automating scientific analytics, where AI becomes a researcher’s partner,' Malashina emphasises.
The AI model is being developed as part of a programme implemented by HSE University's AI Research Centre under a grant from the Russian Ministry of Economic Development.