Image: © fifg/Stock.adobe.com
Yandex said it wants to promote new research into large language models by sharing best practices with the AI developer community.
Russian tech company Yandex said it has made a large language model trained on 100bn parameters available for free to the public, in order to boost the growth of this AI technology.
Large language models are natural language processing (NLP) systems that are trained on a massive volume of text. Yandex said they have become the “pinnacle” of neural networks used in NLP tasks.
The Russian company added that training these models requires millions of dollars, experts and years of development, which means only major companies have access to this technology.
“Researchers and developers all over the world need access to these solutions,” Yandex said in a blogpost. “Without new research, their growth could wane. The only way to avoid this is by sharing best practices with the developer community.”
Meta has also created a large language model that it is giving away to researchers.
But Yandex claimed its YaLM model is currently the world’s largest GPT-like neural network that is freely available for English. The Russian tech giant published the model and training materials on GitHub, under a licence that permits both research and commercial use.
On this GitHub page, Yandex said it took 65 days to train the YaLM model on a cluster of 800 A100 graphics cards and 1.7TB of online texts, books and “countless other sources” in English and Russian.
Yandex’s reasons for sharing its large language model are similar to Meta’s statements last month, when it announced plans to share its model that has 175bn parameters trained on publicly available datasets.
“Meta AI believes that collaboration across research organisations is critical to the responsible development of AI technologies,” the company said at the time.
Yandex and Meta are not the only companies looking into large language models. Last October, tech giants Microsoft and Nvidia teamed up to create a language model with 105 layers and 530bn parameters, three times as many parameters as OpenAI’s GPT-3 model.
Yandex is Russia’s largest tech company, providing online tools and services including a search engine, email, news aggregator and apps for navigation, translation, ride-hailing and more.
The company, which has come under the microscope for its ties to the Kremlin, has had a turbulent few months. Its CEO stepped down earlier this month after the EU included him on list of sanctions against Russian.
10 things you need to know direct to your inbox every weekday. Sign up for the Daily Brief, Silicon Republic’s digest of essential sci-tech news.
