Fraunhofer Institute for Intelligent Analysis and Information Systems IAISBreakthrough for generative AI research in Germany and Europe
Together with AI Sweden, Fraunhofer IAIS gains large-scale computing capacities for training large language models
The Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS and the NLU group of AI Sweden have jointly received computational capacity on the new high-performance computer MareNostrum 5 at the Barcelona Supercomputing Center. This is one of the largest contingents granted by the The European High Performance Computing Joint Undertaking (EuroHPC JU) for the development of European large language models (LLMs) on the EuroHPC infrastructure. From the end of May 2024, the partners will start computing the first multilingual models. The “EuroLingua-GPT” project will run for one year. This means that large European multilingual open source models are now in sight.
The contingent approved via a EuroHPC “Extreme Scale Access” comprises 8.8 million GPU hours on H100 chips and has been available since May. “These computing capacities are a milestone for Germany and Europe. The models trained with it will massively accelerate the use of generative AI in companies and give both business and science a boost – GenAI 'made in Europe' is thus becoming a reality,” says Dr. Joachim Köhler, head of the NetMedia department at Fraunhofer IAIS. With the new computing capacities, small models in the range of 7 to 34 billion parameters and large models with up to 180 billion parameters can be trained from scratch.
One model family, all European languages – Fraunhofer IAIS and AI Sweden combine their expertise
The new EuroLingua models are based on a training dataset consisting of 45 European languages, dialects and codes, including the 24 official European languages. This gives a significant weight to European languages and values – multilingual large language models are still rare. Training will start at the end of May 2024 and the first joint models are expected to be published in the coming months.
Project leader Dr. Nicolas Flores-Herr, team leader Conversational AI at Fraunhofer IAIS says: “The goal of our collaboration with AI Sweden is to train a family of large language models from scratch that will be published open source.” Magnus Sahlgren, head of Research NLU at AI Sweden adds: “Both the public and private sectors in the EU are asking for open, powerful language models trained for European languages. This is one way to meet that need.”
The models developed on the EuroHPC infrastructure are intended on the one hand to serve as generalist basic models to support research and science, and on the other hand – for example in joint transfer projects – to provide specialized models for specific sectors or areas of application for productive use in companies or public administration.
To achieve this, the two organizations are pooling their expertise: Fraunhofer IAIS andAI Sweden's NLU group are two of the leading LLM labs in Europe with proven expertise and years of experience in developing LLMs. For example, Fraunhofer is leading the OpenGPT-X consortium project funded by the Federal Ministry for Economic Affairs and Climate Protection (BMWK), in which large European, multilingual open source models are also being developed. The NLU group at AI Sweden has developed the GPT-SW3 LLM for the Scandinavian languages. The two teams are also working together on other open source community projects. EuroLingua-GPT is also one of three major ongoing EU projects on language models in which Fraunhofer IAIS and AI Sweden collaborate. The other two are TrustLLM and Deploy AI.
About Fraunhofer IAIS
As part of the largest organization for applied research in Europe, the Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, based in Sankt Augustin/Bonn and Dresden, is one of the leading scientific institutes in the fields of Artificial Intelligence (AI), Machine Learning and Big Data in Germany and Europe. Around 380 employees support companies in the optimization of products, services, as well as in the development of new technologies, and processes, and new digital business models. Fraunhofer IAIS is shaping the digital transformation of our working and living environments: with innovative AI applications for industry, health, and sustainability, with forward-looking technologies such as large-scale AI language models or Quantum Machine Learning, with offers for training and education or for the testing of AI applications for security and trustworthiness.
About AI Sweden
AI Sweden is Sweden's national center for applied AI, with offices across the country and in Montreal. AI Sweden brings together more than 120 partners across the public and private sectors as well as academia. The center is funded by the Swedish government and the center’s partners, both public and private. Together, they invest in generating tools and resources to accelerate the use of AI for the benefit of Sweden’s society, competitiveness, and everyone living in Sweden. In November 2023, AI Sweden in collaboration with WASP and RISE, released GPT-SW3, a LLM specifically developed for Swedish and Nordic languages.