Helsinki-based Silo AI, the largest private AI lab in Europe, announced on Wednesday that it has completed the training of the Poro model, together with the University of Turku and HPLT.
According to the company, this represents a significant breakthrough for SiloGen, their generative AI division. It is part of their efforts to strengthen European digital sovereignty and democratise access to large language models (LLMs) for all European languages.
Last year, Silo AI acquired Amsterdam’s Machine2Learn, experts in Machine Learning, to expand its presence in the Benelux area and throughout Western Europe.
Additionally, the acquisition also supports its aim to ensure European digital sovereignty, and to build a leading generative AI and LLM team in the region.
Strengthening European digital sovereignty
Silo AI is working on creating a series of multilingual open source Language Models (LLMs) to strengthen European digital sovereignty and democratising access to LLMs.
It is essential to develop base models that conform to European values and are based on data and information that accurately represent the varied languages, citizens, organisations, and cultural landscape of the European Union.
This approach not only aligns with European values but also allows for independence in how downstream applications and value creation occur.
Outperforms Llama, FinGPT, and others
Silo AI claims that Poro outperforms all existing open language models in the Finnish language, including FinGPT, Mistral, Llama, and the BLUUMI 176 billion parameter model, among others.
This success is attributed to pairing the low-resource Finnish language with high-resource languages.
The team has determined the optimal data reuse frequency for low-resource languages during training and has incorporated translated paired texts between English and Finnish.
This strategy relies on a cross-lingual signal to enhance the model’s understanding of the connections between languages. This approach has proven crucial in achieving superior performance for low-resource languages, without compromising the performance in English.
The Finnish company is releasing Poro as an open-source model that facilitates widespread access and collaborative improvement, particularly for underrepresented European languages.
Having said that here are the key features of Poro 34B:
- Poro Research Checkpoints: Checkpoints for the model are released throughout the training process, providing external researchers with unprecedented access to investigate the model training process.
- Model architecture: Poro 34B has 34.2 billion parameters and uses a BLOOM architecture with ALiBi embeddings to allow for context window extrapolation. While model architecture for the initial model has been kept simple, future models under progress will support additional capabilities, such as flash attention, rotary embeddings, and grouped query attention.
- Multilingual capabilities: Poro is designed to process English and Finnish, and has proficiency with a variety of programming languages. Additionally, it can perform basic translation between English and Finnish.
- Open source: Poro is freely available under the Apache 2.0 License, implying applicability for both commercial and research use.
- Dataset: The model is trained with a dataset of 1 trillion tokens, with English, Finnish, and a variety of programming languages represented.
- Training details: Poro is trained using 512 AMD MI250X GPUs on the LUMI supercomputer in Finland.
Silo AI: Largest private AI laboratories in Europe
Silo AI claims to be one of Europe’s largest private AI laboratories, a reliable AI partner that adds a competitive advantage to product R&D.
We build AI-driven solutions and products to enable smart devices, autonomous vehicles, industry 4.0, and smart cities,” says the company.
Silo AI offers access to AI expertise and the Silo OS infrastructure to speed up AI development and deployment.
Established in 2017, the company is on a mission to build a European flagship AI company, with offices currently in Finland, Sweden, Denmark, and Switzerland.
Read the orginal article: https://siliconcanals.com/news/startups/silo-ai-completes-poro-model-training/
