Meta AI researchers have introduced a new set of AI models called Seamless Communication, designed to make communication across different languages more natural and authentic. This development brings us closer to the idea of a Universal Speech Translator. The models, along with research papers and data, were released this week.
The main model, Seamless, combines features from three other models: SeamlessExpressive, SeamlessStreaming, and SeamlessM4T v2. According to the research, Seamless is the first publicly available system that allows expressive cross-lingual communication in real-time.
Seamless works as a universal real-time translator by integrating three advanced neural network models. It can translate over 100 spoken and written languages while maintaining the speaker’s vocal style, emotion, and prosody. SeamlessExpressive ensures that the vocal style and emotional nuances of the speaker’s voice are preserved during translation. Unlike existing translation tools that often sound monotone and robotic, SeamlessExpressive captures the nuances of human expression.
SeamlessStreaming offers near real-time translation with only about two seconds of delay. It is the first model of its kind to provide such fast translation speeds across nearly 100 languages. SeamlessM4T v2, the foundation for the other two models, is an improved version of the original SeamlessM4T model. It offers better consistency between text and speech output.
The researchers believe that Seamless could revolutionize global communication. It could enable new voice-based communication experiences, such as real-time multilingual conversations using smart glasses or automatically dubbed videos and podcasts. It could also help break down language barriers for immigrants and others who struggle with communication.
By releasing their work publicly, the researchers hope that others will build on their contributions to create technologies that bridge multilingual connections in our interconnected world. However, they also acknowledge the potential for misuse, such as voice phishing scams and deep fakes. To promote safety, they have implemented measures like audio watermarking and techniques to reduce harmful outputs.
The Seamless Communication models are available on Hugging Face and GitHub, in line with Meta’s commitment to open research and collaboration. This release includes the Seamless, SeamlessExpressive, SeamlessStreaming, and SeamlessM4T v2 models, along with metadata. By making these advanced natural language processing models freely available, Meta aims to help researchers and developers connect people across languages and cultures, reinforcing its leadership in open-source AI.