Multilingual AI Language Processing

Most voice AI systems ignore 90% of the world’s languages. Why? Because data is scarce. Meta’s new Omnilingual Speech Recognition suite breaks that cycle. Existing models are trained on internet-rich languages and that dominates the research loop. Omnilingual can transcribe speech in over 1,600 languages, including 500 that no speech AI has ever supported. This is a glimpse into the next wave of AI: models that don’t assume the internet is the world. Highlights: – Transcription accuracy under 10% error for 78% of supported languages – In-context learning: adapt to new languages with just a few audio clips – Fully open-source: models, data, and the 7B Omnilingual w2v 2.0 foundation This isn’t about just recognizing speech. It’s about who gets included. If we can build models that work across dialects, cultures, and scarce data, the future of voice AI in enterprise, customer service, and global markets changes fast. - Announcement blog: https://go.meta.me/ff13fa - Download Omnilingual ASR: https://lnkd.in/g3w4FqY3 - Try the Language Exploration Demo: https://lnkd.in/gVzrcdbd - Try the Transcription Tool: https://lnkd.in/gRdZuZqP - Read the Paper: https://lnkd.in/giKrvniC

Latest research from KAIST and Imperial College London introduces Zero-AVSR, an innovative framework that enables audio-visual speech recognition across languages without requiring training data in target languages. By learning language-agnostic speech representations through romanisation and leveraging LLMs, it can recognise speech even in languages never seen during training. What makes this approach interesting is the scale of language support. The team created MARC, a dataset spanning 2,916 hours of audio-visual speech across 82 languages—far beyond the 9 languages typical systems support. Their results show comparable performance to traditional multilingual systems while supporting this vastly larger language inventory. Zero-AVSR represents a significant advancement for speech tech in low-resource languages, potentially democratising access across thousands of languages without requiring extensive labelled datasets for each. The approach particularly excels when recognising languages from families similar to those in the training data, suggesting promising pathways for further expansion. Paper: https://lnkd.in/dnw_V7XK Authors: Jeong Hun Yeo, Minsu Kim, Chae Won Kim, Stavros Petridis, Yong Man Ro #SpeechRecognition #MultilingualAI #SpeechAI

🌟 Excited to share our latest research on enhancing multilingual capabilities in large language models! 🌟 Introducing SPHINX, a novel multilingual synthetic instruction tuning dataset created to address the performance gap in non-English languages. By translating instruction-response pairs from English into 50 languages, we achieved impressive results. In our study, fine-tuning models PHI-3-SMALL and MISTRAL-7B using SPHINX led to significant performance improvements, surpassing other multilingual datasets in benchmarks. Incorporating N-shot examples further boosted performance, showcasing the effectiveness and efficiency of SPHINX. This advancement marks a significant step forward in making large language models more inclusive and effective across diverse languages. Our research highlights the importance of sample efficiency and diversity while minimizing dataset creation costs. Excited for further discussions and collaborations in the realm of NLP, Multilingual AI, Machine Learning, and Artificial Intelligence! 🚀 Link to the paper : https://lnkd.in/g5CP9EZc Sanchit Ahuja Kumar Tanmay Hardik Chauhan Barun Patra Vishrav Chaudhary Monojit Choudhury Arindam Mitra Luciano Del Corro Tejas Indulal Dhamecha Ahmed Awadallah Sunayana Sitaram #NLP #MultilingualAI #MachineLearning #ArtificialIntelligence #Research #Innovation

Meta went bonkers with this new open-source ASR that works for 1,600+ languages! 🤯 Now, businesses can reach customers in their native tongue, even in low-resource regions, without building ASR from scratch. → Fully open-source, supporting 500+ languages never covered by any ASR before → Trained on 4.3M hours of multilingual speech (1,600+ languages) → Best part: Works zero-shot on languages never seen during training How? Two breakthroughs: Dual-decoder architecture:  • CTC decoder for low-latency, real-time use  • LLM-ASR decoder (Transformer-based) for high-accuracy, context-aware transcription In-context learning: Just 5–10 speech-text examples at inference time, let it transcribe any new language even if the model was never trained on it. Even more surprising: → On FLEURS-81, Omnilingual ASR beats Whisper on 65/81 languages—including 24 of the world’s top 34 most spoken languages → Robust to noise: CER stays <10 even in the noisiest 5% of field recordings → Scales from edge to cloud: 300M (mobile) → 7B (max accuracy) But the real shift isn’t scale, it’s agency. Communities can now extend ASR to their own language with minimal data, compute, or expertise. Check out the carousel to know how it works in simple terms and what the challenges are in detail. Question for you: When building voice tech for underserved languages, do you prioritise zero-shot generalisation or lightweight fine-tuning and why? Follow me, Bhavishya Pandit, for honest takes on AI tools that actually work 🔥 P.S. Model card, inference code, and datasets in the first comment.

Cartesia Sonic-3 is the first AI voice model I’ve seen that nails Hindi perfectly. For years, even the best text-to-speech (TTS) models struggled with Hindi. The rhythm, tonality, and emotional micro-expressions just didn’t sound human and the accent was inaccurate. This model doesn’t just translate Hindi. It is specially trained for it, with precise control over pacing, expressions and  tonality, all rendered in real time. Under the hood, Sonic-3 is engineered for low-latency voice generation optimized for conversational AI agents, clocking in 3–5x faster than OpenAI’s TTS while maintaining superior transcript fidelity. What makes it stand out technically: → 𝗚𝗿𝗮𝗻𝘂𝗹𝗮𝗿 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝘁𝗮𝗴𝘀 let developers dynamically modulate speed, volume, and emotion inside the transcript itself. ("Can you repeat that slower?" now works in production.) → 𝟰𝟮-𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗺𝗼𝗱𝗲𝗹 built on a single unified speaker embedding, so one voice can switch between languages like Hindi, Tamil, and English natively while maintaining accent continuity. → 𝟯-𝘀𝗲𝗰𝗼𝗻𝗱 𝘃𝗼𝗶𝗰𝗲 𝗰𝗹𝗼𝗻𝗶𝗻𝗴 powered by a low-sample adaptive cloning pipeline that enables instant personalization at scale. → 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘀𝘁𝗮𝗰𝗸 achieving sub-300 ms end-to-end latency at p90, tuned for live interactions like support agents, NPCs, and healthcare assistants. → 𝗙𝗶𝗻𝗲-𝗴𝗿𝗮𝗶𝗻𝗲𝗱 𝘁𝗿𝗮𝗻𝘀𝗰𝗿𝗶𝗽𝘁 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 that handles heteronyms, acronyms, and structured text (emails, IDs, phone numbers) which usually break realism in production systems. �� Here is example of me trying Sonic-3’s Hindi. You have to hear it to believe it. If you’re building voice agents, conversational AI, or multimodal assistants, keep an eye on Cartesia. They’ve raised $100M to build the most human-sounding voice models in the world, and Sonic-3 just set a new benchmark for multilingual voice AI. #CartesiaPartner

AI doesn’t speak just one language. It never should. It should speak to, and for, all of us! From the steppes of Mongolia to the villages of India and the ministries of Chile, local AI experts are proving that sovereign, locally useful AI models can flourish even with limited resources. These efforts show that the barriers to multilingual AI can be overcome with creativity, determination, and modest funding. The question now is: how can we support and scale these efforts globally? #Mongolia – Egune AI Very happy to see Bloomberg News highlight Egune AI today, a small startup that built the first Mongolian-language foundation model from scratch. This team made the country 1 of just 8 to develop its own national model. With only $3.5M in local seed funding, they now power over 70% of the nation’s AI market. Their work protects Mongolian language and culture through homegrown AI - a powerful example of what’s possible when communities build for themselves. #India – Bhashini India’s BHASHINI - (Digital India BHASHINI Division) is a government-backed, public–private mission to make AI inclusive for all Indian languages. Launched under the National Language Translation Mission, Bhashini supports over 35 languages through an open-source model which provides real-time translation tools in text -to-text, speech-to-text, and video translation services. Through the “Bhasha Daan” crowdsourcing initiative, thousands of people are contributing text, voice and video data and translations to help the AI learn. Bhashini bridges digital gaps across the country and creates datasets for underrepresented languages. It has  already hit 1 billion+ inferences.     #Chile (Latin America) – #LatamGPT Chile is leading a regional push for AI sovereignty through a Spanish-language foundation model called Latam GPT. Under the leadership of my dear friend Minister Aisen Etcheverry, the Ministry of Science, Technology, Knowledge and Innovation is building a model that reflects Latin America’s own histories, dialects, and values. With support from CENIA and a university-backed supercomputer, the project is advancing on just a few million dollars in funding. The model is designed to be open, adaptable, and shared across countries — “AI by Latin America, for Latin America.”    The call to action: Multilingual AI capacity is often described as a roadblock to universal access. But these efforts prove it doesn’t have to be. 🔹 How do we support and scale grassroots AI infrastructure? 🔹 Can we pool funding, talent, and knowledge to help more countries build their own models? 🔹 What does a global ecosystem look like when every language has a voice in shaping it? #AIforAll #LocalAI #MultilingualAI #Innovation #aipolicy Nick Martin Hugging Face Satwik Mishra Bloomberg News Nick Cain Mary Rodriguez, MBA Mathilde Barge Nagi Otgonshar  Ashwini Vaishnaw S Krishnan Abhishek Singh Tara Chklovski Room to Read Vivian Schiller Aspen Digital

❓ If we ask a multilingual language model a factual question written on different languages, do the answers always refer to the same entity? well..not quite. 🤔 I'm happy to report that our '24 Summer Research Intern Mahardika Krisna Ihsani from @MBZUAI collaboration came to fruition in joint work with Barid Xi Ai! We study crossling consistency across LLMs 🌎🌍🌏. See the ❇️EMNLP Findings🎇preprint https://t.co/zyo37zV9r6 & thread 🧵 for details! In our work, we did the evaluation on code-switched sentence and we expect that by this setting, the model aligns the knowledge in more language-agnostic fashion. We limited scope to only consider English as the pivot language and we examined the top-5 answers rather than top-1. We discovered that query whose language is distinct from the pivot language could elicit model to answer in different entity. This finding is substantially pronounced when the writing script is different than the pivot language. Additionally, we could see that larger model doesnt give substantial consistency improved and we explored why this happened. So we examined the cross-lingual consistency across layer and we discovered that there is no monotonic improvement and this could possibly explain why. Lastly, we also tried several methods to alleviate the inconsistency bottleneck. Among the other methods, we found that training objective that promotes cross-lingual alignment shows the best improvement and alleviates bottleneck as shown by the result of xlm-align and xlm-r-cs. If you're keen to know more about the details, please check out the preprint: https://lnkd.in/gv2gb6zh. Huge thanks to the co first authors Mahardika Krisna Ihsani and Barid Xi Ai.

India just got its own multilingual AI stack Not a demo. A real platform. Most AI still speaks English first. India does not. We keep talking about AI scale. But ignore language reality. Sarvam AI just shipped something important. An open-source foundational model suite built for 10 Indian languages and designed voice-first. That changes who AI is for. Here’s what stands out to me: India’s first open-source 2B Indic LLM trained on ~4 trillion tokens Voice agents deployable via phone WhatsApp and in-app workflows Speech → text → translation → synthesis in a single Indic stack Legal AI workbench for drafting redaction and regulatory Q&A Pricing that starts around ₹1 per minute for multilingual agents This is not chasing Silicon Valley scale. It’s solving Indian constraints. Smaller efficient models that run where India actually is Voice interfaces for users who skip keyboards Agentic workflows not just chat responses And the quiet but big idea: Sovereign AI infrastructure. Data stays local. Models align with Indian regulation. Control stays domestic. That matters for BFSI, legal, telecom and any sector touching sensitive data. The real unlock is inclusion. AI that works in Hindi, Tamil, Telugu Malayalam, Punjabi, Odia Gujarati, Marathi, Kannada, Bengali AI that listens before it types We keep saying India will be an AI market. This is India building AI rails. Open-source, voice-first, enterprise-ready That combination is rare. If this ecosystem compounds India does not just consume AI It exports it. Watching this space closely. Local language AI is the next growth curve. What sectors do you think adopt first?

Everyone says AI is multilingual. But how well does it really work in practice, especially in your business; context?? Here’s what happened: A Dutch user interacted with my chatbot. Not only did the AI understand the question perfectly, but it responded in fluent Dutch, providing detailed steps on how to build a support chatbot with a custom knowledge base. This wasn’t just a direct translation. It was: ✅ Context-aware ✅ Technically accurate—It ✅ Natural Why does this matter? It’s redefining global business communication. Whether your customers are in Amsterdam, Tokyo, or São Paulo, AI can now provide localized, intelligent responses that feel seamless. If you’re still thinking AI is only useful for English-speaking markets, it’s time to rethink your strategy. The future of business is borderless. How do you see AI impacting multilingual communication in your industry?

Conversational #AI just hit a triple milestone 1️⃣ #RAG (Retrieval-Augmented Generation) • Grounds every answer in live, verifiable documents, cutting hallucinations and letting teams update knowledge in minutes, not months. 2️⃣ True text-and-voice #multimodality (#ElevenLabs Conversational AI 2.0) • One agent, any channel. Talk on the phone, type in chat, swap mid-conversation, and it never loses context. 3️⃣ Next-gen turn-taking models (#TurnGPT, VAP) • Predict millisecond hand-offs, so bots stop talking over you and feel as smooth as a real colleague. Why this is a very big deal • Trust climbs, risk falls. Regulated fields like healthcare, finance, and aviation can now adopt AI assistants that cite their sources and understand when to stay quiet. • Single build, global reach. Define a bot once and deploy it across web, mobile, telephony, and smart devices without separate codebases. • Always on, always current. Drop fresh PDFs, policies, or product docs into a vector store and your agent “knows” them instantly. • Human-grade flow. Micro-pause prediction means no awkward gaps, no interruptions, and real empathy cues such as quick back-channels (“mm-hmm… go on”). • Multilingual by default. Automatic language detection flips from English to Spanish (or 29+ other languages) inside the same call, opening whole new markets overnight. • Precision where it matters. Users can speak naturally, then type exact account numbers or medication names without starting over. • Cost and speed gains. Shorter call times, higher self-service rates, and fewer agent hand-offs translate into real bottom-line impact. What tomorrow looks like 🔹 Voice-first knowledge bases that quote chapter-and-verse references while you drive. 🔹 On-the-fly compliance coaches that listen to sales calls and whisper policy reminders before a rep misspeaks. 🔹 Hospital kiosks that greet patients in their native language, switch to text when the lobby is noisy, and sync notes straight into the EHR with full citations. 🔹 Zero-latency product experts embedded in every device, from wearables to smart tractors, updating themselves whenever the manual changes. The line between “chatbot” and “colleague” is getting thinner by the week. This trio of breakthroughs makes conversational AI more reliable, versatile, and human than ever. 💡 Question for you: Which industry will leapfrog first now that bots can know, listen, and speak like this? Drop your thoughts below. Harvey Castro MD #DrGPT #ConversationalAI #RAG #VoiceTech #AIInnovation #FutureOfWork