Devnagri Black Bird tops Voice of India ASR benchmark across 15 Indian languages

Devnagri AI says its new speech recognition model, Black Bird, posted the lowest open Indic word error rate on the independent Voice of India benchmark, outpacing models from Google and Sarvam. The result matters for banks, government services and digital commerce that rely on accurate voice automation in regional languages.

Why it matters: - Devnagri AI’s Black Bird model is now positioned as a stronger option for voice systems that must work in real Indian conditions, not just clean lab audio. - The result is especially relevant for BFSI, government services and digital commerce, where errors in names, amounts, dates or customer intent can disrupt workflows. - Lower transcription errors can reduce manual intervention, rework and record-keeping mistakes.

What happened: - Devnagri AI said its new automatic speech recognition model, Devnagri Black Bird, ranked first on the Voice of India benchmark. - The benchmark is an independent Indian ASR test set developed at IIT Madras with AI4Bharat and Josh Talks. - Devnagri reported an average open Indic word error rate of 8.7 across the systems evaluated. - The company said Black Bird outperformed Sarvam Saras v3, Google Gemini 3 Pro and Google Gemini 3 Flash.

The details: - Devnagri said Black Bird’s error rate was less than half of Google Gemini 3 Pro’s 21.1 score. - The model was 19% better than Sarvam Saras v3, which scored 10.7. - Black Bird was 59% lower than Gemini 3 Pro and 62% lower than Gemini 3 Flash, which scored 23.1. - Devnagri said Black Bird reduced errors by about 25% versus its earlier Shivani ASR model. - The benchmark was created on April 21, 2026 to measure how systems handle real-world Indian phone conversations. - The test set includes 536 hours of ordinary telephone calls in 15 Indian languages, 36,691 native speakers and 306,230 utterances. - The benchmark includes balanced representation across age groups and male and female speakers. - It is designed to reflect phone noise, regional accents and code-switching such as Hindi-English mixing. - Valid spelling and transliteration variants used in natural speech are not counted as errors. - Devnagri said Black Bird is built for that real-world environment. - The company said it evaluated all nine systems the same way. - Each production API was called with the same default settings. - Test data was kept confidential to avoid overfitting, and model versions and logs were retained for verification. - Results for other companies were based on independent measurements through publicly available production APIs, not on vendor-published scores. - Devnagri said Black Bird performed better across all 15 languages, including lower-resource languages such as Bhojpuri and Maithili.

Between the lines: - The benchmark appears designed to reward practical accuracy on messy, multilingual Indian calls rather than performance on polished speech clips. - The comparison also suggests a push by Devnagri to define a more demanding standard for Indian-language speech recognition. - In sectors like banking and government helplines, the difference between clean audio and real call conditions can determine whether automation works at scale.

What’s next: - Devnagri is likely to keep improving Black Bird as it competes in real-world speech environments. - The company’s stated focus is expanding voice AI that can handle customers in their own languages across regulated enterprise workflows. - Broader adoption will likely depend on how well the model performs in live deployments across regional languages and noisy networks.

The bottom line: - Devnagri is using the Voice of India benchmark to argue that Black Bird is not just accurate, but accurate under the conditions that matter most in India.