Loading…Loading…

Measured accuracy

How accurate is our live translation? We measured it.

Public test recordings, run through the moschee.io live translation and scored by the rules Baian describes for its own figures. Our figure is ahead of Baian's on all 6 language pairs we could compare, and the finished line appears sooner. Each lead is within the margin of error, and our weak spots are on this page too.

The three key figures

  • 6 of 6

    language pairs ahead of the figure Baian publishes

    by 4.9 points on average, each lead within the margin of error

  • 1.8 s

    from the last word to the finished translated line (median)

    Baian publishes 2.1 s

  • 98%

    of the Quran verses shown during the khutbah were the right verse

    233 of 237, first measurement on 17 live sermons

Translation quality

Ahead on all 6 language pairs we compared

The test works like this: a speaker reads a sentence aloud, the live translation hears it and translates it. An automatic judge, MetricX-24, then rates our translation of the speech and a professional translation of the written sentence.

The figure is the share of sentences where the judge rates ours higher; a tie counts half. 50% means level with the professional translation. Baian publishes the same figure for its own product, which is what makes a comparison possible.

Share of sentences rated above the professional translation

  • moschee.io, measured
  • Baian, published
  • 95% range
  • 50%: level with the professional translation

From English speech

The language pairs Baian publishes

  • English → Arabic

    moschee.io, measured:63.7%95% range 58.6–⁠69.0%

    Baian, published:58.0%

  • English → Hindi

    moschee.io, measured:70.3%95% range 65.6–⁠75.0%

    Baian, published:65.5%

  • English → Korean

    moschee.io, measured:68.3%95% range 63.3–⁠73.0%

    Baian, published:61.5%

  • English → Hausa

    moschee.io, measured:76.9%95% range 72.4–⁠81.6%

    Baian, published:75.0%

  • English → Swahili

    moschee.io, measured:71.1%95% range 66.6–⁠75.7%

    Baian, published:68.0%

  • English → Urdu

    moschee.io, measured:74.1%95% range 69.7–⁠79.1%

    Baian, published:67.0%

From Arabic speech

This is how an Arabic khutbah is translated. Baian publishes no translation figures for this direction.

  • Arabic → German

    moschee.io, measured:53.0%95% range 47.2–⁠58.8%

  • Arabic → English

    moschee.io, measured:62.4%95% range 56.4–⁠67.7%

  • Arabic → Turkish

    moschee.io, measured:62.2%95% range 56.5–⁠67.7%

  • Arabic → French

    moschee.io, measured:61.5%95% range 55.8–⁠67.0%

  • Arabic → Bosnian

    moschee.io, measured:57.4%95% range 51.9–⁠62.7%

  • Arabic → Albanian

    moschee.io, measured:59.7%95% range 54.1–⁠65.7%

Baian publishes no ranges. The ones shown for Baian are our estimate from its 148 sentences.

Translation quality per language pair: moschee.io measured, Baian published
English → ArabicSentences350moschee.io63.7%95% range 58.6–⁠69.0%Against the professional translationRated higherBaian (published)58.0%Live productLead in points+5.7
English → HindiSentences350moschee.io70.3%95% range 65.6–⁠75.0%Against the professional translationRated higherBaian (published)65.5%Live productLead in points+4.8
English → KoreanSentences350moschee.io68.3%95% range 63.3–⁠73.0%Against the professional translationRated higherBaian (published)61.5%Live productLead in points+6.8
English → HausaSentences350moschee.io76.9%95% range 72.4–⁠81.6%Against the professional translationRated higherBaian (published)75.0%Caption tool for recordingsLead in points+1.9
English → SwahiliSentences350moschee.io71.1%95% range 66.6–⁠75.7%Against the professional translationRated higherBaian (published)68.0%Caption tool for recordingsLead in points+3.1
English → UrduSentences350moschee.io74.1%95% range 69.7–⁠79.1%Against the professional translationRated higherBaian (published)67.0%Caption tool for recordingsLead in points+7.1
Arabic → GermanSentences283moschee.io53.0%95% range 47.2–⁠58.8%Against the professional translationLevelBaian (published)Not publishedLead in pointsNot published
Arabic → EnglishSentences283moschee.io62.4%95% range 56.4–⁠67.7%Against the professional translationLevelBaian (published)Not publishedLead in pointsNot published
Arabic → TurkishSentences283moschee.io62.2%95% range 56.5–⁠67.7%Against the professional translationLevelBaian (published)Not publishedLead in pointsNot published
Arabic → FrenchSentences283moschee.io61.5%95% range 55.8–⁠67.0%Against the professional translationLevelBaian (published)Not publishedLead in pointsNot published
Arabic → BosnianSentences283moschee.io57.4%95% range 51.9–⁠62.7%Against the professional translationLevelBaian (published)Not publishedLead in pointsNot published
Arabic → AlbanianSentences283moschee.io59.7%95% range 54.1–⁠65.7%Against the professional translationLevelBaian (published)Not publishedLead in pointsNot published
  • Baian's figures for Arabic, Hindi, and Korean come from its live product. Those for Hausa, Swahili, and Urdu come from its caption tool for recorded files.
  • “Against the professional translation” compares the judge's average scores: “rated higher” means the 95% range of the difference lies above zero, “level” means it includes zero.
  • Baian also publishes a figure for Tamil (69.0%, caption tool for recordings). We have not measured that pair yet, so it is not part of this comparison.

Delay

The finished line 1.8 s after the last word

We measure from the last voiced moment of a sentence to the moment the finished translated line is on screen. The median is the value half of the sentences stay under; nine in ten sentences stay under the second value.

  • Arabic speech

    402 recordings

    Median 1.80 s

    Nine in ten sentences within 2.92 s

  • English speech

    601 recordings

    Median 1.83 s

    Nine in ten sentences within 3.11 s

  • Baian (published)

    Median 2.1 s

  • Baian states a median of 2.1 s from the end of the sentence to the finished line. Baian streamed 15-minute recordings; ours were single sentences followed by silence. In a running sermon the next sentence follows at once, so the delay there can differ.
  • Timed are the recordings in which the recognizer closed the sentence on its own: 402 of 428 Arabic and 601 of 647 English ones.

Speech recognition

Level with Baian in 4 languages, behind in 2

Before anything can be translated, the speech has to be written down correctly. The word error rate (WER) is the share of words that are wrong, missing or added; for Korean, characters are counted instead (CER). Lower is better.

Our figures use the text cleaning Baian uses for its own: punctuation, capitalization, Arabic vowel marks and the way numbers are written are ignored.

Speech recognition error rates per language: moschee.io measured, Baian published
ArabicRecordings428MeasureWERmoschee.io5.76%95% range 4.71–⁠6.95%Baian (published)5.1%Live productLead in points−0.7StandingLevel
EnglishRecordings647MeasureWERmoschee.io5.06%95% range 4.55–⁠5.61%Baian (published)Not publishedLead in pointsNot publishedStandingNo Baian figure
GermanRecordings200MeasureWERmoschee.io5.98%95% range 4.75–⁠7.25%Baian (published)Not publishedLead in pointsNot publishedStandingNo Baian figure
TurkishRecordings200MeasureWERmoschee.io5.99%95% range 4.79–⁠7.14%Baian (published)Not publishedLead in pointsNot publishedStandingNo Baian figure
FrenchRecordings200MeasureWERmoschee.io5.14%95% range 4.16–⁠6.15%Baian (published)4.3%Caption tool for recordingsLead in points−0.8StandingLevel
RussianRecordings200MeasureWERmoschee.io5.61%95% range 4.59–⁠6.71%Baian (published)6.4%Live product4.7%Caption tool for recordingsLead in points+0.8Live product−0.9Caption tool for recordingsStandingLevel
DanishRecordings200MeasureWERmoschee.io9.14%95% range 7.65–⁠10.68%Baian (published)9.9%Live productLead in points+0.8StandingLevel
KoreanRecordings200MeasureCERmoschee.io5.27%95% range 3.95–⁠6.82%Baian (published)2.9%Live productLead in points−2.4StandingBehind
UrduRecordings200MeasureWERmoschee.io15.86%Stricter cleaning · 95% range 14.15–⁠17.51%13.61%Baian's cleaning · 95% range 12.30–⁠14.81%Baian (published)13.2%Caption tool for recordings · Stricter cleaningLead in points−2.7Stricter cleaningStandingBehind
  • “Lead” is Baian's error rate minus ours, in points: a plus sign means ours is the lower one, a minus sign that it is the higher one. By these plain numbers our error rate is lower than Baian's for Russian and Danish and higher for Arabic, French, Korean, and Urdu. “Level” means that Baian's figure lies inside our 95% range, “behind” that it lies below it.
  • Arabic and English come from the headline run. The other languages come from an earlier run on the same day with the same speech recognition, 200 recordings each.
  • Urdu: Baian's figure comes from its caption tool and is scored with a stricter text cleaning than its other figures. Scored the same way, ours is 15.86%, behind Baian's 13.2%. Under Baian's usual cleaning ours is 13.61%.
  • Korean: 5.27% against Baian's 2.9%, both counted without spaces. This is the clearest gap on this page.
  • The French figure and the second Russian figure come from Baian's caption tool for recorded files, not from its live product. For Russian the standing goes by the live product's figure; against the caption tool's figure our error rate is the higher one, and that figure lies inside our 95% range too.

Quran and hadith

Quran verses and hadith, recognized during the live khutbah

First measurement, November 2026

When the imam quotes a Quran verse or a hadith, moschee.io shows listeners its source. For a verse, a published translation appears instead of a machine translation, where one exists for the language.

We went through 17 live Friday sermons held in the 90 days before the measurement and checked every source that listeners were shown against what was said. Baian publishes no figure for this, so there is nothing to compare with here.

Right when shown
Of all the sources listeners were shown: the share that pointed to the right verse or hadith. This figure protects listeners from a wrong source.
Found
Of all the verses or hadith that were quoted: the share for which a source was shown at all. When a passage is not found, no source appears for it.

Quran verses in the khutbah

Right when shown

98%

233 of 237 verses shown were the right verse

Found

about 94% of the verses quoted

Quran recited in prayer

Right when shown

97%

374 of 387 verses shown were the right verse

Found

Not measured yet

Hadith

Right when shown

85%

98 of 115 hadith shown were the right hadith

Found

about 68% of the hadith quoted

  • Hadith are the harder case: narrations differ in wording, and speakers often give them in their own words. 17 of the 115 hadith sources shown were not the right hadith, and about 32% of the hadith quoted were not found.
  • These are first figures and they are preliminary: a second check of a sample of 30 entries is still pending. Only totals are published: no mosque, no speaker and no sermon text.

Method and limits

How we measured, and what the figures do not show

Comparing two products is only fair if the reader can check it. This is everything that went into the figures above.

Test data

Speech: the test split of FLEURS, a public collection of read sentences, with 428 recordings in Egyptian Arabic and 647 in US English; for the other speech recognition languages, 200 recordings each. For translation, one recording per sentence: 350 English and 283 Arabic sentences. Translations to compare with: the professional translations of the same sentences in FLoRes-200.

How the speech was streamed

Each recording went through the moschee.io live translation pipeline at real speed, one sentence per stream, with the spoken language set in advance. This is how AssemblyAI ran the engines Baian compares itself with. Baian streamed 15-minute recordings with automatic language detection, which is the more demanding setup.

Translation score

MetricX-24 in its reference-free mode rates each translation on its own. We count how often our translation of the speech scores better than the professional translation of the text; a tie counts half. An automatic judge is not a human reader, and such judges can favor machine translation. A share above 50% therefore does not prove that a translation is better than a professional's. The figure is suited to comparing two systems that are scored the same way.

Speech recognition score

Word error rate under Baian's text cleaning, so that the figures can be compared. Internally we also score with stricter cleanings, which give higher word error rates.

Margins of error

Our 95% ranges come from bootstrap resampling of the sentences. Baian publishes no ranges. With 148 sentences, each of its figures carries about ±8 points; ours carry about ±5. For the difference between the two that gives about ±9 points. Each lead lies inside that margin. On average we are 4.9 points ahead.

Delay

From the last voiced audio frame of a sentence to the finished translated line on screen, on the recordings in which the recognizer closed the sentence on its own.

Baian's figures

Copied from baian.ai/accuracy as we read it on 8 November 2026. We did not test Baian's product ourselves, and the sentences are not identical: Baian used 148 English sentences, we used 350. If Baian has updated its page since, the figures there apply.

What the figures do not show

FLEURS is read speech recorded in quiet surroundings. A Friday sermon has reverberation, emotion, dialect and Quran recitation, so accuracy in a real sermon can be lower for every system. It is also a single run: a live translation does not word a sentence the same way twice, and between two runs on the day of the measurement a pair moved by about 2 points.

Questions about the measurement

What does the percentage in the translation table mean?

It is the share of test sentences for which the automatic judge MetricX-24 rated our live translation of the spoken sentence above a professional translation of the written sentence. A tie counts half. 50% means level with the professional translation.

Is moschee.io more accurate than Baian?

On the 6 language pairs we could compare, our measured figure is higher than the one Baian publishes, by 4.9 points on average, and the finished line appears after 1.8 s against 2.1 s. Each lead is within the margin of error, and the two measurements did not use identical sentences. So we say “ahead on every pair we compared”, not “proven better in every language”. In speech recognition we are level with Baian in 4 languages (Arabic, French, Russian, and Danish) and behind in 2 (Korean and Urdu).

Where is moschee.io behind?

In speech recognition, in 2 languages (Korean and Urdu): for Korean our character error rate is 5.27% against Baian's 2.9%; for Urdu our word error rate is 15.86% against 13.2% under the same rule. For Arabic and French our error rate is higher than Baian's too, but Baian's figure lies inside our 95% range, so we count them as level. Hadith are also recognized less reliably than Quran verses: 85% of the hadith shown were the right hadith, and about 68% of the hadith quoted were found.

Why read sentences and not real sermons?

Because public test sets are the only way to compare with figures another provider publishes: anyone can obtain the same recordings and check the result. Real sermons are harder for every system. That is why the Quran and hadith figures on this page come from real live sermons.

How reliable is the Quran and hadith detection?

In 17 live Friday sermons, 98% of the Quran verses shown during the khutbah were the right verse, and about 94% of the verses quoted were found. For hadith the figures are 85% and about 68%. These are first figures; a second check of a sample is still pending.

How current are these figures?

They were measured on 10 November 2026. We repeat the measurement when the live translation changes, and the date is shown at the top of the page. Baian's figures are quoted as we read them on 8 November 2026.

Read on

See it for yourself — free

Start free with prayer times and the website widget. The website, app content, and AI translation come with the Basis plan, which you can try free for 14 days — no contract, GDPR-compliant, hosted in Europe.