Short answer. Pipelines that reach 97% accuracy on Latin documents commonly drop to 85–90% on Arabic or Hebrew. Right-to-left typography is described by Arabic NLP researchers as effectively solved, though not universally implemented — the failures are in the pipeline, not the theory. Three script families drive most of the difficulty: right-to-left with contextual shaping, logographic systems without whitespace, and Indic scripts with complex conjuncts. Arabic characters take isolated, initial, medial and final forms depending on position, multiplying the visual classes a recogniser has to separate.
How Do You Collect Text Data for Right-toLeft and Complex Scripts?
Start with a correction, because most writing on this subject gets the emphasis wrong.
A panoramic survey of Arabic NLP addresses right-to-left typography in a single sentence and then sets it aside: the authors do not include issues of right-to-left Arabic typography, which they describe as an effectively solved problem, although not universally implemented.
That parenthetical matters, and so does the main clause. Rendering right-to-left text is solved at the standards level. Your pipeline may still break on it, because implementation is uneven, but that is an engineering defect rather than a research problem.
The difficulties that actually determine whether a text collection project in Arabic, Hebrew, Urdu, Devanagari or Amazigh succeeds are elsewhere: orthographic variation, diacritics, morphological richness and script heterogeneity.
Those are the subject of this piece.
The three script families and what each breaks
Document processing literature identifies three families that drive most architectural complexity, and the failure mode of each is distinct.
Right-to-left with contextual shaping. Arabic and Hebrew. A pipeline achieving 97% accuracy on Latin-script documents frequently drops to 85 to 90% on Arabic or Hebrew, because right-to-left flow and contextual letter shaping require fundamentally different segmentation logic.
The contextual shaping point is the one that is easy to underestimate. Arabic characters take different shapes depending on position within a word: isolated, initial, medial and final. That multiplies the number of visual classes a recognition system must distinguish. Latin has uppercase and lowercase, but those variations are more limited and do not wholly depend on character position. Add cursive connectivity and ligatures and segmentation becomes genuinely hard.
Logographic systems. Chinese, Japanese, Korean. Glyph vocabularies in the tens of thousands, with more than 20,000 standard CJK Unified Ideographs and over 50,000 in traditional Chinese, and critically, no whitespace separating tokens. The consequence is compounding: word segmentation errors misalign every field that follows a single error.
Indic scripts. Devanagari, Tamil and relatives. Character stacking and vowel diacritics that sit above, below and beside base characters simultaneously, which causes bounding-box-based extractors to misread or skip entire syllables.
One practical recommendation from the same source is worth adopting directly: use per-region script detection rather than document-level language flags, which prevents field boundary bleed in mixed-script documents. A document is not in a language. Regions of it are.
The real problem: there are no spelling rules
This is the finding that should shape collection specifications, and it comes from the Arabic script NLP community in unusually direct language.
Describing what happens when speakers begin writing a dialect, whether for social media, for advertising to low-literacy populations, or for building computational resources:
"They don't use rules for writing the oral message because there are none. Conventions develop but are also easily ignored since, the intent being to communicate, as long as the message is understandable the receiver can be flexible."
That is the situation for a large share of the world's text collection targets. Moroccan Arabic, Darija, is described as still showing substantial variation despite efforts at systematising its writing over time.
The consequences are structural. Text normalisation, defined as mapping nonstandard spellings and orthographic variants to a consistent form, becomes a required pipeline stage rather than a refinement, and the literature reports that explicit normalisation significantly improves downstream performance in machine translation, ASR postprocessing and morphological tagging.
For a collection project this produces a decision that has to be made before any text is gathered: do you collect as written, or collect and normalise? Both are defensible. Collecting as written preserves how people actually write, which is what you need if the model will read user-generated text. Normalising produces a consistent corpus, which is what you need for most training uses. Collecting as written and recording a normalised parallel form is the expensive option that serves both, and it is what the DATASHI parallel corpus was built to enable.
What is not defensible is leaving it to individual annotators, because you will get both and no way to tell which is which.
Diacritics, the specific unresolved case
Arabic diacritics deserve their own treatment because the problem is precisely characterised and still open.
Diacritic restoration is described as a persistent and unresolved problem in Arabic NLP, arising from lexical ambiguity, syntactic variation and the absence of diacritics in most written texts.
That last clause is the crux. Arabic is normally written without the vowel marks that disambiguate it, so readers infer them from context. A collection project gathering naturally occurring Arabic text is gathering undiacritised text, and a model trained on it inherits the ambiguity.
The 2026 KSAA shared task frames the current frontier: automatic diacritisation of speech dictation remains challenging because of the mismatch between speech-based transcriptions and traditional text-only diacritisation approaches. ASR systems produce undiacritised or partially normalised output, while text-based diacritisation models cannot use the acoustic information that would resolve the ambiguity.
Arabic script also carries two distinct diacritic systems worth naming in any annotation guideline: i'jām, the dots that distinguish otherwise identical letter forms, and tashkīl, the vowel marks. They are different problems and a specification that says "handle diacritics" resolves neither.
The comparable case in other scripts is the one I raised in an earlier article: Fongbe tonal diacritics preserved in some corpora and stripped in others, creating incompatibility when datasets are combined. The pattern is general.
Morphology, and why vocabulary explodes
One more Arabic-specific factor with direct data implications.
Arabic words have numerous forms from a rich inflectional system covering gender, number, person, aspect, mood, case and a number of attachable clitics. The survey gives an example worth quoting: wa+sa+ya-drus-uuna+ha, 'and they will study it', a single Arabic word rendering a five-word English sentence.
The consequence for data: a much higher number of unique vocabulary types compared with English, which is challenging for machine learning models and means that a corpus of a given token count contains proportionally fewer examples per type than an equivalent English corpus.
The practical implication is that token-count parity across languages is not coverage parity. A hundred million tokens of Arabic does not give a model the same exposure per word form as a hundred million tokens of English, and a scoping document that specifies volume in tokens without accounting for morphological richness is under-specifying for morphologically rich languages.
Shared script does not mean shared language
A frequently missed distinction with real consequences for language identification and corpus assembly.
Over 1.5 billion people speak languages that share the same script. And when languages share a script, they may use the same characters to represent different words, or different characters to represent the same word.
The Arabic script alone covers Perso-Arabic languages including Persian, Urdu, Pashto, Sorani Kurdish, Azeri, Ottoman Turkish, Sindhi and Uyghur, plus Ajami traditions across Africa including Hausa, Fula, Wolofal, Swahili, Kanuri, Mandingo and Tamazight. Together these communities represent almost one billion speakers, many of them under-resourced in NLP.
Two operational consequences. Script detection is not language identification, so a pipeline routing on script will merge Urdu and Persian. And a corpus assembled by script filter will be multilingual whether or not it was meant to be.
The reverse case is equally real: a single language written in multiple scripts. The Tashlhiyt work describes a hybrid digital orthography coexisting with two scripts plus Tifinagh, amplifying inconsistency and posing structural challenges for text normalisation, tokenisation and corpus alignment.
Where our own work fits
Declaring the interest: Lifewood collects text data across 50-plus languages and dialects, including Arabic-script and Indicscript languages, through delivery centres in more than 30 countries.
Two observations.
The first is that the orthographic decision must be made before collection and it requires native judgement.
Deciding whether to normalise Darija spelling, and if so to what standard, is not a decision a project manager can make from a specification document. It requires people who write the language deciding what counts as the same word, and it needs recording as a documented convention with worked examples rather than left as a shared understanding. I have made this point about Sylheti elsewhere in this series and it generalises: for languages without a settled orthography, the convention is part of the deliverable.
The second is that script complexity changes the annotator requirement, not just the tooling requirement.
Verifying Devanagari transcription where diacritics attach above, below and beside a base character requires someone who reads Devanagari fluently, not someone who can compare two strings. The same holds for Arabic contextual forms, where a visually plausible wrong form is invisible to a non-reader. Projects that budget script-complex languages at Latin-script review rates discover this at the QA stage.
A specification checklist
Separate rendering from linguistics. Right-to-left display is an implementation problem; fix it in engineering and do not confuse it with the data questions.
Decide the orthographic convention before collection, with native-speaker input and worked examples, and state whether you are collecting as-written, normalised, or both in parallel.
Specify diacritic handling explicitly, distinguishing letter-distinguishing marks from vowel marks where the script has both.
Use per-region script detection, not document-level language flags, for mixed-script material.
Do not equate script with language. Add a language identification stage after script detection, and expect it to perform worse on closely related languages sharing a script.
Account for morphology in volume targets. Token parity is not coverage parity for morphologically rich languages.
Budget script-literate reviewers, not string comparators, and price accordingly.
Check what happens to your corpus when it is combined with an existing one. Diacritic stripping, normalisation convention and encoding form are where merges silently corrupt data.
Key takeaways
- Right-to-left typography is described by Arabic NLP researchers as an effectively solved problem, though not universally implemented. Pipeline breakage on it is an engineering defect, not a research problem.
- Three script families drive most complexity: right-to-left with contextual shaping, logographic systems without whitespace, and Indic scripts with stacked diacritics.
- Pipelines achieving 97% accuracy on Latin documents frequently drop to 85 to 90% on Arabic or Hebrew.
- Arabic characters take isolated, initial, medial and final forms depending on position, multiplying the visual classes a recogniser must distinguish, unlike Latin case variation which does not depend wholly on position.
- CJK has over 20,000 standard Unified Ideographs and more than 50,000 in traditional Chinese, with no whitespace, so a single segmentation error misaligns every subsequent field.
- Devanagari and Tamil stack characters and place vowel diacritics above, below and beside base characters simultaneously, causing bounding-box extractors to skip entire syllables.
- Use per-region script detection rather than document-level language flags to prevent field boundary bleed.
- For many written dialects there are no spelling rules, conventions develop and are easily ignored, and Moroccan Darija still shows substantial variation despite systematisation efforts.
- Text normalisation, mapping variant spellings to a consistent form, significantly improves downstream machine translation, ASR post-processing and morphological tagging.
- Decide before collection whether to gather as-written, normalised, or both in parallel. Leaving it to annotators produces both with no way to distinguish them.
- Arabic diacritic restoration remains a persistent unresolved problem, arising from lexical ambiguity, syntactic variation and the absence of diacritics in most written text.
- Arabic script carries two distinct diacritic systems, i'jām distinguishing letters and tashkīl marking vowels, which need separate treatment in a specification.
- Arabic morphology produces single words equivalent to five-word English sentences, giving far more unique vocabulary types, so token-count parity across languages is not coverage parity.
- Over 1.5 billion people speak languages sharing a script, and shared scripts use the same characters for different words and different characters for the same word.
- Arabic script alone covers Perso-Arabic languages and African Ajami traditions representing almost one billion speakers, so script detection is not language identification.
- Some languages are written in several scripts at once, as with Tashlhiyt across a hybrid digital orthography and Tifinagh, posing structural problems for normalisation, tokenisation and alignment.
Sources and further reading
- "A Panoramic Survey of Natural Language Processing in the Arab World", arXiv, on right-to-left typography as effectively solved, and on morphological richness, orthographic ambiguity, dialectal variation, orthographic noise and resource poverty
- Extend, "Multilingual OCR: 100+ Languages", on the 97% Latin to 85-90% Arabic and Hebrew accuracy drop, the three script families, CJK glyph counts and segmentation compounding, Indic character stacking, and per-region script detection
- "Performance Gap Analysis between Latin and Arabic Scripts HTR", arXiv, on contextual character shapes across positional forms, cursive connectivity, ligatures, and the i'jām and tashkīl diacritic systems
- AbjadNLP 2026 workshop, on the absence of writing rules for dialects, Darija orthographic variation, and the scope of Arabic-derived scripts across Perso-Arabic and African Ajami traditions representing almost one billion speakers
- KSAA-2026 Shared Task, on diacritic restoration as a persistent unresolved problem and the mismatch between speech-based transcription and text-only diacritisation
- "DATASHI: A Parallel English-Tashlhiyt Corpus for Orthography Normalization and Low-Resource Language Processing", arXiv, on hybrid digital orthography, script heterogeneity and the downstream benefits of explicit normalisation
- Rustagi, "Multilingual NLP: Why Working with Languages with Complex Scripts is Challenging", on shared scripts across 1.5 billion speakers and character-to-word mapping differences
- Lifewood, multilingual text and speech data collection