Collecting Text Data for Right-to-Left and Complex Scripts

A panoramic survey of Arabic NLP addresses right-to-left typography in a single sentence and then sets it aside, describing it as an effectively solved problem, although not universally implemented. That parenthetical matters: rendering right-to-left text is solved at the standards level, so a pipeline that breaks on it has an implementation defect, not a research gap. The difficulties that actually determine whether a text collection project in Arabic, Hebrew, Urdu, Devanagari or Amazigh succeeds sit elsewhere — in how the language is written, not in which direction it flows.

Contextual shaping is when a single letter changes its printed form depending on its position in a word. Arabic and Hebrew are right-to-left scripts with contextual shaping: a pipeline achieving 97% accuracy on Latin-script documents frequently drops to 85–90% on Arabic or Hebrew because right-to-left flow and contextual shaping require different segmentation logic. Arabic characters take isolated, initial, medial and final forms depending on position, which multiplies the number of visual classes a recognition system must distinguish — Latin case variation is far more limited and does not depend on position at all. Add cursive connectivity and ligatures and segmentation becomes genuinely hard.

Chinese, Japanese and Korean are logographic scripts — writing systems built from word- or morpheme-level glyphs rather than an alphabet — with glyph vocabularies in the tens of thousands (more than 20,000 standard CJK Unified Ideographs, and over 50,000 in traditional Chinese) and no whitespace separating tokens. The consequence compounds: a single word-segmentation error misaligns every field that follows it.

Devanagari, Tamil and related Indic scripts stack characters and place vowel diacritics above, below and beside base characters simultaneously, which causes bounding-box-based extractors to misread or skip entire syllables.

One practical recommendation from the same literature is worth adopting directly: use per-region script detection rather than document-level language flags, which prevents field-boundary bleed in mixed-script documents. A document is not in one language; regions of it are.

The Arabic-script NLP community makes this point in unusually direct language: when speakers write a dialect, for social media, for advertising to low-literacy populations, or for building computational resources, they don't use rules for writing the oral message because there are none — conventions develop but are also easily ignored, since, the intent being to communicate, the receiver can be flexible as long as the message is understandable. Moroccan Arabic, Darija, still shows substantial variation despite efforts to systematise its writing over time.

Text normalisation — mapping nonstandard spellings and orthographic variants to a consistent form — becomes a required pipeline stage rather than a refinement; the literature reports it significantly improves downstream performance in machine translation, ASR postprocessing and morphological tagging. A collection project has to decide, before any text is gathered, whether to collect as written, or collect and normalise. Both are defensible: collecting as written preserves how people actually write, which matters for user-generated text; normalising produces the consistency most training uses need. Collecting as written and recording a normalised parallel form is the expensive option that serves both. What is not defensible is leaving the decision to individual annotators, because the result will mix both with no way to tell which is which.

Diacritic restoration — the task of reconstructing the vowel and disambiguation marks omitted from written text — is described as a persistent and unresolved problem in Arabic NLP, arising from lexical ambiguity, syntactic variation and the absence of diacritics in most written texts. A 2026 shared task frames the current frontier: automatic diacritisation of speech dictation remains challenging because of the mismatch between speech-based transcriptions and traditional text-only diacritisation approaches — ASR systems produce undiacritised or partially normalised output, while text-based diacritisation models cannot use the acoustic information that would resolve the ambiguity.

Arabic script also carries two distinct diacritic systems worth naming in any annotation guideline: i'jām, the dots that distinguish otherwise identical letter forms, and tashkīl, the vowel marks. They are different problems, and a specification that says only "handle diacritics" resolves neither. The comparable case shows up in other scripts too — Fongbe tonal diacritics preserved in some corpora and stripped in others create incompatibility when datasets are combined. The pattern generalises well beyond Arabic.

The survey gives an example worth quoting: wa+sa+ya-drus-uuna+ha, "and they will study it," a single Arabic word rendering a five-word English sentence. The practical implication is that token-count parity across languages is not coverage parity — a hundred million tokens of Arabic does not give a model the same exposure per word form as a hundred million tokens of English. A scoping document that specifies collection volume in tokens without accounting for morphological richness is under-specifying for morphologically rich languages.

The Arabic script alone covers Perso-Arabic languages including Persian, Urdu, Pashto, Sorani Kurdish, Azeri, Ottoman Turkish, Sindhi and Uyghur, plus Ajami traditions across Africa including Hausa, Fula, Wolofal, Swahili, Kanuri, Mandingo and Tamazight — together representing almost one billion speakers, many of them under-resourced in NLP. This has two operational consequences: script detection is not language identification, so a pipeline that routes purely on script will merge Urdu and Persian, and a corpus assembled by script filter will be multilingual whether or not it was meant to be. The reverse case is equally real — a single language written in multiple scripts, such as Tashlhiyt, which coexists as a hybrid digital orthography alongside Tifinagh, amplifying inconsistency and posing structural challenges for normalisation, tokenisation and corpus alignment.

Lifewood collects text data across 50+ languages and dialects, including Arabic-script and Indic-script languages, through 40+ delivery centres across 30+ countries. Two things follow from the analysis above. First, the orthographic decision must be made before collection, and it requires native judgement: deciding whether to normalise Darija spelling, and to what standard, is not a decision a specification document can make on its own — it needs people who write the language deciding what counts as the same word, recorded as a documented convention with worked examples. This is the same point that applies to scoping language coverage at the locale level more generally: for languages without a settled orthography, the convention is part of the deliverable.

Second, script complexity changes the annotator requirement, not just the tooling requirement. Verifying Devanagari transcription, where diacritics attach above, below and beside a base character, requires someone who reads Devanagari fluently, not someone comparing two strings. The same holds for Arabic contextual forms, where a visually plausible wrong form is invisible to a non-reader. Projects that budget script-complex languages at Latin-script review rates tend to discover the shortfall at the QA stage — the same failure mode covered in how human-in-the-loop review improves multilingual data quality and in what actually breaks in multilingual AI data collection.

• Separate rendering from linguistics: right-to-left display is an implementation problem, and treating it as one keeps engineering fixes from being confused with data questions.
• Decide the orthographic convention before collection, with native-speaker input and worked examples, and state whether the project collects as-written, normalised, or both in parallel — the same discipline used when collecting speech and text that mixes languages.
• Specify diacritic handling explicitly, distinguishing letter-distinguishing marks from vowel marks where the script has both.
• Use per-region script detection, not document-level language flags, for mixed-script material.
• Do not equate script with language: add a language-identification stage after script detection, informed by the same reasoning covered in high-resource vs low-resource languages in AI training.
• Account for morphology in volume targets, since token parity is not coverage parity for morphologically rich languages — a point that also shapes how multilingual text data trains better LLMs.
• Budget script-literate reviewers rather than string comparators, and price the work accordingly.
• Check what happens to the corpus when it is merged with an existing one: diacritic stripping, normalisation convention and encoding form are where merges silently corrupt data. Lifewood's own multilingual data collection programs and broader AI data services are built around exactly this checklist.