How to Collect Speech and Text That Mixes Languages

Ask someone in Dhaka how their day went and you may get a sentence that starts in Bangla, carries an English noun phrase in the middle, and finishes with a Bangla verb. Nobody in the conversation notices — it is not two languages taking turns, it is one way of speaking. Put that same sentence through a speech recognition system trained on monolingual data and word error rates spike by 30 to 50% at the language boundary, tokenizers emit unknown-token markers, and real-time streams stall at the switch point. That gap between how a large share of the world actually speaks and what most systems can process is the reason code-switched data collection exists as a discipline, and it is one of the harder things to collect well, for reasons that have less to do with recording and more to do with what you write down.

Code-switching (sometimes called code-mixing in the speech literature) is the use of elements from more than one language within the same utterance or discourse. The distinction that matters operationally is where the switch happens, because each type breaks a different part of the pipeline.

Inter-sentential switching happens between sentences — one sentence in Malay, the next in English — and is the easiest case; some pipelines handle it acceptably. Intra-sentential switching happens within a single utterance. It is the common case in practice and the one that breaks the standard architecture: first-generation systems ran a language classifier over the whole utterance, assigned one language label, and routed to a monolingual model, which fails completely once an utterance contains two languages and one label has to win. Insertional switching embeds single words or short phrases from one language into the matrix of another, and is extremely common but easy to mislabel as noise or an accent artefact. Intra-word switching is the hardest case: the literature's clearest example is Bantu languages, where an English root carries a Bantu affix, blurring phonological and lexical boundaries simultaneously so the word cannot be assigned cleanly to either language.

If a collection specification does not distinguish these four types, the resulting dataset will contain all of them labelled inconsistently, which teaches a model very little.

A substantial proportion of published resources and benchmarks concern Mandarin-English, Hindi-English and Arabic-English, alongside a small number of low-resource pairs such as Frisian-Dutch and Malay-English. Scale is modest even for the well-resourced pairs: the ASRU 2019 Mandarin-English challenge released 500 hours of Mandarin-only audio, 200 hours of intra-sentential code-switched audio, and 40 hours of development data. CS-Dialogue, a more recent corpus, contains 104 hours of spontaneous Mandarin-English dialogue. For South African languages, a frequently cited corpus drawn from soap opera audio provides 14.3 hours spanning English, isiZulu, isiXhosa, Setswana and Sesotho — fourteen hours, for five languages, in one of the most code-switched linguistic environments on earth.

The literature is direct about why: real-world code-switching is data scarce because annotated corpora remain rare and expensive to collect. One specific reason is easy to miss — code-switching is predominantly a spoken, non-literary phenomenon, so scraping text does not produce it. The written record underrepresents exactly the register where it lives, which means for most language pairs this data does not exist and has to be made.

Read speech does not produce natural code-switching — hand a bilingual speaker a script and they read the script. The ASRU challenge data was collected via smartphones in quiet rooms with speakers from 30 provinces, which produces clean audio and controlled conditions but constrains spontaneity. Spontaneous dialogue produces genuine switching and harder audio: the CS-Dialogue corpus was built as spontaneous dialogue with full-length transcriptions, and the South African corpus used broadcast material precisely because scripted elicitation would not have produced the phenomenon. The practical resolution most programmes reach is a topic-prompted conversation — give participants subjects to discuss rather than sentences to read, and let the switching happen naturally.

Speaker demographics need deliberate spread, in the same way they do for accented and non-native speech collection. The ASRU data reports speakers from 30 provinces, 70% under 30, with no significant gender imbalance — a skew worth noting as a limitation rather than a model, since switching patterns differ substantially by age, education and social setting. Recording conditions should match deployment: if the product runs on phones in noisy environments, controlled studio audio produces a model that only works in studios. Domain coverage also matters more than in monolingual collection, because switching behaviour is domain-dependent — technical vocabulary triggers switching in some communities that everyday conversation does not.

The core requirement is language labelling at the word level. The MUSCAT benchmark team, working on multilingual scientific conversation, instructed annotators to mark all words belonging to the embedded language whenever code-switching occurred — the minimum viable convention is not just what was said, but which language each token belongs to. Without it, a transcript is one a monolingual model will misread and a code-switching model cannot learn switch points from.

Several further decisions have to be fixed in advance. Script choice for the embedded language: when a Hindi speaker uses an English word, is it written in Latin script or transliterated into Devanagari? Both conventions exist, and mixing them within one corpus is the failure mode. Intra-word handling: for an English root with a Bantu affix, is the word tagged as one language, both, or a separate category — there is no default answer, and the annotation guideline has to state one. Named entities and borrowings: a word that has entered the matrix language as a loanword is not a code-switch, and where the line sits between an established borrowing and a genuine switch needs a documented rule with examples or two annotators will draw it differently. Language diarization — a distinct layer answering which language was spoken when, alongside the separate question of who spoke when — is the one most frequently forgotten in scoping; the DISPLACE challenges treat it as its own annotation task, related to speech diarization work more broadly. Practitioners on the South African corpus used ELAN for manual annotation of monolingual and mixed segments, capturing duration statistics per language to verify representativity — knowing how many hours exist per language within a code-switched corpus is different from knowing total hours, and only the first tells you whether the balance is usable.

A transcriber fluent in Bangla but not English will mis-transcribe the English segments. One fluent in both but unfamiliar with the local mixing pattern will normalise the switching away, writing what the speaker "meant" in one language rather than what they actually said across two. The MUSCAT team hit a related version of this problem: unable to find external annotators with both language fluency and familiarity with technical scientific discourse, they used automatic transcription as a first pass and had the original speakers correct their own recordings, guaranteeing both linguistic and domain accuracy. That specific approach does not generalise to every project, but the underlying principle does — the verification layer needs someone with the same linguistic repertoire as the speaker, not merely competence in both languages separately.

This is one reason managed multilingual data collection programmes budget separately for code-switched language pairs rather than folding them into standard monolingual scope. Lifewood runs collection through 40+ delivery centres across 30+ countries, and code-switching is one of the clearest cases where in-region presence is a requirement rather than a preference. Bangla-English switching in Dhaka, Malay-English in Kuala Lumpur, Swahili-English in Nairobi: each has its own switch points, its own established borrowings, and its own sense of which mixtures sound natural rather than performed. Recruiting native contributors from the same community matters here specifically because verification means a second speaker from that community, not a bilingual reviewer from anywhere else — the same principle that underpins running a speech data collection programme more broadly.

The cascade approach — classify, then route — added compounding latency and cut words mid-switch. End-to-end multilingual architectures now handle intra-sentential switches natively without LID routing overhead, reportedly reducing word error rate by up to 55% at language boundaries. Frame-level language identification, operating on 10 to 25 millisecond acoustic slices rather than whole utterances, addresses the one-label-must-win problem directly. The collection implication is that data labelled only at utterance level is worth less than it was; word-level and frame-aligned labelling has become the useful format.

Synthetic code-switched data, generated from monolingual transcripts, is being used to supplement real data, and recent work frames this explicitly as a way to reduce reliance on expensive real-world collection. The caveat is the same as everywhere else in this field: synthetic switching reflects the switch patterns of whoever designed the generation process, not the patterns of the community, so it needs real data as an evaluation anchor even where it supplements training. There is also a measurement shift worth noting — standard word error rate averages performance across the whole utterance, which dilutes the thing that actually matters. Point-of-Interest Error Rate, proposed specifically for this problem, measures accuracy at the switch points themselves, so a project commissioning code-switched data should ask what metric the evaluation will use before assuming an acceptable aggregate WER means acceptable switch-point performance.

• The language pair and the direction — Hindi-matrix with English insertions behaves differently from the reverse.
• The switching types in scope, explicitly, including whether intra-word cases are included and how they are tagged.
• Elicitation method, and whether spontaneity or audio quality takes precedence when the two conflict.
• Speaker demographics, with age and setting spread specified rather than assumed, since switching behaviour varies sharply across both.
• Transcription convention in full, with worked examples: word-level language tags, script for the embedded language, the intra-word rule, the borrowing boundary, and diarization layers.
• Per-language duration targets within the corpus, not just total hours.
• The evaluation metric, so collection is optimised for switch-point performance if that is what matters.
• Native-speaker verification from the same community as the speakers, not just bilingual review.

Get the convention right and a modest corpus is useful. Get it wrong and a large one is a mixture of four annotation styles that no model can learn a consistent rule from.