Short answer. Code-switching — using more than one language inside a single utterance — breaks monolingual ASR at the language boundary. First-generation systems classified the whole utterance and routed it to one monolingual model, which cannot work on intra-sentential switching because no single model covers the utterance. Four types matter operationally: inter-sentential, intra-sentential, insertional and intra-word, the last being cases like an English root with Bantu affixes. Published resources cluster heavily on Mandarin-English, Hindi-English and Arabic-English, so for most low-resource pairs there is nothing to fine-tune on and collection is the only route.
Mixes Languages?
Ask someone in Dhaka how their day went and you may get a sentence that starts in Bangla, carries an English noun phrase in the middle, and finishes with a Bangla verb. Nobody in the conversation notices. It is not two languages taking turns. It is one way of speaking.
Now put that sentence through a speech recognition system trained on monolingual data. Word error rates spike by 30 to 50% at language boundaries, tokenizers emit unknown-token markers, and real-time streams stall at the switch point.
That gap between how a large share of the world actually speaks and what most systems can process is the reason codeswitched data collection exists as a discipline. It is also one of the harder things to collect well, for reasons that have less to do with recording and more to do with what you write down.
What code-switching actually is, and why the type matters
Code-switching, sometimes called code-mixing in the speech literature, is the use of elements from more than one language within the same utterance or discourse.
The distinction that matters operationally is where the switch happens, because each type breaks a different part of the pipeline.
Inter-sentential switching happens between sentences. One sentence in Malay, the next in English. This is the easiest case and some pipelines handle it acceptably.
Intra-sentential switching happens within a single utterance. This is the common case in practice and the one that breaks the standard architecture, for a reason worth understanding: first-generation systems ran a language classifier over the whole utterance, assigned one language label, and routed to a monolingual model. That fails completely on intra-sentential switches, because the utterance contains two languages and one label has to win.
Insertional switching embeds single words or short phrases from one language into the matrix of another. Extremely common, and easy to mislabel as noise or as an accent artefact.
Intra-word switching is the hardest. The literature gives the clearest example from Bantu languages: English roots carrying Bantu affixes, which blurs phonological and lexical boundaries simultaneously. A word that is neither one language nor the other cannot be assigned to either.
If your collection specification does not distinguish these, your dataset will contain all four and label them inconsistently, which produces a corpus that teaches a model very little.
Where the existing data is, and where it is not
The concentration is stark and worth knowing before scoping any project.
A substantial proportion of published resources and benchmarks concern Mandarin-English, Hindi-English and Arabic-English, alongside a small number of low-resource pairs such as Frisian-Dutch and Malay-English.
The scale of even the well-resourced pairs is modest. For the ASRU 2019 Mandarin-English challenge, DataTang released 500 hours of Mandarin-only, 200 hours of intra-sentential code-switched, and 40 hours of development data.
CS-Dialogue, a more recent corpus, contains 104 hours of spontaneous Mandarin-English dialogue. For South African languages, a frequently cited corpus drawn from soap opera audio provides 14.3 hours spanning English, isiZulu, isiXhosa, Setswana and Sesotho.
Fourteen hours, for five languages, in one of the most code-switched linguistic environments on earth.
The explanation given in the literature is direct: real-world code-switching is data scarce, because annotated corpora remain rare and expensive to collect. And one specific reason is easy to miss. Code-switching is predominantly a spoken, non-literary phenomenon, so scraping text does not produce it. The written record underrepresents exactly the register where it lives.
Which means for most language pairs, this data does not exist and cannot be found. It has to be made.
The collection design decisions
Elicitation method determines whether you get real switching at all.
Read speech does not produce natural code-switching. Hand a bilingual speaker a script and they read the script. The ASRU challenge data was collected via smartphones in quiet rooms with speakers from 30 provinces, which produces clean audio and controlled conditions but constrains spontaneity.
Spontaneous dialogue produces genuine switching and harder audio. The CS-Dialogue corpus was built specifically as spontaneous dialogue with full-length transcriptions, and the South African corpus used broadcast material precisely because scripted elicitation would not have produced the phenomenon.
The practical resolution most programmes reach is a topic-prompted conversation: give participants subjects to discuss rather than sentences to read, and let the switching happen naturally.
Speaker demographics need deliberate spread. The ASRU data reports speakers from 30 provinces, 70% under 30, with no significant gender imbalance. That skew toward young speakers is worth noting as a limitation rather than a model, because switching patterns differ substantially by age, education and social setting. An older speaker in the same city may switch at different points, at different rates, or barely at all.
Recording conditions should match deployment. If the product runs on phones in noisy environments, controlled studio audio produces a model that works in studios.
Domain coverage matters more than in monolingual collection, because switching behaviour is domain-dependent.
Technical vocabulary triggers switching in some communities that everyday conversation does not.
The part that decides whether the dataset is usable: transcription
convention This is where code-switched projects succeed or fail, and it is a specification decision made before any audio is recorded.
The core requirement is language labelling at the word level. The MUSCAT benchmark team, working on multilingual scientific conversation, instructed annotators to mark all words belonging to the embedded language whenever code-switching occurred. That is the minimum viable convention: not just what was said, but which language each token belongs to.
Without it you have a transcript that a monolingual model will misread and a code-switching model cannot learn switch points from.
Several further decisions have to be fixed in advance:
Script choice for the embedded language. When a Hindi speaker uses an English word, is it written in Latin script or transliterated into Devanagari? Both conventions exist. Mixing them within one corpus is the failure mode.
Intra-word handling. For English roots with Bantu affixes, is the word tagged as one language, both, or a separate category? There is no default answer and the annotation guideline has to state one.
Named entities and borrowings. A word that has entered the matrix language as a loanword is not a code-switch. Where the line sits between an established borrowing and a genuine switch is a judgement call, and it needs a documented rule with examples or two annotators will draw it differently.
Language diarization. For multi-speaker audio, the DISPLACE challenges frame this as two parallel questions: who spoke when, and which language was spoken when. The second is a distinct annotation layer and it is frequently forgotten in scoping.
Practitioners working on the South African corpus used ELAN for manual annotation of monolingual and mixed segments, capturing duration statistics per language to verify representativity. That last detail is worth copying: knowing how many hours you have per language within a code-switched corpus is different from knowing total hours, and only the first tells you whether the balance is usable.
The annotator problem
Code-switched transcription cannot be done by a monolingual speaker of either language, which is a more binding constraint than it sounds.
A transcriber fluent in Bangla but not English will mis-transcribe the English segments. One fluent in both but unfamiliar with how the two are mixed in that specific community will normalise the switching away, writing what the speaker "meant" in one language rather than what they said across two.
The MUSCAT team hit a related version of this and solved it in a way worth knowing about. Unable to find external annotators with both language fluency and familiarity with technical scientific discourse, they used automatic transcription as a first pass and had the original speakers correct their own recordings, which guaranteed both linguistic and domain accuracy.
That approach does not generalise to every project, but the underlying principle does: the verification layer needs someone with the same linguistic repertoire as the speaker, not merely competence in both languages separately.
This is where our own footprint matters, so I will declare the interest. Lifewood runs collection through delivery centres across more than 30 countries, and code-switching is one of the clearest cases where in-region presence is not a preference but a requirement. Bangla-English switching in Dhaka, Malay-English in Kuala Lumpur, Swahili-English in Nairobi: each has its own switch points, its own established borrowings, and its own sense of which mixtures sound natural versus performed.
Recruiting for that means recruiting locally, and verification means a second speaker from the same community rather than a bilingual reviewer from anywhere.
How the field is changing, and what it means for collection
Two developments worth knowing because they affect what data is worth collecting.
Architecture has moved past language identification routing. The cascade approach, classify then route, added compounding latency and cut words mid-switch. End-to-end multilingual architectures handle intra-sentential switches natively without LID routing overhead, reportedly reducing word error rate by up to 55% at language boundaries. Frame-level language identification, operating on 10 to 25 millisecond acoustic slices rather than whole utterances, addresses the one-label-must-win problem directly.
The collection implication: data labelled only at utterance level is worth less than it was. Word-level and frame-aligned labelling has become the useful format.
Synthetic code-switched data is being used to supplement real data, generated from monolingual transcripts.
Recent work explicitly frames this as a way to reduce reliance on expensive real-world collection. The honest caveat is the same as everywhere else in this field: synthetic switching reflects the switch patterns of whoever designed the generation process, not the patterns of the community, and it needs real data as an anchor for evaluation even where it supplements training.
There is also a measurement development worth noting. Standard word error rate averages performance across the whole utterance, which dilutes the thing you actually care about. Point-of-Interest Error Rate, proposed specifically for this problem, measures accuracy at the switch points themselves. If you are commissioning code-switched data, ask what metric the evaluation will use, because an aggregate WER can look acceptable while switch-point performance is poor.
What to specify when scoping a project
The language pair and the direction. Hindi-English is not one thing; Hindi-matrix with English insertions behaves differently from the reverse.
The switching types in scope, explicitly, including whether intra-word cases are included and how they are tagged.
Elicitation method, and whether spontaneity or audio quality takes precedence when they conflict.
Speaker demographics, with age and setting spread specified rather than assumed, since switching behaviour varies sharply across both.
Transcription convention, in full, with worked examples: word-level language tags, script for the embedded language, intra-word rule, borrowing boundary, and diarization layers.
Per-language duration targets within the corpus, not just total hours.
The evaluation metric, so the collection is optimised for switch-point performance if that is what matters.
Native-speaker verification from the same community as the speakers, not just bilingual review.
Get the convention right and a modest corpus is useful. Get it wrong and a large one is a mixture of four annotation styles that no model can learn a consistent rule from.
Key takeaways
- Code-switching is the use of elements from more than one language within a single utterance or discourse, and it breaks monolingual ASR at language boundaries with word error rates spiking 30 to 50%.
- First-generation systems classified the whole utterance and routed to a monolingual model, which fails on intrasentential switching because one label has to win.
- Four types matter operationally: inter-sentential, intra-sentential, insertional and intra-word. Intra-word, such as English roots with Bantu affixes, is hardest because it blurs phonological and lexical boundaries at once.
- Published resources concentrate heavily on Mandarin-English, Hindi-English and Arabic-English, with few lowresource pairs.
- Scale is modest even for well-resourced pairs: 200 hours of code-switched Mandarin-English in the ASRU 2019 release, 104 hours in CS-Dialogue, and 14.3 hours covering five languages in a widely used South African corpus.
- Code-switching is predominantly spoken and non-literary, so text scraping does not produce it. For most pairs the data must be created.
- Read speech does not produce natural switching. Topic-prompted spontaneous conversation is the usual resolution between authenticity and audio quality.
- The ASRU corpus skews to speakers under 30, which is a limitation rather than a model, since switching patterns vary by age, education and setting.
- Word-level language labelling is the minimum viable transcription convention. MUSCAT annotators were instructed to mark all words belonging to the embedded language.
- Five conventions must be fixed before recording: word-level tags, script choice for the embedded language, intraword handling, the borrowing versus switch boundary, and language diarization.
- Language diarization asks which language was spoken when, a distinct annotation layer from speaker diarization and frequently omitted in scoping.
- Transcription requires annotators with the same linguistic repertoire as speakers, not competence in both languages separately. MUSCAT had speakers correct their own ASR-drafted transcripts.
- End-to-end multilingual architectures now handle intra-sentential switching natively, reportedly reducing word error rate by up to 55% at language boundaries, which raises the value of word-level and frame-aligned labels over utterance-level ones.
- Synthetic code-switched data can supplement real data but reflects the switch patterns of its generator, so real data remains necessary as an evaluation anchor.
- Point-of-Interest Error Rate measures accuracy at switch points specifically, where aggregate word error rate dilutes it.
Sources and further reading
- Gladia, "Code Switching in Speech Recognition: ASR Guide 2026", on word error rate spikes at boundaries, utterance-level versus frame-level language identification, and end-to-end architectures reducing WER by up to 55% at language boundaries
- "CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition", arXiv, on corpus scale and the definition of code-switching
- Emergent Mind, "Code-Switching ASR Advances", on the South African five-language corpus, ELAN annotation with per-language duration statistics, intra-word switching with Bantu affixes, and the concentration of resources on Mandarin-English, Hindi-English and Arabic-English
- "The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge", arXiv, on the DataTang dataset composition and speaker demographics
- "MUSCAT: MUltilingual, SCientific ConversATion Benchmark", arXiv, on the ASR-first-pass then speaker-correction workflow and the instruction to mark embedded-language words
- "The Second DISPLACE Challenge: DIarization of SPeaker and LAnguage in Conversational Environments", arXiv, on language diarization as a distinct task
- "Adding Robust Code-Switching Capabilities to High Performance Multilingual ASR", arXiv, on data scarcity, synthetic code-switched data generation and Point-of-Interest Error Rate
- "Code Switched and Code Mixed Speech Recognition for Indic languages", arXiv, on the non-literary nature of codeswitching and consequent corpus scarcity
- Lifewood, multilingual speech data collection and delivery network