Phonemes, prosody, and vocoder intuition

Goal of this lesson

Explain the division of labor among pronunciation, prosody, acoustic representation, and waveform generation without training a model.

Units of pronunciation

A phoneme is the smallest speech unit that distinguishes word meaning; the same letter can represent different phonemes in different words or languages. Grapheme-to-phoneme (G2P) maps written form to a pronunciation sequence. Names, abbreviations, and polyphonic characters often need a lexicon or context. Phoneme inventories, lexicons, and language labels are all part of a model/preprocessing version; do not assume that another engine, accent, or locale will use the same lexicon or IPA convention.

Prosody

Prosody includes rhythm, stress, intonation, pauses, and rate. It affects not just “naturalness” but meaning: interrogative intonation, list pauses, and contrastive stress can all change interpretation. In engineering, do not reduce all prosody to one “emotion” label; define observable controls and evaluation objectives instead.

From text to waveform

A common division of labor is:

1. A text/phoneme encoder builds a linguistic representation.
2. An acoustic model predicts an intermediate acoustic representation such as a mel spectrogram.
3. A vocoder converts that acoustic representation into a waveform.

Tacotron 2 represents the “predict mel spectrogram + neural vocoder” pattern; FastSpeech 2 emphasizes parallelism and controls such as duration, pitch, and energy; VITS illustrates an end-to-end joint-modeling path. These papers help explain the design space, not prove that current products use the same architecture. End-to-end models also do not remove system responsibilities such as text normalization, voice authorization, output encoding, evaluation, and revocation.

Waveform generation is followed by a delivery representation layer: resampling, loudness handling, container/codec encoding, chunking, and playback buffering. An audible defect can come from the model or from later resampling, low-bitrate encoding, clipping, or client decoding. At minimum, distinguish the model output’s sample rate/channels, the encoded file’s parameters, and the chunk size and timeline actually received by the player. Do not attribute every failure to a single “voice model.”

Common distortions and where to investigate

• Wrong pronunciation: inspect normalization, lexicon, language, and phonemes.
• Unnatural rhythm: inspect sentence boundaries, pauses, duration, and prosody controls.
• Metallic sound or pops: inspect the vocoder, sampling, encoding, and clipping.
• Missing words in long sentences: inspect input length, alignment, and segmentation rather than merely changing the voice.
• Incorrect numbers or names: inspect language, text normalization, and pronunciation rules before the model.

Exercise and self-check

• Use a sentence whose intent changes with stress to describe two different readings.
• Must a vocoder take text as input? No; its common input is an acoustic representation.
• Does end-to-end synthesis eliminate normalization and authorization? No; these are outer system responsibilities.

Next step and references

Next, study Data and voice authorization. See the original Tacotron 2, FastSpeech 2, and VITS papers (accessed 2026-07-22). Their results are limited by their data, implementation, and experimental settings; they do not directly equal performance in your scenario.