AI Agent Engineer

AI Agent Engineer
PERSONAL PROGRESS

Mark this course

Course content is public. Sign in to save personal progress.

Text to Speech

5 min read

Text to Speech

Course overview

Text to Speech (TTS) turns text into audible speech. In engineering terms, it is not merely “choosing a voice for a sentence”: input needs normalization, pronunciation and prosody must be controllable, streaming playback must manage latency, voices need authorization, and output must be associated with source_revision, voice-catalog/policy version, purpose, access scope, generation configuration, and release status.

This course introduces text normalization, phonemes, prosody, acoustic representations, vocoders, and end-to-end models before moving to SSML, service engineering, evaluation, and voice-cloning risk. Dynamic material was checked on 2026-07-22. Vendors support different SSML subsets, so confirm a target version’s official documentation and contract for its voices, parameters, upload/streaming behavior, and permissions.

Where this fits in the overall path

Text to speech is in the Extended applications and complex collaboration stage. It is the output channel for a voice agent, usually after Agent Core or Workflow Automation. Together with Speech Recognition, it forms the input/output modules; Real-Time Multimodal Interaction then handles turns, barge-in, low-latency transport, and session recovery.

flowchart LR
    A[Source text\nsource_revision] --> B[Text normalization\nnormalization_revision]
    B --> C[Synthesis-plan gate\nvoice/language/purpose/ACL references]
    C --> D[Controlled synthesis]
    D --> E[Audio asset\nformat, configuration, release_id]
    E --> F[Playback/distribution]
    G[Authorization, disclosure, and revocation policy] -. constrains .-> C
    G -. constrains .-> F

“Plan valid,” “generated,” “playable,” “released,” and “revoked” are distinct states. An offline plan’s acl_reference stores only a reference to the object awaiting authorization; the external identity/authorization system must decide object-level ACL again before generation, reading, and release. In particular, a voice that has already played cannot be “recalled”: cancellation can only stop later generation or transport. See Real-Time Multimodal Interaction for real-time turns, interruption, and tool gates.

Learning objectives

  • Explain the roles of text normalization, phonemes, prosody, acoustic representation, and a vocoder.
  • Use stable SSML concepts to control language, phrasing, pauses, emphasis, and pronunciation, while handling dialect differences.
  • Design contracts for voice selection, batch work, streaming, and caching that state container, codec, sample rate, channels, and player compatibility.
  • Organize naturalness, intelligibility, content-correctness, stability, and preference evaluations.
  • Establish voice authorization, cloning limits, disclosure, provenance records, revocation, and abuse-response processes.
  • Complete an offline project that only creates and validates synthesis plans and SSML; it never produces audio.

Prerequisites

  • Python Fundamentals, JSON, and API are recommended.
  • No linguistics or signal-processing background is required; new terms receive an intuitive introduction.
  1. The TTS pipeline and text normalization: turn source text into a speakable form.
  2. Phonemes, prosody, and vocoder intuition: understand the responsibilities inside a synthesis system.
  3. Data and voice authorization: establish rights boundaries before collecting data or selecting a voice.
  4. SSML and pronunciation control: express pauses, emphasis, and readings safely.
  5. Voice selection, batch processing, and streaming: build a reliable output service.
  6. Quality, intelligibility, latency, and evaluation: validate experience with subjective and objective measures.
  7. Cloning risk, disclosure, and traceability: establish authorization and incident response.
  8. Project: an offline synthesis plan and SSML: create and validate a plan without audio.

Hands-on entry point

Mastery criteria

  • I can distinguish text normalization, pronunciation modeling, acoustic representation, and a vocoder.
  • I can explain how phonemes, stress, intonation, rhythm, and pauses affect intelligibility.
  • I can construct valid basic SSML and identify portability risks in vendor extensions.
  • I can design time-to-first-audio, real-time factor, content-correctness, and human-listening tests.
  • I can demonstrate that a voice and its training/reference data have explicit authorization, purpose limits, and a revocation process.
  • I can run the offline project and verify its plan, SSML, provenance fields, and no-audio boundary.

Connections to other knowledge bases

Primary references

The following sources were checked on 2026-07-22:

Portability and authorization

W3C specifications define semantics, but engines can implement only a subset or add private extensions. “Available” in a voice catalog does not mean usable for every scenario; technical availability, actual authorization, consent, portrait/voiceprint and related rights, and content disclosure must be checked separately. Voice similarity, catalog presence, or an authorization reference in a plan cannot independently prove permission.

Backlinks