Advanced speech settings

Note: Advanced speech settings are only supported by telephony integrations.

This guide provides additional, advanced settings for speech features. You can toggle these advanced settings off and on by navigating to Agent settings > Speech and IVR > Speech-to-Text > Advanced speech settings.

These settings are available in agent settings (applies to entire agent), flow settings (applies to entire flow and overrides agent settings), page settings (applies to page and overrides flow and agent settings), and fulfillment settings (applies to fulfillment and overrides page, flow, and agent settings). A subset of these settings are available at each level, depending on the setting relevance for the level.

To configure these settings at any lower level, first select Enable advanced speech settings in your agent settings.

Updated agent-level settings don't propagate to the flow, page, and fulfillment level when the Customize option is selected on these lower levels. If the Customize option encompasses multiple settings and you want to update only some of them, you might also need to update other settings if you want them to be the same as the agent-level settings.

If you clear Enable advanced speech settings on a flow, the flow doesn't inherit agent speech or Barge-in settings. Keep this option selected on flows (such as Default Start Flow) that inherit agent settings.

Settings availability by level

The following table indicates which advanced speech settings are available at each level:

Setting name	Agent	Flow	Page	Fulfillment
Model selection (Speech-to-Text)	✔	✔	✔
End of speech sensitivity	✔	✔	✔
Advanced timeout-based end of speech sensitivity	✔	✔	✔
Enable smart endpointing	✔
No speech timeout	✔	✔	✔
Barge-in	✔	✔		✔
Allow cancellation of partial response playback				✔
Audio export bucket	✔	✔
DTMF	✔	✔	✔

Model selection (Speech-to-Text)

Can be set at the agent, flow, and page levels.

Sets the speech model used for speech recognition. This setting is language-specific, so you can select different models for different languages. You can also select Override request-level speech model. This option applies the agent, flow, or page model instead of the model in a request or ConversationProfile.

In bidirectional streaming sessions (BidiStreamingAnalyzeContent), use the same Speech-to-Text version across the agent, flows, and pages. Don't mix Speech-to-Text V2 (chirp_3) with Speech-to-Text V1 models (such as telephony_short or latest_short) in the same session.

For Dialogflow CX Phone Gateway, see limitations. For more information, see Speech models.

End of speech sensitivity

Can be set at the agent, flow, and page levels.

Controls the sensitivity for recognizing the end of speech in end-user audio input. The value ranges from 0 (low sensitivity, less likely to end speech) to 100 (high sensitivity, more likely to end speech).

Keep the following recommendations in mind:

• When you use Speech-to-Text V2 chirp_3, start with an End of speech sensitivity value of 75, or 80 on noisy 8 kHz telephony audio. Don't set the value to 100, which can cause misrecognitions on noisy lines. Keep Enable advanced timeout-based end of speech sensitivity enabled in the console. In API requests, set useTimeoutBasedEndpointing to true.
• If you use a ConversationProfile, verify the End of speech sensitivity value in your agent, flow, and page settings. Low values, such as 35, can add multi-second endpointing delays.

Note: This setting only supports the phone_call speech model for the en-US language tag unless you check the Enable advanced timeout-based end of speech sensitivity setting.

Advanced timeout-based end of speech sensitivity

Can be set at the agent level and disabled at the flow and page levels.

If this setting is enabled, the End of speech sensitivity setting value is used as a gauge to establish a relative audio silence time-out to determine the end of speech. If this setting is disabled (the default), the End of speech sensitivity setting value is used to determine the end of speech by the ML model provided by Google Cloud Speech-to-Text.

While the End of speech sensitivity setting supports only the phone_call speech model for the en-US language tag by default, the Enable advanced timeout-based end of speech sensitivity setting allows for configuring end of speech sensitivity for all languages and speech models supported by Dialogflow.

Enable smart endpointing

Can be set at the agent level only.

If this setting is enabled, Dialogflow CX will analyze partial user input to determine the end of speech. For example, if the user says "I would like to" and pauses, Dialogflow CX will wait for the user to continue the sentence.

This is particularly useful for numeric parameter collection, where the user might say "1234" and pause before saying "5678". In order to apply this setting for a specific parameter, you have to set up Smart endpointing in the parameter's form.

This setting is available for the en-US language tag only and is disabled by default.

Important: When you use Speech-to-Text V2 chirp_3, keep Enable smart endpointing disabled. This setting isn't compatible with chirp_3 and adds a 5-second post-speech delay.

No speech timeout

Can be set at the agent, flow, and page levels.

The time duration in seconds for which Dialogflow CX will stop waiting for end-user audio input. The default is 5 seconds, and the maximum value is 60 seconds. For this timeout, Dialogflow CX invokes a no-input event.

When you use Speech-to-Text V2 chirp_3, we recommend a no speech timeout of 8 seconds.

Barge-in

Can be set at the agent, flow, and fulfillment levels.

When enabled, an end-user can interrupt Dialogflow CX response audio. When interrupted, Dialogflow CX will stop sending audio, and it will process the next end-user input.

If there are multiple messages in the message queue, and a message was queued by a fulfillment associated with a page, flow, or agent that has barge-in enabled, then all following messages in the queue will also have barge-in enabled. In this case, the integration will stop playing audio for all of the queued messages with barge-in enabled.

If you use barge-in with a custom telephony or streaming integration, follow these guidelines:

• Per-turn StreamingAnalyzeContent calls: Include barge_in_config in InputAudioConfig on every turn of the conversation. If you omit barge_in_config on later turns, barge-in stops working after the first turn.
• Prerecorded .wav prompts: Trim trailing silence from prerecorded audio files in <audio> tags. Trailing silence lengthens playback, which can delay barge-in eligibility and the no-speech timeout.

Note: Enabling barge-in increases the total billable duration for each turn. Because the agent is actively listening for an interruption while it's speaking, you're billed for both Input Seconds and Output Seconds simultaneously during the response playback, even if the user remains silent.

Allow cancellation of partial response playback

Can be set at the fulfillment level only.

You can enable this setting when the Enable advanced speech settings box is checked in Agent settings > Speech and IVR and partial response is enabled at the fulfillment level. This setting allows cancellation of a partial response playback.

If a message in the message queue is created by a fulfillment that allows cancellation, playback of the message is cancelled if another message is added to the queue. This is useful when you want an initial message to start playback, but for that playback to be cancelled if a working webhook produces another message before playback of the initial message completes.

Audio export bucket

Can be set at the agent and flow levels.

If supplied, any audio data associated with a request will be saved to the Cloud Storage bucket:

Audio Saved	Applicable requests
End-user audio input	DetectIntent, StreamingDetectIntent, AnalyzeContent, StreamingAnalyzeContent
Text-to-Speech (TTS) audio synthesized for a response	AnalyzeContent, StreamingAnalyzeContent

Grant the Storage Object Creator role to the following service accounts in your project:

• To the service account of the format one-click@df-cx-ALPHANUMERIC_VALUE-ALPHANUMERIC_VALUE. if you use a partner built-in telephony integration.

• To the service account of the format service-PROJECT_NUMBER@gcp-sa-dialogflow. if you use the Dialogflow CX Phone Gateway integration. To find this service account in IAM, check the Include Google-provided role grants option.

Streaming text-to-speech for Chirp 3: HD voices

Streaming text-to-speech for Chirp3-HD voices requires access approval. After approval, select Enable streaming text-to-speech in Agent settings > Speech and IVR > Text-to-Speech. Streaming synthesis uses StreamingSynthesize to reduce latency on longer responses.

Keep the following behavior in mind:

• To use streaming text-to-speech with Barge-in, use BidiStreamingAnalyzeContent. Streaming text-to-speech with barge-in isn't supported on per-turn StreamingAnalyzeContent calls.
• Within a single turn, Dialogflow CX concatenates consecutive text responseMessages into one text-to-speech synthesis request.

DTMF

See the DTMF (Dual-tone multi-frequency signaling) documentation for more information about this feature.

If your StreamingDetectIntent client intercepts out-of-band DTMF digits, don't cancel the active gRPC audio stream. Canceling the stream without half-closing it leaves the server stream open until the no-speech timeout expires. Instead, stop sending audio frames, send the DTMF queryInput on the same stream, and half-close the stream.