Streaming response
A streaming response is when an LLM emits its output token-by-token as it generates rather than returning the full text at the end. Streaming dramatically improves perceived latency: the user sees the first words within milliseconds instead of waiting seconds for the whole reply.
Why it matters
Without streaming, a 5-second LLM call feels like a 5-second wait. With streaming, the same call feels almost instant: the first word appears within ~100 milliseconds and the rest unfolds in front of the user. The total time-to-completion is identical; the perceived experience is dramatically better.
Streaming matters most for:
- Voice calls. The voice synthesis can begin speaking the first generated token while the model is still producing the rest. Total latency for the user drops to "barely noticeable" instead of "uncomfortably long."
- Conversational chat. Watching the agent type one word at a time feels like a real conversation; waiting for a complete reply feels like submitting to a server.
- Long responses. A 500-word answer that takes 4 seconds to fully generate is unusable as a single bubble; streamed token-by-token, the user starts reading immediately.
The trade-offs:
- Tool calls cannot stream. The model has to finish the call before the runtime can execute the tool. Mid-call latency is unavoidable for the tool execution.
- Error handling is harder. A streaming response that fails halfway leaves the user with partial output. Recovery patterns matter.
How Ajeris handles this
Ajeris streams every text response. On voice calls, the synthesis pipeline begins speaking as soon as the first ~5-10 tokens are available, so the user hears the agent's response as one continuous flow. On chat surfaces (SMS, WhatsApp), some channels do not support partial-message updates; the agent buffers there and sends complete messages.
For tool-using turns, the visible latency is the tool execution time itself, not the model's reasoning before or after.
