LLM Latency Budget: Where the Time Goes

An LLM latency budget starts with one user-visible target, then allocates time to every operation on the critical path. Set the target at P95 for a specific task, give each span an owner and timeout, reserve slack, and measure observed latency against the allocation.

Do not treat “the model took four seconds” as a diagnosis. Retrieval, queues, tool calls, retries, and validation can consume more time than generation. The production observability trace must expose them separately.

Define the experience before the number

Latency is task-dependent. An autocomplete interaction and a research report should not share an objective. For each important task, record:

• the event that starts the clock;
• the first useful result the user sees;
• the event that means the task is complete;
• the P50, P95, and P99 targets;
• the timeout and user-facing fallback.

Time to first token is useful for chat, but a token is not always useful. A tool-using agent may stream reassuring prose while no work has completed. Track time to first useful value and time to complete separately.

Streaming can improve perceived responsiveness. It does not reduce backend work, total completion time, or resource occupancy. Label it accordingly.

The latency budget worksheet

Copy this table for one task and fill the numbers in milliseconds. The allocations below are illustrative, not universal targets.

Slack is not unowned time. It absorbs normal variance and coordination overhead. If a span repeatedly consumes it, revise the architecture or the objective instead of quietly expanding the total.

Calculate the critical path correctly

For sequential work, the budget is approximately the sum of the spans plus coordination overhead:

T_total = T_ingress + T_retrieval + T_model + T_tools + T_validation + T_slack

For independent parallel work, budget the slowest branch—not the sum—plus fan-out and merge time:

T_parallel_stage = max(T_branch_1, T_branch_2, ...) + T_coordinate

Parallelism helps only when the branches are independent. Starting ten speculative calls can lower wall-clock time while multiplying spend and rate-limit pressure. Measure cost per successful task alongside latency; the model routing framework shows how to keep quality and escalation in the decision.

Find the span that owns the miss

When P95 exceeds the target, segment before optimizing:

1. Compare request slices: intent, tenant, geography, model route, prompt version, cache status, and tool path.
2. Identify the first span whose observed distribution exceeds its allocation.
3. Separate queue time, time to first token, generation time, retries, and downstream tool duration.
4. Test one architectural change against the same slice.
5. Promote the result only if quality, cost, and tail latency remain inside their gates.

Averages hide the users who wait. P50 describes the common case; P95 and P99 reveal saturation, slow dependencies, large contexts, and retry storms.

Use the right lever

Never give every child operation the full request timeout. The parent deadline must flow downward. A tool starting with 300 ms remaining should not begin a two-second retry policy.

Release gate

• Each important task has start, useful-value, and completion events.
• P50, P95, and P99 are segmented by route and task type.
• Every critical-path span has an allocation, owner, and timeout.
• Child deadlines fit inside the remaining parent deadline.
• Fallback behavior is tested, not merely documented.
• Streaming is reported separately from completion latency.
• Changes pass quality and cost gates as well as the latency gate.

Use this worksheet beside the broader first production LLM application checklist. If a customer-facing workflow still misses its objective after instrumentation, JoinAI can help turn the trace into a concrete production improvement plan.

Sources and further reading

• OpenAI: Latency optimization
• OpenTelemetry: Generative AI semantic conventions registry
• Google SRE: Service level objectives