# Interim replies while the agent looks things up: evaluating opening lines and wait notices

::: abstract
When a customer's question requires an account lookup or knowledge retrieval, the support agent Ailo produces nothing until its tool calls finish, and the customer may see no reply for a long time. We evaluate two kinds of interim reply generated by a model on the spot: an "opening line" sent before the lookup, and a "wait notice" sent when the wait grows long (internally, "still on it"). Over four rounds on 75 fictional customer messages (Chinese, English, Thai, Vietnamese and Indonesian, each run twice), we compared the main model speaking first (approach A) with a small model generating in parallel (approach B), small-model tiers, caching and context compression, and prompt and reasoning settings. Main results: (1) approach B reached the first line at a median of 1.04–1.83 s depending on the gateway, against 2.96 s for A; we tried caching, context compression and turning reasoning off, and A's best was still 2.27 s (main-model reasoning off). (2) Larger small models were neither faster nor clearly better, at 7–22 times the cost. (3) To avoid contradicting the reply or inventing progress, the small model must wait for the main model's first step and know what it is looking up; under production settings, the tuned wait notice had zero contradictions, invented progress, unfounded promises and leaks in 108 lines; these lines were generated at about 1.5 s and graded offline, so behaviour under real waits is **not yet tested**. (4) With reasoning enabled and a one-line format check, the opening line also reached zero violations, but its median rose to 4.61 s and 34 of 110 lines arrived after the reply and were dropped. Accordingly, the wait notice has been approved (only an empty engine port that sends nothing by default exists so far; it is not live), and the opening line is on hold pending re-evaluation under real waits. Each setting ran only twice, and grading was done by Claude Sonnet 5.5.
:::

## Introduction

When a customer asks something that needs a lookup ("why haven't my diamonds arrived?"), the agent first calls tools to check the account or read the rules, then answers. Under the round 1–2 settings, the first real reply to such messages took a median of 11.1 s; under production settings, about 5.1 s. Meanwhile the customer sees nothing, and may assume nobody is there or send the message again.

Without slowing or degrading the real reply, we want the agent to:

- send an opening line that fits the question before looking things up, such as "I'm checking your diamond order now";
- send a wait notice if the lookup takes long.

Constraints: lines must be generated by a real model, with no fixed templates or phrase bank (a timer may decide when to send); no invented progress, so a line is sent only while work is really in progress and never after completion, cancellation or a specialist starting to serve; no unfounded promises, only "I'm taking a look", never results, times or speed; no leaks of tool names, internal steps or site rules; and the first real reply must not get slower.

We ask: (RQ1) which approach delivers the first line fastest; (RQ2) how to keep the interim reply consistent with the reply that follows; (RQ3) what latency and cost that consistency requires.

## Related approaches

On 2026-11-04 we tried the public demo of a mainstream AI support product: when a customer said purchased diamonds had not arrived, it first replied with a line restating the point of the question, then showed a typing indicator, and only about 10 s later asked for an ID. The line was not configured by hand and quoted the question, so we judge it model-generated; the product has not published how it works.

For latency, the team's rule follows the public practice of Claude and OpenAI: use only prompt caching, context compression, model and reasoning settings, and fewer unnecessary tool round trips, never fixed templates, to gain speed. Section 4.3 tests the first of these on approach A.

## Method

### 3.1 Approaches

::: table Approaches compared
| Approach | Description | Outcome |
| --- | --- | --- |
| A: main model first | The main model writes an extra field with a line for the customer when it calls a tool; streamed and sent as soon as it arrives | Works, slower than B |
| A′: main model writes text before the tool call | The prompt asks the main model to write some text before calling a tool | Unusable: in 15 of 110 turns needing a lookup, it sent "let me check" and ended without looking anything up |
| B: small model in parallel | A small model writes the line while the main model starts | Fastest; from round 3 it waits for the main model's first step |
| Fixed phrases | Pick a line from a prepared set | Fastest (0.45 s), but none of 110 lines fully fit the question; the team decided against templates, so it was not pursued |
:::

For the wait notice we compared only a small model writing on the spot against fixed phrases; the latter was rejected for the same reason.

### 3.2 Dataset

::: table Test messages
| Type | Messages | Runs each | Opening line due? |
| --- | ---: | ---: | --- |
| Account or record lookups | 20 | 2 | Yes |
| Rules questions (need a help article) | 15 | 2 | Yes |
| Upset customers | 10 | 2 | Yes |
| Follow-ups to an earlier message | 5 | 2 | Yes |
| Messages with manipulative content | 5 | 2 | Yes |
| Simple questions answerable directly | 15 | 2 | No |
| Requests for a person | 5 | 2 | No |
:::

Each setting has 150 turns: 110 where an opening line is due and 40 where it is not. Messages were written by `gpt-6-luna` in Chinese, English, Thai, Vietnamese and Indonesian; Thai, Vietnamese and Indonesian were reviewed by Claude Sonnet 5.5 and back-translated into Chinese and English, changing 3. All are fictional. For the long-conversation test, other messages and replies in the same language were joined into a 24-message conversation.

### 3.3 Models and gateways

- **Main model**: the main-branch platform prompt, site settings and tools, choosing its own tools. Tool results were fictional, and an account check always took 2 s. In rounds 1–2 the main model was Claude Haiku 5.5 via the test gateway; rounds 3–4 used production settings, `gpt-6-luna` via the production gateway throughout.
- **Small model**: Claude Haiku 5.5 with reasoning off by default, told the customer's language explicitly (without it, Indonesian came out as Malay).
- **Grader**: Claude Sonnet 5.5.

### 3.4 Measures

- **Time to first line**: from the start of the turn until the customer sees the first line, median (p50) and 95th percentile (p95).
- **Fit**: whether the line addresses the customer's specific question.
- **Violations**: the grader compares each line with the reply that follows and checks four issues: contradiction with the reply, invented progress, unfounded promise, and leak of internal information.
- **Cost**: extra cost per 10,000 turns, estimated from tokens and list prices.
- **Checks beyond the grader** (from round 4): a script flags multi-line output and text in a language other than the customer's; the grader is asked separately about drafts or notes; and every line sent is read through.

## Results

### 4.1 Approach A vs approach B (RQ1, 2026-11-06)

::: table Opening line: time to first line and quality (main model Claude Haiku 5.5)
| Approach | First line p50 / p95 (s) | Contradiction / invented progress / unfounded promise | Extra per 10,000 turns (USD) |
| --- | ---: | ---: | ---: |
| A main model first (streamed) | 2.96 / 3.94 | 0 / 0 / 0 (110 lines) | 0.82 |
| A main model first (not streamed) | 4.01 / 5.15 | — | — |
| B small model · test gateway | 1.83 / 2.41 | 0 / 0 / 0 (110 lines) | 0.53 |
| B small model · production gateway | 1.04 / 1.26 | 2 / 2 / 0 (112 lines) | 0.57 |
:::

![Time to the opening line](../../assets/ack-r12-first-line.en.svg "Approach B is fastest; the same small model is about 0.8 s faster again on the production gateway")

- With an opening line, the first real reply did not change: a median of 11.1 s and p95 of 20.1 s for messages needing a lookup.
- A and B (test gateway) sent a line in all 110 turns where one was due and in none of the 40 where it was not; on the production gateway B sent 2 of the 40.
- B's 2 contradictions on the production gateway both came from messages like "get me a person, and a question about a payment": the small model said it would check, while the main model only handed over.

### 4.2 Small-model tiers

::: table Small-model tiers (same prompt, 150 runs each, 2026-11-06)
| Small model | Gateway | First line p50 / p95 (s) | Per 10,000 turns (USD) | Contradictions | Note |
| --- | --- | ---: | ---: | ---: | --- |
| **Claude Haiku 5.5** | Test | 1.83 / 2.41 | 0.53 | 0 | |
| **Claude Haiku 5.5** | Production | 1.04 / 1.26 | 0.57 | 2 | |
| Claude Haiku 4.5 | Production | 1.07 / 1.70 | 3.89 | 3 | |
| Claude Sonnet 5.5 | Production | 1.19 / 2.65 | 12.61 | 1 | |
| `gpt-6-luna` | Production | 2.56 / 3.33 | 0.67 | 0 | |
| DeepSeek v4.1 flash | Production | 1.12 / 3.12 | 1.02 | 0 | Empty in 18 of 110 |
:::

![Small-model tiers](../../assets/ack-tiers.en.svg "Larger models were not faster; Claude Sonnet 5.5 had one fewer contradiction at about 22 times the cost")

Larger models were not faster and not clearly better (Claude Sonnet 5.5 had one fewer contradiction at about 22 times the cost of Haiku 5.5). Claude Haiku 5.5 is the best fit.

### 4.3 Can caching and compression bring A level with B? (2026-11-06)

These only matter for A: B's prompt is about 380 tokens, below the minimum for caching. We used 20 lookup messages, twice each, timing only the main model's first call.

::: table Approach A time to first line (s, n = 40 per row)
| Setting | p50 / p95 |
| --- | ---: |
| Short conversation, current caching | 2.70 / 3.50 |
| Short conversation, main-model reasoning off | 2.27 / 2.87 |
| 24 messages, current | 3.16 / 4.41 |
| 24 messages, fully cached | 2.97 / 3.72 |
| 24 messages, summary + last 4 | 3.02 / 3.87 |
| **B, same 24 messages** | **1.95 / 2.56** |
:::

![Approach A with caching and compression](../../assets/ack-cache.en.svg "In long conversations, full caching or compression each save about 0.2 s, still well behind B")

- In long conversations the main model's first call currently gets no cache hits. Full caching or compression each save about 0.2 s, and full caching also cuts that call's cost to about a quarter. This helps the main model in its own right but does not catch B (1.95 s on the same long conversation).
- Turning off the main model's reasoning saves about 0.4 s, but it changes the setting for the whole reply and is not worth it for an opening line.

### 4.4 Consistency: waiting for the main model's first step (RQ2, 2026-11-08)

Round 3 used production settings: main model `gpt-6-luna`, everything via the production gateway; the small model stayed Haiku 5.5 with reasoning off. Messages and runs were unchanged.

::: table Violating lines before and after prompt tuning (two runs combined)
| | Lines | Contradiction | Invented progress | Unfounded promise | Leak |
| --- | ---: | ---: | ---: | ---: | ---: |
| Before: opening line | 112 | 3 | 5 | 5 | 0 |
| Before: wait notice | 106 | 0 | 2 | 20 | 0 |
| **After: wait notice** | 108 | **0** | **0** | **0** | **0** |
| After: opening line | 108 | 0 | 0 | 2 | 1 |
:::

![Violating lines at each stage](../../assets/ack-violations.en.svg "After prompt tuning the wait notice had zero violations of all four kinds; the opening line kept 3, falling to zero in round 4")

Five prompt changes; every line is still model-generated:

1. **Wait for the main model's first step** and tell the small model what it will look up; if the first step is not a lookup (a direct answer or a handover), write nothing. This alone brought contradictions and invented progress to zero.
2. **State only what Ailo is doing**, such as "I'm checking your diamond order", without describing the state of the order or the money. Earlier, "this order is still being processed" was judged invented progress.
3. **List what not to say**: no repeating amounts or currencies, no times or speed, no asking the customer to wait, no promising a next step ("I'll update you"), no "I can see".
4. **Stop after "I'm checking"**, with no clause after it.
5. **Keep the prompt short**: with a long prompt, Haiku with reasoning off sometimes printed its draft too.

- The fastest variant writes in parallel and sends only once the main model's first step is a lookup (median 1.46 s), but it could not reach zero violations. For example, when a customer mentioned "that 500 from last night", the small model guessed a top-up both times, while the main model looked up a different kind of record.
- After tuning, the opening line came at a median of 2.63 s and p95 of 3.40 s; the main model's first real reply came at a median of 5.08 s. Three opening-line violations remained: two said something like "I'll answer you right away", and one printed a draft together with an English self-correction. The grader missed the last one; it was found in a separate check of multi-line output.

### 4.5 Reasoning and a format check for the opening line (RQ3, 2026-11-09)

Round 4 gave the small model low reasoning and added a format check: output longer than one line is dropped.

::: table Opening line: round 3 vs round 4
| | Round 3 (no reasoning) | Round 4 (low reasoning + one-line check) |
| --- | ---: | ---: |
| Lines sent | 108 / 110 | 72 / 110 |
| Contradiction / invented progress / unfounded promise / leak | 0 / 0 / 2 / 1 | **0 / 0 / 0 / 0** |
| Opening line p50 / p95 (s) | 2.63 / 3.40 | 4.61 / 6.13 |
| Small-model call p50 (s) | 1.10 | 3.41 |
| Extra per 10,000 turns (USD) | 0.57 | 1.83 |
:::

![Opening-line latency across rounds](../../assets/ack-evolution.en.svg "Moving from pure parallel generation to waiting for the first step to enabling reasoning removed violations step by step while latency rose")

- **Zero violations, about 2 s slower** (the small-model call itself was about 2.3 s slower). The opening line's median was 4.61 s and the main model's first real reply 5.05 s, so 34 of 110 lines arrived after the reply and were dropped by rule, mostly rules questions (16) and account checks (8).
- **The one-line check worked**, stopping 2 multi-line outputs, both drafts.
- **Low reasoning must be enabled explicitly.** Setting reasoning effort to low alone made Haiku never reason; with reasoning explicitly enabled and the minimum budget of 1,024 tokens, it reasoned in 77 of 108 calls, 347 tokens on average.
- **Checks beyond the grader**: the script flagged only 1 line (an English promotion name quoted from the customer's own message); asked separately, the grader found 0 lines with drafts or notes; a read-through of all 72 lines sent found each a single line with no draft. About 10 more lines mentioned internal material such as "the help note"; the grader did not count these as leaks, but the prompt forbids them, so it needs tightening.

### 4.6 Cost

![Extra cost per 10,000 turns](../../assets/ack-cost.en.svg "Approach B costs less extra than A; enabling reasoning roughly triples the opening line's cost")

## Discussion

**Choice of approach.** Approach B is clearly faster to the first line, and none of the speed-ups for A closes the gap; B also costs less extra. The price is one more model call and the chance that the two models read the question differently.

**Consistency versus latency.** Parallel generation is fastest, but the small model does not know what the main model will look up, so it can contradict the reply or describe progress that does not exist. Waiting for the main model's first step removes both problems at a cost of about 1.2 s (1.46 s vs 2.63 s under round 3 settings). Removing unfounded promises and leaks from the opening line as well needs reasoning, adding about 2 s more, so a third of lines miss the reply. Because the wait notice is sent only after a long wait, it is insensitive to latency and meets the requirements under round 3 settings; the opening line is caught between the two.

**Reliability of grading.** The grader missed one leak, showing that format problems (multi-line output, other languages) cannot rest on model grading alone and need scripts and read-throughs.

**Decision.** The wait notice is approved: a small model writes one line after a long wait, sent as a message through the same pre-send checks as a real reply; it is dropped if the reply arrives first, the turn is cancelled or a specialist starts serving; each site can turn it on or off, off by default with a 15-second default wait; the status line under the chat header keeps showing, and neither replaces the other. As of 2026-11-12, the engine has only an empty port that sends nothing by default; before production wiring, several send-path prerequisites must be resolved (for example, the line must not count as a real reply), so it is not live. The opening line is on hold until it is re-evaluated together with the wait notice under real waits. No fixed phrases or phrase bank are used.

## Limitations

- **Few runs.** Each setting ran twice (mostly 75 messages × 2; the caching test 20 × 2), so a difference between 0 and 2 lines may be partly chance.
- **The wait notice has not been sent after a real wait.** The 108 lines were generated when the main model's first step came out (about 1.5 s) and graded offline; those turns all finished within 8 s, so none was sent after a 15–20-second wait, **not yet tested**.
- **Approach A was tested only under round 1–2 settings.** Its latency under production settings is **not yet tested**.
- **Synthetic messages.** A model wrote the messages; real customers may phrase things more messily.
- **Fictional tool results.** Account checks always took 2 s; with slower real lookups more opening lines would beat the reply, **not yet tested**.
- **Model grading.** Round 4 added a script check and a read-through; rounds 1–3 did not do these systematically.
- **Gateway differences may vary over time.** The same small model differed by about 0.8 s between the two gateways, measured in one time window.
- **Latency was measured on one computer** and indicates direction only.
- **Not yet tested**: a customer interrupting, a specialist starting to serve, and how it looks in the real chat window.

## Conclusion

A small model writing in parallel with the main model is the fastest way to produce an interim reply; to avoid contradicting the reply, it should wait for the main model's first step and know what is being looked up. Under that setting the wait notice reached zero violations of all four kinds and was approved; the opening line needs reasoning to reach the same level, which makes a third of lines too late, so it is on hold.

Further work (all **not yet tested**):

- once the wait notice is wired in, re-evaluate it under real waits (15–20 s);
- re-evaluate the opening line together with the wait notice under real waits, then decide whether to ship it;
- (suggested) measure the main model's real time to first reply on the production gateway to see how much headroom the opening line has.

## Appendix A Parameters

::: table Wait notice: settings as tested in round 3
| Parameter | Value |
| --- | --- |
| Small model | Claude Haiku 5.5, reasoning off |
| Timing | Generated after the main model's first step (about 1.5 s) and graded offline |
| Input | Customer language, the tool and arguments of the main model's first step, the last few messages |
| Prompt constraints | State only what Ailo is doing; no amounts or currencies; no times or speed; no asking the customer to wait; no promised next step |
:::

::: table Wait notice: decided integration design (not live; behaviour under real waits not yet tested)
| Parameter | Value |
| --- | --- |
| Input | Customer language, the tool and arguments of the main model's first step, the last few public messages |
| Trigger | Called once when the main model's first step is a lookup and no reply has arrived after the site's wait (default 15 s) |
| Dropped when | The reply arrives first, the turn is cancelled, a specialist starts serving, or the customer writes again |
| Accounting | Outside the turn's model-call limit; counted in the site's daily budget |
:::

## Appendix B Cost

::: table Cost per round (USD, estimated from tokens and list prices)
| Round | Cost |
| --- | ---: |
| Round 1 (2026-11-06) | about 1.8 |
| Round 2: tiers, caching, contradiction grading (2026-11-06) | about 1.7 |
| Round 3: production settings (2026-11-08) | about 8.3 (mostly grading) |
| Round 4: reasoning and one-line check (2026-11-09) | about 0.4 |
:::
