Recall engine: evaluating knowledge and SOP prefetch before the first model call
Abstract
The recall engine retrieves relevant knowledge passages (and, where applicable, SOP candidates) on the server before the support agent Ailo makes its first model call, and appends them after the customer's message as one already-executed search tool call (knowledge_search) with its result (early rounds used a separate recall tool name); the answering model can still call the existing retrieval tools for more. We evaluated it over several rounds on a fictional support knowledge base for an online game platform (572 articles), with gpt-6-luna as the answering model. Main results: (1) at the recall stage, vectors plus a judge model reached a top-5 hit rate of 90.1% (392 questions) and 92.8% at top 10, against 84.3% for vectors alone at top 5. (2) Placing results in context did not by itself reduce round trips: the model searched again in 81% of turns even when recall had found the answer, and latency rose; once the prompt said to answer directly on a strong match and a supplementary tool (recall_more) was added, the rework rate fell from 97.5% to 18%–23%. (3) The way recall results were written triggered a reasoning block in the answering model; presenting them as an already-executed knowledge_search brought median latency to 7.6 s, 3.0–3.5 s faster than two concurrent control runs (10.6 s and 11.1 s), with equal accuracy (88.3%). (4) In a sweep of pre-placement sizes (210 questions × 3 runs), placing 3 articles with 7 more on request (n3m7) was most accurate (89.2 ± 0.7%), against 83.9 ± 2.7% for the control. (5) In multi-turn dialogues, in the 11-13 sweep and the 11-15 test (where the reference line says read_article can read them back), replacing earlier recalls with references did not lower turn-3 accuracy; adding a "restate the original text when returning to an earlier topic" sentence pointed in opposite directions in two experiments and remains unresolved. (6) With SOPs and knowledge in the same context, misuse rose markedly at 30 SOPs. Accordingly, recall is on by default for new versions, knowledge pre-placement uses n3m7, and SOPs are given in full up to 25. Most experiments ran once, graded blind by Claude Sonnet.
Introduction #
Under the existing retrieval approach (see the knowledge retrieval report), the answering model receives the customer's message, calls knowledge_search itself, reads the results and then answers, usually taking two or more model calls per turn. The recall engine's premise is that the server can retrieve in parallel before the first model call and place the results in the current turn, so the model can usually answer in one call, lowering latency; when the results are not enough, the existing tools remain.
This raises four questions: (RQ1) does a single server-side search using the customer's own words find the right article; (RQ2) once results are placed in context, does the model actually make fewer round trips, and how do latency and accuracy change; (RQ3) how many articles to place up front and how many to offer on request; (RQ4) how to handle earlier recalls in multi-turn dialogues, and whether SOPs and knowledge interfere when given together.
Related approaches #
- Prompt caching: the prompt caches of Anthropic, OpenAI and similar services match only an identical prefix, in the order tools, system prompt, conversation history; everything after the first difference is invalidated. Recall results placed at the end of the system prompt would therefore force the whole conversation history to be recomputed at full price every turn, whereas results placed after the current message leave everything before them cacheable. We therefore wrap recall as a tool call with its result, appended after the customer's message in the current turn.
- Context editing: Anthropic's context editing and the context engineering practices Manus has published replace older tool results with short references when context grows long, to be read back when needed. Section 4.5 evaluates this in multi-turn support dialogues.
- SOP preselection: SOP candidates are preselected by the judge model Jev (see the SOP selection report); the recall engine reuses that path.
Method #
3.1 System #
- Knowledge recall: one search using the customer's words. Two embedding models (
gemini-embedding-2,pplx-embed-v1-4b) take their top 20 each, merged; the judge modeljev-1.14reranks and marks strong and weak matches at 0.9; whole passages are returned. - Placement: recall results are appended after the customer's message as one already-executed
knowledge_searchcall with its result (script round 1 placed results at the end of the system prompt; script round 2 and product-code round 1 used a separaterecalltool name; product-code round 3 switched to the current form, see Section 4.3), without changing the system prompt, so the cache prefix is unchanged. Each turn's recall is stored and replayed verbatim later; earlier turns are never re-searched. - Supplementary tools: first
recall_morefor the next batch of candidates, thenknowledge_searchwith new wording, thenread_articlefor detail. - SOP recall: Jev preselects when the SOP count reaches a threshold. During the evaluations reported here the SOP source in the product code was not yet connected, so all SOP results come from standalone evaluation scripts.
3.2 Datasets #
| Experiment | Date | Questions | Runs |
|---|---|---|---|
| Recall stage (knowledge) | 2026-11-12 | 572-article knowledge base, 392 questions (EN, TH, VI, ID) | 1 |
| Recall stage (SOP) | 2026-11-12 | Existing predictions from the SOP selection study, no cost | — |
| Script round 1 | 2026-11-12 | 109 questions (no no-answer questions sampled) | 1 |
| Script round 2 | 2026-11-12 | 120 questions (20 per scenario, including 20 no-answer) | 1 |
| Product-code rounds 1 and 3 | 2026-11-12 | Same 120 questions | 1; control twice in round 3 |
| Pre-placement sweep | 2026-11-13 | 385 original + 80 new casual questions (retrieval); 210 questions (end to end); 34 three-turn dialogues | 3 (casual set 1) |
| SOP and knowledge together | 2026-11-14 | 10, 30 and 150 SOPs; 6,000 calls in all | Not reported |
| Restate sentence | 2026-11-15 | 120 questions; 30 three-turn dialogues | 2 |
All articles and questions are model-written fiction. The new casual questions include typos, slang and mixed Chinese and English. Three-turn dialogues ask about X, switch to Y, then return to X with "the first thing I asked about…" and ask for another fact from X's article.
3.3 Conditions #
- Tools only (control): nothing placed; the model calls
knowledge_searchitself. - Recall top 5 / top 10 / vectors-only top 5 + tools: script round 1, placement only, prompt unchanged.
- Group A: recall top 5, with the prompt telling the model to answer directly on a strong match.
- Group C: group A plus
recall_more(ranks 6–10). - Group B (lazy catalogue): no recall; titles and one-line summaries stay in the system prompt, and bodies are read on demand.
- Variants A and B (product-code round 3): recall results written as an already-executed
knowledge_search, plus a sentence saying the system has run it; variant B adds "answer only from passages that address what the customer asked". - Pre-placement nXmY: X articles placed, Y more via
recall_more.
3.4 Models and gateways #
The answering model is gpt-6-luna via the test gateway; embedding and judge models go through the production gateway. In a trial run on the morning of 2026-11-12, usage for Claude models through the test gateway showed about 16,000 extra cache-write tokens (source unconfirmed; down to about 1,200 when rechecked that afternoon), and the model called a tool that does not exist, so from that day the answering model was gpt-6-luna; the pre-placement sweep also ran Claude Haiku 5.5 as a reference. Grading was blind by Claude Sonnet: script rounds 1–2 called it through the test gateway; the product-code rounds and later experiments used the team's own Claude Code, with answers shuffled and groups hidden. In product-code round 3, the 720 answers from three runs were pooled by question and regraded under one rubric.
3.5 Measures #
- Hit rate: share of questions whose correct article is in the top k passages.
- Accuracy and wrong-answer rate: as in the knowledge-retrieval report; a no-answer question counts as correct if no answer is invented.
- Rework rate: share of turns where the model called a knowledge tool on top of recall.
- Answered in one call: share of turns where the first model call produced the final reply.
- Latency: median (p50) and 95th percentile (p95) per turn.
- Cost: per turn, estimated from tokens and list prices.
When the control was rerun under identical settings, 3–9 questions flipped between any two runs, so differences of ±2 questions are treated as noise. The sweep reports mean ± standard deviation over three runs.
Results #
4.1 Recall stage (RQ1) #
- Knowledge (392 questions): 90.1% at top 5, 92.8% at top 10, 84.3% for vectors alone at top 5; when the model searched on its own, it found the right article in 90.4% of turns. The casual subset (105 questions) reached only 75.2% at top 5.
- Sweep (11-13): 385 original questions, 80.0% at top 1 and 94.3% at top 10; 80 new casual questions, 86.3% at top 1 and 96.3% at top 10.
- SOPs: Jev's top 3 hit 97.2%–98.7% and top 5 98.7%–100%; top 10 barely improves. Vectors alone were clearly worse (82% at top 10 with 166 SOPs).
- Prefetch latency p50 1.19 s, p95 1.62 s (vectors 0.67 s, judge 0.51 s), about USD 0.46 per thousand.
- "No strong match in the top 5" as a knowledge-gap signal flags 93% of no-answer questions (30) and wrongly flags 6.4% of answerable ones (362). It is logged only, never used to block an answer.
4.2 Does pre-placement reduce round trips? (RQ2) #
| Group | Accuracy | Wrong | Recall hit | Rework | Searched again on a hit | Calls per turn | p50 / p95 (s) | Cost per turn (USD) |
|---|---|---|---|---|---|---|---|---|
| Tools only | 89.9% | 2.8% | — | 98.2% | — | 2.61 | 11.9 / 21.1 | 0.0010 |
| Recall top 5 + tools | 92.7% | 1.8% | 87.2% | 88.1% | 81.1% | 2.19 | 14.8 / 27.4 | 0.0016 |
| Recall top 10 + tools | 89.9% | 5.5% | 88.1% | 92.7% | 87.5% | 2.17 | 13.6 / 22.7 | 0.0017 |
| Vectors-only top 5 + tools | 88.1% | 6.4% | 79.8% | 89.0% | 81.6% | 2.17 | 13.3 / 27.4 | 0.0011 |
With placement alone, the model still searched again in 81% of turns where recall had found the answer, so round trips did not fall; latency rose by about 1.4–2.9 s, and cost rose by about 60% for the top-5 and top-10 groups. Top 5 was 2.8 points above the control, only 5 questions against 2, within noise.
| Group | Accuracy | Rework | Answered in one call | Calls per turn | p50 / p95 (s) | Prompt tokens per turn | Cost per turn (USD) |
|---|---|---|---|---|---|---|---|
| Tools only (control) | 89.2% | 97.5% | 0% | 2.70 | 12.1 / 23.2 | 6,718 | 0.0010 |
| A recall top 5 + answer on strong match | 85.0% | 27.5% | 68% | 1.37 | 11.9 / 24.8 | 4,358 | 0.0011 |
C as A + recall_more |
90.8% | 23.3% | 75% | 1.52 | 10.8 / 25.1 | 5,357 | 0.0011 |
| B lazy catalogue | 75.8% | — | 0% | 2.09 | 12.7 / 27.0 | 47,375 | 0.0008 |
- Group C cut rework to 23.3%, answered 75% of turns in one call and was 1.3 s faster at p50 than the control; the accuracy difference, 4 questions against 2, is within noise.
- Without a supplementary tool, group A fell to 85.0% (1 question against 6, p≈0.12): long-article questions were answered from fragments without reading the full article, or reported as not found. Answering directly must come with a way to get more.
- Group B often picked the wrong article from titles; 21% of answerable questions went unanswered, 45% on long articles.
4.3 Latency in the product code (RQ2) #
Product-code round 1 reproduced group C: accuracy 92.5% (control 89.2%), wrong-answer rate 5.8% for both, rework 18.3%, 1.43 calls per turn and USD 0.00117 per turn (control 0.00127); but p50 was 12.6 s and p95 30.6 s, slower than the control (12.1 / 21.7 s).
Diagnosis: replaying the recorded requests with streaming (10 questions) showed that group C always produced a 4–7-second reasoning block before answering, while the control never reasoned. The recall result was about 5,900 characters, close to the control's roughly 5,200 for the corresponding call, and cutting it to 3 passages or 600 characters per passage did not remove the reasoning. What triggered it was the wording: a separate recall tool name and a long description of the recall. Writing the result as an already-executed knowledge_search, reusing that tool's own description plus one sentence, largely removed the reasoning (variant A reasoned in 0 of 20 probes, variant B in 3 of 20) and cut first-token p50 from 5.6 s to 2.1 s.
| Tools only (two concurrent runs) | Round 1 group C | Variant A | Variant B (adopted) | |
|---|---|---|---|---|
| Accuracy | 88.3% / 88.3% | 90.0% | 87.5% | 88.3% |
| Wrong | 9.2% / 10.0% | 10.0% | 11.7% | 11.7% |
| Rework | 97.5% / 98.3% | 18.3% | 23.3% | 20.8% |
| Answered in one call | 1% / 0% | 76.7% | 70.8% | 72.5% |
| Calls per turn | 2.28 / 2.40 | 1.43 | 1.41 | 1.38 |
| p50 / p95 (s) | 10.6 / 21.7; 11.1 / 23.5 | 12.6 / 30.6 | 7.2 / 16.2 | 7.6 / 16.6 |
| Cost per turn (USD) | 0.00129 / 0.00137 | 0.00117 | 0.00111 | 0.00112 |
Variant B was 3.0–3.5 s faster at p50 and 5.1–6.9 s faster at p95 than the two control runs, with the same accuracy (88.3%), 2 questions below round 1's group C (within noise), 20.8% rework and lower cost. Graded separately, the grader treated "the reply followed a different article that conflicts with the reference" inconsistently, moving 3–8 questions on the same set, so round 3 used a pooled regrade. Of the 20 no-answer questions, the recall arm got 3 wrong, each time treating a nearby related article as the answer.
4.4 Pre-placement size (RQ3) #
The pre-placement sweep was run by the team on 2026-11-13 with the product code and gpt-6-luna as the answering model; all 3,264 answers were shuffled and graded blind by Claude Code Sonnet.
| Arm | Accuracy | Wrong | Calls per turn | p50 (s) |
|---|---|---|---|---|
| Control: nothing placed, model searches | 83.9 ± 2.7 | 12.2 | 2.66 | 9.7 |
| n3m5: 3 placed + 5 more | 86.4 ± 4.0 | 9.4 | 1.61 | 6.7 |
| n3m7: 3 placed + 7 more | 89.2 ± 0.7 | 9.7 | 1.67 | 6.7 |
| 5 + 5 | 86.7 ± 0.7 | 11.7 | 1.65 | 6.7 |
| 8 + 5 | 86.7 ± 0.7 | 11.4 | 1.59 | 6.7 |
| 5 + 5 with "say so if you can't find it" | 87.8 ± 1.7 | 10.3 | 1.59 | 6.1 |
| Claude Haiku 5.5: control / 5 + 5 | 76.4 / 76.1 | 16.9 / 19.2 | 2.24 / 1.36 | 14.9 / 7.2 |
Pre-placement had higher accuracy than letting the model search, cut p50 by about 30%, and answered nearly 60% of turns in one call. n3m7 had the highest accuracy and the smallest spread. Claude Haiku 5.5 as the answering model was clearly weaker and suits only a fallback role. A few questions were graded wrong in every arm because the reference article conflicts with another article and every answer followed the other one, a conflict in the knowledge base itself; a small number of answers were empty or placeholders in every arm, presumably the test script missing the final reply.
The sweep also compared ways of offering tools (answering model gpt-6-luna): Jev preselecting tools 86.7%, listing all tools 69.2%, offering only the search tool 6.7% (the model hardly searched), and the search tool plus a list of tool names 36.3%; Claude Haiku 5.5 was worse in every setting. The question count for this part is not given in the summary.
4.5 Earlier recalls in multi-turn dialogues (RQ4) #
| Experiment | Setting | Turn-3 accuracy |
|---|---|---|
| Script round 2 (30 dialogues, 1 run) | Keep all | 83.3% |
| Drop each turn | 60.0% | |
| Forced to references at turn 3 (keep the latest 1) | 53.3% | |
| Pre-placement sweep (34 dialogues) | Keep all | 92.7% |
| References | 94.1% | |
| References + restate sentence | 95.6% | |
| Restate test (30 dialogues × 2 runs) | References + restate sentence (A) | 86.7% / 86.7% |
| References, no restate sentence (B) | 96.7% / 96.7% |
- In script round 2, after forced replacement the model read the original back by id 0 times in 30, and that turn's cache-read share was only 0.07. In the restate test, the reference line states that
read_articlecan read it back, and the model read the original back in 57%–63% of third turns. - In the sweep, adding the restate sentence raised the restating rate from 9% to 81% and moved accuracy from 94.1% to 95.6%; with 34 dialogues, the 1.5-point gap (about half a dialogue) is within noise.
- The restate test used a sentence we wrote ourselves (the sweep's original was not yet available) and found the opposite: A with the sentence 52/60, B without it 58/60, p≈0.09. A's errors were mostly answering from memory without reading back (wrong figures or steps) or reporting after several searches that it could not confirm; B answered the same questions correctly after reading back. We conjecture, without verification, that the sentence prompts the model to restate what it remembers rather than read first. On 120 single-turn questions the two were equally accurate (A 92.5% / 90.0%, B 90.8% / 92.5%), and A's p50 was about 0.3 s slower, within the 0.5 s by which A's own two runs differed.
The two restate experiments used different sentences and different questions, and possibly different reference settings, so they cannot be pooled. The timeline: on 2026-11-14, based on the sweep, "references + restate sentence" was set as the default and merged into the product code with our own wording; the 11-15 test pointed the other way, and removing the sentence has been proposed and awaits confirmation; the final decision should come from comparing the sweep's original wording against no sentence on the same 30 three-turn dialogues.
4.6 SOPs and knowledge together (RQ4) #
| SOPs | Method | Right SOP when one applies | Used one when none fits | Knowledge question misrouted |
|---|---|---|---|---|
| 10 | Full list | 85.7% | 6.1% | 2.6% |
| 10 | Jev top 3 | 84.9% | 8.4% | 3.3% |
| 30 | Full list | 79.0% | 28.7% | 8.8% |
| 30 | Jev top 3 | 78.4% | 37.3% | 14.3% |
| 150 | Jev top 3 | 81.5% | 27.6% | 39.0% |
- Up to 30 SOPs, the full list had accuracy comparable to Jev's top 3 with less misuse, supporting "full list up to 25". In the sweep, the full list scored 90% with 10 SOPs and 88% with 25, against 77% for Jev preselection at 25 (question counts not given in the summary), consistent with this.
- Beyond 25, with Jev's top 3, the model often used an SOP even when none of the candidates fit: with 150 SOPs, 39.0% of knowledge questions were misrouted. The right SOP was in Jev's top 3 in 100% (10), 97% (30) and 92% (150) of cases, so errors arise mainly when the model chooses among the candidates.
- For answerable knowledge questions, the answer article was in the top 3 in 82.7% of cases.
Script round 2 also tested a lazy catalogue with 100 SOPs (186 synthetic questions): when the main model read the catalogue directly, it used an SOP on questions that needed none 31.7%–52.4% of the time, against 3.7%–4.9% with Jev preselecting the same questions; every layout selected correctly 96%–100% of the time. A resident SOP catalogue was therefore not adopted.
Discussion #
Round trips depend on the prompt, not on placement. With results merely placed in context, the model still searched again almost every turn; only once the prompt said to answer on a strong match and a supplementary tool was offered did rework fall sharply. Without the supplementary tool accuracy dropped, showing that the placed passages are not always enough.
Latency depends on how results are written. In product-code round 1 the recall arm made fewer calls but was slower, because a new tool name and a long description triggered reasoning. Writing the results as a familiar search-tool result largely removed it. When changing context structure, check the model's reasoning behaviour, not just token length.
Pre-placement size. Placing 3 with 7 more on request was slightly better than placing more, and the most stable. Placing more did not raise accuracy, perhaps because more irrelevant passages enter the context. Differences between pre-placement arms are mostly 1–3 points, close to the run-to-run standard deviation, so the ordering is only moderately stable.
Interference between SOPs and knowledge. With few SOPs the full list suffices; with many, Jev's preselection is accurate enough, and the problem is how the answering model handles the candidates. Before the preselection tier ships, it needs safeguards against misuse (such as telling the model not to use an SOP when none fits, or a threshold on Jev's "none applies"), tested first.
Decision. Recall is on by default for versions newly switched to rag_marked-type retrieval, and sites can turn it off; versions switched to rag_marked earlier stay off, with behaviour unchanged. Knowledge pre-placement uses n3m7; earlier recalls in multi-turn dialogues are replaced with references; SOPs are given in full up to 25 with Jev preselecting beyond that (decided, but the SOP source is not yet connected in the product code, and the preselection tier must be tested for misuse before launch). The restate sentence is currently in the default with our own wording; removing it has been proposed and awaits confirmation, and the final decision follows a comparison with the original wording.
Limitations #
- Few runs. Script rounds and product-code round 1 ran once; identical reruns flip 3–9 questions between any two runs, so differences within ±2 questions cannot be read as effects.
- Synthetic data. Articles and questions are model-written; conflicting articles in the knowledge base make some questions wrong in every arm.
- Grading. One Sonnet grader, one pass; separate gradings were inconsistent, hence the pooled regrade in round 3. Borderline questions may shift results by a few points.
- No cross-experiment comparison. Experiments differ in questions, grading batches and runs; compare only within an experiment.
- SOPs not in the product code. SOP results come from evaluation scripts. In the joint SOP and knowledge test the knowledge block was appended to the user message rather than returned by the recall tool; 5%–12% of turns failed to call a tool when they should have and count as "no SOP"; excluding them moves each figure by 1–5 points without changing the conclusions.
- No-answer questions. The recall arm still treats a nearby related article as the answer on some no-answer questions (3 of 20 in round 3); the knowledge-gap log after launch should be watched.
- One answering model. Main results come from
gpt-6-luna; Claude Haiku 5.5 appears only as a reference in the sweep.
Conclusion #
Prefetching knowledge before the first model call, combined with a prompt that answers directly on a strong match and the recall_more tool, cuts rework from about 98% to about 20% without lowering accuracy, and with the right presentation shortens median latency by 3.0–3.5 s. Placing 3 articles with 7 more on request was the most accurate and stable configuration in the sweep. For SOPs, the full list up to 25 beats preselection; beyond that, preselection needs misuse safeguards before launch.
Further work (all not yet tested):
- compare the sweep's original restate sentence against no sentence on the same three-turn dialogues, then decide;
- add and evaluate misuse safeguards for the SOP preselection tier;
- re-measure recall hit rate and accuracy on a real knowledge base and real conversations;
- monitor the knowledge-gap log after launch to assess wrong answers on no-answer questions.
Appendix A Parameters #
| Parameter | Value |
|---|---|
| Switch | On by default for versions newly switched to rag_marked / rag_kw; earlier versions stay off; sites can turn it off |
| Knowledge placed | 3 articles (whole passages, marked strong or weak) |
recall_more |
7 more articles |
| Retrieval | gemini-embedding-2 and pplx-embed-v1-4b, top 20 each, merged; jev-1.14 reranks; strong-match cut-off 0.9 |
| Placement | After the customer's message, as one tool call with its result; stored per turn and replayed verbatim |
| Earlier recalls | Replaced in a batch with one-line references once past a threshold (article passages are usually replaced from the next turn); readable again with read_article |
| SOPs | Decided: full list up to 25, Jev preselects beyond that; SOP source not yet connected in the product code; the preselection tier needs misuse testing before launch |
| Restate sentence | Current default includes our own wording; removal proposed, awaiting confirmation |
| Knowledge gap | Flagged in the run log when recall has no strong match |
Appendix B Cost #
| Experiment | Answering and grading models | Embedding and judge models |
|---|---|---|
| Recall stage and script round 1 (2026-11-12) | about 3.49 | about 0.53 |
| Script round 2 (2026-11-12) | about 5.36 | about 0.34 |
| Product-code rounds 1–3 (2026-11-12) | about 0.55 | about 0.56 |
| Pre-placement sweep (2026-11-13) | 4.50 | 3.73 |
| SOP and knowledge together (2026-11-14) | 2.57 | Not reported |
| Restate test (2026-11-15) | about 0.60 | about 0.74 |
From the product-code rounds of 2026-11-12 onward, grading used the team's own Claude Code and is not included above.