Research

Recall engine: evaluating knowledge and SOP prefetch before the first model call

Product
Harness
Date
2026-11-15
Authors
Wilson
Version
v1.0
Status
Synthetic-data evaluation

Abstract

The recall engine retrieves relevant knowledge passages (and, where applicable, SOP candidates) on the server before the support agent Ailo makes its first model call, and appends them after the customer's message as one already-executed search tool call (knowledge_search) with its result (early rounds used a separate recall tool name); the answering model can still call the existing retrieval tools for more. We evaluated it over several rounds on a fictional support knowledge base for an online game platform (572 articles), with gpt-6-luna as the answering model. Main results: (1) at the recall stage, vectors plus a judge model reached a top-5 hit rate of 90.1% (392 questions) and 92.8% at top 10, against 84.3% for vectors alone at top 5. (2) Placing results in context did not by itself reduce round trips: the model searched again in 81% of turns even when recall had found the answer, and latency rose; once the prompt said to answer directly on a strong match and a supplementary tool (recall_more) was added, the rework rate fell from 97.5% to 18%–23%. (3) The way recall results were written triggered a reasoning block in the answering model; presenting them as an already-executed knowledge_search brought median latency to 7.6 s, 3.0–3.5 s faster than two concurrent control runs (10.6 s and 11.1 s), with equal accuracy (88.3%). (4) In a sweep of pre-placement sizes (210 questions × 3 runs), placing 3 articles with 7 more on request (n3m7) was most accurate (89.2 ± 0.7%), against 83.9 ± 2.7% for the control. (5) In multi-turn dialogues, in the 11-13 sweep and the 11-15 test (where the reference line says read_article can read them back), replacing earlier recalls with references did not lower turn-3 accuracy; adding a "restate the original text when returning to an earlier topic" sentence pointed in opposite directions in two experiments and remains unresolved. (6) With SOPs and knowledge in the same context, misuse rose markedly at 30 SOPs. Accordingly, recall is on by default for new versions, knowledge pre-placement uses n3m7, and SOPs are given in full up to 25. Most experiments ran once, graded blind by Claude Sonnet.

Introduction #

Under the existing retrieval approach (see the knowledge retrieval report), the answering model receives the customer's message, calls knowledge_search itself, reads the results and then answers, usually taking two or more model calls per turn. The recall engine's premise is that the server can retrieve in parallel before the first model call and place the results in the current turn, so the model can usually answer in one call, lowering latency; when the results are not enough, the existing tools remain.

This raises four questions: (RQ1) does a single server-side search using the customer's own words find the right article; (RQ2) once results are placed in context, does the model actually make fewer round trips, and how do latency and accuracy change; (RQ3) how many articles to place up front and how many to offer on request; (RQ4) how to handle earlier recalls in multi-turn dialogues, and whether SOPs and knowledge interfere when given together.

  • Prompt caching: the prompt caches of Anthropic, OpenAI and similar services match only an identical prefix, in the order tools, system prompt, conversation history; everything after the first difference is invalidated. Recall results placed at the end of the system prompt would therefore force the whole conversation history to be recomputed at full price every turn, whereas results placed after the current message leave everything before them cacheable. We therefore wrap recall as a tool call with its result, appended after the customer's message in the current turn.
  • Context editing: Anthropic's context editing and the context engineering practices Manus has published replace older tool results with short references when context grows long, to be read back when needed. Section 4.5 evaluates this in multi-turn support dialogues.
  • SOP preselection: SOP candidates are preselected by the judge model Jev (see the SOP selection report); the recall engine reuses that path.

Method #

3.1 System #

  • Knowledge recall: one search using the customer's words. Two embedding models (gemini-embedding-2, pplx-embed-v1-4b) take their top 20 each, merged; the judge model jev-1.14 reranks and marks strong and weak matches at 0.9; whole passages are returned.
  • Placement: recall results are appended after the customer's message as one already-executed knowledge_search call with its result (script round 1 placed results at the end of the system prompt; script round 2 and product-code round 1 used a separate recall tool name; product-code round 3 switched to the current form, see Section 4.3), without changing the system prompt, so the cache prefix is unchanged. Each turn's recall is stored and replayed verbatim later; earlier turns are never re-searched.
  • Supplementary tools: first recall_more for the next batch of candidates, then knowledge_search with new wording, then read_article for detail.
  • SOP recall: Jev preselects when the SOP count reaches a threshold. During the evaluations reported here the SOP source in the product code was not yet connected, so all SOP results come from standalone evaluation scripts.

3.2 Datasets #

Data per experiment
Experiment Date Questions Runs
Recall stage (knowledge) 2026-11-12 572-article knowledge base, 392 questions (EN, TH, VI, ID) 1
Recall stage (SOP) 2026-11-12 Existing predictions from the SOP selection study, no cost —
Script round 1 2026-11-12 109 questions (no no-answer questions sampled) 1
Script round 2 2026-11-12 120 questions (20 per scenario, including 20 no-answer) 1
Product-code rounds 1 and 3 2026-11-12 Same 120 questions 1; control twice in round 3
Pre-placement sweep 2026-11-13 385 original + 80 new casual questions (retrieval); 210 questions (end to end); 34 three-turn dialogues 3 (casual set 1)
SOP and knowledge together 2026-11-14 10, 30 and 150 SOPs; 6,000 calls in all Not reported
Restate sentence 2026-11-15 120 questions; 30 three-turn dialogues 2

All articles and questions are model-written fiction. The new casual questions include typos, slang and mixed Chinese and English. Three-turn dialogues ask about X, switch to Y, then return to X with "the first thing I asked about…" and ask for another fact from X's article.

3.3 Conditions #

  • Tools only (control): nothing placed; the model calls knowledge_search itself.
  • Recall top 5 / top 10 / vectors-only top 5 + tools: script round 1, placement only, prompt unchanged.
  • Group A: recall top 5, with the prompt telling the model to answer directly on a strong match.
  • Group C: group A plus recall_more (ranks 6–10).
  • Group B (lazy catalogue): no recall; titles and one-line summaries stay in the system prompt, and bodies are read on demand.
  • Variants A and B (product-code round 3): recall results written as an already-executed knowledge_search, plus a sentence saying the system has run it; variant B adds "answer only from passages that address what the customer asked".
  • Pre-placement nXmY: X articles placed, Y more via recall_more.

3.4 Models and gateways #

The answering model is gpt-6-luna via the test gateway; embedding and judge models go through the production gateway. In a trial run on the morning of 2026-11-12, usage for Claude models through the test gateway showed about 16,000 extra cache-write tokens (source unconfirmed; down to about 1,200 when rechecked that afternoon), and the model called a tool that does not exist, so from that day the answering model was gpt-6-luna; the pre-placement sweep also ran Claude Haiku 5.5 as a reference. Grading was blind by Claude Sonnet: script rounds 1–2 called it through the test gateway; the product-code rounds and later experiments used the team's own Claude Code, with answers shuffled and groups hidden. In product-code round 3, the 720 answers from three runs were pooled by question and regraded under one rubric.

3.5 Measures #

  • Hit rate: share of questions whose correct article is in the top k passages.
  • Accuracy and wrong-answer rate: as in the knowledge-retrieval report; a no-answer question counts as correct if no answer is invented.
  • Rework rate: share of turns where the model called a knowledge tool on top of recall.
  • Answered in one call: share of turns where the first model call produced the final reply.
  • Latency: median (p50) and 95th percentile (p95) per turn.
  • Cost: per turn, estimated from tokens and list prices.

When the control was rerun under identical settings, 3–9 questions flipped between any two runs, so differences of ±2 questions are treated as noise. The sweep reports mean ± standard deviation over three runs.

Results #

4.1 Recall stage (RQ1) #

Recall-stage hit rateRecall-stage hit rate
Vectors plus a judge model reach about 90% at top 5, comparable to the whole-turn hit rate when the model searches on its own
  • Knowledge (392 questions): 90.1% at top 5, 92.8% at top 10, 84.3% for vectors alone at top 5; when the model searched on its own, it found the right article in 90.4% of turns. The casual subset (105 questions) reached only 75.2% at top 5.
  • Sweep (11-13): 385 original questions, 80.0% at top 1 and 94.3% at top 10; 80 new casual questions, 86.3% at top 1 and 96.3% at top 10.
  • SOPs: Jev's top 3 hit 97.2%–98.7% and top 5 98.7%–100%; top 10 barely improves. Vectors alone were clearly worse (82% at top 10 with 166 SOPs).
  • Prefetch latency p50 1.19 s, p95 1.62 s (vectors 0.67 s, judge 0.51 s), about USD 0.46 per thousand.
  • "No strong match in the top 5" as a knowledge-gap signal flags 93% of no-answer questions (30) and wrongly flags 6.4% of answerable ones (362). It is logged only, never used to block an answer.

4.2 Does pre-placement reduce round trips? (RQ2) #

Script round 1 (109 questions, placement only, prompt unchanged)
Group Accuracy Wrong Recall hit Rework Searched again on a hit Calls per turn p50 / p95 (s) Cost per turn (USD)
Tools only 89.9% 2.8% — 98.2% — 2.61 11.9 / 21.1 0.0010
Recall top 5 + tools 92.7% 1.8% 87.2% 88.1% 81.1% 2.19 14.8 / 27.4 0.0016
Recall top 10 + tools 89.9% 5.5% 88.1% 92.7% 87.5% 2.17 13.6 / 22.7 0.0017
Vectors-only top 5 + tools 88.1% 6.4% 79.8% 89.0% 81.6% 2.17 13.3 / 27.4 0.0011

With placement alone, the model still searched again in 81% of turns where recall had found the answer, so round trips did not fall; latency rose by about 1.4–2.9 s, and cost rose by about 60% for the top-5 and top-10 groups. Top 5 was 2.8 points above the control, only 5 questions against 2, within noise.

Script round 2 (120 questions, recall placed in the current turn)
Group Accuracy Rework Answered in one call Calls per turn p50 / p95 (s) Prompt tokens per turn Cost per turn (USD)
Tools only (control) 89.2% 97.5% 0% 2.70 12.1 / 23.2 6,718 0.0010
A recall top 5 + answer on strong match 85.0% 27.5% 68% 1.37 11.9 / 24.8 4,358 0.0011
C as A + recall_more 90.8% 23.3% 75% 1.52 10.8 / 25.1 5,357 0.0011
B lazy catalogue 75.8% — 0% 2.09 12.7 / 27.0 47,375 0.0008
  • Group C cut rework to 23.3%, answered 75% of turns in one call and was 1.3 s faster at p50 than the control; the accuracy difference, 4 questions against 2, is within noise.
  • Without a supplementary tool, group A fell to 85.0% (1 question against 6, p≈0.12): long-article questions were answered from fragments without reading the full article, or reported as not found. Answering directly must come with a way to get more.
  • Group B often picked the wrong article from titles; 21% of answerable questions went unanswered, 45% on long articles.
Rework rate across roundsRework rate across rounds
Once the prompt said to answer directly on a strong match, rework fell from about 90% to about 20%

4.3 Latency in the product code (RQ2) #

Product-code round 1 reproduced group C: accuracy 92.5% (control 89.2%), wrong-answer rate 5.8% for both, rework 18.3%, 1.43 calls per turn and USD 0.00117 per turn (control 0.00127); but p50 was 12.6 s and p95 30.6 s, slower than the control (12.1 / 21.7 s).

Diagnosis: replaying the recorded requests with streaming (10 questions) showed that group C always produced a 4–7-second reasoning block before answering, while the control never reasoned. The recall result was about 5,900 characters, close to the control's roughly 5,200 for the corresponding call, and cutting it to 3 passages or 600 characters per passage did not remove the reasoning. What triggered it was the wording: a separate recall tool name and a long description of the recall. Writing the result as an already-executed knowledge_search, reusing that tool's own description plus one sentence, largely removed the reasoning (variant A reasoned in 0 of 20 probes, variant B in 3 of 20) and cut first-token p50 from 5.6 s to 2.1 s.

Product-code round 3 (120 questions, pooled regrade)
Tools only (two concurrent runs) Round 1 group C Variant A Variant B (adopted)
Accuracy 88.3% / 88.3% 90.0% 87.5% 88.3%
Wrong 9.2% / 10.0% 10.0% 11.7% 11.7%
Rework 97.5% / 98.3% 18.3% 23.3% 20.8%
Answered in one call 1% / 0% 76.7% 70.8% 72.5%
Calls per turn 2.28 / 2.40 1.43 1.41 1.38
p50 / p95 (s) 10.6 / 21.7; 11.1 / 23.5 12.6 / 30.6 7.2 / 16.2 7.6 / 16.6
Cost per turn (USD) 0.00129 / 0.00137 0.00117 0.00111 0.00112
Product round 3: seconds per turnProduct round 3: seconds per turn
After rewriting how recall results are presented, p50 is 3.0–3.5 s faster than the two control runs and p95 5.1–6.9 s faster
Median seconds per turn across roundsMedian seconds per turn across rounds
Recall's median latency started above the control and fell below it after the prompt and wording changes

Variant B was 3.0–3.5 s faster at p50 and 5.1–6.9 s faster at p95 than the two control runs, with the same accuracy (88.3%), 2 questions below round 1's group C (within noise), 20.8% rework and lower cost. Graded separately, the grader treated "the reply followed a different article that conflicts with the reference" inconsistently, moving 3–8 questions on the same set, so round 3 used a pooled regrade. Of the 20 no-answer questions, the recall arm got 3 wrong, each time treating a nearby related article as the answer.

4.4 Pre-placement size (RQ3) #

The pre-placement sweep was run by the team on 2026-11-13 with the product code and gpt-6-luna as the answering model; all 3,264 answers were shuffled and graded blind by Claude Code Sonnet.

End-to-end results (210 questions, 3 runs per arm, mean ± SD)
Arm Accuracy Wrong Calls per turn p50 (s)
Control: nothing placed, model searches 83.9 ± 2.7 12.2 2.66 9.7
n3m5: 3 placed + 5 more 86.4 ± 4.0 9.4 1.61 6.7
n3m7: 3 placed + 7 more 89.2 ± 0.7 9.7 1.67 6.7
5 + 5 86.7 ± 0.7 11.7 1.65 6.7
8 + 5 86.7 ± 0.7 11.4 1.59 6.7
5 + 5 with "say so if you can't find it" 87.8 ± 1.7 10.3 1.59 6.1
Claude Haiku 5.5: control / 5 + 5 76.4 / 76.1 16.9 / 19.2 2.24 / 1.36 14.9 / 7.2
Pre-placement sweepPre-placement sweep
n3m7 is most accurate with the least variation; every pre-placement arm is about 30% faster than the control

Pre-placement had higher accuracy than letting the model search, cut p50 by about 30%, and answered nearly 60% of turns in one call. n3m7 had the highest accuracy and the smallest spread. Claude Haiku 5.5 as the answering model was clearly weaker and suits only a fallback role. A few questions were graded wrong in every arm because the reference article conflicts with another article and every answer followed the other one, a conflict in the knowledge base itself; a small number of answers were empty or placeholders in every arm, presumably the test script missing the final reply.

The sweep also compared ways of offering tools (answering model gpt-6-luna): Jev preselecting tools 86.7%, listing all tools 69.2%, offering only the search tool 6.7% (the model hardly searched), and the search tool plus a list of tool names 36.3%; Claude Haiku 5.5 was worse in every setting. The question count for this part is not given in the summary.

4.5 Earlier recalls in multi-turn dialogues (RQ4) #

Turn-3 accuracy in three-turn dialoguesTurn-3 accuracy in three-turn dialogues
When the reference line says the original can be read back, replacing earlier recalls with references does not lower accuracy; the restate sentence pointed in opposite directions in two experiments
Turn-3 accuracy in three-turn dialogues
Experiment Setting Turn-3 accuracy
Script round 2 (30 dialogues, 1 run) Keep all 83.3%
Drop each turn 60.0%
Forced to references at turn 3 (keep the latest 1) 53.3%
Pre-placement sweep (34 dialogues) Keep all 92.7%
References 94.1%
References + restate sentence 95.6%
Restate test (30 dialogues × 2 runs) References + restate sentence (A) 86.7% / 86.7%
References, no restate sentence (B) 96.7% / 96.7%
  • In script round 2, after forced replacement the model read the original back by id 0 times in 30, and that turn's cache-read share was only 0.07. In the restate test, the reference line states that read_article can read it back, and the model read the original back in 57%–63% of third turns.
  • In the sweep, adding the restate sentence raised the restating rate from 9% to 81% and moved accuracy from 94.1% to 95.6%; with 34 dialogues, the 1.5-point gap (about half a dialogue) is within noise.
  • The restate test used a sentence we wrote ourselves (the sweep's original was not yet available) and found the opposite: A with the sentence 52/60, B without it 58/60, p≈0.09. A's errors were mostly answering from memory without reading back (wrong figures or steps) or reporting after several searches that it could not confirm; B answered the same questions correctly after reading back. We conjecture, without verification, that the sentence prompts the model to restate what it remembers rather than read first. On 120 single-turn questions the two were equally accurate (A 92.5% / 90.0%, B 90.8% / 92.5%), and A's p50 was about 0.3 s slower, within the 0.5 s by which A's own two runs differed.

The two restate experiments used different sentences and different questions, and possibly different reference settings, so they cannot be pooled. The timeline: on 2026-11-14, based on the sweep, "references + restate sentence" was set as the default and merged into the product code with our own wording; the 11-15 test pointed the other way, and removing the sentence has been proposed and awaits confirmation; the final decision should come from comparing the sweep's original wording against no sentence on the same 30 three-turn dialogues.

4.6 SOPs and knowledge together (RQ4) #

SOPs and knowledge in the same contextSOPs and knowledge in the same context
Beyond 30 SOPs, using an SOP when none fits and misrouting knowledge questions rise sharply
Joint SOP and knowledge test (2026-11-14, answering model gpt-6-luna)
SOPs Method Right SOP when one applies Used one when none fits Knowledge question misrouted
10 Full list 85.7% 6.1% 2.6%
10 Jev top 3 84.9% 8.4% 3.3%
30 Full list 79.0% 28.7% 8.8%
30 Jev top 3 78.4% 37.3% 14.3%
150 Jev top 3 81.5% 27.6% 39.0%
  • Up to 30 SOPs, the full list had accuracy comparable to Jev's top 3 with less misuse, supporting "full list up to 25". In the sweep, the full list scored 90% with 10 SOPs and 88% with 25, against 77% for Jev preselection at 25 (question counts not given in the summary), consistent with this.
  • Beyond 25, with Jev's top 3, the model often used an SOP even when none of the candidates fit: with 150 SOPs, 39.0% of knowledge questions were misrouted. The right SOP was in Jev's top 3 in 100% (10), 97% (30) and 92% (150) of cases, so errors arise mainly when the model chooses among the candidates.
  • For answerable knowledge questions, the answer article was in the top 3 in 82.7% of cases.
False-trigger rate with 100 SOPsFalse-trigger rate with 100 SOPs
When the main model reads the catalogue directly, false triggers run at 32%–52%; with Jev preselecting, 3.7%–4.9%

Script round 2 also tested a lazy catalogue with 100 SOPs (186 synthetic questions): when the main model read the catalogue directly, it used an SOP on questions that needed none 31.7%–52.4% of the time, against 3.7%–4.9% with Jev preselecting the same questions; every layout selected correctly 96%–100% of the time. A resident SOP catalogue was therefore not adopted.

Discussion #

Round trips depend on the prompt, not on placement. With results merely placed in context, the model still searched again almost every turn; only once the prompt said to answer on a strong match and a supplementary tool was offered did rework fall sharply. Without the supplementary tool accuracy dropped, showing that the placed passages are not always enough.

Latency depends on how results are written. In product-code round 1 the recall arm made fewer calls but was slower, because a new tool name and a long description triggered reasoning. Writing the results as a familiar search-tool result largely removed it. When changing context structure, check the model's reasoning behaviour, not just token length.

Pre-placement size. Placing 3 with 7 more on request was slightly better than placing more, and the most stable. Placing more did not raise accuracy, perhaps because more irrelevant passages enter the context. Differences between pre-placement arms are mostly 1–3 points, close to the run-to-run standard deviation, so the ordering is only moderately stable.

Interference between SOPs and knowledge. With few SOPs the full list suffices; with many, Jev's preselection is accurate enough, and the problem is how the answering model handles the candidates. Before the preselection tier ships, it needs safeguards against misuse (such as telling the model not to use an SOP when none fits, or a threshold on Jev's "none applies"), tested first.

Decision. Recall is on by default for versions newly switched to rag_marked-type retrieval, and sites can turn it off; versions switched to rag_marked earlier stay off, with behaviour unchanged. Knowledge pre-placement uses n3m7; earlier recalls in multi-turn dialogues are replaced with references; SOPs are given in full up to 25 with Jev preselecting beyond that (decided, but the SOP source is not yet connected in the product code, and the preselection tier must be tested for misuse before launch). The restate sentence is currently in the default with our own wording; removing it has been proposed and awaits confirmation, and the final decision follows a comparison with the original wording.

Limitations #

  • Few runs. Script rounds and product-code round 1 ran once; identical reruns flip 3–9 questions between any two runs, so differences within ±2 questions cannot be read as effects.
  • Synthetic data. Articles and questions are model-written; conflicting articles in the knowledge base make some questions wrong in every arm.
  • Grading. One Sonnet grader, one pass; separate gradings were inconsistent, hence the pooled regrade in round 3. Borderline questions may shift results by a few points.
  • No cross-experiment comparison. Experiments differ in questions, grading batches and runs; compare only within an experiment.
  • SOPs not in the product code. SOP results come from evaluation scripts. In the joint SOP and knowledge test the knowledge block was appended to the user message rather than returned by the recall tool; 5%–12% of turns failed to call a tool when they should have and count as "no SOP"; excluding them moves each figure by 1–5 points without changing the conclusions.
  • No-answer questions. The recall arm still treats a nearby related article as the answer on some no-answer questions (3 of 20 in round 3); the knowledge-gap log after launch should be watched.
  • One answering model. Main results come from gpt-6-luna; Claude Haiku 5.5 appears only as a reference in the sweep.

Conclusion #

Prefetching knowledge before the first model call, combined with a prompt that answers directly on a strong match and the recall_more tool, cuts rework from about 98% to about 20% without lowering accuracy, and with the right presentation shortens median latency by 3.0–3.5 s. Placing 3 articles with 7 more on request was the most accurate and stable configuration in the sweep. For SOPs, the full list up to 25 beats preselection; beyond that, preselection needs misuse safeguards before launch.

Further work (all not yet tested):

  • compare the sweep's original restate sentence against no sentence on the same three-turn dialogues, then decide;
  • add and evaluate misuse safeguards for the SOP preselection tier;
  • re-measure recall hit rate and accuracy on a real knowledge base and real conversations;
  • monitor the knowledge-gap log after launch to assess wrong answers on no-answer questions.

Appendix A Parameters #

Recall engine defaults (as of 2026-11-15)
Parameter Value
Switch On by default for versions newly switched to rag_marked / rag_kw; earlier versions stay off; sites can turn it off
Knowledge placed 3 articles (whole passages, marked strong or weak)
recall_more 7 more articles
Retrieval gemini-embedding-2 and pplx-embed-v1-4b, top 20 each, merged; jev-1.14 reranks; strong-match cut-off 0.9
Placement After the customer's message, as one tool call with its result; stored per turn and replayed verbatim
Earlier recalls Replaced in a batch with one-line references once past a threshold (article passages are usually replaced from the next turn); readable again with read_article
SOPs Decided: full list up to 25, Jev preselects beyond that; SOP source not yet connected in the product code; the preselection tier needs misuse testing before launch
Restate sentence Current default includes our own wording; removal proposed, awaiting confirmation
Knowledge gap Flagged in the run log when recall has no strong match

Appendix B Cost #

Cost per experiment (USD, estimated from tokens and list prices)
Experiment Answering and grading models Embedding and judge models
Recall stage and script round 1 (2026-11-12) about 3.49 about 0.53
Script round 2 (2026-11-12) about 5.36 about 0.34
Product-code rounds 1–3 (2026-11-12) about 0.55 about 0.56
Pre-placement sweep (2026-11-13) 4.50 3.73
SOP and knowledge together (2026-11-14) 2.57 Not reported
Restate test (2026-11-15) about 0.60 about 0.74

From the product-code rounds of 2026-11-12 onward, grading used the team's own Claude Code and is not included above.