# Knowledge retrieval for a support agent: semantic search, full-context reading and variants compared

::: abstract
We study how the support agent Ailo obtains evidence from a knowledge base before it answers. On a fictional support knowledge base for an online game platform (572 articles; questions in English, Thai, Vietnamese and Indonesian) we ran three rounds of evaluation. Round 1 evaluated retrieval alone, comparing keywords, generated questions, three embedding models and their combinations. Round 2 compared eight end-to-end retrieval approaches on 392 scored questions. Round 3 compared semantic search with full-context reading (placing the whole knowledge base in the prompt) on a 50-article subset. Main findings: (1) across the four question languages, a single embedding model (`gemini-embedding-2`) achieved a top-5 hit rate of 83.8%–91.2%, against 21.2%–60.0% for keywords; (2) returning whole passages instead of the first 400 characters raised accuracy on long-article details from 51.4% to 91.9%; (3) having a judge model mark strong and weak matches, with the answering model relying on strong matches only (`rag_marked`), gave the highest round 2 score (83.7%; 85.2% on a repeat run); (4) on the 50-article subset, semantic search scored 91.8% against 86.7% for full-context reading, which invented fewer answers (1.9% vs 5.1%) and was faster (median 3.5 s vs 8.6 s) but left more answerable questions unanswered (13.3% vs 3.9%). That comparison ran once, and a paired sign test gives p≈0.17, so chance cannot be ruled out. Customers are therefore offered semantic search only; full-context reading is kept as an internal comparison switch.
:::

## Introduction

Knowledge retrieval for customer support has two competing requirements: find and use the answer whenever it exists, and decline to invent one when it does not. For a customer, a wrong answer costs more than an honest "I can't confirm that yet". The knowledge base studied here is harder than generic question answering in four ways:

- articles are mostly in English, while customers often ask in Thai, Vietnamese or Indonesian, so retrieval must work across languages;
- promotion names are very similar (for example "Weekend top-up bonus" and "Weekday top-up bonus"), and customers use nicknames;
- policy articles are long, and the key facts often sit in the second half;
- some questions have no answer in the library, and the correct behaviour is to say so.

We address three questions: (RQ1) at the retrieval stage, which representation ranks the correct article highest; (RQ2) end to end, in what form retrieved content is best handed to the answering model; (RQ3) when the knowledge base is small enough to fit in context, whether retrieval is still worthwhile.

Our contribution is a three-round comparison on one synthetic dataset, reported with sample sizes, confidence intervals and significance tests, which sets the product default and states plainly what remains untested.

## Related approaches

On 2026-11-04 we reviewed the official documentation and research pages of several mainstream AI support products, referred to here as products A, B and C. We summarise only their public descriptions; performance figures are vendor-reported and unverified.

- **Product A**: single-pass, two-stage retrieval. The conversation is condensed into a short query, vector search takes the top 40, a cross-encoder reranks them, about 5–10 passages are kept within a context budget, and the answer is validated for grounding after generation. Its research page states that retrieval is purely semantic, with no keyword or hybrid component.
- **Product B**: content is chunked, embedded and matched semantically; answers are generated from the article title plus the best chunk. By default it searches once, using the customer's latest message. Customers configure scope (articles, languages, labels, user segments), not retrieval parameters.
- **Product C**: offers both a fixed retrieval node (semantic, keyword or hybrid, with reranking, top K and a threshold) and an agent mode in which the model searches repeatedly and can read full articles.
- **Full context**: none of the products reviewed states that it places the whole knowledge base in context.
- **DecisionRAG** (public paper): a judge model scores all candidates in one request and filters level by level, "folder → article → passage". Section 5.4 re-examines two of its ideas at the retrieval stage.

Our default resembles product A's public design in structure (vector recall, reranking, a few passages, answering from evidence). It differs in merging two embedding models at recall and having a judge model mark each passage as a strong or weak match.

## Method

### 3.1 Datasets

All three rounds used fictional support content for an online game platform. Articles and questions were written by models; no real customer data was used.

::: table Data per round
| | Round 1 (retrieval) | Round 2 (end to end) | Round 3 (50-article subset) |
| --- | ---: | ---: | ---: |
| Date | 2026-11-02, follow-up 11-03 | 2026-11-05 | 2026-11-10 |
| Articles | A few hundred English help-article passages | 572 articles (608 passages) | 50 articles |
| Question languages | EN, TH, VI, ID | EN, TH, VI, ID | EN, TH, VI, ID |
| Questions | 80 per language | 425 | 173 |
| Scored questions | 80 per language | 392 | 158 |
| Runs | 1 | 2 for the top two, 1 for the rest | 1 |
:::

Round 2's 425 questions cover six scenarios: casual phrasing 105, error codes 80, similarly named promotions 80, details in the second half of long articles 60, same-language content 60, no answer 40. Manual review removed 33 questions that contradicted later articles (23 long-article, 10 no-answer), leaving 392 scored. The round 3 subset contains 15 promotions (keeping several groups with similar names), 7 FAQs, 4 long policies, 3 items each in four language versions, and other ordinary articles; its scored questions are error codes 28, casual 36, similar promotions 44, long articles 11, same language 9, no answer 30.

### 3.2 Conditions

::: table Retrieval approaches compared
| Approach | Description | Round |
| --- | --- | --- |
| Keywords | Literal matching | 1 |
| Keywords + generated questions | Questions generated per article at publish time, matched literally as well | 1 |
| Single embedding | `gemini-embedding-2`, `bge-m3` and others; 19 models in the follow-up | 1 |
| Merge | Keywords, questions and vectors merged by rank (RRF); in the follow-up, 6 pairwise embedding merges and 2 vector-plus-keyword mixes | 1 |
| Vector + reranking | A reranker reorders the vector top 20; 7 rerankers in the follow-up | 1 |
| `rag` | Two embedding models take their top 20 each, merged and reranked by a judge model; returns 5 articles, each with the first 400 characters of the matching passage; the answering model can read full articles | 2 |
| `rag_passage` | As `rag`, returning whole passages (up to 1,500 characters) | 2 |
| `rag_gate` / `rag_gate9` | `rag_passage`, dropping passages the judge scores below 0.5 / 0.9 | 2 |
| `rag_marked` (semantic search, default) | `rag_passage` without dropping; each passage is marked strong or weak at 0.9, and the answering model answers from strong matches only | 2, 3 |
| `toc` | Folder and title lists; the answering model chooses articles to read | 2 |
| `full` (full-context reading) | All articles in the prompt; no retrieval; one call | 2, 3 |
| `hybrid` | `rag` retrieval plus the `toc` lists | 2 |
:::

Round 2's two embedding models are `gemini-embedding-2` and `pplx-embed-v1-4b`; Section 5.1 explains the choice. The judge model `jev-1.14` answers "can this passage answer the customer's question?" with a probability from 0 to 1, and its scores also rerank.

### 3.3 Models and gateways

- **Answering model**: Claude Haiku 5.5 in rounds 2 and 3, with the main-branch platform prompt and tools, at most 6 model calls per turn; only the knowledge tool changes. The keyword comparison in Section 5.3 also uses `gpt-6-luna`.
- **Grader**: Claude Sonnet 5.5.
- **Question generation**: in round 1, questions were written by `gpt-6-luna` and search queries by Claude Haiku, so that no model answered its own questions.
- **Gateways**: answering and grading models were called through the test gateway, embedding and judge models through the production gateway; the 2026-11-08 reruns in Section 5.3 used the production gateway throughout. The same approach scores differently on different gateways (Section 6), so we never compare numbers across gateways.

### 3.4 Measures

- **Hit rate** (round 1): share of questions whose correct article is in the top 5 passages; one correct article per question.
- **Accuracy**: share of scored questions judged correct. The grader compares the answer with the correct article and judges correct, wrong or unanswered; for no-answer questions "unanswered" is correct. The grader also sees the other articles the answer drew on; getting the main question right counts as correct, and added common sense is not penalised.
- **Wrong-answer rate**: share of questions with an invented wrong answer, over all 425 questions in round 2 and 158 in round 3.
- **Unanswered rate**: among answerable questions, the share where the model said it could not find the answer or handed over.
- **Latency and cost**: median and 90th percentile seconds per question; cost estimated from token usage and list prices, not reconciled with invoices.

### 3.5 Statistics

95% confidence intervals for proportions are Wilson intervals computed from accuracy and question count, ignoring correlation between questions. Differences between two approaches on the same questions use a paired sign test on discordant questions (one right, the other wrong). With 80 questions per language, round 1 intervals are about ±7 points.

## Results

### 4.1 Retrieval stage (RQ1)

::: table Top-5 hit rate (%, n = 80 per language)
| Approach | EN | TH | VI | ID |
| --- | ---: | ---: | ---: | ---: |
| Keywords | 60.0 | 26.2 | 21.2 | 46.2 |
| Keywords + generated questions | 61.2 | 63.8 | 68.8 | 67.5 |
| Single embedding `gemini-embedding-2` | 86.2 | 91.2 | 83.8 | 83.8 |
| Single embedding `bge-m3` | 82.5 | 87.5 | 82.5 | 85.0 |
| Three-way merge (`gemini-embedding-2`) | 82.5 | 73.8 | 82.5 | 80.0 |
| Vector + reranking (`rerank-4-fast`) | 81.2 | 82.5 | 78.8 | 82.5 |
| Follow-up: `pplx-embed-v1-4b` merged with `gemini-embedding-2` | 87.5 | 93.8 | 91.2 | 86.2 |
| Follow-up: `gemini-embedding-2` + `qwen3-reranker-8b` | 86.2 | 92.5 | 87.5 | 86.2 |
:::

![Top-5 hit rate](../../assets/kr-r1-hit.en.svg "Keyword retrieval does not cross languages; vector retrieval leads in all four. n = 80 per language, one run")

Keyword retrieval does not cross languages: Thai queries against English articles hit only 26.2% of the time. Generated questions close part of the gap but stay well below vectors. An equal-weight three-way merge did not beat vectors alone. Reranking depends on the model: `rerank-4-fast` made results worse, and among the 7 rerankers in the follow-up `qwen3-reranker-8b` helped slightly. Merging two embedding models averaged 89.7% across the four languages, above the best single model (`pplx-embed-v1-4b`, 88.1%). With 80 questions per language, approaches a few points apart cannot be distinguished, and the merge and reranking combinations were chosen on the same questions, so they may be optimistic.

### 4.2 End to end: eight approaches (RQ2)

::: table Round 2 accuracy by approach (%, n = 392; wrong-answer rate over 425)
| Approach | Score | Casual | Error codes | Similar promos | Long | Same lang. | No answer | Wrong |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `rag` | 80.6 | 75.2 | 96.2 | 91.2 | 51.4 | 91.7 | 43.3 | 14.8 |
| `rag_passage` | 81.9 | 75.2 | 88.8 | 93.8 | 91.9 | 86.7 | 33.3 | 15.5 |
| `rag_gate` | 81.1 | 70.5 | 88.8 | 91.2 | 89.2 | 90.0 | 43.3 | — |
| `rag_gate9` | 79.8 | 58.1 | 86.2 | 87.5 | 89.2 | 95.0 | 76.7 | 12.0 |
| **`rag_marked`** | **83.7** | 67.6 | 93.8 | 92.5 | 91.9 | 91.7 | 63.3 | 11.8 |
| `toc` | 53.1 | 46.7 | 65.0 | 41.2 | 29.7 | 63.3 | 83.3 | 20.9 |
| `full` | 77.6 | 81.0 | 90.0 | 65.0 | 48.6 | 88.3 | 80.0 | 9.9 |
| `hybrid` | 75.3 | 65.7 | 90.0 | 82.5 | 48.6 | 93.3 | 46.7 | 14.8 |
:::

![End-to-end score of eight retrieval approaches](../../assets/kr-r2-overall.en.svg "Error bars are 95% Wilson intervals. The first five approaches overlap; toc is clearly behind")

![Accuracy by scenario](../../assets/kr-r2-scenario.en.svg "rag, returning the first 400 characters, falls behind on long articles; full-context reading falls behind on similar promotions and long articles and leads on casual and no-answer questions")

The wrong-answer rate for `rag_gate` was not recorded and is left blank; part of `toc`'s 83.3% on no-answer questions comes from turns that ran out of calls without replying, not genuine refusals. Main findings:

- **On long articles the bottleneck is what is returned, not retrieval.** `rag` often ranked the long article first but returned only its first 400 characters; the answer lay further down and the model did not open the full article. Returning whole passages raised long-article accuracy from 51.4% to 91.9%.
- **How the judge is used trades fewer inventions against answering casual questions.** Dropping passages below 0.9 (`rag_gate9`) raised no-answer accuracy to 76.7% but cut casual accuracy to 58.1%: casual questions are short, and the right passage often scored below 0.9 and was dropped. Marking without dropping (`rag_marked`) sits between and scored best overall.
- **A repeat run kept the order.** `rag_marked` and `rag_passage` scored 85.2% and 80.9% on the repeat, so `rag_marked` was first both times. Single scenarios moved by 8–13 points between runs, and the two approaches swapped places on casual questions.
- **The intervals of the first five approaches overlap** (`rag_marked` 83.7%, 95% interval 79.7%–87.0%), so single-run differences should be read with care; `toc`'s deficit (53.1%, 48.2%–58.0%) exceeds random variation.

### 4.3 A small knowledge base: semantic search vs full context (RQ3)

In round 2, full-context reading held all 572 articles. To test the earlier, untested assumption that full context may do better on a small knowledge base, round 3 built a 50-article subset. Both approaches used the same subset, and the vector index covered only those 50 articles.

::: table Round 3 main measures (2026-11-10, one run)
| Measure | Full context `full` | Semantic search `rag_marked` |
| --- | ---: | ---: |
| Accuracy (n = 158) | 86.7 (80.5–91.1) | **91.8** (86.4–95.1) |
| Wrong-answer rate (n = 158) | **1.9** | 5.1 |
| Unanswered (answerable, n = 128) | 13.3 | **3.9** |
| No-answer: did not invent (n = 30) | **93.3** | 86.7 |
| Empty replies (questions) | 0 | 4 |
| Model calls per question | 1.0 | 2.6 |
| Seconds per question, median / p90 | **3.5 / 4.9** | 8.6 / 13.7 |
| Cost per question (answering + retrieval, est., USD) | 0.0013 | 0.0016 |
| Prompt tokens per call | 21,793 | 4,638 |
:::

Values in parentheses are 95% Wilson intervals.

![Main measures](../../assets/kr-r3-main.en.svg "Semantic search is more accurate and leaves fewer questions unanswered; full context invents fewer answers")

![Accuracy by scenario](../../assets/kr-r3-scenario.en.svg "Full context falls behind on similar promotions, long articles and same-language content, and leads on error codes and no-answer questions")

![Seconds per question](../../assets/kr-r3-latency.en.svg "Full context answers in one call, at less than half the latency of semantic search")

The approaches disagreed on 26 questions: 17 answered correctly only by semantic search and 9 only by full context; paired sign test p≈0.17.

- **Full context mostly failed by saying it could not find an answer that was in its context.** 14 of the 17 were of this kind, concentrated in similarly named promotions (9) and long articles (3), mostly Indonesian, Vietnamese or Thai questions about English articles.
- **Full context's strength is inventing less.** Of its 9 wins, 3 were error codes semantic search failed to retrieve and 3 were no-answer questions where semantic search invented an answer.
- **Cost depends on cache hits.** Each full-context call carries about 22,000 tokens; in this run about 57% of prompt tokens were read from cache, which is why cost per question was close to semantic search. In production the hit rate depends on whether other conversations arrive within a short window, and may be lower.

### 4.4 Summary

In both end-to-end comparisons semantic search scored higher than full context (572 articles: 83.7% vs 77.6%; 50 articles: 91.8% vs 86.7%). Full context's prompt grows with the knowledge base (about 22,000 tokens at 50 articles, about 170,000 at 572), whereas semantic search carries 5 passages per call.

## Additional experiments

### 5.1 Choice of embedding models

In the round 1 follow-up, the top 4 of 19 embedding models were paired, 6 pairs in all. `gemini-embedding-001` with `text-embedding-3-large` averaged 90.6% across the four languages and `pplx-embed-v1-4b` with `gemini-embedding-2` 89.7%, a 0.9-point gap within the margin of error. Counting a hit whenever any article in the top 5 can answer, the latter was slightly higher (99.1% vs 98.5%, also within error). Round 2 therefore used the latter, but neither pair can be called better.

### 5.2 Strict mode

Strict mode is round 2's `rag_gate9`: passages the judge scores below 0.9 are dropped, so the answering model sees strong matches only. Whether it becomes a product switch has not been decided, and it is not live.

![Strict mode vs default semantic search](../../assets/kr-strict.en.svg "Strict mode invents less on no-answer questions but loses clearly on casual questions; the overall wrong-answer rate does not fall")

Strict mode invented less on no-answer questions (76.7% vs 63.3%), while casual accuracy fell by 9.5 points and the overall wrong-answer rate did not fall (12.0% vs 11.8%). It ran once; default search on its repeat run scored 73.3% on no-answer and 71.4% on casual questions, which narrows the gap. On latency (single-threaded, 22 questions each), strict mode had a median of 8.9 s and a slowest of 32.8 s, close to the answering model's 45-second timeout. It fits content where the library often lacks an answer and "I can't confirm" is preferable to a guess, such as money or compliance.

### 5.3 Keyword boost

Keyword boost (`rag_kw`) adds a literal (BM25) top 5 to vector recall; passages not already in the vector top 20 join the candidates, and the judge model scores each one. The interface seen by the answering model is unchanged. It has been decided as an optional switch, off by default, and is not live.

::: table Keyword boost vs default semantic search (n = 392 per run)
| Answering model | Gateway, date, runs | Keyword boost | Default search | Discordant questions |
| --- | --- | ---: | ---: | --- |
| Claude Haiku 5.5 | Test gateway; keyword boost once on 11-06, default twice on 11-05 | 87.2 | 83.7 / 85.2 | — |
| Claude Haiku 5.5 | Production gateway, 2026-11-08, twice each | 88.8 / 86.5 | 85.7 / 86.0 | 41 vs 27 (p≈0.11) |
| `gpt-6-luna` | Production gateway, 2026-11-08, twice each | 84.7 / 84.9 | 86.7 / 87.5 | 24 vs 42 (p≈0.04) |
:::

![Keyword boost vs default semantic search](../../assets/kr-kw.en.svg "With Haiku answering, a small, non-significant gain; with gpt-6-luna, a significant loss. Y axis starts at 70%")

- With Haiku answering, the gain came mainly from similarly named promotions (9 vs 3) and same-language content (5 vs 0) but was not significant; the same approach run twice also flips 25–43 questions.
- With `gpt-6-luna` answering, accuracy fell significantly, with losses on casual and no-answer questions. It appears the added passages look alike in wording but mean something else, and the model uses them.
- No clear help on questions with numbers or error codes (11-06: 93.4%; default search 92.2% in both 11-05 runs). The synthetic error codes are distinctive, so vectors already find them. Whether knowledge bases dense in short codes and IDs benefit is **not yet tested**.

We also tested two word-based approaches (2026-11-06, test gateway, one run each), neither offered: literal search only (like grep over code) scored 76.0% and missed whenever the customer's wording differed from the article's; giving the model both a semantic and a literal tool scored 83.2%, and the model used the literal tool on only 25 of 425 questions.

### 5.4 Re-examining ranking (DecisionRAG)

On 2026-11-10, using round 1's corpus and questions (320 questions across four languages), we re-examined two DecisionRAG ideas: scoring all candidates together in one request followed by one more selection, and filtering level by level, "folder → article → passage". This experiment covers the retrieval stage only, without the answering model, and ran once.

![Top-5 hit rate of six ranking approaches](../../assets/kr-decisionrag.en.svg "Joint scoring with an extra pick and hierarchical filtering both beat the current per-passage scoring; the latter's gain is concentrated in one flat FAQ folder")

::: table Top-5 hit rate (%, n = 320)
| Approach | EN | TH | VI | ID | Average | Cost per question (USD) |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Merged vectors only | 87.5 | 93.8 | 91.2 | 86.2 | 89.7 | — |
| Per-passage scoring (current) | 87.5 | 91.2 | 87.5 | 88.8 | 88.8 | 0.00041 |
| Joint scoring | 87.5 | 91.2 | 91.2 | 90.0 | 90.0 | 0.00018 |
| Joint scoring + extra pick | 91.2 | 95.0 | 91.2 | 93.8 | 92.8 | 0.00034 |
| Hierarchical filtering | 93.8 | 92.5 | 95.0 | 97.5 | 94.7 | 0.00062 |
| Hierarchical filtering + vector candidates | 93.8 | 96.2 | 95.0 | 96.2 | 95.3 | 0.00081 |
:::

- Joint scoring with an extra pick is 4.1 points above per-passage scoring (paired bootstrap 95% interval +1.6 to +6.9) and cheaper per question; joint scoring alone is only 1.2 points higher (−0.9 to +3.4) and cannot be distinguished.
- Hierarchical filtering is 5.9 points above the current approach (+2.8 to +9.1), but nearly all of the gain is in one flat FAQ folder (140 questions); in the topic folders (180 questions) every judge-based approach is close to perfect. The synthetic folders were built by topic and are tidier than a real knowledge base, so this may be optimistic.
- Current per-passage scoring cannot be distinguished from vectors alone (−0.9, −4.7 to +2.8).

This does not change the decision to offer semantic search only; it concerns ranking inside search. Whether full answers improve, and whether the strong-match cut-off of 0.9 needs resetting under joint scoring, are both **not yet tested**.

## Discussion

**Why semantic search.** First, it scored higher in both end-to-end comparisons. Second, it left far fewer answerable questions unanswered (3.9% vs 13.3%), and handing a customer to a person when the answer is in the library is a poor experience. Third, full context's cost and latency grow with the knowledge base, while semantic search carries 5 passages. Fourth, a single method is easier to explain and to keep evaluating.

**The cost we accept.** Semantic search makes slightly more wrong answers (5.1% vs 1.9%), invents more on no-answer questions (did not invent: 86.7% vs 93.3%), and is slower (median 8.6 s vs 3.5 s). These are the targets for further work; the ranking change in Section 5.4 was evaluated only on retrieval hits, and whether it reduces wrong answers or latency is **not yet tested**.

**Gateway effects.** The same `rag_marked` scored 87.5% on another gateway and 83.7% and 85.2% on the test gateway; the gap came mostly from empty replies (the model called the reply tool without writing text). We therefore compare only within one gateway and one batch.

## Limitations

- **Few runs.** Rounds 1 and 3 ran once; in round 2 only the top two ran twice. Round 3's 17-vs-9 difference has p≈0.17 and may be chance.
- **Synthetic data.** Articles and questions were written by models and may be easier than real customers' phrasing; how much real articles overlap is unknown.
- **Single-turn only.** Follow-ups that depend on earlier messages ("how long does that take?") were not included.
- **One answering model.** The main results come from Claude Haiku 5.5; a stronger model could change how full context and table-of-contents reading perform.
- **Empty replies.** Semantic search had 4 empty replies in round 3; whether they relate to the gateway is **not yet tested**.
- **Grader error.** In round 2, Claude Sonnet 5.5 re-graded a sample of 50 turns and agreed with itself on 46 (92%); round 3 did not repeat this check.
- **Post-hoc selection.** The four approaches from `rag_passage` on were added after seeing earlier results on the same questions, and may be optimistic.
- **Not the formal evaluation pipeline.** A standalone evaluation script was used; the formal pipeline's grounding check did not run.
- **Cost and latency.** Costs use list prices with cache prices estimated by ratio, not reconciled with invoices; round 2 latency was measured single-threaded on one office computer, including retries; round 3 ran 4 questions in parallel.

## Conclusion

On synthetic data, semantic search (`rag_marked`) achieved the highest score in both end-to-end rounds. Full-context reading was faster and invented less on a small knowledge base but left significantly more questions unanswered and scored lower overall. Customers are offered semantic search only; full context is an internal comparison switch; keyword boost is an optional switch, off by default; whether strict mode becomes a switch is undecided.

Further work (all **not yet tested**):

- repeat the semantic search vs full-context comparison on a pilot customer's real knowledge content;
- rerun the 50-article subset to confirm the 91.8% vs 86.7% gap;
- evaluate joint scoring with an extra pick end to end and reset the strong-match cut-off;
- reduce invented answers on no-answer questions and shorten latency;
- add multi-turn follow-up questions.

## Appendix A Parameters

::: table Default semantic search (`rag_marked`) parameters
| Parameter | Value |
| --- | --- |
| Embedding models | `gemini-embedding-2`, `pplx-embed-v1-4b`, top 20 each, merged by rank |
| Judge model | `jev-1.14`, scores each passage and reranks |
| Returned | 5 articles, each with its whole matching passage (up to 1,500 characters) |
| Strong-match cut-off | Judge probability ≥ 0.9 |
| Tools for the answering model | `knowledge_search`, `read_article` |
| Model calls per turn | At most 6, including the final reply |
:::

## Appendix B Cost

::: table Cost per experiment (USD, estimated from tokens and list prices)
| Experiment | Answering and grading models | Embedding and judge models | Total |
| --- | ---: | ---: | ---: |
| Round 1 (2026-11-02) | about 2.88 | 2.19 | about 5.1 |
| Round 1 follow-up (2026-11-03) | 1.15 | 1.87 | about 3.0 |
| Round 2: eight approaches (answering / grading and review) | about 6.5 / about 26 | — | — |
| Round 2: repeat of the top two (answering / grading) | about 0.8 / about 5.4 | — | — |
| Keyword follow-up (2026-11-06) | 10.1 | 0.5 | about 10.6 |
| Production-gateway rerun: keyword boost (2026-11-08) | — | — | about 23.9 |
| Round 3: 50-article subset (2026-11-10) | 2.71 | 0.11 | about 2.83 |
| Ranking re-examination (2026-11-10) | — | 0.56 | 0.56 |
:::

For the two round 1 rows (11-02, 11-03) the two columns are split by test and production gateway; round 1 had no answering model. Round 2's embedding and judge cost is a cumulative figure (about USD 6.8) not split by approach, so it is not listed per row.
