Choosing the right SOP as the catalog grows
Preliminary
The test set has not yet been checked by a second reviewer, and the confidence threshold has not been validated on held-out items. This is an offline test, not a result from live traffic. The numbers show a direction, not a final result.
Abstract
- As the number of SOPs grows, a main model choosing for itself gets worse. Claude Haiku 5.5 falls from 90.6% overall accuracy at 3 SOPs to 64.1% at 50, mostly by picking an SOP when none applies.
- GPT 5.6 Luna is the most accurate on small catalogs (91–98% at 3–10 SOPs), but with a long catalog it often picks none: at 150 SOPs it misses 36.8% of the cases that need one.
- The decision model Jev mostly leads from 90 SOPs up. Letting Jev choose, using its answer when it is confident and passing its top 3 to Luna when it is not, reaches 79.5% at 50 SOPs with a 50% threshold (Jev alone 78.4%, everything to Luna 78.9%) and 76.7% at 150 SOPs (Jev alone 73.8%, everything to Luna 74.5%). The gaps are small. These figures combine existing results offline and still need confirming on held-out items.
- Keep examples out of the full catalog Luna reads: at 10 and 50 SOPs they lower accuracy by 5–8 points. They work better in Jev's catalog (+4.5 points at 50 SOPs) or, from 50 SOPs up, in the top 3 passed to Luna (+3.5 to +4.8 points).
Background #
An SOP is a written procedure for Ailo, such as how to handle a customer who forgot their password. For each customer message, Ailo first has to decide which SOP applies, or that none does.
With a handful of SOPs, putting the catalog in the prompt and letting the main model choose is enough. A site may have hundreds. We wanted to know three things: how much worse a main model gets as SOPs are added, whether a dedicated decision model helps as a first step, and how detailed SOP descriptions should be.
Method #
- Items: 152 SOPs and 1,980 items, from anonymized real customer-support conversations and hand-written cases, in several languages, some with earlier turns. Each item is labeled with one correct SOP or "none applies".
- Scales: subsets of 3, 5, 10, 30, 50, 70, 90, 110, 130 and 150 SOPs. If an item's correct SOP is not in the subset, its answer becomes "none applies". Scales of 3–10 use three draws each, pooled. The number of items varies by scale, from 96 at 3 SOPs to 1,980 at 150 SOPs, and about a third of items have the answer "none applies".
- Three models, same catalogs and items: Claude Haiku 5.5 and GPT 5.6 Luna (reasoning effort low) get the production prompt and tools; the first tool call counts as the choice. Jev (
typesafe/jev-1.14) answers one multiple-choice question whose options are the SOPs plus "none applies", and returns a probability for each. - Two catalog formats:
descis the name, title and when to use it;desc_exadds example customer messages for when it does and does not apply, and was run up to 50 SOPs. - Prefilter: Jev's top 3 become the entire catalog the main model sees. If the correct SOP is not in the top 3, the item counts as wrong.
Measures: overall accuracy covers all items; correct first choice is the share of SOP-needing items answered correctly; false trigger means choosing an SOP when none applies; missed means choosing none when one was needed; wrong pick means choosing a different SOP.
Results #
Three models choosing alone #
| SOPs | Haiku 5.5 | Luna | Jev |
|---|---|---|---|
| 3 | 90.6% | 97.9% | 93.8% |
| 5 | 87.7% | 96.3% | 92.0% |
| 10 | 79.6% | 91.1% | 90.2% |
| 30 | 69.1% | 79.1% | 77.8% |
| 50 | 64.1% | 73.8% | 74.0% |
| 70 | 65.0% | 77.5% | 76.4% |
| 90 | 64.2% | 69.1% | 72.1% |
| 110 | 64.0% | 72.0% | 71.5% |
| 130 | 63.9% | 66.2% | 72.4% |
| 150 | 68.5% | 63.2% | 73.8% |
| SOPs | False trigger Haiku | False trigger Luna | False trigger Jev | Missed Haiku | Missed Luna | Missed Jev |
|---|---|---|---|---|---|---|
| 3 | 18.2% | 6.1% | 0.0% | 4.8% | 0.0% | 9.5% |
| 10 | 43.5% | 13.0% | 1.9% | 7.0% | 5.7% | 13.0% |
| 50 | 61.7% | 31.3% | 27.3% | 9.0% | 16.1% | 16.8% |
| 90 | 56.1% | 18.1% | 28.2% | 9.3% | 31.6% | 15.4% |
| 150 | 40.0% | 20.2% | 18.5% | 7.4% | 36.8% | 11.2% |
The three models fail differently. Haiku picks an SOP too eagerly. Luna has fewer false triggers than Haiku, but often picks none when the catalog is long. Jev is more balanced, and leads clearly from 130 SOPs up.
Jev's top 3, then the main model #
| Approach | Overall | Correct first choice | False trigger | Missed | Wrong pick |
|---|---|---|---|---|---|
| Haiku, no prefilter | 64.1% | 78.0% | 61.7% | 9.0% | 13.0% |
| Luna, no prefilter | 73.8% | 76.5% | 31.3% | 16.1% | 7.3% |
| Jev alone (desc_ex) | 78.4% | 76.3% | 17.6% | 16.1% | 7.6% |
| Jev top 3 → Haiku (sees desc) | 65.6% | 81.5% | 63.9% | 8.1% | 10.4% |
| Jev top 3 → Haiku (sees desc_ex) | 69.2% | 76.3% | 44.1% | 16.1% | 7.6% |
| Jev top 3 → Luna (sees desc) | 76.0% | 85.8% | 42.3% | 5.5% | 8.8% |
| Jev top 3 → Luna (sees desc_ex) | 80.7% | 82.9% | 23.3% | 11.1% | 5.9% |
| Approach | Overall | Correct first choice | False trigger | Missed | Wrong pick |
|---|---|---|---|---|---|
| Haiku, no prefilter | 68.5% | 72.5% | 40.0% | 7.4% | 20.1% |
| Luna, no prefilter | 63.2% | 55.5% | 20.2% | 36.8% | 7.7% |
| Jev alone | 73.8% | 70.2% | 18.5% | 11.2% | 18.6% |
| Jev top 3 → Haiku (sees desc) | 70.8% | 74.2% | 36.5% | 6.0% | 19.8% |
| Jev top 3 → Haiku (sees desc_ex) | 69.3% | 66.9% | 25.3% | 11.8% | 21.3% |
| Jev top 3 → Luna (sees desc) | 71.1% | 77.4% | 42.5% | 4.4% | 18.2% |
| Jev top 3 → Luna (sees desc_ex) | 74.5% | 77.7% | 32.3% | 9.2% | 13.1% |
With desc, the prefilter helps Luna most on large catalogs: at 150 SOPs Luna gains 7.9 points (63.2% → 71.1%) and Haiku 2.3 (68.5% → 70.8%); at 10 and 50 SOPs both gain only 1.5–2.4 points. Luna misses far less once it only sees three SOPs: 4–6% with desc, though false triggers rise to about 42%, and adding examples brings those back to 23–32%. Haiku, given three candidates, still often picks one when none applies (false triggers of 36–64% with desc).
More candidates did not help Haiku. At 150 SOPs the top 3, 5 and 10 contain the right answer 90.1%, 94.0% and 96.6% of the time, but Haiku's overall accuracy is 70.8%, 71.0% and 69.9%, essentially flat: having the answer among the candidates more often did not turn into more correct choices.
Use Jev when it is confident, Luna when it is not #
Jev gives a probability for each option. If its confidence reaches a threshold we use its choice directly; below that, its top 3 go to Luna. The table below combines existing results offline and was not run end to end. In the 50-SOP rows Jev uses desc_ex, and one call gives both the decision and the ranking, so "All to Luna" is 78.9%. The 80.7% in the previous section uses the desc ranking, which needs a second Jev call.
| Scale | Threshold | Sent to Luna | Overall | False trigger | Missed | Wrong pick |
|---|---|---|---|---|---|---|
| 50 SOPs (Jev desc_ex) | 0% (Jev alone) | 0% | 78.4% | 17.6% | 16.1% | 7.6% |
| 50% | 12.5% | 79.5% | 20.3% | 13.0% | 7.6% | |
| 70% | 31.1% | 80.9% | 23.3% | 10.7% | 6.2% | |
| All to Luna | 100% | 78.9% | 26.0% | 12.3% | 6.2% | |
| 150 SOPs (Jev desc) | 0% (Jev alone) | 0% | 73.8% | 18.5% | 11.2% | 18.6% |
| 50% | 19.5% | 76.7% | 20.4% | 9.7% | 14.9% | |
| 70% | 38.7% | 77.0% | 24.2% | 8.7% | 13.7% | |
| All to Luna | 100% | 74.5% | 32.3% | 9.2% | 13.1% |
Where examples help #
| Who reads the catalog | 10 SOPs | 50 SOPs | 150 SOPs |
|---|---|---|---|
| Jev alone | 0.0 | +4.5 | Not tested |
| Haiku, full catalog | +0.4 | +0.5 | Not tested |
| Luna, full catalog | −8.3 | −5.4 | Not tested |
| Haiku, Jev's top 3 | −1.1 | +3.5 | −1.5 |
| Luna, Jev's top 3 | −2.2 | +4.8 | +3.5 |
For Haiku and Luna, examples cut false triggers and raise misses, and the net effect depends on how far misses rise. At 50 SOPs, with Luna reading the full catalog, false triggers fall from 31.3% to 12.8% but misses rise from 16.1% to 37.9%, so overall accuracy drops. With only the top 3, false triggers fall from 42.3% to 23.3% and misses rise only from 5.5% to 11.1%, so overall accuracy rises. For Haiku reading the full catalog the two cancel out, leaving overall accuracy almost unchanged (+0.5 at 50 SOPs). With examples Jev's misses barely change (16.8% → 16.1%) while false triggers fall from 27.3% to 17.6%.
Speed #
Jev takes about 0.4–0.6 seconds per call. Luna reading three SOPs has a median of about 2.0–2.3 seconds, slightly faster than reading the full catalog. At 150 SOPs the two steps add up to about 2.7 seconds, about the same as Luna choosing alone at 150 SOPs (2.7 seconds). This is an estimate from adding the two call times.
Recommendations #
| Situation | Recommendation |
|---|---|
| Fewer than 30 SOPs, Luna as main model | No prefilter. Luna reads the full catalog and chooses; no examples in the catalog |
| 30 SOPs or more, Luna as main model | Jev chooses first (this combination was only computed at 50 and 150 SOPs, with examples in its catalog at 50 and desc at 150). If it reaches the threshold, put that SOP's text into this turn's context; otherwise pass its top 3, with examples, to Luna |
| Haiku as main model | At any size, let Jev decide; Haiku only carries out the SOP |
| Setting the threshold | Default 50%: fewer items reach Luna (12.5% at 50 SOPs, 19.5% at 150). At 50 SOPs, with one Jev desc_ex call, overall accuracy is 79.5%, which cannot yet be separated from Jev alone at 78.4%. At 70%, 31–39% of items reach Luna for only 0.3–1.4 more points, within the confidence interval, so we do not recommend raising it for now |
When writing SOP descriptions, describe the situation and do not name other SOPs' IDs. Of 152 SOPs, 80 had descriptions mentioning another SOP. Across all scales, Haiku read an SOP that was not in the catalog 279 times, and in 255 of those the ID was one mentioned in a description.
Limitations #
- The test set has not been checked by a second reviewer, and some labeling rules are still open.
- The threshold was examined on the same items, so results may be optimistic; it needs validating on held-out items.
- Threshold results combine existing runs offline and were not run end to end.
- Jev with examples was only tested up to 50 SOPs. Scaling from measured sizes, each question at 150 SOPs would be over 60,000 tokens; this was not run.
- About 86% of items are in one language and the others are a minority; we draw no per-language conclusions.
- The prefilter was only tested at 10, 50 and 150 SOPs. The boundary of 30 comes from Jev alone (desc_ex, 79.8%) and Luna alone (79.1%) being close there; sizes between 10 and 30 were not tested. At 10 SOPs Jev's top 3 → Luna (desc) reaches 93.5%, slightly above Luna alone at 91.1%, but the confidence intervals overlap and it costs one more call.
- We only tested the first choice of SOP. Whether to continue or switch while a customer is partway through an SOP, and the quality of replies that follow an SOP, were not tested.
- At 50 SOPs there are about 650 items, with a 95% confidence interval of about ±3 points; at 150 SOPs there are 1,980 items, about ±2 points. Differences smaller than that cannot separate approaches.
- The item set differs by scale (96 to 1,980 items), so changes across scales are not caused by catalog length alone, and this may explain why accuracy recovers at some scales.
Next steps #
- Have a second reviewer check the test set, split items into tuning and held-out sets, and re-validate the threshold.
- Test continuing or switching SOPs in multi-turn conversations.
- Validate the recommended approach on a small share of real traffic.