Research

Choosing the right SOP as the catalog grows

Product
Harness
Date
2026-11-11
Authors
Eval team
Version
v0.1
Status
Preliminary

Preliminary

The test set has not yet been checked by a second reviewer, and the confidence threshold has not been validated on held-out items. This is an offline test, not a result from live traffic. The numbers show a direction, not a final result.

Abstract

  • As the number of SOPs grows, a main model choosing for itself gets worse. Claude Haiku 5.5 falls from 90.6% overall accuracy at 3 SOPs to 64.1% at 50, mostly by picking an SOP when none applies.
  • GPT 5.6 Luna is the most accurate on small catalogs (91–98% at 3–10 SOPs), but with a long catalog it often picks none: at 150 SOPs it misses 36.8% of the cases that need one.
  • The decision model Jev mostly leads from 90 SOPs up. Letting Jev choose, using its answer when it is confident and passing its top 3 to Luna when it is not, reaches 79.5% at 50 SOPs with a 50% threshold (Jev alone 78.4%, everything to Luna 78.9%) and 76.7% at 150 SOPs (Jev alone 73.8%, everything to Luna 74.5%). The gaps are small. These figures combine existing results offline and still need confirming on held-out items.
  • Keep examples out of the full catalog Luna reads: at 10 and 50 SOPs they lower accuracy by 5–8 points. They work better in Jev's catalog (+4.5 points at 50 SOPs) or, from 50 SOPs up, in the top 3 passed to Luna (+3.5 to +4.8 points).

Background #

An SOP is a written procedure for Ailo, such as how to handle a customer who forgot their password. For each customer message, Ailo first has to decide which SOP applies, or that none does.

With a handful of SOPs, putting the catalog in the prompt and letting the main model choose is enough. A site may have hundreds. We wanted to know three things: how much worse a main model gets as SOPs are added, whether a dedicated decision model helps as a first step, and how detailed SOP descriptions should be.

Method #

  • Items: 152 SOPs and 1,980 items, from anonymized real customer-support conversations and hand-written cases, in several languages, some with earlier turns. Each item is labeled with one correct SOP or "none applies".
  • Scales: subsets of 3, 5, 10, 30, 50, 70, 90, 110, 130 and 150 SOPs. If an item's correct SOP is not in the subset, its answer becomes "none applies". Scales of 3–10 use three draws each, pooled. The number of items varies by scale, from 96 at 3 SOPs to 1,980 at 150 SOPs, and about a third of items have the answer "none applies".
  • Three models, same catalogs and items: Claude Haiku 5.5 and GPT 5.6 Luna (reasoning effort low) get the production prompt and tools; the first tool call counts as the choice. Jev (typesafe/jev-1.14) answers one multiple-choice question whose options are the SOPs plus "none applies", and returns a probability for each.
  • Two catalog formats: desc is the name, title and when to use it; desc_ex adds example customer messages for when it does and does not apply, and was run up to 50 SOPs.
  • Prefilter: Jev's top 3 become the entire catalog the main model sees. If the correct SOP is not in the top 3, the item counts as wrong.

Measures: overall accuracy covers all items; correct first choice is the share of SOP-needing items answered correctly; false trigger means choosing an SOP when none applies; missed means choosing none when one was needed; wrong pick means choosing a different SOP.

Results #

Three models choosing alone #

Overall accuracy (desc, no prefilter)
SOPs Haiku 5.5 Luna Jev
3 90.6% 97.9% 93.8%
5 87.7% 96.3% 92.0%
10 79.6% 91.1% 90.2%
30 69.1% 79.1% 77.8%
50 64.1% 73.8% 74.0%
70 65.0% 77.5% 76.4%
90 64.2% 69.1% 72.1%
110 64.0% 72.0% 71.5%
130 63.9% 66.2% 72.4%
150 68.5% 63.2% 73.8%
Overall accuracy of the three models by number of SOPsOverall accuracy of the three models by number of SOPs
desc format, no prefilter. Each point covers 96 items (3 SOPs) to 1,980 items (150 SOPs); scales of 3–10 pool three draws. The horizontal axis spaces tested scales evenly; the vertical axis starts at 50%
False triggers and misses (desc, no prefilter)
SOPs False trigger Haiku False trigger Luna False trigger Jev Missed Haiku Missed Luna Missed Jev
3 18.2% 6.1% 0.0% 4.8% 0.0% 9.5%
10 43.5% 13.0% 1.9% 7.0% 5.7% 13.0%
50 61.7% 31.3% 27.3% 9.0% 16.1% 16.8%
90 56.1% 18.1% 28.2% 9.3% 31.6% 15.4%
150 40.0% 20.2% 18.5% 7.4% 36.8% 11.2%
False triggers and misses of the three models by number of SOPsFalse triggers and misses of the three models by number of SOPs
desc format, no prefilter. False triggers count only items whose answer is ‘none applies’ (33 to 628 items); misses count only items that need an SOP (63 to 1,352 items). The chart shows all 10 scales; the table above lists 5

The three models fail differently. Haiku picks an SOP too eagerly. Luna has fewer false triggers than Haiku, but often picks none when the catalog is long. Jev is more balanced, and leads clearly from 130 SOPs up.

Jev's top 3, then the main model #

50 SOPs ("top 3" ranked by Jev with desc)
Approach Overall Correct first choice False trigger Missed Wrong pick
Haiku, no prefilter 64.1% 78.0% 61.7% 9.0% 13.0%
Luna, no prefilter 73.8% 76.5% 31.3% 16.1% 7.3%
Jev alone (desc_ex) 78.4% 76.3% 17.6% 16.1% 7.6%
Jev top 3 → Haiku (sees desc) 65.6% 81.5% 63.9% 8.1% 10.4%
Jev top 3 → Haiku (sees desc_ex) 69.2% 76.3% 44.1% 16.1% 7.6%
Jev top 3 → Luna (sees desc) 76.0% 85.8% 42.3% 5.5% 8.8%
Jev top 3 → Luna (sees desc_ex) 80.7% 82.9% 23.3% 11.1% 5.9%
150 SOPs (Jev ranks with desc)
Approach Overall Correct first choice False trigger Missed Wrong pick
Haiku, no prefilter 68.5% 72.5% 40.0% 7.4% 20.1%
Luna, no prefilter 63.2% 55.5% 20.2% 36.8% 7.7%
Jev alone 73.8% 70.2% 18.5% 11.2% 18.6%
Jev top 3 → Haiku (sees desc) 70.8% 74.2% 36.5% 6.0% 19.8%
Jev top 3 → Haiku (sees desc_ex) 69.3% 66.9% 25.3% 11.8% 21.3%
Jev top 3 → Luna (sees desc) 71.1% 77.4% 42.5% 4.4% 18.2%
Jev top 3 → Luna (sees desc_ex) 74.5% 77.7% 32.3% 9.2% 13.1%

With desc, the prefilter helps Luna most on large catalogs: at 150 SOPs Luna gains 7.9 points (63.2% → 71.1%) and Haiku 2.3 (68.5% → 70.8%); at 10 and 50 SOPs both gain only 1.5–2.4 points. Luna misses far less once it only sees three SOPs: 4–6% with desc, though false triggers rise to about 42%, and adding examples brings those back to 23–32%. Haiku, given three candidates, still often picks one when none applies (false triggers of 36–64% with desc).

More candidates did not help Haiku. At 150 SOPs the top 3, 5 and 10 contain the right answer 90.1%, 94.0% and 96.6% of the time, but Haiku's overall accuracy is 70.8%, 71.0% and 69.9%, essentially flat: having the answer among the candidates more often did not turn into more correct choices.

Use Jev when it is confident, Luna when it is not #

Jev gives a probability for each option. If its confidence reaches a threshold we use its choice directly; below that, its top 3 go to Luna. The table below combines existing results offline and was not run end to end. In the 50-SOP rows Jev uses desc_ex, and one call gives both the decision and the ranking, so "All to Luna" is 78.9%. The 80.7% in the previous section uses the desc ranking, which needs a second Jev call.

Effect of the threshold
Scale Threshold Sent to Luna Overall False trigger Missed Wrong pick
50 SOPs (Jev desc_ex) 0% (Jev alone) 0% 78.4% 17.6% 16.1% 7.6%
50% 12.5% 79.5% 20.3% 13.0% 7.6%
70% 31.1% 80.9% 23.3% 10.7% 6.2%
All to Luna 100% 78.9% 26.0% 12.3% 6.2%
150 SOPs (Jev desc) 0% (Jev alone) 0% 73.8% 18.5% 11.2% 18.6%
50% 19.5% 76.7% 20.4% 9.7% 14.9%
70% 38.7% 77.0% 24.2% 8.7% 13.7%
All to Luna 100% 74.5% 32.3% 9.2% 13.1%

Where examples help #

Change in overall accuracy from adding examples (desc → desc_ex), in points
Who reads the catalog 10 SOPs 50 SOPs 150 SOPs
Jev alone 0.0 +4.5 Not tested
Haiku, full catalog +0.4 +0.5 Not tested
Luna, full catalog −8.3 −5.4 Not tested
Haiku, Jev's top 3 −1.1 +3.5 −1.5
Luna, Jev's top 3 −2.2 +4.8 +3.5

For Haiku and Luna, examples cut false triggers and raise misses, and the net effect depends on how far misses rise. At 50 SOPs, with Luna reading the full catalog, false triggers fall from 31.3% to 12.8% but misses rise from 16.1% to 37.9%, so overall accuracy drops. With only the top 3, false triggers fall from 42.3% to 23.3% and misses rise only from 5.5% to 11.1%, so overall accuracy rises. For Haiku reading the full catalog the two cancel out, leaving overall accuracy almost unchanged (+0.5 at 50 SOPs). With examples Jev's misses barely change (16.8% → 16.1%) while false triggers fall from 27.3% to 17.6%.

Speed #

Jev takes about 0.4–0.6 seconds per call. Luna reading three SOPs has a median of about 2.0–2.3 seconds, slightly faster than reading the full catalog. At 150 SOPs the two steps add up to about 2.7 seconds, about the same as Luna choosing alone at 150 SOPs (2.7 seconds). This is an estimate from adding the two call times.

Recommendations #

Which approach for which situation
Situation Recommendation
Fewer than 30 SOPs, Luna as main model No prefilter. Luna reads the full catalog and chooses; no examples in the catalog
30 SOPs or more, Luna as main model Jev chooses first (this combination was only computed at 50 and 150 SOPs, with examples in its catalog at 50 and desc at 150). If it reaches the threshold, put that SOP's text into this turn's context; otherwise pass its top 3, with examples, to Luna
Haiku as main model At any size, let Jev decide; Haiku only carries out the SOP
Setting the threshold Default 50%: fewer items reach Luna (12.5% at 50 SOPs, 19.5% at 150). At 50 SOPs, with one Jev desc_ex call, overall accuracy is 79.5%, which cannot yet be separated from Jev alone at 78.4%. At 70%, 31–39% of items reach Luna for only 0.3–1.4 more points, within the confidence interval, so we do not recommend raising it for now

When writing SOP descriptions, describe the situation and do not name other SOPs' IDs. Of 152 SOPs, 80 had descriptions mentioning another SOP. Across all scales, Haiku read an SOP that was not in the catalog 279 times, and in 255 of those the ID was one mentioned in a description.

Limitations #

  • The test set has not been checked by a second reviewer, and some labeling rules are still open.
  • The threshold was examined on the same items, so results may be optimistic; it needs validating on held-out items.
  • Threshold results combine existing runs offline and were not run end to end.
  • Jev with examples was only tested up to 50 SOPs. Scaling from measured sizes, each question at 150 SOPs would be over 60,000 tokens; this was not run.
  • About 86% of items are in one language and the others are a minority; we draw no per-language conclusions.
  • The prefilter was only tested at 10, 50 and 150 SOPs. The boundary of 30 comes from Jev alone (desc_ex, 79.8%) and Luna alone (79.1%) being close there; sizes between 10 and 30 were not tested. At 10 SOPs Jev's top 3 → Luna (desc) reaches 93.5%, slightly above Luna alone at 91.1%, but the confidence intervals overlap and it costs one more call.
  • We only tested the first choice of SOP. Whether to continue or switch while a customer is partway through an SOP, and the quality of replies that follow an SOP, were not tested.
  • At 50 SOPs there are about 650 items, with a 95% confidence interval of about ±3 points; at 150 SOPs there are 1,980 items, about ±2 points. Differences smaller than that cannot separate approaches.
  • The item set differs by scale (96 to 1,980 items), so changes across scales are not caused by catalog length alone, and this may explain why accuracy recovers at some scales.

Next steps #

  • Have a second reviewer check the test set, split items into tuning and held-out sets, and re-validate the threshold.
  • Test continuing or switching SOPs in multi-turn conversations.
  • Validate the recommended approach on a small share of real traffic.