AI planner runs

15 runs over 15 cases · 89 turns · 2,941,054 tokens · ≈ 0.442 USD · 0 of 15 scored runs have a failing check

Scorecard

ClaimPassFail?
A search without an area is country-wide by design 15
An interpreted fact reaches the loop and not the account 1 14
A pick that accepts the venue offer gets the search that turn 4 11
A fact the couple declined is never asked again 6 9
Every figure carries a source it can support 6 9
A checklist built from a date is reported with its real task count 9 6
Asking never outruns the two-in-a-row guard 10 5
A turn that writes but unlocks nothing uses the one-line form 10 5
A capability is called by turn 2 11 4
No three consecutive turns ask for a fact 14 1
The interview detector sees every turn 14 1
A readiness offer is made at most twice per capability, then goes quiet 14 1
A year-only plan that talks timing says it is dated from January 4 1
No internal stage name is ever said to the couple 15
No word from inside the machine reaches the couple 15
A turn where the couple states nothing produces no receipt 15
The receipt reports a change, not a state 15
One turn builds the checklist at most once 15
The offer log keeps every real offer and at most the latest quiet turn 15
The run completed without a stream or write error 15
A year-only date gets no months-out figure 5
A year-only plan is not called late without saying why 5
"Next year" resolves to the calendar year after today 1

Every claim, counted across every scored run. ? means the run could not answer it, which is not a pass — a claim answered by nothing is untested, not met. Some passes are vacuous: a search that never ran cannot run without a search area.

Runs, newest first

changing-minds 14 turns ≈ 0.108 USD all checks pass 3 unanswered

Full facts at once, then a guest cut, a date move, a city switch and re-searches. Scores writes, rebuilds and re-ranking over 14 steps.

2026-08-18 23:15 · commit dc9df71daf6 · 733,580 tokens · 2 receipts · tools: executeClientActions, searchWeddingSuppliers, getBudgetBreakdown, getUpcomingTaskAdvice, planningSnapshot
budget-journey 12 turns ≈ 0.077 USD all checks pass 4 unanswered

A budget refusal that later becomes £18,000, then splits, benchmarks and savings questions at their own numbers. 12 steps.

2026-08-18 23:12 · commit dc9df71daf6 · 523,140 tokens · 1 receipt · tools: executeClientActions, getCoachReference, searchWeddingSuppliers, getBudgetBreakdown, getUpcomingTaskAdvice
full-journey 15 turns ≈ 0.108 USD all checks pass 4 unanswered

Complete onboarding then venue work, photographer costs, a reload and a month plan. Scores cadence and state over a 15-step conversation.

2026-08-18 23:08 · commit dc9df71daf6 · 681,846 tokens · 1 receipt · tools: executeClientActions, getBudgetBreakdown, searchWeddingSuppliers, getCoachReference, getUpcomingTaskAdvice, planningSnapshot
all-at-once 4 turns ≈ 0.016 USD all checks pass 4 unanswered

Every core fact in two messages. Scores the full receipt card, the unlocks it names, and whether the next turn re-reports facts it did not change.

2026-08-18 22:58 · commit 4268d1ad2f2 · 110,169 tokens · 1 receipt · tools: executeClientActions, searchWeddingSuppliers, getUpcomingTaskAdvice
supplier-search 4 turns ≈ 0.016 USD all checks pass 6 unanswered

A real place picked from the location field, then a venue request. Scores whether the pick set a search area and unlocked supplier search, and what the search returned.

2026-08-18 22:56 · commit 4268d1ad2f2 · 126,801 tokens · 0 receipts · tools: executeClientActions, searchWeddingSuppliers, compareSuppliers
relative-date 3 turns ≈ 0.010 USD all checks pass 6 unanswered

The couple says "next year" and never names it. Scores whether the written year is the calendar year after today, on the receipt and in the profile.

2026-08-18 22:55 · commit 4268d1ad2f2 · 61,255 tokens · 1 receipt · tools: executeClientActions, getUpcomingTaskAdvice
cost-questions 4 turns ≈ 0.0095 USD all checks pass 5 unanswered

Three direct cost questions. Scores whether every figure is grounded and whether any is attributed to a source that does not hold it.

2026-08-18 22:00 · commit d6dd2848e8d · 94,389 tokens · 1 receipt · tools: getBudgetBreakdown, getCoachReference
stonewaller 6 turns ≈ 0.016 USD all checks pass 7 unanswered

Five turns that state nothing at all. Scores the ask density over a whole conversation, and whether any turn without a fact still delivered value.

2026-08-18 21:58 · commit d6dd2848e8d · 79,587 tokens · 0 receipts · tools: none called
returning-couple 5 turns ≈ 0.016 USD all checks pass 5 unanswered

A first meeting, then a return on another day with the transcript gone. Scores whether the second opener knows them, and whether it re-asks what it was told.

2026-08-18 21:57 · commit d6dd2848e8d · 119,415 tokens · 1 receipt · tools: executeClientActions, getUpcomingTaskAdvice, planningSnapshot
just-show-me 3 turns ≈ 0.0054 USD all checks pass 8 unanswered

A demand for suppliers before anything is known, repeated. Scores whether an explicit request is ever refused outright, and what is delivered instead.

2026-08-18 21:56 · commit d6dd2848e8d · 54,841 tokens · 0 receipts · tools: searchWeddingSuppliers
refusal-reload 5 turns ≈ 0.0099 USD all checks pass 7 unanswered

A budget refusal, then the tab closes and reopens. Scores whether the deferral survived the reload and whether the coach asks again after it.

2026-08-18 21:55 · commit d6dd2848e8d · 64,370 tokens · 0 receipts · tools: getCoachReference
generic-location 4 turns ≈ 0.015 USD all checks pass 4 unanswered

A location too generic to search on. Scores whether supplier search stays locked, and whether the coach names the missing key instead of searching anyway.

2026-08-18 21:54 · commit d6dd2848e8d · 90,198 tokens · 1 receipt · tools: executeClientActions, searchWeddingSuppliers
vision-first 5 turns ≈ 0.018 USD all checks pass 4 unanswered

Vision, then a guest count, then two budget refusals, then a year. Scores what a rapport fact writes, and whether a refusal is honoured.

2026-08-18 21:52 · commit d6dd2848e8d · 94,344 tokens · 2 receipts · tools: executeClientActions, getUpcomingTaskAdvice
year-only 4 turns ≈ 0.015 USD all checks pass 3 unanswered

Two budget refusals, then the bare year 2027. Scores the year-only plan, the receipt for a checklist build, and whether a declined fact is asked again.

2026-08-18 21:50 · commit d6dd2848e8d · 96,729 tokens · 1 receipt · tools: getCoachReference, executeClientActions, getUpcomingTaskAdvice
opener 1 turn ≈ 0.0004 USD all checks pass 12 unanswered

The opening move alone. Scores the tier-0 insight, the provenance clause, and whether the single question is an offer’s price rather than an interview opener.

2026-08-18 21:50 · commit d6dd2848e8d · 10,390 tokens · 0 receipts · tools: none called