A successful request is not a correct answer

Evaluator regression cases passed

31 / 31

Successful requests replayed

7

Answer-level reassessment

2 fail · 5 review

The question

Did the old benchmark confuse successful API responses with correct, complete answers?

How the work was approached

Added observable output checks, preserved semantic-review states, checked truncated code without executing it, and replayed seven saved Bonsai answers.

What the record shows

Seven successful requests did not establish seven good answers. Replay found extra prose violating an exact-bullet instruction and truncated coding output with an unclosed fence/missing brace; five answers still needed semantic/completeness review.

What remains unresolved

31 passing regression fixtures validate the evaluator's specified behavior, not production model quality. Offline replay did not run Bonsai again. Original source files were preserved.

What we are trying to achieve

The goal is to make model-selection decisions on useful, complete answers. We are building a habit of checking the actual output and keeping uncertainty visible, so a technically successful request cannot silently become evidence of a capable system.

Sources & reading notes

1. bonsai-quality-review-20261007.mdSummary and Prompt Results Retained project record Source record retained privately.
2. quality-validation-regressions-20261007.json31 cases and outcomes Retained project record Source record retained privately.
3. results.jsonOriginal request counts and stop reasons Retained project record Source record retained privately.
4. 004-instruction_following.txtActual extra prose after three bullets Retained project record Source record retained privately.
5. 006-coding_helper.txtActual incomplete code/fence Retained project record Source record retained privately.

This is a dated, source-based research account, not a live machine report. Historical results, engineering preparation, research designs, open review gates, and measured pilots remain distinct.