Evaluation
Building a model benchmark that keeps its context
Model selection needs frozen tasks, runtime identity, request failures, and confirmation runs—not a single leaderboard number.
- One standardized JSON-extraction batch
- 12 / 12
- A distinct reasoning batch with the same model
- 9 / 12
- Separately recorded cloud-reference exact passes
- 72 / 72
A task-level study
The LLM Benchmark Study Master records model identities, generation settings, load events, standardized batches, request results, telemetry, incidents, and source artifacts. Its protocol preserves legacy observations without treating mixed-granularity historical notes as new standardized metrics.
In the evidence rollup, Qwen3.5 35B A3B completed one 12-job RAM-only JSON-extraction batch with 12 exact passes. A separate 12-job reasoning batch recorded nine exact passes. Those are different tests; neither supports a universal accuracy claim.
What a provisional shortlist means
The portfolio sheet ranks candidates for specific lanes such as extraction, reasoning, code, and retrieval. It explicitly says that a top-five row is a routing shortlist rather than a blanket model winner.
Best-recorded throughputs come from particular profiles and workloads. They are not a matched head-to-head test of every model or evidence that a text benchmark qualifies vision, OCR, or coding.
Keeping cloud and local evidence separate
The study control records a GPT-6 Luna High cloud reference with 72 of 72 exact passes across extraction and grounded-reasoning tasks at three difficulty levels. That is a cloud-reference result; it is not CPU-only execution or proof that a local system matches the reference.
Hardware and service snapshots are dated. A recorded environment does not establish the R640’s present inventory, spare capacity, or active workload.
What we are trying to achieve
The goal is to choose models by the task they can do reliably. We are building a record that can support narrow routing decisions, reproduce useful comparisons, and show when a stronger model or human review is warranted.
Sources & reading notes
- LLM Benchmark Study MasterStudy Control; Hardware Runtime; Evidence Rollup; Portfolio Review. Drive workbook · access may be restricted Source record retained privately.
- Benchmark Capture ProtocolFrozen packets, runtime identity, acceptance rubrics, and promotion limits. Drive record · access may be restricted Source record retained privately.
This is a dated, source-based research account, not a live machine report. Historical results, engineering preparation, research designs, open review gates, and measured pilots remain distinct.