Evaluation
Reading CPU inference claims carefully
A token-per-second claim is only useful when we know what was counted, on which machine, and under which workload.
- Provider answers preserved
- 28
- Responding providers
- 7
- Answered follow-ups
- 21
The question
Which public claims establish output inference at or above 13 tokens/second on conventional CPU/system RAM?
How the work was approached
Identical seed and focused clarifications across seven providers; preserved original source fields, measurement classifications and corrections.
What the record shows
Created a provenance-preserving benchmark-claims collection and identified common errors: decode versus prefill, single-stream versus aggregate, synthetic work units versus real output, and confounded optimization comparisons.
What remains unresolved
No R640 reproduction. Provider tables remain attributed and are not an independently verified or deduplicated global benchmark database. RTF layout not certified.
What we are trying to achieve
The goal is a fair basis for deciding what to try on our own systems. Careful source and measurement review should narrow the candidate set, reveal uncertainty, and prevent an unrelated headline number from becoming a promised R640 result.
Sources & reading notes
- CHECKPOINT.mdScope / Current Work / Evidence Boundaries Retained project record Source record retained privately.
- collection_integrity.json28 answers, 7 providers, 21 questions Retained project record Source record retained privately.
- discussion_notes.mdExplicit Corrections Preserved / Unresolved Evidence Retained project record Source record retained privately.
This is a dated, source-based research account, not a live machine report. Historical results, engineering preparation, research designs, open review gates, and measured pilots remain distinct.