What it is
A rigorous, evidence-led methodology for evaluating whether a local or open-weight LLM can be trusted for a given task. The work centres on held-out evaluation sets, predeclared scoring thresholds, hash-bound evidence artifacts, and independent review gates — the discipline of proving whether a model or prompting approach works, with explicit gates against overclaiming, before any production trust is extended.
- Hash-bound evidence artifacts and independent review gates at each stage
- Hard rule against claiming progress without verified, reproducible evidence
- Predeclared advancement gates — the methodology fails closed when thresholds are not met
- Baseline and guarded condition comparison to isolate what actually changes outcomes
- Fail-closed non-degradation gate that aborts runs causing host resource regressions
- Honest about what remains open — retrieval and training gated until evaluation results justify them
Why it matters
Most teams evaluating LLMs do not have a rigorous answer to "how do we know this actually works?" They run a demo, look at a few outputs, and ship — or they read a benchmark from a vendor's marketing page. This methodology starts from the opposite position — advancement is blocked until predeclared thresholds are met on a held-out evaluation, and the evidence is reproducible and hash-bound. That honest discipline is what makes the findings credible.
How it could help you
CodeVolt can work with you to apply this methodology to your own LLM selection, prompting strategy or self-hosted inference evaluation. We would design the evaluation set, define the acceptance criteria and scoring thresholds before running anything, run the evaluation against baseline and improved conditions, and report results honestly — including where thresholds were not met. This is active, ongoing work; not a finished product we hand over.
The problem
Teams evaluating LLMs or prompting strategies typically lack a rigorous framework for distinguishing genuine improvement from noise, test-set overfitting, or vendor-controlled benchmarks. Without predeclared gates and reproducible evidence, it is easy to mistake a promising demo for production-ready capability.
Who this is for
Technical teams evaluating LLM providers, prompting strategies, or considering local or self-hosted inference — particularly those who need to justify trust decisions to stakeholders with evidence, not demonstrations.
Evidence base
-
An untouched baseline model scored 93/120 (77.5%) on a 12-case held-out evaluation, with one critical synthetic-secret-handling failure. A guarded condition — deterministic credential redaction, a concise behaviour contract, JSON grammar constraints and case-specific validation — raised the score to 106/120 (88.3%) with zero critical failures. The guarded condition still failed the project's own predeclared advancement gate because evidence and epistemic-reasoning cases scored only 70%, below the required 75% threshold.
https://codevolt.co.uk/capabilities/llm-evaluation-methodology -
Local inference throughput measured at approximately 748 prompt tokens/sec and 57 generation tokens/sec on bounded local inference, with no swap or memory pressure regressions. A fail-closed non-degradation gate aborts evaluation runs that would degrade the host.
https://codevolt.co.uk/capabilities/llm-evaluation-methodology
Open questions
These are questions CodeVolt is still working through. Naming them is part of the honest framing of this capability.
- Evidence and epistemic-reasoning calibration scored 70% against a required 75% threshold — that gate has not yet passed and the methodology acknowledges this openly.
- Retrieval-augmented approaches remain gated until evaluation results justify the additional complexity.
- Model training is not admitted in this project — governance states training is not considered until evaluation and retrieval results justify it, and that gate has not passed.
- Which task domains and evaluation set designs produce the most useful signal for the types of decisions teams actually face?
Prerequisites
- A concrete task or capability to evaluate, with named acceptance criteria defined before any model is run
- A held-out evaluation set that has not been used to guide prompting or model selection
- Agreed scoring thresholds declared before evaluation begins — not adjusted after seeing results
- Reproducible evaluation runs with hash-bound evidence artifacts
Risks to hold
- Evaluation sets can become stale or overfit to the methodology itself over successive iterations
- Scoring thresholds that are too lenient reward marginal improvement; too strict and no reasonable approach passes
- Prompting improvements do not always transfer across model versions or deployment contexts
- This methodology does not certify production readiness — it establishes whether a condition met a predeclared bar on a specific held-out set, under the conditions tested
Discuss this with us
There is genuine thinking behind this capability. If you are working through a similar problem, we would like to hear about it.
Start a conversation We review suitability before agreeing any work.