The standing commitment
What would make me tell you not to deploy
Every buyer of an evaluation has the same private worry, and almost nobody selling one addresses it: that you will find whatever result I paid you to find. The worry is rational. Nothing about a framework, a certificate or a dashboard constrains what an evaluator concludes. This page is the constraint I accept instead, written down so it can be held against me.
Five commitments
Each of these forbids something. That is what makes it a commitment.
- 01
Thresholds before results
Pass and fail criteria are written down and timestamped before I see the system. Not "we will assess performance" — the actual numbers, and what happens either side of them.
Rules outDeciding what counts as good after seeing how the system performed.
- 02
The unflattering result gets published
If the evaluation says do not deploy, the report says do not deploy. If it says the thing you already shipped does not work, it says that. Agreed in writing before the work starts, because agreeing to it afterwards is worth nothing.
Rules outA result that exists only if it is favourable.
- 03
Every claim carries its method
Sample size, fieldwork dates, who funded it, and what result would falsify it. A number without those is an assertion wearing a number’s clothes.
Rules outQuoting a figure whose method I have not read.
- 04
Corrections are published, not edited away
When I get something wrong, the correction is a new dated statement and the old wording is found and fixed everywhere it was repeated. The mistake stays legible.
Rules outQuietly changing a claim so the record looks cleaner than it was.
- 05
The instrument is inspectable
Where the work involves software I built, the properties that matter are testable rather than asserted — and where a participant or a client can check one from the outside, the page tells them how.
Rules outAsking you to trust a description of the system instead of the system.
What it costs
The right to be flattering
Pre-committing to a threshold means giving up the most commercially useful thing an evaluator has: the ability to decide, after the fact, that whatever happened counts as a good outcome.
It also means some engagements end with a report the client did not want, which is the point at which a method either exists or does not. I would rather lose that work than hold a threshold loosely, because a threshold held loosely is not a threshold — it is a preference.
Apply this to me
One question, asked before you hire anyone
Write down, in advance, what result would make you tell me not to deploy.
If they cannot answer, the number they eventually hand you is not evidence about your system. It is evidence about what they were willing to say.
If they will not answer, that is the more informative refusal of the two.
Ask it of me too. If I ever fail to answer it in writing, that is the moment to stop reading this page and start ignoring it.
How to check it
A published method nobody checks is a certificate with more words
That objection is the strongest one against this page, so here is the answer to it rather than a defence of it. Everything above is checkable against something dated that already exists.
The preregistration carries a DOI and a timestamp that predates the data. The Claim Register carries the dates on which each claim was verified, and the corrections where I got one wrong. The study’s treatment isolation is asserted by a test rather than by this sentence, and a participant can confirm it from their own browser.
If any of those turns out not to hold, the commitment above is broken, and I would rather you found that than took the page at face value.
Receipts
Dated things you can open
What this is not
The limits, stated here rather than discovered later
- Not certification. Nothing here is a badge, and no result entitles anyone to display one.
- Not approval. A published threshold constrains my conclusion; it does not make the conclusion correct.
- Not peer review. A preregistration is a timestamp, not a quality guarantee — a point the Register makes at length about other people’s studies and applies to mine.
- Not a guarantee about your system. An evaluation describes what was measured, on the dates it was measured, under the conditions stated.