Measurement
Part of Newsletter measurement that keeps permission, attention, customers and cash apart
Judging a newsletter benchmark: accept, reject or use as context
Judge a newsletter benchmark with a source card and a three-way verdict, worked through Mailchimp, Reuters Institute, OSR and AQuA Book examples.
What to take away
- Every candidate source gets one of three verdictsaccept, use as context only, or reject.
- A source card records who published a figure, for whom and from which population before anyone quotes it.
- A platform's customer figure stays labelled as platform customers, and a UK or global result is not an England result.
- A benchmark that backs a marketing claim needs adequate documentary evidence under CAP Section 3.
- "No defensible external benchmark found" is a valid result.
The three verdicts
Give each candidate benchmark one of three verdicts. Accept it when population, event, denominator, geography and period are alike enough for the decision owner to defend. Use it as context when the differences stay visible and no threshold depends on it. Otherwise reject it.
Reject a source on any of these grounds:
- A definition is missing.
- A UK or global population is presented as England.
- Incompatible figures have been averaged.
- Supplier marketing offers no inspectable methods.
Write the verdict and the reason on the source card, so the next person does not repeat the search. A rejected source is still worth keeping on file, because it explains why the team is using an internal baseline instead.
The source card
A source card is a short record made before any figure is quoted. For each possible reference, capture the publisher, publication and field dates, original purpose, population, geography, coverage, event collection, denominator, exclusions, weighting, revisions and uncertainty. Keep it to one page.
Save the exact table or documentation version rather than only a headline. Benchmark pages change, and a later reader needs the version that informed the decision, together with the date it was retrieved, so that a revision can be spotted.
Four sources, four verdicts
The table runs four real sources through the card. None gives an England threshold, and the verdicts show how different the permitted uses are.
| Source | What it can support | What it cannot support | Verdict |
|---|---|---|---|
| Mailchimp Email Marketing Benchmarks | Context for average open rate | England paid-access entitlements; covers only Mailchimp customers | Context if no threshold depends on it; reject for paid access |
| Reuters Institute Digital News Report | Context for England news behaviour | Open events, which it does not measure | Context only; reject as an open-rate benchmark |
| OSR Code of Practice for Statistics | A way to expose source quality and intended use | Making newsletter analysis official | A method, not a benchmark |
| AQuA Book | Questions about assurance and uncertainty | Commercial email; its scope is government analysis | A method, not a benchmark |
Mailchimp's public Email Marketing Benchmarks show why a platform figure needs a label. They cover Mailchimp customers who meet the platform's reporting thresholds, not all England newsletter operators. Whether verified first paid-access entitlements per eligible checkout are changing cannot be answered from open rates at all.
The Office for Statistics Regulation has said the Code of Practice for Statistics will be based around the three pillars of trustworthiness, quality and value. Newsletter analysis does not become official by borrowing it. The practical lesson is to expose source quality and intended use on the card.
The government's AQuA Book says effort should be appropriate to the risk of the intended use of the analysis. Its scope is government analysis, not commercial email, but the question carries over: is this source precise yet unfit for the decision?
Two traps to check on every card
Check whether sent, delivered, open and click describe the same events in both datasets. RFC 5321 is a transport standard, not a definition of human readership. Apple's UK Mail Privacy Protection page says remote content can download in the background, which limits how an open event can be read.
Open metrics therefore depend on client mix, background downloads and bot filtering. The same trap catches many of the measurement mistakes that make two studies look comparable when they are not.
Money figures need more than a currency conversion. Work out whether a figure is gross receipts, payment-authorised value, settled cash, contribution or taxable profit, then match refund, dispute, supplier-fee, VAT and time boundaries. HMRC's self-employed record guidance distinguishes income and expenses, and qualified accounting and tax reviewers decide the operator's treatment.
When a benchmark backs a claim
If a benchmark informs a marketing claim, CAP Section 3 requires adequate documentary evidence for objective claims. The source card is part of that trail, so file it with the claim. A rejected source should not appear in the copy at all.
If person-level records support the work, the ICO's data-minimisation guidance is a separate privacy gate. The subscriber data rules set out what may be kept and for how long, before any record is joined to an external figure.
The valid conclusion may be "no defensible external benchmark found". That is more useful than an incompatible threshold, because it points to better definitions. Use a stable internal baseline with a version break when definitions change, and keep comparable periods.
Before you act
- Open a source card for each candidate figure.
- Record a verdict and a reason on every card.
- Keep UK, global and platform-only figures under their own labels.
- Match event, refund, dispute, fee, VAT and time boundaries.
- File the card with any claim that relies on the figure.
Common questions
When is "use as context" enough?
When the differences from your case stay visible and no decision threshold depends on the figure. Context can inform a question, but it cannot set a target.
Can a UK result stand in for an England result?
No. A UK result should stay labelled UK, and a supplier's global aggregate should remain global. Only a rule you can state, such as a billing address for one commercial cohort, supports an England label.
What does a source card add to a marketing claim?
It records the evidence behind the figure. CAP Section 3 requires adequate documentary evidence for objective claims, and the card shows population, period and limits in one place.
What if no source passes?
Record that no defensible external benchmark was found, then use a versioned internal baseline. Internal history still needs comparable periods and source checks.
