Measurement
Part of Newsletter measurement that keeps permission, attention, customers and cash apart
Six reconciliation tests and a break marker for a newsletter dashboard
Six reconciliation tests on subscriber, send and payment records, plus a break marker format that shows where a newsletter dashboard definition changed.
Reconcile the data before anyone reads a tile, and version every definition so a change never hides inside a trend. A newsletter dashboard can look finished while its subscriber, send and payment records are duplicated, mis-ordered or missing. Six tests and one marker format keep those faults visible.
What to take away
- Run six reconciliation tests on every refresh, before anyone interprets a number.
- A failed or empty input stays visible on the control view. It is never converted into zero.
- A change to a source schema, bot filter, permission rule or metric formula gets a new definition version.
- Mark the change with a labelled line on the time axis, and keep the old calculation.
- The dashboard is not the statutory record by default. The accountant decides how it connects to the books.
Six tests on a newsletter's records
Run these tests before anyone reads a tile. Each one targets a way that sound-looking numbers go wrong, and each result belongs on one of the dashboard's views, where a failure is plain to everyone who uses it.
| Test | Newsletter example | Where the result shows |
|---|---|---|
| Duplicate event IDs | A payment notification delivered twice, so one new paid subscriber is counted twice | Control view |
| Impossible state order | A paid subscription that ends before it starts | Customer view |
| Mismatched currencies | Pound subscriptions and a dollar sponsorship invoice added into one revenue tile | Money view |
| Missing timestamps | A signup with no date, so it falls into no month or cut-off | Control view |
| Excluded recipients returning | Unsubscribed or suppressed readers back in the send list after an import | Eligibility view |
| Processor totals that do not bridge | A payout total that matches neither the bank deposit nor the accounting entry for the same cut-off | Money view |
The first four are mechanical and can run on every refresh without a person. The last two need a decision about what counts as a match, because they depend on the cut-off and on the records of the provider you contract with.
These tests assume each measure already has a definition. The measurement guide covers how to write one, so a test has something to check the numbers against.
Keep every failure on show
A failed test stays on the first page until someone clears it. The control view shows refresh status, failed tests, missing sources and unresolved reconciliation items, so no reader can take a tile at face value while a check beneath it is red.
Empty or failed inputs remain visible; they are not converted into zero. A zero says something was measured and came to nothing, while a gap says the number is unknown. A manager who reads a zero may act on it as if it were a fact.
Late-arriving data and restatements get the same treatment. Mark them on the tile, and show both numerator and denominator for any rate. A reader can then see when a figure has moved and which side of the fraction moved it.
Do not rename a supplier's accepted or delivered state as a read. That is a common measurement mistake, and a failed test is the place to catch it.
What a break marker looks like
A break marker is a labelled vertical line on the time axis at the date a definition changed. Its label carries the new version number and a one-line reason, so a reader sees the seam before drawing a trend across it.
Take an open-rate chart. A bot filter is added in the middle of the year, so opens before and after the change are counted differently. The chart shows a dashed line at that date labelled "Opens v2: bot filter added", and the v1 calculation stays on file to rerun on the same data.
Issue a new definition version whenever a source schema, bot filter, permission rule or metric formula changes. The old version stays on file, so a chart never mixes two ways of counting under one label without saying so.
- Write the new definition with its owner and the reason for the change.
- Issue it as a new version and leave the previous version untouched.
- Add a break marker on every chart that uses the metric.
- Keep the old calculation so that it can be rerun on the same data.
Do not splice series unless a documented restatement is valid. Anyone comparing the months before and after a change then sees the seam instead of a false trend. Name a definition owner for each metric, as the team roles plan does for the wider work.
Borrow the AQuA Book's logs for decision-critical views
The Government's AQuA Book recommends data, assumption and decision logs plus proportionate verification and validation for public-sector analysis. This is transferable quality guidance, not a rule for private newsletters. Use it as a model, not as a duty.
The AQuA Book describes a decisions log as a register of decisions that comes from the commissioner or from the analysis. For a newsletter, the same habit gives each continue, investigate or pause call a dated entry beside the view that supported it.
Require a second-person check, a rerunnable query and a stated lifespan for each decision-critical view. A view that nobody can rerun, or that has outlived the decision it served, should be retired rather than left for someone to quote.
Where the dashboard stops and the books begin
A dashboard is not the statutory record by default. GOV.UK says you must keep records for business income and outgoings if you are self-employed. Let the accountant and tax adviser decide how reporting connects to them.
Stripe's first-party documentation separates refund outcomes from disputes, so apply the states of the provider you actually contract with and preserve unknowns. That is also what lets the processor-total test bridge to bank and accounting entries by currency and cut-off.
Ask whether an aggregate answers the reporting question before importing recipient-level histories. The data protection guide covers the duties that apply once personal records enter a view.
Before you act
- Run the six reconciliation tests on every refresh.
- Show failed and empty inputs on the control view, never as zero.
- Issue a new definition version for every schema, filter, rule or formula change.
- Add a labelled break marker and keep the old calculation.
- Give each decision-critical view a second-person check, a rerunnable query and a stated lifespan.
- Ask the accountant how the dashboard connects to the books.
Common questions
Which reconciliation tests should run on every refresh?
Six: duplicate event IDs, impossible state order, mismatched currencies, missing timestamps, excluded recipients returning to the send population, and processor totals that do not bridge to bank or accounting records.
What goes on a break marker?
A dashed vertical line at the date of the change, labelled with the new definition version and a one-line reason. The previous calculation stays on file so both can be rerun on the same data.
Why show a failed input instead of a zero?
Because a zero reads as a measured result. A failed or empty input is an unknown, and hiding it behind a zero lets a decision rest on a number nobody measured.
Is the dashboard my tax record?
Not by default. GOV.UK expects self-employed businesses to keep records of business income and outgoings, so the books remain the record and the dashboard reports from them.
