Newsletter ROI needs a causal comparison before it needs a formula

Do not calculate newsletter ROI from views, clicks, attributed purchases or gross receipts. Define the intervention and estimate what would have happened without it. If the evidence cannot support that comparison, report observed outcomes or contribution instead of a return.

This is a blank evaluation method, not financial advice or a performance promise.

What to take away

Define the decision

Name the newsletter change, eligible adult population, England operating context, start and end dates, exposure rule, revenue route and decision owner. Freeze the analysis plan before reading results. State which payment, refund, labour and customer records will be used and who can access them.

Separate the intervention from simultaneous price, product, advertising or sales changes. Record spillover, non-compliance with allocation, missing data and other explanations. HM Treasury's current Magenta Book distinguishes outcome monitoring from impact evaluation and treats the counterfactual as central to causal inference.

Choose a proportionate comparison

The counterfactual estimates the eligible outcome without the change. Choose the design before launch and state its conditions.

A randomised holdout is credible when you can randomly assign eligible units, protect the allocation, and reach enough sample for useful precision. It estimates the effect of the offer on those units, not on everyone.

Difference-in-differences is credible when the intervention and comparison groups followed parallel trends before the change and no other shock hits one group only. Show the pre-period trend and test for spillover.

Matched control is credible when you match on pre-period contribution, tenure, price and acquisition route, and when hidden differences do not drive the result. Report how matching changed the sample.

Qualified statistical review must set the method, power or precision, exclusions and uncertainty. HM Treasury's quality guidance for impact evaluation explains that monitoring a change does not show the intervention caused it.

Before launch, define the smallest decision-relevant effect and the evidence needed to detect it with useful precision. If the available population is too small or allocation cannot be protected, say that a numerical causal estimate is unlikely to answer the decision. Do not extend the test or change exclusions after seeing a favourable result without labelling the analysis exploratory.

Build the money variables

For each eligible unit and period, reconcile gross receipts, supplier deductions, cash settled, refunds, chargebacks and recognised revenue. Then subtract the directly attributable variable cost under the approved accounting policy.

contribution = recognised eligible revenue - directly attributable eligible cost

Keep taxable profit outside the evaluation worksheet until the accountant applies the correct entity, period and rules. HMRC's business-record guidance illustrates the need for transaction evidence but does not set the policy for every organisation.

Calculate the denominator separately:

full economic cost = platform and payment cost + acquisition + authorised labour + accessibility, privacy, advertising and security assurance + support and reversals + evaluation + migration and exit

Use actual dated costs where available. Apply an approved economic value to internal labour. In practice that means a day rate or hourly cost basis agreed by finance. That basis should include salary, employer National Insurance and pension contributions, not salary alone. Where no approved rate exists, record the gap and leave the labour line blank. Unknowns stay blank.

Indicative UK components for the denominator include the email platform or newsletter tool. Pricing typically runs from free tiers up to a few hundred pounds a month as list size grows. Card processing commonly sits near 1.5% plus 20p per transaction. Standard VAT is 20% where it applies.

Add the labour rate above. Acquisition spend, accessibility and privacy assurance, support and reversals, evaluation time, and migration or exit costs are added where they apply.

Apply the blank ROI formula

incremental contribution = observed contribution - estimated counterfactual contribution

ROI = incremental contribution / full economic cost of the defined intervention

State currency, price basis, VAT treatment, population, unit and period. Report an interval or other suitable uncertainty measure beside the point estimate. Do not call a platform-attributed conversion incremental.

Worked example, illustrative only. Eligible units show observed contribution of £12,000. A matched control gives a counterfactual estimate of £9,500. Incremental contribution is £2,500. Full economic cost is £1,800. Point ROI is about 139%. With uncertainty bounds of £1,500 to £3,300 incremental contribution, the ROI interval is about 83% to 183%.

Show the result as a calculation trail, not a decorative percentage. Publish eligible units, missing records, exclusions, observed contribution, counterfactual estimate, difference, full economic cost and uncertainty in that order. Suppress or aggregate personal data where required, and limit access to the event-level file. The privacy review remains independent of the statistical sign-off.

Test sensitivity and decide

Vary the counterfactual, missing-data rule, allocation compliance, cost valuation, refund timing and attribution window within justified bounds. Show which assumptions reverse the decision. HM Treasury's evaluation transparency guidance calls for accessible uncertainty and robustness reporting.

Set the decision rule in advance: continue, investigate, pause or stop. A positive estimate cannot override privacy, consumer, advertising, accessibility, security or tax failures. Retire the result when the offer, audience, platform, payment route, price, comparison design or economic-cost basis changes. If uncertainty remains too wide for the decision, say so rather than publishing a precise return.

Archive the analysis plan, code or query version, source extracts, reconciliation log and reviewer decisions. A later rerun must use the frozen method or explain every change. This protects the decision from silent revisions and allows a qualified analyst to reproduce or challenge the estimate.

Before you act

Common questions

What should be reported if no credible counterfactual is feasible?

A documented contribution analysis can test a causal chain but may not produce a defensible effect size. If the available population is too small or allocation cannot be protected, say that a numerical causal estimate is unlikely to answer the decision.

How is the ROI formula set out?

Incremental contribution equals observed contribution minus estimated counterfactual contribution. ROI equals incremental contribution divided by the full economic cost of the defined intervention. State currency, price basis, VAT treatment, population, unit and period. Report an interval or other suitable uncertainty measure beside the point estimate.

What must be varied in sensitivity testing?

Vary the counterfactual, missing-data rule, allocation compliance, cost valuation, refund timing and attribution window within justified bounds. Show which assumptions reverse the decision. A positive estimate cannot override privacy, consumer, advertising, accessibility, security or tax failures.