Ex post evaluation of health interventions

It is not enough: an indicator improving after an intervention does not show that the improvement was due to it, because it may have come from a trend already under way, from seasonality or from a simultaneous change. This applies to any programme or policy already in place that you want to evaluate on a before-and-after basis. Comparing the value before with the value after credits the intervention with everything that changed in that period, including what would have changed anyway without it. Ex post evaluation of interventions answers the right question: not whether the indicator improved, but whether it improved more, or differently, than it would have without the intervention. To answer it, an explicit counterfactual scenario is built, what would have happened without the intervention, and the effect is estimated as the difference from that scenario, not from the starting point.

Ex post evaluation studies an intervention that has already been introduced; ex ante modelling estimates what might happen before it is implemented. The first uses observed data to estimate the effect attributable to the intervention against a counterfactual; the second combines prior evidence and assumptions to project plausible scenarios. If the intervention is under way or complete and you want to know whether it worked, for whom and by how much, you need ex post evaluation. If you are still deciding whether to introduce it, how to design it or what impact to expect, you need ex ante modelling of interventions. The methods may overlap, but the decisions differ: one evaluates what happened; the other helps you decide what to do before it happens.

At a minimum, we need the implementation date and outcome data from before and after it. The rest depends on the design: a sufficiently long historical series for interrupted time series, comparable units for difference-in-differences, donor units for the synthetic control method, or enough covariates to balance the groups. The data do not have to arrive analysis-ready, but there must be a pre-intervention baseline and some defensible source of comparison. We also need to know how the intervention was introduced: whether implementation was gradual, whether other measures were introduced at the same time, and whether the recording system changed during the period. The client provides the data and that context; we define the time frame, construct the comparison and assess the plausibility of the assumptions before estimating the effect. If a credible counterfactual cannot be supported, we will say so plainly: we may still describe what changed, but we cannot attribute that change causally to the intervention.

The counterfactual is not observed; it is estimated, from the information available on how the unexposed units, or the exposed unit itself before the intervention, were evolving. This applies to every quasi-experimental design, which is precisely the family of methods built to reconstruct that unobserved scenario. Difference-in-differences compares the trajectories of exposed and unexposed units when both exist, or when exposure is staggered over time. Interrupted time series project a unit's pre-intervention trend to see whether it breaks after the intervention. Synthetic control builds a comparison unit by combining several donor units that reproduce the exposed unit's behaviour before the intervention. Each rests on a key assumption, whether parallel trends, continuity of the trend or the validity of the synthetic combination, which must be defensible and is made explicit. If those assumptions are not defensible, the design has to change or the causal conclusion has to be scaled back. The counterfactual is an estimate resting on assumptions, not a fact, and its credibility depends on how defensible those assumptions are.

It is controlled by choosing the method according to how the intervention was assigned: adjustment or balancing on observed characteristics, or quasi-experimental methods when there is an eligibility threshold or a source of exogenous variation, because those who receive an intervention are rarely comparable at the outset to those who don't. This applies to any evaluation where assignment was not random, which is the norm in public health. Depending on the assignment mechanism, propensity scores (matching or weighting) are used when there are enough covariates to improve comparability; regression discontinuity when an eligibility threshold separates those who get access from those who don't; or instrumental variables when there is a defensibly exogenous source of variation in access or exposure to the intervention. The choice is not a matter of habit: it depends on what the data and the real assignment mechanism can defend. The goal is that the difference estimated between groups does not simply reflect one that already existed before the intervention. It is worth remembering that these methods adjust for observed confounding, and the assumptions of each are made explicit and put to the test.

An effect estimate is more defensible when it holds up under tests of the design's key assumption and reasonable changes to the specification, not when it appears in a single analysis. This applies to any quasi-experimental causal estimate, where a figure without sensitivity analysis is a claim, not evidence. That is why the assumption each design depends on is put to the test, such as parallel trends in difference-in-differences, the absence of simultaneous co-interventions or the validity of the synthetic control, and the report sets out how the result changes under alternative specifications. If the effect holds under different reasonable assumptions, the conclusion gains credibility; if it depends heavily on one particular choice, that is reported rather than hidden. The estimate comes with its degree of confidence and its limits made explicit, not with a certainty the design cannot support.

Ex post evaluation of interventions is causal inference applied to a public health intervention or policy that has already been implemented: the object is a specific programme or law, and the question is what attributable effect can be estimated for it. The causal inference and observational study design service is the method in its broad sense, applicable to any causal question without an experiment, inside or outside the field of public health interventions. The relationship is that of an applied use case to a general methodological framework: ex post evaluation is causal inference applied to evaluating an implemented intervention, with the vocabulary and designs of that field (difference-in-differences, interrupted time series, synthetic control). If what you have is a specific intervention or policy already under way whose effect you want to attribute, it belongs in ex post evaluation; if it is a causal inference question in another type of data or context, the cross-sector causal inference service is the better fit.

You receive an estimate of the effect attributable to the intervention against the counterfactual constructed, with the quasi-experimental design justified, the confounding control applied and a sensitivity analysis of the critical assumptions, plus effects by subgroup, area or level of exposure when the data allow. This applies if you need to support a decision about an intervention already under way: to continue it, expand it, adjust it or discontinue it. The deliverable lets you make the case for that decision to a funder or a decision-making body with a causal estimate of the attributable effect, conditional on the design and its assumptions, not with the raw trend in an indicator. The client implements the intervention and makes the policy decision; the service estimates the effect that can reasonably be attributed to the intervention and provides the evidence, with its assumptions and degree of confidence made explicit, so that the decision rests on a defensible basis.