Health surveys

Not if the survey uses a complex sampling design, which is the norm for health surveys: they stratify, sample in clusters and apply weights. This applies to any survey that is not a simple random sample, from your own study to an existing national or regional survey. Calculating the raw percentage, as if each person entered the sample independently and with the same probability, can cause two problems: a biased estimate, because it ignores that some profiles carry more weight than others in the population, and confidence intervals that are too narrow, suggesting a precision the sampling does not support. The correct analysis incorporates the declared design, with its strata, clusters and weights, so that the estimate represents the target population and the reported uncertainty is consistent with the sampling design, not an optimistic figure that does not hold up.

Analysing a survey with its design means building into the calculation the strata, clusters and weights used to construct the sample, instead of treating it as a set of independent, equally likely observations. This applies to every survey with complex sampling. It changes the result on two levels. In the point estimates, because the weights correct for some profiles being over- or under-represented relative to the population. And in the precision, because clusters usually reduce the effective information: two people from the same household or census section are more alike than two people chosen at random, and that widens the confidence intervals compared with what a simple random sample would give. Ignoring the design is not a minor technical nuance: it can change statistical significance or the width of the confidence intervals, or shift an estimated prevalence. That is why the sampling design does not stay in the fieldwork: it shapes every number the analysis produces.

It is not worse. The intervals are wider because they reflect the uncertainty the design imposes: not because the method is worse, but because it stops assuming a precision the design does not justify. This applies to any survey with clusters, where the default calculation gives a false precision. Clustering reduces the effective information in the sample: if two people from the same household tend to be alike, each additional person from the same cluster adds less new information than one chosen at random, and the "effective" sample size is smaller than the nominal one. Design-adjusted variances, calculated by Taylor linearisation or replication methods, capture that design effect, so that the confidence interval expresses what the survey actually allows you to claim. A narrow interval calculated by ignoring the design is not more precise: it is misleading. The right width is the uncertainty consistent with the design, the one that protects you from concluding more than the sampling supports.

Differential non-response is addressed through weighting and calibration, so that the estimate better represents the target population and not just those who responded. This applies whenever some profiles respond less and end up under-represented, which is the usual situation. The process starts from weights based on the inverse probability of selection, which reflect how much each person counts given how they entered the sample. It adjusts them for non-response, to reduce the observable under-representation of the groups that responded less, and calibrates them against known population totals, such as the distribution by age, sex or area, to align the sample with the population. The scope needs to be stated precisely: weighting corrects the observable imbalance on the calibration variables available, but it cannot be relied on to remove all non-response bias if non-respondents differ in ways that were not recorded. It is a defensible, standard correction, not a promise of perfect representativeness, and its assumptions are made explicit.

You need the survey data and, critically, the variables that declare its sampling design: the stratum identifier, the cluster identifier and the weights, plus documentation of how the sample was constructed. This applies to any analysis of a complex sampling survey, whether your own or an existing one. Without those design variables, a complex survey cannot be analysed correctly, because the variance calculation needs to know how cases were clustered and weighted. The client provides the data and the design information, or, for an official survey, this usually comes documented; the service declares the design correctly, checks that the weights are consistent, assesses the pattern of non-response and prepares the calibration before estimating. The client does not need to know how to handle that design, but the information does need to exist: if a survey was collected with a complex design but does not retain the variables that declare it, recovering the design after the fact may be limited or impossible, depending on the documentation available.

The sample size of a cluster survey is calculated by building in the expected design effect, not with the simple random sampling formula, which can leave the survey undersized if the design effect is left out. This applies to the design of a new survey, before fieldwork. A cluster survey usually needs a larger sample than a simple random sample to reach the same precision, because the effective information per individual is lower; calculating the size with the simple formula can leave the survey short. The calculation starts from the desired precision and the design effect expected from the cluster structure, and pays particular attention to the breakdowns that matter: if estimates will be made by subgroup or by area, the sample size must deliver sufficient precision in those partitions, not just in the total. That way the survey does not fall short exactly where the estimate is needed. The service sizes and designs the sampling; fieldwork belongs to the client or the organisation running the survey.

A survey is useful when the administrative record does not capture the variable well or only includes people who come into contact with the system, with the utilisation bias that implies. If you need to estimate prevalence across the whole population, including people who do not seek care, receive a diagnosis or use the service, probability sampling provides a better basis, provided the sampling frame and response are adequate. An administrative record may be enough when it covers the target population reasonably well and contains the variable with sufficient quality and continuity. The two sources can also be combined: records can help calibrate a survey, while the survey can reveal which groups are under-observed in the record. The choice depends on whom each source represents and the question you need to answer.

You receive population estimates with the design built in: proportions, means and regression models with design-adjusted standard errors, confidence intervals calculated by Taylor linearisation or replication, and breakdowns by demographic subgroup or area, with a check that precision remains sufficient in each partition. This applies to anyone who needs valid estimates of prevalence, risk factors or service use in a population, or an analysis of a national or regional survey that is comparable over time. The deliverable documents the declared design and the weights and calibration applied, so that the estimates are traceable and reproducible with each new wave on the same methodological basis. The client or the organisation collects the field data and makes the policy decisions; the service designs the sampling when the survey is new, analyses it with its design and delivers weighted, design-adjusted estimates with uncertainty consistent with the sampling.