The Split Decides the Result: Temporal Validation in Under-Five Mortality Prediction
June 16, 2026
The same model, the same features, the same preprocessing. Three different answers, depending only on how I split the data.
Most machine-learning papers on child mortality in low- and middle-income countries follow a familiar recipe. Take a Demographic and Health Survey (DHS) round, split it 80/20 at random, train a few classifiers, report the AUROC. I have written papers like that. The trouble is that a random split answers a question nobody planning a health programme is asking.
A programme manager wants to know something closer to this: if I build a model on the surveys I already have and use it on the next cohort of children, how many of the children who will die can I find, and how many families will my health workers have to visit to find them?
That is a question about the future. So I tested it against the future.
The setup#
I used four rounds of the Bangladesh DHS (2011, 2014, 2017 and 2022). After eligibility restrictions that gave 33,962 children and 1,290 under-five deaths. I held everything fixed except the validation design: one 26-feature pipeline, with preprocessing fit on each training split only, and three model classes (a small neural network found by a genetic-algorithm architecture search, XGBoost, and L2-penalised logistic regression).
Then I evaluated them under four regimes:
| Regime | Train | Test |
|---|---|---|
| Pooled random | 80% of all rounds mixed together | the other 20% |
| Matched-size pooled random | same, but training set cut to the temporal size (n = 14,380) | same pooled test set |
| 2022-only random | 80% of 2022 | 20% of 2022 |
| Cross-round temporal | 2011 + 2014 (2017 used only for selection and calibration) | all of 2022 |
The temporal design is the one that mimics deployment. The model is built on older surveys, tuned on a middle round, and tested on a round it has never touched. I deliberately kept 2017 out of the training set so that architecture selection and Platt calibration could happen on data that did not leak into the test round.
What changed, and what did not#
For the neural network, AUROC ranged from 0.669 (2022-only random) to 0.775 (pooled random). The temporal estimate sat in between at 0.730.
That spread of 0.106 is large. For comparison, within any single regime the three model classes differed by at most 0.015. Across 13 architectural variants in the ablation, prospective AUROC moved by less than 0.01. If you care about the number you report, the validation design matters several times more than the choice between a neural network, gradient boosting, and logistic regression, and about ten times more than architectural details.
Two details surprised me.
First, pooled random validation over-stated the temporal AUROC by 0.046, and cutting its training set down to the temporal size only reduced that to 0.038. So the optimism is mostly not about having more training data.
Second, the single-round bias does not have a fixed direction. The 2022-only split under-stated temporal performance. When I repeated the single-round design on 2011 and on 2014, both came out above the temporal estimate (0.768 and 0.753). My reading is that in the earlier, higher-mortality rounds the within-round structure is strong and easy to learn, while by 2022 it has weakened. A single-round random split tells you about that round’s internal structure, and nothing about how a model transfers.
The part I had to correct myself on#
My first draft said, in effect, “random splits make models look better than they are.” That is too strong, and a paired analysis showed why.
The regimes do not share the same test rows. To separate the effect of what the model learned from the effect of which children it was tested on, I restricted each pair of regimes to their overlapping test rows and ran a paired bootstrap of the AUROC difference. On the shared rows the differences mostly disappeared. Temporal versus pooled random on 2,387 shared rows: +0.006, with a 95% interval from -0.012 to +0.022. Temporal versus 2022-only on 2,308 shared rows: -0.011, interval -0.065 to +0.041.
So the models themselves are not very different. The headline AUROC gaps are largely a property of test-set composition.
Does that make the whole exercise pointless? I don’t think so, and this is the actual lesson. A screening programme is also evaluated on a particular population. When the test population changes, the number a planner reads off the report changes with it, even if the model does not. The question is which population your reported number describes. For a forward-looking programme, the honest answer is a later survey round, not a shuffled mix of past ones.
Translating AUROC into workload#
AUROC is a poor planning metric. Nobody screens “at AUROC 0.73”. Health programmes work at fixed capacity: we can follow up this many households.
So I fixed capacity at the top 10% of predicted risk and reported what a planner would need.
Under temporal validation, on the 2022 round, the network flagged 1,154 children and caught 152 of the 355 observed deaths. That is a sensitivity of 42.8%, a positive predictive value of 13.2%, and a number needed to screen (NNS) of 7.6: roughly one death identified for every eight children followed up.
The same model under pooled random validation implied an NNS of 5.6. Under 2022-only random validation it implied 11.0. Plug those into a budget for community health worker visits and you get three quite different programmes.
At the same threshold, XGBoost caught 149 deaths and logistic regression 152. Neither difference from the network was significant (exact McNemar p = 0.55 and 1.00). Decision curve analysis agreed: all three gave about the same net benefit at the 5% threshold, which corresponds to roughly the top-10% capacity. I want to be clear that this paper is not evidence that architecture search helps on tabular survey data. It found a reasonable small network. Logistic regression did just as well.
Limits#
A few things I would want a reader to weigh before using these numbers.
- Censoring. DHS records children born in the five years before the survey, so many test children had not yet been observed to age five. AUROC was stable when I restricted to children observed for at least 12, 24 and 36 months. At 48 months it dropped to 0.662, with an interval that no longer overlapped the primary estimate. For predicting deaths after infancy, the model is weaker than the headline suggests.
- Most deaths are early. 71.0% of under-five deaths in this sample were neonatal and 90.9% occurred in the first year. A model like this is mostly a model of early-life risk.
- Recall-based outcomes. Deaths come from maternal birth histories, with the usual recall and reporting issues.
- One country. This is Bangladesh, four rounds. The direction of the single-round bias already changed between rounds, so I would not assume the specific numbers transfer elsewhere without a temporal replication.
- Prediction is not intervention. Identifying high-risk children says nothing about whether any follow-up visit changes their outcome. That needs a different study design.
What I would ask of papers in this area#
If you publish a DHS-based mortality model, I think three things would make it far more useful to someone who has to act on it:
- Report a temporal split across survey rounds, even as a secondary analysis, and say which population your headline number describes.
- Report capacity-based metrics (sensitivity, PPV and NNS at a stated screening fraction) alongside AUROC.
- Compare against logistic regression under the same pipeline. If the fancy model does not beat it, that is worth saying.
None of this is new to epidemiologists who work with prediction models. TRIPOD and PROBAST have asked for most of it for years. It is still rare in the machine-learning papers that use these surveys, and the gap between 0.669 and 0.775 is a good illustration of why it matters.
The full paper, with the paired-bootstrap tables and the censoring analysis, is on arXiv, and there is a short summary on the publication page.