Can machine learning on tax records match randomized trials' accuracy?
Core argument: Chetty, Katz, and co-authors report a double/debiased machine learning estimator, built on tax records for 327 million U.S. adults, correlates 0.93 with estimates from 37 RCTs and has an RMSE of $2,073 versus $2,126 for the RCT estimates themselves.
We train gradient-boosted trees [a sequence of branching prediction models, each correcting errors left by the earlier model] on the population of non-participants; [e.g., were prior earnings below $20,000? If yes, follow one branch; if no, follow another] to predict future earnings based on a rich set of pre-program individual characteristics: work histories, demographics, and residential locations. We then estimate treatment effects by comparing participants’ realized outcomes to the predicted counterfactuals. Finally, we combine the outcome and propensity score models using the double/debiased machine learning (DML) estimator. We use 37 randomized controlled trials (RCTs) to estimate the distribution of their bias. We use the term “digital twins” for an estimator because it yields counterfactuals — statistical twins constructed from data on non-participants — as accurate as those one would obtain from random assignment. Figure 2 plots the RCT estimates against the DML estimates. The estimates are clustered around the 45-degree line, with an unweighted correlation of 0.93. Our method reduces RMSE by 60% relative to parametric estimators used in prior work.

