Follow us
Publications Commentary Research People Events News Resources and Videos About IFS
Home Publications Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies

Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies

Cemmap Working Paper CWP56/18

Currently there is little practical advice on which treatment effect estimator to use when trying to adjust for observable differences. A recent suggestion is to compare the performance of estimators in simulations that somehow mimic the empirical context. Two ways to run such ‘empirical Monte Carlo studies’ (EMCS) have been proposed. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real-world setting where estimator performance is known. We find that in our setting both EMCS approaches are worse than random at selecting estimators which minimise absolute bias. They are better when selecting estimators that minimise mean squared error. However, using a simple bootstrap is at least as good and often better. For now researchers would be best advised to use a range of estimators and compare estimates for robustness.

More on this topic

Cemmap Working Paper CWP28/20
This paper is concerned with learning decision makers’ preferences using data on observed choices from a finite set of risky alternatives.
Cemmap Working Paper CWP29/20
We investigate state-dependent effects of fiscal multipliers and allow for endogenous sample splitting to determine whether the US economy is in a slack state.
Cemmap Working Paper CWP27/20
This paper provides a method to construct simultaneous confidence bands for quantile functions and quantile effects in nonlinear network and panel models with unobserved two-way effects, strictly exogenous covariates, and possibly discrete outcome variables.
Cemmap Working Paper CWP25/20
This paper demonstrates the use of bounds analysis for empirical models of market structure that allow for multiple equilibria.
Cemmap Working Paper CWP24/20
This paper evaluates the dynamic impact of various policies, such as school, business, and restaurant closures, adopted by the US states on the growth rates of confirmed Covid-19 cases and social distancing behavior measured by Google Mobility Reports, where we take into consideration of ...