The crucial role of good evidence in evidence-based policymaking

Published on 25 October 2013

In a time of continuing fiscal austerity, policymakers increasingly want to know ‘what works’ and for whom, in order to target scarce resources on those who will benefit most and to ensure that policy has the desired impact upon those it is designed for. Basing policy decisions on evidence is undoubtedly a good thing - but only if the evidence used is robust, unbiased and methodologically sound. This observation uses recent IFS work on the link between parents’ marital status and relationship stability and child development to illustrate the challenges of using research to inform policymaking.

In a time of continuing fiscal austerity, policymakers increasingly want to know ‘what works’ and for whom, in order to target scarce resources on those who will benefit most and to ensure that policy has the desired impact upon those it is designed for. Basing policy decisions on evidence is undoubtedly a good thing – but only if the evidence used is robust, unbiased and methodologically sound.

Of particular concern is whether the introduction of a new policy or a change in an existing policy would cause (i.e. directly lead to) a change in the relevant outcome of interest, or whether the two are simply correlated. What do we mean by this? Two factors are causally related if movement in one factor causes a change in the other. For example, heating water causes water to boil. Two factors are correlated if they move together, but one factor doesn’t cause the other. This occurs when another factor (or multiple factors) influences both. For example, higher ice-cream sales are correlated with lower water levels in reservoirs, but greater ice-cream consumption does not cause water levels to fall; both factors are driven by external weather conditions.

This distinction is crucial when thinking about how to use evidence to inform policymaking. It would be very worrying indeed if a government thought it could increase water levels in reservoirs by taxing ice-cream sales. But it is easy to think of many rather less silly examples, particularly in the social sciences, where the relationships between factors are more complicated, and it is harder to say with any degree of certainty whether one causes the other, or whether the two are simply correlated, both driven by one or more other factors.

New IFS research funded by the Nuffield Foundation on the link between parents’ marital status and relationship stability and child outcomes illustrates these challenges. It shows that married parents are, on average, less likely to separate while their child is young than cohabiting parents. It also shows that children born to married parents have slightly higher cognitive and socio-emotional development, on average, than children born to cohabiting couples; they are also less likely to engage in risky and antisocial behaviours, such as underage smoking, drinking and cannabis use. Does this evidence prove that marriage causes improvements in relationship stability and child development?

The short answer is no.

We know that married parents differ from cohabiting parents in many ways other than their marital status (and in ways which we can be confident were determined before they decided whether or not to get married). For example, they are from different ethnic backgrounds, have typically grown up in more advantaged homes themselves, and are more highly educated. To the extent that these factors influence both the likelihood that a couple marries, and the likelihood that they stay together (or raise a child with higher cognitive or socio-emotional development), we can think of these as the ‘external weather conditions’ in the example above: ignoring them gives a misleading impression of the effect of marital status on relationship stability or child development.

Our research shows unequivocally that accounting for these other characteristics is important: once we compare cohabiting and married parents who ‘look the same’ in all respects other than their marital status (i.e. are from the same ethnic background, have grown up in similar home environments, have similar education levels, and so on), we substantially reduce or even eliminate the link between marital status and relationship stability or child development. This suggests that much of the raw relationship is due to the fact that different types of people choose to get married, rather than that marriage has a large causal effect on relationship stability or child outcomes.

There is an important caveat to our findings, however: we are not always able to eliminate the difference in outcomes between married and cohabiting parents using only characteristics that are fixed (such as ethnicity) or observed in childhood, long before marriage decisions were taken. In some cases, we must rely on the inclusion of characteristics (such as household income and housing tenure) that are observed after a couple’s decision to marry or cohabit, and so have the potential to be affected by marriage. Our judgement – and the view of most other academic research on this topic – is that these effects are likely to be small. But as a result, it is possible that we could be understating any potential positive impact of marriage if characteristics beneficial to child development or relationship stability are strongly influenced by the decision to marry.

Having said this, however, even if a statistically significant association does remain between parents’ marital status and relationship stability or child development, this estimate does not necessarily represent the causal effect of marriage on these outcomes. This is because there are likely to be many factors that influence both the likelihood of getting married and a couple’s relationship stability or their children’s development that are difficult to observe and account for in our analysis, but that were apparent before their decision to marry. Good examples might be a couple’s level of commitment to one another, or their attitudes to childrearing. Without access to richer data including these characteristics (or a source of exogenous variation – an external factor that affects the likelihood of getting married), it is impossible to know whether any significant association would remain after accounting for all of the ways in which married and cohabiting couples differ.

Understanding whether and to what extent factors are causally related or simply correlated is crucial from a policy perspective. Policy decisions should ideally be based on evidence of a causal relationship. This is not always possible, however, in which case it is important to be clear about exactly what we do, and don’t, know.

The work we are publishing today on the link between parents’ marital status and relationship stability and children’s outcomes is a good example of where one has to be very careful about the conclusions one is drawing. There is a strong correlation between parents being married and children who are more successful academically and in other ways. But there is not good evidence of a causal link – though we can’t say for sure that such a link does not exist. Researchers must be careful not to interpret or present statistically significant associations as evidence of causation. In turn, policymakers must be cautious about using such associations as a basis for policymaking.

The Nuffield Foundation has funded this project but the views expressed are those of the authors and not necessarily those of the Foundation – www.nuffieldfoundation.org