Measuring causal effects is a tricky business. But, it’s necessary if we want to appropriately design effective policies and interventions. Many things are not amenable to manipulation in an experiment and so we rely upon a toolbox of statistical tools to try to identify the effect of interest. These methods are often ingenious, finding sophisticated ways of exploiting different types of variation, but they are essentially uninterpretable without an underlying causal theory. To illustrate this, let’s consider a paper that was featured a few weeks ago in the journal round-up: Effect of democratic reforms on child mortality: a synthetic control analysis.
A large number of countries have undergone democratic reform over the last 20 years. This article aimed to estimate how that reform has impacted upon child mortality. To do this a synthetic control method was used.
Synthetic control
The synthetic control method was formalised by Alberto Abadie, Alexis Diamond, and Jens Hainmueller in an article in the Journal of the American Statistical Association. It’s particularly useful in the situation where there is one area or cluster or country that has undergone a change, and multiple potential countries or clusters to act as controls that did not undergo the change. The eponymous synthetic control is a weighted average of the potential control sites where the weights have been chosen to best replicate the pre-intervention trend in the intervention site. The example given by Abadie and colleagues is estimation of the impact of tobacco control reform in California on tobacco consumption. The other US states are all potential controls. Bayesian synthetic control methods have also been established (by a team at Google), which we will make use of later.
The synthetic control method therefore seems appropriate to analyse the impact of democratic reform in a given country. Measurement of democratic reform was based on a change in the Polity2 index that rates the degree of autocracy/democracy in countries; a switch in the index from negative to positive (the index runs from -10 to 10) was considered ‘democratic reform’. Of the 60 countries identified as having reformed, 33 met the inclusion criteria, and 24 counterfactuals were able to be constructed. The primary outcome is the relative reduction in child mortality after ten years; the results from the 24 countries are shown in the histogram below (Figure 1). It would seem that, on average, democratic reform seems to reduce child mortality.
Figure 1. Histogram of results from the 24 included countries from Pieters et al.
Causes
Perhaps one of the factors that have limited research in the area of political economy and health is the complexity of the relationships between the various macro, micro, economic, and political effects. For example, on the basis of the evidence presented above, we would still not be able to say whether, for a given country, introducing democratic reform would have any impact on child mortality. Let’s consider a couple of examples to explore why.
Figure 2. Results from synthetic control analyses of the impact of democratic change on child mortality. Data from UN Inter-agency Group for Child Mortality Estimation.
Mozambique
Mozambique was engaged in a civil war between 1977 and 1992 as the communist Frelimo battled the anti-communist Renamo for control of the country. At the cessation of hostilities in 1993, wide sweeping reforms were enacted by Joaquim Chissano, and an election was held. We can consider 1993 as the year of democratic reform and conduct our own synthetic control analysis (using the aforementioned Bayesian approach). The results are shown in the figure above (Figure 2). Clearly, there is a significant reduction, but is this due to democratic reform or simply the end of civil war? A counterfactual approach is used as the theory of causation behind much statistical inference. Had an autocratic regime followed the civil war would there have still been declines in child mortality? I would conjecture that there would have been. Democratic reform in this case is either not a cause or a redundant cause.
Zambia
Kenneth Kuanda was removed as the president of Zambia in 1991. Following this, reforms for multiparty democracy were enacted. Figure 2 above reports the estimated impact upon child mortality. A decline is clearly evident, however, this decline does not start until 2003, when the copper price tripled (Figure 3 below; copper constitutes 88% of Zambia’s exports) and GDP per capita almost doubled. Again, whether democratic reform can be inferred as a cause in this instance is questionable, especially when reduced to questions of counterfactuals.
Figure 3. Copper price per metric ton (US$). Data from IMF Cross Country Macroeconomic Statistics Database.
South Africa
The apartheid regime was ousted in 1994 following election of the ANC in South Africa. This regime change opened up the political institutions to the majority of South Africans who had previously been excluded. However, as Figure 2 above shows, this reform appeared to have little impact on child mortality. Indeed, South African healthcare still faces significant challenges, and large structural inequalities in access to quality healthcare persist even today. Taken together, this illustrates that democratic reform is not a sufficient condition for improvement of population health.
Zimbabwe
Finally, consider Zimbabwe, which became more autocratic in 1986 following a deal between Zanu and Zapu. If democratic reform improves child mortality then it seems reasonable that autocratic changes would worsen child mortality. Figure 2 above reports the synthetic control results for Zimbabwe. No change is evident.
What can we conclude?
Democratic reform is neither a sufficient nor necessary condition for improvements in child mortality. We cannot understand the evidence without an underlying theory. The study discussed here is a good data analysis; decent analyses in this area should be encouraged. But, the theory should come before the data.
Credits
[…] a past discussion of an article on democratic reform and child health, I concluded that “Democratic reform is neither a […]
[…] reform on under-five mortality. We previously covered this paper in a journal round-up and a subsequent post, for which we also used the Brodersen et al method described above. We recently featured a paper […]
[…] even when measured in a randomised trials, depends on neighbourhood effects. As we have discussed in other contexts, development of a causal theory is clearly required for the appropriate design of […]