Method of the month: Synthetic control

Once a month we discuss a particular research method that may be of interest to people working in health economics. We’ll consider widely used key methodologies, as well as more novel approaches. Our reviews are not designed to be comprehensive but provide an introduction to the method, its underlying principles, some applied examples, and where to find out more. If you’d like to write a post for this series, get in touch. This month’s method is synthetic control.

Principles

Health researchers are often interested in estimating the effect of a policy of change at the aggregate level. This might include a change in admissions policies at a particular hospital, or a new public health policy applied to a state or city. A common approach to inference in these settings is difference in differences (DiD) methods. Pre- and post-intervention outcomes in a treated unit are compared with outcomes in the same periods for a control unit. The aim is to estimate a counterfactual outcome for the treated unit in the post-intervention period. To do this, DiD assumes that the trend over time in the outcome is the same for both treated and control units.

It is often the case in practice that we have multiple possible control units and multiple time periods of data. To predict the post-intervention counterfactual outcomes, we can note that there are three sources of information: i) the outcomes in the treated unit prior to the intervention, ii) the behaviour of other time series predictive of that in the treated unit, including outcomes in similar but untreated units and exogenous predictors, and iii) prior knowledge of the effect of the intervention. The latter of these only really comes into play in Bayesian set-ups of this method. With longitudinal data we could just throw all this into a regression model and estimate the parameters. However, generally, this doesn’t allow for unobserved confounders to vary over time. The synthetic control method does.

Implementation

Abadie, Diamond, and Haimueller motivate the synthetic control method using the following model:

y_{it} = \delta_t + \theta_t Z_i + \lambda_t \mu_i + \epsilon_{it}

where y_{it} is the outcome for unit i at time t, \delta_t are common time effects, Z_i are observed covariates with time-varying parameters \theta_t, \lambda_t are unobserved common factors with \mu_i as unobserved factor loadings, and \epsilon_{it} is an error term. Abadie et al show in this paper that one can derive a set of weights for the outcomes of control units that can be used to estimate the post-intervention counterfactual outcomes in the treated unit. The weights are estimated as those that would minimise the distance between the outcome and covariates in the treated unit and the weighted outcomes and covariates in the control units. Kreif et al (2016) extended this idea to multiple treated units.

Inference is difficult in this framework. So to produce confidence intervals, ‘placebo’ methods are proposed. The essence of this is to re-estimate the models, but using a non-intervention point in time as the intervention date to determine the frequency with which differences of a given order of magnitude are observed.

Brodersen et al take a different approach to motivating these models. They begin with a structural time-series model, which is a form of state-space model:

y_t = Z'_t \alpha_t + \epsilon_t

\alpha_{t+1} = T_t \alpha_t + R_t \eta_t

where in this case, y_t is the outcome at time t, \alpha_t is the state vector and Z_t is an output vector with \epsilon_t as an error term. The second equation is the state equation that governs the evolution of the state vector over time where T_t is a transition matrix, R_t is a diffusion matrix, and \eta_t is the system error.

From this setup, Brodersen et al expand the model to allow for control time series (e.g. Z_t = X'_t \beta), local linear time trends, seasonal components, and allowing for dynamic effects of covariates. In this sense the model is perhaps more flexible than that of Abadie et al. Not all of the large number of covariates may be necessary, so they propose a ‘slab and spike’ prior, which combines a point mass at zero with a weakly informative distribution over the non-zero values. This lets the data select the coefficients, as it were.

Inference in this framework is simpler than above. The posterior predictive distribution can be ‘simply’ estimated for the counterfactual time series to give posterior probabilities of differences of various magnitudes.

Software

Stata

  • Synth Implements the method of Abadie et al.

R

  • Synth Implements the method of Abadie et al.
  • CausalImpact Implements the method of Brodersen et al.

Applications

Kreif et al (2016) estimate the effect of pay for performance schemes in hospitals in England and compare the synthetic control method to DiD. Pieters et al (2016) estimate the effects of democratic reform on under-five mortality. We previously covered this paper in a journal round-up and a subsequent post, for which we also used the Brodersen et al method described above. We recently featured a paper by Lépine et al (2017) in a discussion of user fees. The synthetic control method was used to estimate the impact that the removal of user fees had in various districts of Zambia on use of health care.

Credit 

Sam Watson’s journal round-up for 30th October 2017

Every Monday our authors provide a round-up of some of the most recently published peer reviewed articles from the field. We don’t cover everything, or even what’s most important – just a few papers that have interested the author. Visit our Resources page for links to more journals or follow the HealthEconBot. If you’d like to write one of our weekly journal round-ups, get in touch.

Conditional cash transfers: the case of Progresa/OportunidadesJournal of Economic Literature [RePEc] Published September 2017

The Progresa/Oportunidades programme was instigated in Mexico in 1995. The main innovation of the programme was a series of cash payments conditional on various human capital investments in children, such as regular school attendance and health check-ups. Beginning principally in rural areas, it expanded to urban areas in 2000-1. Excitingly for researchers, randomised implementation of the programme was built into its rollout, permitting evaluation of its effectiveness. Given it was the first such programme in a low- or middle-income country to do this, there has been a considerable amount of analysis and literature published on the topic. This article provides an in-depth review of this literature – incorporating over one hundred articles from economics and health journals. I’ll just focus on the health-related aspects of the review rather than education, labour market, or nutrition outcomes, but they’re also worth a look. The article provides a simple theoretical model about the effects of conditional cash transfers to start with and suggests that they have both a price effect, through reducing the shadow wage of time in activities other than those to which the payment is targeted, and an income effect, by increasing total income. The latter effect is ambiguous in its direction. For health, a large number of outcomes including child mortality and height, behavioural problems, obesity, and depression have all been assessed. For the most part  this has been through health modules applied to a subsample of people in surveys, which may limit the conclusions one can make for reasons such as attrition in the samples of treated and control households. Generally, the programme has demonstrated positive health effects (of varying magnitudes) in both the short and medium terms. Health care utilisation increased and with it there was a reduction in self-reported illness, behavioural problems, and obesity. However, positive effects are not reported universally. For example, one study reported an increase in child height in the short term, but in the medium term little change was reported in height-for-age z-scores in another study, which may suggest children catch-up in their growth. Nevertheless, it seems as though the programme succeeded in its aims, although there remains the question of its cost-benefit ratio and whether these ends could have been achieved more cost-effectively by other means. There is also the political question about the paternalism of the programme. While some political issues are covered, such as the perception of the programme as a vehicle for buying votes, and strategies for mitigating these issues, the issue of its acceptability to poor Mexicans is not well covered.

Health‐care quality and information failure: evidence from Nigeria. Health Economics [PubMedPublished 23rd October 2017

When we conceive of health care quality we often think of preventable harm to patients. Higher quality institutions make fewer errors such as incorrect diagnoses, mistakes with medication, or surgical gaffes. However, determining when an error has been made is difficult and quality is often poorly correlated with typical measures of performance like standardised mortality ratios. Evaluating quality is harder still in resource-poor settings where there are no routine data for evaluation and often an absence of patient records. Patients may also have less knowledge about what constitutes quality care. This may provide an environment for low-quality providers to remain in business as patients do not discriminate on the basis of quality. Patient satisfaction is another important aspect of quality, but not necessarily related to more ‘technical’ aspects of quality. For example, a patient may feel that they’ve not had to wait long and been treated respectfully even if they have been, unbeknownst to them, misdiagnosed and given the wrong medication. This article looks at data from Nigeria to examine whether measures of patient satisfaction are correlated with technical quality such as diagnostic accuracy and medicines availability. In brief, they report that there is little variation in patient satisfaction reports, which may be due to some reporting bias, and that diagnostic accuracy was correlated with satisfaction but other markers of quality were not. Importantly though, the measures of technical quality did little to explain the overall variation in patient satisfaction.

State intimate partner violence-related firearm laws and intimate partner homicide rates in the United States, 1991 to 2015. Annals of Internal Medicine [PubMedPublished 17th October 2017

Gun violence in the United States is a major health issue. Other major causes of death and injury attract significant financial investment and policy responses. However, the political nature of firearms in the US limit any such response. Indeed, a 1996 law passed by Congress forbade the CDC “to advocate or promote gun control”, which a succession of CDC directors has interpreted as meaning no federally funded research into gun violence at all. As such, for such a serious cause of death and disability, there is disproportionately little research. This article (not federally funded, of course) examines the impact of gun control legislation on inter-partner violence (IPV). Given the large proportion of inter-partner homicides (IPH) carried out with a gun, persons convicted of IPV felonies and, since 1996, misdemeanours are prohibited from possessing a firearm. However, there is variation in states about whether those convicted of an IPV crime have to surrender a weapon already in their possession. This article examines whether states that enacted ‘relinquishment’ laws that force IPV criminals to surrender their weapons reduced the rate of IPHs. They use state-level panel data and a negative binomial fixed effects model and find that relinquishment laws reduced the risk of IPHs by around 10% and firearm-related IPH by around 15%.

Credits

Sam Watson’s journal round-up for 2nd October 2017

Every Monday our authors provide a round-up of some of the most recently published peer reviewed articles from the field. We don’t cover everything, or even what’s most important – just a few papers that have interested the author. Visit our Resources page for links to more journals or follow the HealthEconBot. If you’d like to write one of our weekly journal round-ups, get in touch.

The path to longer and healthier lives for all Africans by 2030: the Lancet Commission on the future of health in sub-Saharan Africa. The Lancet [PubMedPublished 13th September 2017

The African continent has the highest rates of economic growth, the fastest growing populations and rates of urbanisation, but also the highest burden of disease. The challenges for public health and health care provision are great. It is no surprise then that this Lancet commission on the future of health in Sub-Saharan Africa runs to 57 pages yet still has some notable absences. In the space of a few hundred words, it would be impossible to fully discuss the topics in this tome, these will appear in future blog posts. For now, I want to briefly discuss a lack of consideration of the importance of political economy in the Commission’s report. For example, the report notes the damaging effects of IMF and World Bank structural adjustment programs in the 70s and 80s. These led to a dismantling of much of the public sector in indebted African nations in order for them to qualify for further loans. However, these issues have not gone away. Despite strongly emphasizing that countries in Africa must increase their health spending, it does not mention that many countries spend much more servicing debt than on public health and health care. Kenya, for example, will soon no longer qualify for aid as it becomes a middle-income country, and yet it spends almost double (around $6 billion) servicing its debt than it does on health care (around $3 billion). Debt reform and relief may be a major step towards increasing health expenditure. The inequalities in access to basic health services reflect the disparities in income and wealth both between and within countries. The growth of slums across the continent is stark evidence of this. Residents of these communities, despite often facing the worst exposure to major disease risk factors, are often not recognised by authorities and cannot access health services. Even where health services are available there are still difficulties with access. A lack of regulation and oversight can lead the growth of a rentier class within slums as those with access to small amounts of capital, land, or property act as petty landlords. So while some in slum areas can afford the fees for basic health services, the poorest still face a barrier even when services are available. These people are also those who have little access to decent water and sanitation or education and have the highest risk of disease. Finally, the lack of incentives for trained doctors and medical staff to work in poor or rural areas is also identified as a key problem. Many doctors either leave for wealthier countries or work in urban areas. Doctors are often a powerful interest group and can influence macro health policy, distorting it to favour richer urban areas. Political solutions are required, as well as the public health interventions more widely discussed. The Commission’s report is extensive and worth the time to read for anyone with an interest in the subject matter. What also becomes clear upon reading it is the lack of solid evidence on health systems and what works and does not work. From an economic perspective, much of the evidence pertaining to health system functioning and efficiency is still just the results from country-level panel data regressions, which tell us very little about what is actually happening. This results in us being able to identify areas needed for reform with very little idea of how.

The relationship of health insurance and mortality: is lack of insurance deadly? Annals of Internal Medicine [PubMedPublished 19th September 2017

One sure-fire way of increasing your chances of publishing in a top-ranked journal is to do something on a hot political topic. In the UK this has been seven-day services, as well as other issues relating to deficiencies of supply. In the US, health insurance is right up there with the Republicans trying to repeal the Affordable Care Act, a.k.a. Obamacare. This paper systematically reviews the literature on the relationship between health insurance coverage and the risk of mortality. The theory being that health insurance permits access to medical services and therefore treatment and prevention measures that reduce the risk of death. Many readers will be familiar with the Oregon Health Insurance Experiment, in which the US state of Oregon distributed access to increased Medicaid expansion by lottery, therein creating an RCT. This experiment, which takes a top spot in the review, estimated that those who had ‘won’ the lottery had a mortality rate 0.032 percentage points lower than the ‘losers’, whose mortality rate was 0.8%; a relative reduction of around 4%. Similar results were found for the quasi-experimental studies included, and slightly larger effects were found in cohort follow-up studies. These effects are small. But then so is the baseline. Most of these studies only examined non-elderly, non-disabled people, who would otherwise not qualify for any other public health insurance. For people under 45 in the US, the leading cause of death is unintentional injury, and its only above this age that cancer becomes the leading cause of death. If you suffer major trauma in the US you will (for the most part) be treated in an ER insured or uninsured, even if you end up with a large bill afterwards. So it’s no surprise that the effects of insurance coverage on mortality are very small for these people. This is probably the inappropriate endpoint to be looking at for this study. Indeed, the Oregon experiment found that the biggest differences were in reduced out-of-pocket expenses and medical debt, and improved self-reported health. The review’s conclusion that, “The odds of dying among the insured relative to the uninsured is 0.71 to 0.97,” is seemingly unwarranted. If they want to make a political point about the need for insurance, they’re looking in the wrong place.

Smoking, expectations, and health: a dynamic stochastic model of lifetime smoking behavior. Journal of Political Economy [RePEcPublished 24th August 2017

I’ve long been sceptical of mathematical models of complex health behaviours. The most egregious of which is often the ‘rational addiction’ literature. Originating with the late Gary Becker, the rational addiction model, in essence, assumes that addiction is a rational choice made by utility maximising individuals, whose preferences alter with use of a particular drug. The biggest problem I find with this approach is that it is completely out of touch with the reality of addiction and drug dependence, and makes absurd assumptions about the preferences of addicts. Nevertheless, it has spawned a sizable literature. And, one may argue that the model is useful if it makes accurate predictions, regardless of the assumptions underlying it. On this front, I have yet to be convinced. This paper builds a rational addiction-type model for smoking to examine whether learning of one’s health risks reduces smoking. As an illustration of why I dislike this method of understanding addictive behaviours, the authors note that “…the model cannot explain why individuals start smoking. […] The estimated preference parameters in the absence of a chronic illness suggest that, for a never smoker under the age of 25, there is no incentive to begin smoking because the marginal utility of smoking is negative.” But for many, social and cultural factors simply explain why young people start smoking. The weakness of the deductive approach to social science seems to rear its head, but like I said, the aim here may be the development of good predictive models. And, the model does appear to predict smoking behaviour well. However, it is all in-sample prediction, and with the number of parameters it is not surprising it predicts well. This discussion is not meant to be completely excoriating. What is interesting is the discussion and attempt to deal with the endogeneity of smoking – people in poor health may be more likely to smoke and so the estimated effects of smoking on longevity may be overestimated. As a final point of contention though, I’m still trying to work out what the “addictive stock of smoking capital” is.

Credits