Social scientists, especially economists, are concerned with causal inference: understanding whether and how an event causes a certain effect. Typically, we subscribe to the view that causal relations are reducible to sets of counterfactuals, and we use ever more sophisticated methods, such as instrumental variables and propensity score matching, to estimate these counterfactuals. Under the right set of assumptions, like that unobserved differences between study subjects are time invariant or that a treatment causes its effect through a certain mechanism, we can derive estimators for average treatment effects. All uncontroversial stuff indeed.
A recent paper from L.A. Paul and Kieran Healy introduces an argument of potential importance to how we can interpret studies investigating causal relations. In particular, they make the argument that we don’t know if individual preferences persist in a study through treatment. It is in general not possible to distinguish between the case where a treatment has satisfied an underlying revealed preference, or transformed an individual’s preferences. If preferences are changed or transformed, rather than revealed, then they are, in effect, a different population and in a causal inference type study, no longer comparable to the control population.
To quote their thought experiment:
Vampires: In the 21st century, vampires begin to populate North America. Psychologists decide to study the implications this could have for the human population. They put out a call for undergraduates to participate in a randomized controlled experiment, and recruit a local vampire with scientific interests. After securing the necessary permissions, they randomize and divide their population of undergraduates into a control group and a treatment group. At t1, members of each group are given standard psychological assessments measuring their preferences about vampires in general and about becoming a vampire in particular. Then members of the experimental group are bitten by the lab vampire.
Members of both groups are left to go about their daily lives for a period of time. At t2, they are assessed. Members of the control population do not report any difference in their preferences at t2. All members of the treated population, on the other hand, report living richer lives, enjoying rewarding new sensory experiences, and having a new sense of meaning at t2. As a result, they now uniformly report very strong pro-vampire preferences. (Some members of the treatment group also expressed pro-vampire preferences before the experiment, but these were a distinct minority.) In exit interviews, all treated subjects also testify that they have no desire to return to their previous condition.
Should our psychologists conclude that being bitten by a vampire somehow satisfies people’s underlying, previously unrecognized, preferences to become vampires? No. They should conclude that being bitten by a vampire causes you to become a vampire (and thus, to prefer being one). Being bitten by a vampire and then being satisfied with the result does not satisfy or reveal your underlying preference to be a vampire. Being bitten by a vampire transforms you: it changes your preferences in a deep and fundamental way, by replacing your underlying human preferences with vampire preferences, no matter what your previous preferences were.
In our latest journal round-up, I featured a paper that used German reunification in 1989 as a natural experiment to explore the impact of novel food items in the market on consumption and weight gain. The transformative treatments argument comes into play here. Did reunification reveal the preferences of East Germans for the novel food stuffs, or did it change their preferences for foodstuffs overall due to the significant cultural change? If the latter case is true then West Germans do not constitute an appropriate control group. The causal mechanism at play is also important to the development of policy: for example, without reunification there may not have been any impact from novel food products.
This argument is also sometimes skirted around with regards to the valuing of health states. Should it be the preferences of healthy people, or the experienced utility of sick people, that determine health state values? Do physical trauma and disease reveal our underlying preferences for different health states, or do they transform us to have different preferences entirely? Any study looking at the effect of disease on health status or quality of life could not distinguish between the two. Yet the two cases are akin to using the same or different groups of people to do the valuation of health states.
Consider also something like estimating the impact of retirement on health and quality of life. If self-reported quality of life is observed to improve in one of these studies, we don’t know if that is because retirement has satisfied a pre-existing preference for the retired lifestyle, or retirement has transformed a person’s preferences. In the latter case, the appropriate control group to evaluate the causal effect of retirement is not non-retired persons.
Paul and Healy do not make their argument to try to prevent or undermine research in the social sciences, they interpret their conclusion as a “methodological challenge”. The full implications of the above arguments have not been explored but could be potentially great and new innovations in methodology to estimate average causal effects could be warranted. How this may be achieved, I’ll have to admit, I do not know.
Credits
[…] of health care is attributable to patient characteristics. However, I (as ever) remain skeptical: a previous post brought up the challenge of ‘transformative treatments’, which may apply here as this […]