# Tag Archives: statistics

## Data everywhere! – Introduction to Quandl

Economists need data. In this post I want to introduce those of you who don’t know it to a magnificent data source – quandl.com. Quandl is an open source website that indexes a huge range of data – over 6,000,000 data sets according to the website – for almost every country on Earth. The bulk of these data appear to be financial but there is a wealth of socioeconomic data for many countries (see here for their list of health topics, for example).

One of the most useful things about Quandl is its ability to provide that data directly into a usable format. You can even download any of the datasets straight into R; here, I will show you how.

Let’s look at total health spending as a proportion of GDP in the UK. We first find the dataset in Quandl (which is found here) and then click download where we have a number of options. In this case let’s opt for R.

We copy and paste the code into R

df<-read.csv('http://www.quandl.com/api/v1/datasets/WHO/20600_56.csv?&trim_start=1995-12-31&trim_end=2010-12-31&sort_order=desc', colClasses=c('Year'='Date'))

And then we plot

ggplot(aes(Year,Value),data=df)+theme_bw()+labs(x="Year",y="Healthcare spending as % of GDP")+geom_line()

Simple. There is also an R package available in CRAN that enables you to access data from Quandl without using the website and customising the data set (selecting variables and dates). I suspect that this will make finding appropriate data much easier in future.

Tags: , , ,

## Bad science in health economics: complementary medicine, costs and mortality

By Chris Sampson, David Whitehurst and Andrew Street

In December 2012, an article was published in The European Journal of Health Economics with the title ‘Patients whose GP knows complementary medicine tend to have lower costs and live longer’. We spotted a number of shortcomings in the analysis and reporting, to which we felt a response was worthwhile. Subsequently the authors of the original piece, Professor Peter Kooreman and Dr Erik Baars, wrote a reply. In this blog post we summarise the debate and offer some concluding thoughts.

The study

The study employed a large dataset (n~150,000) from a Dutch health insurer. The objective of the study was “to explore the cost-effectiveness of CAM compared with conventional medicine”. The study sought to find out whether different levels of cost or mortality were observed depending on whether or not an individual’s general practitioner (GP) was trained in complementary and alternative medicine (CAM). The authors specifically looked at GPs trained in anthroposophy, homeopathy and acupuncture.

The authors implemented both a linear and log-linear regression model to estimate the cost differences associated with different types of CAM-training. Separate regressions were carried out for each type of CAM, for four different age groups and for five different cost categories. This gave a total of 120 different coefficients (2 (models) x 3 (CAM approaches) x 4 (age groups) x 5 (cost categories)) for the cost difference associated with CAM-training. Eighteen (15%) of these coefficients were negative (indicating positive findings attributable to CAM training) and statistically significant at the 5% level. Three (2.5%) coefficients showed a greater cost associated with CAM training.

For mortality effects, the authors implemented both a fixed effects logit and a fixed effects linear probability model (LPM). In this case the groups were split by sex and, again, by type of CAM-training; additionally an overall effect of CAM-training was included. This gave a total of 24 different coefficients for the mortality difference associated with CAM-training. Four (16.7%) of these were lower and statistically significant at the 5% level; all from the LPM.

The authors concluded that “patients whose GP has additional CAM training have 0–30% lower healthcare costs and mortality rates, depending on age groups and type of CAM”; adding that “since the differences are obtained while controlling for confounders… the lower costs and longer lives are unlikely to be related to differences in socioeconomic status.”

The study’s faults

A major problem with the study is one of selection. Selection is important in this study; there is selection of individuals who decide whether or not to register with CAM-trained GPs and selection of GPs who choose to pursue CAM. Patients that register with CAM-trained GPs may have different characteristics from those who do not, and exhibit different levels of cost and mortality as a result of these characteristics, rather than of CAM itself. The risk-adjustment the authors perform is the only way they deal with selection, and the set of risk-adjusters is very small; including only age, gender and postal code. The authors defend their position by citing a paper suggesting that selection bias might operate in the other direction. Neither we nor the authors can prove this one way or another. To thoroughly address selection, a larger set of risk-adjusters should be included and an approach such as propensity score matching would have been superior to the model adopted by the authors.

In reporting and reflecting upon their analyses, the authors do not recognise the problems associated with multiple testing. The authors appear to misunderstand the familywise error rate and the implications of this for the results that are currently shown as statistically significant. The authors should have accounted for this, using a method such as the Bonferroni correction.

The primary claims of the study are that patients with CAM-trained GPs had “0–30% lower costs” and “0–30% lower mortality rates”. These claims can be found throughout the original study, including the title, and in the authors’ subsequent dealings with the media. We believe that the first claim is a ‘cherry-picked’ finding; the second is simply false.

With regard to costs, as identified in the authors’ reply, the 30% relates specifically to patients “aged 75 and above with an anthroposophic GP-CAM”. But there are some coefficients that show a greater cost associated with CAM-trained GPs. Yet the paper’s title and publicity statements focus on this significant result alone. This is not an accurate reflection of the cost implications for patients in general, and highlighting this cherry-picked result is a misleading representation of the overall effects. A more appropriate way of reporting the results would have been to present the expected cost differences across the whole sample.

The analysis of mortality is simply incorrect. Mortality risk is bounded by 0,1 but the linear probability model is unbounded; making it inappropriate to model mortality data. The logit model is designed for binary outcomes, and when this is employed the significance of the mortality differences disappears or is less than 5%. But even the logit is inappropriate for these data because mortality is an infrequent event (around 3% of the sample died). A probit model would be preferable and we suspect that, had a probit been employed, no significant differences would be found. In short, the ‘significant’ effects that the authors identify are due to incorrect model specification.

In their responses, the authors retreated from their original emphasis on the significance of the mortality results saying that “our results do not show any evidence that patients of GP-CAMs have higher mortality rates”. We agree with this re-statement. Nevertheless, the title of the paper remains “Patients whose GP knows complementary medicine tend to … live longer”, which the authors now appear to admit is false.

Closing remarks

The study was available in its current form, as well as earlier versions, long before it was published in the EJHE. As a result, the study’s inaccurate claims have been repeated in a number of papers that cite the work in relation to herbal medicine and CAM in primary care. The publicity sounding these claims, and the authors’ conduct with the media, has been discussed elsewhere (English translation).

We believe that the original study and the response pieces might be used as a case study to aid teaching. To this end we have provided material to the Health Economics Education website. In addition, please do consider commenting below to develop the discussion – whatever your thoughts on the matter. Do you see other flaws in the study design? Or maybe you think some of our comments are unfounded? Are there better ways of studying important questions such as these?

## To ‘y’ or not to ‘y’: Dichotomous outcomes and endogenous regressors

Empirical research into health and health related outcomes is characterised by a predominance of binary outcomes. One of the most popular outcomes (in a statistical sense) is mortality, for example. This type of outcome warrants its own type of model that allows the outcome to be observed with probability p – the outcome is a draw from a Bernoulli distribution with this probability; is allowed to vary across individuals. One distinction arises between the economics literature and the medical literature in that the linear probability model (LPM) is fairly popular in the former whereas it is never seen in the latter. The LPM is simply OLS estimation of the binary outcome on the regressors and is popular due to its easily interpretable marginal effects (equal to the estimated coefficient) and ease with which other procedures such as instrumental variables estimation can be used. However, the LPM may predict probabilities outside of the zero to one range and may even be inconsistent in many cases (see here). The probit model is also popular in the economics literature but is likewise rarely seen in medical studies. It is the logit which is ubiquitous in these analyses. But, in the medical literature, studies don’t usually try to address endogeneity whereas in economics we often do. Usually instrumental variables are employed to tackle this problem, but how do we use them in logit models?

In a linear model the two stage least squares (2SLS) estimator can be used. The endogenous variables are regressed on all the exogenous variables including the instruments, then the predicted value of the endogenous variable is used in place of its actual value in a regression. But in a model where the outcome is a nonlinear function of the regressors, such as a logit, this method would be inconsistent. To see why, note that we are trying to estimate a model that assumes:

 $E(y|\textbf{x},\textbf{z},c)=m(\textbf{x}_1'\boldsymbol{\beta}+\textbf{z}'\gamma+c)$ (1)

Where x is assumed to be exogenous of which $x_1$ is a subset, c is unobserved and z is allowed to be correlated with c so that it is potentially endogenous. We model:

$\textbf{z}=\textbf{x}'\boldsymbol{\Pi}+\textbf{v}$

And,

$c=\textbf{v}'\boldsymbol{\rho}+e$

This issue is that if ρ≠0 then z is endogenous. For our estimates to be consistent we essentially require the conditional mean in (1) to be correctly specified. We have two options, we can estimate $\hat{\textbf{z}}$ and substitute it for z in (1) or we can eliminate c. In the latter case, assuming $E(e)=0$, we can rewrite (1) as:

 $E(y|\textbf{x},\textbf{z},\textbf{v})=m(\textbf{x}_1'\boldsymbol{\beta}+\textbf{z}'\gamma+\textbf{v}'\boldsymbol{\rho})$ (2)

But, we do not observe v, however, we can consistently estimate it as $\hat{\textbf{v}}=z-\textbf{x}'\hat{\boldsymbol{\Pi}}$ and include these values in our regression. This method is known as two stage residual inclusion (2SRI).

Our other method however is inconsistent; we can use estimates of $\hat{\textbf{z}}$ in place of z but this does not eliminate c since, even though $E(c)=0$, the expectation does not ‘pass through’ the nonlinear function m(.).

A further useful feature of this comes from the fact that exogeneity of z only happens when ρ=0. We can test this empirically when we estimate (2). This is equivalent to a Hausman test for the exogeneity of z.

The standard errors won’t be correct when estimating (2). Calculating the correct standard errors is not too difficult. But, often in health econometric applications, we want to adjust for clustering within hospitals or regions. To accommodate this into our standard errors makes the calculation considerably more difficult, if not intractable. In this case bootstrapping is the preferred solution.

I have noted in previous posts that endogeneity could be a serious issue in health econometrics, particularly when these types of studies are used to inform healthcare policy. Clearly there are methods for dealing with this, though having a suitable methodology is only the first hurdle. The next one is convincing non-economists why you are using it.

See more

2SRI in health econometrics is discussed in this paper. For a more thorough discussion see Wooldridge Chapter 12.