Letter to Editor

Please … do not be more toxic

I came across an article published in the blog of the US AID ASSIST, and tried to post a comment but could not.  Hence I am sharing my comment here, in response to the article “Proudly Toxic” by Edward Broughton.

  Since the post is on a government site I assume it is open to the public to reproduce and am pasting the text below for reference.

Proudly toxic

  Edward Broughton, Director, Research and Evaluation, USAID ASSIST Project/URC
My comment: 

Intriguing title drew me to the article, then I found I was interested in this from the perspective of a social activist.  Of course there is no way for me to do anything like an RCT but still I feel it important to base our campaigns on evidence and to articulate that evidence in terms of concrete data whenever possible.   I also agree that “we’ve got to stop using the excuse of “but it’s a complex social intervention.”   At the same time there are always intangibles which are often the more meaningful reason to pursue a certain approach over another – on this point I was recently challenged by a fellow activist who said that for every intangible there will be tangible data points that give evidence for it.   I have been pondering that ever since.

It is serendipity that I come across your article today – I was reading about the Bhopal Gas disaster, seeing the photographs of the survivors marching, 34 years after the inital gas explosion, and looking up the details of the health impacts, when I came across another article by you which led me to this.

But please, if I may implore you, do not try to be more toxic in the future.

Proudly toxic

Edward Broughton

Director, Research and Evaluation, USAID ASSIST Project/URC

In January, the Institute of Medicine’s “Committee to Support USAID’s Engagement in Health Systems Strengthening” convened an open meeting to discuss methods for improving health care in low- and middle-income countries supported by USAID. I was asked to address the issue of cost-effectiveness analysis (CEA) in improvement.  My basic message was that it’s not easy but that’s no excuse not to try do CEAs as best we can.

Coming after presentations by other researchers advocating randomized control trials (RCTs) as one of the methods to evaluate the effectiveness of improvement interventions, I seconded their call to do them.

Yet others argued that such research is not possible.  One speaker even went so far as to say that RCTs are “toxic” in improvement evaluation, arguing that methods to improve health care are “complex social interventions” and simply can’t be tested by RCTs.

ProdI point out that there are literally thousands of published examples of RCTs involving complex interventions with critical social interactions.  A good example is this recent study published in BMC Pregnancy and Childbirth: A complex intervention to improve pregnancy outcome in obese women; the UPBEAT randomized controlled trial.  And I would argue that such research is not toxic to the larger purpose of making health care interventions more effective. The fact is that RCTs on complex social interventions are done often.

Every scientist I know believes RCTs only answer one basic question of whether something works but do nothing to tell us about how / why they do or do not. Therefore, well designed and executed qualitative and implementation research is also needed to tell us how and under which conditions such interventions can be effective in the real world.

In ASSIST’s improvement work, I believe we should always strive for a “valid proxy for the counter-factual” of having an improvement program. Often control groups fit this purpose best. Randomizing can help balance out complex social factors that can confound the effect of the complex social interventions. Tight budgets, bad timing, and local circumstances mean we often can’t often do RCTs on our interventions. But we should, and do, always try – at least to use control groups, even if we can’t randomize.  USAID has recognized the importance of seeking such rigor in our evaluation of improvement interventions by requiring ASSIST to build in comparison groups, validation of data, and CEAs.

Those outside the health care improvement field can’t be expected to believe improvement works without strong evidence. Controlled trials and RCTs give this. Run charts without controls don’t. We’ve got to stop using the excuse of “but it’s a complex social intervention”. So I hereby declare, I’m proudly toxic, I fully support toxicity, and I’ll try to be more toxic when I can in the future.

Related Countries:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s