A Surprising Microfinance Researcher and Possible Implications for Haiti
After not posting anything on this blog for more than eleven months, I hope at least a few people are still paying attention. If so, they could reasonably ask me about the status of my book on Fonkoze, and of Fonkoze itself. I will largely leave the answers to those questions to future posts. My book is on the backburner for now, though I hope to pick it up and complete it in a couple of years. By that point, I may have more time to write and the Fonkoze story will be less in flux. I will soon be posting a blog about some major developments with Fonkoze and the role I have been playing.
One of the things that has preoccupied me over the last couple of years has been the mostly tumultuous relationship between the microfinance movement on the one hand, and a new generation of researchers that has been studying it on the other. I recall a colleague saying to me in the late 1990s that a new generation of PhDs would warm to the task of trying to debunk the myth of microfinance being an effective tool to fight poverty. To some extent, that confrontation has come to pass.
Certainly, microfinance practitioners and advocates have contributed to this conflict by their (I suppose I should say our) collective lack of curiosity and defensiveness about research findings showing our weaknesses, and also by not being as welcoming as we could be to researchers wanting to study our flagship organizations (which has helped lead them to focus mostly on second and third-rate microfinance institutions). Needless to say, there have been noteworthy exceptions, such as Chris Dunford, formerly the CEO of Freedom From Hunger, and Beth Rhyne, the director of the Center for Financial Inclusion at Accion who recently joined the Grameen Foundation Board of Directors.
Still, researchers are far from blameless. When the media has covered their studies, they have tended toward oversimplified and sensationalistic headlines and stories – which the researchers, perhaps pleased that they were getting any mainstream coverage at all, have done too little to respond to and correct. Furthermore, while the most serious researchers (such as David Roodman) have come to the conclusion that reliable financial serves are extremely valuable to the poor – in fact, more valuable than they are for the non-poor – they have been unwilling to question whether it might be issues with their research methods (rather than entirely with MFIs’ practices) that have caused little positive impact being measured from microfinance in recent studies (aside from contract savings programs), particularly since few dispute that well-run MFIs collectively provide reliable financial services to the poor on a massive scale. (If one casts a broader net in terms of what one considers valid research, as I do, there are quite a few encouraging findings.)
But most damning in my mind is the fact that researchers have failed to do much to help ensure that what they have uncovered about what works well in terms of microfinance program operations is digested by practitioners and turned into better services delivered to the poor. For example, Dean Karlan’s flawed but impressive book More Than Good Intentions, which I reviewed, is chock full of tantalizing nuggets about what researchers have learned about what works best in delivering microfinance services. Unfortunately, those insights usually get buried amongst other findings and rhetoric, and are generally not understood by practitioners.
As a result, I have been trying to argue that we should hold researchers to the standard they hold the rest of us – they should measure whether their research, on which millions of dollars are spent every year, produces any positive change. More tangibly, I have been urging the research community for months to write up in a single paper the “lessons for practice” coming out of their many studies of microfinance, so that we get beyond debates about “whether microfinance works” to more fruitful and action-oriented dialogues about “how it can work better.” My advocacy has been met with deafening silence … until a surprisingly satisfying and productive encounter with a researcher this past summer.
Financial Access Initiative Managing Director Timothy Ogden
By way of background, I first met Tim Ogden, the managing director of the Financial Access Initiative at NYU, at a social enterprise conference organized by the Stern Business School in November 2012. He was generally rather contrarian and pugnacious, but he made some good points. At the end of his session he went out his way to compliment my colleague Matthew Speh, who was also on the panel, for his emphasis on grounding strategy in research on customer needs and behaviors. Maybe there was more to Tim than met the eye.
Fast forward a few months, and Tim was spending a couple of hours with the Microfinance CEO Working Group (MCWG) that I am a founding member of. The MCWG has been grappling with how to better quantify our collective outcomes and impacts on poverty, and Tim was giving us some sensible and constructive feedback. At one point, I raised the issue of whether he or some researcher he might identify could come up with the “lessons for practice” paper I had been advocating (perhaps prompting some eye-rolling by my CEO colleagues who had heard me beat this drum before).
Tim immediately saw how this could be helpful and said he would take a serious look at taking it on himself. Some weeks later I called him to make sure he understood our request (since one CEO colleague had a different recollection), and he and I were on the same page and in fact, he had already made some real progress. Even if he does not produce this paper himself, I admire his can-do attitude and constructive approach.
Coming back to Fonkoze, Tim said something during his time with the MCWG that really struck me. He said that one of the hypotheses that merited testing was whether the hundreds if not thousands of well-run MFIs that provide reliable and reasonably efficient financial services, without requiring bribes, have helped raise the expectations of millions of poor people about the quality of services they deserved in general. If true, this could have profound impact on how the poor interface with their local and national governments as well as the private charities, faith-based institutions and businesses that serve them. More demanding and discriminating customers of these services could cause a chain reaction of improvement and investment, assuming public, charitable and commercial entities are responsive (and certainly some are, based on their desires for, if nothing else, reelection, profits, etc.).
He also raised some other interesting issues related to the indirect impact of microfinance on national development, such as MFIs collective success in training of hundreds of thousands of employees to provide reliable and often excellent customer service and to manage operations efficiently – skills that are often in short supply in developing countries and that are likely transferred to the broader economy as employees leave MFIs to work elsewhere.
The particular theory about raising the expectations of the poor, if proven to be true, may point to one of Fonkoze’s most profound and lasting impacts in rural Haiti – a place where the government, charities and businesses provide few services, and what they do provide is often haphazard and substandard. The process of Haiti developing into a self-sufficient democracy free from abject poverty is obviously going to take many years and a wide variety of complementary strategies. But I believe that progress is possible.
When I recall the Bangladesh I encountered when I first landed there in December 1988, it was not that much farther along than Haiti is today. By 2013 Bangladesh, despite its many problems, has already met its 2015 Millennium Development Goals concerning halving the number of people in extreme poverty and reducing child deaths, and is on course to meet almost all of the rest of those goals. Imbuing its population with higher expectations from the institutions that serve them will certainly be a part of Haiti following a similar course. How interesting it would be to have researchers test that hypothesis in the years ahead!
So let me tip my hat to one researcher whose constructiveness and fresh ideas have impressed me, and who has also given me a possible new reason to appreciate the important work of Fonkoze.
(A follow up post to this blog can be accessed here.)
Alex Counts' Blog about Fonkoze and Microfinance in Haiti 
nice blog
Your post raises several issues, but this one seems to be the lynchpin: we should hold researchers to the standard they hold the rest of us – they should measure whether their research, on which millions of dollars are spent every year, produces any positive change. Ouch! The whole academic research industry is speechless in response. This point sets up your call for a “lessons for practice” paper very nicely. The challenge for researchers is that it is hard for them to draw lessons for practice when generally they are not practitioners and many times not even familiar with or interested in the mechanics of practice. The ideal may be a paper by a researcher and a practitioner together, but even that requires each co-author to be fairly familiar with the other’s field of expertise. Given my research background and practitioner familiarity, I tried to do a brief version of such a paper for Freedom from Hunger (see the last post on The Evidence Project – http://www.microfinanceandworldhunger.org). There are two takeaways I can offer from that effort to summarize research findings in some way that is meaningful for practice. Number One is that you have to look at the research results through the lens of a particular approach to practice in order to glean what is meaningful for that approach. It is conceivable that you could use multiple lenses to broaden the value of the paper to a greater range of practitioners, but then you would have to be quite familiar with multiple practice approaches, which I for one am not so much. On the positive side, there is much in the practice approach of Freedom from Hunger partners that is shared by many others – a lot of the lessons apply generally to group-based financial services (which, by the way, I believe are more likely to produce the societal impacts you and Tim are talking about in Bangladesh and Haiti). Number Two is that you are limited to lessons that research findings can actually offer. The findings are not comprehensive in their coverage of the range of practice approaches and mechanics in microfinance, so you are left with awkward gaps and therefore incomplete lessons, like a recipe from which one ingredient is randomly missing. I wish I could offer you a really compelling example, but I am writing off the top of my head, of course. In short, you have written well a thought-provoking piece on an important topic. I liked reading it.
Chris, Thanks for such a thoughtful comment on my blog and in particular, the promise and problems inherent in my proposal for a “lessons for practice” paper. I agree with you on group based approaches and was struck by the fact that buried in Dean Karlan’s book is reference to a study that talks about how groups that meet more frequently build more social capital amongst clients, which seems to argue against the current conventional wisdom. I love the idea of the paper I am advocating be written by a joint effort by a practitioner and researcher. Perhaps you and Tim !?!? -Alex
Certainly Chris Dunford’s articulation of what he saw as the “lynchpin” of your blog is perhaps the most important point you wanted to make . . . and one that needed making. But as a practitioner myself, I was fascinated by the examples you gave of positive outcomes that the researchers may have missed to this point. The notion that one outcome of microfinance may be that we help the most vulnerable raise their expectations about the quality of services they deserve, whether being provided by the state, an NGO or the private sector servicing the bottom of the pyramid is, I believe true, but rarely measured. The second outcome is the number of employees who learn how to provide reliable and even excellent customer service while delivering services efficiently. Both of these ring true in my experience at Fonkoze, where we put a high premium on both. Over and over again I witnessed women learning how to advocate for themselves, regardless of how vulnerable they were. I also saw so many young Haitians with very little education learn how to put their clients first while managing the concepts of time, productivity and effectiveness.
Thanks for writing this piece. . . . and for always looking for the meanings of what we do.
Perhaps a reason Mr. Ogden’s approach felt so refreshing is that he is a communications professional with a specialization in areas such as financial access, behavioral economics, and poverty. He does not get paid to draw students to University, his tenure (or long-term job security) is not determined by where his work is published, and he’s a good writer and synthesizer. His efforts are compensated when people like you and me read, understand and value his work and find ways to use it in both policy and practice. We need more people like this to “bridge” the work of researchers, who’s professional interests are elsewhere, though many can be personally very interested in practical applications to their work. I believe that many of the efforts of groups such as Ideas42, R4D, and my own firm, EA Consultants show initiatives to create bridges between research and practice in a way that can be more effective to practitioners and policy makers and Mr. Counts’ article suggests there may be great demand for this.
Barbara, I think you are right on in pointing to Tim’s different training and incentives being part of the story here. We do need more people serving as “bridges” — that’s for sure ! Thanks for being part of this conversation! -Alex
Alex,
We were very pleased with the random control trial/anthropological study carried out in Mali with three years between the baseline and the final study. The outcomes documented through the RCT (comparing randomly selected villages where Saving for Change was implemented not just the participants) – a 10% increase in food security and a 13% increase in the value of livestock (that can be sold in time of need) along with increased savings and borrowing. The poorest (daily consumption under $0.47 per day) were only a few percentage points less likely to join than those slightly better off. In addition were surprised to see that 1/3 as many joined Saving for Change like groups in the control villages.
On the other hand the RCT carried out by Innovations in Poverty Action showed no improvement in business income or the use of education and health facilities and no impact on empowerment as IPA defined it. Fortunately this was a mixed methods study and the team of Anthropologists from the Bureau for Applied Research in Anthropology pointed out the growth of social capital measured by the depth of relationships not specific activities.
Overall the conclusion was that participation in groups smoothed income with half of the loans used to purchase food during the lean season. BARA especially pointed out the importance of context. Compared to the baseline conditions in Mali had deteriorated considerably by 2012 when the data was collected with a coup, an Islamist insurgency in the North, a major drought, skyrocketing food prices, an influx of immigrants into the villages of the south where SfC is based, limited opportunities to leave the village to seek work.
The anthropologists speculate that come of the lack of impact reflected current difficult conditions. These findings underscore the value of research to achieve a better understanding of what happened (RCT) and why it happened (anthropological research).
Probably very few are being lifted out of poverty and one service the researchers have done is to provide a more realistic assessment of the impact of our work – as they underscore how important it is – having a little more to eat and more assets is a major advance. The women I talk to say that “life is much less stressful.”
To provide a context for this research that was carried out only in one province, Saving for Change has grown to 445,000 members in 5,000 villages achieved with a staff of 203, less than half that number now. Ninety six percent of the groups trained are still saving and lending.
My book on Savings Groups will be out next year.
Jeff, Thanks for weighing in ! Courtesy of your successor at Oxfam whom I just met and like very much, here is a link to the study you mention: http://www.oxfamblogs.org/fp2p/?p=14604
The really important news here is surely not (with due respect) Tim Ogden, but the fact that you have summarily abandoned your long-running book project without giving any explanation whatsoever to your loyal readers, such as myself. I know you said you would blog about the reasons later, but surely we deserve to know right away the reasons why you are ditching a major GFUSA project after so much time, financial resources and effort by non-GFUSA individuals was invested into it. Inevitably, of course, I am left wondering if your decision has got anything to do with the fact that, from what I am reading and hearing from my own contacts working in P-au-P, the new fangled microcredit sector has precipitated a development disaster in Haiti. I’m sure you know as well as I do that the bulk of microcredit in Haiti has overwhelmingly gone into unsustainable consumption spending and also into expanding street trading to a crazy degree. Many of the most commercialised MFIs have been caught making seriously high profits, paying high salaries and bonuses to managers, and otherwise being as naughty as elsewhere. I also hear that the level of individual over-indebtedness is by all accounts reaching quite scary levels. At the same time, family-based agriculture – the one sector with an historic base and with real promise in terms of import substitution (Haiti is now food import dependent to an alarming degree) – is completely avoided by most MFIs because of its high risk and low returns compared to simply expanding consumption and pushing yet more of the poor into the street trader economy. However way you look at it, and just as in the rest of Latin America – see my recent CEPR blog posting for example at
http://www.cepr.net/index.php/blogs/the-americas-blog/latin-americas-tragic-engagement-with-microcredit – the introduction of microcredit has been a pretty disastrous intervention in Haiti.
Anyway, I look forward to your future blogging – very soon, I hope – on the reasons why you ditched a book project that many of us were hoping would throw some important additional light on what has been going on in Haiti these past few years, and what actual contribution the provision of microcredit has made, if any, to helping the Haitian people recover from one of the world’s worst ever humanitarian disasters.
Milford Bateman
Dr. Bateman, Thank you for raising some interesting issues. I’ll look into them as I try to learn more about Haiti and microfinance. To be clear, my book project is simply on a longer timeframe than it was originally and I am trying to manage expectations of people who have been tracking my effort. The main reason is lack of time to do the subject justice. I have a full time job leading a global organization without any kind of endowment, so finding the time to do research and write is not always easy. Also, Fonkoze and its for-profit sister organization (neither of which is profitable yet) are going through a period of change (that I am involved in shaping myself from a volunteer governance role) and the story will make more sense to tell in book form once those changes play out. This book was never a “major Grameen Foundation project” but rather a personal interest/initiative that had GF’s blessing. Think of it as a kind of moonlighting job, especially since GF has little if any activity in Haiti today. This blog is run by me and is not reviewed by GF communications staff. I am not sure who the non-GF people who invested in my book project are, but I’d be eager to meet them if you know of any! -Alex
Mr Counts, thanks for replying. I look forward to reading at least a summary of what you found in Haiti, since you seem to have spent quite a lot of time there since early 2011. As to your jocular query to me as to who might have supported your book, I’m pretty sure you can tell us about this. After all, you reported on June 29th, 2011, as follows:
“By the way, with the support of Grameen Foundation‘s top notch fund-raising department, I just got a grant yesterday that pays for a big part of the costs of hiring an advisor who specializes in coaching authors to make their books into best-sellers (he is a best-selling author himself). Progress! And my team of volunteer advisors who are helping to create buzz about the book through social media has nearly 30 members and we have set up a listserv to make it easy for them to communicate. What fun!”
In many other postings, you list other forms of internal (sabbatical) and external (financial and volunteer) support for your Haiti book project, such as on June 27th, 2011, and July 18th 2011. You also talk at one stage about finalising a lengthy pre-publication teaser article that summarises the main conclusions in your book and that your agent was going to place at a major magazine (The New Yorker was mentioned I think) or at another major media outlet, but then we hear nothing more after this posting. Did the article get turned down? Did you fail to complete it?
What I’m saying is that your project got a hell of a lot of support, including for numerous trips to Haiti, resources for research that many of us who are quite sceptical of microcredit would never in a million years have had access to, for obvious reasons. So you must have formed an opinion on the viability of microcredit as a post-disaster development and anti-poverty strategy in Haiti, and some of us would just like to hear about it, that’s all. Let’s be honest here: I’m pretty sure had you found a wonderful microcredit-induced economic paradise emerging in Haiti, you and GFUSA and the usual suspects would have been shouting about it from the rooftops. So your abrupt announcement that you are essentially ditching the book project, for no apparent reason given, and after significant resources were expended, and after all the bad news some of us have been getting from our contacts in Haiti, looks to me like you simply cannot honestly report on what you found. Am I right?
Dr. Bateman, It is good to know that there are people out there who have the interest and time on their hands to pay such close attention to what I write. But to answer your question, you are not correct. I am postponing this project, not “ditching” it. During my 2011/12 sabbatical (lasting 3 months) that all qualifying GF employees are due every seven years, I spent some of it working on this book project, which included 3 trips to Haiti over the course of two summers. The GF staff raised $10,000 to pay for me to do a 2-day training for writers (and some follow up training). This training is helping to me in all the writing I do for GF. I was and am grateful for the grant, but in the grand scheme of things, that’s not a lot of money. The long-form article was written but was not accepted for publication. Some of the best parts of it (in my view at least) were published on this blog instead. Overall, I have been very impressed with the the kind of holistic microfinance I have found in Haiti, and this is why I have been keeping a blog about it and intend, in the long term, to write a book about the topic. Thanks for your interest in this project. -Alex
Thanks for the post! As someone who moved last year from the research world (after four years at Innovations for Poverty Action and the Centre for Micro Finance in India) to the practice world (now at the Smart Campaign at the Center for Financial Inclusion), I am lucky to have some perspective from both camps.
I think the practitioner world isn’t fully aware of or exposed to the full breadth and depth of what ‘financial inclusion’ researchers are up to. Perhaps it is the angst that we feel when another microcredit impact study comes out that keeps us firmly focused on these types of studies, but there are literally dozens of other rigorous research projects out there with findings that can help practitioners. There are product tests that look for reasons why people adopt or don’t adopt products, research on how to tweak product features to increase their usage, and research on how certain tweaks in product features can ultimately help beneficiaries, among many other examples. And of course plenty of welfare studies on products beyond traditional microcredit. A few of my favorite examples:
– Rohini Pande and Erica Field’s work on how variations in credit repayment schedules (i.e. including a grace period) can have implications for microentrepreneurs’ businesses, borrower stress levels etc. They also looked at cost implications for the MFI.
– Dean Karlan and others’ work on commitment savings devices that shows tremendous potential for products that allow individuals to voluntarily save to avoid the temptation to spend
– Greg Fischer’s work in the Dominican Republic that shows the effectiveness of simple rules of thumb business training on microentrepreneurs
There are also dozens of new projects being launched in the area of financial capability and mobile money. Many of these studies have practical implications for MFIs and providers. However there is often a bit gap in translating results into practice. In my opinion this is because of a few hurdles:
– One barrier has been that rigorous research projects often focused largely on outputs/outcomes/impacts and not on cost-effectiveness. A final project paper would analyze and discuss the impact or outcome but not in the context of how much it costs the institution (and compared to a cost counterfactual). This often made it hard for interested practitioners to take the results and run with them. This is changing. With the increased emphasis on scaling up innovations that have been found impactful, i.e. at DIV, donors and researchers are putting more effort into understanding which interventions bring the most bang for their buck. For example, IPA recently decided to invest a lot more intellectual energy into understanding what types of ‘reminders to save’ (like SMS’) work best in increasing savings among low income households. Early research found that simple and frequent reminders were powerful in increasing savings balances from between 6-14%. While this may not sound like a huge increase, considering how cheap it is to send out SMS’ this is an extremely cost effective. IPA is now taking these results and looking for practitioner partners to test out messaging variations for even better results for poor savers.
– Researchers’ jobs are to do research, not translate their results into practice. Applying market tests to the utility of research is not fair if the motivating policy questions were important. Plus jumping too quickly to universalizing the results of a single study will strike fear into most academics that have external validity concerns. At the same time far few researchers have any interest in replicating a study that either they or someone else has done in another context because from a professional perspective, it will not add much to their academic credentials. One of the reasons that I enjoyed IPA/JPAL’s approach was that they understand the need to replicate promising findings from one study into another context and sought funding for them. Donors and other actors should incentivize this more and practitioners I’m sure would jump on board.
– Practitioner partners in these studies have, in certain instances, felt that the research was being done ‘to them’ rather than with them. Thus at the end there is sometimes not a lot of ownership of the results and a desire to champion them (especially if the findings are less than stellar). This is changing with the arrival of thematic research funds that not only jump immediately into a full-fledged RCT but rather have options for research preparation that allows for a much richer dialogue between the practitioner and the research team.
I think there is already a wealth of information that practitioners and policymakers can use. I think it’s the job of practitioners and other stakeholders to jump into the fray take results from studies and with careful planning, put them into practice. The Smart Campaign works with many dedicated and engaged practitioners who I’m sure would be a ready audience for them.
Alex, I appreciate the points you’ve raised and I’ve enjoyed reading much of what has been posted here about the relationship between researchers and practitioners. As you pointed out, there is actually a lot of research out there. What I find challenging for all of us as a whole, is a way to put these findings into formats and venues where practitioners can really consider the implications for their work. Is this just the researchers role, or is it our role as technical assistance providers? For those of us who have the time to read through the full researcher reports –and even the research summary reports that the Grameen Foundation has led on (the ones Goldberg and Odell wrote summarizing key research studies)–we may understand some of the implications, but it is challenging to find appropriate times/venues to share these findings with partners so that they can individually consider the implications. For example, the papers that Karlan wrote about the group guarantee vs. individual loans from the study in the Philippines. He found that while people stayed in the group to make repayments, there was no reduction in the repayment rate when some groups moved away from the group guarantee to individual loans. For many of us who work with group-based lending, this seems like an important finding. Of course, many partners might simply say—well that was the Philippines. But how can we spark the debate with organizations so that they do think about how the results could influence their own practice? A challenge too, is that many of the research reports are written in English and never translated to encourage consumption among Spanish and French speaking MFIs, for example. One thing I appreciate about IPA and JPAL is that once they have multiple evaluations that have assessed the same thing, they tend to develop very short technical policy briefs that provide fairly clear implications for practice for different fields. But, they are rarely available in other languages…and again, it’s left up to those of us who know how to read these, to figure out how to share them with people who really need to make the decisions. So, I guess this is all to say, is that we also have to find better ways to take research from the researchers and help others translate the findings into actionable opportunities to change practice. Perhaps more research briefs written in other languages that end with questions for practitioners to consider might be useful (not relying on 20-30 page research reports that are filled with methods, modeling, etc. that are hard for a lay person to understand without joining a statistics course). Otherwise, there seems to be a chasm between important findings making it to the people (MFIs) that would benefit the most.
Bobbi, Your point about the opportunity to translate key research reports into other languages is well taken. Are their allies in Spanish and French-speaking worlds who could be convinced to underwrite the translation of the best reports already out there and the best to come? I feel a strong need to create a single paper that distills the essential “lessons for practice” that leaves out (as you say) all the stuff about methods, modeling, etc. If such a paper was written, I would urge every MFI’s senior management team and board to read it and then huddle to discuss implications for what they do going forward. Maybe there is a better way to help translate research findings into better practices but this seems like low-hanging fruit to me. -Alex
I’ve often thought that if we, as practitioners, want
studies that give us lessons for practice, we need to become
engaged in the generation of the hypotheses and questions to be
studied. While the broad and general questions like “does
microfinance have an impact?” might be interesting from a public
policy or donor point of view (the types of folks that usually pay
for studies), the findings from those types of studies are not
particularly actionable by practitioners. I would argue that these
types of studies are not as useful as they appear to policymakers
either because they tend to lead people to “throw the baby out with
the bathwater”, so to speak, without really understanding the
specifics. As practitioners, we’ve seen microfinance work for some
and not for others. So, it’s understandably hard for us to accept a
“no impact” conclusion. It’s possible that both we and the
researchers are right. As Alex suggests above, it’s time for us to
get specific. We know that there is an enormous diversity of
products and practice that is called “microfinance.” In some of the
“no impact” studies, all these different programs and services were
lumped together and no statistically significant impact on poverty
was found. Accepting that as true doesn’t necessarily mean that
none of the types of programs included in the study had any impact.
We don’t know one way or the other. But, as practitioners trying to
use evidence-based decision-making to improve programs, we need to
know exactly that (and much more). In other studies, a particular
program was studied in a particular context, and we are left
wondering–how does that finding apply to my slightly different
program in a completely different context? To Chris’ point above,
we aren’t sure of the “recipe” that led to the impact (or no
impact) found in the study. We need to understand what factors are
influencing the outcomes. Is it the product? The training? The
delivery channel? The group? The lack of a group? The staff? The
client profile? We don’t know, but we need to know if we are going
to make microfinance better. We need to know what kinds of services
work for what kinds of people in what kinds of circumstances. The
problem is that looking into all these possibilities with RCTs
would take tens of millions of dollars–perhaps hundreds of
millions. It’s truly like looking for a needle in a haystack.
Experimental research design is really best suited for proving or
disproving hypotheses. We need other types of research ranging from
quasi-experimental to qualitative to help us formulate hypotheses
that can be tested by experimental research. For example, the
research that Freedom from Hunger undertook with the support of Tim
Odgen exploring the role of field agents (or loan officers). This
was primarily qualitative research designed to illuminate the ways
in which field agents influence how the program is implemented and
ultimately experienced by clients. On an intuitive level, it makes
sense that field agents influence outcomes, but often in the RCTs
field agents are fuzzy factor–this very human factor–for which it
is hard to control. By learning more about how these folks
influence outcomes, we may be able to design an experiment that
takes their influence more fully into account. This is just one
example of how we can use more practitioner-friendly research
methods to inform practice and the development of hypotheses. At
this point, it seems that the research community has disproven the
hypothesis that all types of microfinance work for all types of
people all the time to reduce poverty–or the “silver bullet”
theory. It’s time for us to get busy generating some new, more
nuanced hypotheses to be tested. The good news is that we,
collectively, already have a great deal of information and research
that can inform this work. By working together with researchers and
analyzing existing research and data in aggregate, we should be
able to pick out trends and patterns that point to hypotheses and
the lessons for practice that we all seek.
Thanks so much to Chris, Anne, Lisa, Bobbi, Barbara, Jeff and Alex for their great comments. I will try to reply as time allows. Thanks for engaging in this discussion.
And just to add, many of the projects that have been mentioned do not cost millions of dollars. Especially if they’re looking at outcomes such as adoption and usage that rely more on administrative data from the partner MFI, they can often be quite inexpensive! Welfare studies are indeed more expensive because they rely on more in-depth face-to-face surveys. But again these are just one type of study within a much larger group of research projects.
Alex, thanks for you original and follow up comment. Certainly there is a range of costs for impact studies based on many factors. I hear about lightweight RCTs and so forth and they sound promising. But I think it is undeniable that in the aggregate, tens of millions of dollars are spent on these studies annually and we should have a way of measuring their effectiveness in leading to better outcomes.
Just to clarify that my comment above was referring to the tens or hundreds of millions that it could cost to try to test out every possibility with an RCT–not the cost of a single study. I definitely appreciate these more targeted, less costly studies and can see their value in testing very practical hypotheses around product uptake and usage, etc. But, even so, I don’t think it is possible to test every possibility with RCTs. So we need other research methods to complement these to help us narrow in on the most promising hypotheses and identify factors that we should try to control for in the RCTs.
Dear all,
This is a great conversation. Thanks Alex for bringing up this very important topic. I would like to state that I think it is OUR job to help bridge this gap, i.e. break down the barriers and ensure research findings are useful to practitioners. By “our” I mean to say the practitioner community. Some very insightful folks within our Savings Led Working group promoted this idea for two years. We were fortunate to get the support of the Gates Foundation to allow us to do it. We are publishing “The Evidence Based Story of Savings Groups: A synthesis of Seven randomized control trials” now. The authors are Meaghan Gash of Freedom from Hunger and Katherine Odell from Domincan Univeristy. However, what made the work so useful is it has received a great deal of input from the Savings Led Working group members (practitoners). It has effectively translated hundreds and hundreds of pages of RCT reports (7 in total) and made it useful to the community of practitioners. There is most certainly other opportunities of this nature. Again, I do feel practitioners need to take the lead to not only help interpret the findings and make the learning more widely accessible, but also as Lisa says, help influence some of the questions being asked. I will mention we designed a series of Research Roundtables at the SEEP Conference this year with the intention of helping to bridge this gap. While we will talk about research findings, the greater good hopefully will be the discussion of research processes and methods and how we can engage to benefit more broadly. It is our first year to try this so we hope we can walk away with some good ideas to sustain these efforts on a more continual basis. Thanks again for this great discussion.
Hi Alex,
Just got to read your blog, and found it so interesting both for content about the researcher issues and to get caught up a bit on where you are focusing some of your time & energies re: Haiti & Fonkoze. We don’t talk much about your expansive work life, including your book writing & participation & leadership in the broad microfinance world, so it is a pleasure to read about it in your blog. Thanks for including me in emailing.
Hope you are doing well & not too awfully busy. Did see that you had lots of good times in Key West. Hope to talk soon.
Lots of Love, Norm
On Fri, Oct 18, 2013 at 10:20 AM, Alex Counts’ Blog about Fonkoze and Microfinance
How nice to get this comment on my blog from my beloved step-mom ! Not sure she knew it could/would be published.
Hi, I published a response to this on the Governance Across Borders Blog. With all due respect, since I understand your article as a challenge against independent research on microfinance, these are the key points of my defense:
1. The claim that impact researchers have focused so far on bad MFIs is questionable.
2. The “noteworthy exceptions” among researchers (above) happen to be MFI affiliates.
3. Negative media reports about microfinance are hardly more sensationalist than positive ones; the allegation that researchers have sought sensationalist headlines is unfair; and Grameen perhaps shouldn’t throw stones when it comes to sensationalism.
4. You suggest the mainstream methods are inadequate for properly finding microfinance impacts; but perhaps the impacts aren’t find-able.
5. The study cited as part of a “broader net” of valid research is a Grameen in-house study.
6. Researchers are hardly responsible for helping practitioners improve their business; and vice-versa.
7. “Positive change” doesn’t only come from finding positive things.
8. Please don’t blame independent researchers for being independent in their findings.
I explain these points in better detail here (http://governancexborders.com/2013/10/29/grameen-foundations-antiscience-eight-contentions/); hopefully this can contribute – so to say – to a cordial and constructive discussion about what a more constructive discussion might be.
Kind regards, Phil Mader
Mr. Mader,
Thank you for joining this conversation. Fundamentally, I am arguing that the relationship between microfinance researchers and practitioners is far from optimal and this blog is a brief attempt to analyze why and point to some ways that the dynamic could be improved. I think both groups bear some responsibility for the less than ideal relationship. I understand from other fields, this situation is far from unique – there is an inherent, and probably healthy, tension between those who do things in an effort to improve society, and those who evaluate whether their efforts are effective.
I will not go through your eight points one by one, but let me make a few additional comments. I have argued that a small number of frequently-cited studies of microfinance’s impact may be valid for those individual cases, but there are questions in my mind about how generalizable they are to microfinance in general. I have further argued that when an independent academic commissioned by Grameen Foundation cast the net wider in terms of research on microfinance, the picture looks more encouraging. (Grameen Foundation had no editorial control over that study, which was written by Professor Kathleen Odell — a report that the thoughtful microfinance skeptic David Roodman called “great” and “a model of public communication about social science research.” ) If raising questions about whether the findings of a small number of studies on MFIs can be generalized to make conclusions about microfinance in general is “anti-science,” I stand guilty as charged.
I have further argued that some researchers have done too little to set the record straight when their studies have been mischaracterized in the media. Certainly several of them recognize that the media has not reported on their studies accurately and have tried to respond; I simply wish they had done more.
Lastly, I take it for granted that most if not all social science and impact research is about improving society by increasing the knowledge of what works, what does not work, and how things can be improved so that we can allocate our scarce human and financial resources better. Indeed, the Jameel Poverty Action Lab’s admirable motto is, “Turning Research Into Action.” The main thrust of my blog, then, is that simply churning out good research is a necessary but insufficient condition for ensuring that more resources are allocated to what works, and less to what does not work, and that what can be improved, is improved. Additionally, I argued that the cost/benefit analysis that researchers rightly apply to anti-poverty programs should be applied to research itself.
Translating research into an improved human society requires, among other things, an effort by researchers to make their findings available in plain language to policy-makers, practitioners and the public. I think that researchers have, in general, been too quick to jump to the next prestigious study and have under-invested their time in making sure their conclusions about things like how microfinance can be improved are widely known and lead to better decisions by policy-makers and implementers. (I also argued that many practitioners have lacked curiosity about research findings, further compounding the problem.) So, in making this case I identified a researcher who seemed to agree on this point and more importantly was willing to take a step in the direction of investing in the “translation” of research into improved practice, and lauded him for doing so. I am not sure this constitutes an “attack on independent research,” as you claim, but I leave that up to you and those reading this blog.
Alex
Dear Mr. Counts,
Thanks for this reasoned and clarificatory reply to my contentions. As in the case of my blog, you raise too many interesting points for me to address without going into excessive length.
In terms of sophisticated researchers with a willingness to communicate intelligibly, I would point you to Duvendack, Palmer-Jones, Copestake, Hooper, Loke and Rao, who published an (in my opinion) very easily-digestible “Policy Brief” to accompany their long Systematic Review “What is the evidence of the impact of microfinance on the well-being of poor people?”. The intelligible researchers aiming for policy relevance of their findings are certainly out there!
As for the balance of Kathleen Odell’s report for Grameen, I would only point to the the intervention on David Roodman’s blog Richard Palmer-Jones, who questions whether a completely untried economist with no experience in local development and no publications prior to this, could do a good objective job on assessing the virtues of microfinance (here: http://international.cgdev.org/blog/literature-review-microfinance-impacts-africa). That’s his assessment, not mine.
Regarding your final statement that good research should facilitate “more resources are allocated to what works”, I agree, and we see eye to eye. Research and practice should not be conflictual but cyclical. However, I think where we disagree fundamentally is on what counts as the basis for evidence on “what works”, and who gets to call the shots.
I will try to follow up tomorrow.
Phil
Just to follow up with a brief remark: it would be very enlightening to have David Roodman’s views on this, as he indeed on the one hand applauded Dr. Odell’s work, but on the other hand he remains responsible for the candid public appraisal of microcredit’s impact as “zero”.
Meanwhile I note with some surprise the very timely recent awarding of a large grant to Grameen Foundation for the measurement of poverty reduction efforts (http://www.microfinancegateway.org/p/site/m/template.rc/1.1.20224/). Given the explicit desire Mr. Counts, CEO of the Foundation. expresses in this article for more positive results from impact evaluations, one may wonder to what extent the results of the measurement exercise are pre-determined.
Mr. Mader,
I am glad you brought up our work on poverty measurement. We have certainly made no secret of our desire to build more robust monitoring and evaluation (M&E) systems into MFIs operations, and that we worked with CGAP and the Ford Foundation to develop the Progress out of Poverty Index (based on Grameen Bank’s “10 Indicators of Poverty” tool) to facilitate that objective. We believe that simply measuring loans made or repaid, or deposits mobilized, is insufficient for MFIs that have a poverty-fighting mandate. We have an entire website (http://www.progressoutofpoverty.org/) dedicated to the use of this tool, and we welcome feedback from you and others on how the tool and the website can be improved. The tool is used by mainly by practitioners and researchers, not by us — we make it available online to anyone who seeks to use it, and we provide online resources on how to use it, and deliver onsite training as often as possible when requested. These dynamic M&E systems complement independent impact research well.
My desire is for microfinance to learn through internal M&E and independent research how it can be more effective as a poverty-fighting strategy. The goal is continuous improvement through cost-effective measurement of outcomes and impact.
Alex
Mr. Mader,
I am heartened by the fact that you see some common ground emerging between us, and thankful that you have brought some papers to my attention.
I think reasonable people can disagree about what constitutes valid evidence. It is for this reason that I think making people aware of as many research findings as possible and letting them make their own judgment is the wisest course.
Regarding Professor Odell’s paper, I think reasonable people can also disagree the merits of what she wrote, but I think one should focus on the quality of her report, rather than on her qualifications. When one looks at that, I agree with David Roodman that she did a great job. I respect your right to come to a different conclusion.
Alex
When I wrote my original blog, I had wanted to give a positive example of researchers trying to set the record straight when the mainstream media mis-reports on their work. I looked for the best example I knew but could not find it online. Now I have found it. It was written by Abhijit Vinayak Banerjee and Esther Duflo of M.I.T, and Dean Karlan of Yale. Here is the link: http://kristof.blogs.nytimes.com/2009/12/28/the-role-of-microfinance/?_r=1. If others know of other examples, please post them. If there are a lot more of these than I was aware of out there, I will certainly need to withdraw my earlier criticism that researchers as a group have not done enough of this type of communication. It would also be nice to have them all in one place.
Hi Alex! Hope all’s well. In response to your question, I’ve read a few articles written by researchers about microfinance for public consumption. I’ve posted links to a few of them below:
The first one I think is particularly powerful as it was in the aftermath/midst of the AP crisis where there was a lot of fear and anxiety about the sector.
http://www.indianexpress.com/news/help-microfinance-dont-kill-it/716105/0
http://www.csmonitor.com/Business/new-economy/2011/0517/Microfinance-works-even-without-a-Hollywood-ending
Mr Counts, you misrepresent the situation. The NYT output you report, and the two reports Alex Rizzi notes, are all penned by researchers who, through their Universities and, increasingly, through the high-profile NGOs that they have privately established, are intimately linked to important funds flows from the microfinance support and advocacy industry. It beggars belief to assume, as you appear to do, that this work can remain independent and is not somehow shaped by the microfinance advocacy institutions that fund it. Indeed, my earlier blog pointed out how such researchers, notably including Professors Karlan and Duflo, appear to go to some considerable length to ensure that the very serious downside problems to microfinance are not fully reflected in their findings:
You also chose not to refer to the rather important fact that researchers with completely opposing views to the mainstream in terms of the supposed ‘positive impact’ of microfinance, like myself, are very routinely denied access to mainstream media outlets to try to provide some balance to the debate, especially in the USA. I’ll admit that it’s getting better, however, because the microfinance industry is coming apart at the seams, but the sceptics still have serious difficulty responding to the continuing flood of hype about microfinance. Meanwhile, very pedestrian contributors are giving as much time as they need to sell their own dodgy microfinance product (Kiva is the embarrassingly obvious example) and to provide a lot of evidence-free tosh about how microfinance is fantastically changing the world for the better. Your problem seems to be that the media downplay what you see as the great things that microfinance has done. I would venture to suggest this is but a tiny insignificant problem compared to the way that the mainstream media have decided for so many years, and with your and GFUSA’s regular encouragement and active support I should add, NOT to report on the many and growing downside problems with microfinance.