The North-South Institute



‘Big data’ is a big deal for development

May 28, 2012


The North-South Institute

A recent (short) essay of mine on why I think “Big Data” is such a big deal for development was just published by USAID as part of their Frontiers in Development book. The book contains interesting contributions by luminaries like Bill Gates, Ellen Johnson Sirleaf (Nobel laureate & President of Liberia), Paul Collier, and the introductions by US Secretary of State Hillary Clinton and USAID Administrator Rajiv Shah capture the essence of the book.

My humble contribution makes the following points in support of a simple thesis: How we think about data and analysis in the field of international development is changing rapidly, and faster than many organizations that “do development” are prepared for.

1. International development as a field of research and practice has been a laggard in using big data and powerful analytics. In essence how we know what we know in development (about what works, what doesn’t and why) and what passes as “evidence” is sometimes shocking when we compare development with other sectors. Much of the data are of poor quality, and there are huge gaps in the information base we rely on.

2. This situation is changing faster than anyone predicted, and the set of tools driving this evolution – which I provide a synopsis of – represents the single most important trend in development.

3. The open data movement has already widened access to a broad range of contextual information in the public sector and now a similar push is required to open private sector data – the main repository of big data – in the service of social good. Consider this, the entire US Library of Congress totals about 235 terabytes of information (April 2011), Walmart processes and stores 2500 terabytes of data per hour, Twitter generates the same amount of data in about two weeks. The unprecedented innovations in tools, techniques and methodologies that are helping make sense of what this means for how we do development differently how we think about success and failure differently are the ones I am most excited about.

4. In particular, analytical tools are becoming cheaper, faster and easier to use than ever before. First, analytics helped retailers discover unlikely trends, most famously that customers who came in to buy diapers also tended to buy beer! It can do the same for complex social systems. Developments in analytics have kept pace with the speed with which big data has grown. Bringing this capacity to bear on development challenges such as food security and urbanization is just getting started. Second, it is the developing world that is leading in the

proliferation of mobile sensors. Mobile phones have grown from less than 750 million with less than a third in developing countries at the start of the 2000s to more than 5 billion and 4 times as many in developing countries as in the developed world today. About 1 billion subscribers live on less than $5 a day. The developing world is leading the big data ‘exhaust’ (autonomously generated transactional, locational, positional, text, voice and other data signatures). Coupling terabytes of mobile big data with census information, research labs have modeled slum development in Kenya, responses to socioeconomic shocks in Rwanda and food security in Uganda. In addition virtualization and visualization tools have similarly ‘democratized’ (becoming cheaper, faster, more open, and easier for non-technical users), allowing us to keep pace with the data deluge and extract meaningful inference for poverty reduction and economic development.

5. For the first time, we have a feedback layer, which has made possible deep and near real-time awareness of what is working, not working, where, and why, with feedback sourced directly from intended beneficiaries and aggregated with other localized data to make sense of highly contextual issues without losing the ability to generalize.

Together, big data, democratized analytics, and the ability to tap deep contexts are rapidly changing the way we think and do development.