Data Collection and Sharing

AI-D is a non-profit organization established to support research on AI for Development (AI-D). A focal point of current AI-D efforts is the coalescence and distribution of data sets in support of research. Data promises to play a critical role in AI-D projects. AI-D is working with potential sources of data (such as telephone operators in different regions of the world) to devise strategies for sharing the data for research purposes. Below you will find a list of international data resources (descriptions and logos taken from the source websites). Please watch this site for updates on the availability of data sets. Please contact data@ai-d.org if you are aware of data on development that hasn’t yet been posted.

The World Bank

worldbank

Key Development Data & StatisticsThe World Bank Group now offers free access to more than 2,000 financial, business, health, economic and human development statistics that had mostly been available only to paying subscribers.
Data by Topic Search for data by topics, such as education, gender, health, macroeconomics, population, and external debt. Each topic shows a  number of key indicators from the World Development Indicators database and publication, as well as providing links to additional online data sources.
 At a Glance Country At a Glance (AAG) tables present key indicators tracing trends in social and economic development from the last three decades.

Source: http://data.worldbank.org
Format: Returns data from disparate sources in tabular form
Access: Query databases by country or by topic
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: April 24, 2010

data.un.org

un

A huge corpus of different datasets appropriate for development.

Source: http://data.un.org
Format: Provides tables and raw data export
Access: Database queries
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: November 4, 2009

International Census

census

Global population trends, links to historical population estimates, population clocks, and estimates of population, births, and deaths occurring each year, day, hour, or second.

Source: http://www.census.gov/ipc/www/idb/
Format: Provides bar charts and tables
Access: Database queries
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Povery Action Lab

bread

Randomized trial data.

Source: http://www.povertyactionlab.org
Format: A list of data sources in various formats
Access: Linked lists to outside databases with different criteria
Usage restrictions: Most data are publicly available
Reporting requirements: Check website for details
Date Listed: April 24, 2010
Other Links: http://poverty-action.org/work/data

BREAD (Buerau for Research in Economic Analysis of Development)

bread

A website that lists both publicly available data and data that can be obtained at modest cost from the data collection agency. BREAD provides links to some of the data that are on-line and explanations of how to obtain others. This page is maintained by the Center for Health and Development at UCLA.

Source: http://ipl.econ.duke.edu/dthomas/dev_data/index.html
Format: A list of data sources in various formats
Access: Linked lists to outside databases with different criteria
Usage restrictions: Most data are publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Stockholm International Peace Research Institute

sipri

The SIPRI Multilateral Peace Operations Database is a uniquely source of transparent and reliable data on all multilateral peace operations (both UN and non-UN) conducted around the world. It contains information on where an operation is located, the dates of its deployment and operation, the mission mandate, the participating countries, the approved and actual number of personnel deployed, broken down into personnel types, the financial costs and the number of fatalities suffered. The database records all peace operations according to the SIPRI's definition. The database currently includes over 500 peace operations in the period 2000-2008. It is updated continually.

Source: http://www.sipri.org/databases
Format: Returns data from disparate sources in report form
Access: Query data from multiple databases
Usage restrictions: Most source data are publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Demographic and Health Surveys

demographic

The demographic and health surveys (DHS) program has collected, analyzed and disseminated accurate and representative data on population, health, HIV, and nutrition through more than 200 surveys in over 75 countries.

Source: http://www.measuredhs.com/
Format: Different formats
Access: Query databases by country or by topic
Usage restrictions: Data are publicly available for research
Reporting requirements: Must register as a DHS user
Date Listed: September 21, 2009

World Trade Organization

wto

Trade Statistics available in an interactive format. Includes the most up-to-date WTO trade statistics. Includes links to trade profiles, tariff profiles and services profiles. The bound and applied tariff for all members at the standardized Harmonized System 6-digit subheading level are available for download.

Source: http://stat.wto.org/
Format: .xls, .html, .zip
Access: Query databases
Usage restrictions: Data are publicly available for research
Reporting requirements: Check website for details
Date Listed: September 21, 2009

United States Department of Agriculture

usda

A number of data sets on agriculture (primarily US but website does have some aggregate level country and region data) covering subjects on agriculture such as Animal Products, Countries & Regions,Crops, Diet, Health, & Safety, Farm Economy, Farm Practices & Management, Food & Nutrition Assistance, Food Sector, Natural Resources & Environment, Policy Topics, Research & Productivity, Rural Economy, Trade & International Markets, U.S. State and Regional

Source: http://www.ers.usda.gov/Browse/view.aspx?subject=CountriesRegions
Format: Different formats – mainly briefs
Access: Different tools depending on data source available by country
Usage restrictions: Data are publicly available for research
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Princeton Data and Statistical Services: Data on Africa

princeton

A compilation of datasets on Africa.

Source: http://dss1.princeton.edu/cgi-bin/dataresources/newdataresources.cgi?term=14
Format: Depends on source
Access: Depends on source
Usage restrictions: Some data are publicly available, Some restricted to Princeton Faculty and students
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Africover

africover

Geographic data produced by the the Africover Project and the participating countries. The information available in the national Multipurpose Africover Databases on Environmental Resources (MADE) is composed by a main geographic information layer (i.e. land cover) and several additional layers that vary for each country (e.g. roads, rivers and water bodies, etc.); the available data produced by Africover is listed for each country in the Africover Data table (here only full resolution data sets and public domain spatially aggregated data sets are listed. Thematic aggregations are available starting from the metadata of these data sets).

Source: http://www.africover.org/
Format:
Access: If you are interested in accessing the data, please first examine the metadata of the specific data set you are interested in. Afterwards you will be able to proceed to access the data with the appropriate procedure.
Usage restrictions: Please consult the data access policies that rule the access to the different types of data sets.
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Factset

factset

A compilation of various international economic data sets

Source: http://www.factset.com
Format: Returns data from disparate sources in tabular form
Access: Query databases by country or by topic
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Airports Council International

aci

Annual reports: A comprehensive overview of passenger, cargo and traffic movements for the most recent year. Monthly Traffic Data: A pay for service, the monthly passenger, cargo and traffic data available on the website is an overview of the main trends in traffic movement around the world.

Source: http://www.aci.aero
Format: Html, tables
Access: Annual reports are query based, Pay for service
Usage restrictions: See website
Reporting requirements: See website
Date Listed: September 21, 2009

Global Distribution of Poverty

povmap

A website with a collection of subnational, spatially explicit, poverty data sets. This page is maintained by The Poverty Mapping Project at CIESIN (The Center for International Earth Science Information Network) at the Earth Institute at Columbia University.

Source: http://sedac.ciesin.columbia.edu/povmap/ds_info.jsp
Format: XLS, CSV, TXT, Shapefile
Access: Click on links to download datasets
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

International Data Resource Center (IDRC)

idrc

Created by Inter-University Consortium for Political and Social Research, the International Data Resource Center acts as a clearinghouse for international data housed there.

Source: http://www.icpsr.umich.edu/IDRC/sites.html
Format: A list of data sources in various formats
Access: Linked lists to outside databases with different criteria
Usage restrictions: Most data are publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

International Agency For Cancer Research

cancer

Maintained by Cancer Information Section of International Agency For Cancer Research Available data includes: Incidence data collected by cancer registries worldwide : CI5 (Cancer Incidence in Five Continents) Volumes I to IX, the data on incidence and survival of children and adolescents in Europe ( ACCIS ), mortality data extracted from the World Health Organization (WHO) databank, the most recent estimates of the cancer incidence, mortality and prevalence, by sex and cancer site, for all the countries of the world (GLOBOCAN 2002), the incidence, mortality and prevalence statistics from 41 major cancers in the Nordic countries, starting from beginning of registration (the NORDCAN project in collaboration with the Association of the Nordic Cancer Registries).

Source: http://www-dep.iarc.fr/dephome.htm
Format: CSV, TXT
Access: Click on download link to download zip file
Usage restrictions: May be freely used and reproduced, but not for sale or for use in conjunction with commercial or promotional purposes.
Reporting requirements: Any use shall be subject to appropriate reference and acknowledgement of the source.
Date Listed: September 21, 2009

State of The World’s Children

unicef

A website with statistical data that includes economic and social data from 195 countries and territories, with particular reference to children's well-being This page is maintained by UNICEF.

Source: http://www.unicef.org/sowc08/statistics/tables.php
Format: XLS
Access: Click on links to download datasets
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Food and Agriculture Organization of the United Nations

aci

FAOSTAT provides time-series and cross sectional data relating to food and agriculture for some 200 countries. This page is maintained by the UN.

Source: http://faostat.fao.org/site/291/default.aspx
Format: Provides tables based on queries.
Access: Database queries
Usage restrictions: Publicly available
Reporting requirements: Check individual websites for details
Date Listed: September 21, 2009

Organization for Economic Co-Operation and Development

oecd

Monitoring financial flows and their allocation is an important aspect of the Development Assistance Committee's programme of work. The DAC Working Party on Statistics brings international aid statisticians together to look at how statistics respond to key development issues. The DAC databases cover bilateral and multilateral donors’ aid and other resource flows to developing countries.

Source: http://www.oecd.org/dataoecd/50/17/5037721.htm
Format: Provides tables based on queries
Access: Database queries
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

The Center for International Comparisons at the University of Pennsylvania (CICUP)

cicup

The Penn World Table provides purchasing power parity and national income accounts converted to international prices for 188 countries for some or all of the years 1950-2004.

Source: http://pwt.econ.upenn.edu/
Format: Provides tables based on queries
Access: Database queries
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Africa Research Program

arp

Data set consists of an aggregate of a number of the most commonly used publicly available variables used in the study of African political economy.

Source: http://africa.gov.harvard.edu/
Format: Stata, ASCII
Access: Click to download
Usage restrictions: Publicly available
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Comparative Welfare States Data Set

lis

The data contained in this data set were collected by a project entitled "The Welfare State in Comparative Perspective: Determinants, Program Characteristics, and Outcomes". Site maintained by the Luxembourg Income Study.

Source: http://www.lisproject.org/publications/welfaredata/welfareaccess.htm
Format: XLS
Access: Click to download
Usage restrictions: Publicly available
Reporting requirements: In any work using data from this data set, please cite both the data set and, where appropriate, the original source.
Date Listed: September 21, 2009

Center For International Development

cid

This page is a depository for data developed through research at the Center for International Development at Harvard University (CID). Often the data are associated with a research paper and, thus, the paper is also available for downloading.

Source: http://www.cid.harvard.edu/ciddata/ciddata.html
Format: A list of data sources in various formats
Access: Linked lists to outside databases with different criteria
Usage restrictions: Most data are publicly available
Reporting requirements: In any work using data from this data set, please cite both the data set and, where appropriate, the original source.
Date Listed: September 21, 2009

Rwandan Factbook

factbook

A good introduction to data about Rwanda.

Source: https://www.cia.gov/library/publications/the-world-factbook/geos/rw.html
Format A list of basic national data in html
Access: Browser / HTML
Usage restrictions: Public
Reporting requirements: Check website for details
Date Listed: September 21, 2009

Centre for Micro Finance

"the knowledge bank from micro finance"

Source: http://ifmr.ac.in/cmf/resources.html
Format STATA
Access: Browser / HTML
Usage restrictions: Public
Reporting requirements: Check website for details
Date Listed: Feb 21, 2010

Annenberg Research Network on Globalization and Communication


WTO Data
Statistical database of WTO trade
http://stat.wto.org/Home/WSDBHome.aspx?Language=

International trade statistics 1998-2004

World Bank Data

World Bank Data
http://www.worldbank.org/data/

Data on globalization
http://www.worldbank.org/data/databytopic/globalization.html

IMF Data

Global financial data and financial data by country
http://www.imf.org/external/fin.htm

UNESCO Data

Data on newspapers by country by year (Number and circulation)
http://stats.uis.unesco.org/TableViewer/tableView.aspx?ReportId=16

Data on culture and communication
http://www.sociology.emory.edu/globalization/data.html

Data on education
http://stats.uis.unesco.org/ReportFolders/reportfolders.aspx

UNICEF Data

Data on the condition of children organized by country
http://www.unicef.org/statistics/index.html

War and Peace Data

http://www.umich.edu/~cowproj/

UN General Data

A master list of all the database of international statistics
http://unstats.un.org/unsd/methods/inter-natlinks/sd_intstat.htm

World Population
http://www.un.org/esa/population/unpop.htm

Global environment data
http://www.unep.org/Evaluation/default.htm

UNAIDS: data on global HIV/AIDS situation
http://www.unaids.org/en/resources/epidemiology.asp

Food and agriculture organization: data on food production and trade, land use, nutrition, etc.
http://apps.fao.org/default.jsp

Department of economic and social affairs
http://unstats.un.org/unsd/methods/inter-natlinks/sd_intstat.htm

UN crime and justice information network
http://www.uncjin.org/Statistics/statistics.html

Demographic Health Surveys

http://www.measuredhs.com/

Global policy Forum

Provide data that measure globalization in terms of economy, politics, culture, law, and technology. Includes many case studies.
http://www.globalpolicy.org/globaliz/charts/

OCED

Data organized by topics, including economy, social welfare, education, science, development, etc.
http://www.oecd.org/statsportal/0,2639,en_2825_293564_1_1_1_1_1,00.html