Open Data and APIs

Why this matters

Some of the best data available costs nothing: population figures from the World Bank, global statistics from the UN, competition datasets on Kaggle. Free is not the same as unconditional, though, and an analysis built on data you were not entitled to use, or whose origin you cannot state, is not an analysis anyone should rely on.

The idea

Open data is published for anyone to use. It comes from international organisations, government agencies, researchers and companies, and reliability varies, so the source matters as much as the contents.

You can search and download from a browser, and often fetch the same data through an API. An API lets your program get the data directly, which matters when there is too much to click through by hand. It runs in three steps: the client sends a request to the endpoint, the URL the request goes to; the server processes it and returns the data in an agreed format; the client takes the parts it needs.

What comes back is usually JSON, which represents data as names paired with values, readable by people and easy to handle in any language.

Two obligations come with all of this. Check the terms of use, which are set per dataset, before you use anything, and do not hammer an API with excessive requests. And give a source citation: provider, dataset name, date obtained, and URL. That is both a claim about reliability and what lets somebody else check your work.

Picture it

sequenceDiagram
  participant P as Your program (client)
  participant A as API endpoint
  P->>A: request for the data you need
  Note over A: server processes the request
  A-->>P: response, usually JSON
  Note over P: extract the parts you need,<br/>record provider, name, date, URL

Worked example

You want ten years of population figures for twenty countries.

Downloading twenty spreadsheets by hand works once and then rots: next year's update means doing it all again, and any mistake in copying is invisible. An API call instead takes the country codes and years as parameters and returns exactly those rows, in JSON, in one request. Re-running it next year is one command.

Before using any of it, read the terms of use. World Bank data is broadly reusable with attribution, but some datasets on other platforms are restricted to non-commercial use, or forbid redistribution. That is a decision you have to make before the analysis, not after somebody asks.

Then record the source citation: the provider, the dataset name, the date you pulled it, and the URL. The date is the part people forget and the part that matters most, because these figures get revised. An analysis citing a pull date can be reproduced; one without it cannot be, even by you.

Your turn

Try the practice questions for this lesson. Judge statements about open data and APIs, match each provider to what it publishes, and identify what you must check before using a free dataset.

Take it further

For a project you want to show how internet access in Egypt has changed over ten years compared with two neighbouring countries. Should you take the first dataset you find, or choose a reliable source and follow its rules?

1. Inspect two sources. Look up two candidate providers for this exact question. For each, note what it actually covers, how current it is, and what its terms of use permit.
2. Choose how to fetch it. Decide whether a browser download or an API call suits better, and say why.
3. Decide. Recommend one source, and write the full citation you would publish: provider, dataset name, date obtained and URL.

The date is the part everyone omits and the part that matters most, because these figures get revised.

Recap

• Open data is free to use but not free of conditions; reliability depends on the provider.
• World Bank Open Data, UN Data and Kaggle are the main sources to know.
• An API request goes to an endpoint and comes back, usually as JSON.
• Always check the terms of use, and always give a source citation including the date obtained.