Data sources

SDMX makes a distinction between data providers and sources:

  • a data provider is the original publisher of statistical information and metadata.

  • a data source is a specific web service that provides access to statistical information.

Each data source might aggregate and provide data or metadata from multiple data providers. Or, an agency might operate a data source that only contains information they provide themselves; in this case, the source and provider are identical.

pandaSDMX identifies each data source using a string such as 'ABS', and has built-in support for a number of data sources. Use list_sources() to list these. Read the following sections, or the file sources.json in the package source code, for more details.

pandaSDMX also supports adding other data sources; see add_source() and Source.

Data source limitations

Each SDMX web service provides a subset of the full SDMX feature set, so the same request made to two different sources may yield different results, or an error message.

A key difference is between sources offering SDMX-ML and SDMX-JSON APIs. SDMX-JSON APIs do not support metadata, or structure queries; only data queries.

Note

For JSON APIs, start by browsing the source’s website to retrieve the dataflow you’re interested in. Then try to fine-tune a planned data request by providing a valid key (= selection of series from the dataset). Because structure metadata is unavailable, pandaSDMX cannot automatically validate keys.

In order to anticipate and handle these differences:

  1. add_source() accepts “data_content_type” and “supported” keys. For example:

    [
      {
        "id": "ABS",
        "data_content_type": "JSON"
      },
      {
        "id": "UNESCO",
        "unsupported": ["datastructure"]
      },
    ]
    

    pandaSDMX will raise NotImplementedError on an attempt to query the “datastructure” API endpoint of either of these data sources.

  2. pandasdmx.source includes adapters (subclasses of Source) with hooks used when querying sources and interpreting their HTTP responses. These are documented below: ABS, ESTAT, and SGR.

ABS: Australian Bureau of Statistics

SDMX-JSON — Website

Note

This is deprecated. Use new XML-based API with ID “ABS_XML”. Se below. In future versions, ABS will refer to the new API.

class pandasdmx.source.abs.Source(*, id: str, api_id: str | None = None, url: HttpUrl | None = None, name: str, documentation: HttpUrl | None = None, headers: Dict[str, Any] = {}, resource_urls: Dict[str, HttpUrl] = {}, default_version: str = 'latest', data_content_type: DataContentType = DataContentType.XML, supports: Dict[str | Resource, bool] = {Resource.data: True})[source]
handle_response(response, content)[source]

Handle ABS’ own text/html error page for some endpoints.

ABS_XML: Australian Bureau of Statistics - XML-based API

SDMX-XML —

This will be renamed to “ABS” in a future version. The new API supports dataflow, datastructure and other SDMX 2.1-compliant resources. It is recommended to use this new API. However, ABS still marks itas Beta (2022-01-25) and hints at potential delays in data publication.

BBK: Bundesbank (German Central Bank)

SDMX-ML — Website

BIS: Bank for International Settlements

SDMX-ML — Website

  • service went live in April 2021

  • Supports preview_data and series-key based key validation.

EC_COMP: European Commission - DG Competition

SDMX-ML — Website

  • Dataflows on competition, mainly on state aids.

EC_EMPL: European Commission - DG Employment

SDMX-ML — Website

  • Dataflows on employment.

EC_GROW: European Commission - DG Growth

SDMX-ML — Website

  • Dataflows on growth / industrial policy.

ECB: European Central Bank

SDMX-ML — Website

  • Supports categorisations of data-flows.

  • Supports preview_data and series-key based key validation.

  • In general short response times.

ESTAT: Eurostat

SDMX-ML — Website

  • Thousands of dataflows on a wide range of topics.

  • Does not return DSDs for dataflow requests with the references='all' query parameter.

class pandasdmx.source.estat.Source(*, id: str, api_id: str | None = None, url: HttpUrl | None = None, name: str, documentation: HttpUrl | None = None, headers: Dict[str, Any] = {}, resource_urls: Dict[str, HttpUrl] = {}, default_version: str = 'latest', data_content_type: DataContentType = DataContentType.XML, supports: Dict[str | Resource, bool] = {Resource.data: True})[source]

ILO: International Labour Organization

SDMX-ML — Website

The ILO SDMX web service API has been updated in 2020. The adapter shipped with pandasdmx until v1.3.0 is now counterproductive and has been removed in v1.3.1.

  • Data flow IDs take on the role of a filter. E.g., there are dataflows for individual countries, ages, sexes etc. rather than merely for different indicators.

  • It is highly recommended to read the User guide.

IMF: International Monetary Fund’s “SDMX Central” source

SDMX-ML — Website

  • The SDMX REST API no longer accepts data queries. But queries for metadata still work. Datasets must be retrieved manually from https://data.imf.org.

INEGI: National Institute of Statistics and Geography (Mexico)

SDMX-ML — Website.

  • Spanish name: Instituto Nacional de Estadística y Geografía.

INSEE: National Institute of Statistics and Economic Studies (France)

SDMX-ML — Website

  • French name: Institut national de la statistique et des études économiques.

Warning

An issue has been reported apparently due to a missing pericite codelist in StructureMessages. This may cause crashes. Avoid downloading this type of message. Prepare the key as string using the web interface, and simply download a dataset.

ISTAT: National Institute of Statistics (Italy)

SDMX-ML — Website

  • Italian name: Istituto Nazionale di Statistica.

  • Similar server platform to Eurostat, with similar capabilities.

LSD: National Institute of Statistics (Lithuania)

SDMX-ML — Website

NB: Norges Bank (Norway)

SDMX-ML — Website

  • Few dataflows. So do not use categoryscheme.

  • It is unknown whether NB supports series-keys-only.

NBB: National Bank of Belgium

SDMX-JSON only. Only data queries supported. Discover dataflows on the website and query the required data sets.

Website — API documentation (en)

OECD: Organisation for Economic Cooperation and Development

SDMX-JSON — Website

SGR: SDMX Global Registry

SDMX-ML — Website

class pandasdmx.source.sgr.Source(*, id: str, api_id: str | None = None, url: HttpUrl | None = None, name: str, documentation: HttpUrl | None = None, headers: Dict[str, Any] = {}, resource_urls: Dict[str, HttpUrl] = {}, default_version: str = 'latest', data_content_type: DataContentType = DataContentType.XML, supports: Dict[str | Resource, bool] = {Resource.data: True})[source]
handle_response(response, content)[source]

SGR responses do not specify content-type; set it directly.

SPC: Pacific Data Hub

SDMX-ML — Website This service also offers SDMXJSON datasets. This feature requires a specific HTTP header as described on the website. There seems to be an on SPC’s side in series-key-only data messages as the reference to the DSD’s is not recognizsed.

STAT_EE: Statistics Estonia

SDMX-JSON. Data queries only. No metadata. Discover dataflows through the website.

Website (et) — API documentation (en), (et)

UNSD: United Nations Statistics Division

SDMX-ML — Website

  • Supports preview_data and series-key based key validation.

Warning

supports categoryscheme even though it offers very few dataflows. Use this feature with caution. Moreover, it seems that categories confusingly include dataflows which UNSD does not actually provide.

UNICEF: UN International Children’s Emergency Fund

SDMX-ML — Website

CD2030: Countdown 2030

SDMX-ML — Website

WB: World Bank Group’s “World Integrated Trade Solution”

SDMX-ML — Website

WB_WDI: World Bank Group’s “World Development Indicators”

SDMX-ML — Website This service also offers SDMXJSON datasets. This feature requires a specific HTTP header as described on the website. Queries for a list of dataflows do not seem to be supported.