Data sources¶
SDMX makes a distinction between data providers and sources:
a data provider is the original publisher of statistical information and metadata.
a data source is a specific web service that provides access to statistical information.
Each data source might aggregate and provide data or metadata from multiple data providers. Or, an agency might operate a data source that only contains information they provide themselves; in this case, the source and provider are identical.
pandaSDMX identifies each data source using a string such as 'ABS'
, and has
built-in support for a number of data sources. Use list_sources()
to list
these. Read the following sections, or the file sources.json
in the
package source code, for more details.
pandaSDMX also supports adding other data sources; see add_source()
and Source
.
Data source limitations¶
Each SDMX web service provides a subset of the full SDMX feature set, so the same request made to two different sources may yield different results, or an error message.
A key difference is between sources offering SDMX-ML and SDMX-JSON APIs. SDMX-JSON APIs do not support metadata, or structure queries; only data queries.
Note
For JSON APIs, start by browsing the source’s website to retrieve the dataflow you’re interested in. Then try to fine-tune a planned data request by providing a valid key (= selection of series from the dataset).
Because structure metadata is unavailable, pandaSDMX
cannot automatically validate keys.
In order to anticipate and handle these differences:
add_source()
accepts “data_content_type” and “supported” keys. For example:[ { "id": "ABS", "data_content_type": "JSON" }, { "id": "UNESCO", "unsupported": ["datastructure"] }, ]
pandaSDMX will raise
NotImplementedError
on an attempt to query the “datastructure” API endpoint of either of these data sources.pandasdmx.source
includes adapters (subclasses ofSource
) with hooks used when querying sources and interpreting their HTTP responses. These are documented below: ABS, ESTAT, and SGR.
ABS
: Australian Bureau of Statistics¶
SDMX-JSON — Website
Note
This is deprecated. Use new XML-based API with ID “ABS_XML”. Se below. In future versions, ABS will refer to the new API.
- class pandasdmx.source.abs.Source(*, id: str, api_id: str | None = None, url: HttpUrl | None = None, name: str, documentation: HttpUrl | None = None, headers: Dict[str, Any] = {}, resource_urls: Dict[str, HttpUrl] = {}, default_version: str = 'latest', data_content_type: DataContentType = DataContentType.XML, supports: Dict[str | Resource, bool] = {Resource.data: True})[source]¶
ABS_XML
: Australian Bureau of Statistics - XML-based API¶
SDMX-XML —
This will be renamed to “ABS” in a future version. The new API supports dataflow, datastructure and other SDMX 2.1-compliant resources. It is recommended to use this new API. However, ABS still marks itas Beta (2022-01-25) and hints at potential delays in data publication.
BBK
: Bundesbank (German Central Bank)¶
SDMX-ML — Website
BIS
: Bank for International Settlements¶
SDMX-ML — Website
service went live in April 2021
Supports preview_data and series-key based key validation.
EC_COMP
: European Commission - DG Competition¶
SDMX-ML — Website
Dataflows on competition, mainly on state aids.
EC_EMPL
: European Commission - DG Employment¶
SDMX-ML — Website
Dataflows on employment.
EC_GROW
: European Commission - DG Growth¶
SDMX-ML — Website
Dataflows on growth / industrial policy.
ECB
: European Central Bank¶
SDMX-ML — Website
Supports categorisations of data-flows.
Supports preview_data and series-key based key validation.
In general short response times.
ESTAT
: Eurostat¶
SDMX-ML — Website
Thousands of dataflows on a wide range of topics.
Does not return DSDs for dataflow requests with the
references='all'
query parameter.
- class pandasdmx.source.estat.Source(*, id: str, api_id: str | None = None, url: HttpUrl | None = None, name: str, documentation: HttpUrl | None = None, headers: Dict[str, Any] = {}, resource_urls: Dict[str, HttpUrl] = {}, default_version: str = 'latest', data_content_type: DataContentType = DataContentType.XML, supports: Dict[str | Resource, bool] = {Resource.data: True})[source]¶
ILO
: International Labour Organization¶
SDMX-ML — Website
The ILO SDMX web service API has been updated in 2020. The adapter shipped with pandasdmx until v1.3.0 is now counterproductive and has been removed in v1.3.1.
Data flow IDs take on the role of a filter. E.g., there are dataflows for individual countries, ages, sexes etc. rather than merely for different indicators.
It is highly recommended to read the User guide.
IMF
: International Monetary Fund’s “SDMX Central” source¶
SDMX-ML — Website
The SDMX REST API no longer accepts data queries. But queries for metadata still work. Datasets must be retrieved manually from https://data.imf.org.
INEGI
: National Institute of Statistics and Geography (Mexico)¶
SDMX-ML — Website.
Spanish name: Instituto Nacional de Estadística y Geografía.
INSEE
: National Institute of Statistics and Economic Studies (France)¶
SDMX-ML — Website
French name: Institut national de la statistique et des études économiques.
Warning
An issue has been reported apparently due to a missing pericite codelist in StructureMessages. This may cause crashes. Avoid downloading this type of message. Prepare the key as string using the web interface, and simply download a dataset.
ISTAT
: National Institute of Statistics (Italy)¶
SDMX-ML — Website
Italian name: Istituto Nazionale di Statistica.
Similar server platform to Eurostat, with similar capabilities.
LSD
: National Institute of Statistics (Lithuania)¶
SDMX-ML — Website
NB
: Norges Bank (Norway)¶
SDMX-ML — Website
Few dataflows. So do not use categoryscheme.
It is unknown whether NB supports series-keys-only.
NBB
: National Bank of Belgium¶
SDMX-JSON only. Only data queries supported. Discover dataflows on the website and query the required data sets.
OECD
: Organisation for Economic Cooperation and Development¶
SDMX-JSON — Website
SGR
: SDMX Global Registry¶
SDMX-ML — Website
- class pandasdmx.source.sgr.Source(*, id: str, api_id: str | None = None, url: HttpUrl | None = None, name: str, documentation: HttpUrl | None = None, headers: Dict[str, Any] = {}, resource_urls: Dict[str, HttpUrl] = {}, default_version: str = 'latest', data_content_type: DataContentType = DataContentType.XML, supports: Dict[str | Resource, bool] = {Resource.data: True})[source]¶
SPC
: Pacific Data Hub¶
SDMX-ML — Website This service also offers SDMXJSON datasets. This feature requires a specific HTTP header as described on the website. There seems to be an on SPC’s side in series-key-only data messages as the reference to the DSD’s is not recognizsed.
STAT_EE
: Statistics Estonia¶
SDMX-JSON. Data queries only. No metadata. Discover dataflows through the website.
UNSD
: United Nations Statistics Division¶
SDMX-ML — Website
Supports preview_data and series-key based key validation.
Warning
supports categoryscheme even though it offers very few dataflows. Use this feature with caution. Moreover, it seems that categories confusingly include dataflows which UNSD does not actually provide.
UNICEF
: UN International Children’s Emergency Fund¶
SDMX-ML — Website
CD2030
: Countdown 2030¶
SDMX-ML — Website
WB
: World Bank Group’s “World Integrated Trade Solution”¶
SDMX-ML — Website
WB_WDI
: World Bank Group’s “World Development Indicators”¶
SDMX-ML — Website This service also offers SDMXJSON datasets. This feature requires a specific HTTP header as described on the website. Queries for a list of dataflows do not seem to be supported.