pandaSDMX: Statistical Data and Metadata eXchange in Python

pandaSDMX is an Apache 2.0-licensed Python client to retrieve and acquire statistical data and metadata disseminated in SDMX 2.1, an ISO-standard widely used by institutions such as statistics offices, central banks, and international organisations. pandaSDMX exposes datasets and related structural metadata including dataflows, codelists, and datastructure definitions as pandas Series or multi-indexed DataFrames. Many other output formats and storage backends are available thanks to Odo.

Main features

  • support for many SDMX 2.1 features
  • SDMXML and SDMXJSON formats
  • pythonic representation of the SDMX information model
  • When requesting datasets, validate column selections against code lists and content-constraints if available
  • export data and structural metadata such as code lists as multi-indexed pandas DataFrames or Series, and many other formats as well as database backends via Odo
  • read and write SDMX messages to and from files
  • configurable HTTP connections
  • support for requests-cache allowing to cache SDMX messages in memory, MongoDB, Redis or SQLite
  • extensible through custom readers and writers for alternative input and output formats
  • growing test suite

Example

Suppose we want to analyze annual unemployment data for some European countries. All we need to know in advance is the data provider, eurostat. pandaSDMX makes it super easy to search the directory of dataflows, and analyze the complete structural metadata about the datasets available through the selected dataflow. We will skip this step here. The impatient reader may directly jump to Basic usage. The dataflow with the ID ‘une_rt_a’ contains the data we want. The dataflow definition references the datastructure definition which contains or references all the metadata describing data sets available through this dataflow: the dimensions, concept schemes, and corresponding code lists.

In [1]: from pandasdmx import Request

In [2]: estat = Request('ESTAT')

# Download the metadata and expose it as a dict mapping resource names to pandas DataFrames
In [3]: flow_response = estat.dataflow('une_rt_a')

In [4]: structure_response = flow_response.dataflow.une_rt_a.structure(request=True, target_only=False)

# Show some code lists.
In [5]: structure_response.write().codelist.loc['GEO'].head()
Out[5]: 
    dim_or_attr      name
GEO           D       GEO
AT            D   Austria
BE            D   Belgium
BG            D  Bulgaria
CY            D    Cyprus

Next we download a dataset. We use codes from the code list ‘GEO’ to obtain data on Greece, Ireland and Spain only.

In [6]: resp = estat.data('une_rt_a', key={'GEO': 'EL+ES+IE'}, params={'startPeriod': '2007'})

# We use a generator expression to select some columns
# and write them to a pandas DataFrame
In [7]: data = resp.write(s for s in resp.data.series if s.key.AGE == 'TOTAL')

# Explore the data set. First, show dimension names
In [8]: data.columns.names
Out[8]: FrozenList(['UNIT', 'AGE', 'SEX', 'GEO', 'FREQ'])

# and corresponding dimension values
In [9]: data.columns.levels
Out[9]: FrozenList([['PC_ACT', 'PC_POP', 'THS_PER'], ['TOTAL'], ['F', 'M', 'T'], ['EL', 'ES', 'IE'], ['A']])

# Show aggregate unemployment rates across ages and sexes as
# percentage of active population
In [10]: data.loc[:, ('PC_ACT', 'TOTAL', 'T')]
Out[10]: 
GEO            EL    ES    IE
FREQ            A     A     A
TIME_PERIOD                  
2018         19.3  15.3   5.8
2017         21.5  17.2   6.7
2016         23.6  19.6   8.4
2015         24.9  22.1  10.0
2014         26.5  24.5  11.9
2013         27.5  26.1  13.8
2012         24.5  24.8  15.5
2011         17.9  21.4  15.4
2010         12.7  19.9  14.6
2009          9.6  17.9  12.6
2008          7.8  11.3   6.8
2007          8.4   8.2   5.0

Quick install

  • pip install pandasdmx

Indices and tables