[
Process JSON using Pandas¶
Let us understand how to process JSON using Pandas.
- We can use
read_json
to read JSON documents from file into a Data Frame. - It works well with customers.json where we have one valid JSON document per line.
In [1]:
import pandas as pd
In [2]:
pd.read_json?
Signature: pd.read_json( path_or_buf=None, orient=None, typ='frame', dtype: 'DtypeArg | None' = None, convert_axes=None, convert_dates=True, keep_default_dates: 'bool' = True, numpy: 'bool' = False, precise_float: 'bool' = False, date_unit=None, encoding=None, encoding_errors: 'str | None' = 'strict', lines: 'bool' = False, chunksize: 'int | None' = None, compression: 'CompressionOptions' = 'infer', nrows: 'int | None' = None, storage_options: 'StorageOptions' = None, ) Docstring: Convert a JSON string to pandas object. Parameters ---------- path_or_buf : a valid JSON str, path object or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.json``. If you want to pass in a path object, pandas accepts any ``os.PathLike``. By file-like object, we refer to objects with a ``read()`` method, such as a file handle (e.g. via builtin ``open`` function) or ``StringIO``. orient : str Indication of expected JSON string format. Compatible JSON strings can be produced by ``to_json()`` with a corresponding orient value. The set of possible orients is: - ``'split'`` : dict like ``{index -> [index], columns -> [columns], data -> [values]}`` - ``'records'`` : list like ``[{column -> value}, ... , {column -> value}]`` - ``'index'`` : dict like ``{index -> {column -> value}}`` - ``'columns'`` : dict like ``{column -> {index -> value}}`` - ``'values'`` : just the values array The allowed and default values depend on the value of the `typ` parameter. * when ``typ == 'series'``, - allowed orients are ``{'split','records','index'}`` - default is ``'index'`` - The Series index must be unique for orient ``'index'``. * when ``typ == 'frame'``, - allowed orients are ``{'split','records','index', 'columns','values', 'table'}`` - default is ``'columns'`` - The DataFrame index must be unique for orients ``'index'`` and ``'columns'``. - The DataFrame columns must be unique for orients ``'index'``, ``'columns'``, and ``'records'``. typ : {'frame', 'series'}, default 'frame' The type of object to recover. dtype : bool or dict, default None If True, infer dtypes; if a dict of column to dtype, then use those; if False, then don't infer dtypes at all, applies only to the data. For all ``orient`` values except ``'table'``, default is True. .. versionchanged:: 0.25.0 Not applicable for ``orient='table'``. convert_axes : bool, default None Try to convert the axes to the proper dtypes. For all ``orient`` values except ``'table'``, default is True. .. versionchanged:: 0.25.0 Not applicable for ``orient='table'``. convert_dates : bool or list of str, default True If True then default datelike columns may be converted (depending on keep_default_dates). If False, no dates will be converted. If a list of column names, then those columns will be converted and default datelike columns may also be converted (depending on keep_default_dates). keep_default_dates : bool, default True If parsing dates (convert_dates is not False), then try to parse the default datelike columns. A column label is datelike if * it ends with ``'_at'``, * it ends with ``'_time'``, * it begins with ``'timestamp'``, * it is ``'modified'``, or * it is ``'date'``. numpy : bool, default False Direct decoding to numpy arrays. Supports numeric data only, but non-numeric column and index labels are supported. Note also that the JSON ordering MUST be the same for each term if numpy=True. .. deprecated:: 1.0.0 precise_float : bool, default False Set to enable usage of higher precision (strtod) function when decoding string to double values. Default (False) is to use fast but less precise builtin functionality. date_unit : str, default None The timestamp unit to detect if converting dates. The default behaviour is to try and detect the correct precision, but if this is not desired then pass one of 's', 'ms', 'us' or 'ns' to force parsing only seconds, milliseconds, microseconds or nanoseconds respectively. encoding : str, default is 'utf-8' The encoding to use to decode py3 bytes. encoding_errors : str, optional, default "strict" How encoding errors are treated. `List of possible values <https://docs.python.org/3/library/codecs.html#error-handlers>`_ . .. versionadded:: 1.3.0 lines : bool, default False Read the file as a json object per line. chunksize : int, optional Return JsonReader object for iteration. See the `line-delimited json docs <https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#line-delimited-json>`_ for more information on ``chunksize``. This can only be passed if `lines=True`. If this is None, the file will be read into memory all at once. .. versionchanged:: 1.2 ``JsonReader`` is a context manager. compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None}, default 'infer' For on-the-fly decompression of on-disk data. If 'infer', then use gzip, bz2, zip or xz if path_or_buf is a string ending in '.gz', '.bz2', '.zip', or 'xz', respectively, and no decompression otherwise. If using 'zip', the ZIP file must contain only one data file to be read in. Set to None for no decompression. nrows : int, optional The number of lines from the line-delimited jsonfile that has to be read. This can only be passed if `lines=True`. If this is None, all the rows will be returned. .. versionadded:: 1.1 storage_options : dict, optional Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to ``urllib`` as header options. For other URLs (e.g. starting with "s3://", and "gcs://") the key-value pairs are forwarded to ``fsspec``. Please see ``fsspec`` and ``urllib`` for more details. .. versionadded:: 1.2.0 Returns ------- Series or DataFrame The type returned depends on the value of `typ`. See Also -------- DataFrame.to_json : Convert a DataFrame to a JSON string. Series.to_json : Convert a Series to a JSON string. Notes ----- Specific to ``orient='table'``, if a :class:`DataFrame` with a literal :class:`Index` name of `index` gets written with :func:`to_json`, the subsequent read operation will incorrectly set the :class:`Index` name to ``None``. This is because `index` is also used by :func:`DataFrame.to_json` to denote a missing :class:`Index` name, and the subsequent :func:`read_json` operation cannot distinguish between the two. The same limitation is encountered with a :class:`MultiIndex` and any names beginning with ``'level_'``. Examples -------- >>> df = pd.DataFrame([['a', 'b'], ['c', 'd']], ... index=['row 1', 'row 2'], ... columns=['col 1', 'col 2']) Encoding/decoding a Dataframe using ``'split'`` formatted JSON: >>> df.to_json(orient='split') '{"columns":["col 1","col 2"],"index":["row 1","row 2"],"data":[["a","b"],["c","d"]]}' >>> pd.read_json(_, orient='split') col 1 col 2 row 1 a b row 2 c d Encoding/decoding a Dataframe using ``'index'`` formatted JSON: >>> df.to_json(orient='index') '{"row 1":{"col 1":"a","col 2":"b"},"row 2":{"col 1":"c","col 2":"d"}}' >>> pd.read_json(_, orient='index') col 1 col 2 row 1 a b row 2 c d Encoding/decoding a Dataframe using ``'records'`` formatted JSON. Note that index labels are not preserved with this encoding. >>> df.to_json(orient='records') '[{"col 1":"a","col 2":"b"},{"col 1":"c","col 2":"d"}]' >>> pd.read_json(_, orient='records') col 1 col 2 0 a b 1 c d Encoding with Table Schema >>> df.to_json(orient='table') '{"schema":{"fields":[{"name":"index","type":"string"},{"name":"col 1","type":"string"},{"name":"col 2","type":"string"}],"primaryKey":["index"],"pandas_version":"0.20.0"},"data":[{"index":"row 1","col 1":"a","col 2":"b"},{"index":"row 2","col 1":"c","col 2":"d"}]}' File: ~/.local/lib/python3.8/site-packages/pandas/io/json/_json.py Type: function
In [3]:
pd.read_json('customers.json', lines=True)
Out[3]:
id | first_name | last_name | gender | ip_address | ||
---|---|---|---|---|---|---|
0 | 1 | Frasco | Necolds | fnecolds0@vk.com | Male | 243.67.63.34 |
1 | 2 | Dulce | Santos | dsantos1@mashable.com | Female | 60.30.246.227 |
2 | 3 | Prissie | Tebbett | ptebbett2@infoseek.co.jp | Genderfluid | 22.21.162.56 |
3 | 4 | Schuyler | Coppledike | scoppledike3@gnu.org | Agender | 120.35.186.161 |
4 | 5 | Leopold | Jarred | ljarred4@wp.com | Agender | 30.119.34.4 |
5 | 6 | Joanna | Teager | jteager5@apache.org | Bigender | 245.221.176.34 |
6 | 7 | Lion | Beere | lbeere6@bloomberg.com | Polygender | 105.54.139.46 |
7 | 8 | Marabel | Wornum | mwornum7@posterous.com | Polygender | 247.229.14.25 |
8 | 9 | Helenka | Mullender | hmullender8@cloudflare.com | Non-binary | 133.216.118.88 |
9 | 10 | Christine | Swane | cswane9@shop-pro.jp | Polygender | 86.16.210.164 |
- It is not straight forward to create data frame using youtube_playlist_items.json where we have one single JSON document with multiple attributes.
- We can extract items and create data frame using
pd.DataFrame
by passing the list of dicts to it.
In [4]:
import json
In [5]:
type(open('youtube_playlist_items.json'))
Out[5]:
_io.TextIOWrapper
In [6]:
json.load(open('youtube_playlist_items.json'))['items']
Out[6]:
[{'kind': 'youtube#playlistItem', 'etag': 'SGHDydc4dLsY2RjfXTPneb_zc_s', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy5EQkE3RTJCQTJEQkFBQTcz', 'contentDetails': {'videoId': 'ETZJln4jtAo', 'videoPublishedAt': '2020-11-28T16:29:47Z'}, 'status': {'privacyStatus': 'public'}}, {'kind': 'youtube#playlistItem', 'etag': '5EFUNhJBvcwXPxO416VYQsXGzMo', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy4yQzk4QTA5QjkzMTFFOEI1', 'contentDetails': {'videoId': '1OVHjHTkP3M', 'videoPublishedAt': '2020-11-28T16:30:12Z'}, 'status': {'privacyStatus': 'public'}}, {'kind': 'youtube#playlistItem', 'etag': 'TiKqB2aeYxJjMGKQ0yLMJY0vpQE', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy45NDlDQUFFOThDMTAxQjUw', 'contentDetails': {'videoId': 'qfUbPLsLQcQ', 'videoPublishedAt': '2020-11-28T16:30:33Z'}, 'status': {'privacyStatus': 'public'}}, {'kind': 'youtube#playlistItem', 'etag': 'vQrJOpYdXmGJuV32kjj2xqvSByc', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy4xN0Y2QjVBOEI2MzQ5OUM5', 'contentDetails': {'videoId': 'rLTbhSaXhSM', 'videoPublishedAt': '2020-11-28T16:30:52Z'}, 'status': {'privacyStatus': 'public'}}, {'kind': 'youtube#playlistItem', 'etag': '2CzGUToIgqywXAr4wuPswj9MuFg', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy5FQUY2Qzk4RUFDN0ZFRkZF', 'contentDetails': {'videoId': 'wP7BhXrJKR8', 'videoPublishedAt': '2020-11-28T16:31:14Z'}, 'status': {'privacyStatus': 'public'}}]
In [7]:
json.load(open('youtube_playlist_items.json'))['items'][0]
Out[7]:
{'kind': 'youtube#playlistItem', 'etag': 'SGHDydc4dLsY2RjfXTPneb_zc_s', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy5EQkE3RTJCQTJEQkFBQTcz', 'contentDetails': {'videoId': 'ETZJln4jtAo', 'videoPublishedAt': '2020-11-28T16:29:47Z'}, 'status': {'privacyStatus': 'public'}}
In [8]:
yt_items = json.load(open('youtube_playlist_items.json'))['items']
In [9]:
type(yt_items)
Out[9]:
list
In [10]:
yt_items
Out[10]:
[{'kind': 'youtube#playlistItem', 'etag': 'SGHDydc4dLsY2RjfXTPneb_zc_s', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy5EQkE3RTJCQTJEQkFBQTcz', 'contentDetails': {'videoId': 'ETZJln4jtAo', 'videoPublishedAt': '2020-11-28T16:29:47Z'}, 'status': {'privacyStatus': 'public'}}, {'kind': 'youtube#playlistItem', 'etag': '5EFUNhJBvcwXPxO416VYQsXGzMo', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy4yQzk4QTA5QjkzMTFFOEI1', 'contentDetails': {'videoId': '1OVHjHTkP3M', 'videoPublishedAt': '2020-11-28T16:30:12Z'}, 'status': {'privacyStatus': 'public'}}, {'kind': 'youtube#playlistItem', 'etag': 'TiKqB2aeYxJjMGKQ0yLMJY0vpQE', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy45NDlDQUFFOThDMTAxQjUw', 'contentDetails': {'videoId': 'qfUbPLsLQcQ', 'videoPublishedAt': '2020-11-28T16:30:33Z'}, 'status': {'privacyStatus': 'public'}}, {'kind': 'youtube#playlistItem', 'etag': 'vQrJOpYdXmGJuV32kjj2xqvSByc', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy4xN0Y2QjVBOEI2MzQ5OUM5', 'contentDetails': {'videoId': 'rLTbhSaXhSM', 'videoPublishedAt': '2020-11-28T16:30:52Z'}, 'status': {'privacyStatus': 'public'}}, {'kind': 'youtube#playlistItem', 'etag': '2CzGUToIgqywXAr4wuPswj9MuFg', 'id': 'UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy5FQUY2Qzk4RUFDN0ZFRkZF', 'contentDetails': {'videoId': 'wP7BhXrJKR8', 'videoPublishedAt': '2020-11-28T16:31:14Z'}, 'status': {'privacyStatus': 'public'}}]
In [11]:
pd.DataFrame(yt_items)
Out[11]:
kind | etag | id | contentDetails | status | |
---|---|---|---|---|---|
0 | youtube#playlistItem | SGHDydc4dLsY2RjfXTPneb_zc_s | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | {‘videoId’: ‘ETZJln4jtAo’, ‘videoPublishedAt’:… | {‘privacyStatus’: ‘public’} |
1 | youtube#playlistItem | 5EFUNhJBvcwXPxO416VYQsXGzMo | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | {‘videoId’: ‘1OVHjHTkP3M’, ‘videoPublishedAt’:… | {‘privacyStatus’: ‘public’} |
2 | youtube#playlistItem | TiKqB2aeYxJjMGKQ0yLMJY0vpQE | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | {‘videoId’: ‘qfUbPLsLQcQ’, ‘videoPublishedAt’:… | {‘privacyStatus’: ‘public’} |
3 | youtube#playlistItem | vQrJOpYdXmGJuV32kjj2xqvSByc | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | {‘videoId’: ‘rLTbhSaXhSM’, ‘videoPublishedAt’:… | {‘privacyStatus’: ‘public’} |
4 | youtube#playlistItem | 2CzGUToIgqywXAr4wuPswj9MuFg | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | {‘videoId’: ‘wP7BhXrJKR8’, ‘videoPublishedAt’:… | {‘privacyStatus’: ‘public’} |
In [12]:
pd.json_normalize(json.load(open('youtube_playlist_items.json'))['items']) # nested jsons are flattened
Out[12]:
kind | etag | id | contentDetails.videoId | contentDetails.videoPublishedAt | status.privacyStatus | |
---|---|---|---|---|---|---|
0 | youtube#playlistItem | SGHDydc4dLsY2RjfXTPneb_zc_s | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | ETZJln4jtAo | 2020-11-28T16:29:47Z | public |
1 | youtube#playlistItem | 5EFUNhJBvcwXPxO416VYQsXGzMo | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | 1OVHjHTkP3M | 2020-11-28T16:30:12Z | public |
2 | youtube#playlistItem | TiKqB2aeYxJjMGKQ0yLMJY0vpQE | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | qfUbPLsLQcQ | 2020-11-28T16:30:33Z | public |
3 | youtube#playlistItem | vQrJOpYdXmGJuV32kjj2xqvSByc | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | rLTbhSaXhSM | 2020-11-28T16:30:52Z | public |
4 | youtube#playlistItem | 2CzGUToIgqywXAr4wuPswj9MuFg | UExmMHN3VEZoVEk4cmtINHlJZm95VEFoZUVHaldJUnRQRy… | wP7BhXrJKR8 | 2020-11-28T16:31:14Z | public |
]