Skip to content

Tutorial 2: Nested JSON to tables

This tutorial parses nested JSON, flattens it into related table-like datasets, and converts output to pandas-backed types.

Setup

>>> import omnipy as om

Parse nested JSON

>>> nested = om.JsonListOfDictsDataset({'items': [
...     {'id': 'a', 'meta': {'x': 1, 'y': 2}, 'tags': [{'k': 't', 'v': 1}, {'k': 'u', 'v': 2}]},
...     {'id': 'b', 'meta': {'x': 3, 'y': 4}, 'tags': []},
... ]})
>>> nested

╭───┬────────────────┬──────────────────────┬────────┬──────────────────╮
│ # │ Data file name │         Type         │ Length │ Size (in memory) │
│   │                │                      │        │                  │
│ 0 │ items          │ JsonListOfDictsModel │      2 │           7.1 kB │
╰───┴────────────────┴──────────────────────┴────────┴──────────────────╯

>>> from omnipy.components.json.flows import flatten_nested_json
>>> flat = flatten_nested_json.run(nested)
>>> sorted(flat.dict().keys())

╭───┬────────────────┬───────────────────────────────┬────────┬──────────────────╮
│ # │ Data file name │             Type              │ Length │ Size (in memory) │
│   │                │                               │        │                  │
│ 0 │ items          │ JsonListOfDictsOfScalarsModel │      2 │           2.9 kB │
│ 1 │ items.meta     │ JsonListOfDictsOfScalarsModel │      2 │           3.0 kB │
│ 2 │ items.tags     │ JsonListOfDictsOfScalarsModel │      2 │           3.0 kB │
╰───┴────────────────┴───────────────────────────────┴────────┴──────────────────╯

Dataset naming preserves relational structure (items, items.meta, items.tags) so table relationships stay explicit.

Convert with .to(PandasModel) and .to(PandasDataset)

>>> flat_pd = flat.to(om.PandasDataset)
>>> one_table = om.JsonListOfDictsModel([{'id': 'a', 'x': 1}, {'id': 'b', 'x': 3}])
>>> one_table_pd = one_table.to(om.PandasModel)
>>> flat_pd
>>> one_table_pd

╭───┬────────────────┬─────────────┬────────┬──────────────────╮
│ # │ Data file name │    Type     │ Length │ Size (in memory) │
│   │                │             │        │                  │
│ 0 │ items          │ PandasModel │      2 │          10.1 kB │
│ 1 │ items.meta     │ PandasModel │      2 │          17.8 kB │
│ 2 │ items.tags     │ PandasModel │      2 │          17.9 kB │
╰───┴────────────────┴─────────────┴────────┴──────────────────╯

Current boundaries

Supported shapes include:

  • list of dicts
  • dict of dicts
  • dict of lists

Not supported in this conversion path:

  • mixed containers/scalars at level 2
  • single-level lists, dicts, or scalars

Flattening supports deeply nested structures where pandas.json_normalize often struggles, but it still has boundaries.

See also: Parse, don't validate

What you learned

  • How to flatten nested JSON into related table datasets with flatten_nested_json.run(...).
  • How to convert table-shaped results with .to(PandasDataset) and .to(PandasModel).

Common pitfalls

  • Expecting one universal flat table for all nested inputs. Many payloads produce multiple related tables (items, items.meta, items.tags, ...).

Next steps