Skip to content

Home

Omnipy logo

Omnipy is a type-driven Python library for:

  • data parsing, wrangling, visualization, and conversion
  • tool and web service interoperability, and
  • scalable dataflow orchestration

Conceptual overview of Omnipy

Why use Omnipy?

A Readable Contract Between You and Your AI Agent

AI agents are good at the time-consuming parts of working with data: reformatting, cleaning, and integrating. Omnipy gives that work a structure a human can inspect. The agent turns exploratory code into data models, conversions, and dataflows declared in Python; the data is validated at every step; and you review the declarations and the data at each junction. The declarations are the contract between you and your agent.

Dataflows, Not Workflows

Traditional workflows rely on command-line tools and intermediate files, adding complexity to data pipelines. Omnipy replaces this with dataflows that operate directly in memory or on standard formats like JSON or CSV. Built on Pydantic models, Omnipy enhances data parsing, conversion, and serialization for structured data processing.

"It's Static Typing!"… "It's Dynamic!"… "It's Omnipy!"

Omnipy blends Python’s dynamic typing with runtime type safety. Models behave like native Python structures while ensuring type guarantees without the rigidity of static typing. Defined in Python, Omnipy models can be as general or specific as needed.

Parse, Don’t Validate

Strict validation often breaks pipelines when data is messy. Inspired by "Parse, don't validate", Omnipy eagerly parses input into structured models that retain integrity throughout the pipeline. This approach aligns with the Robustness Principle: "be liberal in what you accept, and conservative in what you send!"

Self-Constraining Data Models

Omnipy models aren’t just one-time validators. A Model[list[int]]() behaves like a list but ensures its elements are always integers. Every modification parses data to enforce integrity, rolling back invalid operations automatically.

Omnify Your Data Pipelines

Omnipy invites you to "omnify" pipelines — break them into reusable, universal components. By defining dataflows and tasks with structured input and output models, Omnipy simplifies reuse and promotes good coding practices, improving maintainability as projects grow. Small, typed, reviewable components are also what an agent should produce.

Catalog of Components for Interoperability

Omnipy includes components for tasks like asynchronous API requests with rate limiting and retries, posting JSON to web services, building paged URLs, keying Datasets by the primary keys of their records, parsing JSON or tabular data, and flattening nested data into relational tables. Integration with REST APIs and data wrangling/analysis tools like Pandas simplifies interoperability across diverse systems. Expect the catalog to grow as the community expands!

Built to Scale

Omnipy’s hierarchical Dataset structure simplifies batch processing of directory-based data, including parsing, serialization, and metadata handling. With built-in Prefect support, Omnipy scales seamlessly from local experiments to distributed deployment, meeting the demands of projects large and small.

Installing Omnipy

  1. Make sure that your Python version is between 3.11.4 and 3.15, e.g.:

    $ python --version
    Python 3.15
    
  2. Create and activate a virtual environment for your project, e.g.:

    $ python -m venv myproject
    $ source myproject/bin/activate
    

    Tip

    • If you need help with setting up a virtual environment, check out the relevant section in the FastAPI documentation. (Please note that Omnipy does not depend on FastAPI, it is just that their documentation is excellent!)
    • If you are using Omnipy in a Jupyter notebook, you can most likely skip this step.
  3. Install Omnipy using:

    $ pip install omnipy
    

  4. If you work with a coding agent, install the Omnipy skills into your project, running in the project directory:

    $ python -m omnipy.skills install
    

    This links the skills shipped with Omnipy into .agents/skills/, where most agents read them, so that they follow Omnipy through upgrades. Add --claude for Claude Code, which reads .claude/skills/, and see python -m omnipy.skills --help for listing the skills and other options.

Omnipy works with both Pydantic 1 and 2. With Pydantic 2 installed, it runs through Pydantic's v1 compatibility layer; Pydantic 2 models are not supported yet.

Getting started

Parse data into a model

To define an Omnipy data model, concretize the generic Model class by specifying a data type in brackets, e.g. Model[list[int]].

The following creates a data model of a list of integers, and parses some data into that model:

>>> import omnipy as om
>>> data = (123, '234', 345.0)  # Note that the input data is a tuple of mixed types
>>> data_as_list_of_ints = om.Model[list[int]](data)
>>> data_as_list_of_ints  # The data is now parsed into a list of integers
╭──────────────────╮
│ Model[list[int]] │
│                  │
│ [123, 234, 345]  │
╰──────────────────╯

Style the displays

Omnipy's display settings configure the terminal, notebook, and browser separately. The displays below use the dark MEASURED_DARK_T16 theme with a solid background. Use ui.terminal or ui.jupyter in place of ui.browser to style your own console or notebook:

>>> ui_config = om.runtime.config.data.ui.browser
>>> ui_config.color.style = om.DarkHighContrastColorStyles.MEASURED_DARK_T16
>>> ui_config.color.solid_background = True

The list of integers from above now displays in the new theme:

>>> data_as_list_of_ints
╭──────────────────╮
│ Model[list[int]] │
│                  │
│ [123, 234, 345]  │
╰──────────────────╯

Build the URLs of a web API

Let's work with river samples from the water-sample harmonization example. A water quality API would return such samples in pages. Here the two pages for this tutorial are provided JSON files in the Omnipy repository, so the walkthrough runs as it stands from any Python console.

An HttpUrlModel holds a URL as structured fields, so the path is edited in place instead of by string formatting:

>>> data_base_url = om.HttpUrlModel('https://raw.githubusercontent.com/fairtracks/omnipy/main')
>>> data_base_url.path // 'docs' // 'examples' // 'river_samples'
>>> page_0_url = om.HttpUrlModel(data_base_url)
>>> page_0_url.path // 'page_0.json'
>>> page_0_url
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                            HttpUrlModel                                            │
│                                                                                                    │
│ 'https://raw.githubusercontent.com/fairtracks/omnipy/main/docs/examples/river_samples/page_0.json' │
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯

A second page differs only in its file name. An HttpUrlDataset collects the URLs under names that will also name the responses:

>>> page_1_url = om.HttpUrlModel(data_base_url)
>>> page_1_url.path // 'page_1.json'
>>> page_urls = om.HttpUrlDataset(page_0=page_0_url, page_1=page_1_url)
>>> page_urls
╭───┬────────────────┬──────────────┬────────┬──────────────────╮
│ # │ Data file name │     Type     │ Length │ Size (in memory) │
│   │                │              │        │                  │
│ 0 │ page_0         │ HttpUrlModel │      - │           3.2 kB │
│ 1 │ page_1         │ HttpUrlModel │      - │           3.1 kB │
╰───┴────────────────┴──────────────┴────────┴──────────────────╯

Such Datasets are also built by the catalog: GeneratePagedUrlsModel from a page or item count for APIs with many pages, and the get_github_repo_urls task from a directory listing on GitHub.

Load the pages

A Dataset's load() fetches all its URLs, keeping the responses under the URL names. A JsonDataset parses the responses as JSON, including when the server sends them as plain text, as GitHub does here:

>>> pages = om.JsonDataset.load(page_urls)
>>> pages
╭───┬────────────────┬───────────┬────────┬──────────────────╮
│ # │ Data file name │   Type    │ Length │ Size (in memory) │
│   │                │           │        │                  │
│ 0 │ page_0         │ JsonModel │      2 │           6.5 kB │
│ 1 │ page_1         │ JsonModel │      1 │           3.4 kB │
╰───┴────────────────┴───────────┴────────┴──────────────────╯

A Dataset's peek() shows the contents of its items side by side:

>>> pages.peek()
╭─────────────────────────────────────────────────┬───────────────────────────────────────────────────────╮
│                    0. page_0                    │                       1. page_1                       │
│                                                 │                                                       │
│ [                                               │ [                                                     │
│   {                                             │   {                                                   │
│     "river_sample_id": "river-sample-1",        │     "river_sample_id": "river-sample-3",              │
│     "catchment": "glomma-upper",                │     "catchment": "glomma-upper",                      │
│     "sampled_at": "2026-05-03",                 │     "sampled_at": "2026-05-17",                       │
│     "station": "River Mouth",                   │     "station": "Upstream Bend",                       │
│     "sample_alias": "River-R1",                 │     "sample_alias": "River-R3",                       │
│     "measurements": [                           │     "measurements": [                                 │
│       { "name": "Nitrate", "value_mg_l": 1.2 }, │       { "name": "Temperature", "value_celsius": 9.8 } │
│       {                                         │     ]                                                 │
│         "name": "Phosphorus",                   │   }                                                   │
│         "value_ug_l": 120.0                     │ ]                                                     │
│       }                                         │                                                       │
│     ]                                           │                                                       │
│   },                                            │                                                       │
│   {                                             │                                                       │
│     "river_sample_id": "river-sample-2",        │                                                       │
│     "catchment": "glomma-upper",                │                                                       │
│     "sampled_at": "2026-05-10",                 │                                                       │
│     "station": "River Mouth",                   │                                                       │
│     "sample_alias": "River-R2",                 │                                                       │
│     "measurements": [                           │                                                       │
│       { "name": "nitrate", "value_mg_l": 1.4 }  │                                                       │
│     ]                                           │                                                       │
│   }                                             │                                                       │
│ ]                                               │                                                       │
╰─────────────────────────────────────────────────┴───────────────────────────────────────────────────────╯

These are synchronous console calls. In a notebook or other running event loop, use await with .load().

Give one measurement a type

Models mimic their wrapped values, so we can index the loaded JSON directly. The second measurement in the first sample is phosphorus, reported in µg/L:

>>> raw_measurement = pages['page_0'][0]['measurements'][1]
>>> raw_measurement
╭───────────────────────────────────────────────╮
│                 _JsonAnyDictM                 │
│                                               │
│ { "name": "Phosphorus", "value_ug_l": 120.0 } │
╰───────────────────────────────────────────────╯

JSON types alone do not describe what a measurement means. Start with one Pydantic record for a concentration in µg/L, wrapped in an Omnipy Model to keep its field types active during edits. mimics exposes the record's fields to editors and type checkers; the Model supplies field access at runtime.

╭────────────────────────────────────────────────────────────────────────────────╮
│ import omnipy.util.pydantic as pyd                                             │
│                                                                                │
│ class RiverUgConcMeasurementRecord(pyd.BaseModel):                             │
│     name: str                                                                  │
│     value_ug_l: float                                                          │
│                                                                                │
│ @om.mimics(RiverUgConcMeasurementRecord)                                       │
│ class RiverUgConcMeasurementModel(om.Model[RiverUgConcMeasurementRecord]): ... │
╰────────────────────────────────────────────────────────────────────────────────╯

Constructing the Model parses the JSON measurement into the record:

>>> measurement = RiverUgConcMeasurementModel(raw_measurement)
>>> measurement
╭─────────────────────────────────────────────╮
│         RiverUgConcMeasurementModel         │
│                                             │
│ {'name': 'Phosphorus', 'value_ug_l': 120.0} │
╰─────────────────────────────────────────────╯

Give the samples a type

The full sample includes temperature in °C and concentrations in mg/L or µg/L. A union of records tells these shapes apart by their fields. The sample record combines them with a station and a DateModel. The catalog's MergeListsOfRecordsIntoDatasetByPrimaryKeyModel merges the pages and keys the samples by their PrimaryKey, rejecting duplicates. Its contents are a Dataset, so we name the specialization RiverSamplesDataset.

╭──────────────────────────────────────────────────────────────────────────────────────────────╮
│ from typing import Literal                                                                   │
│                                                                                              │
│ class RiverTempMeasurementRecord(pyd.BaseModel):                                             │
│     name: str                                                                                │
│     value_celsius: float                                                                     │
│                                                                                              │
│ class RiverMgConcMeasurementRecord(pyd.BaseModel):                                           │
│     name: str                                                                                │
│     value_mg_l: float                                                                        │
│                                                                                              │
│ RiverMeasurementRecordType = (                                                               │
│     RiverTempMeasurementRecord | RiverMgConcMeasurementRecord | RiverUgConcMeasurementRecord │
│ )                                                                                            │
│                                                                                              │
│ class RiverSampleRecord(pyd.BaseModel):                                                      │
│     river_sample_id: str                                                                     │
│     catchment: str                                                                           │
│     sampled_at: om.DateModel                                                                 │
│     station: str                                                                             │
│     sample_alias: str                                                                        │
│     measurements: list[RiverMeasurementRecordType]                                           │
│                                                                                              │
│ @om.mimics(om.Dataset[om.Model[RiverSampleRecord]])                                          │
│ class RiverSamplesDataset(                                                                   │
│     om.MergeListsOfRecordsIntoDatasetByPrimaryKeyModel[                                      │
│         om.PrimaryKey[Literal['river_sample_id']], RiverSampleRecord                         │
│     ]                                                                                        │
│ ): ...                                                                                       │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯

Constructing the Dataset from the loaded pages parses them into typed samples:

>>> samples = RiverSamplesDataset(pages)
>>> samples.peek()
╭──────────────────────────────────────┬────────────────────────────────────┬────────────────────────────────────╮
│          0. river-sample-1           │         1. river-sample-2          │         2. river-sample-3          │
│                                      │                                    │                                    │
│ {                                    │ {                                  │ {                                  │
│   'river_sample_id': 'river-sample-… │   'river_sample_id': 'river-sampl… │   'river_sample_id': 'river-sampl… │
│   'catchment': 'glomma-upper',       │   'catchment': 'glomma-upper',     │   'catchment': 'glomma-upper',     │
│   'sampled_at': '2026-05-03',        │   'sampled_at': '2026-05-10',      │   'sampled_at': '2026-05-17',      │
│   'station': 'River Mouth',          │   'station': 'River Mouth',        │   'station': 'Upstream Bend',      │
│   'sample_alias': 'River-R1',        │   'sample_alias': 'River-R2',      │   'sample_alias': 'River-R3',      │
│   'measurements': [                  │   'measurements': [                │   'measurements': [                │
│     {                                │     {                              │     {                              │
│       'name': 'Nitrate',             │       'name': 'nitrate',           │       'name': 'Temperature',       │
│       'value_mg_l': 1.2              │       'value_mg_l': 1.4            │       'value_celsius': 9.8         │
│     },                               │     }                              │     }                              │
│     {                                │   ]                                │   ]                                │
│       'name': 'Phosphorus',          │ }                                  │ }                                  │
│       'value_ug_l': 120.0            │                                    │                                    │
│     }                                │                                    │                                    │
│   ]                                  │                                    │                                    │
│ }                                    │                                    │                                    │
╰──────────────────────────────────────┴────────────────────────────────────┴────────────────────────────────────╯

json() displays the data with JSON syntax highlighting. The display methods also take a color style of their own:

>>> samples.json(style='gotham-t16')
╭───────────────────────────────────────────────────────────────────────────────────────────────╮
│                                           JsonModel                                           │
│                                                                                               │
│ {                                                                                             │
│   "river-sample-1": {                                                                         │
│     "river_sample_id": "river-sample-1",                                                      │
│     "catchment": "glomma-upper",                                                              │
│     "sampled_at": "2026-05-03",                                                               │
│     "station": "River Mouth",                                                                 │
│     "sample_alias": "River-R1",                                                               │
│     "measurements": [                                                                         │
│       { "name": "Nitrate", "value_mg_l": 1.2 }, { "name": "Phosphorus", "value_ug_l": 120.0 } │
│     ]                                                                                         │
│   },                                                                                          │
│   "river-sample-2": {                                                                         │
│     "river_sample_id": "river-sample-2",                                                      │
│     "catchment": "glomma-upper",                                                              │
│     "sampled_at": "2026-05-10",                                                               │
│     "station": "River Mouth",                                                                 │
│     "sample_alias": "River-R2",                                                               │
│     "measurements": [ { "name": "nitrate", "value_mg_l": 1.4 } ]                              │
│   },                                                                                          │
│   "river-sample-3": {                                                                         │
│     "river_sample_id": "river-sample-3",                                                      │
│     "catchment": "glomma-upper",                                                              │
│     "sampled_at": "2026-05-17",                                                               │
│     "station": "Upstream Bend",                                                               │
│     "sample_alias": "River-R3",                                                               │
│     "measurements": [ { "name": "Temperature", "value_celsius": 9.8 } ]                       │
│   }                                                                                           │
│ }                                                                                             │
╰───────────────────────────────────────────────────────────────────────────────────────────────╯

Work with the Dataset

A Dataset supports names and integer positions. The third sample can be accessed by its name, and its station read through the Model:

>>> samples['river-sample-3'].station
'Upstream Bend'

Integer indexing starts at zero, so samples[2] selects the same sample:

>>> samples[2].station
'Upstream Bend'

A list of keys selects several items and returns a Dataset:

>>> samples[['river-sample-1', 'river-sample-2']].peek()
╭─────────────────────────────────────────────────┬────────────────────────────────────────────────────────────╮
│                0. river-sample-1                │                     1. river-sample-2                      │
│                                                 │                                                            │
│ {                                               │ {                                                          │
│   'river_sample_id': 'river-sample-1',          │   'river_sample_id': 'river-sample-2',                     │
│   'catchment': 'glomma-upper',                  │   'catchment': 'glomma-upper',                             │
│   'sampled_at': '2026-05-03',                   │   'sampled_at': '2026-05-10',                              │
│   'station': 'River Mouth',                     │   'station': 'River Mouth',                                │
│   'sample_alias': 'River-R1',                   │   'sample_alias': 'River-R2',                              │
│   'measurements': [                             │   'measurements': [{'name': 'nitrate', 'value_mg_l': 1.4}] │
│     {'name': 'Nitrate', 'value_mg_l': 1.2},     │ }                                                          │
│     {'name': 'Phosphorus', 'value_ug_l': 120.0} │                                                            │
│   ]                                             │                                                            │
│ }                                               │                                                            │
╰─────────────────────────────────────────────────┴────────────────────────────────────────────────────────────╯

A slice selects by position. Here it returns the same two items:

>>> samples[:2]
╭───┬────────────────┬──────────────────────────┬────────┬──────────────────╮
│ # │ Data file name │           Type           │ Length │ Size (in memory) │
│   │                │                          │        │                  │
│ 0 │ river-sample-1 │ Model[RiverSampleRecord] │      - │           5.4 kB │
│ 1 │ river-sample-2 │ Model[RiverSampleRecord] │      - │           4.3 kB │
╰───┴────────────────┴──────────────────────────┴────────┴──────────────────╯

Edit safely, without unpacking

Omnipy Models are self-constraining: assigning through the mimicked interface parses the new value against its field type. An invalid concentration raises a validation error:

>>> phosphorus = samples['river-sample-1'].measurements[1]
>>> phosphorus
╭─────────────────────────────────────────────╮
│     Model[RiverUgConcMeasurementRecord]     │
│                                             │
│ {'name': 'Phosphorus', 'value_ug_l': 120.0} │
╰─────────────────────────────────────────────╯
>>> phosphorus.value_ug_l = 'not a number'

ValidationError

2 validation errors for Model[RiverUgConcMeasurementRecord]
__root__ -> value_ug_l
  value is not a valid float (type=type_error.float)
__root__
   (type=assertion_error)

In interactive mode, the failed edit is rolled back to the last valid snapshot. Displaying the measurement again shows the original 120.0 µg/L:

>>> phosphorus
╭─────────────────────────────────────────────╮
│     Model[RiverUgConcMeasurementRecord]     │
│                                             │
│ {'name': 'Phosphorus', 'value_ug_l': 120.0} │
╰─────────────────────────────────────────────╯

A numeric string can be parsed as a float. Let's correct the reading to 150.0 µg/L. The measurement is part of the sample, so the correction is visible there too:

>>> phosphorus.value_ug_l = '150.0'
>>> samples['river-sample-1'].measurements[1]
╭─────────────────────────────────────────────╮
│     Model[RiverUgConcMeasurementRecord]     │
│                                             │
│ {'name': 'Phosphorus', 'value_ug_l': 150.0} │
╰─────────────────────────────────────────────╯

Convert to a common representation

Other sources use different field names and units. A ConverterModel makes a conversion reusable and validates its output. First, a measurement mapper uses the record type to choose the conversion, including µg/L to mg/L:

╭─────────────────────────────────────────────────────────────────────────────────────────╮
│ class NormalizedMeasurementRecord(pyd.BaseModel):                                       │
│     type: Literal['nitrate', 'phosphorus', 'temperature']                               │
│     value: float                                                                        │
│     unit: Literal['mg/L', 'deg_C']                                                      │
│                                                                                         │
│ @om.mimics(NormalizedMeasurementRecord)                                                 │
│ class NormalizedRiverMeasurementMapper(                                                 │
│     om.ConverterModel[RiverMeasurementRecordType, NormalizedMeasurementRecord]          │
│ ):                                                                                      │
│     @classmethod                                                                        │
│     def _convert(cls, data: RiverMeasurementRecordType) -> NormalizedMeasurementRecord: │
│         match data:                                                                     │
│             case RiverTempMeasurementRecord():                                          │
│                 return NormalizedMeasurementRecord(                                     │
│                     type=data.name.lower(), value=data.value_celsius, unit='deg_C'      │
│                 )                                                                       │
│             case RiverMgConcMeasurementRecord():                                        │
│                 return NormalizedMeasurementRecord(                                     │
│                     type=data.name.lower(), value=data.value_mg_l, unit='mg/L'          │
│                 )                                                                       │
│             case RiverUgConcMeasurementRecord():                                        │
│                 return NormalizedMeasurementRecord(                                     │
│                     type=data.name.lower(), value=data.value_ug_l / 1000, unit='mg/L'   │
│                 )                                                                       │
╰─────────────────────────────────────────────────────────────────────────────────────────╯

Constructing the mapper converts the corrected measurement to 0.15 mg/L:

>>> normalized = NormalizedRiverMeasurementMapper(phosphorus)
>>> normalized
╭───────────────────────────────────────────────────────╮
│           NormalizedRiverMeasurementMapper            │
│                                                       │
│ {'type': 'phosphorus', 'value': 0.15, 'unit': 'mg/L'} │
╰───────────────────────────────────────────────────────╯

Next, a sample mapper renames the sample fields and reuses the measurement mapper's convert_all() for its measurements. A Dataset of that mapper converts all the samples:

╭───────────────────────────────────────────────────────────────────────────────────╮
│ class NormalizedSampleRecord(pyd.BaseModel):                                      │
│     sample_key: str                                                               │
│     monitoring_date: om.DateModel                                                 │
│     location_name: str                                                            │
│     measurements: list[NormalizedMeasurementRecord]                               │
│                                                                                   │
│ @om.mimics(NormalizedSampleRecord)                                                │
│ class NormalizedRiverSampleMapper(                                                │
│     om.ConverterModel[RiverSampleRecord, NormalizedSampleRecord]                  │
│ ):                                                                                │
│     @classmethod                                                                  │
│     def _convert(cls, data: RiverSampleRecord) -> NormalizedSampleRecord:         │
│         return NormalizedSampleRecord(                                            │
│             sample_key=data.sample_alias.lower(),                                 │
│             monitoring_date=data.sampled_at,                                      │
│             location_name=data.station,                                           │
│             measurements=list(                                                    │
│                 NormalizedRiverMeasurementMapper.convert_all(data.measurements)   │
│             ),                                                                    │
│         )                                                                         │
│                                                                                   │
│ class NormalizedRiverSamplesDataset(om.Dataset[NormalizedRiverSampleMapper]): ... │
╰───────────────────────────────────────────────────────────────────────────────────╯

The Dataset constructor applies the mapper to every sample, keeping the sample ids as keys. peek() displays the converted samples side by side:

>>> normalized_samples = NormalizedRiverSamplesDataset(samples)
>>> normalized_samples.json()
╭──────────────────────────────────────────────────────────────────────────────────╮
│                                    JsonModel                                     │
│                                                                                  │
│ {                                                                                │
│   "river-sample-1": {                                                            │
│     "sample_key": "river-r1",                                                    │
│     "monitoring_date": "2026-05-03",                                             │
│     "location_name": "River Mouth",                                              │
│     "measurements": [                                                            │
│       { "type": "nitrate"   , "value": 1.2 , "unit": "mg/L" },                   │
│       { "type": "phosphorus", "value": 0.15, "unit": "mg/L" }                    │
│     ]                                                                            │
│   },                                                                             │
│   "river-sample-2": {                                                            │
│     "sample_key": "river-r2",                                                    │
│     "monitoring_date": "2026-05-10",                                             │
│     "location_name": "River Mouth",                                              │
│     "measurements": [ { "type": "nitrate", "value": 1.4, "unit": "mg/L" } ]      │
│   },                                                                             │
│   "river-sample-3": {                                                            │
│     "sample_key": "river-r3",                                                    │
│     "monitoring_date": "2026-05-17",                                             │
│     "location_name": "Upstream Bend",                                            │
│     "measurements": [ { "type": "temperature", "value": 9.8, "unit": "deg_C" } ] │
│   }                                                                              │
│ }                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────╯

For a one-line conversion and display, to() is convenient too:

>>> samples[0].to(NormalizedRiverSampleMapper).json()
╭──────────────────────────────────────────────────────────────╮
│                          JsonModel                           │
│                                                              │
│ {                                                            │
│   "sample_key": "river-r1",                                  │
│   "monitoring_date": "2026-05-03",                           │
│   "location_name": "River Mouth",                            │
│   "measurements": [                                          │
│     { "type": "nitrate"   , "value": 1.2 , "unit": "mg/L" }, │
│     { "type": "phosphorus", "value": 0.15, "unit": "mg/L" }  │
│   ]                                                          │
│ }                                                            │
╰──────────────────────────────────────────────────────────────╯

Explore the hierarchy as nested Datasets

StructuredNestedDataset represents mappings recursively as Datasets and recognizes the structure of the leaves: scalar fields become JsonScalarModel items, and each measurement list, a list of records with the same fields, becomes a column-wise table Model. The conversion preserves the sample ids and field names, while making the samples nested Datasets with their fields as items:

>>> nested_samples = om.StructuredNestedDataset(normalized_samples)
>>> nested_samples
╭───┬────────────────┬─────────────────────────┬────────┬──────────────────╮
│ # │ Data file name │          Type           │ Length │ Size (in memory) │
│   │                │                         │        │                  │
│ 0 │ river-sample-1 │ StructuredNestedDataset │      4 │           5.0 kB │
│ 1 │ river-sample-2 │ StructuredNestedDataset │      4 │           4.8 kB │
│ 2 │ river-sample-3 │ StructuredNestedDataset │      4 │           4.8 kB │
╰───┴────────────────┴─────────────────────────┴────────┴──────────────────╯

Calling peek() on a StructuredNestedDataset shows a hierarchical view of the structure, with tables nested within tables:

>>> nested_samples[0].peek()
╭────────────┬─────────────────┬───────────────┬─────────────────────────────────────────╮
│     0.     │       1.        │      2.       │             3. measurements             │
│ sample_key │ monitoring_date │ location_name │                                         │
│            │                 │               │ ╭───┬────────────┬──────────┬─────────╮ │
│ "river-r1" │ "2026-05-03"    │ "River Mouth" │ │ # │  0. type   │ 1. value │ 2. unit │ │
│            │                 │               │ │   │            │          │         │ │
│            │                 │               │ │ 0 │ nitrate    │ 1.2      │ mg/L    │ │
│            │                 │               │ │ 1 │ phosphorus │ 0.15     │ mg/L    │ │
│            │                 │               │ ╰───┴────────────┴──────────┴─────────╯ │
╰────────────┴─────────────────┴───────────────┴─────────────────────────────────────────╯

Turn the measurements into tables

The measurement tables have type, value, and unit columns. do() applies a placeholder expression to each item, here selecting the measurements from each sample, keeping the sample ids as keys. A PandasDataset then converts each measurement table to a pandas DataFrame:

>>> measurements = nested_samples.do(om.x['measurements'])
>>> tables = om.PandasDataset(measurements)
>>> tables.peek(width=132)
╭─────────────────────────────────────────┬──────────────────────────────────────┬──────────────────────────────────────────╮
│            0. river-sample-1            │          1. river-sample-2           │            2. river-sample-3             │
│                                         │                                      │                                          │
│ ╭───┬────────────┬──────────┬─────────╮ │ ╭───┬─────────┬──────────┬─────────╮ │ ╭───┬─────────────┬──────────┬─────────╮ │
│ │ # │  0. type   │ 1. value │ 2. unit │ │ │ # │ 0. type │ 1. value │ 2. unit │ │ │ # │   0. type   │ 1. value │ 2. unit │ │
│ │   │            │          │         │ │ │   │         │          │         │ │ │   │             │          │         │ │
│ │ 0 │ nitrate    │ 1.2      │ mg/L    │ │ │ 0 │ nitrate │ 1.4      │ mg/L    │ │ │ 0 │ temperature │ 9.8      │ deg_C   │ │
│ │ 1 │ phosphorus │ 0.15     │ mg/L    │ │ ╰───┴─────────┴──────────┴─────────╯ │ ╰───┴─────────────┴──────────┴─────────╯ │
│ ╰───┴────────────┴──────────┴─────────╯ │                                      │                                          │
╰─────────────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────────╯

Compose tasks and flows

The steps so far form a dataflow: fetch the pages, key the samples, normalize them, expose the hierarchy, select measurements, and tabulate. A TaskTemplate wraps a function; a LinearFlowTemplate declares its stages in order. Model and Dataset constructors can be stages directly. resolve() handles the loader's result whether it is already available or still awaiting completion:

╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ @om.TaskTemplate()                                                                          │
│ async def fetch_sample_pages(page_urls: om.HttpUrlDataset) -> om.JsonDataset:               │
│     return await om.resolve(om.JsonDataset.load(page_urls))                                 │
│                                                                                             │
│ @om.TaskTemplate()                                                                          │
│ def select_measurements(samples: om.StructuredNestedDataset) -> om.StructuredNestedDataset: │
│     return samples.do(om.x['measurements'])                                                 │
│                                                                                             │
│ @om.LinearFlowTemplate(                                                                     │
│     fetch_sample_pages,                                                                     │
│     RiverSamplesDataset,                                                                    │
│     NormalizedRiverSamplesDataset,                                                          │
│     om.StructuredNestedDataset,                                                             │
│     select_measurements,                                                                    │
│     om.PandasDataset,                                                                       │
│ )                                                                                           │
│ async def harmonize_river_samples(page_urls: om.HttpUrlDataset) -> om.PandasDataset: ...    │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯

The flow reloads the original pages, so its phosphorus reading starts at 0.12 mg/L. The manual correction above belongs to the earlier in-memory Dataset:

>>> tables = harmonize_river_samples.run(page_urls)
>>> tables.peek(width=132)
╭─────────────────────────────────────────┬──────────────────────────────────────┬──────────────────────────────────────────╮
│            0. river-sample-1            │          1. river-sample-2           │            2. river-sample-3             │
│                                         │                                      │                                          │
│ ╭───┬────────────┬──────────┬─────────╮ │ ╭───┬─────────┬──────────┬─────────╮ │ ╭───┬─────────────┬──────────┬─────────╮ │
│ │ # │  0. type   │ 1. value │ 2. unit │ │ │ # │ 0. type │ 1. value │ 2. unit │ │ │ # │   0. type   │ 1. value │ 2. unit │ │
│ │   │            │          │         │ │ │   │         │          │         │ │ │   │             │          │         │ │
│ │ 0 │ nitrate    │ 1.2      │ mg/L    │ │ │ 0 │ nitrate │ 1.4      │ mg/L    │ │ │ 0 │ temperature │ 9.8      │ deg_C   │ │
│ │ 1 │ phosphorus │ 0.12     │ mg/L    │ │ ╰───┴─────────┴──────────┴─────────╯ │ ╰───┴─────────────┴──────────┴─────────╯ │
│ ╰───┴────────────┴──────────┴─────────╯ │                                      │                                          │
╰─────────────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────────╯

The flow declaration is what a reviewer reads, and what an AI agent working on the pipeline produces and is held to: the stages, their order, and the types passing between them, with every Model and Dataset validating its data at its stage. The same flow runs on Prefect by setting om.runtime.config.engine.choice = 'prefect'; see Tutorial 8: Prefect orchestration.

Where next