Home

Omnipy is a type-driven Python library for:
- data parsing, wrangling, visualization, and conversion
- tool and web service interoperability, and
- scalable dataflow orchestration

Why use Omnipy?
A Readable Contract Between You and Your AI Agent
AI agents are good at the time-consuming parts of working with data: reformatting, cleaning, and integrating. Omnipy gives that work a structure a human can inspect. The agent turns exploratory code into data models, conversions, and dataflows declared in Python; the data is validated at every step; and you review the declarations and the data at each junction. The declarations are the contract between you and your agent.
Dataflows, Not Workflows
Traditional workflows rely on command-line tools and intermediate files, adding complexity to data pipelines. Omnipy replaces this with dataflows that operate directly in memory or on standard formats like JSON or CSV. Built on Pydantic models, Omnipy enhances data parsing, conversion, and serialization for structured data processing.
"It's Static Typing!"… "It's Dynamic!"… "It's Omnipy!"
Omnipy blends Python’s dynamic typing with runtime type safety. Models behave like native Python structures while ensuring type guarantees without the rigidity of static typing. Defined in Python, Omnipy models can be as general or specific as needed.
Parse, Don’t Validate
Strict validation often breaks pipelines when data is messy. Inspired by "Parse, don't validate", Omnipy eagerly parses input into structured models that retain integrity throughout the pipeline. This approach aligns with the Robustness Principle: "be liberal in what you accept, and conservative in what you send!"
Self-Constraining Data Models
Omnipy models aren’t just one-time validators. A Model[list[int]]() behaves like a list but
ensures its elements are always integers. Every modification parses data to enforce integrity,
rolling back invalid operations automatically.
Omnify Your Data Pipelines
Omnipy invites you to "omnify" pipelines — break them into reusable, universal components. By defining dataflows and tasks with structured input and output models, Omnipy simplifies reuse and promotes good coding practices, improving maintainability as projects grow. Small, typed, reviewable components are also what an agent should produce.
Catalog of Components for Interoperability
Omnipy includes components for tasks like asynchronous API requests with rate limiting and retries, posting JSON to web services, building paged URLs, keying Datasets by the primary keys of their records, parsing JSON or tabular data, and flattening nested data into relational tables. Integration with REST APIs and data wrangling/analysis tools like Pandas simplifies interoperability across diverse systems. Expect the catalog to grow as the community expands!
Built to Scale
Omnipy’s hierarchical Dataset structure simplifies batch processing of directory-based data,
including parsing, serialization, and metadata handling. With
built-in Prefect support, Omnipy scales seamlessly from local experiments
to distributed deployment, meeting the demands of projects large and small.
Installing Omnipy
-
Make sure that your Python version is between
3.11.4and3.15, e.g.: -
Create and activate a virtual environment for your project, e.g.:
Tip
- If you need help with setting up a virtual environment, check out the relevant section in the FastAPI documentation. (Please note that Omnipy does not depend on FastAPI, it is just that their documentation is excellent!)
- If you are using Omnipy in a Jupyter notebook, you can most likely skip this step.
-
Install Omnipy using:
-
If you work with a coding agent, install the Omnipy skills into your project, running in the project directory:
This links the skills shipped with Omnipy into
.agents/skills/, where most agents read them, so that they follow Omnipy through upgrades. Add--claudefor Claude Code, which reads.claude/skills/, and seepython -m omnipy.skills --helpfor listing the skills and other options.
Omnipy works with both Pydantic 1 and 2. With Pydantic 2 installed, it runs through Pydantic's v1 compatibility layer; Pydantic 2 models are not supported yet.
Getting started
Parse data into a model
To define an Omnipy data model, concretize the generic
Model class by specifying a
data type in brackets, e.g. Model[list[int]].
The following creates a data model of a list of integers, and parses some data into that model:
>>> import omnipy as om
>>> data = (123, '234', 345.0) # Note that the input data is a tuple of mixed types
>>> data_as_list_of_ints = om.Model[list[int]](data)
>>> data_as_list_of_ints # The data is now parsed into a list of integers
╭──────────────────╮
│ Model[list[int]] │
│ │
│ [123, 234, 345] │
╰──────────────────╯
Style the displays
Omnipy's display settings
configure the terminal, notebook, and browser separately. The displays below use the
dark MEASURED_DARK_T16
theme with a solid background. Use ui.terminal or ui.jupyter in place
of ui.browser to style your own console or notebook:
>>> ui_config = om.runtime.config.data.ui.browser
>>> ui_config.color.style = om.DarkHighContrastColorStyles.MEASURED_DARK_T16
>>> ui_config.color.solid_background = True
The list of integers from above now displays in the new theme:
╭──────────────────╮
│ Model[list[int]] │
│ │
│ [123, 234, 345] │
╰──────────────────╯
Build the URLs of a web API
Let's work with river samples from the water-sample harmonization example. A water quality API would return such samples in pages. Here the two pages for this tutorial are provided JSON files in the Omnipy repository, so the walkthrough runs as it stands from any Python console.
An HttpUrlModel
holds a URL as structured fields, so the path is edited in place instead of by
string formatting:
>>> data_base_url = om.HttpUrlModel('https://raw.githubusercontent.com/fairtracks/omnipy/main')
>>> data_base_url.path // 'docs' // 'examples' // 'river_samples'
>>> page_0_url = om.HttpUrlModel(data_base_url)
>>> page_0_url.path // 'page_0.json'
>>> page_0_url
╭────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ HttpUrlModel │
│ │
│ 'https://raw.githubusercontent.com/fairtracks/omnipy/main/docs/examples/river_samples/page_0.json' │
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯
A second page differs only in its file name. An
HttpUrlDataset
collects the URLs under names that will also name the responses:
>>> page_1_url = om.HttpUrlModel(data_base_url)
>>> page_1_url.path // 'page_1.json'
>>> page_urls = om.HttpUrlDataset(page_0=page_0_url, page_1=page_1_url)
>>> page_urls
╭───┬────────────────┬──────────────┬────────┬──────────────────╮
│ # │ Data file name │ Type │ Length │ Size (in memory) │
│ │ │ │ │ │
│ 0 │ page_0 │ HttpUrlModel │ - │ 3.2 kB │
│ 1 │ page_1 │ HttpUrlModel │ - │ 3.1 kB │
╰───┴────────────────┴──────────────┴────────┴──────────────────╯
Such Datasets are also built by the catalog:
GeneratePagedUrlsModel
from a page or item count for APIs with many pages, and the
get_github_repo_urls
task from a directory listing on GitHub.
Load the pages
A Dataset's load() fetches
all its URLs, keeping the responses under the URL names. A
JsonDataset parses
the responses as JSON, including when the server sends them as plain text, as GitHub does here:
╭───┬────────────────┬───────────┬────────┬──────────────────╮
│ # │ Data file name │ Type │ Length │ Size (in memory) │
│ │ │ │ │ │
│ 0 │ page_0 │ JsonModel │ 2 │ 6.5 kB │
│ 1 │ page_1 │ JsonModel │ 1 │ 3.4 kB │
╰───┴────────────────┴───────────┴────────┴──────────────────╯
A Dataset's peek()
shows the contents of its items side by side:
╭─────────────────────────────────────────────────┬───────────────────────────────────────────────────────╮
│ 0. page_0 │ 1. page_1 │
│ │ │
│ [ │ [ │
│ { │ { │
│ "river_sample_id": "river-sample-1", │ "river_sample_id": "river-sample-3", │
│ "catchment": "glomma-upper", │ "catchment": "glomma-upper", │
│ "sampled_at": "2026-05-03", │ "sampled_at": "2026-05-17", │
│ "station": "River Mouth", │ "station": "Upstream Bend", │
│ "sample_alias": "River-R1", │ "sample_alias": "River-R3", │
│ "measurements": [ │ "measurements": [ │
│ { "name": "Nitrate", "value_mg_l": 1.2 }, │ { "name": "Temperature", "value_celsius": 9.8 } │
│ { │ ] │
│ "name": "Phosphorus", │ } │
│ "value_ug_l": 120.0 │ ] │
│ } │ │
│ ] │ │
│ }, │ │
│ { │ │
│ "river_sample_id": "river-sample-2", │ │
│ "catchment": "glomma-upper", │ │
│ "sampled_at": "2026-05-10", │ │
│ "station": "River Mouth", │ │
│ "sample_alias": "River-R2", │ │
│ "measurements": [ │ │
│ { "name": "nitrate", "value_mg_l": 1.4 } │ │
│ ] │ │
│ } │ │
│ ] │ │
╰─────────────────────────────────────────────────┴───────────────────────────────────────────────────────╯
These are synchronous console calls. In a notebook or other running event loop,
use await with .load().
Give one measurement a type
Models mimic their wrapped values, so we can index the loaded JSON directly. The second measurement in the first sample is phosphorus, reported in µg/L:
╭───────────────────────────────────────────────╮
│ _JsonAnyDictM │
│ │
│ { "name": "Phosphorus", "value_ug_l": 120.0 } │
╰───────────────────────────────────────────────╯
JSON types alone do not describe what a measurement means. Start with one
Pydantic record for
a concentration in µg/L, wrapped in an Omnipy Model to keep its field types active during edits.
mimics exposes the
record's fields to editors and type checkers; the Model supplies field access at runtime.
╭────────────────────────────────────────────────────────────────────────────────╮
│ import omnipy.util.pydantic as pyd │
│ │
│ class RiverUgConcMeasurementRecord(pyd.BaseModel): │
│ name: str │
│ value_ug_l: float │
│ │
│ @om.mimics(RiverUgConcMeasurementRecord) │
│ class RiverUgConcMeasurementModel(om.Model[RiverUgConcMeasurementRecord]): ... │
╰────────────────────────────────────────────────────────────────────────────────╯
Constructing the Model parses the JSON measurement into the record:
╭─────────────────────────────────────────────╮
│ RiverUgConcMeasurementModel │
│ │
│ {'name': 'Phosphorus', 'value_ug_l': 120.0} │
╰─────────────────────────────────────────────╯
Give the samples a type
The full sample includes temperature in °C and concentrations in mg/L or µg/L. A union of records
tells these shapes apart by their fields. The sample record combines them with a station and a
DateModel.
The catalog's
MergeListsOfRecordsIntoDatasetByPrimaryKeyModel
merges the pages and keys the samples by their
PrimaryKey, rejecting
duplicates. Its contents are a Dataset, so we name the specialization RiverSamplesDataset.
╭──────────────────────────────────────────────────────────────────────────────────────────────╮
│ from typing import Literal │
│ │
│ class RiverTempMeasurementRecord(pyd.BaseModel): │
│ name: str │
│ value_celsius: float │
│ │
│ class RiverMgConcMeasurementRecord(pyd.BaseModel): │
│ name: str │
│ value_mg_l: float │
│ │
│ RiverMeasurementRecordType = ( │
│ RiverTempMeasurementRecord | RiverMgConcMeasurementRecord | RiverUgConcMeasurementRecord │
│ ) │
│ │
│ class RiverSampleRecord(pyd.BaseModel): │
│ river_sample_id: str │
│ catchment: str │
│ sampled_at: om.DateModel │
│ station: str │
│ sample_alias: str │
│ measurements: list[RiverMeasurementRecordType] │
│ │
│ @om.mimics(om.Dataset[om.Model[RiverSampleRecord]]) │
│ class RiverSamplesDataset( │
│ om.MergeListsOfRecordsIntoDatasetByPrimaryKeyModel[ │
│ om.PrimaryKey[Literal['river_sample_id']], RiverSampleRecord │
│ ] │
│ ): ... │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
Constructing the Dataset from the loaded pages parses them into typed samples:
╭──────────────────────────────────────┬────────────────────────────────────┬────────────────────────────────────╮
│ 0. river-sample-1 │ 1. river-sample-2 │ 2. river-sample-3 │
│ │ │ │
│ { │ { │ { │
│ 'river_sample_id': 'river-sample-… │ 'river_sample_id': 'river-sampl… │ 'river_sample_id': 'river-sampl… │
│ 'catchment': 'glomma-upper', │ 'catchment': 'glomma-upper', │ 'catchment': 'glomma-upper', │
│ 'sampled_at': '2026-05-03', │ 'sampled_at': '2026-05-10', │ 'sampled_at': '2026-05-17', │
│ 'station': 'River Mouth', │ 'station': 'River Mouth', │ 'station': 'Upstream Bend', │
│ 'sample_alias': 'River-R1', │ 'sample_alias': 'River-R2', │ 'sample_alias': 'River-R3', │
│ 'measurements': [ │ 'measurements': [ │ 'measurements': [ │
│ { │ { │ { │
│ 'name': 'Nitrate', │ 'name': 'nitrate', │ 'name': 'Temperature', │
│ 'value_mg_l': 1.2 │ 'value_mg_l': 1.4 │ 'value_celsius': 9.8 │
│ }, │ } │ } │
│ { │ ] │ ] │
│ 'name': 'Phosphorus', │ } │ } │
│ 'value_ug_l': 120.0 │ │ │
│ } │ │ │
│ ] │ │ │
│ } │ │ │
╰──────────────────────────────────────┴────────────────────────────────────┴────────────────────────────────────╯
json() displays the data
with JSON syntax highlighting. The display methods also take a color style of their own:
╭───────────────────────────────────────────────────────────────────────────────────────────────╮
│ JsonModel │
│ │
│ { │
│ "river-sample-1": { │
│ "river_sample_id": "river-sample-1", │
│ "catchment": "glomma-upper", │
│ "sampled_at": "2026-05-03", │
│ "station": "River Mouth", │
│ "sample_alias": "River-R1", │
│ "measurements": [ │
│ { "name": "Nitrate", "value_mg_l": 1.2 }, { "name": "Phosphorus", "value_ug_l": 120.0 } │
│ ] │
│ }, │
│ "river-sample-2": { │
│ "river_sample_id": "river-sample-2", │
│ "catchment": "glomma-upper", │
│ "sampled_at": "2026-05-10", │
│ "station": "River Mouth", │
│ "sample_alias": "River-R2", │
│ "measurements": [ { "name": "nitrate", "value_mg_l": 1.4 } ] │
│ }, │
│ "river-sample-3": { │
│ "river_sample_id": "river-sample-3", │
│ "catchment": "glomma-upper", │
│ "sampled_at": "2026-05-17", │
│ "station": "Upstream Bend", │
│ "sample_alias": "River-R3", │
│ "measurements": [ { "name": "Temperature", "value_celsius": 9.8 } ] │
│ } │
│ } │
╰───────────────────────────────────────────────────────────────────────────────────────────────╯
Work with the Dataset
A Dataset supports names and
integer positions. The third sample can be accessed by its name, and its station read through
the Model:
Integer indexing starts at zero, so samples[2] selects the same sample:
A list of keys selects several items and returns a Dataset:
╭─────────────────────────────────────────────────┬────────────────────────────────────────────────────────────╮
│ 0. river-sample-1 │ 1. river-sample-2 │
│ │ │
│ { │ { │
│ 'river_sample_id': 'river-sample-1', │ 'river_sample_id': 'river-sample-2', │
│ 'catchment': 'glomma-upper', │ 'catchment': 'glomma-upper', │
│ 'sampled_at': '2026-05-03', │ 'sampled_at': '2026-05-10', │
│ 'station': 'River Mouth', │ 'station': 'River Mouth', │
│ 'sample_alias': 'River-R1', │ 'sample_alias': 'River-R2', │
│ 'measurements': [ │ 'measurements': [{'name': 'nitrate', 'value_mg_l': 1.4}] │
│ {'name': 'Nitrate', 'value_mg_l': 1.2}, │ } │
│ {'name': 'Phosphorus', 'value_ug_l': 120.0} │ │
│ ] │ │
│ } │ │
╰─────────────────────────────────────────────────┴────────────────────────────────────────────────────────────╯
A slice selects by position. Here it returns the same two items:
╭───┬────────────────┬──────────────────────────┬────────┬──────────────────╮
│ # │ Data file name │ Type │ Length │ Size (in memory) │
│ │ │ │ │ │
│ 0 │ river-sample-1 │ Model[RiverSampleRecord] │ - │ 5.4 kB │
│ 1 │ river-sample-2 │ Model[RiverSampleRecord] │ - │ 4.3 kB │
╰───┴────────────────┴──────────────────────────┴────────┴──────────────────╯
Edit safely, without unpacking
Omnipy Models are self-constraining: assigning through the mimicked interface parses the new value against its field type. An invalid concentration raises a validation error:
╭─────────────────────────────────────────────╮
│ Model[RiverUgConcMeasurementRecord] │
│ │
│ {'name': 'Phosphorus', 'value_ug_l': 120.0} │
╰─────────────────────────────────────────────╯
ValidationError
In interactive mode, the failed edit is rolled back to the
last valid snapshot. Displaying the measurement again shows the original 120.0 µg/L:
╭─────────────────────────────────────────────╮
│ Model[RiverUgConcMeasurementRecord] │
│ │
│ {'name': 'Phosphorus', 'value_ug_l': 120.0} │
╰─────────────────────────────────────────────╯
A numeric string can be parsed as a float. Let's correct the
reading to 150.0 µg/L. The measurement is part of the sample, so the correction
is visible there too:
╭─────────────────────────────────────────────╮
│ Model[RiverUgConcMeasurementRecord] │
│ │
│ {'name': 'Phosphorus', 'value_ug_l': 150.0} │
╰─────────────────────────────────────────────╯
Convert to a common representation
Other sources use different field names and units. A
ConverterModel
makes a conversion reusable and validates its output. First, a measurement mapper uses the record
type to choose the conversion, including µg/L to mg/L:
╭─────────────────────────────────────────────────────────────────────────────────────────╮
│ class NormalizedMeasurementRecord(pyd.BaseModel): │
│ type: Literal['nitrate', 'phosphorus', 'temperature'] │
│ value: float │
│ unit: Literal['mg/L', 'deg_C'] │
│ │
│ @om.mimics(NormalizedMeasurementRecord) │
│ class NormalizedRiverMeasurementMapper( │
│ om.ConverterModel[RiverMeasurementRecordType, NormalizedMeasurementRecord] │
│ ): │
│ @classmethod │
│ def _convert(cls, data: RiverMeasurementRecordType) -> NormalizedMeasurementRecord: │
│ match data: │
│ case RiverTempMeasurementRecord(): │
│ return NormalizedMeasurementRecord( │
│ type=data.name.lower(), value=data.value_celsius, unit='deg_C' │
│ ) │
│ case RiverMgConcMeasurementRecord(): │
│ return NormalizedMeasurementRecord( │
│ type=data.name.lower(), value=data.value_mg_l, unit='mg/L' │
│ ) │
│ case RiverUgConcMeasurementRecord(): │
│ return NormalizedMeasurementRecord( │
│ type=data.name.lower(), value=data.value_ug_l / 1000, unit='mg/L' │
│ ) │
╰─────────────────────────────────────────────────────────────────────────────────────────╯
Constructing the mapper converts the corrected measurement to 0.15 mg/L:
╭───────────────────────────────────────────────────────╮
│ NormalizedRiverMeasurementMapper │
│ │
│ {'type': 'phosphorus', 'value': 0.15, 'unit': 'mg/L'} │
╰───────────────────────────────────────────────────────╯
Next, a sample mapper renames the sample fields and reuses the measurement mapper's
convert_all()
for its measurements. A Dataset of that mapper converts all the samples:
╭───────────────────────────────────────────────────────────────────────────────────╮
│ class NormalizedSampleRecord(pyd.BaseModel): │
│ sample_key: str │
│ monitoring_date: om.DateModel │
│ location_name: str │
│ measurements: list[NormalizedMeasurementRecord] │
│ │
│ @om.mimics(NormalizedSampleRecord) │
│ class NormalizedRiverSampleMapper( │
│ om.ConverterModel[RiverSampleRecord, NormalizedSampleRecord] │
│ ): │
│ @classmethod │
│ def _convert(cls, data: RiverSampleRecord) -> NormalizedSampleRecord: │
│ return NormalizedSampleRecord( │
│ sample_key=data.sample_alias.lower(), │
│ monitoring_date=data.sampled_at, │
│ location_name=data.station, │
│ measurements=list( │
│ NormalizedRiverMeasurementMapper.convert_all(data.measurements) │
│ ), │
│ ) │
│ │
│ class NormalizedRiverSamplesDataset(om.Dataset[NormalizedRiverSampleMapper]): ... │
╰───────────────────────────────────────────────────────────────────────────────────╯
The Dataset constructor applies the mapper to every sample, keeping the sample ids as keys.
peek() displays the converted samples side by side:
╭──────────────────────────────────────────────────────────────────────────────────╮
│ JsonModel │
│ │
│ { │
│ "river-sample-1": { │
│ "sample_key": "river-r1", │
│ "monitoring_date": "2026-05-03", │
│ "location_name": "River Mouth", │
│ "measurements": [ │
│ { "type": "nitrate" , "value": 1.2 , "unit": "mg/L" }, │
│ { "type": "phosphorus", "value": 0.15, "unit": "mg/L" } │
│ ] │
│ }, │
│ "river-sample-2": { │
│ "sample_key": "river-r2", │
│ "monitoring_date": "2026-05-10", │
│ "location_name": "River Mouth", │
│ "measurements": [ { "type": "nitrate", "value": 1.4, "unit": "mg/L" } ] │
│ }, │
│ "river-sample-3": { │
│ "sample_key": "river-r3", │
│ "monitoring_date": "2026-05-17", │
│ "location_name": "Upstream Bend", │
│ "measurements": [ { "type": "temperature", "value": 9.8, "unit": "deg_C" } ] │
│ } │
│ } │
╰──────────────────────────────────────────────────────────────────────────────────╯
For a one-line conversion and display,
to() is convenient too:
╭──────────────────────────────────────────────────────────────╮
│ JsonModel │
│ │
│ { │
│ "sample_key": "river-r1", │
│ "monitoring_date": "2026-05-03", │
│ "location_name": "River Mouth", │
│ "measurements": [ │
│ { "type": "nitrate" , "value": 1.2 , "unit": "mg/L" }, │
│ { "type": "phosphorus", "value": 0.15, "unit": "mg/L" } │
│ ] │
│ } │
╰──────────────────────────────────────────────────────────────╯
Explore the hierarchy as nested Datasets
StructuredNestedDataset
represents mappings recursively as Datasets and recognizes the structure of the leaves: scalar
fields become
JsonScalarModel
items, and each measurement list, a list of records with the same fields, becomes a column-wise
table Model. The conversion preserves the sample ids and field names, while making the samples
nested Datasets with their fields as items:
╭───┬────────────────┬─────────────────────────┬────────┬──────────────────╮
│ # │ Data file name │ Type │ Length │ Size (in memory) │
│ │ │ │ │ │
│ 0 │ river-sample-1 │ StructuredNestedDataset │ 4 │ 5.0 kB │
│ 1 │ river-sample-2 │ StructuredNestedDataset │ 4 │ 4.8 kB │
│ 2 │ river-sample-3 │ StructuredNestedDataset │ 4 │ 4.8 kB │
╰───┴────────────────┴─────────────────────────┴────────┴──────────────────╯
Calling peek() on a StructuredNestedDataset shows a hierarchical view of the structure, with
tables nested within tables:
╭────────────┬─────────────────┬───────────────┬─────────────────────────────────────────╮
│ 0. │ 1. │ 2. │ 3. measurements │
│ sample_key │ monitoring_date │ location_name │ │
│ │ │ │ ╭───┬────────────┬──────────┬─────────╮ │
│ "river-r1" │ "2026-05-03" │ "River Mouth" │ │ # │ 0. type │ 1. value │ 2. unit │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ │ 0 │ nitrate │ 1.2 │ mg/L │ │
│ │ │ │ │ 1 │ phosphorus │ 0.15 │ mg/L │ │
│ │ │ │ ╰───┴────────────┴──────────┴─────────╯ │
╰────────────┴─────────────────┴───────────────┴─────────────────────────────────────────╯
Turn the measurements into tables
The measurement tables have type, value, and unit columns.
do() applies a
placeholder expression to each item, here selecting the measurements from each sample, keeping
the sample ids as keys. A
PandasDataset
then converts each measurement table to a pandas DataFrame:
>>> measurements = nested_samples.do(om.x['measurements'])
>>> tables = om.PandasDataset(measurements)
╭─────────────────────────────────────────┬──────────────────────────────────────┬──────────────────────────────────────────╮
│ 0. river-sample-1 │ 1. river-sample-2 │ 2. river-sample-3 │
│ │ │ │
│ ╭───┬────────────┬──────────┬─────────╮ │ ╭───┬─────────┬──────────┬─────────╮ │ ╭───┬─────────────┬──────────┬─────────╮ │
│ │ # │ 0. type │ 1. value │ 2. unit │ │ │ # │ 0. type │ 1. value │ 2. unit │ │ │ # │ 0. type │ 1. value │ 2. unit │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ 0 │ nitrate │ 1.2 │ mg/L │ │ │ 0 │ nitrate │ 1.4 │ mg/L │ │ │ 0 │ temperature │ 9.8 │ deg_C │ │
│ │ 1 │ phosphorus │ 0.15 │ mg/L │ │ ╰───┴─────────┴──────────┴─────────╯ │ ╰───┴─────────────┴──────────┴─────────╯ │
│ ╰───┴────────────┴──────────┴─────────╯ │ │ │
╰─────────────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────────╯
Compose tasks and flows
The steps so far form a dataflow: fetch the pages, key the samples, normalize them, expose the
hierarchy, select measurements, and tabulate. A
TaskTemplate wraps a function;
a LinearFlowTemplate
declares its stages in order. Model and Dataset constructors can be stages directly.
resolve() handles the loader's
result whether it is already available or still awaiting completion:
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ @om.TaskTemplate() │
│ async def fetch_sample_pages(page_urls: om.HttpUrlDataset) -> om.JsonDataset: │
│ return await om.resolve(om.JsonDataset.load(page_urls)) │
│ │
│ @om.TaskTemplate() │
│ def select_measurements(samples: om.StructuredNestedDataset) -> om.StructuredNestedDataset: │
│ return samples.do(om.x['measurements']) │
│ │
│ @om.LinearFlowTemplate( │
│ fetch_sample_pages, │
│ RiverSamplesDataset, │
│ NormalizedRiverSamplesDataset, │
│ om.StructuredNestedDataset, │
│ select_measurements, │
│ om.PandasDataset, │
│ ) │
│ async def harmonize_river_samples(page_urls: om.HttpUrlDataset) -> om.PandasDataset: ... │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
The flow reloads the original pages, so its phosphorus reading starts at 0.12 mg/L.
The manual correction above belongs to the earlier in-memory Dataset:
╭─────────────────────────────────────────┬──────────────────────────────────────┬──────────────────────────────────────────╮
│ 0. river-sample-1 │ 1. river-sample-2 │ 2. river-sample-3 │
│ │ │ │
│ ╭───┬────────────┬──────────┬─────────╮ │ ╭───┬─────────┬──────────┬─────────╮ │ ╭───┬─────────────┬──────────┬─────────╮ │
│ │ # │ 0. type │ 1. value │ 2. unit │ │ │ # │ 0. type │ 1. value │ 2. unit │ │ │ # │ 0. type │ 1. value │ 2. unit │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ 0 │ nitrate │ 1.2 │ mg/L │ │ │ 0 │ nitrate │ 1.4 │ mg/L │ │ │ 0 │ temperature │ 9.8 │ deg_C │ │
│ │ 1 │ phosphorus │ 0.12 │ mg/L │ │ ╰───┴─────────┴──────────┴─────────╯ │ ╰───┴─────────────┴──────────┴─────────╯ │
│ ╰───┴────────────┴──────────┴─────────╯ │ │ │
╰─────────────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────────╯
The flow declaration is what a reviewer reads, and what an AI agent working on the pipeline
produces and is held to: the stages, their order, and the types passing between them, with every
Model and Dataset validating its data at its stage. The same flow runs on
Prefect by setting om.runtime.config.engine.choice = 'prefect';
see Tutorial 8: Prefect orchestration.
Where next
- Meet the comics roster: the Omnificients and the Un-FAIR Alliance turn parsing, validation, batch processing, API retrieval, and visualization into stories.
- The full water-sample harmonization scenario extends this walkthrough to river and wastewater sources, reusable child flows, and Pandas tables of samples and measurements.
- The sequence data brokering scenario shows a larger dataflow: a typed submission with Datasets of samples and files, mapped requests posted to mock web services, derived status, and DAG child flows. A tutorial based on it is in preparation.
- Jupyter notebooks with real-world Omnipy code: ihec_encode, isa_json_data_wrangling_example, and nobel_idw2025.
- Tutorial 8: Prefect orchestration runs a flow on the Prefect engine.
- The Omnipy skills are what a coding agent reads to work with Omnipy. The first, omnipy-dataflows, covers designing dataflows: typed contracts first, reusable tasks and flows, and narrative tests that double as tutorials. The skills ship with Omnipy, see step 4 of Installing Omnipy above.
- The workshop Structure your AI-powered research data pipelines with Omnipy at Digital Scholarship Days 2027, University of Oslo, is where this direction is taken further.
- The omnipy-examples package has stand-alone
example scripts (
pip install omnipy-examples, thenomnipy-examples --help). It is updated separately and may lag behind the Omnipy release.