Data loading - Cirron Documentation

ci.load() is the single entry point for data access. One function, flat kwargs, local-first by default. Nothing hits the network unless you explicitly opt in via source="platform" or a scheme in the source string. A scheme in the name string (s3://, gs://, postgres://, …) always overrides the source= kwarg. Without a scheme and with the default source="local", ci.load() probes the local filesystem and never calls the platform. For the full signature and parameter table, see ci.load.

Where the data comes from

# Single file
df = ci.load("./data/training/events.parquet")

# Directory: probes ./training-data/, then ./data/training-data/
df = ci.load("training-data")

# Multi-source union: parallel load, concatenate
df = ci.load(["./data/a/", "./data/b/"])

df = ci.load("s3://ml-data/events/")
df = ci.load("gs://analytics-bucket/events/")
df = ci.load("azure://container/events/")
df = ci.load("file:///absolute/path/events.parquet")

df = ci.load("postgres://prod/events",   where="created_at > '2025-01-01'")
df = ci.load("mysql://analytics/clicks", where="country = 'US'")
df = ci.load("databricks://analytics.public.clicks", where="country = 'US'")
df = ci.load("snowflake://warehouse/db/schema/table", where="region = 'EMEA'")

# Resolve a registered dataset on the Cirron platform. The SDK does not
# hold raw credentials; it asks the platform for a scoped, short-lived
# token per call.
df = ci.load("bucket1", source="platform")

Data staged by a pipeline

If a Load Data step runs before your training step, the dataset is already prepared. Point ci.load() at the staged location (default /data/input) instead of wiring up the source yourself:

import os
import cirron as ci

DATA_DIR = ci.env("CIRRON_DATA_DIR")
df = ci.load(os.path.join(DATA_DIR, "input"))

See Reading training data for the full contract.

Filtering and selection

match= and ext= work on any filesystem-backed source (local, S3, GCS, Azure, file://).

# Glob pattern
df = ci.load("s3://ml-data/events/", match="year=2025/month=*/*.parquet")

# Structured match: path glob + filename regex + columns pushdown
df = ci.load(
    "s3://ml-data/events/",
    match={
        "path": "year=2025/month=*/",
        "filename": r"events_.*\.parquet",
        "columns": ["user_id", "ts", "event_type"],
    },
)

# Extension shorthand
df = ci.load("./data/", ext=["csv", "parquet"])

# Column selection: pushed down to Parquet / SQL readers
df = ci.load("./events.parquet", columns=["user_id", "ts", "event_type"])

where= is passed through to SQL sources unescaped. It’s your query, against your data.

Streaming SQL results

as_="iter" on a SQL source streams. Each driver reads through a server-side cursor (a named cursor on Postgres, an unbuffered SSCursor on MySQL, and the default paging cursors on Snowflake and Databricks), so only one batch of batch_size rows is in memory at a time, however large the table:

for batch in ci.load("postgres://prod/events", as_="iter", batch_size=10_000):
    train_on(batch)   # list of row dicts

The connection opens and the query runs inside ci.load(), so a bad credential or table fails there rather than on your first loop iteration. It stays open while you iterate and closes when the iterator is exhausted or you call .close() on it, which is worth doing if you stop early. lazy=True defers all of this until .collect(). Rows carry the driver’s own values (None for NULL, Decimal, datetime) rather than pandas-coerced ones. Every other as_= value returns a single in-memory object, so those still read the full result; bound them with LIMIT or where=.

Transforms at load time

# Row-wise: receives one row at a time
df = ci.load(
    "./raw/",
    columns=["raw_text", "label"],
    map=lambda row: {"text": row["raw_text"].lower(), "label": int(row["label"])},
)

# Batch-wise: receives the full frame at once
@ci.map
def to_features(frame):
    frame["text"] = frame["raw_text"].str.lower()
    return frame

df = ci.load("./raw/", map=to_features)

Use @ci.map when the transform is vectorizable against pandas/polars; use plain callables for per-row work. The @ci.map decorator sets a _cirron_batch_map=True attribute that ci.load() checks at runtime: present means batch, absent means row-by-row. On a streaming SQL load (as_="iter"), a row-wise callable runs batch by batch as rows arrive. A @ci.map function is rejected with a CirronError before any connection opens, because its contract is a single call over the whole result, which a streaming load never holds.

Size guardrails

Before downloading anything, ci.load() sums the matched bytes across all sources and applies a three-tier policy on the total:

Size	Behavior
< 1 GB	Silent
< 10 GB	`WARNING` log with narrowing hints (use `match=`, etc.)
≥ 10 GB	Raises `CirronDataSizeError` unless `confirm_large=True`

Configurable per Cirron instance via load_warn_bytes and load_max_bytes. SQL sources opt out (they can’t report a size before executing the query). Stream large results with as_="iter", or bound them with LIMIT.

Credential resolution for SQL sources

Credentials resolve in this order, first match wins:

URI inline: postgres://user:pass@host/db
Platform integrations: GET /api/integrations/resolve with a scoped, short-lived token (requires a configured Cirron integration for that host)
ci.secret("<scheme>-<host>"): platform-mounted secret
Driver env var: PGPASSWORD / MYSQL_PWD / SNOWFLAKE_PASSWORD / DATABRICKS_TOKEN

Same code works across cloud, on-prem, and air-gapped environments; only the credential source changes.

Not-yet-shipped

search= / top_k= accept input today for API stability but raise the stdlib NotImplementedError until the platform vector-index feature ships. The docs will update when it does.

Errors

CirronDependencyError, CirronDataSizeError, CirronDatasetNotFound, CirronPlatformRequired: see Errors for what triggers each.

Configuration

The Cirron class and where credentials come from.

ci.load reference

Full signature, return types, lazy loading.

​Where the data comes from

​Data staged by a pipeline

​Filtering and selection

​Streaming SQL results

​Transforms at load time

​Size guardrails

​Credential resolution for SQL sources

​Not-yet-shipped

​Errors

​Next