Data loading - Cirron Documentation

ci.load() is the single entry point for data access. One function, flat kwargs, local-first by default. Nothing hits the network unless you explicitly opt in via source="platform" or a scheme in the source string. A scheme in the name string (s3://, gs://, postgres://, …) always overrides the source= kwarg. Without a scheme and with the default source="local", ci.load() probes the local filesystem and never calls the platform. For the full signature and parameter table, see ci.load.

Where the data comes from

# Single file
df = ci.load("./data/training/events.parquet")

# Directory: probes ./training-data/, then ./data/training-data/
df = ci.load("training-data")

# Multi-source union: parallel load, concatenate
df = ci.load(["./data/a/", "./data/b/"])

df = ci.load("s3://ml-data/events/")
df = ci.load("gs://analytics-bucket/events/")
df = ci.load("azure://container/events/")
df = ci.load("file:///absolute/path/events.parquet")

df = ci.load("postgres://prod/events",   where="created_at > '2025-01-01'")
df = ci.load("mysql://analytics/clicks", where="country = 'US'")
df = ci.load("databricks://analytics.public.clicks", where="country = 'US'")
df = ci.load("snowflake://warehouse/db/schema/table", where="region = 'EMEA'")

# Resolve a registered dataset on the Cirron platform. The SDK does not
# hold raw credentials; it asks the platform for a scoped, short-lived
# token per call.
df = ci.load("bucket1", source="platform")

Data staged by a pipeline

If a Load Data step runs before your training step, the dataset is already prepared. Point ci.load() at the staged location (default /data/input) instead of wiring up the source yourself:

import os
import cirron as ci

DATA_DIR = ci.env("CIRRON_DATA_DIR")
df = ci.load(os.path.join(DATA_DIR, "input"))

See Reading training data for the full contract.

Filtering and selection

match= and ext= work on any filesystem-backed source (local, S3, GCS, Azure, file://).

# Glob pattern
df = ci.load("s3://ml-data/events/", match="year=2025/month=*/*.parquet")

# Structured match: path glob + filename regex + columns pushdown
df = ci.load(
    "s3://ml-data/events/",
    match={
        "path": "year=2025/month=*/",
        "filename": r"events_.*\.parquet",
        "columns": ["user_id", "ts", "event_type"],
    },
)

# Extension shorthand
df = ci.load("./data/", ext=["csv", "parquet"])

# Column selection: pushed down to Parquet / SQL readers
df = ci.load("./events.parquet", columns=["user_id", "ts", "event_type"])

where= is passed through to SQL sources unescaped. It’s your query, against your data. Bound result size with LIMIT when you can.

Transforms at load time

# Row-wise: receives one row at a time
df = ci.load(
    "./raw/",
    columns=["raw_text", "label"],
    map=lambda row: {"text": row["raw_text"].lower(), "label": int(row["label"])},
)

# Batch-wise: receives the full frame at once
@ci.map
def to_features(frame):
    frame["text"] = frame["raw_text"].str.lower()
    return frame

df = ci.load("./raw/", map=to_features)

Use @ci.map when the transform is vectorizable against pandas/polars; use plain callables for per-row work. The @ci.map decorator sets a _cirron_batch_map=True attribute that ci.load() checks at runtime: present means batch, absent means row-by-row.

Size guardrails

Before downloading anything, ci.load() sums the matched bytes across all sources and applies a three-tier policy on the total:

Size	Behavior
< 1 GB	Silent
< 10 GB	`WARNING` log with narrowing hints (use `match=`, etc.)
≥ 10 GB	Raises `CirronDataSizeError` unless `confirm_large=True`

Configurable per Cirron instance via load_warn_bytes and load_max_bytes. SQL sources opt out (they can’t report a size before executing the query). Use LIMIT to bound results.

Credential resolution for SQL sources

Credentials resolve in this order, first match wins:

URI inline: postgres://user:pass@host/db
Platform integrations: GET /api/integrations/resolve with a scoped, short-lived token (requires a configured Cirron integration for that host)
ci.secret("<scheme>-<host>"): platform-mounted secret
Driver env var: PGPASSWORD / MYSQL_PWD / SNOWFLAKE_PASSWORD / DATABRICKS_TOKEN

Same code works across cloud, on-prem, and air-gapped environments; only the credential source changes.

Not-yet-shipped

search= / top_k= accept input today for API stability but raise the stdlib NotImplementedError until the platform vector-index feature ships. The docs will update when it does.

Errors

CirronDependencyError, CirronDataSizeError, CirronDatasetNotFound, CirronPlatformRequired: see Errors for what triggers each.

Configuration

The Cirron class and where credentials come from.

ci.load reference

Full signature, return types, lazy loading.

​Where the data comes from

​Data staged by a pipeline

​Filtering and selection

​Transforms at load time

​Size guardrails

​Credential resolution for SQL sources

​Not-yet-shipped

​Errors

​Next