> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cirron.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data loading

> Unified data access across local, cloud, and SQL.

`ci.load()` is the single entry point for data access. One function,
flat kwargs, local-first by default. Nothing hits the network unless you
explicitly opt in via `source="platform"` or a scheme in the source
string.

A scheme in the `name` string (`s3://`, `gs://`, `postgres://`, ...)
**always overrides** the `source=` kwarg. Without a scheme and with the
default `source="local"`, `ci.load()` probes the local filesystem and
never calls the platform.

For the full signature and parameter table, see
[`ci.load`](/sdk/reference/load).

## Where the data comes from

<CodeGroup>
  ```python Local theme={null}
  # Single file
  df = ci.load("./data/training/events.parquet")

  # Directory: probes ./training-data/, then ./data/training-data/
  df = ci.load("training-data")

  # Multi-source union: parallel load, concatenate
  df = ci.load(["./data/a/", "./data/b/"])
  ```

  ```python Cloud theme={null}
  df = ci.load("s3://ml-data/events/")
  df = ci.load("gs://analytics-bucket/events/")
  df = ci.load("azure://container/events/")
  df = ci.load("file:///absolute/path/events.parquet")
  ```

  ```python SQL theme={null}
  df = ci.load("postgres://prod/events",   where="created_at > '2025-01-01'")
  df = ci.load("mysql://analytics/clicks", where="country = 'US'")
  df = ci.load("databricks://analytics.public.clicks", where="country = 'US'")
  df = ci.load("snowflake://warehouse/db/schema/table", where="region = 'EMEA'")
  ```

  ```python Platform theme={null}
  # Resolve a registered dataset on the Cirron platform. The SDK does not
  # hold raw credentials; it asks the platform for a scoped, short-lived
  # token per call.
  df = ci.load("bucket1", source="platform")
  ```
</CodeGroup>

### Data staged by a pipeline

If a **Load Data** step runs before your training step, the dataset is
already prepared. Point `ci.load()` at the staged location (default
`/data/input`) instead of wiring up the source yourself:

```python theme={null}
import os
import cirron as ci

DATA_DIR = ci.env("CIRRON_DATA_DIR")
df = ci.load(os.path.join(DATA_DIR, "input"))
```

See [Reading training data](/guides/models/training-jobs#reading-training-data)
for the full contract.

## Filtering and selection

`match=` and `ext=` work on any filesystem-backed source (local, S3,
GCS, Azure, `file://`).

```python theme={null}
# Glob pattern
df = ci.load("s3://ml-data/events/", match="year=2025/month=*/*.parquet")

# Structured match: path glob + filename regex + columns pushdown
df = ci.load(
    "s3://ml-data/events/",
    match={
        "path": "year=2025/month=*/",
        "filename": r"events_.*\.parquet",
        "columns": ["user_id", "ts", "event_type"],
    },
)

# Extension shorthand
df = ci.load("./data/", ext=["csv", "parquet"])

# Column selection: pushed down to Parquet / SQL readers
df = ci.load("./events.parquet", columns=["user_id", "ts", "event_type"])
```

`where=` is passed through to SQL sources unescaped. It's your query,
against your data. Bound result size with `LIMIT` when you can.

## Transforms at load time

```python theme={null}
# Row-wise: receives one row at a time
df = ci.load(
    "./raw/",
    columns=["raw_text", "label"],
    map=lambda row: {"text": row["raw_text"].lower(), "label": int(row["label"])},
)

# Batch-wise: receives the full frame at once
@ci.map
def to_features(frame):
    frame["text"] = frame["raw_text"].str.lower()
    return frame

df = ci.load("./raw/", map=to_features)
```

Use `@ci.map` when the transform is vectorizable against pandas/polars;
use plain callables for per-row work. The `@ci.map` decorator sets a
`_cirron_batch_map=True` attribute that `ci.load()` checks at runtime:
present means batch, absent means row-by-row.

## Size guardrails

Before downloading anything, `ci.load()` sums the matched bytes across
all sources and applies a three-tier policy on the total:

| Size     | Behavior                                                 |
| -------- | -------------------------------------------------------- |
| \< 1 GB  | Silent                                                   |
| \< 10 GB | `WARNING` log with narrowing hints (use `match=`, etc.)  |
| ≥ 10 GB  | Raises `CirronDataSizeError` unless `confirm_large=True` |

Configurable per `Cirron` instance via `load_warn_bytes` and
`load_max_bytes`. SQL sources opt out (they can't report a size before
executing the query). Use `LIMIT` to bound results.

## Credential resolution for SQL sources

Credentials resolve in this order, first match wins:

1. **URI inline**: `postgres://user:pass@host/db`
2. **Platform integrations**: `GET /api/integrations/resolve` with a
   scoped, short-lived token (requires a configured Cirron integration
   for that host)
3. **`ci.secret("<scheme>-<host>")`**: platform-mounted secret
4. **Driver env var**: `PGPASSWORD` / `MYSQL_PWD` /
   `SNOWFLAKE_PASSWORD` / `DATABRICKS_TOKEN`

Same code works across cloud, on-prem, and air-gapped environments; only
the credential source changes.

## Not-yet-shipped

`search=` / `top_k=` accept input today for API stability but raise the
stdlib `NotImplementedError` until the platform vector-index feature
ships. The docs will update when it does.

## Errors

`CirronDependencyError`, `CirronDataSizeError`, `CirronDatasetNotFound`,
`CirronPlatformRequired`: see [Errors](/sdk/errors) for what triggers
each.

## Next

<CardGroup cols={2}>
  <Card title="Configuration" icon="gear" href="/sdk/configuration">
    The `Cirron` class and where credentials come from.
  </Card>

  <Card title="ci.load reference" icon="code" href="/sdk/reference/load">
    Full signature, return types, lazy loading.
  </Card>
</CardGroup>