> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cirron.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Profiling training

> Instrument training runs end-to-end.

This guide walks the training-side surface as one story, from the
one-line zero-touch setup to custom scopes and marks. For a flat
signature reference, jump to [Reference](/sdk/reference/profile).

For inference instrumentation, see the [Inference guide](/sdk/inference).

## The happy path

One call, called once per process. Framework hooks do the work.

```python theme={null}
import cirron as ci

ci.profile()
```

That's the whole setup. The SDK detects installed frameworks and
installs hooks for each. It opens the `cirron.session` root scope,
starts a background flush thread, and registers clean-shutdown handlers.

| Framework                | Triggered by           | What you get                                                          |
| ------------------------ | ---------------------- | --------------------------------------------------------------------- |
| **Keras**                | `model.fit()`          | `epoch` and `batch` scopes; logged metrics as marks                   |
| **HuggingFace Trainer**  | `trainer.train()`      | `epoch` and `step` scopes; end-of-epoch values as summary marks       |
| **PyTorch + DataLoader** | `for batch in loader:` | `data_load`, `forward`, `backward`, `optimizer_step`, implicit `step` |

<CodeGroup>
  ```python Keras theme={null}
  import cirron as ci
  ci.profile()

  model.fit(X, y, epochs=20)
  ```

  ```python HuggingFace theme={null}
  import cirron as ci
  ci.profile()

  trainer = Trainer(model=model, args=args, train_dataset=ds)
  trainer.train()
  ```

  ```python PyTorch + DataLoader theme={null}
  import cirron as ci
  ci.profile()

  for epoch in range(20):
      for batch in loader:
          loss = train_step(batch)
  ```
</CodeGroup>

### Distributed training

Every rank calls `ci.profile()`. The SDK reads `RANK` / `LOCAL_RANK` /
`WORLD_SIZE` from the environment and tags every span with the rank.
The platform merges views at query time.

## Custom loops

If your loop doesn't fit the hook patterns (generator-based iteration,
custom samplers, step counters without a DataLoader), wrap the
iterables. They're transparent: `ci.epochs(range(20))` yields `0..19`
exactly while opening an indexed `epoch` scope around each iteration.

```python theme={null}
for epoch in ci.epochs(range(20)):
    for batch in ci.batches(loader):
        loss = train_step(batch)
        ci.mark("loss", loss.item())
```

`ci.batches()` additionally measures DataLoader stall time (waiting on
data vs. compute) when the iterable is a `torch.utils.data.DataLoader`.
See [`ci.epochs / ci.batches`](/sdk/reference/loop-wrappers).

## Scopes and marks

Use [`ci.scope`](/sdk/reference/scope) for explicit spans the hooks and
wrappers don't cover (augmentation, beam search, custom schedulers).
Use [`ci.mark`](/sdk/reference/mark) to attach scalar values to the
innermost open scope.

```python theme={null}
with ci.scope("augmentation"):
    batch = augment(batch)

ci.mark("loss", loss.item())                       # point: time-series
ci.mark("epoch_accuracy", val_acc, kind="summary") # one canonical value per span
```

Scopes nest arbitrarily under whatever scope is already open, so the
hooks' `epoch` / `batch` / `forward` tree stays intact and your custom
scope slots in at the right level. Max depth: 64.

## Framework hooks

Hooks install automatically by `ci.profile()` when the framework is
importable. Each hook is wrapped in a top-level `try/except`, so a
failing hook logs a warning and training continues.

When multiple frameworks are installed, hooks fire in priority order
**`transformers > tensorflow > torch`**. Higher-level frameworks claim
ownership of the semantic scopes (`epoch`, `step`) first; lower-level
frameworks yield, so no semantic scope is duplicated.

* **PyTorch** uses module pre/post hooks, autograd hooks,
  optimizer-step hooks, and DataLoader iterator wrapping; gradient
  accumulation collapses to a single `step` span. CUDA timing via
  `torch.cuda.Event` pairs lands on the `gpu_ns` attribute.
* **Keras** auto-registers a `Callback` on `Model.fit`; logged metrics
  become marks.
* **HuggingFace** auto-registers a `TrainerCallback` on
  `Trainer.__init__`; end-of-epoch values are marked `kind="summary"`.
* **scikit-learn** has no auto-hook; wrap the estimator with
  [`ci.wrap`](/sdk/reference/wrap).

See [`ci.profile`](/sdk/reference/profile) for the full hook table and
per-framework details.

## Snapshots

At each detected epoch boundary, the SDK captures per-tensor statistics
for every parameter in the model being profiled. Three modes, controlled
by `ci.profile(snapshots=...)`.

| Mode        | Cost per boundary         | What's captured                                         |
| ----------- | ------------------------- | ------------------------------------------------------- |
| `"stats"`   | ≤ 50 ms (typical)         | `{mean, std, min, max, norm, histogram[16]}` per tensor |
| `"sampled"` | ≤ 200 ms on sampled steps | Stats + raw tensors for `random() < sample_rate` epochs |
| `"full"`    | unbounded; debug-only     | Stats + raw tensors every epoch                         |

In `"sampled"` and `"full"` modes, raw tensors are written as
safetensors blobs at `./.cirron/snapshots/<span_id>/weights.safetensors`
(and `gradients.safetensors` when gradients are non-`None`).

Keras and HuggingFace hooks discover the model from their callback
kwargs. **Bare PyTorch loops** that don't use `ci.epochs()` should
register the model once with [`ci.watch(model)`](/sdk/reference/watch)
before training.

<Note>
  `"full"` mode is not recommended for models over 100M parameters. At
  7B+, even `"sampled"` is expensive. Drop `sample_rate`.
</Note>

## Output sinks

By default `ci.profile()` writes each batch as a JSON file under
`./.cirron/spool/`. Swap or fan-out via `output=`:

```python theme={null}
ci.profile(output="stdout")            # live [cirron] lines per closed span
ci.profile(output=["spool", "log"])    # disk + log
ci.profile(output="none")              # nothing local; pair with ci.trace()
```

Sinks are independent of the platform transport. `output="none"` inside
a Cirron pipeline still ships traces to the platform over the kernel
event stream. See [`ci.profile`](/sdk/reference/profile#output-sinks)
for the full table.

## In-process read-back

[`ci.trace()`](/sdk/reference/trace) returns the current session's scope
tree without leaving the process, useful in notebooks (cell renders
inline) and for ad-hoc analysis (flat DataFrame for quantiles and
group-bys):

```python theme={null}
ci.trace()                  # text tree
ci.trace(format="df")       # pandas DataFrame, one row per span
ci.trace(name="epoch")      # filter by scope name
```

Works with or without an active profiler. When no profiler is attached,
the call is purely in-memory and never writes a spool file (safe on
read-only filesystems).

## Lifecycle

The `atexit` handler registered by `ci.profile()` flushes and shuts down
on process exit. Reach for the manual helpers only when you need
deterministic behavior in tests or hot-reload scenarios:

```python theme={null}
ci.flush()       # synchronously drain buffers to spool
ci.health()      # dict: enabled, drop counts, hook handles, transport
ci.shutdown()    # close root scope, flush, stop thread, clear singleton
```

See [Lifecycle](/sdk/reference/lifecycle).

## Next

<CardGroup cols={3}>
  <Card title="Inference guide" icon="server" href="/sdk/inference">
    `@ci.inference`, LLM detection, config-driven capture.
  </Card>

  <Card title="ci.profile reference" icon="code" href="/sdk/reference/profile">
    Full signature and parameter table.
  </Card>

  <Card title="ci.trace reference" icon="diagram-project" href="/sdk/reference/trace">
    Read-back formats and filters.
  </Card>
</CardGroup>