Profiling training - Cirron Documentation

This guide walks the training-side surface as one story, from the one-line zero-touch setup to custom scopes and marks. For a flat signature reference, jump to Reference. For inference instrumentation, see the Inference guide.

The happy path

One call, called once per process. Framework hooks do the work.

import cirron as ci

ci.profile()

That’s the whole setup. The SDK detects installed frameworks and installs hooks for each. It opens the cirron.session root scope, starts a background flush thread, and registers clean-shutdown handlers.

Framework	Triggered by	What you get
Keras	`model.fit()`	`epoch` and `batch` scopes; logged metrics as marks
HuggingFace Trainer	`trainer.train()`	`epoch` and `step` scopes; end-of-epoch values as summary marks
PyTorch + DataLoader	`for batch in loader:`	`data_load`, `forward`, `backward`, `optimizer_step`, implicit `step`

import cirron as ci
ci.profile()

model.fit(X, y, epochs=20)

import cirron as ci
ci.profile()

trainer = Trainer(model=model, args=args, train_dataset=ds)
trainer.train()

import cirron as ci
ci.profile()

for epoch in range(20):
    for batch in loader:
        loss = train_step(batch)

Distributed training

Every rank calls ci.profile(). The SDK reads RANK / LOCAL_RANK / WORLD_SIZE from the environment and tags every span with the rank. The platform merges views at query time.

Custom loops

If your loop doesn’t fit the hook patterns (generator-based iteration, custom samplers, step counters without a DataLoader), wrap the iterables. They’re transparent: ci.epochs(range(20)) yields 0..19 exactly while opening an indexed epoch scope around each iteration.

for epoch in ci.epochs(range(20)):
    for batch in ci.batches(loader):
        loss = train_step(batch)
        ci.mark("loss", loss.item())

ci.batches() additionally measures DataLoader stall time (waiting on data vs. compute) when the iterable is a torch.utils.data.DataLoader. See ci.epochs / ci.batches.

Scopes and marks

Use ci.scope for explicit spans the hooks and wrappers don’t cover (augmentation, beam search, custom schedulers). Use ci.mark to attach scalar values to the innermost open scope.

with ci.scope("augmentation"):
    batch = augment(batch)

ci.mark("loss", loss.item())                       # point: time-series
ci.mark("epoch_accuracy", val_acc, kind="summary") # one canonical value per span

Scopes nest arbitrarily under whatever scope is already open, so the hooks’ epoch / batch / forward tree stays intact and your custom scope slots in at the right level. Max depth: 64.

Framework hooks

Hooks install automatically by ci.profile() when the framework is importable. Each hook is wrapped in a top-level try/except, so a failing hook logs a warning and training continues. When multiple frameworks are installed, hooks fire in priority order transformers > tensorflow > torch. Higher-level frameworks claim ownership of the semantic scopes (epoch, step) first; lower-level frameworks yield, so no semantic scope is duplicated.

PyTorch uses module pre/post hooks, autograd hooks, optimizer-step hooks, and DataLoader iterator wrapping; gradient accumulation collapses to a single step span. CUDA timing via torch.cuda.Event pairs lands on the gpu_ns attribute.
Keras auto-registers a Callback on Model.fit; logged metrics become marks.
HuggingFace auto-registers a TrainerCallback on Trainer.__init__; end-of-epoch values are marked kind="summary".
scikit-learn has no auto-hook; wrap the estimator with ci.wrap.

See ci.profile for the full hook table and per-framework details.

Snapshots

At each detected epoch boundary, the SDK captures per-tensor statistics for every parameter in the model being profiled. Three modes, controlled by ci.profile(snapshots=...).

Mode	Cost per boundary	What’s captured
`"stats"`	≤ 50 ms (typical)	`{mean, std, min, max, norm, histogram[16]}` per tensor
`"sampled"`	≤ 200 ms on sampled steps	Stats + raw tensors for `random() < sample_rate` epochs
`"full"`	unbounded; debug-only	Stats + raw tensors every epoch

In "sampled" and "full" modes, raw tensors are written as safetensors blobs at ./.cirron/snapshots/<span_id>/weights.safetensors (and gradients.safetensors when gradients are non-None). Keras and HuggingFace hooks discover the model from their callback kwargs. Bare PyTorch loops that don’t use ci.epochs() should register the model once with ci.watch(model) before training.

"full" mode is not recommended for models over 100M parameters. At 7B+, even "sampled" is expensive. Drop sample_rate.

Output sinks

By default ci.profile() writes each batch as a JSON file under ./.cirron/spool/. Swap or fan-out via output=:

ci.profile(output="stdout")            # live [cirron] lines per closed span
ci.profile(output=["spool", "log"])    # disk + log
ci.profile(output="none")              # nothing local; pair with ci.trace()

Sinks are independent of the platform transport. output="none" inside a Cirron pipeline still ships traces to the platform over the kernel event stream. See ci.profile for the full table.

In-process read-back

ci.trace() returns the current session’s scope tree without leaving the process, useful in notebooks (cell renders inline) and for ad-hoc analysis (flat DataFrame for quantiles and group-bys):

ci.trace()                  # text tree
ci.trace(format="df")       # pandas DataFrame, one row per span
ci.trace(name="epoch")      # filter by scope name

Works with or without an active profiler. When no profiler is attached, the call is purely in-memory and never writes a spool file (safe on read-only filesystems).

Lifecycle

The atexit handler registered by ci.profile() flushes and shuts down on process exit. Reach for the manual helpers only when you need deterministic behavior in tests or hot-reload scenarios:

ci.flush()       # synchronously drain buffers to spool
ci.health()      # dict: enabled, drop counts, hook handles, transport
ci.shutdown()    # close root scope, flush, stop thread, clear singleton

See Lifecycle.

Inference guide

@ci.inference, LLM detection, config-driven capture.

ci.profile reference

Full signature and parameter table.

ci.trace reference

Read-back formats and filters.

​The happy path

​Distributed training

​Custom loops

​Scopes and marks

​Framework hooks

​Snapshots

​Output sinks

​In-process read-back

​Lifecycle

​Next

Inference guide

ci.profile reference

ci.trace reference

The happy path

Distributed training

Custom loops

Scopes and marks

Framework hooks

Snapshots

Output sinks

In-process read-back

Lifecycle

Next