cirron test

Run comprehensive tests for your ML projects, including environment validation, model testing, and data pipeline verification.

Usage

cirron test [options]

Options

Option	Description	Default
`--env`	Test environment setup (Python, CUDA, dependencies)	`false`
`--build`	Test Docker container build process	`false`
`--requirements`	Validate Python requirements.txt	`false`
`--unit`	Run unit tests (pytest/unittest)	`false`
`--lint`	Code quality checks	`false`
`--model`	Test model loading and instantiation	`false`
`--data`	Test data loading functionality	`false`
`--inference`	Test model inference pipeline	`false`
`-v, --val`	Run validation tests on model accuracy	`false`
`-p, --path <path>`	Path to validation data (used with —val)	`auto-detect`
`-e, --endpoint <url>`	Test deployed endpoint for performance	-
`--pipeline`	End-to-end ML pipeline testing	`false`
`-w, --watch`	Watch mode for continuous testing	`false`
`--json`	Output results in JSON format	`false`
`--strict`	Fail fast on any errors (useful for CI)	`false`
`-i, --interactive`	Smart test selection with presets and custom options	`false`

Test Types

Environment Tests

# Test environment compatibility
cirron test --env

Tests:

Python version compatibility (checks against pythonVersion in cirron.yaml)
CUDA availability for GPU projects (if gpuRequired: true)
Framework-specific GPU checks (PyTorch/TensorFlow)
Virtual environment detection

Build Tests

# Test Docker build process
cirron test --build

Tests:

Dockerfile existence
Docker build success with test image
Test image cleanup

Requirements Tests

# Test requirements.txt validation
cirron test --requirements

Tests:

requirements.txt existence
Dependency conflict detection via pip check
Package resolution via dry-run installation

Unit Tests

# Run unit tests
cirron test --unit

Tests:

pytest (preferred) or unittest discovery
Test directory existence
Test execution

Lint Tests

# Run code quality checks
cirron test --lint

Tests:

flake8 (preferred) or pylint execution
Code style validation
Source directory checks

Model Tests

# Test model loading and instantiation
cirron test --model

Tests:

src/model.py file existence
create_model() function import and execution
Model method validation (fit, predict, or forward)
Framework-specific functionality

Data Tests

# Test data loading and validation
cirron test --data

Tests:

src/data_loader.py file existence
Sample data availability in data/sample/
Data loading functionality with sample data
.cirronignore filtering

Inference Tests

# Test inference pipeline
cirron test --inference

Tests:

src/inference.py file existence
ModelInference class instantiation
Model loading from models/model.joblib
Prediction functionality with real or dummy data

Validation Tests

# Test model accuracy on validation data
cirron test --val -p data/validation/

Tests:

Model accuracy on validation data
Performance metrics (accuracy, MSE, MAE)
Inference latency and throughput
Supports CSV files and automatic data path detection

Endpoint Tests

# Test deployed endpoint performance
cirron test --endpoint https://your-endpoint.com/predict

Tests:

Endpoint availability and response
Performance metrics (latency, throughput, success rate)
Multiple request testing (10 requests by default)
Error handling and timeout detection

Pipeline Tests

# End-to-end ML pipeline testing
cirron test --pipeline

Tests:

Complete pipeline execution in sequence
Environment → Data Loading → Model Creation → Inference → Validation
Step-by-step timing and success tracking
Comprehensive pipeline validation

Default Behavior

When no specific tests are specified, runs:

Environment tests
Requirements tests
Unit tests
Model tests
Data tests

Examples

Basic Testing

# Run default tests (env, requirements, unit, model, data)
cirron test

# Test specific component
cirron test --model

# Test multiple components
cirron test --unit --lint

Advanced Testing

# Test build process
cirron test --build

# Test inference pipeline
cirron test --inference

# Test validation with specific data path
cirron test --val -p data/test/

# Test endpoint performance
cirron test --endpoint https://api.example.com/predict

# Test complete pipeline
cirron test --pipeline

# Watch mode for development
cirron test --watch

Test Configuration

Tests are automatically configured based on your project structure. The CLI looks for:

Model tests: src/model.py (requires create_model() function)
Data tests: src/data_loader.py
Inference tests: src/inference.py (requires ModelInference class)
Unit tests: tests/ directory
Requirements: requirements.txt
Build: Dockerfile

Framework-specific tests are automatically detected from your cirron.yaml:

{
  "name": "my-model",
  "version": "1.0.0",
  "framework": "pytorch",
  "pythonVersion": "3.9",
  "gpuRequired": false,
  "test": {
    "dataPaths": {
      "validation": "data/validation/",
      "sample": "data/sample/"
    },
    "fallbackToDummy": true
  }
}

Test Output

Success Output

🧪 Test Results

✓ Environment tests
✓ Requirements tests
✓ Unit tests
✓ Model tests
✓ Data tests

All 5 test suites passed! 🎉

Failure Output

🧪 Test Results

✓ Environment tests
✗ Requirements tests
   requirements.txt not found
✓ Unit tests
✗ Model tests
   Model file not found
✓ Data tests

2 of 5 test suites failed

Watch Mode

Use --watch for development with automatic test re-runs:

cirron test --watch

Watches for changes in:

src/**/*.py
tests/**/*.py
cirron.yaml

Automatically re-runs a subset of tests when files change:

Unit tests
Model tests
Data tests
Lint tests

How It Works

Test Execution

Tests run sequentially with individual error handling
Each test shows progress with spinner
Failed tests don’t stop execution of remaining tests
Results summary shows pass/fail status for each test

Validation Data Detection

Uses -p path if provided
Falls back to cirron.yaml test configuration
Checks common paths: data/validation/, data/val/, data/test/, data/sample/
Supports both single files and directories
Applies .cirronignore filtering

Model Training Integration

Automatically trains model if no trained model exists
Uses train.py with Trainer class for model training
Saves model to models/model.joblib
Supports fallback to dummy data for inference tests

Performance Testing

Endpoint tests measure latency, throughput, and success rate
Validation tests measure accuracy and inference performance
Pipeline tests track timing for each step
All tests have appropriate timeouts

Test Workflows

Development Workflow

# Quick test during development
cirron test --unit --model

# Watch mode for continuous testing
cirron test --watch

CI/CD Workflow

# Run all tests in CI
cirron test

# Test specific components
cirron test --build --requirements

Production Validation

# Validate before deployment
cirron test --env --build --model --inference

# Test with validation data
cirron test --val -p data/validation/

# Test endpoint performance
cirron test --endpoint https://production-api.com/predict

Framework-Specific Tests

The CLI automatically runs framework-specific tests based on your project configuration:

PyTorch Tests

Tests torch.cuda.is_available() if GPU is required
Tests model forward pass with dummy data
Validates model structure and methods

TensorFlow Tests

Tests GPU device availability if required
Tests model prediction with dummy data
Validates model methods and interface

Scikit-Learn Tests

Tests model has required methods (fit, predict)
Validates model interface and functionality

Test Dependencies

The CLI automatically detects and uses available testing tools:

pytest (preferred) or unittest for unit tests
flake8 (preferred) or pylint for code quality
Docker for build tests
pip for requirements validation
requests for endpoint testing
pandas for data validation

Troubleshooting

Missing Files

# Error: Model file not found
# Solution: Ensure src/model.py exists with create_model() function

# Error: Data loader file not found  
# Solution: Ensure src/data_loader.py exists

# Error: Inference file not found
# Solution: Ensure src/inference.py exists with ModelInference class

# Error: Tests directory not found
# Solution: Create tests/ directory with test files

Python Version Issues

# Error: Python version check failed
python3 --version

# Ensure Python version matches cirron.yaml pythonVersion

GPU/CUDA Issues

# Error: CUDA not available but required
nvidia-smi

# Check if gpuRequired is set correctly in cirron.yaml

Docker Issues

# Error: Docker build failed
docker --version

# Ensure Docker is running and accessible

Linting Issues

# Error: No linter found
pip install flake8

# Or install pylint as alternative
pip install pylint

Validation Data Issues

# Error: No validation data found
# Solution: Provide path with -p option or configure in cirron.yaml

# Error: Validation data is empty
# Solution: Ensure CSV files contain data and aren't filtered by .cirronignore

Best Practices

1. Project Structure

my-project/
├── src/
│   ├── model.py          # Required: create_model() function
│   ├── data_loader.py    # Required for data tests
│   ├── inference.py      # Required: ModelInference class
│   └── train.py          # Required: Trainer class
├── tests/
│   └── test_*.py         # Unit tests
├── data/
│   ├── sample/           # Sample data for testing
│   └── validation/       # Validation data
├── models/               # Trained models
├── requirements.txt      # Required for requirements tests
├── Dockerfile           # Required for build tests
└── cirron.yaml          # Project configuration

2. Test Development

# Use watch mode during development
cirron test --watch

# Test specific components as you work
cirron test --unit --model

# Run full test suite before commits
cirron test

3. Framework-Specific Setup

# Ensure model.py has create_model() function
# Ensure inference.py has ModelInference class
# Ensure train.py has Trainer class
# Ensure data_loader.py can load sample data

4. CI/CD Integration

# Run all tests in CI
cirron test

# Test critical components in staging
cirron test --env --build --model

# Test with validation data
cirron test --val -p data/validation/

Cirron CLI

Commands

Utils

Templates

Documentation Index

​cirron test

​Usage

​Options

​Test Types

​Environment Tests

​Build Tests

​Requirements Tests

​Unit Tests

​Lint Tests

​Model Tests

​Data Tests

​Inference Tests

​Validation Tests

​Endpoint Tests

​Pipeline Tests

​Default Behavior

​Examples

​Basic Testing

​Advanced Testing

​Test Configuration

​Test Output

​Success Output

​Failure Output

​Watch Mode

​How It Works

​Test Execution

​Validation Data Detection

​Model Training Integration

​Performance Testing

​Test Workflows

​Development Workflow

​CI/CD Workflow

​Production Validation

​Framework-Specific Tests

​PyTorch Tests

​TensorFlow Tests

​Scikit-Learn Tests

​Test Dependencies

​Troubleshooting

​Missing Files

​Python Version Issues

​GPU/CUDA Issues

​Docker Issues

​Linting Issues

​Validation Data Issues

​Best Practices

​1. Project Structure

​2. Test Development

​3. Framework-Specific Setup

​4. CI/CD Integration

cirron test

Usage

Options

Test Types

Environment Tests

Build Tests

Requirements Tests

Unit Tests

Lint Tests

Model Tests

Data Tests

Inference Tests

Validation Tests

Endpoint Tests

Pipeline Tests

Default Behavior

Examples

Basic Testing

Advanced Testing

Test Configuration

Test Output

Success Output

Failure Output

Watch Mode

How It Works

Test Execution

Validation Data Detection

Model Training Integration

Performance Testing

Test Workflows

Development Workflow

CI/CD Workflow

Production Validation

Framework-Specific Tests

PyTorch Tests

TensorFlow Tests

Scikit-Learn Tests

Test Dependencies

Troubleshooting

Missing Files

Python Version Issues

GPU/CUDA Issues

Docker Issues

Linting Issues

Validation Data Issues

Best Practices

1. Project Structure

2. Test Development

3. Framework-Specific Setup

4. CI/CD Integration