Skip to content

Pipelines#

A pipeline is a declarative ML workflow: a sequence (or graph) of agents that work together to produce a trained, evaluated, audited, deployed model.

Three built-in templates#

Template Agents When to use Typical runtime
fast_prototype 5 agents, 3 teams Quick experiments on a laptop 3-8 minutes
default_ml_pipeline 22 agents, 6 teams Production-quality runs 15-45 minutes
parallel_research 18 agents, parallel branches Hyperparameter sweep / algorithm bake-off 20-60 minutes

See ml_team/config/pipelines/ for the YAML source.

A pipeline is YAML#

# config/pipelines/fast_prototype.yaml
name: fast_prototype
description: Rapid iteration pipeline for small datasets + laptop-scale models.
teams:
  - management
  - data
  - algorithm
  - training
  - evaluation
agents:
  - ml_director          # orchestrates
  - data_profiler        # understands the data shape
  - algorithm_selector   # picks the model family
  - trainer              # fits the model
  - model_evaluator      # scores it
flow: sequential         # or `graph` for parallel
max_iterations: 3        # agent ReAct loop cap
evaluator_grading: true  # run clean-context grader on each agent's output

Running a pipeline#

From the CLI:

swarm pipelines run \
  --problem "Detect fraudulent credit card transactions" \
  --dataset fraud_train.csv \
  --template default_ml_pipeline \
  --compliance rbi_free_ai

From REST:

curl -X POST http://localhost:8000/api/v1/pipelines \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "problem_statement": "Detect fraudulent credit card transactions",
    "dataset_path": "fraud_train.csv",
    "template": "default_ml_pipeline",
    "compliance_profile": "rbi_free_ai"
  }'

From the dashboard: Pipelines → New.

What a pipeline produces#

Every run creates a directory at pipeline_runs/{run_id}/:

pipeline_runs/7f8e9a2b/
├── conversations/          # per-agent JSONL journals
│   ├── ml_director.jsonl
│   ├── trainer.jsonl
│   └── model_evaluator.jsonl
├── models/                 # trained model artefacts
│   └── model.joblib
├── reports/
│   ├── model_card.md
│   ├── fairness_audit.json
│   └── shap_explanation.json
├── audit/
│   └── audit_report.pdf    # regulator-format PDF with SHA-256 pin
├── checkpoints/            # batch-runner checkpoint files
├── lessons_learned.md      # post-run feedback
└── run_events.jsonl        # append-only event log

Flow: sequential vs graph#

  • Sequential (flow: sequential) — agents run one at a time, each seeing the previous agent's output. Simpler. Used by fast_prototype and default_ml_pipeline.
  • Graph (flow: graph) — agents run in parallel where dependencies allow. Requires dependencies: block per agent. Used by parallel_research (multiple trainers fan out, model_comparator fans in).

Max iterations#

Each agent runs a ReAct loop (LLM thinks → picks tool → observes result → thinks again). max_iterations caps that at N before the agent is forced to produce a terminal response. Default: 5. For a cautious/expensive agent (like trainer), you might bump to 8; for a fast one (data_profiler), cap at 2.

Compliance profile#

At pipeline start, a compliance profile (e.g. rbi_free_ai, hipaa, eu_ai_act_high_risk) can be attached. This:

  1. Injects the right guardrail agents into the pipeline (fairness_auditor for RBI, phi_redactor for HIPAA, etc.)
  2. Enforces additional permission rules (e.g. HIPAA disables the export_raw_data tool)
  3. Triggers the right audit-PDF template at the end

See Compliance profiles.

Observability during a run#

  • WebSocket: ws://localhost:8000/api/v1/pipelines/{run_id}/stream streams agent-message events live to the dashboard
  • Prometheus: /metrics exposes swarm_pipeline_* counters + histograms
  • OpenTelemetry: spans exported if OTEL_EXPORTER_OTLP_ENDPOINT set
  • Logs: per-agent JSONL journal written in real-time

Cancellation#

swarm pipelines cancel <run_id>

Sends a soft cancel. Running agent completes its current tool call; subsequent agents are skipped. Artefacts produced before cancellation are preserved. Useful for "I meant to point at the other dataset" moments.

Checkpointing#

Pipelines checkpoint automatically after each agent completes. If the API process restarts mid-pipeline, resume with:

swarm pipelines resume <run_id>

Extending: your own template#

  1. Drop a new YAML in ml_team/config/pipelines/<my_template>.yaml
  2. Reference any agents from the 40-agent catalogue (or plugin-contributed agents)
  3. Pick sequential or graph flow
  4. Run it: swarm pipelines run --template <my_template> ...

Next#