Skip to content

Generate an audit PDF#

Goal: produce a regulator-format PDF of a pipeline run (or a date range, or a promotion). Time: 2 minutes to generate, 10 minutes to read.

swarm produces three kinds of audit PDF:

Kind When it's produced Scope
Run audit At end of a pipeline run (if compliance profile enabled) Everything that happened in one run
Promotion audit At model promotion Shadow evidence + comparison + approval
Rollup audit On-demand or via cron All runs in a date range + aggregate metrics

Run audit — automatic#

If you ran with --compliance rbi_free_ai (or HIPAA / EU AI Act), the audit PDF is produced at the end. Find it at:

pipeline_runs/<run_id>/audit/audit_report.pdf
pipeline_runs/<run_id>/audit/audit_report.sig     # manifest + SHA

Preview:

open pipeline_runs/7f8e9a2b/audit/audit_report.pdf

What's inside#

A 10-25 page PDF containing:

  1. Cover — run id, timestamp, model name, compliance profile, SHA pin
  2. Executive summary — 1 paragraph, problem statement, final model, verdict
  3. Model card — algorithm, parameters, training data hash, performance
  4. Data governance — profile of training data, features used, PII/PHI handling
  5. Fairness audit — demographic parity, equalized odds on protected attributes
  6. Explainability — global + local SHAP feature importances
  7. Drift baseline — feature distributions pinned for future comparison
  8. Agent trail — every agent's major decisions, tools called, approvals granted
  9. Permission denials — anything that was blocked + why
  10. Tamper-evident manifest — SHA-256 of every included artefact

See Reading the audit PDF for a section-by-section walkthrough.

Run audit — manual (for a run without compliance profile)#

If you forgot to set --compliance at pipeline-start time:

swarm audit generate \
  --run-id 7f8e9a2b \
  --profile rbi_free_ai \
  --output-dir pipeline_runs/7f8e9a2b/audit/

Caveat: if the run didn't produce certain required artefacts (fairness audit, SHAP), the audit generator runs them retroactively. This adds a few minutes and requires the original model + data to still be available.

Promotion audit#

Produced automatically on model promotion. See Deploy a model for the flow.

On-demand:

swarm deployments audit \
  --model fraud \
  --from-version v1 \
  --to-version v2

Goes into pipeline_runs/<promotion_run_id>/audit/promotion_audit.pdf.

Rollup audit#

For a date range:

swarm audit rollup \
  --model fraud \
  --from 2026-04-01 \
  --to 2026-04-15 \
  --profile rbi_free_ai \
  --output fraud_apr_first_half.pdf

Contents: - Every run in the window, summarized (1 row each) - Drift trend across the window - Fairness metric trend - Every promotion + approver - Every alert + resolution - Tamper-evident manifest for the aggregated bundle

Scheduled rollups#

Cron task kind audit_pdf:

swarm cron create \
  --name "fraud_monthly_audit" \
  --schedule "0 8 1 * *" \
  --task audit_pdf \
  --config '{
    "model": "fraud",
    "profile": "rbi_free_ai",
    "window_days": 30,
    "email": "compliance@yourorg.com"
  }'

Fires 1st of each month at 08:00. Emails a link (or attached PDF if <10MB).

Verifying tamper-evidence#

Every audit PDF ships with a .sig companion:

swarm audit verify pipeline_runs/7f8e9a2b/audit/audit_report.pdf

Verifying pipeline_runs/7f8e9a2b/audit/audit_report.pdf...

  Manifest found: audit_report.sig
  PDF SHA-256:    ok  (matches manifest)
  model.joblib:   ok  (matches manifest)
  model_card.md:  ok  (matches manifest)
  fairness.json:  ok  (matches manifest)
  shap.json:      ok  (matches manifest)
  run_events.jsonl: ok  (matches manifest)

Verdict: UNTAMPERED.
Audited at 2026-04-15T14:35:02+05:30 by run_id=7f8e9a2b, swarm=v0.11.0.

If any file has been modified since the audit manifest was computed, verify fails with the specific mismatch. That's what auditors want — evidence the bundle hasn't been edited post-hoc.

Custom audit template (BFSI internal governance)#

The RBI profile ships with a template that looks like the regulator expects. For internal audit / model risk committees, customize:

cp ml_team/tools/audit_templates/rbi_free_ai.yaml \
   ml_team/tools/audit_templates/acme_bank_internal.yaml
# Edit sections, branding, required artefacts

swarm audit generate \
  --run-id 7f8e9a2b \
  --template acme_bank_internal \
  --output acme_internal.pdf

See ml_team/tools/audit_pdf.py for the template DSL.

Exporting evidence bundles#

Some auditors want the full artefact set, not just the PDF:

swarm audit bundle \
  --run-id 7f8e9a2b \
  --output fraud_v2_evidence.tar.gz

Contents: - audit_report.pdf - audit_report.sig - Every source artefact (model, model card, fairness, SHAP, logs) - MANIFEST.txt with SHA-256 of each - README.md explaining what each file is

Tamper-evident via the manifest.

Retention#

Audit PDFs are retained by the retention daemon according to policy: - Default: 7 years (RBI BFSI norm) - HIPAA: 6 years (per § 164.316) - EU AI Act: 10 years (Art. 12) - Override: SWARM_RETENTION_AUDIT_PDF_DAYS env var

Retained audit PDFs are in pipeline_runs/<run_id>/audit/ until rotation. After rotation, moved to audit_archive/ then to configured long-term storage (S3, GCS, Azure Blob via storage_backend).

Next#