AI Governance Dashboard
Glossary — AI Governance
Lifecycle & deployment
- Portfolio — Inventory of AI models by indication, site, and version.
- Indication — Clinical problem the model targets (e.g., PE on CT).
- Operating Mode — Shadow (silent), Canary (limited exposure), Live (full use).
- Promotion/Demotion — Moving between modes per criteria; recorded in audit trail.
- Release — A new model/software version for a site with notes and validation.
- Rollback — Reverting to a previous version when guardrails are breached.
- Toggle — Enable/disable a model or alert stream without changing version.
Ground truth & adjudication
- Adjudication — Expert review creating a gold-standard label.
- Ground Truth — Adjudicated label used to score model outputs.
- Silent-Mode Gap — Difference between shadow performance and adjudicated outcomes.
Core performance metrics
- TP/FP/FN/TN — Counts from comparing model outputs to adjudicated labels.
- Sensitivity (Recall) — TP/(TP+FN); ability to detect positives.
- Specificity — TN/(TN+FP); ability to avoid false alarms.
- PPV (Precision) — TP/(TP+FP); fraction of alerts that are true positives.
- NPV — TN/(TN+FN); fraction of negatives that are true negatives.
- AUROC — Threshold-independent discrimination summary.
- Calibration Error — Gap between predicted probabilities and observed outcomes.
- Baseline — Reference performance used for comparisons (e.g., canary guardrails).
Fairness & subgroup analysis
- Subgroup — Patient slice used for equity analysis (e.g., age bands, sex).
- Fairness Gap (Δ) — Difference in a metric between subgroups (percentage points).
- Small-N Warning — Flag for low sample sizes; interpret with caution.
- Equity Threshold — Maximum allowed subgroup gap before review (e.g., ≤ 10 pp).
Drift & data quality
- Drift — Shifts after deployment: input, output, or performance drift.
- PSI — Population Stability Index; higher values suggest distribution shift.
- Score Distribution — Spread of model scores (e.g., p50, p90) to spot shifts.
- Data Quality Flags — Missing/invalid metadata that may undermine monitoring.
Workflow & burden
- Alert Rate — Alerts per 1,000 studies or per reader/shift.
- Time-to-Action — Median seconds from alert to documented action.
- Dismiss Rate — % of alerts dismissed without action (fatigue signal).
- TAT Change — Percent change in turnaround time post-deployment.
Incidents & learning system
- Incident — Event where AI contributed to potential or actual harm.
- Near-Miss — Averted harm; valuable for preventive learning.
- Severity — Minor/Major/Sentinel categorization.
- Closure Rate — % of incidents closed within the monitoring window.
Change control & guardrails
- Release Notes — What changed, why, and supporting validation.
- Guardrails — Limits that must hold in canary/live (e.g., PPV ≥ 0.85).
- Canary Criteria — Metrics/sample sizes required before promotion to live.
Audit, logging, evidence
- Audit Trail — Immutable record of changes and approvals.
- Logging Coverage — % of inferences with complete logs (target ≥ 95%).
- Input Fingerprint — Non-PHI hash enabling traceability.
- Evidence Binder — Exportable package of charts, notes, bias reports, SOPs, and logs.
- DMC View — Data Monitoring Committee lens on adjudication, fairness, incidents.
Configuration & provenance
- Benchmarks — Editable thresholds that drive “Met / Not Met.”
- Conditions — Status statements that compare KPIs to benchmarks.
- Provenance — Source of each row (manual/csv/demo) with filename/timestamp.
- Filters — Global controls (date, site, model, version, mode) applied across tabs.
- CSV Templates — Schemas for importing data to the correct panel.
How to add information to the dashboard: Start on the Start tab, then click the section you want to update (Portfolio, Performance, Fairness, Drift, Workflow, Incidents, Change Control, or Audit). At the top-right, choose Import CSV if you already have a spreadsheet; be sure your CSV’s first row uses the exact header names for that section (e.g., Performance needs columns like date, site, model_label, adjudicated_label, age_band, sex, time_to_action_s). The importer will recognize the headers and automatically place the rows in the right panel. Prefer to type directly? Scroll to Manual Input; Add Row in any section, fill in the fields, and click Add Row. Nothing is pre-loaded by default, so every number you see comes from the rows you add. If you’re doing a demo, you can enable the optional Load Sample Data button. Finally, open Benchmarks to set your targets (e.g., PPV and Sensitivity thresholds, subgroup gap limits, PSI drift limits, incident closure goals); these drive the “Met / Not Met” summaries you’ll see on each tab.
What the dashboard tells you once data is in: Each tab translates raw rows into clear, decision-ready indicators. You’ll see KPI tiles (counts, percentages, medians) and a concise Conditions box that states whether your policy targets are Met or Not Met (for example, “PPV ≥ 0.85 — Met,” “Input drift PSI < 0.20 — Not Met”). The Portfolio view shows what models are live, where, and which versions. Performance computes core accuracy measures (TP, FP, FN, PPV, Sensitivity) and time-to-action. Fairness highlights subgroup gaps (by age and sex) and flags any gap beyond your limit. Drift surfaces distribution shifts (e.g., PSI) and changes in alert rate. The workflow summarizes frontline burden (alerts per reader and median acknowledgment time) for incidents, tracks, near-misses, and closure rates. Change Control displays release notes and governance actions; Audit reports logging coverage and missing events. Use the filters (date, site, model, version, mode) to narrow the view and click through to see the underlying records. A small Data badge at the top of each section shows where the information came from (manual entry, CSV file name, or sample data), so the team can trust what they’re seeing and act with confidence.