ClassifierDriftMonitor

Classifier Drift Monitor

Liquid Biopsy жидкостная биопсия cfDNA ctDNA CTC exosomes NGS qPCR

Utility description: Classifier Drift Monitor

Classifier Drift Monitor — ML Classifier Drift Monitoring

ℹ️ Utility performs continuous monitoring of ML model stability in production according to FDA SaMD, EU AI Act and GMLP requirements:
• PSI (Population Stability Index): Quantitative assessment of input feature and output prediction distribution changes relative to baseline.
• KLD (Kullback-Leibler Divergence): Measurement of information distance between current and reference distributions.
• Mean Shift: Detection of systematic shift in model predictions.
• Graded Alerts: Separation into STABLE / WARNING / DRIFT_DETECTED for proportional response.

⚠️ IMPORTANT:
• Model drift often occurs silently and accumulates gradually.
• Absence of monitoring may lead to mass diagnostic errors before problem detection.

Usage:
ClassifierDriftMonitor.exe → demo mode (console output)
ClassifierDriftMonitor.exe input.csv output.json → evaluate your data

Input format:
ModelName,ModelVersion,MonitoringWindow,PSI_Feature_Avg,PSI_Prediction,Max_PSI_Threshold,Warning_PSI_Threshold,KLD_Score,Max_KLD_Threshold,Prediction_Mean_Shift,Max_Mean_Shift,SampleSize_Current,Min_SampleSize,Baseline_Accuracy,Current_Proxy_Accuracy

Example:
OncoClass,v3.2,W22,0.03,0.05,0.2,0.1,0.02,0.1,0.01,0.05,450,100,0.94,0.93

📍 Scope of Application (Usage Where):
• Clinical ML Systems: Continuous post-market model surveillance.
• Regulatory Compliance: Documentation of SaMD stability for FDA/EMA.
• MLOps: Automatic trigger for retraining pipeline.
• AI Audit: Confirmation of EU AI Act compliance for high-risk AI systems.

— WHY IS THIS NEEDED?
ML models are validated on historical data, but real patient population changes.
Changes in equipment, data collection protocols, or demographics cause drift.
Without systematic monitoring, model degradation is detected only after patient harm.
Automated monitoring ensures proactive AI lifecycle management.

⚠️ CRITICAL:
• PSI Thresholds: 0.1 = warning, 0.2 = action (industry standard).
• Sample Size: Drift metrics are unreliable with small samples (<100).
• Feature vs Prediction Drift: Feature drift may precede prediction drift.
• Ground Truth: Use proxy metrics for indirect assessment when labels unavailable.

Key features:
• Multi-metric drift assessment (PSI + KLD + Mean Shift)
• Three-tier alert system
• Support for monitoring without ground truth
• Report generation for regulatory authorities
• Compliance with GMLP Principle 6 (Continuous Monitoring)

Critical parameters:
• PSI Features: < 0.1 (stable), 0.1-0.2 (warning), > 0.2 (drift)
• PSI Predictions: < 0.2
• KLD Score: < Threshold
• Mean Shift: Within tolerance
• Sample Size: ≥ Minimum

💡 Usage tips:
1. Baseline: Use validation set as reference, not training set.
2. Monitoring Window: Balance window size (not too small for noise, not too large for delay).
3. Stratified Analysis: Analyze drift separately by subgroups (age, sex, sample type).
4. Automation: Integrate with CI/CD for automatic retraining trigger.
5. Documentation: Maintain log of all alerts and actions taken.

⚠️ Note: This utility is a post-market ML model surveillance tool. It does not replace periodic full revalidation but ensures timely detection of revalidation necessity.

input.csv

ModelName,ModelVersion,MonitoringWindow,PSI_Feature_Avg,PSI_Prediction,Max_PSI_Threshold,Warning_PSI_Threshold,KLD_Score,Max_KLD_Threshold,Prediction_Mean_Shift,Max_Mean_Shift,SampleSize_Current,Min_SampleSize,Baseline_Accuracy,Current_Proxy_Accuracy
OncoClassify_v3,v3.2.1,2026-W22,0.03,0.05,0.2,0.1,0.02,0.1,0.01,0.05,450,100,0.94,0.93
MCED_TissueOrigin_v2,v2.1.0,2026-W23,0.25,0.35,0.2,0.1,0.18,0.1,-0.12,0.05,380,100,0.89,0.76
VariantPathogenicity_v1,v1.4.2,2026-W23,0.08,0.06,0.2,0.1,0.04,0.1,0.02,0.05,520,100,0.91,0.90

URS & FS — User Requirements and Functional Specification

This document describes the controlled interface and behaviour of ClassifierDriftMonitor for Classifier Drift Monitor.

Domain limits URS input.csv FS IQ/OQ/PQ

Domain limits and critical parameters

Key fragments from the source description are shown below. Before production use, limits must be verified against the approved specification, registration dossier and local SOPs.

⚠️ CRITICAL:
• Sample Size: Drift metrics are unreliable with small samples (<100).
Critical parameters:
• PSI Features: < 0.1 (stable), 0.1-0.2 (warning), > 0.2 (drift)
• PSI Predictions: < 0.2
• KLD Score: < Threshold
• Sample Size: ≥ Minimum

URS — User Requirements Specification

ID	Requirement	Criticality	Acceptance criterion
URS-001	The utility shall accept an input.csv file for Classifier Drift Monitor with headers defined in the data contract.	High	The file is processed without manual header editing.
URS-002	The utility shall perform deterministic QC evaluation without machine learning and without probabilistic conformance decisions.	High	Identical input data, rule version and configuration produce reproducible results.
URS-003	The utility shall validate mandatory fields, data types, ranges, units and domain plausibility.	High	Schema, conversion and range errors are explicitly reported.
URS-004	The utility shall apply domain limits and rules from the description, approved specification, registration dossier and local SOPs.	High	Each check has PASS/WARNING/FAIL and a clear message.
URS-005	The utility shall generate output.json with machine-readable results, source values, warnings, failures and critical findings.	High	JSON is suitable for LIMS/ELN/MES integration and QA/QC review.
URS-006	The utility shall preserve traceability between batch/sample, input file, applied rules and final status.	High	Output contains identifiers, checked parameters and audit metadata.
URS-007	The documentation shall support IQ/OQ/PQ, CSV/CSA and review by internal QA or inspectors.	Medium	URS, FS, input/output contract and test scenarios are supplied with the utility.
URS-008	The utility shall be used as a QC decision-support tool and not as a substitute for approved specifications and QA/QP release decision.	Medium	Documentation states change control and limit-verification expectations.

input.csv contract

#	Field	Type	Sample	Purpose
1	`ModelName`	string / controlled vocabulary	`OncoClassify_v3`	Controlled input parameter for deterministic QC rules.
2	`ModelVersion`	string / controlled vocabulary	`v3.2.1`	Controlled input parameter for deterministic QC rules.
3	`MonitoringWindow`	string / controlled vocabulary	`2026-W22`	Controlled input parameter for deterministic QC rules.
4	`PSI_Feature_Avg`	decimal	`0.03`	Controlled input parameter for deterministic QC rules.
5	`PSI_Prediction`	integer / decimal	`0.05`	Controlled input parameter for deterministic QC rules.
6	`Max_PSI_Threshold`	decimal	`0.2`	Controlled input parameter for deterministic QC rules.
7	`Warning_PSI_Threshold`	decimal	`0.1`	Controlled input parameter for deterministic QC rules.
8	`KLD_Score`	decimal	`0.02`	Controlled input parameter for deterministic QC rules.
9	`Max_KLD_Threshold`	decimal	`0.1`	Controlled input parameter for deterministic QC rules.
10	`Prediction_Mean_Shift`	decimal	`0.01`	Controlled input parameter for deterministic QC rules.
11	`Max_Mean_Shift`	decimal	`0.05`	Controlled input parameter for deterministic QC rules.
12	`SampleSize_Current`	string / controlled vocabulary	`450`	Sample or laboratory specimen identifier.
13	`Min_SampleSize`	string / controlled vocabulary	`100`	Sample or laboratory specimen identifier.
14	`Baseline_Accuracy`	decimal	`0.94`	Controlled input parameter for deterministic QC rules.
15	`Current_Proxy_Accuracy`	decimal	`0.93`	Controlled input parameter for deterministic QC rules.

ModelName,ModelVersion,MonitoringWindow,PSI_Feature_Avg,PSI_Prediction,Max_PSI_Threshold,Warning_PSI_Threshold,KLD_Score,Max_KLD_Threshold,Prediction_Mean_Shift,Max_Mean_Shift,SampleSize_Current,Min_SampleSize,Baseline_Accuracy,Current_Proxy_Accuracy
OncoClassify_v3,v3.2.1,2026-W22,0.03,0.05,0.2,0.1,0.02,0.1,0.01,0.05,450,100,0.94,0.93
MCED_TissueOrigin_v2,v2.1.0,2026-W23,0.25,0.35,0.2,0.1,0.18,0.1,-0.12,0.05,380,100,0.89,0.76
VariantPathogenicity_v1,v1.4.2,2026-W23,0.08,0.06,0.2,0.1,0.04,0.1,0.02,0.05,520,100,0.91,0.90

Input validation rules

ID	Field	Rule	Criticality
VR-001	`ModelName`	The field shall match an approved dictionary or accepted string representation.	High
VR-002	`ModelVersion`	The field shall match an approved dictionary or accepted string representation.	High
VR-003	`MonitoringWindow`	The field shall match an approved dictionary or accepted string representation.	High
VR-004	`PSI_Feature_Avg`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-005	`PSI_Prediction`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-006	`Max_PSI_Threshold`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-007	`Warning_PSI_Threshold`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-008	`KLD_Score`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-009	`Max_KLD_Threshold`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-010	`Prediction_Mean_Shift`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-011	`Max_Mean_Shift`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-012	`SampleSize_Current`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-013	`Min_SampleSize`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-014	`Baseline_Accuracy`	The field shall match an approved dictionary or accepted string representation.	Medium
VR-015	`Current_Proxy_Accuracy`	The field shall match an approved dictionary or accepted string representation.	Medium

FS — Functional Specification

ID	Function	Implementation
FS-001	CLI execution	Support execution modes: demo mode without arguments and production mode input.csv output.json.
FS-002	CSV import	Read input.csv in UTF-8/CSV-compatible format and validate header and expected columns.
FS-003	Schema validation	Check mandatory fields, column count, unknown key fields and empty mandatory values.
FS-004	Type conversion	Convert numeric, flag and text values; invalid format is recorded as a row-level error.
FS-005	Domain rule engine	Apply rules for Classifier Drift Monitor, including critical limits from the description and approved specification.
FS-006	Status aggregation	Produce final status: FAIL for critical failure, WARNING for non-critical deviation, PASS for conformance.
FS-007	JSON export	Write output.json with detailed checks, source values, warnings, failures and critical findings.
FS-008	Audit support	Keep result structure suitable for review, deviation investigation and calculation reproduction.
FS-009	Integration contract	Support the scenario LIMS/ELN/MES → input.csv → utility → output.json → portal/admin review.
FS-010	Error handling	Return explicit messages for missing file, empty CSV, invalid schema, output write failure and invalid format.

Example output.json

{
  "utilityId": "classifierdriftmonitor",
  "utilityFolder": "ClassifierDriftMonitor",
  "package": "LiquidBiopsy",
  "overallStatus": "PASS|WARNING|FAIL",
  "sourceFile": "input.csv",
  "processedAtUtc": "2026-06-10T00:00:00Z",
  "checks": [
    {
      "parameter": "ModelName",
      "value": "OncoClassify_v3",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-001"
    },
    {
      "parameter": "ModelVersion",
      "value": "v3.2.1",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-002"
    },
    {
      "parameter": "MonitoringWindow",
      "value": "2026-W22",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-003"
    },
    {
      "parameter": "PSI_Feature_Avg",
      "value": "0.03",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-004"
    },
    {
      "parameter": "PSI_Prediction",
      "value": "0.05",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-005"
    },
    {
      "parameter": "Max_PSI_Threshold",
      "value": "0.2",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-006"
    },
    {
      "parameter": "Warning_PSI_Threshold",
      "value": "0.1",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-007"
    },
    {
      "parameter": "KLD_Score",
      "value": "0.02",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-008"
    },
    {
      "parameter": "Max_KLD_Threshold",
      "value": "0.1",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-009"
    },
    {
      "parameter": "Prediction_Mean_Shift",
      "value": "0.01",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-010"
    },
    {
      "parameter": "Max_Mean_Shift",
      "value": "0.05",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-011"
    },
    {
      "parameter": "SampleSize_Current",
      "value": "450",
      "status": "PASS|WARNING|FAIL",
      "message": "Deterministic rule-based check result",
      "ruleReference": "FS-RULE-012"
    }
  ],
  "criticalFindings": [],
  "warnings": [],
  "audit": {
    "inputHash": "sha256:<calculated at runtime>",
    "rulesVersion": "<utility executable version>",
    "documentation": "ClassifierDriftMonitor.documentation.html"
  }
}

Traceability matrix

URS	FS	Test	Evidence
URS-001	FS-001, FS-002	OQ-001	Verify execution and import of valid input.csv.
URS-002	FS-005, FS-006	OQ-004	Repeat the same dataset and compare output.json.
URS-003	FS-003, FS-004, FS-010	OQ-002, OQ-003	Verify missing columns and invalid types.
URS-004	FS-005, FS-006	OQ-004, PQ-001	Verify critical deviations on real/boundary data.
URS-005	FS-007, FS-009	OQ-005	Verify JSON schema and downstream-system suitability.
URS-006	FS-008	OQ-006	Verify identifiers and audit metadata.
URS-007	FS-008, FS-010	IQ-001, OQ-007	Verify documentation completeness and control evidence.
URS-008	FS-005, FS-008	PQ-002	Verify review workflow and no replacement of QA decision.

IQ/OQ/PQ test scenarios

ID	Scenario	Expected result
IQ-001	Verify executable, input.csv, documentation and checksum availability.	Delivery set is complete; version is recorded.
OQ-001	Valid sample row from input.csv.	PASS or acceptable WARNING according to rules.
OQ-002	Remove a mandatory CSV column.	Schema error or FAIL with missing-column reference.
OQ-003	Place a non-numeric value into a numeric field.	Type-conversion error with row/field reference.
OQ-004	Set a critical parameter outside the limit.	FAIL and critical finding.
OQ-005	Verify output.json structure.	All mandatory sections are present and JSON is valid.
OQ-006	Verify batch/sample traceability.	Input and result identifiers match.
PQ-001	Verify 3–5 real user batches/samples.	Result is confirmed by QC/QA review.
PQ-002	Verify deviation workflow and manual QA decision.	Utility supports review but does not replace approved decision.

QA/QC and change control

Do not rename columns without updating validator, documentation and test set.
Retain input.csv, output.json, executable version and checksum.
Before production use, perform IQ/OQ/PQ or equivalent CSV/CSA verification.
Critical limits shall be verified against the approved specification, registration dossier and local SOPs.
The utility provides structured QC decision support; final release decision remains with QA/QP and approved procedures.

Included in packages

Liquid Biopsy QC Suite

QC and pre-analytical control package for liquid biopsy workflows: cfDNA/ctDNA, CTC, EV/exosomes, methylation, NGS/qPCR/ddPCR, sample quality, contamination, sensitivity and reporting checks.

Open