Setting your AI-powered medical device for success, with Qity AIMS
- 01A model can perform well while the evidence explaining how it was created, tested, integrated, released, and maintained stays fragmented. That gap is what reviewers find.
- 02No single standard covers the chain, so seven are used together: ISO/IEC 42001, ISO/IEC 23894, ISO/IEC 23053, IEC 62304, ISO 14971 with ISO/TS 24971-2, IEC 62366-1, and IEC 81001-5-1.
- 03The most common weakness in AI submissions is treating the model as if it were the product. Production software can select the wrong image series, apply another normalisation, or associate output with the wrong specimen while the model evaluation stays valid.
- 04ETL is frequently treated as technical plumbing. In medical AI it can determine whether the evidence is valid at all, because the model learns from the output of the pipeline rather than the intention behind it.
- 05Validated does not mean released, and drift is a signal for investigation rather than an instruction to retrain.
The model is not the product
ISO/IEC 23053 begins before training. It starts with definition of the problem, because the task determines what the data, labels, metrics, model, and system need to achieve. "Segment tumour tissue" is a technical objective. It is not yet a complete product requirement.
The programme still needs to define the medical indication, specimen type, intended user, supported scanners, input-quality limits, output format, clinical role of the segmentation, meaningful error conditions, and the response when the system cannot produce a dependable result. Those decisions shape the dataset specification, annotation protocol, performance criteria, interface, risk analysis, and clinical evidence.
A model can perform correctly while the complete system fails. The model evaluation remains valid for the tested artefact. It does not validate the software and workflow surrounding it.
Production software may select the wrong image series, use a different resolution, apply another normalisation process, associate the output with the wrong patient or specimen, or display uncertainty in a way that encourages excessive reliance. IEC 62304 adds the software-engineering process required for that wider system, and the two lifecycles need to meet at controlled interfaces: the model input specification becomes a software requirement, preprocessing becomes a configuration item, model weights become a controlled artefact, and output handling, logging, error behaviour, and fallback enter the software architecture and risk-control design.
Traceability then has to follow the technical reality. Source data connect to the governed dataset, the dataset to its ETL and annotation evidence, its role in training or testing to a specific model version, the model to the software and system baseline, and operational evidence back to the exact configuration in use. Large datasets, repositories, weights, and logs can stay in specialised systems, provided the controlled record preserves their identity, version, location, integrity value, owner, approval state, retention, and relationships.
The lifecycle is represented through distinct AI System, AI Use Case, Dataset, AI Model, and Dataset Model Association records, so each model-data relationship identifies its purpose, version, approval, applicable protocol, evidence, and downstream system configuration without forcing the underlying technical artefacts into Jira or Confluence.
Data acquisition, ETL, and dataset quality
Data governance begins before the first file enters the training environment. The intended purpose defines what the development data need to represent; the available data cannot silently define the eventual claim after the model has been trained. A dataset containing tens of thousands of images may still offer weak evidence when those images come from a narrow technical and clinical environment.
The work starts with a data specification derived from the intended use. Compare that specification with the available sources and record the gaps. A gap can lead to further collection, a narrower claim, a technical control, a subgroup limitation, a separate acceptance criterion, or focused post-market monitoring. Provenance extends well beyond the identity of the supplying hospital: follow data through collection, transfer, pseudonymisation, transformation, quality review, annotation, assembly, correction, approval, use, and retirement.
ETL deserves separate attention. A mapping defect can connect an image to the wrong outcome without causing the training software to fail. A filtering rule can remove difficult cases and make the dataset cleaner than clinical practice. A transformation can introduce a feature correlated with the label but unrelated to the disease. So ETL needs a defined input, transformation logic, output, exception process, acceptance criteria, software version, configuration, validation method, and execution record.
Dataset splitting deserves its own analysis. Training, tuning, and test data can remain stored separately while sharing information through patients, episodes, specimens, slides, sites, preprocessing statistics, duplicates, or repeated exposure of test results to developers. Independence needs to reflect how the clinical data were generated.
Each Dataset has a governed identity, source, purpose, intended development role, version, quality status, provenance status, annotation status, limitations, applicable agreements, linked ETL evidence, approval route, and relationships to every AI Model that uses it, making unsupported or unapproved data visible before model approval.
Annotations and reference standards
The phrase "ground truth" often gives medical annotations more certainty than they possess. A reference can represent a diagnosis, disease grade, anatomical boundary, image-quality assessment, treatment response, or later clinical outcome. Some references arise from objective measurements; others depend on specialist interpretation, consensus, adjudication, or several imperfect sources combined. The development record needs to explain what the reference represents and why it is suitable for the claim.
For a pathology model, "tumour" requires a controlled clinical definition. The annotation protocol needs to state whether the target includes invasive and in situ disease, how uncertain boundaries, treatment effects, inflammation, and artefacts are handled, whether reviewers can use additional stains or clinical information, and whether the unit of analysis is the patient, specimen, slide, region, or pixel. Those decisions define the task the model learns.
Annotator competence needs to match the judgement represented by the label, and a professional title alone may not establish it for a particular indication, modality, or digital workflow. The annotation tool forms part of the evidence too, because how it presents source information, records decisions, manages versions, protects blinding, and retains the audit trail can affect the reference. Agreement analysis should reveal where reviewers differ rather than reduce the result to one favourable statistic: disagreement clustering around a grade, scanner type, or rare presentation can expose an unclear protocol or a claim that exceeds what qualified users can determine reliably.
The annotation method and protocol version are linked to the governed Dataset and carried into every Dataset Model Association, so the organisation can identify which labels supported training, tuning, or final testing, and reassess affected Models whenever the reference definition or protocol changes.
Training, model selection, and independent evaluation
Research depends on iteration. Regulated development depends on recognising when iteration has produced a candidate that can be evaluated without further influence from the final test evidence. While work is genuinely exploratory the records can stay proportionate; once an experiment begins to support a product candidate, its configuration, inputs, conclusions, and limitations enter the controlled design history.
The controlled training record needs enough detail to reconstruct the development: architecture, initial weights, pretrained components, dataset and annotation versions, input representation, preprocessing, loss functions, optimisation, hyperparameters, augmentation, sampling, class balancing, stopping criteria, code and library versions, environment, hardware, randomisation controls, period, deviations, and the resulting model identifier. Exact numerical reproduction is not always possible, but the configuration must be reconstructable and repeated execution must produce results consistent with the conclusions.
Candidate selection needs predefined criteria, because the model with the highest aggregate accuracy may not be the safest candidate. Another may provide stronger sensitivity for critical cases, more stable calibration, better performance across sites, clearer uncertainty behaviour, or simpler deployment. The clinical purpose determines which trade-offs matter.
The final evaluation needs a frozen subject: model, preprocessing, postprocessing, thresholds, code, test dataset, annotation version, statistical plan, and acceptance criteria fixed before testing begins. Changes after the result create a new candidate. The final test set cannot become a repair tool.
Every controlled AI Model has its own lifecycle, owner, architecture, version, linked Dataset roles, evaluation criteria, results, limitations, risks, approvals, and integrity reference, while the Dataset Model Association preserves exactly which Dataset version supported training, tuning, validation, robustness testing, or independent evaluation.
Governing the model and the complete AI system
A trained model needs a stable identity. Names such as "final model", "candidate 3", or "July version" become unsafe once several indications, partners, and releases exist. And validated does not mean released: a model can meet its statistical acceptance criteria while still lacking system validation, clinical approval, cybersecurity assessment, risk-control verification, release approval, or regulatory authorisation.
The complete AI system includes more than the trained artefact. It may contain input association, ingestion, quality checks, preprocessing, the model, postprocessing, threshold and confidence logic, storage, interfaces, logging, cloud or edge infrastructure, update mechanisms, and human review. One system can contain several models, and one model can be considered for several systems or use cases.
The system configuration baseline defines what users actually receive, and deployment verification confirms the production environment matches it. Development and production can differ in hardware, libraries, network behaviour, storage, scaling, access control, and supplier services, and any difference affecting input, output, timing, security, or user interaction can affect the evidence supporting the device.
Operational monitoring then separates system health from model behaviour. Availability, latency, failed jobs, and interface errors show whether the service operates. Input distributions, rejection rates, confidence patterns, site and subgroup mix, user corrections, confirmed errors, and later clinical outcomes show whether the AI remains suitable. Drift is a signal for investigation, not an instruction to retrain automatically.
The approved AI Model remains distinct from the AI System that deploys it, while links to the Release, configuration baseline, monitoring triggers, Quality Events, changes, suppliers, deployment evidence, replacement Model, and retirement decision preserve the history of the complete product rather than only the weights.
Human oversight, transparency, and use-related risk
A clinician remaining in the loop does not automatically create an effective risk control. The user needs authority to intervene, access to the information required to judge the result, enough time to act, and a usable alternative when the AI output cannot be followed safely. When any of these is absent, human oversight can exist in the product description while failing in practice.
Autonomy needs analysis at task level. A product can be assistive when displaying findings, advisory when ranking interpretations, semi-autonomous when prioritising cases, and highly autonomous when selecting acquisition parameters or executing an action. The human oversight specification defines what the AI performs, which decisions remain with the user, which outputs require confirmation, when intervention is possible, how an action can be reversed, what happens when the user does nothing, and how uncertainty or technical failure is handled.
Reliable automation changes behaviour. Repeated success can reduce independent review, a polished output can appear more authoritative than the underlying evidence, and confidence scores can create false precision when users do not understand what they represent. One visible failure can have the opposite effect. Explainability needs a purpose too, and an explanation can itself create risk when it appears more precise than it is, which makes heat maps, attention displays, and explanatory text user-interface elements requiring evaluation.
These controls sit primarily at AI Use Case level, where the user, workflow, clinical decision, autonomy, required confirmation, fallback, transparency, limitations, usability evidence, related risks, training, and monitoring can differ even when several Use Cases rely on the same Model.
Cybersecurity across the AI lifecycle
An AI-enabled medical device can generate an unsafe result because the model fails under normal conditions. It can generate the same result because the input, model, software, infrastructure, or configuration was altered. Cybersecurity is therefore part of the device safety case.
The attack surface extends well beyond the released application: the data source, transfer service, ETL pipeline, annotation platform, training environment, code repository, dependency registry, pretrained model, model registry, inference service, cloud infrastructure, monitoring system, administrative interface, deployment pipeline, and update mechanism. Threat modelling needs to follow the real system boundary.
Security and safety risk remain separate processes. Security risk considers threats, vulnerabilities, exploitability, and security consequences; safety risk follows the resulting device behaviour into hazardous situations and possible harm. The interface between the two matters, because a security control can introduce a safety risk and a safety control can create a new attack path.
Model and data integrity need explicit treatment. The production environment needs to confirm that the deployed model matches the approved artefact, and access to data, labels, training initiation, test data, model registries, production configuration, and deployment needs suitable authorisation, logging, and separation of duties.
Is one AI standard enough for an AI-enabled medical device
No. No single standard covers the chain from data acquisition to post-market monitoring. ISO/IEC 42001 provides the management system, ISO/IEC 23894 the risk method, ISO/IEC 23053 the technical lifecycle, IEC 62304 the software process, ISO 14971 and ISO/TS 24971-2 the safety risk, and IEC 62366-1 the usability engineering.
Does a validated model mean a releasable product
No. A model can meet its statistical acceptance criteria while still lacking system validation, clinical approval, cybersecurity assessment, risk-control verification, release approval, or regulatory authorisation. Lifecycle states need to preserve those distinctions.
Does keeping a clinician in the loop count as a risk control
Only when the user has authority to intervene, the information required to judge the result, enough time to act, and a usable alternative when the output cannot be followed safely. When any of those is absent, human oversight exists in the product description while failing in practice.
Should drift trigger automatic retraining
No. Drift is a signal for investigation. Determine what changed, whether the system remains within its intended use, which risks are affected, and whether the evidence supporting the current release remains valid.
Bring one AI-enabled product. We record its AI System, Use Cases, Datasets, and Models in Qity AIMS while you watch, and you leave knowing which records you already hold and which are missing.
