Guidelines for the Performance Evaluation of Pathological Image AI Analysis Software: Ensuring Accuracy and Compliance in Medical Diagnostics
The "Key Points for the Performance Evaluation Review of Pathological Image Artificial Intelligence Analysis Software" provides a comprehensive set of guidelines aimed at guiding applicants in preparing and writing the non-clinical evaluation section of their registration materials for pathological image artificial intelligence analysis software. This document also serves as a reference for technical review departments, with a focus on software research material requirements, including demand specification, algorithm research data, and more.
Demand specifications consider requirements for data collection, algorithm performance, and usage restrictions. Data collection must take into account the compliance, sufficiency, and diversity of data sources, the scientific and rational distribution of data, and the adequacy, effectiveness, and accuracy of data quality control. Data must come from different regions and no less than three institutions; these institutions must use the slide preparation process, tissue staining, and immunohistochemistry techniques described in the manual. Algorithm performance must consider the product's intended use, comprehensively evaluating analysis speed, sensitivity, specificity, repeatability and reproducibility, and generalizability of performance indicators, as well as factors affecting algorithm performance such as gradient vanishing, gradient explosion, overfitting, and underfitting. Usage restrictions should consider scenarios where the product is prohibited or used with caution, accurately describing product usage scenarios and providing necessary warning information.
Algorithm research material must include a report for each artificial intelligence algorithm or combination of algorithms, covering basic information, risk management, demand specifications, data collection, algorithm training, performance evaluation, and traceability analysis.
The safety level of such software is classified as severe. Data collection must include a compliance statement for data sources, listing the name, location, data collection volume, and ethical approval number of the institutions. Data collection must provide a standardized operation document, including the data collection plan and standard operating procedures; it is mainly carried out by clinical institutions, with the process involving numbering and encrypting sample data according to included rules.
Data organization must clearly define the data cleaning/preprocessing procedures and briefly describe the software used in data processing, submitting research materials for each software. Data annotation must specify the qualifications and training content of annotators and arbitrators, who should be pathologists, with data being annotated by no fewer than two people; a certain proportion of data can be selected for quality assessment by non-annotators. Dataset construction must clearly define the method and basis for dividing each dataset, ensuring that the samples in the training, tuning, and test sets have no overlap and are verified through duplication checks.
Algorithm training must consider factors such as the target population, data source institutions, collection equipment, and sample types, providing a distribution of disease composition data for the training and tuning sets (if available). Training and tuning based on these sets must specify the evaluation metrics, training methods, objectives, and tuning methods used. Algorithm performance evaluation must be based on the test set, assessing the design of the algorithm to confirm the efficiency, sensitivity, and specificity of the software's algorithm performance, which must meet the design requirements.
TX Medical: Empowering International Manufacturers with AI Software Clinical Expertise for Market Entry into China.
For collaboration and inquiries about entering the Chinese market with your AI software solutions, please reach out to us at:
Email: contact@txcro.com