Computer-Assisted Detection (CADe) Device Submissions: Reader Studies

The FDA has defined CADe devices as “computerized systems that incorporate pattern recognition and data analysis capabilities.”

In January 2020 the FDA updated its guidance for industry on CADe devices. The new guidance provides insight on compliance with special controls which have been applied to a wider range of systems “intended to identify, mark, highlight, or in any other manner direct attention to portions of an image, or aspects of radiology device data, that may reveal abnormalities during interpretation of patient radiology images or patient radiology device data” by the clinician.  With the new guidance, more CADe devices will be required to follow the more rigorous 510(k) pathway.

Regulatory approval of CADe devices require demonstration of clinical benefits and advances from existing imaging technologies, which generally necessitate reader studies.

The FDA assesses the effectiveness of imaging devices by evaluating (1) study designs for these devices and (2) the statistical significance of data collected from these studies. The sample size of the study should be large enough such that the study has adequate power to detect with statistical significance your proposed performance claims. When you make multiple performance claims, a pre-specified statistical adjustment for the testing of multiple subsets would be necessary.

Reader studies involve a large number of patients, readers and readers’ interpolations. The objective of the study is to examine how well the technology is incorporated into clinical practice based on image interpretation of a group of readers.

A reader study requires an accurate design, with special care to addressing bias on the one hand and maximizing the likelihood of success on the other hand. 

Although bias cannot be eliminated completely, a good study design would minimize the potential sources of bias and expected unavoidable bias should be described in study reporting and used to provide correct interpretation of results.

Some tips for initiating multiple-reader studies:

• Study Design: You should randomize readers, cases, and reading scenarios to reduce bias in performance measures.
• Endpoint Definitions: The endpoint chosen should be carefully selected in consultation with a statistician in order to maximize chances of success while taking into consideration your target label and overall Company strategy.
• Case Selection: Cases collected for your clinical trial should be independent of the cases used during your device development. Enrichment of diseased or abnormal cases is permissible along with the knowledge that it may affect the reader performance.
• Readers Selection: Choosing the right readers to review the cases and the right number of readers should be based on precedent and statistical power.

How many readers are needed in clinical studies of medical imaging?

The sample size should be sufficient to demonstrate performance claims. Simulations examining a wide range of effect sizes are recommended to determine the robustness of the sample size estimate. The required sample size will depend (beyond the technology performance) on the endpoints selected for the confirmatory analyses — whether lesion-based, patient-based, or based on another unit of measure and whether or not the sample is enriched.

Study populations (both diseased and normal cases) are appropriately representative of or sampled from the intended use population.

The selection of lesion-based, patient-based, or another unit-based measure of performance as a primary or secondary endpoint will depend on the intended use and the expected impact of the device on clinical practice.