5D-06 - Automated Causality Assessments of Individual Case Safety Reports

Background: Causality assessment is performed for all individual case safety reports (ICSR) to support reportability requirements and may assist with signal detection. The assessment may be time consuming, especially with the increasing number of ICSRs, and can be fraught with limitations, such as a lack of precision and high inter-rater variability or lack of reproducibility.

Objectives: To develop, validate, and evaluate the performance of a fully automated causality assessment method to support post-marketing ICSR reportability.

Methods: A systematic literature review was performed and internal expertise was leveraged to develop a prioritized list of key clinically relevant variables. Each variable was given a numeric weight, consistent with existing methods in the published literature (positive if suggestive of a causal association, negative if not, and zero if no data), and an aggregate overall causality score was determined. A retrospective reference data set of medically serious ICSRs (N=150), representing a broad range of attributes was created. Each ICSR was assessed and causality determined by 2 independent pharmacovigilance experts. The final reference dataset consisted of ICSRs where both reviewers agreed on causality (N=106). Using this reference dataset, we compared performance to our current manual process. A sensitivity analysis was also performed to assess performance across a range of variables (e.g. pharma versus vaccines, different numeric weightings, etc.). Success was defined if the automated algorithm had a sensitivity >85%, was non-inferior to the current process, and was better than random.

Results: Performance of the current manual process yielded a sensitivity of 98%, a specificity of 22%, and an accuracy of 57%. Our automated approach for all ICSRs yielded a sensitivity of 94%, a specificity of 19%, and an accuracy of 53%. Sensitivity analysis showed a lower sensitivity but higher specificity and accuracy for vaccines as compared to pharma products (sensitivity of 86% versus 94%, specificity of 63% versus 21%, and an accuracy of 71% versus 57%). Additionally, an AUC ROC was generated and had a confidence interval of 0.50 to 0.63. We noted a positive correlation between performance and the amount of data contained within the ICSR.

Conclusions: Our automated approach to causality assessment met all of our predefined success criteria and performed at a level consistent with our current manual review process. Use of such an approach would save time, allow for transparency and consistency in assessment. Limitations include a lack of precision and implementation challenges of the method, as well as our relatively small reference data set that only focused on serious cases.