(C-130) Comparing Large Language Model Categorization of Nominal Group Technique Results to Human-Labeled Data in a Study on Juvenile Idiopathic Arthritis Treatment Change Preferences

Background: Patient preference in chronic diseases like juvenile idiopathic arthritis (JIA) are often identified using qualitative methods which require significant human labeling and time.

Objectives: To determine how effectively a large language model (LLM) could align categorization with human-labeled data from nominal group technique (NGT) sessions of adolescents with JIA and parents of patients with JIA.

Methods: We conducted NGT sessions to identify information patients want to have before making a medication change for JIA; 1 adolescent (age 14-18) and 2 parent sessions by virtual conference and 1 adolescent NGT via web-based asynchronous platform. Qualitative responses to discussion questions were collected from each participant, reviewed as a group, and then each participant selected and ranked their top 5 responses. The responses were assigned primary themes and additional themes, when needed, by 3 of the authors independently. Disagreements were reviewed together to achieve consensus. We then evaluated a LLM (Llama 3.1) to assign the same label set. A statement was considered correctly categorized if at least one of the LLM’s labels appeared in the human-assigned set, and overall accuracy was computed as the percentage of statements meeting this criterion.

Results: There were 22 adolescent participants (8 live, 14 asynchronous) and 21 parent participants. The most important themes before treatment change were medication risks, efficacy, lifestyle effects, reason, and cost. The adolescent groups had 12 human-labeled primary themes and 10 additional themes (for 38% of statements) and 11 LLM-assigned primary themes and 10 additional themes (for 47% of statements). The parent groups had 10 human-labeled primary themes and 10 additional themes (for 35% of statements) and 9 LLM-assigned primary themes and 10 additional themes (for 78% of statements). The accuracy of LLM theme assignment was 90% for the adolescent statements and 84% for the parent statements. The micro-averaged Cohen’s kappa for human and LLM agreement was 0.65 for adolescent statements and 0.59 for parent statements. Human interrater Cohen’s kappa was 0.54 for adolescents and 0.59 for parents.

Conclusions: Use of a LLM to assign themes to qualitative research statements from NGT sessions was similar to human-labeled themes, as evidenced by robust quantitative measures. The LLM achieved an accuracy of 90%/84% with a micro-averaged Cohen’s kappa of 0.65/0.59 for adolescent/parent statements respectively. The model performed best on statements with clear and explicit themes but had challenges with overlapping statements where multiple themes could apply. Future work will utilize the LLM to determine statement importance as well as analysis of other qualitative data sources.