ABSTRACT

In deductive content analysis, text data is categorized into previously developed themes. We present a three-GenAI consensus model for deductive analysis of text data. In the model, the same prompt and data are provided to three GenAIs, and their agreement with each other is used to cluster data to facilitate deductive coding by researchers. First, data is split into trial and corpus sets. Trial data is used to iteratively revise the prompt provided to the three GenAIs. Next, the revised prompt is provided to the three GenAIs to deductively code the larger corpus data. Based on the responses of the three GenAIs, data is clustered into three groups: all three GenAIs agree, only 2 GenAIs agree, and all three GenAIs disagree. The data in the three clusters is separately coded by researchers, with the GenAI coding in each cluster used to narrow coding choices for the data in that cluster. To illustrate the use of the three-GenAI consensus model, we used it to deductively analyze 496 student responses to a self-directedness survey that had been previously coded by researchers. We calculated that the researchers would have saved nearly 70% of coding time if they had used the three-GenAI consensus model first. Deductive analysis of text data is labor-intensive, involving multiple researchers and multiple iterations. The three-GenAI consensus model enables researchers to redistribute time from exhaustively analyzing all data to closely reviewing only the data that does not clearly fit any of the themes of deductive analysis.

Keywords

1. INTRODUCTION

Qualitative content analysis is used to explore meanings, themes, and patterns in text, allowing researchers to understand social reality in a subjective, but scientific manner [25]. Research questions in qualitative investigations aim to probe the “what”, “why”, “who”, “when”, “where”, and “how” of social interactions or human behavior, rather than quantifying occurrences or measurements to test hypotheses [10, 14]. Qualitative content analysis can be inductive or deductive [2]. In inductive (bottom-up) methods, researchers identify and describe themes that emerge during an iterative process of analyzing bodies of text. In deductive (top-down) methods, researchers use previously developed sets of themes or models or theories as a coding scheme/book to analyze large bodies of text, attempting to fit the data into existing themes. Deductive coding can be binary (whether a theme exists or not) or multi-level (identify one of several themes).

Qualitative analysis techniques are iterative in nature, involving multiple passes through the data to build a deeper, more refined understanding of the subject matter [19]. For example, in deductive coding, data analysis can result in the refinement of existing themes as well as the identification of new themes, which can lead to the recoding of already coded data. For large data sets, this iterative process is labor-intensive [20], and involves the time-consuming work of abstraction and interpretation [11].

Research teams doing qualitative data analysis must collaboratively review notes and do close readings of text [17]. Many of the software tools available for qualitative data analysis (such as ATLAS, Dedoose, NVivo) assist with data organization only and not with the data interpretation done by the research team. Furthermore, they are based on a subscription model that may not be within reach for all researchers. With the recent advent of powerful generative AI tools (GenAI) that can quickly find patterns in large bodies of text, researchers now have a new tool to conduct qualitative analysis. A drawback of GenAI is that they are prone to hallucinate, which reduces the reliability of their responses.

In order to reliably use GenAIs for deductive analysis, we propose a three-GenAI consensus model that uses three GenAIs instead of one, uses the consensus in their responses instead of raw responses, and uses the consensus to narrow choices for deductive coding by researchers. We present the savings in time realized using this model and present a case study to illustrate its use and highlight its savings.

2. RELATED WORK

Qualitative analysis has been used extensively in computing education research. It has been used to identify themes in code interviews [9], student reflection papers [3], survey responses [6], online forum discussions [15], transcripts of think-aloud interviews [16, 23] and semi-structured interviews [8], among others. Typically, qualitative analysis involves processing text data, and involves multiple researchers engaged in a labor-intensive iterative process. As a result, qualitative analysis is hard to scale to larger data sets because of the time, cost and complexity involved.

Machine learning (ML) techniques have been used to process educational data (e.g., see a systematic literature review in [5]). Techniques used to analyze text data include latent semantic analysis [22], bag-of-words [7], etc. GenAI improves upon these traditional ML techniques in that it does not require significant training.

Given that GenAI has been publicly available for just over three years, the use of GenAI for qualitative coding is new. GenAIs have been used for both inductive and deductive coding. In one study, ChatGPT was used to help in codebook development during an inductive-deductive process [21]. In another study, ChatGPT-4 was found to be valuable for inductively developing codebooks, but human involvement in the process was deemed essential [1].

For deductive coding, prompts based on codebook were found to be more effective than those based on examples - combining GenAI (GPT-3) with expert-drafted codebooks achieved fair to substantial agreement with expert-coded results [24]. In another study, codebooks designed for researchers were found to need iterative refining for use with GenAIs [4]. For binary deductive coding, zero-shot prompting, i.e., not providing examples in the prompt, yielded good results for well-defined constructs, but was not as effective when the constructs were context-dependent [12, 13].

We propose a model to use three GenAIs for deductive coding. In the model, we propose to use a prompt based on the codebook designed for researchers. We use it in zero-shot mode, although this is not required for the model to work. The model narrows coding choices and helps researchers redistribute time from exhaustively analyzing all data to closely reviewing only the data that does not clearly fit any of the themes of deductive analysis.

3. THE THREE-GENAI CONSENSUSMODEL

In deductive qualitative analysis or coding, researchers use a previously developed set of themes, called a codebook, to identify the theme in each data item. In this text, we will refer to a data item as a statement, and a theme as a category.

We propose to use three GenAIs to aid in deductive multi-level coding, in what we refer to as the three-GenAI consensus model, as follows:

1. Shuffle the data and split it into a trial set (10-15% of the data) and corpus.
2. Draft a codebook-based prompt. Use an iterative cycle of deductively coding trial data using the three GenAIs, followed by revising the prompt provided to them, until the deductive coding of trial data is satisfactory in terms of inter-rater reliability with researchers.
3. Use the revised prompt to have the three GenAIs code the statements in the corpus data.

For both trial and corpus data, the consensus among the three GenAIs is used to split the data into three clusters:

• Unanimous, the set of statements that all three GenAIs code/categorize the same;
• 2-agree, the set of statements that exactly two GenAIs categorize the same - this excludes the statements in unanimous cluster;
• 3-disagree, the set of statements that all three GenAIs categorize differently.

The three clusters are used by researchers for deductive coding as follows:

• Unanimous: Researchers first conduct binary coding of each statement to check whether it belongs in the category identified unanimously by all three GenAIs. If not, then and only then do they consider the remaining categories for coding the statement;
• 2-agree: Researchers first consider the two categories identified by the three GenAIs: one by two concurring GenAIs and the other by the dissenting GenAI. If neither category applies to the statement, they consider the remaining categories for coding the statement.
• 3-disagree: Researchers first consider the three categories identified by the three GenAIs and consider the rest of the categories only if these three do not apply to the statement.

The utility of this model depends on whether the correct category of a statement is among the one/two/three categories identified by the three GenAIs in the three clusters.

In the three-GenAI consensus model, note that:

1. Three GenAIs are used instead of one to reduce the effect of hallucinations on the outcome.
2. The consensus in the responses of the three GenAIs is leveraged.
3. The clusters obtained based on the degree of consensus among the three GenAIs are used to reduce the number of categories researchers first consider when coding statements.
4. The final categorization of statements is still done by researchers and not GenAIs. The three-GenAI model is used to facilitate the qualitative coding done by researchers.

We will next present a case study of using the three-GenAI consensus model for multi-level deductive coding. In order to illustrate its feasibility and merit, we will use it to verify coding previously done by researchers. The case study will provide an example of using the model and its utility in facilitating deductive coding in practical settings.

4. CASE STUDY

For our case study, we used data from a study of the self-directed disposition of undergraduate computing students [18]. The study was conducted between 2022 and 2024 in five computing courses at four higher education institutions in the US. Data was collected from 264 students who consented to participate in the study. The study was approved by the institutional review boards of all four institutions (IRB-FY2022-53, IRB-FY2022-6, Protocol# 2021-08-005, and IRB-627).

In the study, students were presented the following definition of being self-directed: "Learn new tools/techniques on your own to complete a task, even when the tool/technique is only minimally used or discussed in class, and you may not receive additional credit just for learning it." They were then asked to respond to an open-ended question: "Describe an example of you being self-directed when completing this assignment. Otherwise, describe the circumstances that prevented you from being self-directed." Students were asked to respond to this question after multiple assignments in a course.

A total of 496 responses were collected and deductively coded using the codebook shown in Table 1 developed by researchers  [18]. This coding by researchers (henceforth referred to as prior coding) was used to verify the correctness of coding of the three-GenAI consensus model and for calculating the time saved by using it.

We used three GenAIs to which we had ready access: Claude Sonnet 4.5, ChatGPT 5.2 Pro and Gemini 3. The prompt statement provided to the GenAIs was structured as follows:

1. Persona: GenAI was asked to act as a "qualitative analysis expert".
2. Goal: The goal was to conduct deductive coding of statements written by undergraduate students in computing courses in response to the open-ended question listed earlier in this section.
3. Context: The definition of self-directed disposition given to students [18] was listed next.
4. Instructions: GenAI was asked to identify one of the first eight categories in the codebook (Table 1) for each statement.
5. Codebook: The entire codebook was provided. GenAI was asked to consider the categories in the order listed in the codebook, as was done by researchers during their coding.

6. Constraints: GenAI was further instructed to do the following:

   1. If a statement belonged to more than one category, identify the dominant category as determined by intent.
   2. If a statement did not contain any category, code it as "Not pertinent".
   3. If a statement explained why self-directed was not applied, code it as "Not applied".
   4. Ignore the word "self-directed" in a statement unless the statement is about not being self-directed.
7. Purpose: GenAI was told that its analysis would be used by computing education researchers to summarize how students understood self-directed disposition in undergraduate computing courses.

The prompt was zero-shot—no examples were provided for the categories—to avoid introducing potential biases based on those examples.

The 496 responses were shuffled and split into 74 trial statements (15%) and 422 corpus statements. The same prompt was provided to all three GenAIs and they were instructed to deductively code the 74 trial statements. Based on the responses of the three GenAIs, we clustered the 74 trial statements into unanimous, 2-agree and 3-disagree, as shown in Table 2:

• Unanimous cluster consisted of 52 trial statements (70.3% of trial data). The three GenAIs identified the same category as prior coding for 42 of the 52 statements (80.8%), as shown in Table 3. The inter-rater reliability between the three GenAIs and prior coding was substantial (Cohen’s kappa = 0.7508).
• 2-agree cluster contained 18 trial statements (24.3% of trial data). As shown in Table 3, the two concurring GenAIs identified the same category as prior coding for 6 statements and the dissenting GenAI identified the same category as prior coding for 5 more. In 61.1% of the 2-agree cases, the correct category was one of the two identified by the GenAIs.
• Only 4 out of 74 trial statements were in the 3-disagree cluster. For one of the 4 statements, one of the three GenAIs identified the same category as prior coding, as shown in Table 3.

Given the large percentage of trial statements in unanimous cluster (70.3%) and the substantial inter-rater reliability of coding in this group (0.7508), we chose not to iteratively improve the prompt. We provided the same prompt to the three GenAIs and instructed them to deductively code the 422 corpus statements. The cluster sizes for corpus statements are shown in Table 2.

• Unanimous cluster consisted of 289 corpus statements (68.5% of corpus data). The three GenAIs identified the same category as prior coding for 238 of the 289 statements (82.4%), as shown in Table 4. The inter-rater reliability between the three GenAIs and prior coding was substantial, with Cohen’s kappa of 0.7678.
• 2-agree cluster contained 109 corpus statements (25.8% of corpus data). As shown in Table 4, the two concurring GenAIs identified the same category as prior coding for 57 statements and the dissenting GenAI identified the same category as prior coding for 32 more. In 81.7% of the 2-agree cases, the correct category was one of the two identified by the GenAIs.
• Only 24 out of 422 corpus statements were in the 3-disagree cluster. For 20 out of the 24 statements (83.3%), one of the three GenAIs identified the same category as prior coding, as shown in Table 4.

Note that in all three corpus clusters, the correct category was one of those identified by the GenAIs for at least 80% of the statements.

For the statements for which the GenAIs did not identify the prior coding category, we conducted an additional validation step: we had two researchers independently re-identify the category, and compared their 1 - 2 (for 2 researchers) categories against prior coding category as well as the 1-3 categories identified by GenAIs in unanimous/2-agree/3-disagree clusters. The category identified by the two researchers was one of the 1-3 categories identified by GenAIs instead of the prior coding category in 52 - 61% of the statements in unanimous cluster (the range based on the two researchers’ independent coding), 74 - 89% of 2-agree cluster and 71 - 100% of 3-disagree cluster. This suggests that the statements for which the three GenAIs did not identify the prior coding category were harder to categorize for researchers too.

In the next section, we will calculate the time saved by using the three-GenAI consensus model for qualitative coding.

5. TIME SAVINGS

Assume the data set contains \(s\) statements (data items), the codebook contains \(c\) categories (themes), and during multi-level deductive coding, it takes one unit of time to consider one category for one statement. Therefore, full-fledged deductive coding of \(s\) statements takes \(s * c\) units of time.

If \(n_u\), \(n_a\) and \(n_d\) are the number of statements in unanimous, 2-agree and 3-disagree clusters, respectively:

If \(p_u\) is the percentage of statements in unanimous cluster whose GenAI categorization is correct, the time taken \(t_u\) to categorize all the statements in the cluster is

For \(p_u\) percentage of statements in unanimous cluster, researchers need to consider only one category. For the remaining \((1 - p_u)\) percentage of statements, after confirming that the category chosen by the GenAIs is incorrect, researchers must consider the remaining \((c - 1)\) categories.

Similarly, if \(p_a\) is the percentage of statements in 2-agree cluster for which the correct category is one of the two picked by the three GenAIs, the time taken \(t_a\) to categorize all the statements in the cluster is:

Finally, if \(p_d\) is the percentage of statements in 3-disagree cluster for which the correct category is one of the three picked by the three GenAIs, the time taken \(t_d\) to categorize all the statements in the cluster is:

The time needed to deductively code using the three-GenAI consensus model, as a percentage of the time needed to code without the model is:

The percentage reduction in time is given by:

The larger the number of categories \(c\), and the greater the percentages \(p_u\), \(p_a\), and \(p_d\), the larger the savings in time. Note that the savings are per researcher, and can be substantial considering that typically, three or more researchers are involved in deductive coding.

In the case study in Section 4, combining trial and corpus data, we have from Table 2:

From Table 1, we have \(c = 10\). From Tables 3 and 4, we have:

Note that \(p_u\), \(p_a\), and \(p_d\) are all large: at least 75%, one of the requirements for large savings in time. If the three-GenAI consensus model had been used for prior coding of the case study dataset, the time spent on coding could have been reduced by 69.88% per researcher involved in prior coding.

6. DISCUSSION

We proposed the three-GenAI consensus model for multi-level deductive coding of qualitative data. We used a codebook and prior coding of a dataset of 496 statements to demonstrate the use of the model for deductive coding and the savings in coding time that can be realized using the model. In the future, we plan to start with a codebook, use the three-GenAI model to cluster data into unanimous, 2-agree and 3-disagree, and then deductively code statements in the three clusters.

The percentage reduction in time (expression 1, Section 5) is a theoretical maximum. Using the model requires the following preparatory steps, all of which take additional time and effort: designing the prompt, splitting dataset into trial and corpus, using trial data to iteratively refine the prompt, using the refined prompt to deductively code corpus data, and tabulating the responses of three GenAIs into unanimous, 2-agree and 3-disagree clusters for both trial and corpus data. However, for large data sets, the time and effort needed for these preparatory steps will be largely offset by the time saved by researchers coding the data using the three-GenAI consensus model.

An advantage of the model is that researchers can reallocate some of the time saved coding statements in unanimous and 2-agree clusters to carefully code the statements in 3-disagree cluster. Statements in the 3-disagree cluster are typically the statements that are harder to code because they land on the boundary between categories. Prior research has found that the statements GenAI finds difficult to code are the same as those that researchers find difficult to code [12].

The unique features of the three-GenAI consensus model are: in order to counter the hallucinatory potential of GenAIs, three GenAIs are used instead of one; the consensus in their responses is leveraged; and the clusters obtained based on the degree of consensus are used to reduce the number of categories researchers need to consider when coding data. The final coding is still done by researchers; GenAIs ease their task so that researchers can spend more time on statements that are harder to code.

The Three-GenAI consensus model could have saved nearly 70% of deductive coding time in the case study. However, the case study involved coding statements that were responses to a single specific survey question. Therefore, the context of the data was both limited and clear, which may have played to the strengths of the three-GenAI consensus model. In contrast, deductive coding is often applied to large and ill-structured text such as interview transcripts, reflection papers, and forum discussions in which the context is large and diffuse. The applicability and benefits of using the three-GenAI consensus model to such data remain to be explored.

We considered only one category per statement. When multiple categories applied to a statement, we considered only the dominant category. If we were to consider all the categories in each statement, qualitative coding using the three-GenAI consensus model would have to consider sets of categories instead of individual categories. Unanimous, 2-agree and 3-disagree clusters would be based on set intersection, complicating the application of the model.

We used a zero-shot prompt. A significant performance improvement has been observed going from zero-shot to one-shot prompt [24]. We plan to evaluate the improvement in inter-rater reliability and time savings of the model with one-shot prompt in the future.

In another study with a different data set, we observed how the stochastic nature of GenAIs affects their behavior when used for deductive coding: when three GenAIs were instructed to code the same data set with the same prompt twice, a week apart, the inter-rater reliability (Cohen’s kappa) between the two runs was substantial, but not perfect: 0.82 for Claude Sonnet 4.5, 0.78 for ChatGPT 5 Pro and 0.74 for Gemini 2.5 Flash. Given this observation, it may be beneficial to pick the category identified by a GenAI for a statement as the best of n deductive coding runs of the GenAI.

ChatGPT offered an excellent critique of the inter-rater reliability readiness of the codebook (Table 1). Although the three-GenAI consensus model does not include codebook improvement as part of the process, we would like to enhance the model in the future by using trial data to improve the codebook before refining the prompt.

7. ACKNOWLEDGMENTS

Partial support for this work was provided by the National Science Foundation under grants DUE-2216121, 2216031, 2215970 and 2215166. The authors thank Mihaela Sabin and Stephanos Matsumoto for their contribution to prior coding.

8. REFERENCES

1. A. Barany, N. Nasiar, C. Porter, A. F. Zambrano, A. L. Andres, D. Bright, M. Shah, X. Liu, S. Gao, J. Zhang, S. Mehta, J. Choi, C. Giordano, and R. S. Baker. Chatgpt for education research: Exploring the potential of large language models for qualitative codebook development. In A. M. Olney, I.-A. Chounta, Z. Liu, O. C. Santos, and I. I. Bittencourt, editors, Artificial Intelligence in Education, pages 134–149, Cham, 2024. Springer Nature Switzerland.
2. A. J. Bingham. From data management to actionable findings: A five-phase process of qualitative data analysis. International Journal of Qualitative Methods, 22:1–11, 2023.
3. M. Celepkolu and K. E. Boyer. Thematic analysis of students’ reflections on pair programming in cs1. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education, SIGCSE ’18, page 771–776, New York, NY, USA, 2018. Association for Computing Machinery.
4. Z. Dunivin. Scaling hermeneutics: a guide to qualitative coding with llms for reflexive content analysis. EPJ Data Sci., 14(28), April 2025.
5. Z. Ersozlu and I. Taheri, Sona and· Koch. A review of machine learning methods used for educational data. Education and Information Technologies, 29:22125–22145, 2024.
6. M. Fowler, D. H. Smith IV, B. Chen, and C. Zilles. "i don’t gamble to make my livelihood": Understanding the incentives for, needs of, and motivations surrounding open educational resources in computing. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1, ICER ’23, page 430–443, New York, NY, USA, 2023. Association for Computing Machinery.
7. M. G. Graff, D. Moctezuma, and E. S. Téllez. Bag-of-word approach is not dead: A performance analysis on a myriad of text classification challenges. Natural Language Processing Journal, 11, 2025.
8. I. Hou, O. Man, K. Hamilton, S. Muthusekaran, J. Johnykutty, L. Zadeh, and S. MacNeil. ’all roads lead to chatgpt’: How generative ai is eroding social interactions and student learning communities. In Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2025, page 79–85, New York, NY, USA, 2025. Association for Computing Machinery.
9. S. Kannam, Y. Yang, A. Dharm, and K. Lin. Code interviews: Design and evaluation of a more authentic assessment for introductory programming assignments. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, SIGCSETS 2025, page 554–560, New York, NY, USA, 2025. Association for Computing Machinery.
10. W. M. Lim. What is qualitative research? an overview and guidelines. Australasian Marketing Journal, 33(2):199–229, 2024.
11. B.-M. Lindgren, B. Lundman, and U. H. Graneheim. Abstraction and interpretation during the qualitative content analysis process. International Journal of Nursing Studies, 108:103632, 2020.
12. X. Liu, A. F. Zambrano, R. S. Baker, A. Barany, J. Ocumpaugh, J. Zhang, M. Pankiewicz, N. Nasiar, and Z. Wei. Qualitative coding with gpt-4: Where it works better. Journal of Learning Analytics, 12(1):169–185, Mar. 2025.
13. X. Liu, J. Zhang, A. Barany, M. Pankiewicz, and R. S. Baker. Assessing the potential and limits of large language models in qualitative coding. In Y. J. Kim and Z. Swiecki, editors, Advances in Quantitative Ethnography, pages 89–103, Cham, 2024. Springer Nature Switzerland.
14. P. Mayring. Qualitative content analysis. Forum: Qualitative Social Research, 1(2), 6 2000.
15. P. Mulgund, R. Singh, M. Gillenson, Y. Li, and C. Oliver. “not my first choice”: Why software testing struggles to attract early-career professionals. In Proceedings of the 2025 Computers and People Research Conference, SIGMIS-CPR ’25, New York, NY, USA, 2025. Association for Computing Machinery.
16. H. Reichert, S. Sthapit, B. T. Tabarsi, A. Limke, T. Price, and T. Barnes. Experience helps, but it isn’t everything: Exploring causes of affective state in novice programmers. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2, SIGCSE 2024, page 1792–1793, New York, NY, USA, 2024. Association for Computing Machinery.
17. S. M. Renz, J. M. Carrington, and T. A. Badger. Two strategies for qualitative content analysis: An intramethod approach to triangulation. Qualitative Health Research, 28(5):824–831, 2018.
18. M. Sabin, A. N. Kumar, B. K. MacKellar, R. McCauley, T. VanDeGrift, and S. Matsumoto. The self-directed disposition: What computing students say. In Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2025, page 107–113, New York, NY, USA, 2025. Association for Computing Machinery.
19. J. Saldaña. The Coding Manual for Qualitative Researchers. Sage, 2013.
20. S. A. Silverio, J. Hall, and J. Sandall. Time and qualitative research: Principles, pitfalls, and perils. In The Sixth Annual Qualitative Research Symposium, 2020.
21. R. Sinha, I. Solola, H. Nguyen, H. Swanson, and L. Lawrence. The role of generative ai in qualitative research: Gpt-4’s contributions to a grounded theory analysis. In Proceedings of the 2024 Symposium on Learning, Design and Technology, LDT ’24, page 17–25, New York, NY, USA, 2024. Association for Computing Machinery.
22. K. Wantanabe. Latent semantic scaling: A semisupervised text analysistechnique for new domains and languages. Communicataion Methods and Measures, 15(2):81–102, 2021.
23. J. Whalley, A. Settle, and A. Luxton-Reilly. A think-aloud study of novice debugging. ACM Trans. Comput. Educ., 23(2), 2023.
24. Z. Xiao, X. Yuan, Q. V. Liao, R. Abdelghani, and P.-Y. Oudeyer. Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23 Companion, page 75–78, New York, NY, USA, 2023. Association for Computing Machinery.
25. Y. Zhang and B. M. Wildemuth. Applications of Social Research Methods to Questions in Information and Library Science, chapter Qualitative Analysis of Content. Libraries Unlimited, 2009.

© 2026 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.