Define what a theme count should mean
Open-ended comments can explain experiences that a rating alone leaves unclear. Before asking AI to summarise them, decide what the analysis should produce: recurring topics, reported barriers, requested improvements, or a deeper qualitative interpretation. Define the unit being coded. A whole response, a sentence, and a distinct issue within a response can produce different counts.
For an illustrative student feedback study, one comment might mention unclear deadlines and helpful tutors. A multiple-theme scheme can preserve both observations. Count respondents mentioning each theme if that is the intended measure, and state that percentages may exceed one hundred when a respondent can receive several codes. Keep topic coding separate from any sentiment classification.
Prepare an approved analysis copy
Keep the original export secure and create a separate working copy with stable response identifiers. Remove direct identifiers and review free text for information that might identify students or staff. Confirm the institution's permitted processing arrangements before placing comments in an AI tool. Limit included fields to those needed for the agreed analysis and validation.
Preserve the original language and record any translation. Comments from India, the US, and the UK may include local education terms, abbreviations, and mixed-language expressions. Pew's research on translating open-ended answers shows that translation choices can affect coding quality. Review meaning with appropriate language expertise instead of assuming every translated phrase retains its original context.
Create a codebook that handles difficult cases
Read a varied initial set of comments and draft theme definitions. For each code, specify inclusion rules, exclusion rules, and illustrative examples. Add categories for unclear content, irrelevant responses, and issues requiring a separate institutional process. Test whether reviewers can distinguish neighbouring themes, such as access to support and the usefulness of support once received.
Allow genuinely new topics to be proposed during review, with a controlled process for updating the codebook. If a definition changes, identify which previously coded responses need another pass. Avoid forcing every comment into a small set of categories simply to make a neat chart. Rare issues can matter even when they do not form a dominant theme.
Validate AI coding against independent human decisions
Give the AI the approved codebook, clear output fields, and instructions to identify uncertain cases. Build a separate validation set that reflects different lengths, languages, topics, and respondent groups. Have trained reviewers code it independently of the AI output, then resolve human disagreements. Use the adjudicated decisions to assess how reliably the proposed workflow applies each theme.
AAPOR's guidance on generative AI in survey research calls for explaining how accuracy will be assessed and reporting that validation. Measure errors by theme rather than relying only on overall agreement. A system that frequently misses a rare but important access barrier can appear accurate when most comments concern other topics. Revise the workflow and reassess before full production.
Keep the final themes connected to actual comments
After coding, use deterministic calculations to produce counts from the reviewed records. Report the base: all survey respondents, people shown the comment question, or people who provided a usable comment. Those denominators answer different questions. Do not describe the distribution of comment themes as the prevalence of experiences among all students unless the study design supports that inference.
Select representative quotations from the original responses and review them for privacy and context. Do not publish an AI-generated sentence as a student's quotation. If you paraphrase to protect identity, label it as a paraphrase. Link internal theme summaries to response identifiers so an authorised reviewer can inspect the evidence without exposing raw comments to every dashboard user.
Turn interpretation into accountable follow-up
The reporting note should describe the codebook, AI role, human review, validation findings, translation approach, and remaining limitations. Assign an owner to each proposed action and explain what further evidence is needed. An illustrative scheduling theme might prompt timetable interviews before a major redesign, while a documented broken resource link can be investigated directly by the course team.
White Storm Technologies can build a custom comment-analysis workflow with controlled data handling, versioned coding instructions, review queues, and reproducible summaries. Delivery can include the reviewed dataset and handover guidance alongside the report. Agree the institution's review responsibilities in advance so that sensitive or ambiguous material reaches a qualified person rather than disappearing inside an automated summary.
Sources & further reading
Primary references checked for this article. Practical examples are illustrative, not client results.
Apply this to your project.
Talk to us about turning your survey or business data into useful findings, with data-quality checks, clear explanations, and human review of the conclusions.
Explore AI Data Analysis
