How to Evaluate an AI Tutoring Pilot

Choose the decision the pilot must inform

A pilot should answer a bounded investment or delivery question. Can the assistant provide useful hints for a particular unit? Can students navigate approved explanations? Can educators maintain the source material and review defects? Write the decision before choosing metrics. Otherwise a pilot can collect impressive conversation statistics without resolving whether the institution should continue.

Distinguish feasibility, usability, and learning impact. A short pilot may show that an integration works and that students understand its interface. It may provide limited evidence about learning, especially without an appropriate comparison and enough time. Describe the conclusion at the level the design supports. Do not turn a successful demonstration into a general claim about improved attainment.

Build a representative evaluation set

Map test cases to learning objectives, content types, and plausible learner errors. Include straightforward questions, incomplete attempts, misconceptions, ambiguous wording, unsupported topics, and requests to reveal an assessed answer. Have subject specialists define acceptable behaviour. A response need not match one exact sentence, but it should satisfy explicit educational requirements.

An illustrative chemistry pilot could include interpreting a simple equation, explaining a relevant term, and identifying an error in a learner's reasoning. Record which approved source supports each case. Keep some cases separate from examples used to tune the assistant. If every evaluation question has already shaped the implementation, the results provide a weaker test of behaviour on unfamiliar but relevant questions.

Use an educator review rubric

Ask reviewers to score factual correctness, curriculum alignment, appropriateness for learner level, usefulness of the next step, and handling of uncertainty. Define unacceptable failures separately, such as exposing restricted assessment material or giving a materially false explanation. A high average score should not conceal a serious failure that needs correction before learner use.

Have more than one reviewer inspect a suitable subset and discuss disagreements. A disagreement can reveal an unclear objective or multiple valid teaching approaches, not just an unreliable reviewer. Record the resolution and revise the rubric where needed. Preserve examples showing why an output passed or failed so future changes can be assessed against the same educational intent.

Observe learners doing real tasks

Invite intended users to complete realistic activities with the pilot under the institution's approved process. Watch where they ask for help, misread a response, stop working, or treat an unsupported statement as authoritative. Include relevant accessibility and language needs. A satisfaction question can supplement observation but should not replace evidence about what the learner actually did.

If evaluating learning, use an independent task that requires applying the concept beyond copying the tutor's answer. Decide in advance how performance will be assessed and whether a comparison is feasible. Explain selection effects, missing data, and the limits of a small local pilot. Any illustrative scoring thresholds in a proposal should be labelled as proposed criteria and agreed with educators before results are reviewed.

Check operations and boundaries

Test course access, content updates, unavailable providers, safe stopping, and educator escalation. Measure response behaviour in the conditions the institution expects, including ordinary device and network constraints. Review whether operational logs support incident investigation without retaining unnecessary learner information. Document which model, source collection, and configuration were evaluated.

NIST's framework supplies a structured risk-management reference, while UNESCO emphasises educational validation and human oversight. Neither substitutes for local acceptance evidence. The pilot report should connect each identified issue to an owner, proposed correction, retest requirement, and current deployment decision. A limited release can remain limited while unresolved cases are investigated.

Make the expansion decision explicit

Finish with a record of supported uses, observed strengths, failed cases, unresolved uncertainties, operating responsibilities, and the next decision. Choose among revising the pilot, maintaining the current scope, expanding a justified part, or stopping. Expansion to a different subject, learner age, or language requires additional evidence because it changes the task boundary.

• Agree evaluation questions and acceptance criteria before exposing learners to the pilot.
• Review educational quality, user experience, and system reliability separately.
• Keep serious failures visible alongside aggregate results and positive feedback.
• Retest affected cases after changes to sources, prompts, models, integrations, or permissions.