TL;DR: Most XR training programs say they measure performance and end up reporting completions. Completion tells you someone wore the headset, not that they can do the job. Useful measurement is designed before the first scene is built: agree with your subject matter experts what good and unacceptable performance look like, instrument the scenario around those steps, check that the score actually separates experienced people from novices, and plan from day one how you will show the skill turning up on the job. Do that and the data becomes the reason the program keeps its budget. Skip it and you are running an expensive demo with a sign-in sheet.
Ask anyone commissioning VR or AR training whether it will be measured and the answer is always yes. Ask six months after launch what the dashboard shows and the honest answer is usually a completion rate, an average session time and a pass rate that sits near 100 percent because the pass mark was set to make the pilot look good.
That is not a technology failure. Headsets can capture far more about how a person performs a task than a classroom ever could. It is a design failure: measurement was treated as a reporting feature to add at the end, rather than as the thing the scenario is built around.
We build XR for settings where that difference matters, including ERIS XR for ARCortex, which helps firefighters pre-plan and train, and Nystag, VR eye-tracking diagnostics on the Vive Focus 3 where clinical precision is the whole point. Our pieces on VR training for high-risk teams and enterprise AR training each say "bake measurement in from the start." This article is the long version of that sentence.
Completion is not competence
A completion record answers one question: did this person reach the end of the module. It does not tell you whether they skipped the isolation check and got lucky, took four attempts, needed the hint on every step, or did it perfectly in half the expected time.
Those differences are exactly what a supervisor wants to know before they let someone near the real equipment. They are also exactly what XR is good at capturing, because every grab, gaze, step and pause happens inside software you control. A program that reduces all of it to a green tick has thrown away the main advantage it paid for.
Define good before anyone builds a scene
The most valuable measurement work happens in a meeting room, not in the engine. Sit down with the people who actually do and assess the task and break it into its parts:
- Critical steps. The steps that must happen for the task to be done at all.
- Order-sensitive steps. The ones where doing them in the wrong sequence is itself a failure, such as testing before isolating.
- Time-sensitive steps. Where speed genuinely matters, and where it does not. Rewarding speed on a task where care matters more will train the wrong habit.
- Critical errors. Actions that fail the attempt regardless of everything else, because in real life they hurt someone or destroy something.
- Acceptable variation. The different but valid ways experienced people do the same job. If the scenario only accepts the path the designer had in mind, your best people will score badly.
This list becomes the spine of the scenario and of the score. It also exposes disagreement early. It is common to discover that two senior instructors teach the same procedure differently, and far cheaper to settle that before a scene is built than after trainees have been failed for following one of them.
Four layers of evidence
It helps to think of measurement in layers, each harder to collect and more convincing than the one before. A good program plans for all four, even if it starts with two.
Did they do it
Completion, attempts and time on task. Necessary, cheap and the weakest signal. Useful for scheduling and compliance, not for judging skill.
How did they do it
This is where XR earns its keep. Which steps happened, in what order, which were missed, where the trainee hesitated, whether they checked what they were supposed to check, and how often they needed help. In some tasks, where a person looked matters as much as what they touched, and eye-tracking headsets can record it. That is the same capability we used on Nystag for clinical assessment, and it carries over to training when gaze is part of the skill, such as scanning a scene or a panel. It is also sensitive data, which we come back to below.
Did it get better
A single score is a snapshot. The useful picture is the trend: does the same person make fewer errors on the third attempt, and do they still perform after a gap of weeks. Retention after time away is the honest test of whether training stuck, and it is easy to schedule in XR because repeating a scenario costs almost nothing.
Did it show up on the job
The layer that matters to whoever funds the program, and the one most often skipped. Does the trained group pass real-world assessments at a different rate, reach sign-off sooner, need less supervisor time or show fewer errors in the field. You rarely get this from the headset alone. It needs data from systems the training team does not own, so it has to be agreed with operations at the start, not requested after launch.
A score instructors will trust
Instructors will ignore a score they do not believe, and they are right to. Three habits make the difference.
Make critical errors gates, not deductions. If skipping a safety step costs ten points out of a hundred, someone can skip it and still pass with a high score. Critical errors should fail the attempt outright, with the rest of the rubric scoring everything else.
Calibrate against real people. Before launch, run experienced practitioners and genuine novices through the scenario. If the experts do not clearly outscore the novices, the score is measuring something other than skill. Very often that something is comfort with the headset and controllers.
Separate the interface from the task. First-time users lose points fumbling with menus, hand tracking or locomotion, and that noise looks like incompetence. An unscored familiarization scene before the real one removes most of it. Good interaction design removes more, and so does taking comfort seriously, because someone fighting motion sickness is not showing you their skill. If your trainees include people with different abilities, check the score does not penalize the accommodations covered in XR accessibility.
For free-form tasks, such as a conversation with a simulated witness or patient, there is no checklist of hotspots to score. That needs a rubric of observable behaviors and a review pattern for borderline runs, which we cover in AI characters in XR training.
Instrument the scenario, not the session
A common mistake is to log everything the device can produce, head position many times a second, every controller event, every frame, and hope the insight can be extracted later. It rarely is. Raw telemetry is huge, hard to interpret and a privacy liability.
The better approach is to define meaningful events that map onto the step list: valve closed, test performed, hazard identified, hint requested, critical error committed. Each carries a timestamp, the attempt it belongs to and the version of the scenario that produced it. Keep raw telemetry only where a specific question needs it, and only for as long as it needs it.
The version tag is the detail people forget. Scenarios change after launch as procedures change, equipment changes and confusing moments get fixed. If you cannot tell which version produced a score, you cannot compare this quarter with last quarter, and a content fix can look like a sudden jump in competence. Treat scoring logic as part of the product that has to be maintained, which is the point of our guide to XR app maintenance.
Proving it worked to the people who pay
When the budget review comes, the question is not "did people enjoy it." It is "did this change anything we care about." You can only answer that if you planned the comparison.
Capture a baseline. Before the program starts, record how the same task is currently assessed and what the results look like. Without a before, there is no after.
Compare like with like. A volunteer pilot group is usually more motivated than the general population, so comparing them with everyone else flatters the technology. Where possible, compare similar groups, or the same group before and after.
Claim only what you can show. If assessment results improved in the same quarter a new supervisor arrived, say so. Funders trust a modest, well-supported result far more than a dramatic one that falls apart under one question.
Keep the instructors' view. Numbers explain what changed. The people running the sessions explain why, and their observations often point to the next improvement.
Where the data goes and who sees it
A score nobody sees does nothing. Agree early whether results flow into your LMS, a learning record store using xAPI, a compliance system or a health record in clinical settings, and what the record contains beyond a pass. That integration work is covered in more detail in enterprise XR rollout.
Also agree who can see what. Detailed performance data about identifiable people, especially gaze, voice and recordings of mistakes, raises fair questions about whether it can be used in performance reviews, how long it lives and who can replay it. Answer those before trainees ask, because a workforce that suspects the headset is grading them for HR will behave differently in it. The framework we use is in XR data and privacy, and in clinical settings, where a measurement has to stand up as evidence, the bar rises further, as described in clinical VR diagnostics.
Questions to ask before you commission XR training
- Who on our side defines good performance and critical errors, and when do they sit down with the builders?
- What will the scenario record, and how does each event map to a step our instructors recognize?
- Will the score be checked against experienced staff and novices before launch?
- How do we separate headset unfamiliarity from lack of skill?
- What is our baseline, and what on-the-job result will we compare against?
- Where do results go, who can see them, and how long are they kept?
If a vendor answers these with screenshots of a dashboard, keep asking. The dashboard is the last step, not the first. Measurement design also affects cost and scope, so it belongs in the conversation covered in what an XR app costs.
The bottom line
XR training is persuasive in a demo and has to be defensible in a budget review. The bridge between the two is measurement that was designed with the scenario rather than added after it: a clear definition of good, critical errors that fail the attempt, events that map to real steps, a score that separates experts from novices, and a plan to look for the result on the job. Programs built that way tend to survive their first renewal, because they can show what they changed.
Planning VR or AR training and want it to stand up in front of the people who fund it? Book a demo and we will help you define what to measure, how to score it and how to connect it to the systems you already use. See our XR work: ERIS XR and Planes XR for ARCortex, Nystag clinical VR eye-tracking on the Vive Focus 3, and MR Camera shared mixed reality.