Can you put a conversational AI character into VR training?

Yes, and it is the clearest fix for the problem most XR training programs hit around year two: the content stops. Branching logic is most of the actual software in a training build, every path is written by hand by a subject-matter expert, and a fixed tree has a shelf life measured in reps, because by the fourth run the trainee is recalling which option the tree rewarded rather than making a decision. Where the skill being trained is a conversation, a model can generate those branches at runtime instead, so the author defines who the character is, what they know and what would change their behavior rather than enumerating paths. It only helps for skills taught through talking: clinical history taking, patient handover, de-escalation, incident command, intake, witness questioning. For motor and procedural skills, where your hands go and in what order, a conversational model adds nothing and the money belongs in tracking and interaction quality instead. It is also the wrong call when the exact words are the point, since a scripted line plus speech recognition is cheaper and easier to defend, and when volume is low, because one scenario run by twenty people a year does not repay the engineering.

Does an AI character in a training simulation have to run in the cloud?

Not necessarily, and the deciding factor is usually the network in the room rather than the model. Training gets delivered in station basements, plant floors and hospital sub-levels, and a character that stalls when the Wi-Fi dips is worse than no character at all. The constraint to design against is the turn-taking budget: capturing audio, deciding the trainee has finished speaking, transcribing, generating a reply and synthesizing speech all have to complete fast enough that a character standing in front of you does not appear to freeze, which is a harsher bar than the same pipeline on a phone call. Practical options are pre-generated openings and stock lines, a graceful degrade when the connection drops mid-scenario, and an honest evaluation of whether a smaller on-device model clears the quality bar for this scenario. One constraint is not negotiable either way: comfortable XR means holding a stable frame rate, so nothing in the speech pipeline may compete with the render loop for it. That is an architecture decision taken at the start, not a tuning pass at the end.

AI Characters in XR Training: When the Simulation Talks Back

Q: How do you grade a free-form conversation in VR training?

With a rubric of observable behaviors defined by the same expert who wrote the scenario, scored from the transcript by an evaluation pass that is separate from the character itself. Did the trainee identify themselves. Did they ask open questions before narrowing. Did they avoid leading the answer. Each of those is checkable against a transcript, and a separate pass scores them consistently across hundreds of runs in a way no instructor has time to. Two rules keep it honest. The actor should never be the grader, because a character that is also scoring you has an incentive to be agreeable. And automated scoring is strong enough to triage and to trend a cohort, not to certify anyone by itself, so the workable pattern is machine scoring on every run with flagged and borderline runs going to an instructor with the transcript attached. Remember what the transcript is: a recording of an identifiable person doing their job badly on purpose. Decide up front who can play it back, how long it lives, and whether it can ever be used in a performance review, because that is a governance question before it is a storage question.

TL;DR: Hand-authored branching is why most XR training programs stall at three scenarios. Putting a conversational AI behind a character in the simulation removes that ceiling for anything taught through talking: intake, history taking, de-escalation, handover, questioning a witness. It does nothing for motor skills, and it brings three problems of its own. The speech loop has a latency budget that is harsher inside a headset than on a phone call. The character has to stay inside the scenario, which means the simulation's state and not the model's memory is the source of truth. And you now have to score a conversation nobody wrote in advance.

Ask a team two years into an XR training program what went wrong and you rarely hear about headsets. You hear that the content stopped. They shipped a few scenarios, trainees liked them, and now everyone who ran them knows exactly what happens on every path.

That is not a failure of immersion. It is arithmetic. We build both halves of what fixes it: XR platforms like ERIS XR for ARCortex, where firefighters pre-plan and train against real buildings, and production voice AI like the systems behind CallGuard and Fortell. Here is what we have learned about putting the second inside the first, including where it does not belong.

The authoring ceiling is the real constraint

Branching logic is most of the actual software in a training build. Every decision point, every consequence, every recoverable mistake is written by hand, usually by a subject-matter expert whose day job is not writing scenarios, and who is the scarcest person on the project.

So programs ship a small number of deep scenarios and then plateau. A fixed tree also has a shelf life measured in reps: the first run teaches judgment, and by the fourth the trainee is not making decisions, they are recalling which option the tree rewarded.

Where the skill being trained is a conversation, a model can generate those branches at runtime instead. The scenario author defines who the character is, what they know, what state they are in and what would change their behavior, and the paths stop being enumerated in advance.

What a talking character adds, and what it does not

Immersive training teaches three different things, and separating them tells you whether any of this applies.

Motor and procedural skills. Where your hands go, in what order, under time pressure. XR has always been good at this and a conversational model adds nothing.
Spatial and decision skills. Where to move, what to prioritize, what the room is telling you. Conventional scenario logic serves this well.
Communication skills. What you say, how you ask it, whether you can hold a line under pressure. This has always needed a human role-player, an instructor's time and a scheduled seat.

Only the third is the case for an AI character, and it is a big case. Clinical history taking, patient handover, de-escalation, incident command over a radio, benefits intake: in all of these the difficulty is that people hold back, panic, contradict themselves or say the same thing four ways. You cannot write that convincingly as a menu of three options.

Language is the other unlock. The voice work we do for Fortell handles intake in more than a hundred languages, the same capability that lets a training character speak the language a trainee will meet in the field. If the population you serve is multilingual, practicing only in English trains for the easy version of the job, and the trade-offs are the ones we cover in multilingual voice AI.

The speech loop is a latency budget in disguise

On a phone call, a beat of silence reads as thinking. In a headset it reads as broken, because the character has a body and eyes, and bodies do not freeze for two seconds before answering. The turn-taking budget is the constraint everything else bends around.

Five things sit inside every turn: capturing audio, deciding the trainee has finished speaking, transcribing, generating a reply, and synthesizing speech. Endpointing is the underrated one: wait too long and the character feels slow, cut in early and it interrupts a trainee who was pausing to think, which is what a stressed trainee does.

Three facts make this harder than the same pipeline on a phone:

The microphone is on a headset in a shared room. In team drills it hears the other trainee, the instructor and the extraction fan, and shared mixed-reality sessions like the MR Camera environment we built turn one audio problem into several.
The frame rate is not negotiable. Comfortable XR means holding a stable frame rate, and nothing in the speech pipeline may compete with the render loop for it. That is an architecture decision.
Training rooms have bad networks. A character that stalls when the Wi-Fi dips is worse than no character, so the design needs pre-generated openings, a graceful degrade, and an honest answer about whether a smaller on-device model is the right call. Those trade-offs are the subject of choosing an AI model.

The scenario owns the truth, not the model

The most common failure mode is a character that is delightful and wrong. It invents a symptom the scenario never specified, agrees the trainee did the right thing when they did not, or wanders into a topic the training is not about. In a chatbot that is embarrassing. In training it is worse, because the trainee walks out having practiced the wrong thing and believing they got it right.

The fix is a boundary most demos skip. The simulation holds the state: what is true about this patient, this building, this shift. The character reads that state rather than remembering it, and anything it is not told, it does not know. When the trainee changes the world, the change is written to the simulation and the character reads the new state.

Two more rules earn their keep. The actor should not be the grader, because a character that is also scoring you has an incentive to be agreeable. And the character needs a tested way to stay in role when a trainee tries to break it, which they will, usually by asking the simulated patient to explain the correct answer.

None of it holds still. Prompt edits, model changes and content updates regress character behavior silently, so the honesty cases belong in a regression set that runs on every change, the same discipline we describe in monitoring an AI voice agent in production.

Scoring a conversation nobody wrote

Measurement is what separates a training program from an expensive novelty, and free-form conversation breaks the usual approach: there is no hotspot to check.

What works is a rubric of observable behaviors defined by the same expert who wrote the scenario. Did the trainee identify themselves. Did they ask open questions before narrowing. Did they avoid leading the answer. Each is checkable against a transcript, and a separate evaluation pass scores them consistently across hundreds of runs in a way no instructor has time to.

Be honest about the ceiling: automated scoring is strong enough to triage and to trend a cohort, not to certify anyone by itself. The workable pattern is machine scoring on every run, with flagged and borderline runs going to an instructor with the transcript attached.

That transcript is a recording of an identifiable person doing their job badly on purpose, which is a governance question before it is a storage question. Decide up front who can play it back, how long it lives, and whether it can ever be used in a performance review. The framing we use sits in XR data and privacy, and the retention mechanics are the ones in recording, consent and disclosure.

Where it does not belong

Four cases where the answer is no:

Motor skill drills. If the training value is in the hands, spend the money on tracking and interaction quality instead. That is the subject of XR interaction design.
Fixed-phrase protocols. When the exact words are the point, a scripted line plus speech recognition is cheaper, more reliable and easier to defend than a generative character.
Unreviewed high-stakes content. If a wrong statement from the character could teach an unsafe practice and nobody reads transcripts, the guardrails are not yet good enough for that scenario.
Low volume. One scenario run by twenty people a year does not repay the engineering, and a human role-player is the correct answer.

What it costs and how to scope it

Treat this as an increment on an XR build rather than a separate project. The cost drivers are the number of distinct characters, whether the interaction is voice or text, how many languages, cloud or on-device inference, whether scoring is automated, and whether sessions are shared. The underlying XR budget behaves the way we describe in what an XR app costs.

Prove one character, in one scenario, on the real headset, in the real room, with the real network, in week one. If the turn-taking feels wrong there, no amount of content fixes it later. The rest of the program still needs the discipline we cover in rolling XR out past the pilot and in enterprise AR training.

Questions worth asking before you commission a build

Is the skill being trained actually a conversation? If it is procedure or motor memory, this is the wrong investment.
What is the target turn-taking latency on the real hardware, and when do we first measure it?
Where does scenario truth live, and how does the character read it? If the answer is "in the prompt", state will drift.
Who defines the scoring rubric, and who reviews the flagged runs?
What is the retention and access policy for trainee transcripts?
What is in the regression set, and does it run on model updates we did not schedule?

The bottom line

The ceiling on XR training has moved from what a headset can render to how much scenario content an organization can afford to write. For anything taught through talking, a conversational character lifts that ceiling, because the branches stop being enumerated and start being generated. It only pays off if the simulation keeps hold of the truth, the speech loop is fast enough to feel like a person, and somebody decided in advance how a conversation gets scored.

Have a training scenario where the difficult part is the conversation? Book a demo and we'll scope it honestly, including telling you when a scripted scenario or a human role-player is the better answer. See our work: ERIS XR for ARCortex, Nystag clinical VR eye-tracking, MR Camera shared mixed reality, and the voice systems behind CallGuard and Fortell.