Generative AI sometimes gives an answer that sounds certain but isn’t true. We call this a hallucination. It isn’t a bug, it isn’t a quirk of “early” AI, and it isn’t the result of poor training — it is a natural consequence of how intelligence works.
It’s worth asking how an unreliable tool can fit the nature of learning. What matters is that we decide — with that unreliability in mind — which tasks we hand it and how far we let it go.
The first question is how reliable it actually is. Not long after GoSchool launched, some of our first teachers examined an AI’s answers across four university courses in a study and found that most answers are usable, though a few can be wrong or misleading. 95% of the answers were good enough to help students master the material.
What Is a Hallucination?
Intelligence isn’t about reciting knowledge you already have; it’s about reasoning in new situations. Today’s language models don’t carry out a rule spelled out in advance. They work through machine learning — they answer on the basis of patterns learned from many examples. That inevitably leaves situations where they decide differently than we would. But the same is true of people: the more complex the situation, the more we lean on estimates, analogies, and experience.
Every answer generative AI gives is a kind of “good guess” at how the text would continue, based on earlier examples. When the model has seen many good examples, the guess is right. When the pattern is uncertain, the guess can go astray. A hallucination, then, isn’t a malfunction; it’s the flip side of the model being able to respond to situations that are new (to it) — even when its information is incomplete.
What makes it intelligent is exactly this: it doesn’t only repeat what it has seen before but can produce new combinations, new formulations, new connections. Here creativity and error are two sides of the same coin. If we demand that it never err, we give up genuinely intelligent behavior.
Hallucination is a natural part of the technology — but that doesn’t make the real risks and difficulties in an educational setting go away.
The Risks of Using AI in Education
There’s good reason we’re wary of using AI. Generative AI shapes the learning process in complex ways.
1. It can mislead the student
A generative AI’s answer is often coherent and convincing, so a learner readily takes it to be true. That’s especially problematic during concept formation and the early build-up of knowledge, when the student doesn’t yet have stable internal checks. At that stage a wrong but plausible-sounding answer can lead to a lasting misconception.
2. It solves the task in their place
AI can hand over a complete solution, letting the student skip the steps of cognitive processing and problem-solving. That reinforces surface learning and blocks the deep learning in which a student truly grasps how things connect. Dropping out of the problem-solving process weakens the long-term formation of knowledge.
3. It hides misunderstandings
A well-structured AI answer can easily create the impression that the student understands the material, when in fact they are only accepting the solution. That gets in the way of diagnostic assessment, since the teacher may not notice the deeper conceptual errors. And on the student’s side it reduces metacognitive monitoring: they don’t notice when they haven’t understood something.
4. It can weaken motivation
If there’s a quick, ready answer for everything, one of learning’s most important engines can disappear: the sense of competence. The “I figured it out!”, “I get it!” moment never comes, and tasks easily turn mechanical. Over time that can erode intrinsic motivation, especially in students who are already less sure of themselves in the subject.
The teacher’s role becomes more important
AI feels threatening not because it knows more, but because it works differently. Yet a teacher’s job was never mere delivery of knowledge: it is guiding the learning process, supporting the environment, giving feedback, and organizing what students actually do. AI cannot take those over. The sense of uncertainty really comes from teachers having to redefine their own professional space — and that space doesn’t shrink, it grows in importance.
Can It Be Used Well?
Most of the risks above don’t fall in the essential phases of learning; if anything, understanding, practice, transfer, and independent thinking grow stronger in these situations. AI helps in plenty of cases where the goal of learning isn’t a flawless answer at all, but getting thought moving, practicing, or building independence.
1. Creating a first version of understanding
Learning is rarely linear. A student doesn’t always understand the textbook on the first pass, and that’s entirely normal. In those moments another phrasing, an alternative example, or a quick summary helps a lot — anything that “opens up” the material. Generative AI is excellent at this: it offers a first version the student can measure against, and from there it’s easier to move toward the more detailed, more precise material.
2. Recalling the details
One of the hardest parts of learning isn’t understanding but recall: will we later remember what we once understood? Generative AI is remarkably effective here. If a student can’t remember a formula, the exact wording of a concept, or a detail mentioned in a lecture, AI retrieves it in seconds — from their notes, or even from the lecture recording, if that is part of the course materials. This frees up working memory and helps the learner concentrate on what matters: understanding how things connect.
3. A diagnostic warm-up before class
A short AI-led opening question (“What do you already know about this topic?”) shows where a student is starting from. It replaces the whole-class “what have you covered so far?” round and helps the teacher spot misconceptions before diving into the material.
2. Finding alternative explanations
A strength of AI is that it can say the same thing in several ways. When a student gets stuck, they can quickly get a new example, a different analogy, or another way of putting it. The teacher saves time, and the student finds the version that works for them.
3. Providing structured practice
A student can ask for a new exercise, a new level, a new variation at any time. A teacher can’t be present 24 hours a day — AI can. This is especially useful in concept- and skill-level learning, where the sheer amount of practice matters.
4. Prompts for thinking and starting a project
For an essay, a research task, or a presentation, a student can ask for a list of ideas, a concept map, or talking points. Here what matters isn’t whether the content is true, but that they get moving and find their own direction.
5. Creating a “safe space to fail”
Students dare to ask an AI things they would hesitate to ask their teachers. There are no “wrong questions” here. This builds independent learning and lowers anxiety, especially for those who are afraid to speak up.
6. Instant feedback
Learning needs a steady rhythm: frequent attempts and frequent feedback. AI often responds in seconds rather than minutes, which speeds up the learning cycle.
What Can I Do as a Teacher?
Students use AI even when they’d be better off not — often in exactly the ways that strip the point out of learning: to write the homework, or to churn out quick outlines, flashcards, or quizzes. These rarely help real learning. So our job isn’t to keep them away from the technology with bans and rules; it is to help them find the ways of using it that genuinely support understanding the subject.
That means we first have to choose which good use fits our own subject and our own learning goals. One thing works in a theoretical subject, another in a hands-on, calculation-heavy one, and something else again where reading comprehension is the central element.
Once we’ve taken that step, our job is no longer to regulate everything, but to walk the student through the situations where AI really can help: the first version of understanding, recalling the details, the rhythm of practice, alternative explanations. AI doesn’t replace the teacher — but without the teacher it easily leads learning astray. Our task is to show how to use it well, and to keep, in the process, what makes learning learning: independent thought, personal responsibility, interpretation.
We designed GoSchool to fit into teaching as simply as possible. In the first prototype we minimized hallucination so the system stays faithful to the material you give it — so your students see the connections you actually consider important.
The system puts shared use first: the teacher sees how students think, what they ask, where they get stuck. That helps build trust, and helps you find together the way of using AI that fits your subject best.
If you’d like to try it, or see how it would work in your own course, write to us — we’ll put together a short demo and help you fit it into your teaching.
Contact