Explainability Is Not Optional in Education AI
Artificial intelligence is rapidly becoming a partner in classrooms, career counseling, and student support. It can personalize instruction, identify learning patterns, and recommend educational pathways. But when AI influences decisions about how students learn - or how they see themselves, it must do more than generate answers. It must explain them.
This challenge is known as explainability - the ability of an AI system to communicate how it reached a conclusion in ways that people can understand, evaluate, and question. The National Institute of Standards and Technology (NIST) identifies explainability as one of the defining characteristics of trustworthy AI, recognizing that users should be able to understand why an AI system produces a particular recommendation rather than simply accepting it as correct
(https://www.nist.gov/artificial-intelligence/ai-research-explainability)
Unlike recommending a movie or a product, educational AI can influence a learner's confidence, aspirations, and future opportunities. Black-box systems - those whose reasoning remains hidden - can unintentionally reinforce misconceptions, amplify existing inequities, or discourage students from pursuing interests simply because no one can examine the rationale behind the recommendation.
Organizations such as UNESCO have therefore emphasized that AI in education should remain human-centered, transparent, and accountable, supporting educators rather than replacing professional judgment (https://www.unesco.org/en/artificial-intelligence?).
At ESTE, we believe explainability is about more than understanding technology. It is about understanding learners.
Students deserve to know why an assessment identifies particular strengths, how those conclusions were reached, what evidence supports them, and how those insights can help guide future learning. AI should illuminate a learner's potential, not obscure it behind an algorithm.
Educational AI should never become the authority on who a student is. Instead, it should provide transparent insights that help students better understand themselves while empowering teachers and families to participate meaningfully in interpreting those insights.
Trust Through Transparency
Transparency is not simply about revealing computer code or mathematical models. Most educators do not need to understand the underlying algorithms. They need explanations they can evaluate using their professional expertise.
Trust grows when AI makes its reasoning visible. Instead of asking educators to accept recommendations on faith, explainable AI allows them to ask:
What evidence contributed to this recommendation?
Which learner characteristics were most influential?
What assumptions might the system be making?
What information may be missing?
How confident is the system in its recommendation?
Equally important, trustworthy AI should communicate uncertainty. A recommendation accompanied by an appropriate level of confidence is often more valuable than one presented with unwarranted certainty. Recognizing uncertainty encourages thoughtful human judgment rather than unquestioning acceptance.
The OECD's AI Literacy Framework similarly argues that learners should understand how AI systems generate recommendations, critically evaluate their outputs, and recognize both their capabilities and limitations (https://www.oecd.org/en/publications/empowering-learners-for-the-age-of-ai_65cd27d4-en.html?).
The ESTE Lens: Engineering + Science
The ESTE Framework reminds us that effective educational systems combine rigorous science with thoughtful engineering.
Science encourages curiosity, evidence, observation, and continuous questioning.
Engineering focuses on designing systems that people can understand, evaluate, improve, and trust.
Explainability sits at the intersection of both disciplines.
An engineered system should not simply generate recommendations. It should make its reasoning visible. Likewise, a scientifically grounded educational tool should invite inquiry rather than discourage it.
When educational AI explains its reasoning, teachers remain empowered as decision-makers, students become active participants in their own learning, and families gain confidence that recommendations are grounded in evidence rather than hidden processes.
This is especially important for assessments that explore learner identity. Understanding why a learner demonstrates particular strengths is often more valuable than the label itself.
Call to Practice
This month, invite students to become AI investigators rather than AI consumers.
Ask students to submit the same educational question to two different AI systems.
Then compare:
How did each system explain its recommendation?
What evidence or sources did it provide?
Did one acknowledge uncertainty while the other did not?
Which explanation was more convincing - and why?
Would you make the same recommendation yourself?
The objective is not to determine which AI is "correct." Rather, it is to help students recognize that good thinking requires examining the reasoning behind the answer, not simply accepting the answer itself.
A Bright Spot
Fortunately, educational AI is beginning to move toward greater transparency.
Google Learn About encourages exploration rather than delivering a single definitive answer. Learners are guided through related concepts, multiple perspectives, and follow-up questions that help make the learning process visible rather than simply providing an endpoint (https://learning.google.com/experiments/learn-about).
Similarly, Khan Academy's Khanmigo functions as a learning coach rather than an answer generator. By asking guiding questions and encouraging productive struggle, Khanmigo helps students understand how they arrive at an answer instead of simply providing one
(https://www.khanacademy.org/khan-labs)
While no AI system is perfectly explainable, these approaches demonstrate an encouraging shift toward educational technologies that support reasoning instead of replacing it.
Action Items
For Educators
Ask students to explain why they agree or disagree with AI-generated recommendations.
Choose AI tools that provide reasoning, evidence, citations, or confidence indicators.
Treat AI outputs as conversation starters rather than final answers.
For School Leaders
Include explainability among the criteria used when evaluating educational AI platforms.
Develop policies requiring meaningful human oversight for AI-assisted educational decisions.
Invest in professional learning that builds AI literacy among educators.
For Students
Ask how an AI reached its recommendation before deciding whether to trust it.
Look for evidence, not just answers.
Remember that AI can support your learning, but it should never replace your own reasoning or self-reflection.
Explainability is ultimately about respecting learners. Students should never be expected to shape their educational journeys around recommendations they cannot understand or question. As AI becomes more deeply integrated into education, the goal is not simply to build more intelligent systems - it is to build systems that make their reasoning transparent and invite thoughtful dialogue.
At ESTE, we believe the most valuable educational technologies do more than predict outcomes. They help learners better understand themselves. AI should empower students to explore their strengths, question assumptions, and make informed decisions about their futures, not because an algorithm declared it so, but because the evidence is visible, understandable, and open to discussion.
ESTE® Leverage - founded in the belief that Entrepreneurship, Science, Technology, and Engineering are innate in each of us - grounded in the science of learning & assessment - dedicated to the realized potential in every individual.