When Artificial Intelligence Becomes a Risk, Not a Resource
Artificial Intelligence (AI) can expand learning opportunities but without safeguards, it can also expose students and schools to risks they are not prepared to manage.
AI is rapidly becoming part of the educational landscape. From personalized learning platforms to AI-powered tutoring and career exploration, schools are discovering new ways to support students and educators. Yet as these technologies become more accessible, an equally important question emerges: Are we designing AI systems that are safe enough for the students who depend on them?
For K–16 institutions, AI security and safety extend beyond protecting networks and passwords. They include safeguarding student information, preventing misleading or harmful outputs, establishing accountability, and ensuring that technology operates within appropriate educational boundaries. The challenge is not simply to make AI more capable. It is to make AI more trustworthy.
The LLM Gap: When Powerful Technology Creates New Vulnerabilities
Large language models (LLMs) can generate remarkably convincing responses, but fluency should not be mistaken for reliability or safety.
AI systems may produce inaccurate information, reinforce biases, disclose sensitive information through inappropriate data handling, or respond unpredictably to malicious or manipulative prompts. These vulnerabilities become particularly concerning when AI interacts with children and young adults who may not recognize the limitations of the technology.
Unlike traditional educational software, general-purpose LLMs can respond to an enormous range of questions and requests, including those beyond their intended educational purpose. This flexibility creates opportunity, but it also introduces uncertainty.
The National Institute of Standards and Technology (NIST) addresses these challenges in its AI Risk Management Framework, which emphasizes that trustworthy AI requires attention to safety, security, privacy, transparency, and accountability throughout the technology lifecycle.
For schools, the lesson is clear. An AI system's ability to provide an answer does not mean that the answer is appropriate, accurate, or safe for educational use.
The K–16 Lens: Student Data, Institutional Responsibility, and Trust
Schools manage some of the most sensitive information about young people, including academic records, learning needs, behavioral information, and personal identifiers.
When AI systems are introduced into classrooms or administrative processes, institutions must consider how student information is collected, processed, stored, shared, and potentially reused.
The Family Educational Rights and Privacy Act (FERPA) establishes important protections for student education records at covered institutions. However, compliance is not achieved simply by selecting a technology provider that claims to be secure.
Schools must also understand their own responsibilities regarding vendor oversight, access permissions, data-sharing arrangements, and appropriate uses of student information.
The U.S. Department of Education's Student Privacy and Data Security Resources provide practical guidance for protecting educational information and strengthening institutional safeguards.
Beyond privacy, schools must consider the educational consequences of unsafe AI outputs. A misleading recommendation, inappropriate response, or biased interpretation could influence how students understand their abilities, opportunities, or future pathways.
For AI systems involved in academic guidance or career exploration, these concerns are especially significant. Protecting students means safeguarding both their personal information and the integrity of the guidance they receive.
Why Constrained AI Can Be Safer AI
Not every educational AI system needs unlimited conversational freedom. In fact, one of the most promising approaches to safer educational AI is to establish intentional boundaries around what a system can access, generate, recommend, and do.
Constrained AI systems can be designed to operate within defined educational objectives, approved knowledge sources, age-appropriate interactions, and established institutional policies.
Rather than relying exclusively on a general-purpose model to determine an appropriate response, developers can incorporate safeguards such as restricted data access, validated content, structured decision rules, human oversight, and ongoing monitoring.
For example, an AI-powered career exploration tool might limit its recommendations to verified educational and occupational information while avoiding unsupported judgments about a student's potential or future success. Such constraints do not eliminate risk. However, they can reduce unnecessary exposure and make system behavior more predictable, testable, and accountable.
NIST's Generative AI Risk Management Profile offers additional guidance on identifying and managing risks specific to generative AI systems. The goal is not to restrict innovation. It is to create the conditions in which innovation can be trusted.
The ESTE Lens: Engineering + Technology — Safety by Design
At ESTE Leverage, the intersection of Engineering and Technology offers an important perspective on responsible AI development.
Engineering asks how systems should be structured, tested, and maintained. Technology provides the tools to translate those designs into practical solutions. Together, these disciplines reinforce a fundamental principle. Safety is a design choice, not a feature added after deployment. For educational AI, this means considering privacy, security, explainability, and appropriate use from the earliest stages of system development.
A responsible architecture should address who can access information, how recommendations are generated, what safeguards govern interactions, and when human review is necessary.
It should also recognize that students are not simply users of technology. They are individuals whose interests, development, and future opportunities may be influenced by its outputs.
By combining engineering discipline with technological innovation, educational institutions and developers can move beyond asking what AI can do toward asking what AI should be permitted and trusted to do.
Call to Practice: Conduct a Student-Informed Digital Safety Audit
One of the most practical steps schools can take is to examine their current use of AI and digital learning tools through the lens of student safety.
Invite students, educators, administrators, and technology personnel to participate in a digital safety audit.
Begin with five questions:
Data privacy: What student information do our AI tools collect, and who can access it?
Output reliability: How do we identify inaccurate, biased, or inappropriate AI-generated responses?
Age appropriateness: Are safeguards aligned with the developmental needs of different student populations?
Accountability: Who is responsible when an AI system produces an unsafe or misleading result?
Student voice: Do students have meaningful opportunities to report concerns and help shape responsible AI practices?
Students should participate in evaluating policies and experiences without being asked to disclose sensitive information or deliberately expose themselves to harmful content.
The free TeachAI Guidance for Schools Toolkit offers adaptable resources that schools can use when developing policies and discussing responsible AI adoption. The purpose of the audit is not to discourage AI use. It is to identify gaps before those gaps become problems.
Bright Spot: Giving Students a Seat at the AI Safety Table
An encouraging direction in responsible educational technology is the growing emphasis on involving students and school communities in AI governance. Schools can build on this approach by establishing AI safety councils that include students alongside educators, technology specialists, administrators, and, where appropriate, parents.
These councils can review acceptable-use policies, discuss emerging concerns, evaluate the student experience, and recommend improvements to institutional safeguards. Student participation brings an important perspective: young people often encounter the practical benefits and limitations of educational technology differently from the adults responsible for selecting and managing it.
The U.S. Department of Education's free Empowering Education Leaders: A Toolkit for Safe, Ethical, and Equitable AI Integration provides a useful foundation for collaborative decision-making and responsible implementation. By giving students a meaningful voice in AI safety discussions, schools can strengthen digital literacy, encourage responsible technology use, and build a culture of shared accountability.
The best prepared institutions for AI's future will be those that treat safety as a shared responsibility and not merely a technical requirement.
Looking Ahead: Trust Must Be Engineered
Artificial intelligence has the potential to transform education, but its long-term value will depend on whether students, families, and educators can trust the systems they use. Security, privacy, accuracy, and accountability are not obstacles to innovation. They are essential foundations for sustainable adoption.
As AI becomes more deeply embedded in K–16 learning environments, educational leaders must look beyond the excitement of new capabilities and examine the safeguards that make those capabilities appropriate for students. The future of educational AI should not be defined only by how much technology can do. It should be defined by how responsibly we design it to serve the people who depend on it.
Call to Action
Make AI safety a shared priority. Bring students, educators, and technology leaders together to review your institution's AI tools, identify potential risks, and establish clear safeguards. Start with a student-informed digital safety audit and turn the findings into an actionable plan for safer, more trustworthy AI in education.
ESTE® Leverage - founded in the belief that Entrepreneurship, Science, Technology, and Engineering are innate in each of us - grounded in the science of learning & assessment - dedicated to the realized potential in every individual.