AI-Powered Surveillance in Schools: A Double-Edged Sword
In a growing number of schools across the United States, artificial intelligence is being used to monitor students’ online activity around the clock. Designed to identify potential threats and provide support during a mental health crisis, this technology, while promising to improve student safety, is also raising significant concerns about privacy and its unintended consequences.
Instances of the technology’s use have revealed that students are using school-issued devices for far more than just schoolwork, including coping with personal struggles. Students have written about depression, heartbreak, suicide, addiction, bullying, and eating disorders. They are also composing poems, writing college essays and participating in online role-playing sessions.
Vancouver Public Schools in Washington state, like many other districts nationwide, has been using AI-powered surveillance software. The aim is to detect signs of danger or distress in students’ online communications.
However, the use of this technology has led to serious privacy breaches. Reporters from the Seattle Times and the Associated Press inadvertently gained access to nearly 3,500 unredacted student documents through a records request about the district’s surveillance technology. Experts warned that this was a massive security risk. These documents included sensitive personal information that would have been otherwise private.
The Education Reporting Collaborative, a coalition of eight newsrooms, is investigating the unintended consequences of AI-powered surveillance at schools.
How the Surveillance System Works
The software, like Gaggle Safety Management, uses machine-learning algorithms to scan what students search or write online. If the algorithm detects potential indicators of problems, such as bullying or self-harm, it flags them for review. Human reviewers then assess the alerts and contact school officials if necessary. For immediate threats, the company contacts school officials directly. In rare cases, law enforcement may be involved.
The cost of this kind of technology is not insignificant. The contract that Vancouver signed in the summer of 2024 cost $328,036 for three school years — approximately the cost of employing one extra counselor.
A Vancouver school counselor, who asked to remain anonymous, says they receive three or four student Gaggle alerts per month. In about half the cases, the district contacts parents immediately. “A lot of times, families don’t know. We open that door for that help,” the counselor stated.
Unintended Consequences
While the technology can help identify students in need, there are serious downsides. For example, LGBTQ+ students are particularly vulnerable. The released documents revealed that at least six students were potentially outed to school officials after writing about being gay, transgender, or struggling with gender dysphoria. LGBTQ+ students may be more likely to suffer from depression and thoughts of suicide, often turning to the internet as a form of support. Katy Pearce, a University of Washington professor who researches technology in authoritarian states, notes, “We know that gay youth, especially those in more isolated environments, absolutely use the internet as a life preserver.”
In another case, after North Carolina’s Durham Public Schools piloted Gaggle in 2021, community members raised concerns when a student was outed to their family, who were not supportive, after an alert about self-harm triggered by a teacher’s assignment. Glenn Thompson, a Durham School of the Arts graduate, spoke at a board meeting during his senior year, decrying the lack of transparency. “You can’t just (surveil) people and not tell them. That’s a horrible breach of security and trust,” said Thompson, now a college student, in an interview.
Despite the intentions, many alerts turn out to be false alarms, such as a student essay about consent or a casual conversation between friends. During the 2023-24 school year, Owasso schools received nearly 1,000 Gaggle alerts, with 168 for harassment and 281 for suicide. Russell Thornton, the district’s technology executive director, has noted that one tool can’t solve all the world’s problems, when asked why bullying still occurs despite the surveillance.
Parental Awareness and Concerns
Parents are often unaware of such surveillance technology. Even when they are aware, they may not have an option to opt out. Some parents fear that their children are being monitored, and that their data is being tracked. Tim Reiland, the parent of two teenagers, was unaware of his district’s use of Gaggle until he inquired about allowing his daughter to use her personal laptop for privacy reasons. His request was denied. Reiland’s daughter, Zoe, says she felt so “freaked out” that she stopped Googling anything personal on her school-issued Chromebook.
Weighing the Benefits vs. Risks
While schools continue to use surveillance technology, its long-term effects on student safety have yet to be determined. There is no conclusive research showing that it measurably lowers suicide rates or reduces violence. A 2023 RAND study found “scant evidence” of either benefits or risks from AI surveillance, concluding that more comprehensive research is needed.
Benjamin Boudreaux, an AI ethics researcher, argues that simply issuing more alerts will not improve matters. “If you don’t have the right number of mental health counselors, issuing more alerts is not actually going to improve suicide prevention,” he said.
In some instances, surveillance technology has helped to avert tragedies. For example, Highline School District in Washington’s former Superintendent Susan Enfield noted that a middle school student who was potentially being trafficked used the system to communicate with staff members. “They knew that the staff member was reading what they were writing,” Enfield said. “It was, in essence, that student’s way of asking for help.”
However, there are concerns that constant surveillance may hinder the necessary space that teens need to explore their thoughts and seek support. Boudreaux stated, “The idea that kids are constantly under surveillance by adults — I think that would make it hard to develop a private life, a space to make mistakes, a space to go through hard feelings without adults jumping in.”
There are undeniable tensions between the potential benefits of online surveillance and the risks to student privacy and well being. It remains vital to have an open conversation about these potentially life saving tools that could cause more harm than good. Schools must carefully determine how best to leverage technology while safeguarding students’ rights to privacy and fostering a supportive environment.
