The increasing sophistication of artificial intelligence (AI) chatbots is raising alarms among military experts, who caution that these systems are vulnerable to exploitation. Hackers from nations like China and Russia are reportedly using AI to manipulate these chatbots into actions such as data theft and misinformation dissemination.
AI chatbots rely on large language models to interpret and respond to user inputs. However, these models have a critical flaw: they cannot differentiate between legitimate user prompts and malicious instructions. As Liav Caspi, a former cyberwarfare officer in the Israel Defense Forces, explained, “The AI is not smart enough to understand that it has an injection inside, so it carries out something it’s not supposed to do.” Caspi, who co-founded Legit Security, highlighted a security flaw discovered in Microsoft’s Copilot chatbot that illustrates this vulnerability.
Adversaries could potentially manipulate these chatbots into executing harmful actions, such as altering records or skewing decisions. “It’s like having a spy in your ranks,” Caspi remarked. With hackers already leveraging AI tools like Google’s Gemini and OpenAI’s ChatGPT to generate malware and create fake identities, the threat of prompt injection attacks is growing.
According to Microsoft’s annual digital defense report, AI systems have become significant targets, with prompt injection attacks rising. However, solutions to this problem are elusive. Security researchers and companies like OpenAI acknowledge the difficulty in countering these threats.
The attacks often involve embedding hidden malicious instructions within text that the chatbot processes, such as blog posts or PDFs. Recent examples include a successful prompt injection attack on OpenAI’s AI-based browser, ChatGPT Atlas, and a vulnerability in Microsoft’s Copilot that could have exposed sensitive data.
Microsoft is proactively working to mitigate these risks by continuously testing Copilot for vulnerabilities, blocking exploit attempts, and monitoring for suspicious behavior. “Microsoft ensures its generative AI systems remain resilient against evolving threats for all our customers, including defense and national security,” the company stated.
Dane Stuckey, OpenAI’s chief information security officer, emphasized the ongoing challenge, stating that “prompt injection remains a frontier, unsolved security problem.” To mitigate risks, Caspi suggests limiting an AI assistant’s access to sensitive data and restricting user access to other organizational data.
The U.S. Army, for instance, is employing tools like Ask Sage, which restricts AI model access to specific data and isolates Army information from external sources. This approach aims to contain potential attacks, likened by Caspi to a scenario where an insider threat is confined to a single room.
Despite these defenses, the speed and sophistication of AI make it a formidable adversary. Andre Slonopas of the Virginia Army National Guard highlighted the challenge, stating that AI’s rapid processing capabilities make it difficult for humans to counter without accessible and affordable defensive AI solutions.
Army contractors are also under threat from state-sponsored AI attacks, with China being particularly aggressive. Nicolas Chaillan, founder of Ask Sage and former chief software officer for the U.S. Air Force and Space Force, noted the frequency of these attacks, though his company has successfully thwarted them.
In the complex landscape of AI-driven cyber threats, the ability to spoof languages and identities further complicates attribution. A military official noted that technologies like ChatGPT enable users to operate in languages they do not speak, facilitating the potential for cross-national impersonation among cybercriminals.
