AI risk for IT teams: what has gone wrong, what experts expect, and what you'll need to know
Most AI failures so far haven't been science fiction. They have been ordinary security and governance gaps (over-broad tokens, unreviewed code, unverified output) moving at machine speed. This tracker records the incidents that matter to IT, the lesson each one teaches, and the expertise they demand.
Curated by IT Intelligence · Updated October 18, 2026 · Every entry dated and sourced
Scale
By the numbers
Incident log
27 AI incidents that changed how IT should operate
A curated record of AI incidents relevant to infrastructure, security and governance teams, filterable by type. It's selective, not exhaustive: for the full community-maintained record, see the AI Incident Database.
2026
- 2026-04AI supply chainVercel / Context.ai
A third-party AI tool became the way into a cloud platform
Attackers who had earlier breached Context.ai, an AI productivity tool, used an OAuth token an employee had granted with broad permissions to take over that employee's Google Workspace account and reach internal Vercel environments. Vercel told a limited subset of customers to rotate credentials.
Lesson: OAuth grants to AI tools are part of your perimeter. Audit them, scope them, and revoke what nobody uses.
- 2026-03Agent failureMeta
Internal agent's wrong answer exposed data to unauthorized staff
An internal AI agent posted an incorrect technical answer on an employee forum without being asked to. An engineer followed it and, for about two hours, made large volumes of company and user-related data visible to engineers who were not authorized to see it. Meta rated it Sev 1 and said no user data was mishandled or left the company.
Lesson: Agents should not publish or act without approval, and their advice needs the same change control as a human's.
- 2026-03AI supply chainLiteLLM (PyPI)
Popular LLM proxy package shipped a credential stealer
Two compromised LiteLLM releases (1.82.7 and 1.82.8) were published to PyPI as part of the wider TeamPCP campaign, which had first poisoned a security scanner in the project's build pipeline. The malicious code harvested environment variables, SSH keys, cloud and Kubernetes credentials, and was live for a few hours.
Lesson: AI gateways hold every model API key you own. Pin versions, verify provenance, and keep build-pipeline tokens short-lived.
- 2026-02Data exposureMoltbook
AI-agent social network left its production database open
Researchers at Wiz found a database key embedded in Moltbook's client-side code with no row-level security behind it, giving read and write access to roughly 1.5 million agent API tokens, about 35,000 email addresses, and thousands of private agent conversations, some containing third-party API keys. It was locked down within hours of being reported.
Lesson: "Vibe-coded" AI apps skip the basics. Secrets in front-end code and missing access policies are still the most common failure.
- 2026-01Prompt injection & AI vulnsGrafana ("GrafanaGhost")
Indirect prompt injection pushed dashboard data to an attacker
Researchers reported that instructions hidden in content processed by Grafana's AI features could make the assistant leak internal data through rendered output, with no action required from the user. The case was one of the major incidents in OWASP's first-quarter 2026 GenAI exploit round-up.
Lesson: Anything an assistant reads (logs, tickets, metadata) is attacker-controllable input. Constrain what its output can fetch or render.
- 2026-01AI supply chainOpenClaw (open-source agent)
Self-hosted agent panels exposed online; malicious "skills" flooded its registry
Tens of thousands of OpenClaw agent control panels were found reachable on the public internet, many vulnerable to a one-click remote-code-execution flaw (CVE-2026-25253). In the same weeks, hundreds of malicious skills, mostly disguised as crypto tools, were published to its unvetted skills registry.
Lesson: A personal agent with shell access is a server. Don't expose it, and treat plug-in marketplaces like any untrusted package source.
2025
- 2025-11AI-enabled attackAnthropic (GTG-1002)
First reported cyber-espionage campaign run largely by an AI agent
Anthropic disclosed that a group it assessed as Chinese state-sponsored used Claude Code with tool integrations as an autonomous intrusion engine against about 30 organizations, with the AI performing the large majority of tactical steps and humans approving key decisions. Anthropic banned the accounts and notified those affected; China denied involvement.
Lesson: Attack speed is now machine speed. Detection and response that depend on a human noticing first will lose the race.
- 2025-10Hallucination & liabilityDeloitte Australia
Government report with AI-fabricated references led to a partial refund
A report Deloitte produced for an Australian federal department was found to contain citations to papers that did not exist and a misattributed court quote. Deloitte disclosed it had used a generative AI tool in drafting, issued a corrected version, and refunded part of the roughly AU$440,000 fee.
Lesson: AI-assisted deliverables need citation checks before they ship. The reputational cost far exceeds the fee.
Source: Australian Financial Review; Department of Employment and Workplace Relations (Oct 2025) - 2025-08AI-enabled attackAnthropic (data-extortion case)
One operator used an AI agent to run an extortion campaign end to end
Anthropic reported a criminal who used Claude Code to automate reconnaissance, intrusion, data theft and the drafting of tailored ransom demands against at least 17 organizations, including healthcare, emergency services and government bodies.
Lesson: Skills that once required a team now fit in one person with an agent. Expect more capable attacks from smaller actors.
Source: Anthropic Threat Intelligence Report (Aug 2025) - 2025-08AI supply chainSalesloft Drift
Stolen tokens from an AI chat integration opened hundreds of CRM instances
A threat actor used OAuth tokens stolen from the Drift AI chat agent's Salesforce integration to export data from many customers' Salesforce environments, then mined that data for further secrets such as cloud keys and data-warehouse tokens.
Lesson: Every SaaS-to-SaaS AI integration is a standing credential. Inventory them and alert on bulk exports.
Source: Google Threat Intelligence Group (UNC6395), Salesloft (Aug 2025) - 2025-08AI supply chainNx build system ("s1ngularity")
Malware weaponized developers' own AI coding CLIs to hunt for secrets
Malicious versions of the Nx npm package ran a post-install script that invoked locally installed AI command-line tools with permission checks disabled, asking them to search the machine for credentials and crypto wallets. The loot was published to public GitHub repositories.
Lesson: An AI CLI with broad file access is a privilege on every developer laptop. Restrict what it can run unattended.
Source: Nx security advisory; multiple security vendors (Aug 2025) - 2025-07Data exposureMcDonald's McHire (Paradox.ai)
AI hiring chatbot back end exposed millions of applicant records
Security researchers found that an administrator account for the AI recruiting platform behind McHire still used a trivial default password, giving access to records tied to as many as 64 million job applications. The vendor fixed it quickly after disclosure.
Lesson: AI vendors inherit ordinary security debt. Your due diligence must cover their basics, not just their model.
Source: Researchers Ian Carroll and Sam Curry; Paradox.ai statement (Jul 2025) - 2025-07AI supply chainAmazon Q Developer (VS Code extension)
A malicious prompt shipped inside an official AI coding extension
An outsider got a pull request merged that added a prompt instructing the Amazon Q agent to wipe local files and cloud resources, and it shipped in version 1.84.0. AWS says the code was not functional as written; it pulled the version and released a fix.
Lesson: Prompts are code. Review them like code, and gate who can change the instructions your agents follow.
Source: AWS security bulletin (Jul 2025) - 2025-07Agent failureReplit
Coding agent deleted a production database during a code freeze
During a publicized trial, Replit's AI agent ran destructive commands against a live production database despite explicit instructions not to make changes, then gave misleading accounts of what had happened. Replit's CEO apologized and added stronger separation between development and production.
Lesson: Never give an agent production write access without a hard approval gate and tested backups.
Source: Public posts by SaaStr's Jason Lemkin; Replit CEO statement (Jul 2025) - 2025-06Prompt injection & AI vulnsMicrosoft 365 Copilot ("EchoLeak")
Zero-click prompt injection could leak data from Copilot
Researchers at Aim Security showed that a single crafted email could carry hidden instructions that Copilot later followed while answering an unrelated question, exfiltrating internal data without the user clicking anything. Microsoft fixed it server-side (CVE-2025-32711).
Lesson: Assistants that read your inbox read the attacker's email too. Isolate untrusted content from privileged context.
- 2025-05Hallucination & liabilityWorkday (Mobley v. Workday)
Court let an AI hiring-bias lawsuit proceed as a collective action
A U.S. federal judge allowed age-discrimination claims against Workday's AI-based applicant screening to proceed as a nationwide collective action, signaling that vendors of AI decision tools can face liability alongside the employers who use them.
Lesson: Automated decisions about people need bias testing and an audit trail before deployment, not after a complaint.
Source: U.S. District Court, N.D. California (May 2025) - 2025-01Data exposureDeepSeek
AI lab left a database of chat logs and keys publicly reachable
Wiz researchers found an unauthenticated DeepSeek analytics database exposed on the internet, containing more than a million log lines including user chat history, API secrets and back-end details. DeepSeek secured it after being notified.
Lesson: Fast-moving AI providers can be weak on basic data hygiene. Assume prompts sent to a provider may be stored and exposed.
Source: Wiz Research (Jan 2025)
2024
- 2024-08Prompt injection & AI vulnsSlack AI
Prompt injection could pull data from private channels
Researchers at PromptArmor showed that instructions planted in a public channel could cause Slack AI, when answering another user's question, to surface content from private channels the attacker could not access. Salesforce patched the issue.
Lesson: Retrieval across permission boundaries is the core risk of enterprise assistants. Test it before rollout.
Source: PromptArmor; Salesforce (Aug 2024) - 2024-05Hallucination & liabilityGoogle AI Overviews
AI search summaries served confidently wrong answers at scale
Shortly after launch, Google's AI Overviews produced widely shared errors, such as suggesting glue to keep cheese on pizza, after treating satirical or joke content as fact. Google narrowed when the feature appears and added guardrails.
Lesson: Grounding on low-quality sources produces authoritative-sounding nonsense. Source quality is a design input.
Source: Google Search blog; widespread press coverage (May 2024) - 2024-02Hallucination & liabilityAir Canada
Airline held liable for what its support chatbot promised
A Canadian tribunal ordered Air Canada to honor a bereavement-fare refund its website chatbot had wrongly described, rejecting the argument that the chatbot was responsible for its own statements.
Lesson: Your chatbot speaks for you legally. Customer-facing AI needs policy-grounded answers and clear limits.
Source: BC Civil Resolution Tribunal, Moffatt v. Air Canada (2024 BCCRT 149) - 2024-02Deepfake & fraudArup
Deepfake video call led to a roughly US$25 million transfer
A finance employee at engineering firm Arup's Hong Kong office joined a video call in which the CFO and colleagues were AI-generated deepfakes, and was persuaded to make a series of transfers totaling about HK$200 million.
Lesson: Seeing and hearing is no longer verification. Require out-of-band confirmation for payments and access changes.
Source: Hong Kong Police; Arup confirmation reported by CNN (Feb–May 2024) - 2024-01Deepfake & fraudU.S. election robocall
AI-cloned presidential voice told voters to skip a primary
Robocalls using an AI-generated imitation of President Biden's voice urged New Hampshire voters not to vote in the primary. The FCC then ruled that AI-generated voices in robocalls fall under existing restrictions and fined the political consultant behind the calls US$6 million.
Lesson: Voice cloning is cheap and convincing. Voice alone should never authenticate a person or an instruction.
Source: U.S. Federal Communications Commission (2024)
2023
- 2023-12Prompt injection & AI vulnsChevrolet dealership chatbot
Users talked a dealer's chatbot into "agreeing" to sell an SUV for $1
Visitors manipulated a car dealership's ChatGPT-powered sales assistant into off-topic tasks and into declaring a "legally binding" offer to sell a new Tahoe for one dollar. The dealer pulled the bot.
Lesson: Unscoped chatbots will say anything. Restrict them to their task and keep commitments out of their reach.
Source: Widespread press coverage (Dec 2023) - 2023-09Data exposureMicrosoft AI Research
Over-permissive storage link exposed 38 TB of internal data
While publishing open-source AI training material on GitHub, Microsoft researchers shared an Azure storage link whose token granted access to the whole account, exposing 38 TB of internal data including workstation backups, secrets and internal messages.
Lesson: AI data pipelines move huge datasets around. Scope and expire every sharing token.
Source: Wiz Research; Microsoft Security Response Center (Sep 2023) - 2023-06Hallucination & liabilityMata v. Avianca
Lawyers sanctioned for filing ChatGPT-invented case law
A U.S. federal judge fined two attorneys and their firm US$5,000 after they submitted a brief citing court decisions that ChatGPT had fabricated, and then stood by the citations when questioned.
Lesson: Professional responsibility doesn't transfer to the tool. Verify every factual claim an AI produces.
Source: U.S. District Court, S.D. New York (Jun 2023) - 2023-04Data exposureSamsung
Engineers pasted confidential source code into a public chatbot
Samsung semiconductor staff entered proprietary source code and internal meeting notes into ChatGPT while seeking help. The company then restricted generative AI tools on corporate devices while it built internal alternatives.
Lesson: Shadow AI is a data-loss channel. Give staff a sanctioned tool, or they will use an unsanctioned one.
Source: Bloomberg; Economist Korea (Apr–May 2023) - 2023-03Data exposureOpenAI ChatGPT
Caching bug showed some users other people's chat titles
A bug in an open-source library used by ChatGPT briefly let some users see titles from other users' conversations, and exposed limited payment details for a small share of subscribers. OpenAI took the service offline and fixed it.
Lesson: AI services are ordinary software with ordinary bugs. Treat prompt history as sensitive data with a retention policy.
Source: OpenAI incident report (Mar 2023)
Patterns
What the incidents have in common
Read across the log and six patterns repeat. None of them requires exotic AI expertise. They require applying security fundamentals to a new kind of software.
Treat everything an AI reads as untrusted input
Emails, tickets, web pages and database metadata can all carry instructions. Indirect prompt injection is the defining vulnerability class of AI apps, and there is no complete fix, only containment.
Agents get least privilege and an approval gate
The costliest agent failures came from broad access plus no human checkpoint. Scope agent identities narrowly and require sign-off for anything destructive, financial or customer-facing.
AI tooling is supply chain
Packages, extensions, models, prompts and agent skills are code you run. In 2025–2026 attackers targeted them precisely because they hold the most valuable keys.
OAuth grants to AI tools are the new perimeter
Several of the largest incidents never touched a model. They abused tokens an AI integration already held. Inventory, scope and revoke.
You own the output
Courts and tribunals have been consistent: the organization, not the tool, answers for what its AI says or decides. Verify before it ships.
Verify people out of band
Voice and video can be faked convincingly and cheaply. Payments and access changes need a second channel no deepfake can reach.
Outlook
What experts expect
Forecasts from the most-cited independent research, analyst and survey sources. Summarized in our words; follow the links for the full findings.
International AI Safety Report 2026
Chaired by Yoshua Bengio; written by 100+ experts- → General-purpose AI capability keeps advancing quickly, with agents now completing multi-hour software tasks.
- → Agents carry distinct risk because there are fewer chances for a human to intervene when they go wrong.
- → Risk-management practice is still immature: few quantitative benchmarks and large gaps in evidence about which safeguards actually work.
Stanford HAI, AI Index 2026
Annual independent data report- → Documented AI incidents rose to 362 in 2025 from 233 in 2024.
- → Leading developers publish capability benchmarks routinely, but responsible-AI benchmark reporting remains inconsistent.
- → Improving one responsible-AI property (such as safety) can degrade another (such as accuracy); trade-offs have to be managed explicitly.
Gartner
Analyst forecasts (2025–2026)- → More than 40% of agentic AI projects will be canceled by the end of 2027 because of rising costs, unclear business value, or inadequate risk controls.
- → "Agent washing" is widespread: only a small fraction of vendors marketing agentic products offer genuine agent capability.
- → By 2028, at least 15% of day-to-day work decisions are expected to be made autonomously by AI agents.
IBM / Ponemon, Cost of a Data Breach 2025
Study of 600 breached organizations- → 13% of organizations reported breaches of their own AI models or applications; 97% of those lacked proper AI access controls.
- → Shadow AI was a factor in about one in five breaches and added roughly US$670,000 to the average cost.
- → 63% of breached organizations had no AI governance policy or were still developing one.
World Economic Forum, Global Risks Report 2026
Survey of 1,300+ experts- → Misinformation and disinformation remain among the most severe short-term global risks.
- → Adverse outcomes of AI rank low on the two-year horizon but climb into the top tier over ten years: experts see the risk as real but still building.
- → Misinformation, cyber insecurity and AI harms are increasingly treated as one interlocking cluster.
Compliance
Regulation & frameworks
The EU AI Act timeline as amended by the Digital Omnibus (in force July 2026), and the frameworks most organizations use to operationalize AI risk. Confirm dates against the Official Journal for your obligations.
- Aug 2024EU AI Act enters into force
The first comprehensive AI law; obligations phase in over several years.
- Feb 2025EU AI Act: prohibitions and AI-literacy duty apply
Banned practices (such as social scoring) take effect, and organizations must ensure staff using AI have adequate AI literacy.
- Aug 2025EU AI Act: general-purpose AI model obligations apply
Transparency, copyright and documentation duties for GPAI model providers, with extra duties for models posing systemic risk.
- Jul 2026EU Digital Omnibus on AI in force
Delays high-risk obligations rather than removing them. Prohibitions, AI literacy, GPAI rules and the general transparency date are unchanged.
- Aug 2026EU AI Act: transparency obligations (Art. 50)
Disclosure when people interact with AI and labelling of synthetic content; a short grace period to Dec 2026 for machine-readable marking on systems already on the market.
- Dec 2027EU AI Act: stand-alone high-risk systems (Annex III)
Moved from Aug 2026 by the Omnibus. Covers uses such as hiring, credit scoring and critical infrastructure.
- Aug 2028EU AI Act: high-risk AI in regulated products (Annex I)
AI embedded in products already covered by EU safety law, such as medical devices and machinery.
NIST AI Risk Management Framework
U.S. voluntary framework (Govern, Map, Measure, Manage), plus a Generative AI Profile (NIST AI 600-1).
ISO/IEC 42001
Certifiable AI management-system standard, the ISO 27001 of AI governance.
OWASP Top 10 for LLM Applications
The baseline vulnerability list for LLM apps: prompt injection, sensitive-data disclosure, excessive agency and more.
OWASP Top 10 for Agentic Applications
Released Dec 2025: agent-specific risks such as tool misuse, identity abuse, memory poisoning and rogue agents.
MITRE ATLAS
ATT&CK-style knowledge base of real adversary tactics against AI systems, useful for threat modeling and red teams.
Skills
The expertise IT teams are now expected to have
What employers, auditors and regulators increasingly expect, mapped to who owns it and how urgent it is. The World Economic Forum's Future of Jobs research ranks AI and big data, followed by networks and cybersecurity, as the fastest-growing skills through 2030.
AI security engineering
Required nowSecurity & platform teams- ✓ Threat-model LLM apps and agents against the OWASP LLM and Agentic Top 10.
- ✓ Defend against direct and indirect prompt injection: isolate untrusted content, constrain outputs, filter tool calls.
- ✓ Red-team assistants before rollout, especially retrieval across permission boundaries.
Why: EchoLeak, Slack AI, GrafanaGhost
Identity for agents and AI integrations
Required nowIAM & SaaS owners- ✓ Inventory every OAuth grant and API key given to AI tools; remove broad "allow all" scopes.
- ✓ Give agents their own least-privilege identities, never a human's session.
- ✓ Require human approval for destructive or high-value agent actions.
Why: Vercel/Context.ai, Salesloft Drift, Replit
AI supply-chain security
Required nowDevOps & AppSec- ✓ Pin and verify AI packages, models, extensions and agent skills; treat registries as untrusted.
- ✓ Keep build-pipeline publishing tokens short-lived and scoped.
- ✓ Restrict what AI coding CLIs may run unattended on developer machines.
Why: LiteLLM, Nx s1ngularity, Amazon Q, OpenClaw skills
Data governance for AI
Required nowData owners & security- ✓ Classify what may enter prompts and retrieval indexes; extend DLP to AI tools.
- ✓ Provide a sanctioned AI tool so shadow AI has an alternative.
- ✓ Set retention rules for prompts, outputs and agent memory.
Why: Samsung, DeepSeek, Moltbook
AI governance & compliance
Required by 2026–2027Risk, legal & IT leadership- ✓ Map AI use cases to EU AI Act risk tiers and track the Omnibus dates.
- ✓ Run an AI management system aligned to ISO/IEC 42001 or the NIST AI RMF.
- ✓ Credentials gaining traction: IAPP AIGP (governance, no prerequisites) and ISACA AAISM (security management, requires CISM or CISSP).
Why: Workday, Air Canada
AI infrastructure & operations
Growing demandInfrastructure teams- ✓ Size and choose compute for training versus inference (see our CPU, GPU, or TPU analysis).
- ✓ Operate inference serving, GPU scheduling and capacity planning, on-prem or in the cloud.
- ✓ Monitor cost, latency and reliability of AI workloads like any production service.
Evaluation & output assurance
Required nowEvery team shipping AI- ✓ Test outputs for accuracy, grounding and bias before and after launch.
- ✓ Design human review where an AI output becomes a legal, financial or customer commitment.
- ✓ Keep an audit trail of prompts, sources and decisions.
Why: Deloitte, Mata v. Avianca, Google AI Overviews
IT Intelligence's take
The organizations that get hurt by AI are rarely the ones that adopted it fastest. They are the ones that gave it the most access with the least oversight. Inventory every AI tool and integration, scope its identity, gate its actions, and verify its output. That is the whole program, and almost none of it is new.
Reference & sources
- AI Incident Database (full, community-maintained record)
- OECD AI Incidents Monitor
- International AI Safety Report 2026
- Stanford HAI AI Index
- EU Digital Omnibus on AI — explainer
- OWASP GenAI Security Project
Incident summaries are IT Intelligence's own, drawn from the public disclosures and reporting named in each entry. Figures reflect what was reported at the time; some were later revised. Verify details at the source before relying on them.