How do you attack AI, and how do you protect it?
AI security.
AI is a tool, an attack surface and a weapon at the same time. Anyone who uses it must know all 3 sides.
In 60 seconds
- AI is a tool, attack surface and weapon at the same time.
- The OWASP list names the ten most important risks for language model applications.
- Minimal permissions, separated instructions and data, logs and tests are effective.
Legal status: 5 November 2026
3 sides of the same coin.#
Attacks on AI systems
Language models and their applications can be manipulated, spied on or poisoned. New vulnerabilities that classic security does not cover.
Attacks with AI
Attackers use AI for convincing phishing emails, voice and video clones and automated searches for vulnerabilities.
Misuse and shadow AI
Well-intentioned use without rules: confidential data ends up in private tools, results are adopted unchecked.
The attack surface of an AI application.#
6 places where things can go wrong.
The ten biggest risks (OWASP).#
The list of the OWASP GenAI Security Project for applications with language models, 2025 edition.
| No. | Risk | What happens | What helps |
|---|---|---|---|
| LLM01 | Prompt injection | External text changes the model's behaviour, directly or hidden in documents and websites | Separate input and sources, minimise permissions, require confirmation for risky actions |
| LLM02 | Disclosure of sensitive information | The model reveals personal data, secrets or other people's data | Classify and minimise data and filter output |
| LLM03 | Supply chain | Manipulated models, datasets, libraries or plugins | Check sources, pin versions, inventory components |
| LLM04 | Data and model poisoning | Manipulated training or knowledge data changes the behaviour | Check data provenance, validate data, test models |
| LLM05 | Improper output handling | Answers are executed unchecked as code, commands or requests | Treat and validate output like external input |
| LLM06 | Excessive permissions | An agent is allowed to do more than its task requires | Minimal permissions, approvals, limits and logs |
| LLM07 | System prompt leakage | The application's hidden instructions are coaxed out | No secrets in instructions, do not rely on secrecy for protection |
| LLM08 | Vector and embedding weaknesses | Attacks on knowledge bases (RAG), for example through external content or permission problems | Enforce access rights in the knowledge base, check sources |
| LLM09 | Misinformation | False, invented or misleading answers are believed | Show sources, cross-check, human oversight |
| LLM10 | Unbounded consumption | Mass requests or loops drive up costs and bring services down | Limits, budgets, monitoring and rate limiting |
Source: OWASP GenAI Security Project, Top 10 for LLM Applications 2025, summarised independently and simplified. The list is being developed further. Current version at genai.owasp.org.
Attacks with AI.#
The barrier to fraud is falling. Procedures beat appearances.
AI phishing
Flawless, personally tailored emails in any language. Typical spelling mistakes no longer serve as a warning sign.
Voice and video clones
A few seconds of audio are often enough to imitate a voice. Popular for CEO fraud and grandparent scams.
Disinformation
Fake images, videos and news spread quickly and are hard to refute.
Automated attacks
AI helps find vulnerabilities faster, generate deceptive texts and adapt attacks.
Countermeasure: procedures
Call back on a known number, code word, four-eyes principle for payments, MFA and passkeys.
Countermeasure: awareness
Training with current examples. Pressure, secrecy and haste are warning signs, regardless of the technology.
Shadow AI.#
When use comes before the rules.
Shadow AI arises when employees use AI tools without approval, usually out of pragmatism: the approved tool is missing or inconvenient. The risk lies less in intent than in loss of control.
- Confidential data ends up in private accounts without a contract.
- Nobody knows which AI is used for what.
- Data protection, copyright and the AI Act are violated unnoticed.
What really helps
- Offer an approved alternative that is at least as convenient.
- Clear rules with the data traffic light instead of general bans.
- Training on benefits and risks.
- Technical guardrails such as data loss prevention and access control.
- An open culture: whoever reports use is not punished.
Protection in 6 layers.#
Governance and inventory
Which AI exists, who is responsible, which risk class applies?
Access and identity
SSO, MFA and minimal permissions for AI tools and agents.
Data and data protection
Classification, data traffic light and protection against leakage.
Application security
Check input and output, no secrets in instructions, sandbox for actions.
Supply chain
Check and inventory models, libraries, plugins and data sources.
Monitoring and testing
Logs, abuse detection and regular attack tests (red teaming).
Check: AI security in the company.#
6 questions about your AI landscape.
Rough guidance, not an audit. Your selection stays in your browser.
Frequently asked questions.#
Can prompt injection be fully prevented?
Not reliably as things stand. So the rule is: limit the risk instead of hoping. Minimal permissions, separating instructions from data, confirmation for risky actions and never executing output unchecked.
Are locally run models more secure?
They prevent data from going to a provider and give you more control. In return, operation, updates and the supply chain are your responsibility. Security comes from operation, not from location alone.
Can I spot deepfakes?
Sometimes, but only reliably with procedures: call back, code word, cross-check via a second channel. Never rely on what you see or hear alone.
Do I need separate AI security alongside IT security?
It builds on IT security (identities, network, data, monitoring) and adds AI-specific risks. Both belong together: IT security, Monitoring, NIS2.
Setting up AI securely?
Tell me which AI scenario you want to secure, and we will add to the content.
Sources and further reading
Was this page helpful?
Found a mistake? Let me know.
Suggested citation: Russo, K.: "AI security". sicherheitsdenker.de/en/ki-sicherheit/, as of 6 November 2026.