AI security: prompt injection and other risks

AI features add new attack surfaces on top of traditional software security. Because a model can interpret text from users or content it reads as instructions, these risks are especially important. For general security basics, see our cybersecurity guide.

Prompt injection

Prompt injection is an attacker embedding instructions in text given to the model to change its behaviour.

• Direct: a user tries to trick the system with phrases like "ignore previous instructions".
• Indirect: an email, document or web page the model reads contains hidden instructions, especially dangerous for tool-using AI agents.

Other risks

• Data leakage: the model revealing documents or system instructions the user should not see
• Unauthorised actions: misuse of agents to send emails, delete data or start payments
• Abuse: using the product for spam, fraud or harmful content
• Cost attacks: inflating the bill through excessive use; see AI costs

Defence principles

1. Do not trust model output: treat it like untrusted user input; never execute it as code or queries directly.
2. Least privilege: give models and agents only the tools and data the task needs; let the application, not the model, enforce permissions.
3. Human approval: require user confirmation before irreversible, sensitive actions.
4. Separate instructions from data: keep system instructions clearly apart from user content and external documents; see our prompt guide.
5. Output filters: check for sensitive data, harmful content and unexpected formats.
6. Rate limiting and monitoring: detect abnormal usage patterns.
7. Logging: record prompts, tool calls and decisions auditably (mindful of personal data); see AI and KVKK.

Security testing

• Add attack scenarios to your eval set; see model evaluation.
• Run internal "red team" exercises before launch.
• Open a channel for security researchers to report issues.

Conclusion

Prompt injection cannot be fully prevented today, so the real goal is limiting the damage a successful attack can do. Products that design for security from day one earn enterprise trust; see responsible AI.