Risk
Insecure System Prompt Design
Description
Poorly designed system prompts that are easily bypassed, manipulated (jailbreaking), or that inadvertently leak sensitive contextual information or instructions.
Example
A system prompt for an LLM includes internal API endpoint details that a user extracts via a crafted query.
Assets Affected
System Prompt
Mitigation
- Employ robust prompt engineering techniques
- Sanitize user inputs intended for prompts
- Minimize sensitive data in prompts
- Iteratively test prompts for vulnerabilities
- Document prompt design and rationale
Standards Mapping
- ISO 42001: A.6.2.3, A.8.2
- OWASP Top 10 for LLM: LLM07
- NIST AI RMF: MAP 2.2, MEASURE 2.9
- DASF v2: MODEL SERVING 9.1