Detecting prompt injection

• Pre-filter regex: block "ignore previous", "system prompt", "<|im_start|>" — naive but catches 60% of attempts
• Embedding filter: cosine similarity of user input to a known-jailbreak corpus; threshold 0.85
• Post-filter LLM judge: second call with "Is this response unsafe / off-topic?"
• Output canary token: embed a unique string in the system prompt, check for it in responses (leak = jailbreak)
• Heartbeat monitor on /chat endpoint: response P95 normal but spike in blocked_count → active attack

# Simple pre-filter in Python
import re

BLOCKED = [
    r'ignore (?:previous|prior|all) (?:instructions|prompts)',
    r'system\s*prompt',
    r'<\|im_start\|>',
    r'\bDAN\b',  # 'Do Anything Now' jailbreak
    r'jailbreak',
]

def is_suspicious(text: str) -> bool:
    t = text.lower()
    return any(re.search(p, t) for p in BLOCKED)

# In production: log_to_enterno_heartbeat('blocked' if is_suspicious(input) else 'ok')
# Alert: > 10 blocked/min over the last 5 min → notify Slack

• What is prompt injection
• LLM security monitoring

To detect prompt injection in a Large Language Model (LLM) application, implement input validation and sanitization, monitor API usage for anomalies, and utilize anomaly detection algorithms. Regularly review user inputs and model outputs for unexpected behavior and consider integrating tools like OpenAI's safety features to flag potential injections.

Prompt injection occurs when an attacker manipulates the inputs to an LLM to alter its behavior. This can result in unauthorized data access, incorrect outputs, or triggering unintended actions. Understanding the mechanics of prompt injection is crucial for developing effective detection strategies.

LLMs process inputs as prompts, often without strict validation. An attacker can craft inputs that exploit this openness, leading to unintended consequences. For example, if an LLM is asked to generate a summary of a document, an attacker might include instructions within the input that cause the model to divulge sensitive information instead.

Common techniques for prompt injection include:

• Command Injection: Inserting commands that the LLM interprets as valid instructions.
• Context Manipulation: Crafting inputs that alter the context in which the LLM operates.
• Input Chaining: Combining multiple inputs to create a complex instruction set that the LLM cannot parse correctly.

To mitigate these risks, it's important to implement robust monitoring and validation systems that can identify suspicious patterns in inputs.

To effectively detect prompt injection in your LLM application, consider the following strategies:

1. Input Validation and Sanitization

Implement strict validation rules for user inputs. Use libraries such as validator.js in Node.js applications to sanitize inputs before they reach your LLM.

const validator = require('validator');

function sanitizeInput(input) {
    return validator.escape(input);
}

2. Anomaly Detection

Utilize machine learning models to analyze patterns in API usage. For instance, if you notice a sudden spike in requests with unusual parameters, flag these for review. Tools like TensorFlow or Scikit-Learn can help you build models that detect anomalies based on historical usage.

3. Monitoring and Logging

Implement comprehensive logging of all user inputs and model outputs. This allows you to trace back any anomalous behavior to specific user sessions. Use tools like Loggly or Splunk to manage and analyze your logs effectively.

4. Regular Audits

Conduct regular audits of your LLM application. Review logs for any signs of prompt injection attempts, such as unusual input patterns or outputs that deviate from the expected norms. Create a checklist for these audits that includes:

• Review of input patterns for anomalies
• Analysis of model output consistency
• Assessment of user behavior

By implementing these strategies, you can significantly reduce the risk of prompt injection in your LLM applications, ensuring that both your model and your users remain secure.