Prompt injection detection (simplified - only pattern matching)

[dorien-koelemeijer]

Overview

This PR implements prompt injection detection using pattern-matching techniques, providing users with security warnings when potentially malicious tool calls are detected.

Working design doc with requirements

These notes are a result of pairing sessions with Douwe: https://docs.google.com/document/d/1OWh7Ab_eu8STaoplPA6AKF6AnXQNqXc8JYiAwmUYDTs/edit?tab=t.0

Implementation approach

Implemented pattern-based detection for prompt injection, scanning tool calls for potentially dangerous commands. Pattern-matching provides immediate protection while laying groundwork for future ML-based scanning with BERT models, which will be the next iteration of this work.

Security scanning was added in the reply_internal() method after permission checking, leveraging the existing tool approval workflow. The security scanner receives both proposed tool calls and full conversation history for contextual threat assessment (using BERT models and potentially other tools). In this iteration, the full conversation history is not considered, but we will do that once model scanning is integrated.

Security can be enabled/disabled via security.enabled config parameter with configurable confidence thresholds

Key changes

crates/goose/src/security/: New security module with scanner setup and patterns. Can be extended to support model-based scanning.
agent.rs: Added apply_security_results_to_permissions() to move security-flagged tools from approved to needs_approval
tool_execution.rs: Enhanced handle_approval_tool_requests_with_security() to include security warnings in confirmation prompts
ToolCallConfirmation.tsx: Added support for custom security warning prompts
GooseMessage.tsx: Updated to pass security context from backend to UI components

Configuration

security:
  enabled: true
  confidence_threshold: 0.5  

Future work

• Scan context using BERT model(s) - do some more research on what model/models we want to use + if there's a workaround for the token limit that these models can ingest.
• Integrate security scanning into ToolMonitor
• Make sure we cover all different avenues for prompt injection (files and images, google drive links, recipes, MCP tool results, etc)

[DOsinga]

sorry got to this late - already day for you so sending you what I have

[DOsinga]

did you mean to delete this?

[dorien-koelemeijer]

We want to make sure users can't always allow a tool call. I wasn't entirely sure whether you had mentioned to take it away entirely, or if we just want to take away that option for these security findings. Let me know what you think the better way is.

[michaelneale]

I think we want to leave that in, as it is a common mode

[dorien-koelemeijer]

Will revert these changes, thanks for clarifying

[DOsinga]

As discussed before, we need to move this all out into its own infra rather than having the same logic for all the inspectors that decide whether to allow for a tool and then do the mixing

[dorien-koelemeijer]

Have tried my best to address this comment, let me know if I've fully misunderstood what you meant here 🫠

[DOsinga]

are we using this?

[dorien-koelemeijer]

We're not, getting rid of this, probably copied that over from the previous branch.

[DOsinga]

this code is duplicated from above. also I don't think we use .context. we use other things though