Robustness Testing for Large Language Models
This challenge focuses on developing and implementing methods to test the robustness of large language models (LLMs) against prompt injection attacks. Participants will create a system to identify and mitigate vulnerabilities to attacks that manipulate the LLM's behavior via specially crafted prompts. This is crucial for ensuring the safety and reliability of deployed AI systems.
What you are building
The core problem, expected build, and operating context for this challenge.
Develop and implement methods to test the robustness of LLMs against prompt injection attacks.
How work is evaluated
Evaluation will be based on the effectiveness of the designed techniques in detecting and mitigating prompt injection attacks. The ability to generalize to different LLMs will also be assessed.
Challenge datasets
Public datasets, plus the private datasets you have access to.
What you should walk away with
Document findings and propose improvements to LLM security.
Design and implement techniques to detect and mitigate prompt injection attacks.
Understand the concept of prompt injection attacks in LLMs.
Evaluate the effectiveness of the developed techniques.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
Evaluation will be based on the effectiveness of the designed techniques in detecting and mitigating prompt injection attacks. The ability to generalize to different LLMs will also be assessed.
Challenge input
A sequence of prompts, some benign and some malicious.
Agent execution
No agent tool is listed for this challenge.
Evaluated output
A list indicating which prompts are potentially malicious.
- No evaluation checks are configured.
- Detection Accuracy target: 0.9
- Python execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- No evaluation checks are configured.
Metrics
- Detection Accuracy · Target 0.9
- Mitigation Success Rate · Target 0.8
Workflow tasks
- Task 1
Input: A sequence of prompts, some benign and some malicious.
Output: A list indicating which prompts are potentially malicious.
Execution harness: Python
- Task 2
Input: A set of malicious prompts and an LLM.
Output: The LLM's response after applying the mitigation techniques.
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.