Robustness Testing for Large Language Models
This challenge focuses on developing and implementing methods to test the robustness of large language models (LLMs) against prompt injection attacks. Participants will create a system to identify and mitigate vulnerabilities to attacks that manipulate the LLM's behavior via specially crafted prompts. This is crucial for ensuring the safety and reliability of deployed AI systems.
What you are building
Develop and implement methods to test the robustness of LLMs against prompt injection attacks.
How work is evaluated
Evaluation will be based on the effectiveness of the designed techniques in detecting and mitigating prompt injection attacks. The ability to generalize to different LLMs will also be assessed.
Datasets
Public datasets, plus the private datasets you have access to.
Learning goals
Document findings and propose improvements to LLM security.
Design and implement techniques to detect and mitigate prompt injection attacks.
Understand the concept of prompt injection attacks in LLMs.
Evaluate the effectiveness of the developed techniques.
Resources
Sample data for 2 tasks
Challenge setup
Evaluation will be based on the effectiveness of the designed techniques in detecting and mitigating prompt injection attacks. The ability to generalize to different LLMs will also be assessed.
Challenge input
A sequence of prompts, some benign and some malicious.
Agent execution
No agent tool is listed for this challenge.
Evaluated output
A list indicating which prompts are potentially malicious.
- No evaluation checks are configured.
- Detection Accuracy target: 0.9
- Python execution harness
Shows the first task, up to three tools, and up to three evidence items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- No evaluation checks are configured.
Metrics
- Detection Accuracy · Target 0.9
- Mitigation Success Rate · Target 0.8
Workflow tasks
- Task 1
Input: A sequence of prompts, some benign and some malicious.
Output: A list indicating which prompts are potentially malicious.
Execution harness: Python
- Task 2
Input: A set of malicious prompts and an LLM.
Output: The LLM's response after applying the mitigation techniques.
Execution harness: Python
Recipe freeze marker
No recipe freeze time is recorded.