Operator-ready prompt for reuse, tuning, and workspace runs.
This item is set up for developers who want to inspect the original language, fork it into Workspace, and adapt the evidence model without losing the source prompt structure.
Implementation handoffs, eval setup, and prompt tuning where you need the original structure intact.
Inspect first, copy once, then fork into Workspace when you want variants, notes, and model settings attached to the same run.
Swap domain facts, examples, and any hard-coded entities for your own context.
Tighten the evidence or verification requirement if this is headed toward production.
Decide which failure mode you want to evaluate first before you branch the prompt.
This prompt already carries implementation detail, tool context, and a final-output instruction. Keep that structure intact when you tune it, or your comparison runs get noisy fast.
Open this prompt inside Workspace when you want a live iteration loop.
Copy for quick reuse, or run it in Workspace to keep prompt variants, model settings, and prompt-history changes in one place.
Structured source with 22 active lines to adapt.
Already linked to a challenge workflow.
Sign in to keep private prompt variations.
Prompt content
Original prompt text with formatting preserved for inspection and clean copy.
Design an evaluation pipeline using Evidently AI to assess the 'ThreatDetectionAndClassification' task. Describe how you would collect the agent's outputs and ground truth, and define key metrics Evidently AI should track, such as 'ThreatTypeAccuracy' and 'SeverityClassificationF1Score'. Provide a Python snippet demonstrating how to initialize an Evidently AI monitoring dashboard and log relevant data from your agent's performance.
```python
import evidently
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
from evidently.metric_preset import TextOverviewPreset # For LLM specific outputs
def evaluate_threat_detection(actual_outputs, ground_truth):
# Prepare data for Evidently AI
# ... e.g., combine actual_outputs and ground_truth into a DataFrame ...
report = Report(metrics=[
DataDriftPreset(), # For input/output data drift
# Add custom metrics for classification accuracy, F1 score etc.
# Evidently AI might require custom metric definitions for direct LLM eval,
# or conversion of LLM outputs to structured data first.
])
# report.run(reference_data=ref_df, current_data=current_df)
# report.save_html("threat_detection_report.html")
# Conceptual usage:
# agent_output = threat_detection_agent(sample_input)
# expected_output = get_ground_truth(sample_input)
# evaluate_threat_detection([agent_output], [expected_output])
```Adaptation plan
Keep the source stable, then branch your edits in a predictable order so the next prompt run is easier to evaluate.
Preserve the rubric, target behavior, and pass-fail criteria as the baseline for evaluation.
Adjust fixtures, mocks, and thresholds to the system under test instead of weakening the assertions.
Make sure the prompt catches regressions instead of just mirroring the happy-path examples.
Copy once for a pristine source snapshot, then move the prompt into Workspace when you want variants, run history, and side-by-side tuning without losing the original.
Prompt diagnostics
Quick signals for how structured this prompt already is and where adaptation work is likely to happen first.
This prompt already mixes executable detail with instructions, so the safest path is to tune examples and interfaces before you rewrite the overall scaffold.
Cyberthreat Orchestrator Agent
This challenge requires building an autonomous cyber threat detection and remediation system using the LangChain framework, specifically leveraging LangGraph for complex, stateful multi-agent workflows. Developers will design a team of specialized agents that work together to identify threats from simulated log data, analyze their severity, formulate a remediation plan, and orchestrate protective actions. The system must be capable of dynamic decision-making and adapting its response based on the evolving threat landscape. The focus is on robust agent collaboration patterns, sophisticated tool integration, and continuous evaluation of the agent system's effectiveness.
Use the challenge page to recover the original task boundaries before you tune the prompt. That keeps your variants grounded in the same evaluation target instead of drifting into a different problem.