Skip to main content
Back to challenges
Challenge

Offshore BPO Cyber Risk Triage with LlamaIndex

Unauthorized AI development tools used across offshore consulting teams introduce software supply chain vulnerabilities. Build an automated code repository and logs auditing agent using LlamaIndex to identify shadow AI usage and credential leaks. Achieve a 94% recall rate in identifying unauthorized developer activities.

Machine LearningHosted by Vera
Sign in free to start
Challenge brief

What you are building

The core problem, expected build, and operating context for this challenge.

Build an intelligent code and log audit engine with LlamaIndex to audit offshore team repositories for unauthorized AI tools and security risks.

Delivery guide

How work is evaluated

Evaluation

Evaluates detection recall and precision for unauthorized AI tool signatures in commit logs.

Datasets

Challenge datasets

Public datasets, plus the private datasets you have access to.

Evaluation rubric

How submissions are scored

These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

flagged_type_check

flagged must be a boolean.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

recall

Recall rate on identifying unauthorized tool commits • target: 0.94 • range: 0.8-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

What you should walk away with

  • Build LlamaIndex VectorStoreIndex over repository commits and developer command logs

  • Construct custom metadata extractors for identifying shadow AI tool usage (e.g. Cursor, DeepSeek)

  • Implement security alert workflow for credential leaks and unapproved AI API keys

  • Achieve 94% recall on unauthorized AI activity detection

Resources and assets

Reference links and supporting material

Dataset notes

Synthetic offshore developer commit history containing 800 pull requests and command logs with embedded security risks.

Challenge setup

Evaluates detection recall and precision for unauthorized AI tool signatures in commit logs.

Challenge input

JSON containing commit_id, developer_id, diff_text, and environment_variables

LlamaIndex

Provides indexing and retrieval engines over structured repository diffs.

Evaluated output

JSON with flagged (boolean), tool_detected (string), and severity (HIGH|MEDIUM|LOW)

Evaluation checks
  • flagged must be a boolean.
  • Recall rate on identifying unauthorized tool commits • target: 0.94 • range: 0.8-1
Configured evaluation evidence
  • Benchmark: SecCodeAuditBench
  • Recall target: 0.94
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • Zed · Optional
  • llama.cpp · Optional

Evaluation contract

  • flagged_type_check · Weight 1
  • recall · Weight 1

Metrics

  • recall · Target 0.94

Benchmarks

  • SecCodeAuditBench
  • security

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing commit_id, developer_id, diff_text, and environment_variables

    Output: JSON with flagged (boolean), tool_detected (string), and severity (HIGH|MEDIUM|LOW)

    Execution harness: Python

Evaluation tools

  • LlamaIndex · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Offshore BPO Cyber Risk Triage with LlamaIndex