Skip to main content
Loading navigation
Back to challenges

Offshore BPO Cyber Risk Triage with LlamaIndex

Unauthorized AI development tools used across offshore consulting teams introduce software supply chain vulnerabilities. Build an automated code repository and logs auditing agent using LlamaIndex to identify shadow AI usage and credential leaks. Achieve a 94% recall rate in identifying unauthorized developer activities.

Machine LearningHosted by Vera
Sign in free to start

What you are building

Build an intelligent code and log audit engine with LlamaIndex to audit offshore team repositories for unauthorized AI tools and security risks.

How work is evaluated

Evaluation

Evaluates detection recall and precision for unauthorized AI tool signatures in commit logs.

Datasets

Public datasets, plus the private datasets you have access to.

How submissions are scored

Dimensions
2 scoring checks
Binary
2 pass or fail dimensions
Ordinal
0 scaled dimensions
Dimension 1

flagged_type_check

flagged must be a boolean.

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Dimension 2

recall

Recall rate on identifying unauthorized tool commits • target: 0.94 • range: 0.8-1

Binary check

This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.

Learning goals

  • Build LlamaIndex VectorStoreIndex over repository commits and developer command logs

  • Construct custom metadata extractors for identifying shadow AI tool usage (e.g. Cursor, DeepSeek)

  • Implement security alert workflow for credential leaks and unapproved AI API keys

  • Achieve 94% recall on unauthorized AI activity detection

Resources

Dataset notes

Synthetic offshore developer commit history containing 800 pull requests and command logs with embedded security risks.

Challenge setup

Evaluates detection recall and precision for unauthorized AI tool signatures in commit logs.

Challenge input

JSON containing commit_id, developer_id, diff_text, and environment_variables

LlamaIndex

Provides indexing and retrieval engines over structured repository diffs.

Evaluated output

JSON with flagged (boolean), tool_detected (string), and severity (HIGH|MEDIUM|LOW)

Evaluation checks
  • flagged must be a boolean.
  • Recall rate on identifying unauthorized tool commits • target: 0.94 • range: 0.8-1
Configured evaluation evidence
  • Benchmark: SecCodeAuditBench
  • Recall target: 0.94
  • 1 public reference case
Configured runtime
  • Python execution harness
  • Python sandbox (unavailable on Versalist)

Shows the first task, up to three tools, and up to three evidence items.

View technical recipe

Configured tools

Action Space
  • Llama Index · Required
  • Zed · Optional
  • llama.cpp · Optional

Evaluation contract

  • flagged_type_check · Weight 1
  • recall · Weight 1

Metrics

  • recall · Target 0.94

Benchmarks

  • SecCodeAuditBench
  • security

Reference cases

  • 1 public reference case

Workflow tasks

  1. Task 1

    Input: JSON containing commit_id, developer_id, diff_text, and environment_variables

    Output: JSON with flagged (boolean), tool_detected (string), and severity (HIGH|MEDIUM|LOW)

    Execution harness: Python

Evaluation tools

  • LlamaIndex · Agent Framework

Recipe freeze marker

No recipe freeze time is recorded.

Frequently Asked Questions about Offshore BPO Cyber Risk Triage with LlamaIndex