SkillProof Document Agent
This challenge involves creating an AI agent using the Mastra AI TypeScript framework that generates and verifies 'SkillProof' documents – an AI-native standard designed to replace traditional credentials for displaying professional skills and 'vibe coding'. The agent will leverage Qwen 2 for nuanced skill description and personality assessment generation, while integrating custom tools for external verification of AI proficiency levels. A key aspect is the implementation of a conversational voice interface via Sarvam AI, allowing users to interact naturally to create and modify their SkillProof documents, demonstrating advanced natural language processing and structured output generation.
What you are building
The core problem, expected build, and operating context for this challenge.
Build an AI agent using Mastra AI to generate and verify an AI-native 'SkillProof' document, integrating Qwen 2 for content generation and Sarvam AI for a voice interface.
How work is evaluated
The evaluation module will assess the Mastra AI agent's ability to generate accurate, well-structured 'SkillProof' documents and correctly apply verification logic. Focus areas include document content quality, adherence to schema, and verification accuracy.
Challenge datasets
Public datasets, plus the private datasets you have access to.
How submissions are scored
These dimensions define what the evaluator checks and which criteria separate a passable run from a strong one.
DocumentSchemaAdherence
Generated document strictly adheres to the defined JSON schema.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
CorrectVerificationStatus
Verification status is correct based on proficiency and threshold.
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
DescriptionQualityScore
LLM-generated description quality and relevance (1-5). • target: 4 • range: 1-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
VibeCodingRelevance
Relevance of the generated vibe coding assessment to the skill (1-5). • target: 4 • range: 1-5
This dimension contributes its full weight only when the submission satisfies the requirement. Partial credit is not awarded.
What you should walk away with
Master Mastra AI's core concepts including agent definition, memory system, and tool integration for building complex workflows.
Implement structured output generation using Qwen 2 to adhere to a predefined 'SkillProof' document schema (e.g., JSON-LD or similar AI-native format).
Utilize Mastra AI's built-in memory capabilities (e.g., with Redis) to persistently store draft documents and user preferences, enabling multi-turn document refinement.
Develop a custom tool, 'verify_ai_proficiency', that takes skill descriptions and provides a mock verification status and score.
Integrate Sarvam AI to create a conversational voice interface, allowing users to verbally dictate document content and request verification checks.
Design prompts for OpenAI o3 to parse natural language input into structured data for the 'SkillProof' document and to generate a 'vibe coding' personality assessment.
Implement a document versioning and storage mechanism (e.g., saving to a local file system or mock cloud storage) as part of the agent's capabilities.
Reference links and supporting material
Sample data for 2 tasks
Challenge setup
The evaluation module will assess the Mastra AI agent's ability to generate accurate, well-structured 'SkillProof' documents and correctly apply verification logic. Focus areas include document content quality, adhere...
Challenge input
{"skill_description": "string", "user_profile": {"name": "string", "experience_years": "number"}}
Agent execution
No agent tool is listed for this challenge.
Evaluated output
{"document_json": {"skill_name": "string", "proficiency_level": "string", "description_generated": "string", "vibe_coding_assessment": "string"}}
- Generated document strictly adheres to the defined JSON schema.
- Verification status is correct based on proficiency and threshold.
- LLM-generated description quality and relevance (1-5). • target: 4 • range: 1-5
- DescriptionQualityScore target: 4
- 2 public reference cases
- JavaScript execution harness
Summary shows the first task, up to three tools, and up to three evidence items. Open the technical recipe to view all configured items.
View technical recipe
Configured tools
No tool records are attached.
Evaluation contract
- DocumentSchemaAdherence · Weight 1
- CorrectVerificationStatus · Weight 1
- DescriptionQualityScore · Weight 1
- VibeCodingRelevance · Weight 1
Metrics
- DescriptionQualityScore · Target 4
- VibeCodingRelevance · Target 4
Reference cases
- 2 public reference cases
Workflow tasks
- Task 1
Input: {"skill_description": "string", "user_profile": {"name": "string", "experience_years": "number"}}
Output: {"document_json": {"skill_name": "string", "proficiency_level": "string", "description_generated": "string", "vibe_coding_assessment": "string"}}
Execution harness: JavaScript
- Task 2
Input: {"skill_name": "string", "proficiency_level": "string", "verification_threshold": "number"}
Output: {"verification_status": "string", "verification_score": "number", "verified_by": "string"}
Execution harness: JavaScript
Recipe freeze marker
No recipe freeze time is recorded.