Installing a bundle creates a private copy you can edit. The published bundle stays unchanged.
109 bundles
Track eval suite changes over time, detect regressions when evals are updated
Build evals for specialized verticals (legal, medical, finance, engineering)
Ensure training data and eval data don't overlap
Create evals specifically designed to find failure modes and edge cases
Identify when a model has maxed out an eval and needs harder/different benchmarks
Measure whether your eval suite covers the actual distribution of production tasks
Design evals for tasks with multiple valid solutions (writing, design, open-ended code)
When to reward final results vs. intermediate steps, and how to blend both
Ensure reward functions produce consistent, well-scaled signals across different task types and difficulties
Design interfaces and protocols for collecting human preference judgments at scale
Identify when agents exploit reward function loopholes to get high scores without doing the task correctly
Add intermediate reward signals that guide learning without changing the optimal policy