Apply Here

https://on-contra.typeform.com/to/KwttPXwR

Who we are and what we do

Contra is building the world’s first professional network for independent creatives and the companies that hire them. We are commission-free, global, and focused on enabling the future of flexible work. Alongside our network and marketplace, we’re launching new product lines including Contra for Companies, Contra Ads Network, and Creative RLHF & Evaluation Services for AI labs and creative-tool companies.

We believe human creativity is the defining force of the next era of AI. That’s why we’re building the Human Creativity Benchmark—a new standard for evaluating how foundation models perform on creative, subjective, and open-ended tasks.

We've raised over $51M from leading investors like NEA, Unusual Ventures, and Cowboy Ventures—and we're just getting started.

Responsibilities:

We are seeking a AI Project Lead to lead Contra’s human data and evaluation business. This is a pivotal role that combines AI evaluation strategy, human data operations, and benchmark design. You will manage and own the quantitative frameworks that define how creative foundation models are evaluated across the industry. You will be responsible for:

• Designing, building, and operating Contra’s Human Creativity Benchmark (our flagship evaluation framework for creative foundation models).
• Scaling Contra’s human-data services - from one-off annotation projects to ongoing evaluation partnerships with top AI labs and creative-tool companies.
• Develop mathematical scoring systems, annotation protocols, rubrics, and evaluation methodologies to measure creativity, usability, factuality, and safety across multiple model outputs.
• Applying probability, statistics, and experimental design (A/B testing, randomized trials, inter-rater reliability studies) to validate benchmark integrity, analyze performance, and ensure reproducibility.
• Use sampling techniques, regression analysis, and variance/error analysis to ensure statistical rigor and representative data coverage.
• Establishing quality control systems to ensure reliability, reproducibility, and credibility of results.