Materials · case/employers/corvid.md

# Corvid Labs: research packet

Fictional organization. This packet stands in for what a person would learn from Corvid's site, blog, papers, and public talks.

## What Corvid sells

Evaluation environments for computer-use agents. A customer brings an agent, Corvid runs it through hundreds of realistic software tasks, and reports a score per task family. Customers include two model labs and several enterprises deciding whether to deploy an agent internally.

## What Corvid has published

- **Engineering blog, March 2026: "Our graders disagree with people on long tasks."** Corvid's automated graders score agent trajectories. On tasks shorter than ten steps the graders match human raters well. On long tasks, graders and humans disagree on about a third of trajectories, mostly where the agent reached the right end state by a route a person would reject, such as deleting and recreating a record instead of editing it. The post ends: "We do not yet know which disagreements are grader errors and which are rater errors."
- **Paper, 2025, workshop venue: "Task families for computer-use evaluation."** Describes the environment design. The human baseline in the paper was ten Corvid employees, which the paper notes as a limitation.
- **Talk by the head of research, June 2026.** Says the company's main risk is "a customer deploying an agent because our score said it was ready, when the score was measuring the wrong thing." Says the next hire should "know how to tell whether a measurement means what we think it means."

## Team

Twelve people. Eight engineers, three researchers with machine-learning backgrounds, and the head of research, who came from a model lab. No one on the team has a background in human-subjects research or psychometrics.

## Application mechanism

Greenhouse-style applicant-tracking portal. The portal parses the résumé into fields before a person reads it, so dates and titles must be machine-readable. The research team reads every cover letter that is submitted.