Materials · corvid/for-candidate/cover-letter.txt

Jordan Hale
https://jordanhale.example | [ASK JORDAN: email] | [ASK JORDAN: phone] | [ASK JORDAN: city and state]

[ASK JORDAN: submission date]

Corvid Labs hiring team
Re: Research Scientist, Agent Evaluation

Dear Corvid Labs hiring team,

In your March engineering blog post, you wrote that your automated graders and human raters often disagree on long tasks, mostly where an agent reached the right end state by a route a person would reject, and that you do not yet know which disagreements are grader errors and which are rater errors. I have worked on a smaller problem with the same structure.

Since 2024 I have built and funded Relay, a testbed for studying how information changes as it passes along chains of language-model agents. I used language-model judges to label each message, for example whether a fact had been dropped, or a qualification such as a hedge or caveat. Before trusting the judges, a research assistant and I labeled 300 messages by hand, and I compared the judges with our labels.

The judges agreed with us well on dropped facts and poorly on dropped qualifications, so one overall agreement figure would have hidden the property where the judge was weak. [ASK JORDAN: what you did with the qualification labels after the judges agreed poorly.] A paper reporting the Relay experiments is under review at Management Science, in its first round. I gave an invited talk on when a language-model judge can be trusted at the Measurement in AI Workshop at Varrow University in May 2025.

Relay messages are text, not computer-use trajectories, and I have not built reinforcement-learning environments or trained models. The parallel is partial. My judges scored qualifications unreliably, and some of your disagreements may be of that kind. Others may arise because a grader checks the end state and does not score the route at all. In Relay I treated the hand labels as the reference. For you, whether the raters can serve as the reference is part of the open question.

So I would first settle whether the route is part of success. Then I would estimate rater agreement on agreed as well as disputed trajectories, across task lengths, and sort each disagreement into grader error, rater error, or a task specification that allows both readings. [ASK JORDAN: whether you measured agreement between yourself and the research assistant in Relay. If so, say so here.]

Your head of research named the main risk as a customer deploying an agent because the score said it was ready, when the score was measuring the wrong thing. At Kestrel Media I met that problem with people as respondents. The main item in our satisfaction survey tracked recent service outages more closely than satisfaction with the catalog. I rebuilt the measure and checked it against which viewers later canceled. A task-family score can be checked the same way, against what happens after deployment.

Your 2025 paper notes that its human baseline came from your own staff. During my postdoc I ran laboratory experiments with student participants and managed the lab's participant pool. A comparison group drawn from outside the company is the study I would design.

Sincerely,
Jordan Hale