Materials · corvid/drafts/cover-letter.txt

Jordan Hale
jordanhale.example | [ASK JORDAN: email] | [ASK JORDAN: phone] | [ASK JORDAN: city and state]

October 12, 2026

Corvid Labs research team
Re: Research Scientist, Agent Evaluation

Dear Corvid research team,

You have said in public that your automated graders and your human raters often disagree on long tasks, that most disagreements are cases where the agent reached the right end state by a route a person would reject, and that you do not know which side is in error. I have worked on a smaller problem with the same structure.

Since 2024 I have built and funded Relay, a testbed for studying how information changes as it passes along chains and networks of language-model agents. To measure those changes at scale, I used language-model judges to code each message, for example whether a fact or a qualification had been dropped. Before trusting the judges, a research assistant and I coded a set of messages by hand, and I compared the judges with that hand coding. The judges agreed well on whether facts were dropped and poorly on whether qualifications were dropped. The decision to trust the cheap judge or the slow human one therefore had a different answer for each property, and a single overall agreement figure would have hidden the property where the judge failed. [ASK JORDAN: what you did with the qualification codes after the judges agreed poorly, for example hand-coded them, dropped them, or reported them with a caveat.] Before the first full run I also found that the model-access route was adding hidden context to every agent call, which would have contaminated every condition. The paper reporting this work is under review at Management Science, and I gave an invited talk, "When can a language-model judge be trusted?", at Varrow University in May 2025.

Relay messages are text passed between agents, not multi-step computer-use trajectories. The parallel is in the structure. The automated judge handled the blunt outcome, whether the fact was kept, and failed on the subtler one, whether the qualification was kept. Your graders and raters split along a similar line, between the end state and the route.

I have asked whether a measure means what we think it means on human respondents as well. At Kestrel Media I found that the main item in the viewer-satisfaction survey tracked recent outages more than satisfaction with the catalog. I rebuilt the measure and validated it against later cancellation.

On the grader-versus-rater question, I would separate two questions before scoring anything. The first is whether the route is part of success. That is a definition to settle with customers, and graders and raters cannot be judged against each other until it is settled. The second is whether the raters can serve as the reference. I would measure how well raters agree with one another on the disputed trajectories, score route and end state as separate properties, and only then ask which disagreements are grader errors. [ASK JORDAN: did you measure agreement between yourself and the research assistant in Relay? If so, this paragraph can say so.] For the human-versus-agent studies, my experience with human participants comes from laboratory experiments on group decisions with student participants during my postdoc, where I also managed the participant pool, and from the Kestrel survey work. I have not built reinforcement-learning environments or trained models.

Sincerely,
Jordan Hale