Materials · corvid/reviews/round2-role.md

# Round 2 role review: Corvid Labs, Research Scientist, Agent Evaluation

Reviewer stance: Corvid's head of research, reading the letter and résumé against our own blog post, paper, and talk.

## 1. What Corvid is actually selecting for

The problem is measurement validity, not environment engineering. Our scores drive customers' deployment decisions. On long tasks our graders disagree with people on about a third of trajectories, and we don't know which side is wrong. Our human baseline is ten employees. Nobody on the team has a background in psychometrics or human-subjects research. We need someone who can tell whether a score means what we think it means: someone who can build a defensible human reference (raters, reliability, adjudication, a real baseline sample) and validate graders against it. Whether they can build RL environments is secondary. The posting lists it only as "preferred."

## 2. Does the package answer it?

**Where it does:**
- The Relay judge validation is the right kind of evidence. It checks judges against hand labels, breaks agreement down by property, and shows that one overall agreement figure hides the weak property. That is the reasoning we need.
- The Kestrel survey rebuild maps directly onto our risk of "measuring the wrong thing." Validating the measure against later cancellation is criterion validation against a downstream outcome, which is exactly our customer's problem.
- The proposed first step is right: decide whether the route is part of success before comparing graders and raters.
- The candidate is honest about having no RL or training work. Because the letter puts the stronger fit first, that disclosure doesn't sink it.

**Where it misses or only gestures (ranked by how much it hurts):**
1. **The two open placeholders carry the case.** If Jordan never measured agreement between Jordan and the RA, then the hand labels used as the "reference" in Relay were never shown to be reliable. Meanwhile the letter tells us to measure rater agreement first. A sharp reader will see that contradiction. Likewise, if the qualification labels were simply dropped after the judges did poorly, the story goes from "found the weak property and handled it" to "found it and walked away." These two answers decide whether the Relay paragraph is strong or a liability.
2. **No numbers.** "Agreed well" and "poorly" mean nothing to an evaluation specialist. We need kappa or percent agreement per property, and the n (the 300 appears in the résumé but not the letter). One figure per property would move this a lot.
3. **The human baseline problem is ignored.** The posting asks for agent-versus-human studies, and our own paper names the ten-employee baseline as a limitation. Jordan ran a lab participant pool and laboratory experiments. That is the direct answer to this gap, but the letter only mentions it in the closing disclosure paragraph and never connects it to our baseline.
4. **The most relevant credential isn't used.** The invited talk "When can a language-model judge be trusted?" (May 2025) appears only in the résumé. It is nearly a restatement of our hiring brief.
5. **The Kestrel link stops short.** The letter says Jordan rebuilt the measure and checked it against cancellations, but never draws the parallel: our per-family scores should be validated against whether deployed agents actually work. That is the strongest bridge in the package, and it's left implicit.
6. **The hidden-context anecdote** shows good pipeline hygiene, but it takes up space without connecting to graders. It would be stronger in one clause tied to environment contamination.

## 3. Is the analogy sound, and is the approach credible?

**The analogy is mostly sound and appropriately hedged.** "Structure, not trajectories" is the right caveat. One flaw: the letter maps "blunt vs. subtle property" onto "end state vs. route," which implies our graders are weak on route the way Relay's judges were weak on qualifications. Our blog points to something different. The graders may not be failing at route; they may not be scoring route at all, which is a construct mismatch, not a judge-accuracy problem. The letter's own first step ("settle whether the route is part of success") actually acknowledges this, so the analogy paragraph should match that framing instead of the judge-weakness framing.

**The approach is credible, not naive, with three specific gaps an evaluation specialist would flag:**
- **Selecting on the outcome.** "Measure how well raters agree with one another on the disputed trajectories" restricts reliability estimation to cases already selected for disagreement. That biases the estimate and misses cases where grader and rater agree for the wrong reason. The sample should cover both agreed and disputed trajectories, stratified by length.
- **More than two error categories.** "Grader errors and rater errors" leaves out a third category: the task spec is ambiguous and neither side is wrong. Step one partly covers this, but the sorting step should name it.
- **Length as a moderator.** Our disagreement appears above about ten steps. A plan that doesn't model how agreement degrades with trajectory length (and where along the trajectory the disagreement originates) misses the most distinctive feature of our data.

No adjudication protocol, rater guidelines, or rater-pool composition is mentioned. That's acceptable in a letter, but expect to be asked about it.

## 4. What a hiring manager would doubt, and first-interview questions

**Doubts:** no ML publications or ML-venue track record; the main AI paper is unreviewed and at a management journal; two-plus years self-funded and independent; can't build environments; Relay is text-only and on a small scale.

**Likely questions:**
1. What were the judge–human agreement numbers per property, and how many messages per property?
2. Did you measure agreement between yourself and the RA? If not, why treat those labels as ground truth?
3. What did you do about qualifications once the judge failed on them?
4. Walk us through designing a rater study for our long-task disagreements. Who are the raters, how many, what instructions, how do you adjudicate?
5. How would you replace or extend our ten-employee human baseline? What population, what sample size, how do you handle task familiarity?
6. How would you validate a per-family score against real deployment outcomes? (Kestrel analogue.)
7. How did you validate the Kestrel measure against cancellation, and what confounds did you rule out (for example, outages driving both)?
8. How exactly did you detect the hidden context in the model route? Could similar leakage affect our environments?
9. The pricing rehearsal had a "client" during a self-funded independent period. What was that arrangement?
10. Why leave industry for independent work, and why join a 12-person startup now?
11. How comfortable are you writing production-quality Python in an engineering-heavy team? Show us Relay code.

## 5. Accuracy of statements about Corvid

- "March engineering blog post": correct (March 2026).
- "graders and human raters often disagree on long tasks": accurate but softer than the source. The post says "about a third of trajectories." Quoting the figure would be more precise and shows close reading.
- "mostly where an agent reached the right end state by a route a person would reject": accurate.
- "you do not yet know which side is in error": slight paraphrase drift. The post says "which disagreements are grader errors and which are rater errors," which is per-disagreement. "Which side" suggests one party is wholly wrong. Minor, but this reader will notice. Use the post's wording.
- "head of research has described the main risk as a score that measures the wrong thing": a fair compression, not a misquote, and there are no quotation marks. It drops the "customer deploying an agent" part, which is the hook for the Kestrel criterion-validation point. Consider restoring it.
- Nothing is misdated or invented.

## 6. Overall

**Likely advances to an interview.** The candidate's measurement, reliability, and human-subjects background targets the exact gap our team and talk describe. This assumes the two placeholders come back with real agreement numbers and a real follow-up on qualifications. If they come back "not measured" and "dropped," the Relay paragraph undercuts the pitch.