Materials · corvid/brief.md

# Editorial brief: Corvid Labs, Research Scientist, Agent Evaluation

Written by the orchestrator before drafting. Sources: `case/board/corvid-research-scientist.md`, `case/employers/corvid.md`, `case/candidate/record.md`, `case/candidate/writing-preferences.md`, and the scout report `screening/scout-private-ai.md`. The CV is not a source.

Deliverables: a résumé (PDF, parsed by an applicant-tracking system) and a cover letter. The letter is optional, but Corvid's research team reads every one, so we write it. It carries the argument. The résumé carries the record.

## 1. What Corvid is selecting for

The posting asks for a quantitative PhD, Python, experiment and measurement design, and language-model research experience. Corvid's own work points to something narrower.

- **Its product is a score that customers use to decide whether to deploy an agent.** The head of research named the main risk: a customer deploys "because our score said it was ready, when the score was measuring the wrong thing." The same talk said the next hire should "know how to tell whether a measurement means what we think it means." This is a validity problem, not a modeling problem.
- **Corvid has an open grader problem it has stated in public.** On long tasks, its automated graders and human raters disagree on about a third of trajectories. Most disagreements are cases where the agent reached the right end state by a route a person would reject. Corvid says it does not know which disagreements are grader errors and which are rater errors. That puts two questions in front of the hire:
  - Is the route part of what "success" means? That is a construct definition question.
  - Can the human raters themselves be trusted as the reference? That is a rater reliability question.
- **Its human baseline is weak and it knows it.** The 2025 paper used ten employees as the human comparison and lists that as a limitation. The posting says the hire will run human-versus-agent studies.
- **No one on the twelve-person team has human-subjects or psychometrics training.** The hire is meant to supply that judgment, not more engineering.

In short, Corvid is selecting for someone who can decide whether an automated judge measures what it claims, and who can design the human comparison it is judged against.

## 2. The lead: validating the language-model judges in Relay

**What it is, in terms an outsider can follow.**
- Relay is a testbed Jordan built to study how information changes as it passes along chains and networks of language-model agents.
- To measure those changes at scale, Jordan used language-model judges to code each message, for example whether a fact or a qualification had been dropped.
- Before trusting the judges, Jordan and one research assistant coded a set of messages by hand, and Jordan compared the judges against that hand coding.
- The judges agreed well with the hand coding on whether facts were dropped. They agreed poorly on whether qualifications were dropped.

**What the reader must understand.**
- **The decision it required:** whether to trust a cheap automated judge or a slow human one, and the answer was different for different properties. One overall agreement figure would have hidden the property where the judge failed.
- **Why Corvid should care:** this has the same structure as Corvid's grader problem. The automated judge handles the blunt outcome (was the fact kept, was the end state reached). It fails on the subtler property (was the qualification kept, was the route acceptable).
- **The parallel is structural, not literal.** Relay messages are text passed between agents. They are not multi-step computer-use trajectories. The letter must draw the parallel without claiming Jordan has graded trajectories.

**Supporting detail from the same project.** Before the first full run, Jordan found that the model-access route was adding hidden context to every agent call, which would have contaminated every condition. For a company that sells evaluation environments, this shows Jordan checking the instrument before trusting the results. Use it as one sentence at most.

**Status, exactly.**
- Relay: self-funded, 2024 to present, used only with language-model agents, never with people.
- The paper reporting it: under review at *Management Science*, first round, submitted July 2026, not accepted.
- The talk "When can a language-model judge be trusted?" (Varrow, May 2025): invited. It belongs on the résumé and can be named in the letter.

**Second piece: the Kestrel satisfaction measure.** Jordan found that the main survey item tracked recent outages more than satisfaction with the catalog. Jordan rebuilt the measure and validated it against later cancellation. This is the head of research's question, "does the measurement mean what we think it means," answered on a human measure in a product company. It also shows that Jordan has built measures for human respondents, which the team lacks. It should be one paragraph after the lead, not a second lead.

## 3. Alternative leads and why they lost

- **The Kestrel measure as the lead.** It is the cleanest validity story in the record and includes validation against a real outcome. It lost because it concerns a survey of people, not an automated judge. A reader from a model lab would have to make the translation themselves. Relay puts the same judgment on a language-model judge, which is Corvid's problem. Kestrel stays as the second piece.
- **The pricing-experiment rehearsal.** It is tempting because it puts language-model agents next to an experimental design, and Corvid compares agents with people. It lost for three reasons:
  - It uses agents as stand-ins for people, the opposite of Corvid's question.
  - It is simulated only, and the working paper is not submitted.
  - It invites the overstatement the CV already makes. The CV's title for it claims to predict real shopper response, which the record says the rehearsal cannot do.
  - Résumé only, under its true title and status. Leave it out of the letter.
- **Parcel experiments and the holdout process.** These are solid experimental work, but they are about running experiments, not about whether a measure is valid. Use them on the résumé only.
- **Postdoc participant pool.** Jordan ran lab experiments with student participants and managed the pool. That is relevant to Corvid's weak human baseline, but it is too thin to lead. The drafter may use one clause in the letter when discussing the human comparison, stated exactly.

## 4. Gaps the reader will notice, and how to handle them

| Gap | Will the reader notice? | Handling |
|---|---|---|
| No experience building reinforcement-learning environments or training models (posting lists it as preferred) | Yes, it is in the posting | State it once in the letter, plainly, without apology. Do not add related skills to the résumé to compensate. |
| No work with computer-use agents or long multi-step trajectories | Yes | Do not state it as a gap. Do not claim it either. Draw the Relay parallel as structural. |
| No study comparing agents with people. Relay has never been used with people | Likely, since the posting asks for these studies | Never imply Relay involved people. The human-subjects evidence is the postdoc lab work and the Kestrel survey. Say only that. |
| AI work is recent and not yet peer reviewed | Yes, from the dates and statuses | Handle through exact status (under review, working paper, invited, self-produced). No comment needed. |
| Independent since 2024, no employer | Visible on the résumé | Do not raise it. The record gives no reason to treat it as a gap. |

## 5. Rules for the drafter that this application adds

- **The résumé follows the record, not the CV.** In particular:
  - Parcel: mentored two analysts informally, no direct reports. Not "managed analysts."
  - Kestrel: led a team of three direct reports, and owned the measurement plan for the product teams' experiments but did not run them.
  - The *Management Science* paper: marked under review.
  - The working paper: under its true title, "Rehearsing experiments with simulated participants," marked as a working paper.
  - "Building an agent testbed": either omit it or list it as a self-produced recorded walkthrough, not under talks.
  - No "causal inference."
  - Include the 2021 *Journal of Applied Psychology* paper with Jordan's position, third of four authors. Contribution detail is optional.
- **The ATS parses the résumé.** Dates must be month and year ranges, and the record gives years only. Use `[ASK JORDAN: month]` and do not guess.
- **Teaching:** the TA role and guest lecture may be omitted from this résumé. They do not answer Corvid.
- **Numbers in the letter:** leave out the 300-message count and Corvid's "about a third." The letter needs the shape of each finding, not its size. The 300-message count may appear on the résumé.
- **What the letter should do:** connect Jordan's work to Corvid's stated problem, and describe how Jordan would approach the grader-versus-rater question as an approach. Do not claim results Jordan has not produced, and do not state fit.

## 6. Open questions for Jordan (the drafter marks them, not answers them)

1. Start and end months for every position and project.
2. What did Jordan do with the qualification codes after the judges agreed poorly? Did Jordan hand-code them, drop them, or report them with a caveat? The record does not say, and the letter is stronger if it can.
3. Was the two human coders' agreement with each other measured? The record lists inter-rater reliability as a skill but does not say it was applied here.
4. AoM 2019: was it an invited or a competitively selected paper session? Until Jordan answers, list it as a paper presentation, not under invited talks.
5. Contact details for the résumé header. The record has none.