Materials · corvid/reviews/unfamiliar-reader.md

# Unfamiliar-reader review: Corvid Labs, Research Scientist, Agent Evaluation

Reader stance: a Corvid researcher who knows nothing about Jordan except the posting, the cover letter and the résumé. Findings are ranked by how much each would hurt the application.

## What works

The Relay judge-validation story is concrete and fits Corvid's grader-trust problem. The letter says plainly what Jordan has not done ("not multi-step computer-use trajectories", "have not built reinforcement-learning environments"). The two-question plan in paragraph 5 reads like someone who has done measurement work.

## Ranked findings

### 1. The opening claim about Corvid has no source (section 4)
> "You have said in public that your automated graders and your human raters often disagree on long tasks, that most disagreements are cases where the agent reached the right end state by a route a person would reject, and that you do not know which side is in error."

None of this is in the posting. The reader will ask where Corvid said it: a talk, a blog post, a paper? If the reader can't place it, or doesn't recognise the paraphrase, the letter starts with a claim about the reader's own company that seems invented, and the rest of the letter is built on it. Name the source in a few words.

### 2. Jordan's own reference standard has no reliability figure, while the letter asks Corvid to measure exactly that (sections 4 and 1)
> "a research assistant and I coded a set of messages by hand, and I compared the judges with that hand coding."
> "I would measure how well raters agree with one another on the disputed trajectories"

The letter tells Corvid to check whether the human raters can serve as the reference. It never says whether Jordan checked that the two hand-coders agreed with each other. A careful reader will notice this gap in the main piece of evidence. The placeholder `[ASK JORDAN: did you measure agreement between yourself and the research assistant in Relay?]` sits on this exact gap. If it is dropped without an answer, the contradiction stays.

### 3. The main story stops before it ends (section 1; placeholder needed)
> "a single overall agreement figure would have hidden the property where the judge failed. [ASK JORDAN: what you did with the qualification codes ...]"

The reader learns the judge failed on qualifications but not what Jordan did about it. "Determine when those graders can be trusted" is a core duty in the posting, so the decision taken after the failure is the most relevant part. This placeholder must be filled. Cutting it leaves the paragraph unfinished.

### 4. "Agreed well" and "agreed poorly" have no numbers (section 4)
> "The judges agreed well on whether facts were dropped and poorly on whether qualifications were dropped."

The posting asks for measurement experience, and the résumé lists "inter-rater reliability". The letter gives no statistic (kappa, percent agreement) and no counts. The résumé's "300 messages" does not appear in the letter. A measurement team will want at least one number for each property.

### 5. The "Strong Python" requirement has almost no support (section 3)
The only evidence is "Python" in the Skills list. Relay is described as "set in code", without saying what language, how large it is, or whether Jordan wrote it alone. The posting lists Python as a requirement. One clause would cover it, for example "Relay is N lines of Python I wrote", or a link to the code or to the "Building an agent testbed" walkthrough.

### 6. The analogy assumes the answer the letter later says not to assume (section 4)
> "The automated judge handled the blunt outcome ... and failed on the subtler one ... Your graders and raters split along a similar line, between the end state and the route."

This lines Corvid's graders up with Jordan's failing judge, which implies the graders are the ones in error. The opening, though, says Corvid does not know which side is wrong, and paragraph 5 says to settle the reference "and only then ask which disagreements are grader errors." A sharp reader will see the tension. Present the analogy as a hypothesis to test.

### 7. "Model-access route" and "hidden context" are not explained (section 1)
> "the model-access route was adding hidden context to every agent call, which would have contaminated every condition."

The reader can't tell what the route was (an API proxy, a wrapper, a provider default system prompt?), what the added context was, or how Jordan found it. Without that, it reads as a vague claim rather than a debugging skill. One clause of mechanism would make it count.

### 8. The evidence for human-versus-agent studies is old and indirect (section 3)
> "my experience with human participants comes from laboratory experiments on group decisions with student participants during my postdoc"

That work is from 2016 to 2018. Kestrel was survey work, Relay "Used only with language-model agents, not with people", and the pricing rehearsal was "simulated only". The letter is honest about this but never says how the experience carries over to comparing agents and humans on the same task. Posting duty 3 is answered weakly.

### 9. "A client" conflicts with "no employer" (section 4)
> Résumé: "Self-funded independent research, no employer."
> Résumé: "rehearse the design of a tiered-pricing experiment before a client ran it."

So is Jordan consulting? Who is the client, and was the work paid? The reader can't tell. The project's outcome is also missing: did the design detect the planted effect? This project is also the closest match to "write up methods and findings for customers", and the letter never mentions it.

### 10. Kestrel credit is unclear (section 2)
> "Led a team of three researchers (direct reports)." / "Found that the main item ... tracked recent outages"

The letter's "I found ... I rebuilt" reads as solo work, while the résumé shows a team of three. Was the finding Jordan's own analysis or the team's? "Validated it against later cancellation" also skips the method (did it predict cancellation? with what strength?).

### 11. Terms an ML reader may misread (section 1)
- "code each message" means "label" in the social-science sense. An ML reader may first read it as programming.
- "qualification" means a hedge or caveat in the message. It could be read as a credential or condition. Gloss it once, for example "qualifications (hedges and caveats)".
- "language-model judges": say whether this was one model or several and which ones, if that matters to the claim.

### 12. Smaller questions about fit (section 3)
- "PhD, Organizational Behavior": the posting asks for "a quantitative field". The résumé supports it (multilevel models, experiments), but nothing says so outright. A screener could stop at the degree title.
- The Parcel experiment work (2018 to 2022, including "first holdout process") is the longest stretch of experiment design, and the letter never connects it to the role.
- Relay is a "testbed", which is close to Corvid's "evaluation environments", but the drafts never make that link.
- The Management Science paper (submitted July 2026) and the May 2025 talk are both cited, but the reader can't tell whether the talk presented the same validation result.

## Placeholders
- The cover letter's two `[ASK JORDAN]` notes in paragraphs 2 and 5 need answers, not deletion (findings 2 and 3).
- The date and contact placeholders in the résumé are routine.