Materials · corvid/reviews/round2-evidence.md

# Round 2 evidence-selection review: Corvid Labs, Research Scientist, Agent Evaluation

## 1. What Corvid is selecting for, and the best evidence for it

From the packet, Corvid's open problem is **whether its graders can be trusted**. On long tasks its graders and human raters disagree on about a third of trajectories, and Corvid cannot say which side is wrong. The head of research names the main risk as "a score measuring the wrong thing" and wants a hire who can "tell whether a measurement means what we think it means." The **human baseline is weak** (ten employees, which their own paper lists as a limitation), and **no one on the team has a human-subjects or psychometrics background**. RL experience appears in the posting only as "preferred," and Jordan lacks it.

Ranked evidence from the record:

1. **Relay judge validation.** Jordan checked the language-model judges against 300 messages hand-coded with one RA. The judges agreed well on dropped facts and poorly on dropped qualifications. This is the closest match to the grader-vs-human problem.
2. **Kestrel satisfaction-measure rebuild.** The main item tracked outages rather than satisfaction. Jordan rebuilt the measure and validated it against later cancellation. This is a direct case of "a score measuring the wrong thing," caught and fixed.
3. **Invited talk, "When can a language-model judge be trusted?"** (Measurement in AI Workshop, Varrow, May 2025). It is outside recognition on exactly Corvid's question.
4. **Human-subjects and psychometric depth, which fills the team's gap.** This covers the postdoc lab experiments with human participants, running the participant pool, survey design and validation, inter-rater reliability, and the *JAP* 2021 paper where Jordan built the measurement model and robustness analyses for scoring a construct from meeting transcripts (scoring a latent property from long text records).
5. **Relay configuration catch.** Hidden context was being added to every call, which would have contaminated every condition. It shows care about evaluation infrastructure.
6. **Relay as a built environment.** Jordan wired the agents in code and kept the lineage of every message. This is the nearest analogue to "design and build evaluation environments," though it is text chains and not computer use.
7. **Parcel experiments and the first holdout process for long-running experiments.** This is general experimental-design strength.
8. **Pricing rehearsal with a planted effect.** Jordan checked whether a design detects a known effect, which amounts to a sensitivity check on a measurement. It is useful but secondary.

## 2. Was the strongest evidence chosen and made prominent?

**Cover letter: mostly yes.**
- Item 1 opens the letter and gets the most space. Item 2 is tied explicitly to the head of research's stated risk. Item 5 is included. The letter's ranking is right.
- Items 3 and 4 are missing or weak. The invited talk is absent. The human-participants experience appears only in the closing paragraph, as a hedge next to the RL disclaimer, and is not tied to Corvid's human-baseline problem. The letter therefore ends on a limitation.
- "A set of messages" leaves out the record's concrete figure of 300.

**Résumé: right content, weak placement.**
- The strongest evidence (Relay judge validation) is in **Projects**, after all four positions. The top role, Independent Researcher, has only a pointer ("see Projects"). A screener reading top-down reaches Kestrel, Parcel and the postdoc before the judge-validation bullet.
- Under Kestrel, the first bullet is "Led a team of three." The measure rebuild, which is the relevant item, comes second.
- The invited talk on judge trust is near the bottom, under Talks.
- There is no summary line pointing the reader to measurement validity or grader trust.

## 3. Material left out or understated (ranked), with placement and what it displaces

1. **Move the Relay bullets under Independent Researcher (résumé).** Put the three Relay bullets (built the testbed, validated the judges against 300 hand-coded messages, caught the configuration error) directly under the 2024 to present role and delete the separate Relay project entry. This puts the best evidence first on page 1 at no extra length and keeps the dated title machine-readable for the ATS parser. *Displaces:* the "see Projects" pointer and the duplicate Projects entry.
2. **Invited talk, "When can a language-model judge be trusted?"**
   - *Letter:* add one clause to the Relay paragraph, e.g. "I spoke on when a language-model judge can be trusted at an invited workshop talk (Varrow, May 2025)."
   - *Résumé:* move Talks above Papers, or add it as a final bullet under Independent Researcher.
   - *Displaces:* in the letter, shorten "The choice between a cheap automated judge and slow human labeling therefore had a different answer for each property" to just the point about one overall figure hiding the weak property. In the résumé it displaces nothing; the cut items in section 4 free the space.
3. **Human-subjects and psychometric depth, tied to the human baseline (letter).** Rewrite the closing paragraph so the postdoc lab experiments, the participant-pool management, and survey design and validation answer the posting's "compare agent performance with human performance" and the team's lack of human-subjects background. It should not stand only as a caveat. Keep the RL sentence, but don't make it the last line. Do not claim Jordan has run agent-vs-human comparisons; the record has none. *Displaces:* nothing, since it rewrites the existing final paragraph at about the same length.
4. **The JAP measurement-model contribution (letter).** Add one clause to the Kestrel paragraph saying Jordan also built the measurement model and robustness analyses for a published *JAP* paper that measured psychological safety from meeting transcripts. Keep it framed as third of four authors with Jordan's specific contribution, as the résumé already does. *Displaces:* the trim in item 2. If the page overflows, drop this clause before the talk.
5. **The figure of 300 (letter).** Replace "a set of messages" with "300 messages." This costs no space.
6. **Kestrel bullet order (résumé).** Lead with the rebuilt and validated measure, then the measurement plan, then team leadership. This costs no space.
7. **Optional two-line summary at the top of the résumé.** For example: measurement validity, validating automated judges against human coding, human-subjects experiments, Python. Every element is in the record. *Displaces:* the space freed by the cuts in section 4.

## 4. Weak or distracting items that could be cut

1. **"Other work: self-produced recorded walkthrough" (résumé).** It has low signal, and the label "self-produced" invites the reader to discount it. Cut it.
2. **"Mentored two analysts informally" (Parcel, résumé).** It is weak and not relevant to this role. Cut it.
3. **Pricing rehearsal (résumé).** Compress the two bullets to one that leads with the planted-effect detection check and keeps one short simulation-only caveat. The current version spends more words on caveats than on what Jordan did.
4. **"Used only with language-model agents, not with people" (Relay, résumé).** It repeats the first bullet, which already says "language-model agents." It can go, and the honesty is preserved. This is optional.
5. **The dissertation line and the AOM 2019 presentation (résumé).** These are low relevance. Trim them if space is needed for item 7 in section 3. Lowest priority.
6. **The letter's ending.** It closes on "I have not built reinforcement-learning environments or trained models." The disclosure is fine. Move it before a final sentence that restates fit.

## 5. Statements the record doesn't support

There are **no material misstatements.** Titles, dates, authorship positions, paper status ("under review," "working paper") and the talk labels (invited vs. competitively selected) all match the record. Minor points, none requiring change:
- "Hedges and caveats" (letter and résumé) is a gloss on the record's "qualifications." It is acceptable.
- "To measure those changes at scale" and "the choice between a cheap automated judge and slow human labeling" are the letter's framing, not claims in the record. They are harmless, but the trim suggested in section 3, item 2 removes the second one.
- The [ASK JORDAN] placeholder about agreement between Jordan and the RA is right to stay open. The record lists inter-rater reliability as a skill but does not say it was measured in Relay. Do not fill it in without Jordan's answer.