Materials · corvid/reviews/editorial.md

# Editorial review: Corvid Labs, Research Scientist, Agent Evaluation

Reviewer: editorial. Files read: writing-preferences.md, cover-letter.txt, resume.txt, cover-letter.pdf, resume.pdf. Mechanical checks: `/private/tmp/mapp/run/work/corvid/reviews/editorial-check.py` (raw output below). `[ASK JORDAN: ...]` placeholders are not counted as violations.

## Rule-by-rule

| # | Rule | Result | Evidence |
|---|------|--------|----------|
| 1 | Describe work so an outsider can follow: what, decision, why it mattered | Pass (minor) | Relay paragraph is a model of the rule: what was built ("a testbed for studying how information changes as it passes along chains and networks of language-model agents"), the decision ("The decision to trust the cheap judge or the slow human one therefore had a different answer for each property"), and why it mattered ("a single overall agreement figure would have hidden the property where the judge failed"). Two weak spots. (a) "the model-access route was adding hidden context to every agent call" is jargon to an outsider: what is a "model-access route"? (b) Kestrel gives the finding and the fix but not the decision it required or what changed because of it: "I found that the main item ... tracked recent outages more than satisfaction with the catalog. I rebuilt the measure and validated it against later cancellation." |
| 2 | Do not state fit; produce it | Pass | No fit claims. Fit is produced by structure: "I have worked on a smaller problem with the same structure." and "Your graders and raters split along a similar line, between the end state and the route." The closing paragraph proposes how Jordan would approach Corvid's question rather than asserting suitability. |
| 3 | No numbers in letters unless magnitude is needed | Pass | Only dates appear ("Since 2024", "May 2025", "October 12, 2026"). The "300 messages" count stays in the résumé, not the letter. No volume counts. |
| 4 | Do not list projects one sentence each; choose and explain | Partial | Relay is chosen and explained at length. The Kestrel paragraph is a two-sentence project mention whose link to Corvid is left implicit: "I have asked whether a measure means what we think it means on human respondents as well. At Kestrel Media I found ...". The postdoc lab work appears only as a clause in a list: "my experience with human participants comes from laboratory experiments on group decisions ... and from the Kestrel survey work." Either tie Kestrel to Corvid's rater problem (a measure that tracks the wrong thing is the same risk as a rater or grader that tracks the wrong thing) or cut it. |
| 5 | State noticeable gaps once, without apology; do not raise unprompted ones | Pass (check against posting) | Gaps stated once, flatly: "Relay messages are text passed between agents, not multi-step computer-use trajectories." and "I have not built reinforcement-learning environments or trained models." No apology language. Résumé handles the independent-researcher period plainly ("Self-funded independent research, no employer.") and scopes Relay ("Used only with language-model agents, not with people."). I could not open the posting, so I cannot confirm that RL environments or model training are things the reader will look for. If the posting does not ask for them, that sentence raises a gap the documents give no reason to raise and should go. |
| 6 | No em dashes, no semicolons, no contractions, American spelling | Pass | Script: 0 em dashes, 0 en dashes, 0 "--", 0 semicolons, 0 contractions in both .txt and both PDFs. Possessives "company's", "lab's" correctly not counted. No British spellings ("catalog", "analyzed", "Organizational" are American). |
| 7 | No promotional language | Pass | Script: 0 hits on rule-7 list plus synonyms (thrilled, eager, state-of-the-art, groundbreaking, utilize, spearhead, impactful, novel, align, etc.). "Competitively selected paper session" is a status descriptor, not promotion. |
| 8 | Exact status | Pass (minor) | Letter: "under review at Management Science", "invited talk". Résumé: "Under review at Management Science (first round, submitted July 2026)", "Working paper, posted on jordanhale.example", "Published" (x2) with authorship position, "Invited talk", "Self-produced recorded walkthrough". All exact. Minor: the letter says the invited talk was "at Varrow University", the résumé says "Measurement in AI Workshop, Varrow University". The letter version can be read as a university colloquium. Name the workshop in the letter so the two documents match. |
| 9 | Résumé: at most 2 pages, sentence-case headings, one serif, black text, parseable month-year ranges | Partial (pending placeholders) | Pages: 2 (page 2 text ends at 38% of the page). Headings: all seven in sentence case. Fonts that draw text: Times-Roman and Times-Bold only (one serif family). A Helvetica resource is declared on every page but draws no glyphs (a generator default). This is harmless to a reader, but a font-listing check would report Helvetica, so regenerate without it if easy. Text color: all 84 spans #000000; no colored rules. Dates: all six role and project ranges are "[ASK JORDAN: month] YYYY to [month] YYYY / present", so they will parse once months are filled. Filling them shortens each line, so page count is not at risk. Education is year only ("2016", "2010"), which is standard for degree dates and is not a range. The 2019 Academy of Management talk is year only and the script flags it, but it is a single event and needs no range. A month would be more consistent with "May 2025". |
| 10 | Cover letter: one page, addressed to hiring team, no restated résumé | Partial | One page (text ends at 83% of page). The two long placeholders will be replaced by text of similar length, so it should still fit, but the margin is small. Addressed to "Corvid Labs research team" / "Dear Corvid research team", not "hiring team" as the rule states. Restated résumé, nearly verbatim in two places. Letter: "At Kestrel Media I found that the main item in the viewer-satisfaction survey tracked recent outages more than satisfaction with the catalog. I rebuilt the measure and validated it against later cancellation." Résumé: "Found that the main item in the viewer-satisfaction survey tracked recent outages more than satisfaction with the catalog. Rebuilt the measure in 2023 and validated it against later cancellation." Letter: "the model-access route was adding hidden context to every agent call, which would have contaminated every condition." Résumé: "The model-access route was adding hidden context to every agent call, which would have contaminated every condition." |

## Findings, ranked by importance

1. **Rule 10 / Rule 4: Kestrel paragraph restates the résumé and is not explained.** The sentence is copied almost word for word from the résumé bullet and gets no explanation of why it matters to Corvid. Either connect it to the grader-versus-rater problem in Jordan's own words (a measure that tracks something other than its target is what Corvid fears in its graders or raters) or delete it. The paragraph is also the cheapest place to recover space if the filled placeholders push the letter past one page.
2. **Rule 10: the hidden-context sentence is verbatim from the résumé.** In the letter it also stands out as a separate fact attached to the judge-validation story. Either rephrase it and say why it matters to agent evaluation (silent contamination of every condition is the kind of error an evaluation team must catch before scoring), or drop it from the letter and leave it to the résumé.
3. **Rule 10: salutation.** The rule says "addressed to the hiring team"; the draft says "Corvid research team". Change it to "Dear Corvid Labs hiring team" (and the address line to match), unless Jordan intends the deviation.
4. **Rule 5: confirm the RL/model-training disclaimer against the posting.** It is well stated, but it belongs only if the posting signals those skills. Otherwise it raises a gap without cause.
5. **Rule 1: "model-access route" is opaque to an outsider.** Say in plain terms what added the hidden context (for example, the API wrapper or provider gateway).
6. **Rule 8: name the venue in the letter.** Change it to "at the Measurement in AI Workshop at Varrow University" so it matches the résumé and does not read as a university-hosted invitation.
7. **Rule 9: housekeeping.** Fill the 10 month placeholders, which also completes the parseable ranges. Consider a month for the 2019 talk. Optionally regenerate the PDFs without the unused Helvetica resource.
8. **Rule 10: watch length after filling placeholders.** The letter is at 83% of the page with two long placeholders still in it. Re-run the page count after Jordan answers.
9. **Outside the ten rules, for the fact reviewer:** the letter opens with "You have said in public that your automated graders and your human raters often disagree ...". This claim needs a source. Separately, the résumé's "before a client ran it" (pricing rehearsal) sits beside "Self-funded independent research, no employer". A reader may ask who the client was.

## Script raw output

```
=== TEXT: cover-letter ===
[ASK JORDAN] placeholders: 5 (excluded from checks below)
em dash (U+2014): 0
en dash (U+2013): 0
double hyphen '--': 0
spaced hyphen ' - ' (dash stand-in): 0
semicolon: 0
contractions: 0 
possessives (allowed, listed for audit): 0 
banned/promotional words: 0 
possible British spellings: 0 

=== TEXT: resume ===
[ASK JORDAN] placeholders: 13 (excluded from checks below)
em dash (U+2014): 0
en dash (U+2013): 0
double hyphen '--': 0
spaced hyphen ' - ' (dash stand-in): 0
semicolon: 0
contractions: 0 
possessives (allowed, listed for audit): 2 [(16, "company's"), (21, "lab's")]
banned/promotional words: 0 
possible British spellings: 0 
-- résumé date check (rule 9) --
   line 6: PARSEABLE ONCE MONTH FILLED placeholder month: Independent Researcher, [ASK JORDAN: month] 2024 to present
   line 9: PARSEABLE ONCE MONTH FILLED placeholder month: Research Lead, Measurement, Kestrel Media (streaming service), [ASK JORDAN: month] 2022 to [ASK JORD
   line 14: PARSEABLE ONCE MONTH FILLED placeholder month: Senior Research Scientist, Parcel (consumer delivery app), [ASK JORDAN: month] 2018 to [ASK JORDAN: 
   line 19: PARSEABLE ONCE MONTH FILLED placeholder month: Postdoctoral Fellow, Ridgeview University Decision Lab, [ASK JORDAN: month] 2016 to [ASK JORDAN: mon
   line 25: PARSEABLE ONCE MONTH FILLED placeholder month: Relay, self-funded testbed for language-model agents, [ASK JORDAN: month] 2024 to present
   line 31: PARSEABLE ONCE MONTH FILLED placeholder month: Pricing-experiment rehearsal (simulated), [ASK JORDAN: month] 2025 to [ASK JORDAN: month] 2025
   line 45: NOT PARSEABLE : - Paper presentation. "Late dissent." Academy of Management annual meeting, 2019. Competitively sele
   line 53: education, year only (single date, not a range): PhD, Organizational Behavior, Ridgeview University, 2016
   line 56: education, year only (single date, not a range): BA, Psychology, Carrow College, 2010
-- résumé headings (sentence case) --
   line 4: 'Experience' -> sentence case
   line 23: 'Projects' -> sentence case
   line 35: 'Papers' -> sentence case
   line 42: 'Talks' -> sentence case
   line 47: 'Other work' -> sentence case
   line 51: 'Education' -> sentence case
   line 58: 'Skills' -> sentence case

=== PDF: cover-letter ===
pages: 1 (limit 1) -> OK
fonts used (span counts):
   Times-Roman: 38
   Times-Bold: 1
font resources: ['Helvetica', 'Times-Bold', 'Times-Roman']
font families: ['Times'] -> one family
sans-serif fonts detected: none
text colors (span counts): {'#000000': 39}
non-black text spans: 0
last page fill: text ends at y=661 of 792 (83%)
placeholders in PDF (whitespace-normalized): 5
fonts that draw any glyph (incl. whitespace): ['Times-Bold', 'Times-Roman']
font resources declared but drawing no text: ['Helvetica']
vector drawing colors (rules/lines, not text): none
em dash in PDF text: 0
en dash in PDF text: 0
semicolon in PDF text: 0
PDF text matches .txt (whitespace-normalized): True

=== PDF: resume ===
pages: 2 (limit 2) -> OK
fonts used (span counts):
   Times-Roman: 65
   Times-Bold: 19
font resources: ['Helvetica', 'Times-Bold', 'Times-Roman']
font families: ['Times'] -> one family
sans-serif fonts detected: none
text colors (span counts): {'#000000': 84}
non-black text spans: 0
last page fill: text ends at y=298 of 792 (38%)
placeholders in PDF (whitespace-normalized): 13
fonts that draw any glyph (incl. whitespace): ['Times-Bold', 'Times-Roman']
font resources declared but drawing no text: ['Helvetica']
vector drawing colors (rules/lines, not text): none
em dash in PDF text: 0
en dash in PDF text: 0
semicolon in PDF text: 0
PDF text matches .txt (whitespace-normalized): True
```

Note on the "NOT PARSEABLE" line 45: it is a single-event date (a 2019 conference talk), not an employment range. It is listed for completeness, not as a violation.