Materials · corvid/for-candidate/HANDOFF.md
# Handoff to Jordan: Corvid Labs, Research Scientist, Agent Evaluation
Nothing has been submitted, and no one has been contacted. This folder holds a draft cover letter and résumé (.txt and .pdf) plus `render.py`, which rebuilds the PDFs from the .txt files. You decide whether to send them, and you are the one who sends them.
How the drafts were made: an author agent drafted from your record, your writing preferences, the posting, and an editorial brief. Six reviewers checked the drafts across two rounds. I revised twice. No reviewer has seen this final version. Treat every sentence as unverified until you have checked it.
## 1. What you must verify personally before submitting
**Answer these first. They decide the letter.**
1. **What did you do with the qualification labels after the judges agreed poorly?** (Placeholder in the letter.)
2. **Did you measure agreement between yourself and the research assistant on the 300 messages?** (Placeholder in the letter.) If you did not, the letter currently asks Corvid to do something you did not do yourself. In that case rewrite the approach paragraph and say plainly that you did not, rather than leave the gap for an interviewer to find.
**Check these facts against your own memory and files.**
3. **Relay:**
- 300 messages, hand-labeled by you and one research assistant.
- Judges agreed well on dropped facts and poorly on dropped qualifications.
- The hidden-context error was caught before the first full run.
4. **The Management Science paper:**
- It is still under review, first round. Check again on the day you submit.
- The letter says only that it "reports the Relay experiments." It does not say it contains the judge validation. Add that only if it does.
5. **Kestrel:**
- The survey item tracked outages more than satisfaction with the catalog.
- The measure was rebuilt in 2023 and validated against later cancellation.
- You led a team of three direct reports.
6. **Parcel and the postdoc:** the bullets are as your record states them.
7. **Talks:**
- Is the 2019 Academy of Management "Late dissent" talk a competitively selected paper session, not an invited talk? Your record files it under "Invited" but describes it as competitively selected. The résumé lists it as competitively selected.
- Are the Varrow talk's title, venue, and date correct?
8. **Pricing rehearsal:**
- The résumé lists it under "Self-funded independent research," yet it was done before a client ran their experiment. Is that the right description of the arrangement?
- Is the planted-effect description accurate?
**Check that you are free to disclose these.**
9. **Kestrel's internal finding** that its satisfaction survey tracked outages. The letter and the résumé both describe it.
10. **The pricing client's experiment.** The client is not named, but the work is described.
**Check the claims made about Corvid.**
11. The letter cites three things:
- a March 2026 engineering blog post on grader and rater disagreement
- the head of research's June 2026 statement of the main risk (paraphrased, not quoted)
- Corvid's 2025 paper, whose human baseline came from staff
These come from a research packet, not from Corvid's site. Read the originals before you send.
**Fill these in.**
12. Months for every date range, email, phone, location, and the submission date.
13. Confirm that https://jordanhale.example is live and shows the working paper.
**Decisions that are yours.**
14. **The 300-message count.** It is in the letter. Your rule 3 keeps numbers out unless the reader needs the magnitude. I judged that a measurement team does need it. Remove it if you disagree.
15. **Things I cut from the Corvid résumé.** Confirm you accept these cuts:
- teaching
- informal mentoring at Parcel
- the self-produced "Building an agent testbed" walkthrough
**After editing.**
16. Run `python3 render.py` in this folder. Confirm the letter is still one page and the résumé is no more than two. The letter has about six lines free.
17. Submit through Corvid's applicant-tracking portal:
- résumé as a PDF
- cover letter (optional, but read by the research team)
Before you submit, check the fields the portal parses from the résumé: title, employer, and dates for each role.
## 2. What the package does not establish
The letter is honest about the first two items. The rest a reader may still notice.
- Any work with computer-use agents or long multi-step trajectories.
- Any experience building reinforcement-learning environments or training models. The posting lists this as preferred, and the letter states the gap once.
- Any study comparing agents with people. Relay has never been used with people. Your human-subjects work is lab experiments from 2016 to 2018.
- That the hand labels used as Relay's reference were themselves reliable. This depends on question 2.
- Any numeric agreement figures. Your record has none.
- Peer-reviewed AI work. The Relay paper is under review, and the simulated-participants paper is an unsubmitted working paper.
- Engineering strength for an eight-engineer team. "Python" appears only in Skills, and there is no public code link. If Relay's code can be shared, a link would help.
- That the approach in the letter works. It is a proposal, not a result.
## 3. Likely interview questions
These are drawn from the role reviewer (`../reviews/round2-role.md`):
- What were the agreement figures for each property, and how many messages per property?
- If you did not measure agreement between you and the assistant, why treat those labels as ground truth?
- How would you design a rater study for our long-task disagreements, and how would you replace a staff-only human baseline?
- How did you validate the Kestrel measure, and did you rule out outages driving both low satisfaction and cancellation?
- How did you find the hidden context, and could the same thing happen in our environments?
- Why independent work since 2024, and why a twelve-person company now?
## 4. Recommendation
**Apply.** Corvid's main stated risk is a score that measures the wrong thing, and its team has no one trained in measurement or human-subjects research. Your strongest record evidence is aimed at exactly that: validating language-model judges against hand labels, the Kestrel measure rebuild, and the invited talk on judge trust. The missing reinforcement-learning experience is listed only as preferred. Applying costs a résumé and a one-page letter.
There is one condition. Answer questions 1 and 2 before sending. If the labels' agreement was never measured, do not drop the application. Revise the approach paragraph so the letter does not ask Corvid for something you have not done, and be ready to say why you trusted the hand labels.