Walkthrough · Materials

Materials from the multi-agent application walkthrough

These are the input files and written outputs of the recorded demonstration. The candidate, Jordan Hale, and the employers are fictional. The displayed copies rename a university and a web address because their original fictional names matched a real institution and a live domain. The underlying session files are unchanged. Documents are shown as text. The PDFs are not reproduced.

The six reviewers

Each reviewer had a distinct task and a restricted set of sources. The roster lists what its prompt permitted it to open and the main sources it could not use. These exclusions are selected to explain the design, not to enumerate every other file in the project. The visual reviewer saw page images, not the draft PDFs or text files.

ReviewerRoundAllowed sourcesKey exclusionsAsked about
Unfamiliar reader1Posting, first-draft letter and résuméCandidate record, editorial brief, employer packetWhat a reader without the background files cannot follow, attribute to Jordan, or connect to the role
Factual1Candidate record, first-draft letter and résuméPosting, employer packet, editorial briefMaterially misleading claims about Jordan's roles, findings, credit, dates, or publication status
Editorial1Writing preferences, first-draft text and PDFsCandidate record, posting, employer packetAll ten writing rules, including scripted punctuation, vocabulary, font, color, date, and page-count checks
Role2Posting, full employer packet, revision 1 draftsCandidate record, editorial briefWhether the research argument meets Corvid's actual needs, including the grader analogy and proposed study
Evidence selection2Candidate record, posting, full employer packet, revision 1 draftsEditorial briefWhether the strongest available record evidence was selected, placed prominently, or omitted
Visual2Three page images, posting, employer packet's application-mechanism sectionCandidate record, draft text and PDFs, rest of employer packetFirst scan, visual hierarchy, legibility, and parser-field errors predicted from page images

How the cover letter changed

These passages compare the first draft with the version given to Jordan at the handoff. Ellipses mark omitted words within quoted excerpts. The last column traces changes to reviewer findings or to Claude's own revision decisions. The handoff letter still contains two unanswered questions about the Relay evidence, and no reviewer checked that final version.

PassageFirst draftHandoff draftWhat changed it
Source of the opening claimYou have said in public that your automated graders and your human raters often disagree on long tasks, ... and that you do not know which side is in error.In your March engineering blog post, you wrote that your automated graders and human raters often disagree on long tasks, ... and that you do not yet know which disagreements are grader errors and which are rater errors.Unfamiliar reader (U1) identified the missing source. Role reviewer (R5a) noted that "which side" misrepresented Corvid's per-disagreement uncertainty. Claude applied the changes in revisions 1 and 2.
The analogyThe parallel is in the structure. The automated judge handled the blunt outcome, whether the fact was kept, and failed on the subtler one, whether the qualification was kept. Your graders and raters split along a similar line, between the end state and the route.The parallel is partial. My judges scored qualifications unreliably, and some of your disagreements may be of that kind. Others may arise because a grader checks the end state and does not score the route at all.Unfamiliar reader (U6) identified the implied assumption that Corvid's graders were wrong. Role reviewer (R2) distinguished scoring a property unreliably from not scoring that property at all.
The proposed approachI would measure how well raters agree with one another on the disputed trajectories, score route and end state as separate properties, and only then ask which disagreements are grader errors.Then I would estimate rater agreement on agreed as well as disputed trajectories, across task lengths, and sort each disagreement into grader error, rater error, or a task specification that allows both readings.Role reviewer (R3) identified the selection problem in examining only disputed cases, the missing task-specification category, and the omission of trajectory length.
Status of the paperThe paper reporting this work is under review at Management ScienceA paper reporting the Relay experiments is under review at Management Science, in its first round.Factual reviewer (F1) flagged an implication that the under-review paper contained the judge validation. Claude clarified the paper's scope and added first-round status under writing rule 8.
Scale of the hand checka research assistant and I coded a set of messages by handa research assistant and I labeled 300 messages by handUnfamiliar reader (U11) found "code" ambiguous. Evidence-selection reviewer (E5) argued that the 300-message count was material to judging the validation. Claude added it in revision 2, reversing the brief.
Experience with human participantsmy experience with human participants comes from laboratory experiments on group decisions with student participants during my postdoc, where I also managed the participant pool, and from the Kestrel survey work.Your 2025 paper notes that its human baseline came from your own staff. During my postdoc I ran laboratory experiments with student participants and managed the lab's participant pool. A comparison group drawn from outside the company is the study I would design.Role (R4) and evidence-selection (E3) reviewers connected the postdoc work to Corvid's staff-only human baseline. Claude revised the paragraph without claiming Jordan had studied agents against people.
A claim about the employerThat is a definition to settle with customers(removed)Claude removed this unsupported suggestion about Corvid's customer process in revision 1. The change log identifies it as an orchestrator edit, not a reviewer finding.

Changes to the résumé

Files

Inputs given to the session

Agent prompts

Screening and the brief

Reviews and triage

Documents at each stage