Scout report, private AI research

Materials · screening/scout-private-ai.md

# Scout report: Private AI research track

Sources read: the Corvid and Brightwave postings, the Corvid and Brightwave employer packets, and record.md. No other files were read.

Record note: record.md lists the 2019 Academy of Management talk under "Invited," but its own text says it was a competitively selected paper session, not an invited talk. It is treated below as a competitively selected paper session. The only invited talk is the May 2025 Varrow workshop.

---

## 1. Corvid Labs: Research Scientist, Agent Evaluation

**What Corvid is actually selecting for**
- Someone who can tell whether a measurement means what Corvid thinks it means. The head of research said this about the next hire (June 2026 talk).
- A known open problem: on long tasks, graders and human raters disagree on about a third of trajectories. Corvid says "We do not yet know which disagreements are grader errors and which are rater errors" (March 2026 blog).
- Better human baselines. The 2025 paper's baseline was ten employees, which the paper itself lists as a limitation.
- A skill the team lacks: no one of the twelve has a background in human-subjects research or psychometrics.

**Jordan's evidence (from the record)**
- Validating judges against human coding: Relay's LLM judges were checked against 300 messages that Jordan and one research assistant hand-coded. The judges agreed well on dropped facts and poorly on dropped qualifications. This is the same type of problem as Corvid's grader-versus-rater question.
- Checking whether a measure measures the right thing: at Kestrel, Jordan found that the main satisfaction item tracked outages rather than catalog satisfaction, rebuilt the measure, and validated it against later cancellation.
- Invited talk: "When can a language-model judge be trusted?" at the Measurement in AI Workshop, Varrow, May 2025.
- Human-subjects background: lab experiments with student participants and management of the participant pool (postdoc, 2016 to 2018). PhD in Organizational Behavior.
- Paper 2 (JAP 2021, published, third of four authors) supplied the measurement model and robustness analyses.
- Under review: "Message drift in chains of language-model agents," Management Science, first round, submitted July 2026. Not accepted. Covers LLM agents only.
- Working paper: "Rehearsing experiments with simulated participants" (2025), posted on jordanhale.example, not submitted. The pricing rehearsal behind it used only simulated agents and says nothing about real shoppers.
- Self-produced: a recorded walkthrough, "Building an agent testbed" (2025). It was not invited and not presented to an audience.
- Skills: Python, inter-rater reliability, language-model evaluation, experimental design.

**Missing**
- No experience building RL environments or training models. The record says so explicitly, and the posting lists RL environments as preferred.
- No work with computer-use or software-operating agents. Relay studies message passing between LLM agents.
- No grading of long multi-step trajectories.
- No agent-versus-human comparison studies. Relay has never been used with people.
- No AI-specific work is peer-reviewed or published yet.

**Materials and submission**
- Upload a résumé as a PDF through a Greenhouse-style ATS. The ATS parses fields before a person reads it, so dates and titles must be machine-readable.
- A cover letter is optional, but the research team reads every one. In practice it is the main place to make the measurement-validity case.

**Strongest case against applying**
- Corvid is an ML and engineering shop (eight engineers, three ML researchers, a head of research from a model lab). Jordan's background is organizational behavior and social science.
- Jordan has no RL environment, model training, or computer-use agent experience, and all AI-specific work is from 2024 or later, self-funded, and not yet peer-reviewed.
- A reader could screen Jordan out as an outsider to ML.

**Title vs actual work**
- Mild divergence. The title and posting read like a standard ML research-scientist role, with RL environments preferred.
- The packet shows the real gap is measurement validity and human baselines, which is psychometrics and human-subjects work the team lacks.
- This divergence favors Jordan, but only a cover letter will make that visible past the posting's wording.

**Verdict: pursue.** The employer's stated top need (grader validity and human baselines) matches Jordan's strongest documented evidence, and the missing RL experience is listed only as preferred.

---

## 2. Brightwave Analytics: Staff Data Scientist, Production Machine Learning

**What Brightwave is actually selecting for**
- Production ML operations: on-call reliability and model monitoring (the hiring manager's public posts).
- Hands-on Spark, Airflow, and cloud work.
- A tech lead or manager for four data scientists.
- The company has no research function and does not publish.

**Jordan's evidence (from the record)**
- Management: led three direct reports at Kestrel (2022 to 2024) and mentored two analysts informally at Parcel.
- Python and SQL.

**Missing**
- No ML models deployed to production. The posting requires five or more years.
- No Spark, Airflow, or cloud platform experience in the record.
- No forecasting, retraining, monitoring, or incident-response experience.
- No model training. The record states this explicitly.

**Materials and submission**
- Résumé only, through the ATS portal.
- Shortlisted candidates take a timed coding assessment.
- On-site in Austin five days a week.

**Strongest case against applying**
- Jordan fails the core hard requirements: production deployment, the named stack, and model training.
- None of Jordan's research or measurement strengths matter here, because the company does no research.
- The five-day on-site Austin requirement is an added constraint. The record does not say where Jordan is located.

**Title vs actual work**
- No material divergence. "Staff Data Scientist, Production ML" describes ML engineering, operations, and team leadership, and that is the actual work. This role does not belong in a private AI research track.

**Verdict: drop.** Jordan meets none of the hard technical requirements, and the role involves no research.