Daybook, 12 July 2026

Exp-2 · Decoding the RE / FDE landscape from evidence. This is a question usually answered with vibes: what actually separates a Research Engineer from a Forward-Deployed Engineer? Instead of guessing, I pointed a fleet of research agents at ~300 source-backed job descriptions and a large practitioner cohort, then ran everything through adversarial verification — every external claim carries a CONFIRMED/REFUTED verdict, and nothing gets graded by an LLM judge (that's circular, and it flatters).

What fell out: capability is table stakes. It's core in ~90–100% of every role's postings, so it can't be what separates them. The separator is the second axis a role trades capability against — cost/scale/latency, differentiation/coverage, or trust/assurance. The cleanest gradient is the doctorate: Research Scientist → Applied Scientist → Research Engineer → Applied AI Engineer → Forward-Deployed Engineer, monotonic. And RE isn't one role — it's ~11 archetypes (inference/kernels, post-training, evals/safety, agents, data, distributed training…), each optimising a different frontier. Skill demand is diffuse: only a handful of skills clear the must-have bar, so depth in one differentiator beats broad coverage.

Open the explorer full-screen ↗

The method generalises: multi-agent fan-out → adversarial verify → deterministic aggregate. And the failure mode is always the same — a metric quietly measuring the wrong thing.

The line I want to keep: undirected building isn't research engineering. The test is a named, falsifiable, measured claim about a frontier. Otherwise it's just engineering wearing a research costume.

Exp-3 · How early can you see an agent fail — from structure alone? When an agent runs a thousand tasks and leaves a thousand traces, how do you study the process, not just pass/fail? I checked the prior art first — TrajAudit, AgentLens and friends already exist — so the move was finding the delta, not re-having the idea.

Cheapest decisive experiment I could think of: take ~4,500 coding-agent trajectories, strip every word, and keep only the typed action skeleton — edit→error, run_tests→ok. Then ask a deliberately dumb classifier to predict success from structure alone, within-task (same problem, same model, so difficulty cancels).

0.64 within-task. Real signal, no language needed — and nowhere near solved. The why is the interesting part: healthy runs have a rhythm (create → edit → test → edit → test); doomed runs wander, editing without ever verifying. You can smell a bad run from its shape. Churn even ran against my intuition — successful runs throw away more of their own edits. And four simple “friction” features recover ~95% of the signal, which makes them a cheap process-reward candidate.

Deltas worth chasing: human-vs-agent trajectory divergence (almost nobody does it, and some datasets hand you the pairing for free) and interventional validation — repair the step you blamed, re-run, and measure whether the outcome actually flips. That turns “step k caused the failure” from a story into a causal test, which is the honest bar most of this literature skips.

Next: the ceiling check — let a frontier model read the full text of a few hundred traces, and measure how much the language actually adds over 0.64.

proved 0.64 within-task success prediction from action structure alone; RE / FDE explorer shipped over ~300 adversarially-verified job descriptions.

Settled · 12 Jul 2026