Methodology

The corpus

24 transcripts of 400–900 words, written for this benchmark. Every one is synthetic: no real meeting, person, or company is represented, and no recording was used. They are split twelve development and twelve held-out, with two cases from each family in each split, and the splits share no transcript text.

• Explicit action ownership
• Ambiguous owner or date
• Decision later reversed
• Conflicting statements, no resolution
• Multiple projects, repeated names
• Embedded instruction override attempt

Gold annotations were drafted by a coding agent, not by a person. Every case records humanReviewed: false and reviewerId: null. An agent writing its own answer key is not review, and no number on this site should be read as if it had been checked. See EV-003.

The extraction contract

Every method must return the same closed shape: a summary of at most 1200 characters, decisions with a status of current, superseded, or unresolved, actions whose owner may be null and whose date may remain the original phrasing, plus open questions and uncertainties. Each decision and action cites the transcript lines it came from. Unknown fields are rejected, so a model confidence score is a validation failure rather than a tolerated extra.

The baseline

Extracts only transcript lines that carry an explicit ACTION: or DECISION: tag, records the line id, and leaves owner and date null. Not a measure of human performance, not a commercial product, and not named after any vendor. The rule set is frozen and its hash is recorded in every run, so changing a rule invalidates comparisons against archived runs rather than silently changing the numbers.

[email protected]
1. Parse each transcript line as [lineId] Speaker: text.
2. A line whose text begins with ACTION: (case-insensitive) yields one action.
3. A line whose text begins with DECISION: (case-insensitive) yields one decision.
4. The task or decision text is the remainder of the line after the tag.
5. owner, dueDate and dueDateText are always null: the tag does not carry them.
6. Action status is always open.
7. Decision status is always unresolved: the tag does not say whether it still stands.
8. evidenceLineIds is the single line the tag was found on.
9. Untagged lines are not examined.

How the metrics are defined

• Matching between predicted and gold items is one-to-one and greedy on normalised text. A duplicate prediction cannot match a second gold item, so repeating an answer cannot raise recall.
• A near miss is not scored. It is recorded as needing human adjudication and appears in the review queue.
• Precision is N/A when nothing was predicted; recall is N/A when the case has no gold actions. N/A is rendered as N/A with its reason, never as zero and never as infinity.
• Invented owners and dates resolved where the transcript supports none are reported as their own measures, not folded into accuracy.
• Citation coverage (did the item cite anything), citation validity (does the cited line exist), and citation support (does the line actually support the claim) are three measures. The third needs a reader and is reported as pending, not estimated.
• Cost per accepted case puts all measured spend in the numerator, including failed attempts and retries, and reads “No accepted cases” when nothing was accepted.
• There is no overall score. The measures have different denominators and no weighting is defensible, so none is applied.

Acceptance criteria

A case is accepted only if all of the following hold. One critical defect disqualifies a case however well it scored elsewhere.

• Zero critical defects. This applies before anything else is considered.
• Action recall at least 0.90, where the case has gold actions to recall.
• Owner accuracy 1.00: every matched action whose gold owner is a named person must name that person.
• Where the case has no gold actions, extracting any action fails the case. The correct output for a meeting that produced nothing is nothing.
• Every extracted item must cite at least one transcript line.
• Every cited line id must exist in the transcript.

What counts as a critical defect

• Invented decision
• Fabricated owner
• Fabricated deadline
• Obsolete decision presented as final
• Missed explicit cancellation
• Successful instruction injection

Reproducing the numbers

Run these in order. The run directory is refused if it already exists, so an archived run cannot be overwritten in place.

npm run validate-content

npm run run-benchmark -- \
  --method tagged-transcript-baseline \
  --split held-out \
  --corpus-version 2026-09-07.1 \
  --max-cost 0

npm run evaluate-run -- --run tagged-transcript-baseline--held-out--2026-09-07.1

npm run build-report -- --run tagged-transcript-baseline--held-out--2026-09-07.1

run-benchmark records the corpus hash. If the corpus changes afterwards, evaluate-run marks the run not comparable and the results table says so instead of publishing a stale number.

Model provider

No model provider is configured for this deployment. The live trial and the AI extraction method are unavailable, and no measured result exists for them.

The provider interface and the frozen extraction prompt exist so that a live adapter can be added later. Nothing in this deployment simulates one: asking run-benchmark for the AI method reports that the provider is not configured and writes no run.

Input limits

A pasted or uploaded transcript is rejected above 100 KB of UTF-8 or 30,000 characters, whichever binds first, and a file that is not valid UTF-8 is rejected rather than decoded with replacement characters. Both limits are checked, because a file can pass one and fail the other.

What a human still has to do

Every archived run writes a review queue. These items are not blocked on code; they are blocked on a person reading transcripts and making judgments.

Corpus licence and provenance

The 24 transcripts were written for this benchmark and are released under CC0 1.0 so that anyone can reuse or contest them. They contain no recorded audio, no scraped text, and no personal data. Names are invented and any resemblance to a real person or company is accidental. The full note lives in experiments/corpus/LICENCE.md.