Complex identification with codebooks

Point it at footage, and get back a coded report against YOUR standard: every distinct thing identified once, coded once, graded deterministically, with the evidence saved. You bring the codebook; the model classifies; the engine decides. New in 0.6.0 and early-release: feedback is always appreciated. This page explains how it works; the field guide is the one-page reference to keep open while running it.

The problem it solves

Ask a vision model the same question on every frame and you get two failures at once. You pay per frame for footage that mostly repeats, and the model disagrees with itself: on identical frames a fresh call can return a different answer a third of the time in our measurements. Single frames are also the wrong unit of evidence. A still cannot separate a real mark from a glare hotspot, and a per-frame view cannot know whether fourteen sightings are fourteen things or one thing seen fourteen times.

Complex identification splits the job the way it should be split:

• Drive one tracks. Every frame is annotated and every distinct thing is followed across the footage with no model calls: colour, motion, and persistence do the work locally.
• Drive two codes. Each distinct thing is coded ONCE, from a dossier of its best views with the location burned in, against the codes and rules you supply. One considered call per real finding instead of a nervous guess per frame.

So the same finding seen on 80 frames becomes one coded record, not 80 observations, and cost tracks the number of actual findings, not the number of frames. On footage dense with findings that can still cost more than a single naive sweep, and that is the point: this mode buys accuracy, not savings. The economics are set out below.

What is a codebook?

A codebook is a folder you author. It carries your codes and what each means, your grading tables, your legality rules, and the plain-language rules the model must follow. Nothing about your domain is baked into NeuraFrame™: the engine is deliberately ignorant, and everything it enforces comes from your pack.

Some codes carry a numeric reading taken against a datum your standard defines (a level as a percent of a known reference, for example). The manifest declares which codes, the ask text in your words, and the valid range; out-of-range values are ignored and flagged, and readings drive change detection for STATE-mode codes. The full schema for every file, key by key: Codebook authoring.

A worked example ships in every install (vision/example_codebook, a fictional garden survey). It exists to show every mechanism; copy it and replace the content with your standard. Validate any pack with neuraframe codebook check <dir>: every problem is listed, nothing is silently ignored.

A codebook, in full

This is the shipped example, trimmed to the shape. A fictional garden survey, chosen so the mechanics are obvious; your pack has the same anatomy with your content.

manifest.json declares your domain's tables: what context exists, which field selects the grading column, your percentage bands, and your legality rules.

{
  "name": "garden-survey-example",
  "grading_field": "garden_use",
  "grading_columns": ["ornamental", "vegetable", "orchard"],
  "context_fields": [
    {"key": "garden_use", "assumed_note": "grading column assumed; grades may shift if wrong"},
    {"key": "survey_purpose", "note": "small-tier codes are only legal on a detailed survey"},
    {"key": "netting_present", "note": "netting codes are only legal when netting is recorded"}
  ],
  "defaults": {"garden_use": "ornamental"},
  "band_sets": {"A": ["<=10", "10-20", "20-30", ">30"]},
  "conditional_legality": [
    {"codes": ["SGS"], "allow_when": {"survey_purpose": ["detailed"]},
     "message": "small-tier growth codes are only legal on a detailed survey"}
  ],
  "surface_rules": [
    {"prefix": "MP", "require": ["bed", "path"],
     "message": "pooled moisture coded off the ground - pooled water sits low"}
  ]
}

registry.json is your codes. Three entries showing the three grading styles: banded (the model gives a band, the table gives the grade), fixed-per-column (the same finding grades differently by context), and ungraded state:

{"code": "WG", "title": "Growth, general", "family": "growth",
 "record_mode": "PROMOTABLE", "graded": true, "band_set": "A",
 "grading": {"ornamental": {"type": "threshold", "rules": [
     {"band": "<=10", "grade": 2}, {"band": "10-20", "grade": 3},
     {"band": "20-30", "grade": 4}, {"band": ">30", "grade": 5}]}}}

{"code": "FSB", "title": "Fixture, stake broken", "family": "fixtures",
 "record_mode": "SINGULAR", "graded": true,
 "grading": {"ornamental": {"type": "fixed", "grade": 2},
             "vegetable":  {"type": "fixed", "grade": 4}}}

{"code": "MW", "title": "Moisture, wet state", "family": "moisture",
 "record_mode": "STATE", "graded": false}

rules.md is plain language, and it rides with every model call. Short, imperative, about what the camera can decide:

- Use ONLY a code from the allowed list supplied. If nothing fits, answer NONE.
- Give a percentage as a BAND from the choices, never a number.
- A seam between beds is a border, never dense growth.
- A stake leaning is not a stake broken. Broken means separated fibres or two parts.

One finding, end to end

A patch of growth appears across forty frames of a survey pass. Drive one tracks it (no model calls) and keeps its sharpest views. Drive two sends ONE dossier and gets back the model's whole contribution, which is deliberately small:

{"real": true, "code": "WG", "band": ">30", "surface": "bed",
 "note": "dense growth across most of the bed width"}

The engine does the rest: the code is verified against the registry, legality passes, the band and the grading column resolve the grade from YOUR table, and the location comes from the tracked geometry. Forty frames become one record:

{"code": "WG", "title": "Growth, general", "shape": "run",
 "t_ms_start": 12400, "t_ms_finish": 31200,
 "location": {"mode": "clock", "clock": 7},
 "band": ">30",
 "grade": {"grade": 5, "determinacy": "band_resolves"},
 "seen": 38,
 "assumed_defaults": [{"field": "garden_use", "value": "ornamental",
   "note": "grading column assumed; grades may shift if wrong"}],
 "note": "dense growth across most of the bed width"}

Note what the record carries that the model never said: the grade (looked up, not guessed), the run span (it persisted past the continuous threshold, so PROMOTABLE promoted it), the clock position (computed from tracking), how many frames saw it, and the assumption the grading rested on. And note what did NOT happen: no invented percentage, no invented severity, no forty duplicate observations.

Had the model answered with a code not in the book, the finding would be kept and flagged for recoding, never dropped and never accepted. Had this been a routine survey and the code a small-tier one, the engine would have blocked it as invalid regardless of the model's confidence. That is the division of labour: the model sees; your standard decides.

The discipline: the model classifies, the engine decides

A camera supports classification and location. It does not support measurement, and it knows nothing about your asset records. So the model is only ever asked for what a camera can give:

Every record also carries assumed_defaults: any context field that fell back to a default is listed, so a reviewer can always see what a grading rested on. Defaulting silently is the failure mode this exists to prevent.

State codes are readers, not findings

A STATE-mode code describes a property of the survey, and "not a finding" does not apply to it. State codes never take a confirmation call: at most a cheap reading call for their declared value, asked in your pack's own words and range-validated. The registry's state_trigger splits the class in two. A survey_start state (a level: the pipe always has one) is anchored to the first frame unconditionally - if its value was never legible, the record still exists with the value marked unknown, because a missing start-state is a wiring error, not an observation - and it re-records only when the value genuinely moves. An on_change state (a material change, a lining change) exists ONLY at a transition: a change code with an unknown value is a claim that something changed with no idea what, so its absence is the normal, informative output. And a visible mark that merely carries a level-style reading is not a state at all - it is found or not found like any other observation, in a point record mode.

Under-call is a failure too, and residuals are last

Most quality machinery points one way: gates against inventing findings. A judge optimised only against over-call finds the answer that satisfies every constraint by claiming nothing - the "Other" bucket, the vague catch-all with no grade, no band, no host, and no region to justify. So your registry can mark codes residual_code: true, and residuals become structurally LAST: an answer naming one must explicitly reject the specific alternatives (no rejections, the answer goes to review), a specific proposal answered with a residual is logged as a downgrade event with both codes, and a residual overriding a proposal with broad frame support routes to review - one call overriding hundreds of detections needs more than a preference. The confirmation prompt states the distinction outright: caution means declining to overstate severity, never declining to name what is present. Unsure how much is the specific code with a wide band; a catch-all as the answer to everything is abstention, not caution.

Two free metrics ride on every run because under-call leaves no artifact to argue with: the per-code proposed-to-recorded conversion table (a code proposed hundreds of times and recorded zero is a systematic override, not a series of judgments) and code-mix concentration (most of the findings on one code is a degenerate distribution whichever code it is).

The mirror-image gate exists too: your registry can put an EXTENT FLOOR on codes that describe something with physical extent (min_extent: a minimum sighting count AND a minimum clock span, both required). An installed repair covers a section; a few frames of different-looking surface at one position is not one, however convincing its edge - appearance alone can never carry an extent-gated code, and a finding below the floor routes to review. Pack policy on a generic mechanism: the thresholds are your data, not engine opinion.

Record modes: how many records is one thing?

Annotating every frame that shows a condition is correct and is never thinned: that is what lets you scrub to any moment and see what is wrong right there. What your report needs is different, and your registry says which projection each code uses:

The verification pass: how many things are there?

First pass asks "what is this?" per view. Second pass asks "how many things are there?", which only a look across frames can answer. Two checks run before anything is reported. Identity: the same code in the same place close in time is one physical thing, however many tracks it became, so a finding is counted once. And the artifact test: a real feature is fixed to the scene (it looms, drifts, and exits as the camera moves), while a lens artifact is fixed to the camera (constant frame position while the scene advances). Camera-locked detections are withdrawn, and the test only runs when the scene is genuinely moving, so a static camera never triggers it.

Evidence channels

Different codes are proved by different kinds of evidence. A structural mark is a shading discontinuity: colour contributes nothing. Discoloration is hue: strip colour and there is nothing left. A rate (something flowing, something moving) is not in any still frame at all. So the registry can declare, per code, which channel proves it (evidence_primary):

Three behaviours follow. The confirmation call is handed the L and C renders beside the natural views, so the model judges a code in the channel where its evidence lives rather than being asked to simply look again. The L channel also runs as a cheap extra detector at identify (use_l_detector): features that lighting flattened out of the colour imagery become net-new candidates. And disagreement between channels is deliberately asymmetric, because "confirmed in both" would bias against the few colour codes: the one check that earns its keep is a structural code whose region has no luminance support but real chroma. That is probably a stain read as structure, and it goes to the review queue.

Rate codes get the same treatment percentages get: until rate metrics resolve the rung, the grade is a range, and the measured motion rides on the record for the reviewer. A classifier guessing a five-grade span, including an emergency call, from a still frame of a wet surface is exactly what this prevents.

The residual pass

Conditions overlap: one thing on top of another at the same place is normal footage, and a single-label detector reports only the dominant one. With residual_pass on, the run masks each confirmed finding's box-and-time extent, looks once more at what remains, and codes what emerges. Strictly additive (the second look can never revise the first), one extra pass maximum, and priced as a second detection sweep, so it is off by default.

The candidate funnel: what earns a model call

The model is the most accurate filter in the pipeline and the most expensive, so it must never be the FIRST filter. Everything cheap runs first, and the run reports the count at every stage. The rule the whole funnel follows: gates on the answer catch mistakes; gates on the proposal prevent the call.

The run log and segment.json print the whole funnel: raw detections, past the floor, tracks, screened in and out, pre-gated, features, confirmation calls. Two numbers deserve a look on every run. not_a_finding_rate: when most confirmation calls come back "nothing there", the funnel is leaking and the model is being paid to be your noise filter - it should sit well under 20%. And calls_avoided_vs_per_frame: the whole point of grouping; it reading zero is an alarm, not a statistic.

Running it

# validate your pack, set it, and run. --stub proves the whole spine with no model calls, free.
neuraframe codebook check /path/to/your-pack
neuraframe codebook use /path/to/your-pack
neuraframe codebook run --input footage.mp4 --out coded_out --stub
neuraframe codebook run --input footage.mp4 --out coded_out

Output: report.md (one row per finding: time, code, severity, where, the note - runs as start-finish spans), segment.json (the records, ratings, review list, cost block, and the exact configuration in force), observations.jsonl (every per-frame annotation), portal.jsonl (one row for every sighting of every coded finding, all rows of one finding tagged the same feature_id - the layer a viewing portal wants: scrub to any moment, see the finding and which feature it belongs to; the report collapses, this layer never does), and dossiers/ (exactly what was sent to the model, so you can always see what a decision was based on).

Every knob

Everything is configurable, with defaults if you cannot be bothered. The JSON config file has four blocks: run (sampling, detection, tracking; these are also command-line flags, listed by neuraframe codebook run --help), coder (the model endpoint, the dossier, location; config file only), records (projection and verification thresholds), and context (the survey facts). Every run records the configuration in force. Highlights:

Reuse while coding

When coding, the reuse gate is off by default: drive one already collapses repetition by tracking, and coding each distinct finding once is the cost model. If your footage revisits the same scenes across runs, turn reuse on and give it a go (neuraframe codebook reuse on); it can only skip frames that are materially the same, and the model remains the detector either way.

What it costs

Vision reuse exists to save model calls. Complex identification exists to produce a record you can stand behind, and that is a different economy. A dossier call is a bigger request than a single frame (several views ride in it), and on footage dense with findings the coding pass can cost MORE than one naive per-frame sweep would have. That is intended. This mode is for when accuracy outranks cost. What the candidate funnel guarantees is that the calls you do pay for are spent on plausible, distinct candidates: noise never reaches the model, refuted proposals never reach the model, and one thing seen in fragments is confirmed once, not once per fragment.

The comparison that actually matters is not one clean pass, because a per-frame pass does not give you one clean pass: it gives you a model disagreeing with itself across frames, duplicate observations of the same thing, and an output you re-run and hand-reconcile until something is usable. That loop is the real baseline, and one considered call per finding, on the best evidence, with the decision trail saved, replaces it. Every run writes an exact cost block (calls and tokens, and dollars if you set your prices), so the economics are measured on your footage, not promised on ours.

Choose per job: reuse for volume, codebooks for accuracy, and the reuse gate is there to turn on when your footage revisits the same scenes.

What it does not claim

• The model still classifies, and models are imperfect: the gates catch off-book, illegal, and contradictory answers, and route them to review rather than pretending they did not happen.
• A code only ever comes from a confirmation call. When a confirmation call fails, the finding is kept uncoded in the review queue with its evidence - it is never turned into a coded, graded record by a fallback. The enum lock has no side doors.
• Nothing here measures. Where your standard needs a dimension, the field stays blank or comes from your own records; where a grade needs a percentage, a band either resolves it or you get a range.
• Verification runs on the evidence available: it improves counts; it does not make them perfect.

Run the shipped board (python3 vision/nf_codebook_board.py) on your own install to see every mechanism proven, free.