Vision reuse

Reuse in front of a vision model. NeuraFrame™ sits between your app and any vision model served over HTTP (ResNet, YOLO, RF-DETR, a custom classifier or detector) and reuses the model's own answer whenever an image recurs, so the heavy model runs once per distinct scene and every repeat or near-repeat is served from memory.

Connecting your model: HTTP or ONNX

NeuraFrame™ always sits in front of your model over HTTP. It does not replace your model or bundle one, and there is no separate ONNX loader inside the gateway: your model runs where it runs, and NeuraFrame™ is the reuse-aware proxy in front of it. Point the gateway's upstream at wherever your model listens, and every miss is forwarded there unchanged.

Either way the reuse layer is identical. Scene mode reuses per entity and region regardless of how the model is hosted: with a whole-image model it reuses the scene safely and escalates any material change; with a model you can query about a region (a detector, or any crop-answerable endpoint) it also sends only the changed crop, for the largest pixel savings.

What it does

Four reuse modes, differing in what each keys on, from the whole image up to a thing followed across a video (plus off to disable reuse):

The model's response is replayed opaquely, so this works the same for a classifier (a label) or a detector (bounding boxes): whatever JSON the model returned is what gets replayed. Novel images always reach the model. NeuraFrame™ never invents an answer.

How near reuse works

On each image NeuraFrame™ computes a compact perceptual code, looks it up in an approximate-nearest-neighbour index for the closest image already seen, and if the closest match is within the configured similarity threshold, replays the stored response without calling the model. Otherwise the request is forwarded and the new image and response are remembered.

Scene mode: reuse by what is in the image, not just the whole image

Exact and near reuse key on the whole frame, so any real change to the image is a miss and the entire image goes back to the model. Scene mode reuses at the level of the things in the image. It keeps a visual memory of the entities it has seen and, on each new image, works out what is the same, what merely moved, and what genuinely changed, then reuses the known parts and sends the model only the small unresolved region. In scene mode NeuraFrame™:

• recognises a known thing that has moved, rotated, changed size, or changed lighting, and reuses it instead of asking the model again;
• tells the same thing moved apart from another thing that merely looks like it - two identical items side by side stay separate instances;
• keeps identity separate from condition: a known object that is now dirty, wet, damaged, open, or missing a part stays the same object, and only the changed region is escalated;
• when a new object enters a known scene, reuses everything already known and sends the model only the new object;
• never silently reuses over a material change - a damaged or altered region is always escalated. A single similarity threshold cannot guarantee this: loose enough to save calls, it will replay a stale answer over a small but real change; strict enough to be safe, it stops saving.

Because it escalates only the small unresolved crop instead of the whole frame, scene mode sends the model a fraction of the pixels a whole-image cache sends, while never serving a stale answer over a real change. On a recurring, monitoring-style stream it sent about one percent of the pixels a near-duplicate cache sends at a safe threshold, with zero stale answers over material changes. The benefit is largest for recurring cameras, video, and repeated-object workflows in front of a heavy model, and smallest for streams of unique, unrelated images. Scene mode suits a model you can query about a region (a detector, or any endpoint that accepts a crop), since its advantage is sending small regions rather than whole frames. Labels are optional throughout: NeuraFrame™ recognises a structure whether or not it has a name, so a missing or changed label never triggers a repeat call.

Continuous mode: video, not a pile of frames

Scene mode treats each image on its own. Continuous mode treats a video (a live feed or a recorded file) as one connected experience. It carries what it has seen from frame to frame, so a thing that has already been classified is followed as it moves rather than re-sent every frame. On each new frame it estimates how the camera moved, predicts where the known things went, and asks the model only about a feature that is genuinely new or has materially changed. This is the mode for cameras that pan and walk, for drones and robots, and for moving through a space. In continuous mode NeuraFrame™:

• keeps a thing's identity across frames as the camera pans, zooms, or moves forward, so it is classified once and then tracked, not re-classified every frame;
• separates camera motion from object motion: the whole scene shifting because the camera turned does not create new things;
• handles occlusion and return: a thing that leaves the frame or is briefly hidden and then comes back is recognised as the same thing, not a new one;
• injects a request for the pixel location whenever it does ask the model, so the returned label binds to a track and can be followed from then on;
• reads forward motion (moving forward through a space, so things grow and pass) and 360 / equirectangular video (the frame wraps left-to-right), so forward-motion and panoramic feeds are first-class;
• learns the normal moving background (water, foliage, a running conveyor) and does not keep escalating it, while still escalating a real new feature within it;
• on a recorded file, uses a short look-ahead so it waits for a better view of an uncertain thing instead of calling the model on a poor first glimpse.

The same safety rule as scene mode holds throughout: a material change is always escalated, never silently reused, and a thing that only lacks a name is never re-sent just for that. Continuous mode is the natural fit for a video model you can query about a region; it sends the model a small crop for a new or changed feature and reuses everything else it is already following.

Two ways to run it. Live, as a gateway mode: set neuraframe vision mode continuous and stream your frames through the gateway in order, tagging each request with an X-NeuraFrame-Stream header so frames from the same camera are followed together. Or over a file or feed, with the bundled runner nf_continuous.py, which reads a video file, an RTSP or camera feed, or a folder of frames, calls your model (or an OpenAI-style VLM) only when a track needs it, and writes a run report plus optional feature-track, timeline, and unknown-region exports.

# over a recorded file or a live feed, with the runner:
python3 nf_continuous.py --mode continuous --input walk.mp4 \
    --teacher http://127.0.0.1:8080/detect --lookahead 8 \
    --export-feature-tracks --export-timeline
python3 nf_continuous.py --mode continuous --input rtsp://cam/live --equirect   # 360 feed
python3 nf_continuous.py --mode continuous --input frames/ --max-ai-calls 20    # folder of frames, capped

The runner ships in the Studio package alongside the gateway. The full runner options (--device, --teacher / --teacher-model, --equirect, --lookahead, --max-ai-calls, --offline, --output, and the --export-* flags) are listed by python3 nf_continuous.py --help.

Getting the tracking out: by time, as it goes

Continuous mode does not just save model calls; it builds a record of what it saw, when. Every dial below is optional and off by default, so you take exactly the visibility you need and nothing you do not. What the tracker keeps for each thing it follows: its identity across frames, a progressive label, its status (active, occluded, closed), its condition, its full time-ordered path, and its best (representative) view with the time that view was captured. There are three ways to reach it.

1. A live event stream, as it happens. With the runner, --emit-events <file|-> writes one JSON line per frame the moment the frame is processed (- means stdout), so you can watch or pipe the tracking live without waiting for the run to end. Each line carries the frame number, source time, camera-motion state, the number of active tracks, and, for every thing in view, its track id, current label, the route the tracker took (reused, escalated, watching), and its box. This is the time-ordered tracking, frame by frame:

python3 nf_continuous.py --mode continuous --input walk.mp4 --emit-events -
# {"frame": 12, "t": 11, "motion": "sideways", "active_tracks": 2,
#  "routes": [{"track": "track_0001", "label": "valve", "route": "REUSE_AND_UPDATE_LOCATION", "bbox": [..]}]}

2. Export files at the end of a run (with --output <dir>), each opt-in:

3. Live, from the gateway. When you stream frames through the gateway in continuous mode, each response carries an X-NeuraFrame-Tracking header with the same compact per-frame readout (motion, active tracks, and each track's id, label, route and box), so an HTTP client gets the same visibility as the runner without changing its request. The gateway also keeps a per-stream workspace (its neuraframe vision memory file) with the current tracks, their best views, and the latest frame summary, readable on demand. Best-view images are kept in memory by default; on a very long live stream you can turn that off with the keep_best_view_images knob to save memory while still getting metrics and trajectories.

Continuous mode is an early release. It is validated and safe by the same rule as the rest of NeuraFrame™ (a material change is never silently reused), and we are actively refining it as more people run it on their own video; if you put it to work on your footage we would genuinely value your feedback on what helped and what you needed that was not there.

Enable it

Point your app's base URL at the NeuraFrame™ Gateway and set the upstream to your vision model (see Configuration & profiles), then configure vision reuse from the CLI:

neuraframe vision on                 # turn on image reuse
neuraframe vision mode near          # off | exact | near | scene | continuous
neuraframe vision mode scene         # reuse by entities and regions, escalate only what changed
neuraframe vision mode continuous    # video: follow a thing across frames (tag each frame with X-NeuraFrame-Stream)
neuraframe vision threshold 90       # near-match strictness, 0..100 (higher = stricter)
neuraframe vision memory             # what the visual memory knows and what it has avoided sending
neuraframe vision status             # show current settings
sudo systemctl restart neuraframe-studio

If your endpoint takes the image as a base64 field inside a JSON body rather than a raw image POST, tell NeuraFrame™ where to find it, and set which request paths are vision endpoints if they are not the defaults (/v1/vision, /predict, /infer, /detect, /classify):

neuraframe vision field instances.0.image     # dotted path to the base64 image field
neuraframe vision match /v2/detect,/api/predict

Choosing the mode and threshold

All controls

Every setting can be set from the CLI (it writes the config; restart the service to apply) or edited directly in the gateway config. Restart after a change.

Equivalent config block:

"gateway": {
  "upstream_url": "http://127.0.0.1:8080",   // your model (HTTP, or ONNX served behind HTTP)
  "vision": {
    "enabled": true,
    "mode": "scene",                          // off | exact | near | scene | continuous
    "threshold": 90,                          // near mode strictness
    "image_field": "instances.0.image",       // omit for a raw image POST
    "match": ["/detect", "/classify"],
    "boxes_field": "detections",              // detectors only: enables the region path (pixel savings)
    "box_format": "xyxy_abs",                 // xyxy|xywh|cxcywh, _abs (pixels) or _norm (0..1)
    "box_key": "box"
  }
}

Two ways scene mode reuses, by whether a box format is set. With no boxes_field (a classifier, or any opaque whole-image model) scene mode reuses the whole-image answer only when the scene is unmoved and unchanged, and always escalates a material change: safe, but it forwards the whole image on a miss. With boxes_field set (a detector), it also reuses a moved object's boxes by shifting them (no model call), and on a real change sends only the changed entity's crop and merges the result, which is where the large pixel savings come from. If a response ever fails to parse in the configured format, scene mode falls back to forwarding the whole image, so it can never emit a broken answer.

Measured behaviour

On a recurring stream, near mode makes one model call per distinct scene and reuses the rest at the model's own answer. At 10x recurrence that is roughly 90 percent of model calls avoided at 100 percent agreement with the always-on model. The wall-time speedup grows with how heavy the model is: the more expensive each call, the closer the speedup tracks the calls avoided. See the Benchmarks for the measured ResNet-152 figures.

On a synthetic panning video (a moving camera, one object damaged mid-stream, one object leaving and returning), continuous mode made 3 model calls across 13 frames and sent about 1 percent of the pixels a whole-image cache sends, with zero stale answers over the material change, while a whole-image cache, which cannot follow a thing across a pan, served several stale answers at the same recurrence. It classified each object once, followed both through the pan, and recognised the object that left and came back as the same thing. The comparison harness is vision_bench3.py, re-run on every release.

Requirements and limits

Near and scene modes use numpy and Pillow; if they are not present the gateway degrades cleanly to exact-only reuse. Near reuse is keyed on whole-image appearance. Scene mode reuses per entity and keeps separate instances and concepts, so it recognises the same object moved or changed as well as different instances of the same kind, and escalates only the changed region. Its advantage depends on being able to separate the scene into entities: it is strongest on recurring cameras, monitoring, and repeated objects in front of a region-answerable model, and lighter on dense, cluttered natural scenes. As with all NeuraFrame™ reuse, novel inputs still require the model, a material change is always escalated rather than silently reused, and production use should be validated on your own stream.

Beyond reuse: complex identification

Vision reuse answers "have I seen this before?". Its 0.6.0 sibling answers a bigger question: "what are all the distinct things in this footage, and what does each one mean under MY standard?" You author a codebook (your codes, your grading, your rules), the tracker identifies every distinct thing across the footage with no model calls, and each one is coded once from a dossier of its best views, with every graded attribute a deterministic lookup rather than a model guess. The reuse gate is off by default in that mode and can be turned on. See Complex identification.