---
title: Reference ingest
description: Read one reference clip into a timestamped manifest (shots, source-resolution OCR, audio facts, a labeled strip) in one call. Dest first.
---

> **Dest first (#7944).** `POST /v1/reference-ingest` and the MCP tool
> `reference_ingest` are listed where `SUME_COM_REFERENCE_INGEST_ENABLED`
> allows it (development auto-on, production opt-in). Everything else —
> [video inspect](/models/video-inspect), `video_frames`, `video_trim`,
> `audio_detach` — is unchanged; this surface is additive.

Reference ingest 1.0 is the **deterministic preflight** a reference clip goes
through before an agent plans a remix, a brief or a face swap. It reads
**one** `media.sume.com` clip the workspace already owns (≤ 300 s) on the
Modal media runtime and answers a `ReferenceVideoManifest`:

- `shots[]` — frame-exact cuts (ffmpeg `scdet` and PySceneDetect voting
  together), one sharpest source-resolution keyframe per shot, palette,
  luma, a motion class; the list tiles `[0, duration]` with no gap.
- `text_tracks[]` — PP-OCRv5 (Korean + Latin) read at **source resolution**
  on deduplicated frame states, merged across frames into lines with text,
  normalised box, span, confidence, persistence, a `card_id` grouping and a
  `role_hint`. Nothing is corrected: a line under the confidence threshold is
  `needs_verification` and its native-resolution crop is returned.
- `audio` — loudness gate (`silent` when integrated loudness ≤ −60 LUFS or
  the true peak is −inf), Silero VAD speech presence, librosa beats when the
  track is music, and — only with `speech.allow_billed_stt` — a Sume STT 1.0
  transcript with words and sentence segments.
- `overview` — one labeled strip (up to six tiles, gutter labels such as
  `S0 0.00–4.28s (t=2.14)`) that survives every model transport.
- `coverage`, `uncertain[]`, `provenance.timings_ms` — what ran, the only
  reasons to look again, and where the time went.

The job id **is** the resource id (`sume/reference-ingest-1.0`, type
`reference_ingest`). Read-only: the source is never re-encoded and no MP4 is
produced.

```text
POST /v1/reference-ingest
GET  /v1/reference-ingest/:id
```

Hosted MCP: `reference_ingest` (text result). The Sume Agent host adds
`sume-agent__reference_ingest`, which returns the same manifest **and attaches
the strip plus up to four low-confidence crops as inline images**, so the
model reads facts and pixels in one turn without a download.

## Create an ingest

Required: `video_url` (this workspace’s `media.sume.com` artifact, asset or
chat attachment). `Idempotency-Key` is required. Default `mode` is **`sync`**:
the handler waits up to 30 seconds and answers `200` with the finished
manifest, else `202` with the queued job (`jobs_wait` → `jobs_result`, result
kind `reference_video_manifest`).

Optional fields:

| Field | Meaning |
|---|---|
| `purpose` | `reference_remix` (default), `brief_format`, `face_swap`, `qa`. Stored, not interpreted. |
| `ocr.languages` | `["ko", "en"]` (default). |
| `ocr.fps` | 0.5–2 (default 1): text-state sampling rate. OCR runs on deduplicated states, at most five frames. |
| `ocr.min_confidence_attach_crop` | 0–1 (default 0.85): lines under it are `needs_verification` with a native crop. |
| `speech.allow_billed_stt` | Reserve the STT 1.0 per-minute rate and transcribe **only** when the track is not silent and VAD finds speech. Settled to zero otherwise. |
| `speech.language_code` | STT hint; only with `allow_billed_stt`. |
| `delivery.inline_strip`, `delivery.inline_crops` | What the agent host attaches as images: strip on by default; crops `none` / `low_confidence_only` (default) / `all`. |
| `duration_seconds` | STT reservation hint (≤ 300); only with `allow_billed_stt`. |

There is no `vf`, `filtergraph`, `codec` or argv field: Sume compiles every
pass, and those keys answer `400 ffmpeg_fields_rejected`. `semantic: true` is
refused with `reference_ingest_semantic_unavailable` until the enrichment
pass ships.

## Errors that name the fix

| Code | Meaning |
|---|---|
| `source_too_long_for_reference_ingest` | The clip is over 300 s. Use `video_inspect` (and `video_segment` where listed). |
| `source_no_video_stream` | The file has no video stream. |
| `reference_ingest_unavailable` | The media runtime has no `reference-ingest` Function yet; use `video_inspect`. |
| `reference_ingest_stt_required` | `speech.language_code` / `duration_seconds` without `speech.allow_billed_stt`. |
| `unsupported_media_source`, `source_not_found` | Same admission as the other media tools. |

## Reading the manifest

- `shots[].start` / `end` and `text_tracks[].text` are the facts; the strip is
  orientation, not measurement.
- `audio.silent: true` means plan new music and discard the source audio —
  never request STT on it.
- `uncertain[]` is the only reason to look again: read the attached crop,
  then `video_frames_create` at a manifest time, at most once per entry.
- Warnings: `stt_skipped_silent`, `stt_skipped_no_speech`,
  `stt_skipped_no_audio_track`, `reference_ingest_output_missing:<file>`.

## Pricing

The manifest is unbilled (CPU work on the media runtime, like
`video_frames`). `speech.allow_billed_stt` reserves the
`sume/video-inspect-1.0#transcript` rate per ceil(minute) of the hint (one
minute when absent) and settles to what ran.