---
title: Reference ingest
description: Read one reference clip into a timestamped manifest (shots, source-resolution OCR, audio facts, a labeled strip) in one call. Dest first.
---

> **Dest first (#7944).** Sume lists `POST /v1/reference-ingest` and the MCP
> tool `reference_ingest` only where `SUME_COM_REFERENCE_INGEST_ENABLED`
> permits them (development auto-on, production opt-in). This surface does not
> change the other tools: [video inspect](/models/video-inspect),
> `video_frames`, `video_trim`, and `audio_detach`. This surface is an
> addition, not a replacement.

Reference ingest 1.0 is the **deterministic preflight** for a reference clip
before an agent plans a remix, a brief, or a face swap. It reads **one**
`media.sume.com` clip that the workspace already owns (≤ 300 s) on the Modal
media runtime. It answers with a `ReferenceVideoManifest`:

- `shots[]` — frame-exact cuts (ffmpeg `scdet` and PySceneDetect vote
  together), one sharpest source-resolution keyframe per shot, palette,
  luma, and a motion class. The list tiles `[0, duration]` with no gap.
- `text_tracks[]` — PP-OCRv5 (Korean + Latin) reads at **source resolution**
  on deduplicated frame states. The tool merges the results across frames into
  lines. Each line has text, a normalized box, span, confidence, persistence, a
  `card_id` group, and a `role_hint`. The tool corrects nothing. A line under
  the confidence threshold is `needs_verification`, and the tool returns its
  native-resolution crop.
- `audio` — loudness gate (`silent` when integrated loudness ≤ −60 LUFS or
  the true peak is −inf), Silero VAD speech presence, and librosa beats when
  the track is music. When the read transcribes (by default for
  `reference_remix`, refer to `speech.allow_billed_stt`), it also has a Sume
  STT 1.0 transcript with words and sentence segments.
- `boundaries[]` — visual change candidates `{at, score}`. The tool takes 12
  samples a second, scales them to 32×32 RGB, and calculates the mean absolute
  difference of neighbors. It keeps the candidates at ≥ 0.1. These candidates
  are mechanical evidence, not editorial
  shots. They sit beside `shots[]`.
- `overview` — one labeled strip (up to six tiles, gutter labels such as
  `S0 0.00–4.28s (t=2.14)`) that every model transport can carry.
- `coverage`, `uncertain[]`, `provenance.timings_ms` — what ran, the only
  reasons to look again, and where the time went.

The job id **is** the resource id (`sume/reference-ingest-1.0`, type
`reference_ingest`). The ingest is read-only. It never re-encodes the source
and does not produce an MP4.

```text
POST /v1/reference-ingest
GET  /v1/reference-ingest/:id
```

The hosted MCP tool is `reference_ingest` (text result). The Sume Agent host
adds `sume-agent__reference_ingest`. This tool returns the same manifest **and
attaches the strip plus up to four low-confidence crops as inline images**.
Thus, the model reads facts and pixels in one turn without a download.

## Create an ingest

The required field is `video_url` (this workspace’s `media.sume.com`
artifact, asset, or chat attachment). `Idempotency-Key` is also required. The
default `mode` is **`sync`**. The handler waits up to 30 seconds and answers
`200` with the finished manifest. If the manifest is not ready, the handler
answers `202` with the queued job (`jobs_wait` → `jobs_result`, result kind
`reference_video_manifest`).

The optional fields are:

| Field | Meaning |
|---|---|
| `purpose` | `reference_remix` (default), `brief_format`, `face_swap`, `qa`. Sume stores this value but does not interpret it. |
| `ocr.languages` | `["ko", "en"]` (default). |
| `ocr.fps` | 0.5–2 (default 1): the sample rate for text states. OCR runs on deduplicated states, at most five frames. |
| `ocr.min_confidence_attach_crop` | 0–1 (default 0.85): lines under this value are `needs_verification` with a native crop. |
| `speech.allow_billed_stt` | Reserves STT 1.0 credits per minute and transcribes **only** when the track is not silent and VAD finds speech. In other conditions, the charge settles to zero. If you omit it, the value is `true` for `purpose: reference_remix` and `false` for the other purposes. `false` opts out. |
| `speech.language_code` | STT hint. Use it only when the read transcribes. |
| `delivery.inline_strip`, `delivery.inline_crops` | What the agent host attaches as images. The strip is on by default. Crops are `none` / `low_confidence_only` (default) / `all`. |
| `duration_seconds` | STT reservation hint (≤ 300). Use it only when the read transcribes. |

The request has no `vf`, `filtergraph`, `codec`, or argv field. Sume compiles
every pass, and those keys answer `400 ffmpeg_fields_rejected`. Until the
enrichment pass ships, Sume refuses `semantic: true` with
`reference_ingest_semantic_unavailable`.

## Errors that name the fix

| Code | Meaning |
|---|---|
| `source_too_long_for_reference_ingest` | The clip is longer than 300 s. Use `video_inspect` (and `video_segment` where Sume lists it). |
| `source_no_video_stream` | The file has no video stream. |
| `reference_ingest_unavailable` | The media runtime has no `reference-ingest` Function yet. Use `video_inspect`. |
| `reference_ingest_stt_required` | The request sends `speech.language_code` / `duration_seconds` on a read that does not transcribe. |
| `unsupported_media_source`, `source_not_found` | The admission rules are the same as for the other media tools. |

## Reading the manifest

- `shots[].start` / `end` and `boundaries[]` locate cuts.
  `audio.speech.words[]` carries every spoken word with its time.
  `text_tracks[]` are OCR hits on at most five sampled frames, not caption
  coverage. On the agent host, the agent reads captions, MG, and UI from
  `reference_grid` pages (each cell has a label with its timecode and the words
  spoken at that time). The strip is for orientation, not for measurement.
- `audio.silent: true` means that you must plan new music and discard the
  source audio. Never request STT on that audio.
- `uncertain[]` is the only reason to inspect a frame again. Read the attached
  crop. Then call `video_frames_create` at a manifest time, at most one time
  for each entry.
- Warnings: `stt_skipped_silent`, `stt_skipped_no_speech`,
  `stt_skipped_no_audio_track`, `reference_ingest_output_missing:<file>`.

## Credits

Sume charges the manifest in credits by its compute time, as for
`video_frames`. Every read reserves the compute ceiling of the ingest, and the
charge is never more than the hold it reserved. A read that transcribes
(`speech.allow_billed_stt`, on by default for `reference_remix`) adds the
`sume/video-inspect-1.0#transcript` credits to the reservation. These credits
scale with the minutes of the hint, rounded up (one minute when the hint is
absent). The read then settles to what ran. The
[API rate card](https://www.sume.com/pricing/api) lists the current credits.