Reference ingest

Dest first (#7944). POST /v1/reference-ingest and the MCP tool reference_ingest are listed where SUME_COM_REFERENCE_INGEST_ENABLED allows it (development auto-on, production opt-in). Everything else — video inspect, video_frames, video_trim, audio_detach — is unchanged; this surface is additive.

Reference ingest 1.0 is the deterministic preflight a reference clip goes through before an agent plans a remix, a brief or a face swap. It reads one media.sume.com clip the workspace already owns (≤ 300 s) on the Modal media runtime and answers a ReferenceVideoManifest:

shots[] — frame-exact cuts (ffmpeg scdet and PySceneDetect voting together), one sharpest source-resolution keyframe per shot, palette, luma, a motion class; the list tiles [0, duration] with no gap.
text_tracks[] — PP-OCRv5 (Korean + Latin) read at source resolution on deduplicated frame states, merged across frames into lines with text, normalised box, span, confidence, persistence, a card_id grouping and a role_hint. Nothing is corrected: a line under the confidence threshold is needs_verification and its native-resolution crop is returned.
audio — loudness gate (silent when integrated loudness ≤ −60 LUFS or the true peak is −inf), Silero VAD speech presence, librosa beats when the track is music, and — only with speech.allow_billed_stt — a Sume STT 1.0 transcript with words and sentence segments.
overview — one labeled strip (up to six tiles, gutter labels such as S0 0.00–4.28s (t=2.14)) that survives every model transport.
coverage, uncertain[], provenance.timings_ms — what ran, the only reasons to look again, and where the time went.

The job id is the resource id (sume/reference-ingest-1.0, type reference_ingest). Read-only: the source is never re-encoded and no MP4 is produced.

Hosted MCP: reference_ingest (text result). The Sume Agent host adds sume-agent__reference_ingest, which returns the same manifest and attaches the strip plus up to four low-confidence crops as inline images, so the model reads facts and pixels in one turn without a download.

Create an ingest

Required: video_url (this workspace’s media.sume.com artifact, asset or chat attachment). Idempotency-Key is required. Default mode is sync: the handler waits up to 30 seconds and answers 200 with the finished manifest, else 202 with the queued job (jobs_wait → jobs_result, result kind reference_video_manifest).

Optional fields:

Field	Meaning
`purpose`	`reference_remix` (default), `brief_format`, `face_swap`, `qa`. Stored, not interpreted.
`ocr.languages`	`["ko", "en"]` (default).
`ocr.fps`	0.5–2 (default 1): text-state sampling rate. OCR runs on deduplicated states, at most five frames.
`ocr.min_confidence_attach_crop`	0–1 (default 0.85): lines under it are `needs_verification` with a native crop.
`speech.allow_billed_stt`	Reserve the STT 1.0 per-minute rate and transcribe only when the track is not silent and VAD finds speech. Settled to zero otherwise.
`speech.language_code`	STT hint; only with `allow_billed_stt`.
`delivery.inline_strip`, `delivery.inline_crops`	What the agent host attaches as images: strip on by default; crops `none` / `low_confidence_only` (default) / `all`.
`duration_seconds`	STT reservation hint (≤ 300); only with `allow_billed_stt`.

There is no vf, filtergraph, codec or argv field: Sume compiles every pass, and those keys answer 400 ffmpeg_fields_rejected. semantic: true is refused with reference_ingest_semantic_unavailable until the enrichment pass ships.

Errors that name the fix

Code	Meaning
`source_too_long_for_reference_ingest`	The clip is over 300 s. Use `video_inspect` (and `video_segment` where listed).
`source_no_video_stream`	The file has no video stream.
`reference_ingest_unavailable`	The media runtime has no `reference-ingest` Function yet; use `video_inspect`.
`reference_ingest_stt_required`	`speech.language_code` / `duration_seconds` without `speech.allow_billed_stt`.
`unsupported_media_source`, `source_not_found`	Same admission as the other media tools.

Reading the manifest

shots[].start / end and text_tracks[].text are the facts; the strip is orientation, not measurement.
audio.silent: true means plan new music and discard the source audio — never request STT on it.
uncertain[] is the only reason to look again: read the attached crop, then video_frames_create at a manifest time, at most once per entry.
Warnings: stt_skipped_silent, stt_skipped_no_speech, stt_skipped_no_audio_track, reference_ingest_output_missing:<file>.

Pricing

The manifest is unbilled (CPU work on the media runtime, like video_frames). speech.allow_billed_stt reserves the sume/video-inspect-1.0#transcript rate per ceil(minute) of the hint (one minute when absent) and settles to what ran.