Reference ingest

한국어 번역이 준비되기 전까지 이 페이지는 영어 원문으로 표시됩니다.

Dest first (#7944). Sume lists POST /v1/reference-ingest and the MCP tool reference_ingest only where SUME_COM_REFERENCE_INGEST_ENABLED permits them (development auto-on, production opt-in). This surface does not change the other tools: video inspect, video_frames, video_trim, and audio_detach. This surface is an addition, not a replacement.

Reference ingest 1.0 is the deterministic preflight for a reference clip before an agent plans a remix, a brief, or a face swap. It reads one media.sume.com clip that the workspace already owns (≤ 300 s) on the Modal media runtime. It answers with a ReferenceVideoManifest:

shots[] — frame-exact cuts (ffmpeg scdet and PySceneDetect vote together), one sharpest source-resolution keyframe per shot, palette, luma, and a motion class. The list tiles [0, duration] with no gap.
text_tracks[] — PP-OCRv5 (Korean + Latin) reads at source resolution on deduplicated frame states. The tool merges the results across frames into lines. Each line has text, a normalized box, span, confidence, persistence, a card_id group, and a role_hint. The tool corrects nothing. A line under the confidence threshold is needs_verification, and the tool returns its native-resolution crop.
audio — loudness gate (silent when integrated loudness ≤ −60 LUFS or the true peak is −inf), Silero VAD speech presence, and librosa beats when the track is music. When the read transcribes (by default for reference_remix, refer to speech.allow_billed_stt), it also has a Sume STT 1.0 transcript with words and sentence segments.
boundaries[] — visual change candidates {at, score}. The tool takes 12 samples a second, scales them to 32×32 RGB, and calculates the mean absolute difference of neighbors. It keeps the candidates at ≥ 0.1. These candidates are mechanical evidence, not editorial shots. They sit beside shots[].
overview — one labeled strip (up to six tiles, gutter labels such as S0 0.00–4.28s (t=2.14)) that every model transport can carry.
coverage, uncertain[], provenance.timings_ms — what ran, the only reasons to look again, and where the time went.

The job id is the resource id (sume/reference-ingest-1.0, type reference_ingest). The ingest is read-only. It never re-encodes the source and does not produce an MP4.

The hosted MCP tool is reference_ingest (text result). The Sume Agent host adds sume-agent__reference_ingest. This tool returns the same manifest and attaches the strip plus up to four low-confidence crops as inline images. Thus, the model reads facts and pixels in one turn without a download.

Create an ingest

The required field is video_url (this workspace’s media.sume.com artifact, asset, or chat attachment). Idempotency-Key is also required. The default mode is sync. The handler waits up to 30 seconds and answers 200 with the finished manifest. If the manifest is not ready, the handler answers 202 with the queued job (jobs_wait → jobs_result, result kind reference_video_manifest).

The optional fields are:

Field	Meaning
`purpose`	`reference_remix` (default), `brief_format`, `face_swap`, `qa`. Sume stores this value but does not interpret it.
`ocr.languages`	`["ko", "en"]` (default).
`ocr.fps`	0.5–2 (default 1): the sample rate for text states. OCR runs on deduplicated states, at most five frames.
`ocr.min_confidence_attach_crop`	0–1 (default 0.85): lines under this value are `needs_verification` with a native crop.
`speech.allow_billed_stt`	Reserves STT 1.0 credits per minute and transcribes only when the track is not silent and VAD finds speech. In other conditions, the charge settles to zero. If you omit it, the value is `true` for `purpose: reference_remix` and `false` for the other purposes. `false` opts out.
`speech.language_code`	STT hint. Use it only when the read transcribes.
`delivery.inline_strip`, `delivery.inline_crops`	What the agent host attaches as images. The strip is on by default. Crops are `none` / `low_confidence_only` (default) / `all`.
`duration_seconds`	STT reservation hint (≤ 300). Use it only when the read transcribes.

The request has no vf, filtergraph, codec, or argv field. Sume compiles every pass, and those keys answer 400 ffmpeg_fields_rejected. Until the enrichment pass ships, Sume refuses semantic: true with reference_ingest_semantic_unavailable.

Errors that name the fix

Code	Meaning
`source_too_long_for_reference_ingest`	The clip is longer than 300 s. Use `video_inspect` (and `video_segment` where Sume lists it).
`source_no_video_stream`	The file has no video stream.
`reference_ingest_unavailable`	The media runtime has no `reference-ingest` Function yet. Use `video_inspect`.
`reference_ingest_stt_required`	The request sends `speech.language_code` / `duration_seconds` on a read that does not transcribe.
`unsupported_media_source`, `source_not_found`	The admission rules are the same as for the other media tools.

Reading the manifest

shots[].start / end and boundaries[] locate cuts. audio.speech.words[] carries every spoken word with its time. text_tracks[] are OCR hits on at most five sampled frames, not caption coverage. On the agent host, the agent reads captions, MG, and UI from reference_grid pages (each cell has a label with its timecode and the words spoken at that time). The strip is for orientation, not for measurement.
audio.silent: true means that you must plan new music and discard the source audio. Never request STT on that audio.
uncertain[] is the only reason to inspect a frame again. Read the attached crop. Then call video_frames_create at a manifest time, at most one time for each entry.
Warnings: stt_skipped_silent, stt_skipped_no_speech, stt_skipped_no_audio_track, reference_ingest_output_missing:<file>.

Credits

Sume charges the manifest in credits by its compute time, as for video_frames. Every read reserves the compute ceiling of the ingest, and the charge is never more than the hold it reserved. A read that transcribes (speech.allow_billed_stt, on by default for reference_remix) adds the sume/video-inspect-1.0#transcript credits to the reservation. These credits scale with the minutes of the hint, rounded up (one minute when the hint is absent). The read then settles to what ran. The API rate card lists the current credits.