Models

Video inspect

Probe, sample stills from, and optionally transcribe one Sume-hosted clip. Default clip inspection on dest and prod.

Current SoT. For new clip inspection, use this surface and not video analyses.

  • Dest and prod: POST /v1/video-inspect (MCP video_inspect). Sume charges probe + stills in credits by their compute time. The default is mode: sync.
  • Not typed scenes. Inspect returns probe facts, stills, and optional STT. On dest, use video_analyze / video_segment for semantic questions / intervals, but only when those names are in tools_list.
  • Legacy POST /v1/video-analyses: dest returns 410 video_analysis_retired. On prod, it stays available until #5953 PR-C2.

Video inspect 1.0 reads one media.sume.com clip that the workspace already owns. It never re-encodes the source, and it never makes an MP4. The job ID is the resource ID (sume/video-inspect-1.0, type video_inspect).

Hosted MCP: video_inspect (packages/mcp-server/src/mcp.ts). A write must have idempotency_key (and mcp:write under OAuth). There is no GET MCP wrapper. When the submit returns 202, poll with jobs_wait and then jobs_result.

Create an inspect

You must send video_url (a media.sume.com artifact or asset of this workspace). The API does not fetch from the open internet. First, import the clip (POST /v1/media-imports). You must send Idempotency-Key.

Optional fields: frames, transcribe, and (only with transcribe: true) language_code, segmentation, duration_seconds. You can also send the usual mode / webhook_url / wait_timeout_seconds communication fields.

The default mode is sync. The handler waits a maximum of 30 seconds. If the inspect completes in that time, the handler returns 200 with the result. If not, it returns 202 with the queued job, and you poll that job.

If the submit is successful, the API returns request_id = video_inspect_id (the job ID) and a video_inspect object. Read the object later:

When the inspect is complete, the resource contains probe and frames[{t,url,width,height}] as durable media.sume.com image artifacts. If the request included a transcript, the resource also contains transcript (text, words[], optional sentence segments[], audio_url). It also contains warnings[].

Frames program

framesEffect
not sent8 mid-bin stills (1 fps if the clip is shorter than 8 s)
falseOnly the probe, with no stills
{ at: [seconds…] }Explicit timestamps, 1–24 values, each ≥ 0
{ fps: n }Sample rate, 0 < n ≤ 2, mid-bin, with a limit of 24

An object must contain only one of at[] or fps. Optional fields on that object: format jpeg (default) or png, and max_edge 64–2160 (default 768). For a first-frame restage, send the source edge.

seek on that object selects how Sume finds each still:

seekEffect
precise (default)Decodes to the accurate instant. Programs that you already use do not change.
fastMoves each still to the keyframe at or before its instant, and does not do the decode. The still can be earlier by a maximum of one GOP (approximately 0–5 s on typical sources), but it is never later. The quality, the resolution, and the transcript do not change.

Use fast for a quick look at a clip. Use precise when the timestamp must be accurate (explicit at[] instants, the default 8-still midpoints). With fast, each grid has seek: "fast". sample_times are the instants that the tiles show. requested_times are the instants in the program.

Limits (from packages/api-contract/src/index.ts): source ≤ 1800 s, and 24 stills for each call.

For accurate frames at the source size at one t, use video frames, not this route.

Transcript (optional, billed)

transcribe: true runs Sume STT 1.0 on the audio of the clip. Each inspect reserves its compute ceiling. Sume charges the job in credits for its own compute time, never more than the hold it reserved. The transcript adds its per-minute credits to that reservation.

  • Credits scale with the audio minutes. The API rate card lists the current credits.
  • Without duration_seconds, Sume reserves 1 minute. Maximum hint: 600 s.
  • language_code (for example en or ko) is an STT hint. If you do not send it, STT uses auto-detect.
  • segmentation.mode: "sentence" also returns sentence segments[] with no gaps (in the shape of caption lines). You can also set silence_split_seconds in the range 0.2–3.
  • If you send language_code / segmentation / duration_seconds without transcribe: true, the API returns 400 video_inspect_transcribe_required.
  • A silent clip gives inspect_source_has_no_audio. First, examine probe.has_audio (a frames: false inspect is sufficient).

Refusals (stable codes)

CodeWhen
ffmpeg_fields_rejectedThe client sent vf / filter / ffmpeg / cmd / codec / crf / related fields. The server compiles ffmpeg.
video_inspect_frames_program_conflictThe request has frames.at[] and frames.fps together.
video_inspect_frames_program_requiredframes is an object that has no at[] and no fps.
video_inspect_transcribe_requiredThe request has STT-only fields without transcribe: true.
source_not_foundThe media.sume.com URL is dead or is not from this workspace.
frame_time_out_of_rangeAn at value is outside [0, duration). The error gives the duration.
inspect_source_has_no_audioThe request has transcribe: true for a clip with no audio track.

The API rejects off-host URLs (https://example.com/…) at admit. Import the clip first.

Not this surface

NeedUse
Typed scenes[] / TwelveLabs PegasusLegacy video analyses (prod until PR-C2, dest 410)
Dest semantic Q&A / intervalsvideo_analyze / video_segment, when they are in the list on mcp.dev.sume.com
The accurate frame at t, at the source sizeVideo frames
Burn captions onto a public URLVideo captions
A new MP4 cutVideo trim
Audio track as wav / mp3Audio detach
Pixel pass (dim / crop)Video filter
Put several clips in a sequenceTimeline 1.0