Video inspect

Current SoT. New clip inspection is this surface, not video analyses.

  • Dest and prod: POST /v1/video-inspect (MCP video_inspect). Probe + stills are unbilled. Default mode: sync.
  • Not typed scenes. Inspect returns probe facts, stills, and optional STT. Semantic questions / intervals on dest are video_analyze / video_segment only when those names appear in tools_list.
  • Legacy POST /v1/video-analyses: dest answers 410 video_analysis_retired; prod stays on until #5953 PR-C2.

Video inspect 1.0 reads one media.sume.com clip already owned by the workspace. It never re-encodes the source and never produces an MP4. The job id is the resource id (sume/video-inspect-1.0, type video_inspect).

Hosted MCP: video_inspect (packages/mcp-server/src/mcp.ts). Writes need idempotency_key (and mcp:write under OAuth). There is no GET MCP wrapper; poll with jobs_wait then jobs_result when the submit is 202.

Create an inspect

Required: video_url (this workspace’s media.sume.com artifact or asset). There is no open-internet fetch — import first (POST /v1/media-imports). Idempotency-Key is required.

Optional: frames, transcribe, and (only with transcribe: true) language_code, segmentation, duration_seconds, plus the usual mode / webhook_url / wait_timeout_seconds communication fields.

Default mode is sync. The handler waits up to 30 seconds and answers 200 with the finished inspect, or 202 with the queued job to poll.

A successful submit returns request_id = video_inspect_id (the job id) and a video_inspect object. Read it later:

When ready, the resource carries probe, frames[{t,url,width,height}] as durable media.sume.com image artifacts, transcript when requested (text, words[], optional sentence segments[], audio_url), and warnings[].

Frames program

framesEffect
omitted8 mid-bin stills (1 fps when the clip is shorter than 8 s)
falseProbe only — no stills
{ at: [seconds…] }Explicit timestamps, 1–24 values, each ≥ 0
{ fps: n }Sampling rate, 0 < n ≤ 2, mid-bin, capped at 24

An object must pass exactly one of at[] or fps. Optional on that object: format jpeg (default) or png; max_edge 64–2160 (default 768). Pass the source edge when you need a first-frame restage.

seek on that object picks how each still is found:

seekEffect
precise (default)Decodes to the exact instant. Existing programs are unchanged.
fastSnaps each still to the keyframe at or before its instant and skips the decode — earlier by up to one GOP (roughly 0–5 s on typical sources), never later. Quality, resolution and the transcript are unchanged.

Use fast when skimming a clip; keep precise when the timestamp must match (explicit at[] instants, the default 8-still midpoints). With fast each grid carries seek: "fast", sample_times are the instants the tiles actually show, and requested_times the ones the program asked for.

Caps (from packages/api-contract/src/index.ts): source ≤ 1800 s; 24 stills per call.

Exact source-size frames at one t are video frames, not this route.

Transcript (optional, billed)

transcribe: true runs Sume STT 1.0 on the clip’s audio. Probe and stills stay unbilled; only this half reserves.

  • Public rate: $0.0088 per audio minute (STT_PUBLIC_PRICING; confirm live in GET /v1/catalog).
  • Omit duration_seconds → reserve 1 minute. Max hint 600 s.
  • language_code (for example en or ko) is an STT hint; omit for auto-detect.
  • segmentation.mode: "sentence" also returns gapless sentence segments[] (caption-line shaped). Optional silence_split_seconds 0.2–3.
  • language_code / segmentation / duration_seconds without transcribe: true → 400 video_inspect_transcribe_required.
  • A silent clip → inspect_source_has_no_audio. Check probe.has_audio first (a frames: false inspect is enough).

Refusals (stable codes)

CodeWhen
ffmpeg_fields_rejectedClient sent vf / filter / ffmpeg / cmd / codec / crf / friends. The server compiles ffmpeg.
video_inspect_frames_program_conflictBoth frames.at[] and frames.fps.
video_inspect_frames_program_requiredframes is an object with neither at[] nor fps.
video_inspect_transcribe_requiredSTT-only fields without transcribe: true.
source_not_foundDead or foreign media.sume.com URL.
frame_time_out_of_rangeAn at value outside [0, duration). The error names the duration.
inspect_source_has_no_audiotranscribe: true on a clip with no audio track.

Off-host URLs (https://example.com/…) are rejected at admit. Import first.

Not this surface

NeedUse
Typed scenes[] / TwelveLabs PegasusLegacy video analyses (prod until PR-C2; dest 410)
Dest semantic Q&A / intervalsvideo_analyze / video_segment when listed on mcp.dev.sume.com
Exact frame at t, source sizeVideo frames
Burn captions onto a public URLVideo captions
A new MP4 cutVideo trim
Audio track as wav / mp3Audio detach
Pixel pass (dim / crop)Video filter
Sequence several clipsTimeline 1.0