Video analyses

Current SoT. Do not start new work on this surface.

  • Default on dest and prod: Video inspect — POST /v1/video-inspect (MCP video_inspect) for probe + stills + optional transcript. Probe and stills are free. Default mode: sync.
  • Dest (api.dev.sume.com): SUME_COM_VIDEO_ANALYSIS_ENABLED=false. POST /v1/video-analyses answers 410 video_analysis_retired. MCP video-analyses_* leave tools_list. Stored vana_ rows stay readable via GET. Semantic questions / intervals on dest: video_analyze / video_segment only when those names appear in tools_list (trusted api.dev.sume.com origin + TwelveLabs key — never production).
  • Prod (api.sume.com): create stays on until #5953 PR-C2. Dest 410 is not a production outage.

This page documents the legacy video_analysis / vana_ resource (typed scenes[]). It is not trend discovery, not virality prediction, and it does not generate video. A reusable Format from clip structure is a remix job, not this analysis.

When create is still enabled, a video-understanding model (TwelveLabs Pegasus) watches the MP4 and segments it into typed scenes; ffmpeg then extracts at least one JPEG still per second of each scene (keyframes) plus a representative keyframe_url. Current analysis_version is 1.1.

Remote MCP wrappers video-analyses_create / _get / _list follow the same flag: listed on production until PR-C2; delisted on dest. Poll remaining jobs with jobs_wait (typical runtime 3–5 minutes; poll every 30–60s).

Accuracy caveat: longer videos are less accurate; short clips give the most reliable results.

Create an analysis job (production until PR-C2)

Dest callers: do not POST this. Use video inspect, or dest-only video_analyze / video_segment when listed.

Required: video_url. Optional: max_scenes (2–40, default 24), include_transcript (when true, each scene's audio carries the spoken speech for that scene; otherwise audio is null), plus the usual mode / webhook_url / wait_timeout_seconds communication fields.

A successful submit returns 202 with a vana_… resource id, a video_analysis job (request_id), and status_url / result_url.

Prefer a durable media.sume.com URL from media imports (POST /v1/media-imports) or a completed Sume generation artifact. There is no Higgsfield-style media_id intake gate.

Pricing note

Each accepted analysis reserves and captures $0.30 USD of Sume usage under the current fixed estimate. Confirm live pricing in GET /v1/catalog and OpenAPI.

Duration limits

CapBehavior
Soft (~90s)Job succeeds; response may include warning low_confidence_long_video.
Hard (300s)Rejected after probe with a stable duration error.

Unsupported inputs

InputResult
YouTube URLs422 unsupported_source
Non-HTTPS / private / localhostRejected at admit
Non-video assetsFail with a stable, agent-legible code

Poll and read

When ready, the resource includes whole-video metadata plus a typed scenes[] array. Each scene has contiguous start_seconds / end_seconds, duration_seconds, summary, visual, optional shot_type / camera_motion, on_screen_text, audio ({ speech, has_speech, has_music } when include_transcript was requested, otherwise null), keyframe_url (the representative JPEG still — the 1s sample nearest 40% of the scene, or the historical 40% pick on sub-1s scenes — or null with a keyframe_mirror_failed:scene_N warning), keyframes (an array of { t, url } stills covering every whole second of that scene; a failed second is url: null plus keyframe_mirror_failed:scene_N:t_T and does not drop the scene), and confidence (0–1). The 300s hard duration cap is also the stills ceiling (~300 JPEGs). There is no separate 1-second text/context track.

Scopes

ScopeOperations
video_analyses:writePOST /v1/video-analyses
video_analyses:readGET /v1/video-analyses, GET /v1/video-analyses/:id