Video frames
Current SoT. Media L2 exact-frame extract (
video_frames, #5831). Dest and prod. This is not clip inspection — that is video inspect (probe + sampled stills + optional STT). It is not a new MP4 — range cuts are video trim.
Video frames takes one workspace media.sume.com clip plus a program
(at[] or fps) and returns durable media.sume.com image artifacts.
The source is untouched. The server compiles ffmpeg on the worker media
runtime (apps/api/src/routes.ts submitVideoFramesJob /
getVideoFrames; apps/api/src/schemas.ts createVideoFramesSchema;
apps/worker/src/video-frames-executor.ts).
There is no /v1/models/sume/…/runs alias on this family. The
resource id is the job id (request_id = video_frames_id).
Submit is always 202. submitVideoFramesJob pins
communicationMode: "async" — do not send mode: "sync" expecting a
200. Poll this GET (or GET /v1/jobs/:id/status).
Hosted MCP: video_frames_create / video_frames_get
(packages/mcp-server/src/mcp.ts). Writes need idempotency_key (and
mcp:write under OAuth). Flow: video_frames_create → jobs_wait →
video_frames_get.
Unbilled (same class as a screenshot hop — no seat, no reservation).
Create an extract
Required: video_url (this workspace’s media.sume.com artifact or
asset) and exactly one of at[] or fps. There is no open-internet
fetch — import first (POST /v1/media-imports). Idempotency-Key is
required on MCP writes; send it on REST so a retry does not queue a
second extract.
Optional: format jpeg (default) or png; max_edge 16–2160
(long-edge clamp; omit to keep the source frame size).
A successful submit returns 202 with request_id = video_frames_id
(the job id) and a video_frames object. Read it later:
When resource_status is ready, frames[{t,url,width,height}] are
durable artf_ images. One instant whose extract failed comes back with
url null; that does not fail the job. source_duration_seconds
is what the worker probed.
Program
| Field | Effect |
|---|---|
at[] | Explicit seconds. 1–24 values, each ≥ 0. Every value must satisfy 0 <= t < duration or the worker fails frame_time_out_of_range and names the probed duration. |
fps | Sampling rate instead of a list. 0 < fps ≤ 2. Expanded to mid-bin samples (0.5/fps, 1.5/fps, …) and capped at 24 frames. |
format | jpeg (default) or png (lossless inspection). |
max_edge | Optional long-edge clamp, 16–2160. Omitted keeps the source size (the restage path). |
Pass exactly one of at[] or fps. Caps: source ≤ 300 s
(VIDEO_ANALYSIS_HARD_MAX_DURATION_SECONDS); 24 frames per call.
Whole-clip evidence (probe, eight mid-bin stills, optional STT) is
video inspect. Inspect stills default
max_edge 768; this route omits that clamp.
Refusals (stable codes)
| Code | When |
|---|---|
ffmpeg_fields_rejected | Client sent vf / filter / filter_complex / select / ffmpeg / cmd / codec / crf / preset. The server compiles the extract. |
400 (schema) | Both at[] and fps, neither, more than 24 at values, fps > 2, or video_url not on media.sume.com. |
frame_time_out_of_range | An at value outside [0, duration) (worker, after probe). |
duration_out_of_range | Source longer than 300 s (worker). |
invalid_source_url | Stored job has no usable video_url (worker). |
Off-host URLs (https://example.com/…) are rejected at admit. Import first.
warnings[] may include low_confidence_long_video when the source is
longer than 90 s (probe reuse). That does not fail the job.
Not this surface
| Need | Use |
|---|---|
| Probe / sampled stills / optional STT | Video inspect |
| A new MP4 cut | Video trim |
| Audio track as a durable wav / mp3 | Audio detach |
| Pixel pass (dim / crop) | Video filter |
| Sequence several clips | Timeline 1.0 |
| Look at an unrendered HyperFrames composition | hyperframes_check then hyperframes_snapshot |