Video frames
Current SoT. Media L2 exact-frame extract (
video_frames, #5831). It is available on dest and prod. This API does not inspect a clip. For clip inspection, use video inspect (probe + sampled stills + optional STT). This API does not make a new MP4. For range cuts, use video trim.
Video frames uses one workspace media.sume.com clip and a program
(at[] or fps) as input. It returns durable media.sume.com image
artifacts. The source does not change. The server compiles ffmpeg on the
worker media runtime (apps/api/src/routes.ts submitVideoFramesJob /
getVideoFrames, apps/api/src/schemas.ts createVideoFramesSchema, and
apps/worker/src/video-frames-executor.ts).
This family has no /v1/models/sume/…/runs alias. The resource ID
is the job ID (request_id = video_frames_id).
A submit always returns 202. submitVideoFramesJob pins
communicationMode: "async". Do not send mode: "sync". It does not give a
200. Poll this GET (or GET /v1/jobs/:id/status).
Hosted MCP: video_frames_create / video_frames_get
(packages/mcp-server/src/mcp.ts). A write must have idempotency_key
(and mcp:write under OAuth). Flow: video_frames_create → jobs_wait →
video_frames_get.
This job does not use an admission seat (same class as a screenshot hop). At submit, it reserves its compute ceiling. Sume charges the job in credits for its own compute time, never more than the hold it reserved.
Create an extract
You must send video_url (a media.sume.com artifact or asset of this
workspace) and only one of at[] or fps. The API does not fetch from
the open internet. First, import the clip (POST /v1/media-imports). MCP
writes must have Idempotency-Key. Also send it on REST, so that a retry
does not queue a second extract.
Optional fields: format jpeg (default) or png, and max_edge
16–2160 (a long-edge clamp). If you do not send the clamp, the frames
keep the source frame size.
If the submit is successful, the API returns 202 with request_id =
video_frames_id (the job ID) and a video_frames object. Read the object
later:
When resource_status is ready, frames[{t,url,width,height}] are
durable artf_ images. If the extract fails for one instant, that frame
has url null. This does not cause the job to fail.
source_duration_seconds is the duration that the worker probed.
Program
| Field | Effect |
|---|---|
at[] | Explicit seconds. 1–24 values, each ≥ 0. Each value must be in the range 0 <= t < duration. If not, the worker fails with frame_time_out_of_range and gives the probed duration. |
fps | Sample rate, as an alternative to a list. 0 < fps ≤ 2. Sume expands it to mid-bin samples (0.5/fps, 1.5/fps, …), with a limit of 24 frames. |
format | jpeg (default) or png (lossless inspection). |
max_edge | Optional long-edge clamp, 16–2160. If you do not send it, the frames keep the source size (the restage path). |
Send only one of at[] or fps. Limits: source ≤ 300 s
(VIDEO_ANALYSIS_HARD_MAX_DURATION_SECONDS), and 24 frames for each
call.
For evidence on the full clip (probe, eight mid-bin stills, optional
STT), use video inspect. Inspect stills use a
default max_edge of 768. This route does not use that clamp.
Refusals (stable codes)
| Code | When |
|---|---|
ffmpeg_fields_rejected | The client sent vf / filter / filter_complex / select / ffmpeg / cmd / codec / crf / preset. The server compiles the extract. |
400 (schema) | The request has at[] and fps together, or has none of them. Or it has more than 24 at values, fps > 2, or a video_url that is not on media.sume.com. |
frame_time_out_of_range | An at value is outside [0, duration) (worker, after the probe). |
duration_out_of_range | The source is longer than 300 s (worker). |
invalid_source_url | The stored job has no video_url that the worker can use (worker). |
The API rejects off-host URLs (https://example.com/…) at admit. Import the
clip first.
When the source is longer than 90 s (probe reuse), warnings[] can include
low_confidence_long_video. This warning does not cause the job to fail.
Not this surface
| Need | Use |
|---|---|
| Probe / sampled stills / optional STT | Video inspect |
| A new MP4 cut | Video trim |
| Audio track as a durable wav / mp3 | Audio detach |
| Pixel pass (dim / crop) | Video filter |
| Put several clips in a sequence | Timeline 1.0 |
| Look at an unrendered HyperFrames composition | hyperframes_check then hyperframes_snapshot |