Video filter
Current SoT. Media L2 pixel pass (
sume/video-filter-1.0, #5820 / #5839). Dest and prod. This is not a range cut — that is video trim. It is not assembly — sequencing, transitions, and the audio spine stay on Timeline 1.0.
Video filter 1.0 takes one workspace media.sume.com clip plus a
validated program (ops[] dim/crop and/or a filters-only filtergraph)
and returns a new MP4. The source is untouched. The server compiles
ffmpeg on the worker media runtime (apps/api/src/routes.ts
createVideoFilterV1 / submitSumeVideoFilterJob;
packages/timeline-compiler/src/filter.ts compileVideoFilterProgram).
There is no GET /v1/video-filter/:id. Poll the job envelope:
Hosted MCP: video_filter (packages/mcp-server/src/mcp.ts). Writes
need idempotency_key (and mcp:write under OAuth). Flow:
video_filter with check_only: true (free) → video_filter →
jobs_wait → jobs_result.
Check a program (unbilled)
POST /v1/video-filter/check runs the same schema, op whitelist,
filtergraph allowlist, and Sume-host / HEAD source preflight as the
encode, and returns diagnostics instead of a 400. It does not
create a job, reserve credits, boot a box, or touch the encoder. A
program that passes here can still fail on the box (bad expression,
memory, time) — that comes back as a structured job error.
Idempotency-Key is not required on the check. A valid response is
object: video_filter_check with valid, encode: "not_run",
diagnostics[], compiled program.filters (names only, no argv), an
estimate when valid, and next_action
submit_video_filter | fix_program_and_recheck.
Encode
Required: video_url (this workspace’s media.sume.com artifact or
asset) and a program — at least one of ops[] or a non-empty
filtergraph. There is no open-internet fetch — import first
(POST /v1/media-imports). Idempotency-Key is required.
Optional: ops, filtergraph, metadata, plus the usual mode /
webhook_url / wait_timeout_seconds communication fields.
Default mode is async (readCommunicationOptions). Pass
mode: "sync" to wait up to 30 seconds for a 200 finished job, or
get 202 and poll.
A successful submit returns a job (request_id is the job id). When
result_ready, GET /v1/jobs/:id/result is kind: video_filter with
video_url (new artf_, never the source), duration_seconds,
ops_applied, filtergraph, compiled filters[], and optional
warnings[]. Drop that MP4 into timeline_create video[].
Public rate: $0.02 per encode job (VIDEO_FILTER_PUBLIC_PRICING;
confirm live in GET /v1/catalog). The check is free. No provider
inference — worker ffmpeg only.
Program
| Field | Effect |
|---|---|
ops[] | Ordered program, applied before filtergraph. Max 8. At least one op or a filtergraph is required. |
ops[].op: "dim" | Whole-clip luma multiply. amount in (0, 1] — 0.45 is darker, 1 is unchanged. 0 and >1 are refused. Black stays black; chroma does not shift. |
ops[].op: "crop" | Rectangle as fractions of the source frame: x, y in [0, 1]; width, height in [0.05, 1]; x+width ≤ 1 and y+height ≤ 1. The compiler even-rounds for yuv420p. |
filtergraph | Filters-only ffmpeg graph, applied after ops[]. Max 2048 chars, 32 named filters. No inputs, no outputs, no paths — the server wraps [0:v]…[vout]. Internal labels (split[a][b]) are fine; stream specifiers ([0:v]) are not. |
Caps (from packages/api-contract/src/index.ts): source ≤ 300 s
(compose-band clip ceiling). Output inherits source geometry, frame
rate, and audio unless the program changes them.
Allowlisted filter names live in
packages/timeline-compiler/src/filter.ts
VIDEO_FILTER_GRAPH_ALLOWED_FILTERS (tone, blur, geometry, fade,
internal compositing). Not on that list: trim / setpts (use
video trim), drawtext / subtitles / movie /
lut3d, and anything that reads a file or a socket.
Refusals (stable codes)
| Code | When |
|---|---|
video_filter_ops_empty | Neither ops[] nor a non-empty filtergraph. |
video_filter_too_many_ops | More than 8 ops. |
unsupported_filter_op | ops[].op is not dim or crop. Other pixel work goes in filtergraph. |
unsupported_filter_op_field | Extra key on an op (dim takes {op, amount} only; crop takes {op, x, y, width, height}). |
video_filter_amount_out_of_range | dim amount not in (0, 1]. |
video_filter_crop_out_of_bounds | Crop fractions miss the source frame or a side is under 0.05. |
invalid_filtergraph | Empty, too long, unknown filter, illegal alphabet, or stream specifier. unknown_filter names the token and the allowlist. |
ffmpeg_fields_rejected | Client sent vf / filter / ffmpeg / cmd / codec / crf / -i / friends. The server compiles ffmpeg. |
unsupported_media_source | video_url is not on the Sume media host. |
source_not_found | Dead or foreign media.sume.com URL. |
unsupported_media_type | HEAD is not a video. |
source_too_large | Source exceeds the timeline download budget. |
output_duration_exceeded | Source longer than 300 s (worker). |
Off-host URLs (https://example.com/…) are rejected at admit. Import first.
Not this surface
| Need | Use |
|---|---|
| Probe / stills / optional STT | Video inspect |
| A new MP4 cut | Video trim |
| Audio track as a durable wav / mp3 | Audio detach |
| Sequence several clips | Timeline 1.0 |
| Caption plates | HyperFrames compose / caption assembler |
Exact frame at t, source size | Video frames |