Timeline compose
Current SoT. Timeline 1.0 compose (
sume/timeline-1.0/compose, #3976). Dest and prod. This builds one shot. Assembly — sequencing, transitions, and the audio spine — stays on Timeline 1.0. Sequential image-then-video is not a compose mode: adjacentvideo[]slots on the render already do that.
Compose takes one still + one video and returns one MP4 with
both on screen at once. The server compiles ffmpeg on the worker media
runtime (apps/api/src/routes.ts createTimelineV1Compose /
submitSumeTimelineComposeJob;
packages/timeline-compiler/src/compose.ts). Callers never send
filtergraphs, codecs, or shell fragments.
There is no GET /v1/timeline-1.0/compose/:id. Poll the job envelope:
Hosted MCP: timeline_compose (packages/mcp-server/src/mcp.ts). Writes
need idempotency_key (and mcp:write under OAuth). Flow:
timeline_compose → jobs_wait → jobs_result → drop video_url into
timeline_create video[].
The HTTP field for stack vs overlay is operation, not mode.
mode is the usual async / sync / webhook communication option.
Create a compose
Required: operation (stack | overlay), image.url, video.url.
Both URLs must already be this workspace’s media.sume.com artifact or
asset. image.url must probe as a still (compose_image_not_still);
video.url must probe as video (compose_video_not_video). Import first
(POST /v1/media-imports). Idempotency-Key is required.
Optional: layout, output, video.source_in, video.duration, plus
the usual mode / webhook_url / wait_timeout_seconds communication
fields.
Default mode is async. Pass mode: "sync" to wait up to
30 seconds for a 200 finished job, or get 202 and poll.
Output length always comes from the video layer (video.duration,
else the rest of the file from source_in). The still is held for the
whole clip and can never lengthen it. Ceiling 300 s
(TIMELINE_COMPOSE_MAX_DURATION_SECONDS). Past the source it clamps
(compose_duration_clamped_to_source).
A successful submit returns a job (model: sume/timeline-1.0/compose).
When result_ready, GET /v1/jobs/:id/result is
kind: timeline_compose with video_url (new artf_) and
duration_seconds. Drop that MP4 into
Timeline 1.0 video[].
Public rate: $0.02 flat per job (TIMELINE_COMPOSE_PUBLIC_PRICING;
confirm live in GET /v1/catalog). Flat because video.duration may be
omitted until the worker probes. No provider inference — worker ffmpeg
only.
Default output is 1080×1920 MP4 at the video layer's own frame rate —
the only rate that repeats or drops no frame. Set output.width /
output.height to the timeline you assemble into so the shot is not
rescaled twice; an explicit output.fps that differs from the clip's rate
warns output_fps_resamples_sources.
Audio passes through from the video. A mute video is a warning
(compose_video_has_no_audio), not a failure — the clip still renders,
and the Timeline 1.0 spine supplies the audio at assemble time.
Layout
stack tiles two regions of one frame. Defaults
horizontal / top / 0.5 are 반배너 (half-banner: still on top,
video underneath). ratio is the still’s share (0.1–0.9); the video
takes the exact remainder.
layout key | stack | overlay |
|---|---|---|
split | horizontal | vertical | illegal |
image_region | top | bottom on a horizontal split; left | right on a vertical split | illegal |
ratio | still’s share of the frame | illegal |
image_fit / video_fit | cover | contain | stretch (blur is not a compose fit) | video_fit only |
position | illegal | top | center | bottom |
width_ratio | illegal | 0.05–1 of width (default 0.9); plate keeps aspect |
margin_ratio | illegal | 0–0.45 of height (default 0.05) |
Mixing stack keys with overlay keys is 400
(compose_stack_takes_no_overlay_layout /
compose_overlay_takes_no_stack_layout).
compose_image_region_wrong_axis when the region does not match the
split.
Refusals (stable codes)
| Code | When |
|---|---|
compose_image_not_still | image.url is not a still. |
compose_video_not_video | video.url is not a video. |
compose_image_region_wrong_axis | left/right on a horizontal split, or top/bottom on a vertical split. |
compose_stack_takes_no_overlay_layout / compose_overlay_takes_no_stack_layout | Mixed layout vocabulary. |
compose_duration_clamped_to_source | Warning: video.duration ran past the file. Job still succeeds. |
compose_video_has_no_audio | Warning: mute video source. Job still succeeds. |
unsupported_media_source / source_not_found | Off-host or dead URL. |
| Provider / ffmpeg keys | 400 — filtergraph, ffmpeg_args, codec, crf, friends. |
Off-host URLs (https://example.com/…) are rejected at admit. Import first.
Not this surface
| Need | Use |
|---|---|
| Sequence several clips | Timeline 1.0 |
| Concat / split audio into reusable files | Timeline audio |
[start, end) of one clip | Video trim |
| Pixel pass (dim / crop) | Video filter |
| Audio track as a durable wav / mp3 | Audio detach |