Timeline 1.0
Assemble one audio spine plus ordered video slots into a long-form MP4. The only public assembly surface.
Current SoT. Timeline 1.0 render (
sume/timeline-1.0). It is available on dest and prod. This job is assembly: the sequence, transitions, and the audio spine. Material preparation stays on video trim, audio detach, video filter, and timeline compose.
Timeline 1.0 takes a declarative document (one audio spine + ordered
video[] slots) and returns one MP4. The server compiles ffmpeg on
the worker media runtime (apps/api/src/routes.ts renderTimelineV1 /
submitSumeTimelineRenderJob). Callers never send filtergraphs, codecs,
or shell fragments.
This surface has no GET /v1/timeline-1.0/:id. Poll the job envelope:
The hosted MCP flow is timeline_create, then jobs_wait, then
timeline_get (packages/mcp-server/src/mcp.ts). timeline_get is
GET /v1/jobs/:id/result. Writes need idempotency_key (and mcp:write
under OAuth).
Plan (unbilled)
POST /v1/timeline-1.0/plan (planTimelineV1) runs schema + Sume-host
URL checks + the pure compiler. It returns
object: timeline_plan with duration_seconds, segment_count,
billable_minutes, and a filtergraph_summary. It does not create a
job, reserve credits, or download media. Idempotency-Key is not required.
A plan cannot predict short-source pad/loop warnings.
Render
The required fields are audio.duration_seconds (1–1800), plus either
audio.url or audio.parts[] (unless audio.mode is "silence"), and
video[] (1–200 slots). Every URL must already be this workspace’s
media.sume.com artifact or asset. Import the files first
(POST /v1/media-imports). Idempotency-Key is required.
The default mode is async (readCommunicationOptions). To wait up to
30 seconds for a 200 finished job, pass mode: "sync". Otherwise, you
get 202. Then poll the job.
A successful submit returns a job (type: timeline_render,
model: sume/timeline-1.0). When the job is result_ready,
GET /v1/jobs/:id/result returns kind: timeline_render with video_url,
duration_seconds, segment_count, billable_minutes, and optional
warnings[]. Soft warnings (padded/looped short sources, snapped
transitions, ignored still motion) are not failures.
Credits scale with the output minutes, rounded up to whole minutes. The
API rate card lists the current credits.
The reserve is ceil(audio.duration_seconds / 60) minutes. The job uses no
provider inference, only worker ffmpeg.
The default output is 1080×1920 MP4. If you omit output.fps, the job
renders at the rate of the sources (the longest video sources decide, stills
have no rate, and 30 applies only when no source has a rate). If a rate is
different from the rate of a source, the job repeats or drops a frame every
few frames. This causes judder on motion. The job reports this as
output_fps_resamples_sources with the rate that the sources wanted.
Program
| Field | Effect |
|---|---|
audio.duration_seconds | Output length. Required. 1–1800 s. |
audio.url | One Sume-hosted spine. Not permitted with parts. |
audio.parts[] | ≤20 gapless slices (url + optional source_in / duration). Sample-domain join, no re-TTS. Not permitted with url. |
audio.mode | "silence" — declared length with no spine file. In this mode, do not send url / parts / gain_db / source_in. |
audio.source_in | In-point into a single url spine. Not permitted with parts. The output length is still duration_seconds. |
audio.gain_db | −60…12. Not permitted with silence. |
video[].source_url | Sume-hosted clip or still. Stills are static holds (the job accepts motion, ignores it, and reports motion_ignored). |
video[].start | On-spine start. video[0].start must be 0. Later starts must increase. Declared starts are authoritative. The compiler compensates for xfade and never pre-shifts. |
video[].duration | On-screen length, ≥ 0.2 s. Coverage can stop at most 0.5 s before the end of the spine. |
video[].source_in | In-point into the file. |
video[].fit | cover (default) | contain | stretch | blur. |
video[].transition | Use only on slots after the first. type ∈ fade | wipeleft | wiperight | slideup | slidedown | dissolve. Duration ≤ 1 s, ≤ 50% of the shorter neighbor, and at least one output frame. |
output.width / height | Even integers 256–2160. |
output.fps | 24 | 25 | 30 | 60. Omit it to match the sources. |
output.fade_in_seconds / fade_out_seconds | 0–5 s. The sum must be ≤ output length. |
soundtrack | Optional bed: url, gain_db, loop, fade_out_seconds ≤ 10, duck_db 0–20 (needs a real spine, not silence). |
render.strategy | auto (default, chunks past 12 segments) | chunked | single (Sume refuses single above 12 slots: render_strategy_unsafe). |
If you need sliced VO only inside this render, use audio.parts[]. For
a reusable merged file, use timeline audio. For two
sources on screen at the same time, use
timeline compose. Then put that MP4 into
video[].
Refusals (stable codes)
| Code | When |
|---|---|
audio_url_required | No url / parts and not silence. |
audio_url_and_parts_exclusive | Both url and parts. |
silent_audio_takes_no_url / _parts / _gain / _source_in | Silence plus a spine field. |
audio_source_in_requires_single_spine | source_in with parts. |
audio_parts_shorter_than_duration | The sum of the declared part lengths is less than duration_seconds. |
timeline_must_start_at_zero | video[0].start ≠ 0. |
transition_on_first_segment | video[0].transition. |
invalid_segment_timing / segment_overlap | Starts do not increase, or slots overlap past the xfade. |
transition_too_long / transition_not_frame_aligned | Duration compared with neighbors / fps. |
too_many_chained_transitions | More than 8 adjacent fades. Insert a hard cut. |
edge_fades_exceed_output / soundtrack_fade_exceeds_output | Fade longer than the spine. |
duck_requires_audio_spine | soundtrack.duck_db with silence. |
render_strategy_unsafe | strategy: "single" with more than 12 slots. |
unsupported_media_source / source_not_found | Off-host or dead URL. |
| Provider / ffmpeg keys | 400 — model, filtergraph, ffmpeg_args, codec, crf, and similar keys (TIMELINE_REJECTED_PROVIDER_KEYS). |
Sume rejects off-host URLs (https://example.com/…) at admit. Import the
files first.
A dim / crop / pixel pass is not a render option. Use video filter.
Not this surface
| Need | Use |
|---|---|
[start, end) of one clip | Video trim |
| Audio track as a durable wav / mp3 | Audio detach |
| Concat / split audio into reusable files | Timeline audio |
| Still + video in one frame (반배너) | Timeline compose |
| Pixel pass (dim / crop) | Video filter |
| Probe / stills / optional STT | Video inspect |