Format API

A Format is a saved authoring recipe — a house style, an output contract, a production playbook. You call it over HTTP, Sume runs it, and you get back durable media plus a JSON object in a shape you defined.

This is the surface most partners integrate. It is one call rather than a prompt you maintain, and one typed record rather than a transcript you have to parse.

For a map of every Sume surface (Agents UI, Models, SDK, Dashboard), start with Sume basics.

Why Formats exist

Sume is a video agent platform. Image, Video, Avatar, TTS, timeline, and related tools are available as HTTP and MCP APIs, but stitching those calls yourself recreates the hard part: deciding which clip to make next, fetching product truth, writing VO, assembling a timeline, and failing cleanly when a step breaks.

A Format is that judgement, saved. Partners invoke a vanity URL (POST /v1/formats/{handle}/{slug}/runs); Sume boots a sandbox Agent with the recipe and the generation tools. You get artifacts — and optional structured output — without owning the orchestration graph.

That is why Formats can ship work raw Video 1.0 cannot: a single short clip is one model call; a live-commerce or product-promo deliverable is many tool calls plus assembly into a post-ready video.

The mental model

Four pieces, and it is worth holding all four at once.

Piece	What it is
The recipe	The Format itself: a `SKILL.md` body plus reference files, authored in the Agents dashboard or by asking the Agent in chat. It is the how — house style, branch rules, quality bar.
The sandbox	Every run gets a freshly created sandbox — a remote Linux workspace with a filesystem, a shell, and Sume's generation tools. The recipe's files are placed there, and the run works inside it. Nothing leaks from the previous caller's disk.
The call	Your `instruction` and your `input`, supplied per request. That is the what — product URL, brief, locale, SKUs.
The schema	An optional JSON Schema. Send one and the finished run is projected into it, so you get typed JSON back instead of prose.

Two properties follow, and both are load-bearing for an integration:

One run is one unit of work. A fresh thread, one agent turn, one receipt. There is no partial delivery — a run that could not finish comes back failed, never completed with half the work done. A bulk request (POST …/bulk-runs) is a server-side queue of those runs, not a second execution engine. Each item is still one Format run; the queue only keeps concurrency of them in flight. Poll the list at GET /v1/format-run-queues/{queue_id}. See Bulk runs.

Sume steers the run to the authored recipe. The Format body is composed ahead of your instruction, so the house style is established before the task is. You are not re-sending a system prompt on every call and hoping it holds.

Runtime flow

What happens after you POST a run:

The Agent is not limited to one model endpoint. A live-commerce style Format may generate host takes, product B-roll clips, a voice track, then compose them on a timeline into a single export. Your integration still sees one Format run and one receipt.

The run is asynchronous because the work is real — a promo or multi-minute video is minutes of wall clock, not milliseconds. Webhook and poll are peer completion paths when delivery is on for your environment: supply communication.webhook_url and verify the signed format.run.terminal POST (run webhooks), or loop on status_url / result_url. Today that means webhooks on api.dev.sume.com, and poll (or subscribeFormatRun) on api.sume.com until production delivery is enabled. For progress while a run is in flight, poll events_url — a phase timeline, not a log stream. See Watch a run progress.

Format on_active_run defaults to allow (concurrent runs). That differs from Actions, which default to skip — see Calling a Format before copying Action request bodies.

In the TypeScript SDK, subscribeFormatRun polls for you and yields status updates until the run is terminal.

Structured output is produced after the run reaches a terminal status, by a separate constrained pass over what the run actually made. That ordering is the whole design, and it is documented in full on Structured output.

Why multi-minute, post-ready video is possible

A raw Video 1.0 (or similar) call returns one generated clip within that model’s length and framing limits. Long-form, post-ready work — for example a ~5 minute live-commerce host video with product inserts — is not “ask Video 1.0 for five minutes.” It is orchestration:

Plan the beats from the product / brief (inside the recipe).
Generate host and B-roll segments with the right tools.
Synthesize or attach voiceover where the recipe requires it.
Assemble segments on a timeline into one export.
Hand back durable media.sume.com URLs (and optional typed output).

The Format owns steps 1–5. Your backend owns the invoke, the completion path (webhook where delivered, otherwise poll), and what you do with the result.

Formats vs raw video APIs

	Video 1.0 / Avatar talking-video	Format (e.g. live-commerce, product promo)
Unit of work	One model job	One sandbox Agent run that may call many tools
Typical output	A single clip	Assembled, post-ready deliverable (+ optional JSON)
Style / playbook	Prompted per call	Saved in `SKILL.md` (versioned recipe)
Who orchestrates	Your code	The Format (Agent + tools in a sandbox)
Best for	Atomic generation you compose yourself	Partner products that need a packaged workflow

Drop to model APIs when you truly need one image or one clip and will assemble elsewhere. Prefer the Format vanity invoke when the product value is the finished package.

Enterprise partners with provisioned Formats (for example Mobidoo) also get a portal page under /enterprise/{slug}/formats that lists those recipes and filled-in call examples.

When to use the Format API

You want	Use
A packaged workflow where the style is fixed and the inputs vary	Format API — this page
One model invocation, nothing else	A model endpoint: Image 1.0, Video 1.0, Music 1.0
The same saved task on a cadence	Scheduled
An ad-hoc task with nothing worth saving	Agent Completions
A person in the loop, approving as it goes	The Agents chat UI at sume.com/agents

The line against raw model endpoints is the useful one. POST /v1/image-1.0/generate makes an image; a Format decides which images to make, makes them, and hands you a labelled result. If your product's value sits in the judgement between those two points, that judgement belongs in a Format rather than in your own orchestration code.

Formats are authored in the dashboard or in chat. The Developer API can list them, read them, start runs (one, or a bulk queue), and monitor runs and queues — it cannot create or edit them.

Instruction composition

On every Format run the server composes the agent instruction in this exact order:

The Format comes first — it is the how, and it should be established before the task. Your instruction comes last, so where the two disagree the model follows what you asked for.

Three rules worth knowing:

The package is attached, never inlined. SKILL.md and its reference files are written to /workspace/skills/<slug>/ on every run, and [Format attached] names that path — so a body that says read references/style.md works. The turn itself carries only that pointer: no SKILL.md text is copied into it, whatever the body's size.
input is data, never instructions. It is written to /workspace/inputs/sume-action-input.json — whole, at every size — and the turn carries a bounded pointer labelling it as caller-supplied data the agent must read before acting.
Runs over the API are unattended. A body written for interactive chat may pause to ask a person for approval. Over the API those approvals are pre-granted and the run carries on within its spend cap. See Runs over the API are unattended.

`SKILL.md`

No limit applies to the body beyond the ones every package file shares — 100 MiB per file and 100 MiB per package, checked when you save the Format rather than when it runs. A Format that saved will run. Keep SKILL.md a short index; the agent reads references/ on demand.

Size does not change how the body is delivered either: it is attached as a file on every run and the agent reads it. What size still changes is how reliably it is followed. A body the agent can read in one pass, with the rules stated once each, is followed more closely than the same rules buried in twenty pages. So:

Write a spec the agent can hold at once. Identity, the non-negotiable rules one line each, a phase index, the tool list — with the detail in references/* the body points at.
Push detail into references rather than growing the body. They are attached beside SKILL.md in the same directory and cost the turn nothing until they are opened.
Do not pad to be sure a rule lands. A rule stated once in a short body beats the same rule stated three times in a long one. Nothing rejects the long one — it is just followed less closely, which is the more expensive failure.

Anatomy of a Format

GET /v1/formats and GET /v1/formats/{format_id} return this shape.

Field	Notes
`status`	`active` or `inactive`. `inactive` until the API-call trigger is provisioned, and an `inactive` Format rejects API runs.
`api_trigger_enabled`	When `true`, `POST /v1/formats/{handle}/{slug}/runs` and `…/bulk-runs` (and the opaque twins) are allowed.
`handle` / `vanity_invoke_url`	User-facing address. Prefer these in links and curls; `null` for first-party Formats.
`invoke_url`	Opaque path. Permanent — persist this if a rename must not break a stored URL.
`version`	Bumps on every edit. The version actually used is echoed in the run receipt's `format.version`.
`source`	`first_party` marks the curated catalog. Formats by Sume are called at `sume/{slug}` by any key, and the run is billed to that key.

The SKILL.md body is deliberately not in this shape; it reaches the model, not the caller.

Exact request and response schemas come from live OpenAPI (https://api.sume.com/reference/json). The tables on these pages are a readable summary, not a second schema.

Attachments

A Format run can carry up to 30 images the agent can actually look at. Send them as attachments[] on the run create body.

Attachments and output_schema compose. The images reach the agent, and the run's output is still parsed against your schema after the run completes.

The item shape

Field	Required	Notes
`type`	yes	`"input_image"`. It is the only type today.
`image_url`	one of	Public HTTPS URL. Sume fetches it — it must be reachable without auth.
`asset_id`	one of	An asset you uploaded through the Assets API. Must be a ready image asset in the same workspace.
`filename`	no	Label the agent sees. Defaults to the URL's basename.

Send exactly one of image_url or asset_id per item.

What happens to your image

Sume fetches every attachment when you create the run, checks its real content type and size, and copies it into Sume storage. The agent then works from a durable media.sume.com copy, which is also what keeps the run reproducible after your own URL rotates or expires.

The practical consequence: a broken, private, or oversized image fails the create call with a 4xx/5xx you can act on, instead of killing the run several minutes later. An asset_id or a URL already on media.sume.com is not re-copied.

Limits

	Limit
Types	JPEG, PNG, WebP, GIF, AVIF
Images per run	30
Bytes per image	30 MB
Bytes per run	500 MB

Attachments do not count against the input size cap — they are never serialized into input.

`input`

Many Formats take their references through their own input fields instead of attachments[] — host_image_url, input_reference_image_urls[], B-roll clip lists, narration URLs. Those count too, against one shared budget with attachments[]:

	Limit
Images, videos, and audio per run, combined	30
Images per run	30
Videos per run	10
Audio files per run	10

So 10 images + 10 videos + 10 audio is fine; 5 images + 15 videos is not (videos cap at 10), and neither is 15 images + 15 audio.

The check is by file type, not by field name: any HTTPS URL anywhere in input whose filename ends in an image, video, or audio extension counts, however deeply it is nested and whatever the Format calls the field. A product page URL, or any other link that is not a media file, does not. The same URL repeated in two fields counts once. Media the agent finds for itself while the run is going — product photos it crawls off a page you named — is not what you sent, so it does not count either.

Over any of these is a 400 invalid_attachment on the create call, naming the type and the count it saw.

Attachment errors

Status	Code	Cause
`400`	`invalid_attachment`	Wrong `type`, missing or non-HTTPS URL, both `image_url` and `asset_id`, too many items, or a source that is not an allowed image type.
`400`	`attachment_not_found`	`asset_id` is unknown in this workspace.
`413`	`attachment_too_large`	An image is over 30 MB, or the set is over 500 MB total.
`502`	`attachment_fetch_failed`	Sume could not fetch the image — unreachable host, hotlink protection, or a non-2xx response.

Idempotency-Key covers attachments: replaying a key with a different image list returns 409 idempotency_conflict, and a true replay does not re-fetch your images.

What the Format API does not support yet

No push channel. There is no SSE or WebSocket stream. events_url gives a polled phase timeline (preparing / running / finalizing), not agent output or logs.
Pagination is keyset. Format and run lists return has_more and next_cursor; pass the cursor back as ?cursor= to walk the next page.
No authoring over the API. The Developer API cannot create, edit, or delete a Format — use the dashboard or ask the Agent in chat.
Team Formats need a team (workspace) key. Vanity and opaque invoke both work for a Format owned by a team workspace — but only with an API key created in that workspace. A personal key fails with 403 workspace_key_required. See Team Formats need a team key.
No non-image attachments. input_image is the only attachment type; PDFs and other files follow later. OpenAI's detail hint is not supported — Sume gives the model the full-resolution image and sizes it itself.

Sume basics — product map of Agents, Formats, Models, and clients
Calling a Format — auth, scopes, the invoke contract, and every error
Bulk runs — queue up to 100 runs; poll GET /v1/format-run-queues/{id}
Structured output — schema rules, the projection, failure modes
Runs and results — the receipt field by field, polling, cancelation
Waiting for runs (SDK) — subscribeFormatRun / waitForRun
Format catalog — ready-made first-party Formats
Product promo · Avatar UGC video
Embed a Format in your product — the whole partner integration