Async Agents & Workflows

Some work is not a per-row scalar: calling an LLM, hitting an external API, or running a multi-step process takes seconds to minutes and must be durable and retry-safe. ParticleDB models these as two async Particle kinds driven by an external durable-execution engine — Conductor. ParticleDB does not hand-roll a workflow engine, retry scheduler, or timer service; it owns exactly one durable thing: the identity mapping between a submission and its Conductor execution.

The two async kinds (planned)

@agent — a single async step of bounded external I/O (an LLM call, an API call, a tool use). Maps to a Conductor task.
@workflow — a durable multi-step DAG. You express the steps in Python; the deploy tool lowers them to a Conductor workflow definition.

The model — submit returns a `run_id`

Async Particles are never allowed in an ordinary scalar predicate (that would block the query and let a WHERE clause fan out unbounded external calls). The surface is explicit submission verbs that return a run_id immediately:

-- Submit an @agent or @workflow. Returns ONE row with a run_id, immediately —
-- it does NOT wait for the run to finish.
SUBMIT PARTICLE triage_and_reply(ticket_id => 'T-4821')
       WITH ( deadline => '10m', idempotency_key => 'T-4821' );
-- run_id
-- ────────────────────────────────
-- 9b1f…   (a UUID minted by ParticleDB)

-- Apply an @agent across a query result set — one run_id per input row:
ASYNC APPLY classify_ticket(subject, body)
       OVER (SELECT subject, body FROM tickets WHERE new)
       WITH (max_concurrency => 8);

A submit does three things, then returns: (1) mint a run_id, (2) record a durable PENDING run + an outbox entry inside the caller’s SQL transaction, (3) return the run_id. The Conductor start happens after commit via an outbox relay — SQL-transaction atomicity is never coupled to Conductor start.

Checking status and result by `run_id`

Because the run is asynchronous, you observe it through governed, tenant-scoped table functions keyed by the run_id:

SELECT particle_run_status('9b1f…');
--   PENDING | SUBMITTED | RUNNING | COMPLETED | FAILED | TIMED_OUT | CANCELED

SELECT particle_run_result('9b1f…');   -- result payload (NULL until COMPLETED)

SELECT * FROM particle_run_steps('9b1f…');  -- per-step rows (status, timestamps)

These are cheap, blocking catalog reads — the async thing is the run itself, not the status read. A tenant sees only its own runs; admin sees all.

Durability & retry-safety (design)

Transactional outbox. The submit writes a durable intent in the caller’s transaction; a post-commit relay starts the run on Conductor. If Conductor is down, the run stays PENDING and the intent is never lost — the relay retries.
Idempotency. Every submission carries an idempotency key (explicit or derived from the particle, version, and canonical arguments). It becomes Conductor’s correlation/dedup key, so a duplicated submit resolves to the same run rather than starting a second one.
Monotonic status. Status updates are applied monotonically — a terminal run never regresses, and stale/out-of-order updates are dropped.
Bounded concurrency, backpressure, deadlines. Per-tenant and per-destination caps map to Conductor rate limits; over capacity, a submit still returns a run_id but the run stays queued; deadlines map to Conductor timeouts.

What exists today

A testable skeleton (behind particles-conductor, OFF by default) implements the two async kinds, the pluggable backend trait, an in-memory mock Conductor backend, and the submit → relay → reconcile dispatcher with the outbox / idempotency / monotonicity invariants — all round-tripped in unit tests with no real Conductor. A real end-to-end run needs a live Conductor and the follow-up work listed in the callout above.

Next steps

Overview — how Particles fit together.
Rust Particles — the native cdylib preview tier.