Cadence Workflows: How Uber's Engine Replays State, and How It Compares to Temporal

What this article covers

Cadence is Uber’s open-source distributed workflow orchestration platform. To reproduce the same business process after failures, deploys, or process restarts, you need a model that durably records execution state outside the application process and lets workers deterministically re-run workflow code following that record. Cadence provides that via a service-style execution engine and an event history.

This guide covers placement into domains and task lists, separation of workflow vs. activity responsibilities, signals and queries for asynchronous integration with the outside world, retry policies tuned to external systems, a comparison with widely adopted Temporal, and a practical order-processing workflow. Cadence’s Go and Java SDKs are mature, so examples skew Go-style, but the ideas apply across languages.

---

The problem Cadence solves

Implementing flows like “create order → pay → reserve stock → request shipping” directly in a distributed environment scatters state storage, retries, timeouts, idempotency, and manual recovery scripts across the codebase. Cadence folds those cross-cutting concerns into a workflow execution service and a recorded event history. If a worker process dies, the server keeps the history and a new worker replays the same workflow definition to restore progress.

From an operations standpoint, unlike batch scripts, you can look up, signal, or cancel in-flight procedures by identifier (workflow ID). Support, finance, and field ops can treat “which order is stuck at which step” like a product surface.

---

Architecture overview

A Cadence deployment is usually understood as the following components:

• Frontend Service: The gRPC entry point for clients and workers. Accepts workflow starts, signals, and queries.
• History Service: The core that stores and replays per-workflow event histories. This is what durability rests on.
• Matching Service: Queues tasks for task lists and delivers them to workers.
• Worker: A process you operate that polls for workflow and activity tasks. This is the unit of horizontal scaling.

Data flow is a loop: client starts an execution → events append to history → the matching service enqueues tasks on a task list → workers process → results are written back to history. As long as this loop holds, a started procedure is tracked to completion under the defined policies.

---

Core concepts

Durable execution and determinism

When a Cadence workflow function replays the same event history, it must always reproduce the same branches and the same activity schedule order. This is workflow determinism. It is therefore unsafe to put the following directly in workflow code:

• Direct calls to the network, databases, or message queues
• Non-fixed-seed randomness or arbitrary reads of “now” (language-specific safe APIs exist)
• Branches that depend on mutable global state shared outside the workflow

Side effects belong in activities; the workflow handles orchestration only—scheduling, branching, timers, waiting on signals.

Workflow execution and run ID

A workflow is a registered function-shaped procedure. When a client starts an execution with a workflow ID (often mapped to a business key), the server distinguishes each attempt with a run ID. Restarts, continue-as-new, and similar flows can chain multiple runs under one workflow ID. For ops logs and support tickets, recording both workflow ID and run ID aids tracing.

Activity

An activity is a unit of work with side effects: HTTP calls, DB transactions, message publishing. Activities can have their own timeouts, heartbeats, and retry policies, aligned with external API SLAs. The same activity may run more than once, so idempotency keys are effectively mandatory.

Child workflows and continue-as-new

Long runs or high event volume grow history and replay cost. Continue-as-new closes the current execution and continues under the same workflow ID with a new run, trimming history. Use child workflows when a sub-process should have its own lifecycle.

---

Domains and task lists

Domain

A domain is Cadence’s top-level isolation boundary for workflow executions. Splitting by team, product, or environment (e.g., production vs. staging) makes it easier to separate configuration (history retention, archival) and operational access. Workflow IDs need only be unique within a domain; they do not collide across domains.

When designing domains, consider:

• Regulation and data sovereignty: If certain users’ data must be isolated, domain-level separation is a candidate.
• Failure blast radius: Boundaries so overload in one domain does not destabilize other product lines.
• Deploys and versions: Document worker binaries alongside domain settings so which domain handles which workflow types is explicit.

Task list

A task list is the logical name of the queue from which workers poll tasks. Workflow tasks and activity tasks are each scheduled to task lists; a worker subscribes to one or more lists.

In practice, teams often split as follows:

• Service or team: Align with binary boundaries, e.g. orders-worker, payments-worker.
• Priority and SLA: Separate real-time orders from batch settlement to reduce mutual backlog.
• Deploy isolation: During canaries, have only some workers poll a new task list for gradual rollout.

A task list name is “just a string,” but in operations it ties directly to monitoring, alerts, and autoscaling, so a team-wide naming convention pays off.

---

Workflows and activities — contract and execution

Registering workflows

At startup, a worker process registers workflow and activity functions with names. When a client starts an execution under that name, schedule events accumulate in history and workflow tasks are delivered to a task list.

Activity options

Activity execution typically includes:

• ScheduleToStartTimeout: Upper bound from enqueue to assignment to a worker. Surfaces worker shortage and backlog.
• StartToCloseTimeout: Upper bound for the worker to run the activity to completion. Caps business logic duration.
• HeartbeatTimeout: Checks that long work is still alive. Used together with heartbeats.

In order domains, “wait for payment gateway response” vs. “integrate with warehouse WMS” often need different timeouts, so per-activity-type options are common.

Versioning and compatibility

Changing workflow source can conflict with replay. While long-running instances still exist, conditional branches (e.g., build ID, domain constants, explicit version flags) let old and new logic coexist; remove branches after old executions drain. For activity input/output schema changes, backward-compatible field additions or new activity names are safer migration paths.

---

Signals and queries

Signal

A signal is an external event delivered asynchronously to a running workflow instance. Examples: “payment webhook received,” “customer cancellation,” “stock reservation complete.” The workflow receives events on signal channels and can wait on multiple sources (timers and signals) with a Selector pattern (Go).

Signal handlers are replayed too, so they must deterministically update only in-workflow state. Explicitly validate ordering against business rules (e.g., whether cancellation must be processed before payment confirmation).

Query

A query is a read-only path to inspect the workflow’s current state. Support tools and internal dashboards use it to show “what step is this order in?” Query handlers must not cause side effects and should return a consistent snapshot of state the workflow maintains.

---

Retry policies

Activity retries

External APIs fail due to timeouts, transient errors, and rate limits. Cadence attaches a RetryPolicy to activities to tune:

• InitialInterval: Wait before the first retry
• BackoffCoefficient: Multiplier for exponential backoff
• MaximumInterval: Cap on backoff
• MaximumAttempts: Max attempts (check docs—0 may mean unlimited depending on version)
• NonRetriableErrorReasons: Classify errors where retry is pointless (e.g., invalid order ID) as immediate failure

Independently of retries, design for duplicate activity executions. For payments and inventory, use request keys or tokens supported by external systems.

Workflow tasks and worker-side failure

If a workflow decision cycle takes too long or replay fails due to a bug, it surfaces as a workflow-task-level error. That differs from activity failure, so keep workflow code to light control logic and push heavy work to activities.

---

Comparison with Temporal

Temporal is a separate project that forked and evolved the Cadence codebase. Core ideas—event history, deterministic workflows, activities, signals and queries, task queues—are the same, but product roadmaps, SDKs, and managed services diverged.

Aspect	Cadence	Temporal
Origin	Uber open source	Independent community after Cadence fork
Namespace	Domain	Namespace (similar concept)
Task routing	Task list	Task queue
SDK maturity	Go- and Java-centric	Broad: Go, Java, TypeScript, Python, PHP, .NET, …
Managed service	Self-operated / community	Temporal Cloud, etc.

When to read about Cadence: Legacy systems already on Cadence, or when alignment with Uber open-source docs and patterns matters. When choosing a new standard, teams usually compare Temporal and Cadence by language SDKs, cloud ops needs, and enterprise support. Many teams read Cadence for concepts and implement with the Temporal SDK.

---

Hands-on: order-processing workflow (Go style)

Below is a simplified create order → reserve inventory → request payment → wait for payment signal → release inventory on failure. Production needs transaction boundaries, fraud checks, audit logs, and PSP-specific APIs.

Signals, queries, and state

// Conceptual example: adapt real package paths and client construction to your environment.
const (
    SignalPaymentResult = "payment_result"
    QueryOrderState     = "order_state"
)

type PaymentResultPayload struct {
    Success bool
    TxID    string
}

type OrderState string

const (
    StatePending    OrderState = "pending"
    StateReserved   OrderState = "inventory_reserved"
    StatePaid       OrderState = "paid"
    StateCancelled  OrderState = "cancelled"
    StateFailed     OrderState = "failed"
)

Exposing OrderState from the workflow lets query handlers pass it straight to support UIs.

Workflow body

import (
    "fmt"
    "time"

    "go.uber.org/cadence"
    "go.uber.org/cadence/workflow"
)

func OrderWorkflow(ctx workflow.Context, orderID string, amountCents int64) error {
    logger := workflow.GetLogger(ctx)
    state := StatePending

    workflow.SetQueryHandler(ctx, QueryOrderState, func() (OrderState, error) {
        return state, nil
    })

    paymentCh := workflow.GetSignalChannel(ctx, SignalPaymentResult)
    var pay PaymentResultPayload
    timerFired := false

    ao := workflow.ActivityOptions{
        StartToCloseTimeout: time.Minute,
        RetryPolicy: &cadence.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 2,
            MaximumInterval:    30 * time.Second,
            MaximumAttempts:    5,
        },
    }
    actx := workflow.WithActivityOptions(ctx, ao)

    if err := workflow.ExecuteActivity(actx, ReserveInventoryActivity, orderID).Get(actx, nil); err != nil {
        logger.Error("reserve failed", "order", orderID, "err", err)
        state = StateFailed
        return err
    }
    state = StateReserved

    if err := workflow.ExecuteActivity(actx, RequestPaymentActivity, orderID, amountCents).Get(actx, nil); err != nil {
        _ = workflow.ExecuteActivity(actx, ReleaseInventoryActivity, orderID).Get(actx, nil)
        state = StateCancelled
        return err
    }

    sel := workflow.NewSelector(ctx)
    sel.AddReceive(paymentCh, func(c workflow.Channel, more bool) {
        c.Receive(ctx, &pay)
    })
    tf := workflow.NewTimer(ctx, 30*time.Minute)
    sel.AddFuture(tf, func(f workflow.Future) {
        timerFired = true
    })
    sel.Select(ctx)

    if timerFired {
        _ = workflow.ExecuteActivity(actx, ReleaseInventoryActivity, orderID).Get(actx, nil)
        state = StateCancelled
        return fmt.Errorf("payment timeout")
    }
    if !pay.Success {
        _ = workflow.ExecuteActivity(actx, ReleaseInventoryActivity, orderID).Get(actx, nil)
        state = StateCancelled
        return fmt.Errorf("payment declined")
    }

    // 4) Confirm payment (capture, etc.)
    if err := workflow.ExecuteActivity(actx, CapturePaymentActivity, orderID, pay.TxID).Get(actx, nil); err != nil {
        state = StateFailed
        return err
    }
    state = StatePaid
    return nil
}

Using a Selector to wait on both signal and timer lets you encode policies like “if payment is not confirmed within 30 minutes, cancel.” On the timeout branch, release-inventory activity runs as a compensating transaction.

Activities (conceptual)

func ReserveInventoryActivity(ctx context.Context, orderID string) error {
    // Inventory service HTTP/gRPC call — idempotency key: orderID
    return nil
}

func RequestPaymentActivity(ctx context.Context, orderID string, amountCents int64) error {
    // Create a payment session with the payment gateway — the client sends a signal with the Workflow ID from the webhook
    return nil
}

func ReleaseInventoryActivity(ctx context.Context, orderID string) error {
    return nil
}

func CapturePaymentActivity(ctx context.Context, orderID, txID string) error {
    return nil
}

The webhook handler maps the order ID to a workflow ID and calls SignalWorkflow on the Cadence client. Webhooks need signature verification and handling for duplicate delivery; consider deduplication by event ID so duplicate signals still leave state consistent.

Workers and task lists

// At worker startup: register OrderWorkflow and subscribe to the "orders" task list.
// Binding it with workflow.RegisterOptions{Name: "OrderWorkflow"} is the typical pattern.

In operations, splitting activity-specific workers and task lists for payment, inventory, and notifications reduces one failing segment blocking all polling.

---

Best practices and pitfalls

• Keep workflows thin: Modularize long procedures with child workflows; separate per-step timeouts and compensations.
• Idempotency first: Do not assume an activity runs only once. Fix order, payment, and inventory keys to external contracts.
• Signal ordering: Test concurrent signals against business rules.
• Secrets: Do not pass raw card numbers into workflow arguments—only tokens or references.
• Observability: Structure logs and metrics with workflow ID, run ID, and activity type.

---

Changing workflow code while executions are running

Determinism has a consequence that surfaces only after the first deploy: workflows that started under the old code will be replayed by workers running the new code. If you add, remove or reorder an activity call, a timer or a signal wait, the replayed history no longer matches the commands the new code produces, and the worker reports a non-deterministic error and stops making progress on those executions. Long-running workflows, such as orders that wait days for shipping, make this likely.

Wrap such changes in workflow.GetVersion (Go) or Workflow.getVersion (Java): executions that recorded the old version keep taking the old branch, new executions take the new one, and the old branch can be deleted once no execution still uses it. Before deploying workflow changes, replay a few recorded production histories against the new code in a test; the SDKs provide a workflow replayer for this.

---

Related Articles

• Event Streaming with Kafka
• RabbitMQ in Production
• Go Web Development Guide | REST APIs, Middleware

---