Managing Long-Running Processes — Cross-Cutting Pattern

Technologies that implement this pattern: Kafka · PostgreSQL · DynamoDB · Redis · ZooKeeper & etcd · Flink & Stream Processing

Why This Matters#

Long-running processes are where the request/response model quietly stops being true. Every layer of a web stack assumes work finishes in seconds: API gateways cut requests at ~29–60s, load balancers drop idle connections at 60s, serverless functions die at 15 minutes, Kubernetes gives a terminating pod 30 seconds. Then the product asks for a 40-minute video export, a 3-day onboarding workflow with a human approval step, or a payout that waits for a bank settlement file. The work outlives every process, connection, and deploy that touches it.

Most candidates treat this as a queue question: "put it on a queue and have a worker process it." Staff engineers treat it as a durable state question: where does the progress of this work live, so that any worker can crash, any deploy can roll, and the work resumes from the last meaningful step instead of starting over or — worse — running twice? The queue moves work; it does not remember how far the work got. That memory — checkpoints, heartbeats, leases, compensation records — is the design.

In interviews, the pattern hides inside job schedulers, payment flows, report generation, video processing, ML training, data backfills, and order fulfillment. The L5 answer is "async job + queue + worker." The L6 answer adds the job state machine, heartbeats, checkpoint/resume, idempotent steps, cancellation, and a saga for multi-service work. The L7 answer asks whether the org should run one workflow platform instead of fifteen ad-hoc job tables, and what happens to 200K in-flight workflows when someone deploys incompatible code.

The 60-Second Version#

• Anything over ~10 seconds leaves the request path. Return 202 Accepted with a job ID in < 200 ms; let the client poll (every 2–5 s with backoff) or receive a webhook/push when state changes.
• Progress must live outside the worker. Checkpoint every 30–60 s or every N items to a durable store. A 2-hour job without checkpoints loses an average of 1 hour of work per crash.
• Heartbeats detect death; leases prevent duplicates. Heartbeat every ~10–30 s; declare dead after 3 missed. SQS visibility timeout defaults to 30 s (max 12 h) — a job longer than its lease will run twice.
• Every step must be idempotent. At-least-once delivery plus retries means each step runs ≥ 1 time. Key side effects on (job_id, step, attempt-independent key).
• Multi-step, multi-service, or human-in-the-loop? Use a workflow engine. Temporal, AWS Step Functions (Standard: up to 1 year per execution), or similar give durable timers, retries, and history for free. Hand-rolling this is 12–18 engineer-months to reach parity.
• Cancellation and compensation are features, not afterthoughts. A saga across 4 services needs a compensating action for each completed step, and cancellation must be cooperative — checked at every checkpoint boundary.

The Problem#

A user clicks "Export all my data," a merchant triggers a payout, an admin starts a 50M-row backfill, or a customer's order kicks off payment → inventory → shipping → notification across four services, one of which waits two days for a carrier pickup. Each takes minutes to days. During that time, workers are redeployed several times, a node dies, a downstream service has a 20-minute outage, and the user clicks "cancel" halfway. The system must report accurate progress, never lose work, never double-charge or double-ship, resume after crashes without restarting from zero, and cleanly undo partial work when asked to stop.

Case Studies That Use This Pattern#

• Distributed Job Scheduler — Leases, heartbeats, and exactly-once-effect execution of scheduled work
• Payment Processing — Payouts and settlement as multi-day workflows with compensation
• Code Execution — Sandboxed runs with timeouts, cancellation, and async result delivery
• Web Crawler — Checkpointed crawl frontiers that survive restarts
• Ride Hailing & Delivery — Order lifecycle as a long-running state machine with human steps
• Reservation Systems — Hold → pay → confirm sagas with timeouts and compensation
• Message Queues — Visibility timeouts, redelivery, and DLQs underpin every job system

The Core Tradeoff#

The L5 → L6 → L7 Contrast#

Staff Default Position#

Get the work out of the request path, keep its progress in durable state, and make every step idempotent.

Return 202 with a job ID; persist a job record with an explicit state machine; process with workers that hold a lease, heartbeat, and checkpoint at meaningful boundaries; key every side effect on a stable idempotency key so retries are harmless. Use a plain queue + job table for single-step jobs under ~15 minutes. Use a durable workflow engine as soon as the work has multiple steps across services, waits longer than a worker can live (timers, human approvals, external callbacks), or needs compensation. Cancellation is cooperative and checked at every checkpoint.

When to Deviate#

• Work reliably under ~5–10 seconds. Keep it synchronous with a timeout and idempotency key; async machinery adds polling latency and a state machine for no benefit.
• Pure data processing at high volume. A 10 TB transformation is a batch/stream job (Spark, Flink), not 10 million workflow executions — engines like Temporal charge per action and aren't built for per-record orchestration.
• One team, one simple job type, low volume. A Postgres job table with SELECT ... FOR UPDATE SKIP LOCKED is fine for thousands of jobs/day; adopting a workflow platform for one cron-like job is over-engineering.
• Latency-critical multi-step flows. A workflow engine adds ~10–50 ms per step of persistence overhead. For a sub-100 ms checkout path, do the critical steps synchronously and hand only the tail (fulfillment, notifications) to the workflow.

Common Interview Mistakes#

Quick Reference#

Staff Sentence Templates#

"This takes [N minutes], which is longer than our gateway timeout and longer than a deploy cycle, so I'll return 202 with a job ID and make the job's progress durable — the worker is disposable, the checkpoint is not."

"Each worker holds a lease of [N] seconds and heartbeats every [N/3]. If it misses three heartbeats, another worker takes over from the last checkpoint. Because delivery is at-least-once, every side effect is keyed on [job_id + step]."

"This flow crosses [N] services and waits on [a human / an external callback] for up to [N days], so I'd model it as a workflow in [Temporal / Step Functions] rather than a chain of queue consumers — we get durable timers, retries, and a single place to answer 'where is order X?'"

"If the user cancels after [step K], steps 1 through K are already committed. Cancellation runs the compensations in reverse — [refund, release inventory, void label] — and each compensation is itself idempotent and retried until it succeeds."

Implementation Deep Dive#

1. The Async Job Pattern — 202, Job Resource, and Status Delivery#

The API contract is the first design decision, because clients will build around it for years.

# Job resource — the contract clients poll
GET /jobs/j_81
{
  "id": "j_81",
  "state": "RUNNING",              # QUEUED | RUNNING | SUCCEEDED | FAILED | CANCELLING | CANCELLED
  "progress": { "done": 4_200_000, "total": 10_000_000 },
  "created_at": "...", "updated_at": "...",
  "result_url": null,              # set on SUCCEEDED; short-lived signed URL
  "error": null,                   # { code, message, retryable } on FAILED
  "links": { "cancel": "/jobs/j_81/cancel" }
}
# Response header: Retry-After: 5   -> server controls polling rate

🎯 Staff Insight: The Idempotency-Key on job creation is not optional. A user double-clicking "Export" or a mobile client retrying on a flaky network will otherwise create two 40-minute jobs. A unique constraint on (tenant_id, idempotency_key) turns the retry into "here's the job you already started."

2. Leases and Heartbeats — Knowing When a Worker Is Dead#

A worker that stops making progress looks identical to a worker that's slow — until you add heartbeats. The lease is the mechanism that lets someone else take over, and it is also the mechanism that causes duplicate execution if you get it wrong.

LEASE_TTL = 60s
HEARTBEAT_EVERY = 15s          # ~1/4 of lease: tolerates 3 missed beats

function run(job):
    lease = acquire(job.id, owner=worker_id, ttl=LEASE_TTL)   # CAS on job row
    stop = start_heartbeat(job.id, lease, every=HEARTBEAT_EVERY)
    try:
        for chunk in job.remaining_chunks(from=job.checkpoint):
            if stop.lease_lost:                  # heartbeat CAS failed: someone took over
                return abandon()                 # stop immediately, do NOT commit
            process(chunk)                       # idempotent
            checkpoint(job.id, chunk.end, fencing=lease.token)
    finally:
        stop()

function checkpoint(job_id, cursor, fencing):
    # Fencing token rejects writes from a worker whose lease already expired
    ok = db.exec("UPDATE jobs SET cursor=$1, updated_at=now()
                  WHERE id=$2 AND lease_token=$3", cursor, job_id, fencing)
    if not ok: raise LeaseLost

Queue-native equivalents: SQS visibility timeout (default 30 s, max 12 h) extended with ChangeMessageVisibility as the heartbeat; Temporal activity heartbeat() with a HeartbeatTimeout; Kubernetes Jobs with activeDeadlineSeconds as a hard cap.

🎯 Staff Insight: A lease without a fencing token is only a hint. If a paused worker wakes up after its lease expired, it will happily write its stale checkpoint over the new owner's progress. The fencing token (a monotonically increasing lease version checked on every write) is what makes takeover safe.

3. Checkpointing — Bounding the Cost of a Crash#

Checkpoint frequency is a direct tradeoff between overhead and rework. The expected rework per crash is roughly half the checkpoint interval.

# Cursor-based checkpoint for a 50M-row backfill
function backfill(job):
    cursor = job.checkpoint or MIN_ID
    while cursor < job.max_id:
        batch = db.query("SELECT ... WHERE id > $1 ORDER BY id LIMIT 5000", cursor)
        write_idempotently(batch)                 # upsert keyed on id: replay-safe
        cursor = batch.last_id
        if time_since_last_checkpoint() > 30s or rows_since_checkpoint() > 50_000:
            checkpoint(job.id, cursor)
        throttle_to(db.replica_lag < 2s)          # don't melt the primary

The consistency trap: the checkpoint and the side effects are two writes. If you write side effects and crash before checkpointing, the replay repeats those side effects — which is fine only if they're idempotent. If you checkpoint first and crash before side effects, you skip work. Rule: side effects first, idempotently; checkpoint after. Or commit both in the same transaction when they share a database.

🎯 Staff Insight: Checkpoint meaning, not just position. A cursor says "I've processed up to row 4.2M." It doesn't say whether the aggregate you're building includes rows 4.1–4.2M. For stateful jobs, the checkpoint must include the partial state (or its location) — this is exactly what Flink's checkpoint barriers do for stream jobs.

4. Durable Workflow Engines — Temporal and Step Functions#

When work spans services, timers, and humans, a workflow engine persists every step's result in an event history. Workflow code is replayed from that history after a crash, so the code looks sequential but survives process death.

# Temporal-style workflow (pseudocode): order fulfillment
@workflow
def fulfill_order(order):
    opts = ActivityOptions(start_to_close=30s, retry=Retry(max_attempts=10,
                           backoff=2.0, max_interval=5min))
    charge = activity(charge_card, order, opts)           # idempotency key = order.id
    try:
        activity(reserve_inventory, order, opts)
        label = activity(create_shipping_label, order, opts)
        # Durable timer: survives worker restarts and deploys, costs nothing while waiting
        picked_up = wait_for_signal("carrier_pickup", timeout=72h)
        if not picked_up:
            activity(escalate_to_ops, order, opts)
            wait_for_signal("carrier_pickup", timeout=7d)
        activity(send_shipped_email, order, opts)
    except (CancelledError, ActivityFailure) as e:
        # Compensation in reverse order; each is idempotent and retried
        activity(release_inventory, order, opts)
        activity(refund_charge, charge, opts)
        raise

# Rules the engine imposes: workflow code must be deterministic —
# no wall-clock reads, random(), direct I/O, or unversioned logic changes.

🎯 Staff Insight: The event history limit is the one production teams learn the hard way. A workflow that loops over 100K items as activities will hit ~50K events and be terminated. Long-lived or high-iteration workflows must use continue-as-new (restart with carried-over state) or fan out to child workflows — design for it on day one.

5. Sagas, Cancellation, and Compensation#

A multi-service long-running process has no global transaction. Each step commits locally; failure after step K means undoing steps 1..K with semantic compensations — which are business operations, not database rollbacks.

function cancel(job_id, reason):
    db.cas(job_id, from=["QUEUED","RUNNING","WAITING"], to="CANCELLING", reason)
    # Cooperative: worker/workflow observes CANCELLING at its next checkpoint or await
    # QUEUED jobs are cancelled immediately — no work to undo

# Worker side, at every checkpoint boundary:
if job.state == "CANCELLING":
    run_compensations(completed_steps.reversed())
    db.cas(job.id, from="CANCELLING", to="CANCELLED")
    return

Pivot step: order the saga so that irreversible steps (shipping, external transfers) come last, after all compensable steps succeed. Once past the pivot, the saga can only move forward — retries, not compensation.

🎯 Staff Insight: Cancellation latency is a product decision. "Cancel stops within one checkpoint interval (≤ 60 s)" is an SLA you can state and meet. "Cancel stops immediately" is a promise you can't keep for a worker mid-way through a 5-minute external call. Say the bound out loud.

Architecture Diagram#

Reading the diagram: two execution paths share one control plane. Single-step jobs go queue → worker with leases and checkpoints in the job store. Multi-service flows go to the workflow engine, which owns timers and compensation. The reaper and DLQ are the correctness backstops — every stuck or poisoned job ends up somewhere a human owns.

Failure Scenarios#

1. Visibility Timeout Shorter Than the Job — Eight Copies of Every Report#

Context: A reporting service uses SQS with the default 30-second visibility timeout. Reports used to take ~10 s. A new enterprise customer's reports take 4–6 minutes. Workers don't extend visibility.

t=0        Enterprise customer schedules 2,000 monthly reports at 09:00
t=+30s     Visibility expires on in-progress messages; SQS redelivers to other workers
t=+2min    Each report now running on 3-5 workers simultaneously
t=+5min    Worker pool (200) fully consumed by duplicates; other tenants' jobs queue
t=+8min    Reporting DB replica CPU 98%; reports emailed 3-8 times each
t=+25min   On-call raises visibility to 15 min by config; duplicates stop after drain

Detection: jobs.duplicate_start (same job_id started > 1 time), sqs.ApproximateReceiveCount p99 > 2, worker.utilization pinned at 100% while jobs.completed_rate falls.

Blast radius: All tenants on the shared worker pool; customer-visible duplicate emails; reporting replica saturation.

Mitigation: Raise visibility timeout; purge duplicate in-flight work by checking a job_id lease before starting.

Prevention: (1) Heartbeat with ChangeMessageVisibility every 1/4 of the timeout; (2) lease + fencing on the job row so a second receive can't start the job; (3) dedup email sends on (job_id, recipient); (4) per-tenant concurrency caps so one tenant's large jobs can't consume the pool.

Owner: Reporting team owns the worker; the jobs platform team owns the SDK default that should have heartbeated automatically.

🎯 Staff Insight: Duration is a property of the workload, not the code — it changes when customers change. Any lease or timeout set as a constant will eventually be shorter than some job. Heartbeat-extended leases make duration irrelevant; fixed timeouts make it a ticking bomb.

2. The Nondeterministic Deploy — 40K Workflows Stuck Mid-Flight#

Context: An order workflow in a replay-based engine is modified to add a fraud check between charge and inventory reservation. The change is deployed without versioning. ~40K orders are in-flight, many waiting on 72-hour carrier-pickup timers.

t=0        Deploy rolls out new workflow code to all workers
t=+1min    Workers replay in-flight histories; new code schedules fraud_check where
           history recorded reserve_inventory -> nondeterminism error
t=+5min    Every in-flight workflow that wakes (signal, timer) gets stuck; new orders fine
t=+40min   Ops notices stuck-workflow count climbing; shipping confirmations stop
t=+1h      Rollback to old code; stuck workflows resume on replay
t=+1h30    New code re-deployed behind a version gate: old executions keep old path

Detection: workflow.task_failures{type="nondeterminism"} > 0 (should always be 0), workflow.stuck_count by workflow type, and business signal orders.shipped_rate dropping while orders.created_rate is flat.

Blast radius: Every in-flight execution of that workflow type — orders placed over the previous 3 days — while new orders looked healthy, masking the incident.

Mitigation: Roll back worker code; replay recovers without data loss because history is intact.

Prevention: (1) Version gates (if version >= 2: fraud_check()) for any change to step order; (2) CI replay tests that run new code against a sample of recorded production histories; (3) a deploy check that blocks if in-flight executions exist on an unsupported version.

Owner: Order team for the workflow code; workflow platform team for the replay-test tooling and deploy gate.

🎯 Staff Insight: Long-lived workflows turn every code change into a schema migration against in-flight data. The question at review isn't "does the new code work?" — it's "does the new code correctly continue an execution that started under the old code three days ago?"

3. The Compensation That Never Ran — Charged but Not Shipped#

Context: A choreographed saga: OrderPlaced → PaymentCaptured → InventoryReserved → ShipmentCreated. The inventory service fails permanently for a discontinued SKU and publishes InventoryFailed. The payment service's consumer for that event was removed during a refactor six weeks earlier.

Week 0       Refactor removes payment-service handler for InventoryFailed
Week 0-6     ~1,900 orders: charged, never reserved, never shipped, never refunded
Week 6       Chargeback rate for the merchant category doubles; finance escalates
Week 6+2d    Reconciliation query finds charged-but-unshipped orders older than 7 days
Week 6+5d    Manual refunds issued; saga moved to an orchestrator with explicit compensation

Detection (should have been): A reconciliation invariant — count(orders WHERE captured AND NOT shipped AND age > 72h AND NOT refunded) — alerting at > 0; saga.incomplete_age_p99 per flow.

Blast radius: ~1,900 customers charged for goods never shipped; chargeback fees and a payments-risk review.

Mitigation: Batch refunds with idempotency keys; customer outreach.

Prevention: Orchestrated saga where compensation is code in one place, not an event subscription someone can delete; end-to-end invariant checks run hourly; contract tests that every failure event has a registered handler.

Owner: Checkout team owns the orchestrated saga; finance ops owns the reconciliation invariant.

🧭 Principal Insight: Choreography distributes the happy path nicely and distributes the failure path dangerously — no one team owns "what happens when step 3 fails." For flows that move money or goods, the org should mandate orchestration with a named owner of the end-to-end state.

Operational Reality Matrix#

The Principal Lens#

Why L7 Sees This Problem Differently#

A Staff engineer designs one durable, idempotent, cancellable job system. A Principal engineer walks the org and finds fifteen: a jobs table in billing, a Redis-backed queue in growth, a cron-plus-status-column in data, Celery in ML, Step Functions in one AWS-heavy team, and an in-house "workflow framework" whose author left. Each has its own lease bugs, its own DLQ nobody watches, and its own answer to "where is this customer's request?" The L7 insight is that durable execution is infrastructure, like a database — it should be provided once, operated by a team that specializes in it, and consumed through an SDK that makes heartbeats, idempotency, and versioning the default rather than the exception.

The Org-Level Fault Line#

One durable-execution platform vs fit-for-purpose tools per team. A single platform (e.g., Temporal) gives consistent semantics, visibility across flows, and one on-call for the hard parts. But it concentrates risk — an engine outage pauses every workflow in the company — and it imposes a programming model (determinism, versioning) that some workloads don't need. The Principal position: one paved-road workflow engine for multi-step, cross-service, or long-waiting flows; one paved-road job queue SDK for single-step async work; batch/stream engines for bulk data. Three tools with clear boundaries beats one tool stretched over everything, and beats fifteen tools chosen by accident.

Cost Model#

Assumptions: $300K fully loaded FTE; managed pricing at list order of magnitude per action (Step Functions Standard ~$25 per million transitions; hosted Temporal pricing is per action in a similar order); persistence on Cassandra/Postgres for self-hosted. The biggest saving is not infra — it's retiring 10+ bespoke job systems and their incidents.

🧭 Principal Move: "I don't want a workflow engine because it's elegant. I want it because we currently have fifteen homemade ones, three of which caused Sev1s last year. The business case is headcount returned to product teams and incidents that stop happening."

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Long-Running and Asynchronous Work

Scope: Any operation that can exceed 10 seconds, spans more than one service, or waits on timers, humans, or external callbacks.

Mandatory requirements:

• Operations that can exceed 10 s MUST return 202 with a job resource; synchronous endpoints MUST NOT exceed the gateway timeout.
• Job creation MUST accept an idempotency key; every side-effecting step MUST be idempotent on a stable key.
• Workers MUST use heartbeat-extended leases with fencing tokens via the paved-road job SDK; fixed visibility timeouts are prohibited for jobs over 30 s.
• Jobs over 2 minutes MUST checkpoint at least every 60 s.
• Flows that move money or goods across services MUST use the orchestrated workflow engine with an explicit compensation for every compensable step and a documented pivot step.
• Workflow code changes MUST pass replay tests against recorded production histories; step-order changes MUST be version-gated.
• Every queue and DLQ MUST have an owning team and a depth alert.

Exceptions: Bulk data processing uses batch/stream platforms instead; exceptions approved by the workflow platform lead, reviewed every 12 months.

Success metrics: Bespoke job systems retired (target: 15 → 2 in 24 months); jobs.duplicate_start rate < 0.01%; zero nondeterminism incidents reaching production; saga.incomplete_age_p99 under SLA for every money flow.

What I'd Tell the VP#

We have about fifteen different homemade systems for running background work, and three of them caused major incidents last year — duplicate charges, stuck orders, and reports sent eight times. I want to replace them with one supported system for multi-step business processes and one standard library for simple background jobs, run by a small platform team. Product teams stop maintaining plumbing, and we get a single place to answer "where is this customer's order?" The cost is a platform team of four to five engineers and a vendor or infrastructure bill that is smaller than the engineering time we already spend on these systems.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "I'd use Temporal for this flow" — right for one flow, silent on the other fourteen job systems and on who runs the engine.
• "We'll add heartbeats and checkpoints" — correct mechanics, but no mechanism to make every team do it by default (an SDK, a standard, a lint).
• "Compensation handles failures" — without asking who owns compensations that span teams, or how the org detects the one that silently never ran.

In the Wild#

Uber: Cadence, and the Birth of Temporal#

Uber built Cadence, an open-source durable workflow engine, to run long-lived business processes — the kind that previously lived in ad-hoc combinations of queues, cron jobs, and state tables across many teams. Workflow code is written as ordinary sequential code; the engine persists an event history and replays it to rebuild state after failures. Cadence's creators later founded Temporal, which forked and evolved the project into what is now one of the most widely adopted durable-execution engines.

Staff insight: The origin story is the argument. A large engineering org found that every team was independently building fragile versions of the same thing — retries, timers, state persistence — and the fix was a platform with a programming model, not better code reviews. In an interview, citing Cadence/Temporal signals you understand why workflow engines exist, not just that they do.

Netflix: Conductor and the Orchestration Choice#

Netflix built and open-sourced Conductor (2016), a JSON-defined orchestration engine, to coordinate microservice workflows such as content ingestion and encoding pipelines. Netflix's public writing on it explicitly favored central orchestration over pure event choreography, citing visibility and control over complex flows. In late 2023 Netflix announced it would stop maintaining the open-source project; community and commercial forks continue it.

Staff insight: Two lessons in one. First, orchestration won inside a company famous for microservice autonomy because someone must be able to see the whole flow. Second, even a strong in-house engine became a maintenance burden its creator chose to hand off — the build-vs-buy question applies to workflow engines too.

AWS Step Functions: Workflows as a Managed Service#

AWS Step Functions (launched 2016) offers state-machine workflows defined in the Amazon States Language. Standard workflows can run for up to one year with exactly-once step semantics and full execution history; Express workflows run up to five minutes at much higher throughput with at-least-once semantics. Pricing for Standard is per state transition.

Staff insight: The Standard/Express split is the whole pattern in one product decision: long-lived, auditable, per-step-billed durability vs short, high-volume, cheap execution. Mentioning which one you'd pick — and that 10M items as individual Standard transitions is an expensive mistake — shows you think about cost shape, not just features.

Practice Drill#

Prompt: "Design 'Export my account data' for a SaaS product. Exports can be 50 GB and take up to 2 hours. Users should see progress, be able to cancel, and get a download link when done."

Prompt: "An order workflow has been 'processing' for 9 days for 3,000 customers. You're called in. What do you do?"

Staff Interview Application#

How to Introduce This Pattern#

"This operation takes longer than any request timeout and longer than a deploy cycle, so I'll take it out of the request path and focus on where its progress lives. The API returns a job ID; workers hold heartbeat-extended leases and checkpoint durably so any crash resumes, not restarts; every side effect is idempotent because delivery is at-least-once. If the flow spans services or waits on humans or timers, I'd put it in a workflow engine and define compensations for each step."

Lead with durable progress, not with the queue — that's the sentence that signals Staff.

When NOT to Use This Pattern#

• Work that reliably finishes in a few seconds. Keep it synchronous; a job resource and polling add latency and complexity for nothing.
• High-volume, per-record data transformation. Use Spark/Flink/batch jobs with their own checkpointing, not one workflow per record.
• Fire-and-forget best-effort work. Analytics pings or cache warmups can go on a queue without leases, checkpoints, or sagas — losing some is acceptable.
• A single team's low-volume job. A Postgres table with SKIP LOCKED beats adopting and operating a workflow platform.

Follow-Up Questions to Anticipate#

Evaluation Rubric#

Strong hire signals: asks where progress lives before drawing a queue; sets lease/heartbeat/checkpoint numbers; designs cancellation and compensation unprompted; knows the engine's constraints (determinism, history limits).

Lean no-hire signals: raises the HTTP timeout; assumes exactly-once execution; "rollback" across services; no DLQ owner.

Common false positive: Fluency in a specific engine's API ≠ long-running-process judgment. A candidate can write perfect Temporal code and still put a 100K-iteration loop in one workflow, skip versioning, and choreograph a money flow nobody owns.

Capacity Planning Quick Reference#

Pitfalls checklist:

• Is anything over ~10 s still synchronous?
• Where does progress live if the worker dies right now?
• Is the lease heartbeat-extended and fenced, or a fixed timeout?
• Is every side effect idempotent on a stable key?
• Does every wait have a timeout and an escalation path?
• Does every compensable step have a compensation, and are irreversible steps after the pivot?
• How do in-flight executions survive a code change?
• Who owns the DLQ, and what alerts on it?