Kishore Adimulam
Kishore Adimulam

Learnings from building Durable Workflows

Kishore Adimulam · July 4, 2026 · 3 min read

What is durability?

The concept of “durable workflows” is encountered repeatedly when dealing with LLM systems, but due to the constant evolving of systems in this space, has many varying definitions. The simplest I could find was from Temporal’s post on the topic, where they define it as crash-proof execution.

When I was first building systems using LLMs, my thought was to naively bolt it on to any existing backend. Ex. The backend would expose an API, and inside that API, a chain of multiple LLM calls would exist.

Frontend calls backend with a payloadBackend passes payload into an LLMwith system prompt + user prompt + payloadLLM output is deterministically processedResult passed into a new LLM callOutput returned to the frontend

While the theory was sound at the time, numerous problems would quickly arise, including:

  • Frontend API call timing out, and breaking connection with backend after 60ish seconds
  • Backend would crash somewhere in one of the LLM/deterministic steps, pin-pointing the exact location of failure was difficult

Solution attempt #1 - Server Background tasks

The initial idea was to outsource the actual logic to a server background task. Thus allowing the server to immediately reply with a Job ID, and sending server sent events to frontend. However, was extremely fragile as the entire task ran in server memory. Any crash on the server, and say goodbye to the whole process. We’d have to re-initiate the workflow from the beginning.

Frontend calls the backendBackend instantly replies with a Job IDBackground task runs the LLM chainentirely in the server's memorySSE progressServer crashes → task is gonerestart the whole workflow from the beginning

Solution attempt #2 - Redis streams

Using a Redis stream along with a server background task, helped greatly in terms of visibility. Every major event I wanted to record could be pushed to the stream, and on event of failure, all events were persisted and easy to review. However, the same problem of server crashes wiping out the whole process existed. We’d need to re-initiate the workflow from the top.

Frontend calls the backendBackend instantly replies with a Job IDBackground task runs the LLM chainentirely in the server's memorySSE events losteventsRedis streamevery event persistedreads eventsServer crashes → task is goneevents survive & are easy to review,but the workflow restarts from the top

Solution attempt #3 - Celery tasks

Outsourcing the job to a Celery worker once again helped in sense of not taking down the entire backend server if something crashed, and we got retries for free. However, we still had to retry the entire task from the beginning if something failed. Furthermore, the management overhead of celery was tedious, as we had to maintain task queues, workers, and closely monitor everything to see what failed. It was essentially the entire job of building a durable workflow engine from scratch :(

Frontend calls the backendBackend replies with a Job IDand enqueues the taskTask queue (broker)task waits for a free workerCelery worker runs the LLM chainin a separate worker processretries for freeA step fails → full retrythe backend stays up on a crash,but queues & workers need babysitting

Trying out a Dedicated Durable Workflow engine: Trigger.dev

I finally tried to setup a workflow using a dedicated durable engine: Trigger.dev. The promise of this system was to resolve the major issue of the previous attempts: Allow the checkpointing of failures within the system, and not have to re-try each step.

Frontend triggers the taskDurable engine runs the workflowas a series of checkpointed stepsStep 1 · fetch dataStep 2 · LLM callStep 3 · save resultA step fails → resume from itcompleted steps stay checkpointed,no replay from the topresume at failed step

For the full walkthrough of building it, see Building an Email Sentiment Classifier.

Looking back at the three failed attempts, each one was me rebuilding a part of a durable workflow engine by hand. Background tasks gave me async execution, Redis streams gave me visibility, Celery gave me retries. A dedicated engine gave all three at once, plus checkpointing, which I am yet to explore in more detail with the new tools.

The best part was how much code I could delete. No backend server, database, complex redis logging, or separate workers. The whole workflow is just a few task definitions. Although this isn’t a one size fits all, I am likely going to use a durable engine for any task that runs for an extended period of time.