When your training job outlives its compute: Orchestrating ML workflows on Modal

The MLOps team builds the tools that Epidemic Sound’s ML engineers rely on. Now, they’re open-sourcing the feature those engineers use every day: a small workflow orchestrator that turns a long, fragile sequence of ML steps into something repeatable, resumable, and observable on Modal.

They describe it as “the unglamorous machinery that has to run reliably, for hours, before anyone gets to argue about whether a soundtrack is any good.”

The tool lives at github.com/epidemicsound/modal-workflows.

In their own words, this is why the MLOPs team built it, what it does, and the one constraint that shaped the whole design.

The 24-hour wall

Most of our ML work is not a single step. A typical pipeline loads a dataset, preprocesses it, trains a model, evaluates it across a few splits, and writes the results somewhere durable. On a good day, that’s a clean chain of functions. But good days aren’t realistic. On a real day, it retries: a GPU that takes a while to schedule, an embedding job that fans out over thousands of items, and a training run that simply takes a long time.

We run this on Modal, which we like a great deal. But Modal functions have a hard ceiling: A single function can run for at most 24 hours. For a lot of serverless work, that’s an eternity. For training a model on a large catalog of music, it’s just not enough.

We have jobs that comfortably need more wall-clock time than that, and the moment a function hits the time limit, Modal kills it. No appeal. And the 24-hour cap isn’t even the only way a function can die mid-run. Modal can also preempt it earlier. Either way, the job stops before the work is done.

The real problem was never “How do we chain a few functions together?”. Plenty of tools do that. The problem was: “How do we run a multi-hour pipeline that survives being killed and picks up exactly where it left off?”, without every ML engineer hand-rolling their own state machine.

What we did before: Reserved VMs

For long training jobs, the 24-hour cap meant serverless was simply off the table. As a result, our ML engineers didn’t run them on Modal at all.

The fallback was a long-lived virtual machine. A GPU box provisioned and reserved for the duration of the work, sometimes months at a time, and billed for every one of those hours whether it was actually busy or not.

That’s a frustrating place to be. You pay for a reserved machine around the clock, even when it idly waits for new training jobs. Scaling up means reserving another box; scaling down means remembering to tear one away. The elastic, pay-for-what-you-use model we relied on everywhere else stopped applying exactly where the compute was most expensive.

In principle, you could make a long job fit on serverless. You’d hand-roll a self-continuing orchestrator that spawns its own work, persists progress somewhere durable like a Modal Dict, and re-spawns itself just before the timeout to carry on. Roughly:

@app.function(timeout=24 * 3600)  # Modal's hard ceiling
def orchestrate(orchestration_id: str | None = None):
    if orchestration_id is None:
        orchestration_id = str(uuid.uuid4())
    state_dict = modal.Dict.from_name(f"run-{orchestration_id}", create_if_missing=True)
    state = state_dict.get("state", {"a_id": None, "b_id": None, ...})
    # Spawn work only if we haven't already, on a previous incarnation
    if state["a_id"] is None:
        state["a_id"] = A.spawn().object_id
        state_dict["state"] = state
    # ... poll for results within our time budget ...
    # ... if we're about to time out, spawn ourselves again and hand off ...

Nobody wanted that in their training loop, and reasonably so. The state-management dict, the object_id book-keeping, the timeout arithmetic, the hand-off…none of it was about the model. It was pure plumbing. But it was the plumbing most likely to leak at 3 a.m., and getting it subtly wrong meant silently re-running work or losing it. So long jobs stayed on the reserved VMs, and serverless stayed for everything short.

One piece of insight turned this recurring problem into the modal-workflows solution: That self-continuing machine is the same every time.

Track which steps have completed, persist their results, and on restart, replay the completed ones instantly and resume from the first one that hasn’t run. If we wrote that once, correctly, nobody would have to write it again. Long jobs could finally move onto the same elastic compute as everything else.

The shape of the abstraction

We landed on two decorators and a context object. The goal was that the workflow code should read like the diagram you’d draw on a whiteboard, and nothing else.

You mark the individual steps with @workflow_function. These are just Modal functions; anything you can pass to a Modal function (a GPU, secrets, volumes) you can pass here:

from modal_workflows.workflows import workflow_function
@workflow_function(app, gpu="A10G")
def train_model(train_df, val_df) -> Model:
    ...

Then you compose them into a workflow with @workflow. The first argument to a workflow function is always a WorkflowContext (the decorator injects it for you), and your workflow body does nothing but describe the graph, offloading the actual work through the context:

from modal_workflows.workflows import workflow, WorkflowContext
@workflow(app)
def train_pipeline(ctx: WorkflowContext) -> None:
    train_df, val_df = ctx.step(load_dataset, step_id="load-dataset", path="...")
    model = ctx.step(train_model, step_id="train", train_df=train_df, val_df=val_df)
    ctx.parallel(
        [(evaluate, model, val_df), (evaluate, model, train_df)],
        step_id="evaluate",
    )
    ctx.finish()

load_dataset and evaluate are @workflow_function-decorated too, just like train_model above. step_id is the unsung hero of the whole design, though. It’s a stable name for a step, and it’s the cache key that makes resumption work. When a workflow restarts, modal_workflows doesn’t re-run load_dataset; it already has the result keyed under that ID, handing it back instantly and moving on to the first step it never finished. Name your steps once, and resumption comes for free.

The context gives you four ways to run work, which you pick by the shape of your dependency graph:

1. ctx.step(...): Run one job, wait for the result. For when the next line needs it.
2. ctx.spawn(...): Start a job, get a handle back immediately, collect with .get() later. For independent work that runs in parallel.
3. ctx.map(fn, items, ...): Run the same function over a list, in parallel. For fan-out (embed every track, score every split).
4. ctx.parallel([...]): Run different functions concurrently in one batch.

None of these mention timeouts, state, or restarts. That’s the point.

The part we’re most proud of: Surviving the wall

Here’s what happens when a workflow approaches the 24-hour limit, with none of it in your code.

Each workflow has a self_managed_timeout: 23 hours by default, deliberately under Modal’s ceiling. modal_workflows sets Modal’s own timeout to the full 24 hours and watches the clock itself. When the workflow reaches the self-managed limit, instead of waiting to be killed, it gracefully restarts, spawning a fresh instance of itself and handing it over. The new instance reads the persisted step results, replays everything already completed (instantly, since it’s just reading cached values), and resumes from the first unfinished step.

From the engineer’s side, a long pipeline that takes 30 hours of wall-clock just...finishes. Under the hood it may have lived two separate lives, each under the 24-hour cap, stitched together by the step cache — or even more than two lives, if a preemption cut one of them short. The same restart-from-cache machinery handles either case. The only place this leaks into your code is if you .spawn() the workflow yourself and want to follow the chain across a restart:

result = train_pipeline.spawn()
output = result.get()
if isinstance(output, WorkflowRestartedResult):
    output = output.to_modal_function().get()

When a workflow finishes for real, call ctx.finish() to clear the saved state, so the next run starts clean instead of inheriting stale results. We learned to make this explicit the way you’d expect, wondering why a “fresh” run suspiciously had its first three steps already done.

Porting Lightning code: Resumption at the checkpoint level

A lot of our training code was already written in PyTorch Lightning, which has its own checkpointing story. To move it onto modal-workflows with barely any changes, we reused the exact same idea one level down, at the checkpoint instead of the step.

A parallel @training decorator and a @TrainingCheckpointRestartCallback do the same timeout-and-handoff dance around @trainer.fit(). As the self-managed timeout approaches, the callback checkpoints, hands off to a fresh instance, and training resumes from the last checkpoint rather than from scratch. Same mechanism, even simpler to adopt, and existing Lightning code hardly moves.

The invisible features that matter most

Resumption is the headline, but the things that make a tool livable are the ones you don’t even notice:

• Artifacts that outlive restarts. modal_workflows mounts a workflow_artifacts volume automatically. Write your checkpoints, models, and logs to ctx.artifacts_run_path, call ctx.commit_artifacts_volume(), and they persist across restarts and outlive the app that produced them. No bucket-wrangling for intermediate state.
• Alerts you don’t have to wire up. Give a workflow a SLACK_WEBHOOK_URL (through a Modal Secret) to alert it on any unhandled exception by default: function name, error, the function-call ID, and a direct link to the Modal dashboard. That’s the whole setup; there's no alerting code in your pipeline. A long job failing at hour six should find you, not the other way around. There’s an @alert_on_error decorator for standalone functions and an alert(message, app_id=...) helper for “validation accuracy just fell off a cliff” messages, too.
• Observability for free. Because every step is a real Modal function with a stable ID, the Modal dashboard already shows you the logs, status, and history of each one. The orchestrator didn’t have to reinvent a UI; it just had to name things well.

Why we’re open-sourcing it

This is a reliable, foundational tool that lets our ML engineers devote their time and talent to genuinely hard, interesting problems. The 24-hour wall isn’t unique to us; anyone with long-running ML jobs and workflows on serverless compute hits it. If we’ve already built the solution, keeping it to ourselves helps no one.

So we’re putting it out there. If you run ML workloads on Modal and have ever hand-rolled a self-restarting orchestrator with a dict and a prayer, we think you’ll find this familiar…and thankfully, with much less code than you’re used to.

Try the Modal workflow

It installs straight from the GitHub repository rather than from PyPI. Pin a release tag, as the project is pre-1.0; minor versions may still change the API schema.

pip install "modal-workflows @ git+https://github.com/epidemicsound/modal-workflows@v0.1.1"
# with the PyTorch Lightning training callback
pip install "modal-workflows[training]@git+https://github.com/epidemicsound/modal-workflows@v0.1.1"

from modal_workflows.workflows import workflow, workflow_function, WorkflowContext

The source and documentation live at github.com/epidemicsound/modal-workflows. It’s Apache-2.0 licensed. Issues and pull requests are welcome.

→ Try modal workflows

→ Read more research