The short version: one pod, one event loop, one translation at a time. We replaced a network call with a blocking, CPU-bound model call inside an async def activity. From then on, each Python worker could run only one translation at a time, no matter how many it had accepted, and the translations waiting their turn couldn’t even tell Temporal they were alive.

As before, the domain is anonymized, and it’s snacks again. Our app collects tasting notes from all over the world, and to show them in one feed, we translate them. The code is simplified. The settings and timeouts are the real ones.

Act I: two branches that look the same

Translation could go one of two ways, depending on how a client was configured:

if (translationParameters.kind === "languageModel") {
  // TypeScript activity: a direct call to an LLM
  return translateTastingNoteText({
    tastingNote: { text },
    outputLanguages: translationParameters.outputLanguages,
    model: translationParameters.languageModel,
  });
} else {
  // Python activity: our own translation model
  return translateWithOwnModel({
    text,
    outputLanguages: translationParameters.outputLanguages,
    ...(translationParameters.selectedTranslationModel && {
      selectedTranslationModel: translationParameters.selectedTranslationModel,
    }),
  });
}

Read it as the workflow sees it: same inputs, same output, two interchangeable implementations. Nothing here tells you that the first branch is an HTTPS request and the second one, most of the time, is a matrix multiplication.

The TypeScript branch was boring in the best way. The activity sent a request to an LLM and awaited the response. While it waited, the Node.js event loop was free to run other activities. In Grafana, activity execution time was low and so was schedule-to-start latency. The worker spent most of its life waiting on sockets.

At our scale, though, paying per token added up. So the company decided to translate with its own model, served by a new Python worker (one per Kubernetes pod) on a new task queue. Clients were moved to it gradually, which is why, for a while, only some of them were affected.

Act II: one pod, one lane

The Python activity did more than the TypeScript one. It detected the source language with fastText, translated locally with a Marian model if the language pair was supported, and otherwise fell back to an LLM over HTTPS. While it worked, a keepalive sent a heartbeat every 3 seconds. Simplified:

@activity.defn
async def translate_text(self, args: dict[str, Any]) -> dict[str, Any]:
    stop_heartbeat = self._start_heartbeat_keepalive()  # heartbeat every 3 s
    try:
        source = self.factory.identify_language(args["text"])  # CPU
        result = self.factory.translate(                         # CPU, or blocking HTTP
            args["text"], source, args["target_language"]
        )
    finally:
        stop_heartbeat()
    return {"output_text": result}

And the worker setting that mattered:

temporal_max_concurrent_activities = 50

The async def decides everything here. In the Temporal Python SDK, an async def activity runs on the worker’s asyncio event loop. A plain def activity runs on a thread pool, which you have to give the worker explicitly. The SDK README is explicit about which to prefer: threaded activities are the initial recommendation, and there is a boxed warning not to block the thread in an async def function.

The configuration also had temporal_activity_max_threads = 10, which suggests someone expected threads to be involved. With only async def activities, they weren’t: no thread pool ran any of this code, and the setting had no effect.

Look at the body again: there isn’t a single await in it. Language detection, local inference and even the LLM fallback were ordinary blocking calls. An async def with no await doesn’t give the event loop back until it returns.

So while one translation was running, every other translation on that pod simply stood still. The worker accepted up to 50 of them and ran them strictly one after another. A pod configured for 50 concurrent activities behaved like a pod configured for one.

Act III: timeouts on a three-minute activity

That alone would have made translations slow. What made them fail was this, on the workflow side:

proxyActivities<TranslationPythonActivities>({
  taskQueue: TASK_QUEUES.TRANSLATION,
  startToCloseTimeout: "3 minutes",
  heartbeatTimeout: "10 seconds",
  retry: { initialInterval: "1 second", maximumAttempts: 2 },
});

Three minutes for a translation looks generous, and the activity sent a heartbeat every 3 seconds, well inside 10. So why did so many attempts die on the heartbeat timeout?

Because of two details that are easy to miss:

• The timeout clock starts when the attempt starts, and an attempt starts when the worker receives the task, not when your code gets to run. The translations waiting their turn on a pod weren’t waiting in Temporal’s queue. As far as Temporal was concerned, they had started.
• Heartbeats need the event loop too. A translation waiting its turn hadn’t started its keepalive yet, so it sent nothing. The one being translated had a keepalive, but its heartbeats couldn’t be sent while the loop was blocked. This doesn’t depend on how the keepalive is built: in the Python SDK, every heartbeat is handed to the worker’s event loop to be sent, including one called from an activity’s thread.

So a translation timed out if 10 seconds passed without a heartbeat reaching the server. For one waiting in line, that meant its turn didn’t come within 10 seconds of being accepted. With a few translations ahead of it, that was most of them.

It got worse from there. A translation that had already timed out while waiting still ran in full when its turn came: nothing had told the worker, and even if something had, there was no await at which to stop it. So part of each pod’s single lane went into results the server would reject.

Meanwhile, Temporal retried the timed-out attempts. All of the activity’s own errors were marked non-retryable, so maximumAttempts: 2 came into play only for timeouts, and there were plenty of those. A retry could just as well land on another pod that was equally stuck.

With the TypeScript activity, none of this showed. An LLM call yields at its await, so many requests can wait in parallel on one event loop, and a short heartbeat timeout is harmless. The Python activity kept the same workflow call but stopped yielding, and that removed the conditions that had made the timeout harmless.

Act IV: why 60 replicas made it worse

When a queue backs up, the natural reaction is to add workers. The deployment already scaled with KEDA on the task-queue backlog:

minReplicaCount: 5
maxReplicaCount: 60
pollingInterval: 3
advanced:
  horizontalPodAutoscalerConfig:
    behavior:
      scaleUp:
        stabilizationWindowSeconds: 0
        policies:
          - type: Percent
            value: 100
            periodSeconds: 15
      scaleDown:
        stabilizationWindowSeconds: 3600
triggers:
  - type: temporal
    metadata:
      taskQueue: "translation"
      queueTypes: "activity"
      targetQueueSize: "1"

The backlog on the translation queue grew, and with it schedule-to-start latency, so the autoscaler did exactly what it was told. targetQueueSize: "1" asks for roughly one pod per waiting task, the replica count can double every 15 seconds, and the one-hour scale-down window keeps the pods once they exist. We went to the maximum quickly and stayed there.

Every new pod did add a lane. But it also brought the same problem with it:

1. The pod accepts up to 50 tasks and starts 50 clocks.
2. It works through them one at a time, and most of them time out, some of them while still taking up the lane.
3. Each timeout puts a retry back on the queue, alongside the new work.
4. The backlog grows, KEDA adds pods, and we go back to step 1.

Every pod also loaded its own copy of the models, so 60 replicas meant 60 copies in memory. The cluster paid for all that capacity and the queue barely moved, because much of it went into work that had already timed out. The number of pods wasn’t the problem. Each pod accepting far more work than it could start was.

Act V: threads, and what they didn’t change

The first change was the obvious one: turn async def into def, so each activity runs on a thread from the pool and the event loop stays free.

Threads come with their own limitation. The model and tokenizer aren’t safe to use from several threads at once, so local inference sits behind a lock. The lock covers only tokenization, model.generate() and model loading. Language detection and the LLM fallback run outside it.

class MarianTranslator:
    def __init__(self) -> None:
        self._lock = threading.RLock()

    def translate(self, text: str) -> str:
        with self._lock:
            ...  # tokenize, model.generate(), decode

The second change was making sure every task the worker accepts gets a thread right away. Switching to def and keeping the old configuration would have meant 50 concurrent activities on 10 threads, with 40 tasks sitting in the executor’s queue. They wouldn’t have started their keepalive, so they’d time out exactly as before. The SDK logs a warning about this, but a warning is easy to miss, so I made the worker refuse to start instead:

concurrency = settings.temporal_max_concurrent_activities
pool_size = settings.temporal_activity_max_threads
has_sync_activity = any(not asyncio.iscoroutinefunction(fn) for fn in activities)
if has_sync_activity and pool_size < concurrency:
    raise ValueError(
        f"temporal_activity_max_threads ({pool_size}) must be >= "
        f"temporal_max_concurrent_activities ({concurrency}) when running "
        "synchronous activities; otherwise activities queue without a worker "
        "thread and miss their heartbeat timeout."
    )

worker = Worker(
    client,
    task_queue=settings.temporal_task_queue,
    activities=activities,
    activity_executor=ThreadPoolExecutor(max_workers=pool_size),
    max_concurrent_activities=concurrency,
)

I set both values to 10: 10 threads and 10 concurrent activities per pod.

Here’s what that changed:

• Every accepted task starts heartbeating immediately. It has its own thread, so its keepalive runs from the first second, even while it waits for the inference lock. Waiting no longer costs a heartbeat timeout. The limit for a waiting task is now the 3-minute startToCloseTimeout.
• The LLM fallback no longer waits behind inference. While one thread holds the lock, others detect languages and wait on LLM responses in parallel.
• Each pod accepts only what it can start. 10 tasks instead of 50.

And what it didn’t change: local inference still runs one at a time per pod. Threads don’t add inference capacity; they only make waiting safe. If translations needing the local model arrive faster than the pods can run them, they still queue up, now on the lock or in Temporal’s queue instead of behind a blocked event loop.

The keepalive has a cost worth knowing about. It proves that the thread is alive, not that the translation is making progress. If model.generate() ever hangs, the keepalive will keep heartbeating until the 3-minute limit. And while threads fixed the cancellation problem from Act III in general (the SDK can raise a cancellation inside an activity’s thread), it still can’t interrupt a thread stuck inside a native call like inference.

What I would do next

None of the following is implemented, so treat it as my reading of the problem rather than a proven fix.

• Size concurrency for the lock, not the thread pool. If one inference takes t seconds, a pod finishes at most 1/t local translations per second, however many threads it has. That’s Little’s law from the previous post again. Tasks waiting on one pod’s lock can’t be picked up by an idle pod, so accepting more than the lock can serve soon only moves the queue to the wrong place.
• Split the two execution profiles. Language detection plus the LLM fallback is mostly I/O and can run with high concurrency. Local inference is CPU-bound and serialized. Two activities on two task queues, with the workflow choosing between them, would let each queue have its own worker settings and its own scaling. Today, one setting has to fit both.
• Scale on something the pods can actually drain. Every pod adds one inference lane and one more copy of the models in memory. A target of one pod per waiting task buys mostly memory.
• Heartbeat with progress, not just a pulse. Including how far the translation got in the heartbeat details would at least make a stuck attempt visible.

Why not keep async def and offload the work?

A fair question for Python users: why not keep the activity async and wrap the blocking calls in await asyncio.to_thread(...)?

You can, and the event loop would stay free. But you then have two concurrency limits, max_concurrent_activities and the size of whichever executor you offload to, and they must agree for the same reason as above. You also have to find and wrap every blocking call, and the next one someone adds will freeze the loop again. A plain def activity with an explicit activity_executor puts both limits in one place, in the worker configuration, where the next person will look. It also matches what the SDK recommends as the default.

Takeaways

1. An activity’s execution profile is part of its contract. The workflow call and the types stayed the same. Timeouts, worker concurrency and autoscaling were set up as if it were still a network call.
2. A blocking call in an async def activity stops the whole worker. Not just that activity: every activity on that pod waits, whatever max_concurrent_activities says.
3. “Started” means handed to a worker, not running. Timers run from the moment the worker receives a task, including the time it waits inside the pod.
4. A task can only heartbeat once it is running. On a blocked event loop or without a free thread, it can’t. For synchronous activities that heartbeat from their own thread, that means the pool must have at least as many threads as concurrent activities.
5. A lock defines the real capacity. Threads make waiting safe; they don’t make serialized work parallel.
6. Replicas can’t fix a bottleneck inside each pod. With a backlog-based scaler and retries, they can turn it into a feedback loop.

How it ended

Moving the activity off the event loop, and running it with 10 threads and 10 concurrent activities per pod instead of 50, made the timeouts largely go away. It didn’t solve the underlying problem, though: for translations that need the local model, the capacity of the whole system is still the number of pods divided by the time of one inference.

The practical lesson: when you swap the implementation behind an activity, check its execution profile, not just its signature. Ask where it spends its time, what that does to the worker it runs on, and whether the timeouts, concurrency and scaling settings still describe it.

The short version: a started workflow is not a finished one. Our importer waited for the first and ignored the second.

As before, the domain is anonymized, and it’s snacks again. Instead of the real data, we import a tasting archive: a large export from a snack-tasting app, where each tasting note can carry attachments such as photos and scanned wrappers. The code is simplified, but the shape of the problem and the numbers are real. The names are not.

Act I: one workflow per tasting note

The design was simple, and for small archives it worked. An import activity reads the archive and starts a tastingNoteWorkflow for every note. If a note has attachments, the note workflow starts a child workflow per attachment: one workflow type for photos, another for wrapper scans.

Each workflow type has its own task queue and its own workers. Some activities are shared, though. tagFlavors, for example, is called both from note workflows and from photo workflows, and all of its executions run on one dedicated activity task queue.

Everybody knew that one workflow per note was not the most efficient design. It was the easiest one, and while the archives were small, nobody felt the cost.

The importer looked roughly like this (simplified):

const BATCH_SIZE = 200;

export async function importTastingArchiveActivity({ archiveId, archiveUrl }: Args) {
  const resumeFrom = parseHeartbeat(activityInfo().heartbeatDetails)?.batchIndex ?? 0;
  const notes = await readWholeArchive(archiveUrl); // everything in memory
  using client = await getWorkflowClient();

  let batchIndex = resumeFrom;
  for (const batch of chunk(notes, BATCH_SIZE).slice(resumeFrom)) {
    heartbeat({ batchIndex });
    await Promise.all(
      batch.map((note) =>
        client.workflow.start(tastingNoteWorkflow, {
          workflowId: `tasting-note-${note.id}`,
          taskQueue: TASK_QUEUES.TASTING_NOTES,
          searchAttributes: { ArchiveId: [archiveId] },
          args: [{ note }],
          workflowIdConflictPolicy: "USE_EXISTING",
        }),
      ),
    );
    batchIndex++;
  }
}

Look at what the loop waits for. client.workflow.start resolves when the Temporal frontend has accepted the start request: the workflow exists and its first workflow task is scheduled. Nothing has run yet.

So the batching doesn’t throttle the system. Send 200 starts, wait for 200 acknowledgements, send the next 200. The whole archive is handed to Temporal as fast as the server can say “OK”.

Act II: the freeze

Then the archives got big. The largest one I processed had around 200,000 tasting notes, many of them with attachments. I have seen an export with over 400,000, but I never ran anything that large through this pipeline.

What we saw:

• Temporal UI showed an enormous number of running workflows, and almost none of them made progress.
• Some workflows waited hours for their first workflow task. Not an activity: the very first step of the workflow code.
• The PostgreSQL database behind Temporal (where workflow histories live), with 5 CPUs in Kubernetes, sat at its CPU limit.
• Processing a single archive took up to about 15 hours.

Watching it live was strange. A batch of 200 workflows appeared in Temporal UI, and some of them started making progress. A fraction of a second later came the next 200, and a smaller share of those got going. Then the next 200, with an even smaller share. And so on. Things began to unclog, slowly, only once the importer stopped scheduling new workflows.

From the outside it looked like Temporal had frozen. It hadn’t. It was doing exactly what we had asked it to do.

Act III: firefighting with synchronous_commit

The database was the visible bottleneck, so I went there first. Temporal persistence runs a lot of small transactions. By default, PostgreSQL waits for each commit’s WAL record to be flushed to disk before acknowledging it. Setting synchronous_commit = off removes that wait, and it helped a little.

It is a trade-off, not a fix. With asynchronous commit, a database crash can lose the last fraction of a second of transactions that were already reported as committed. The database stays consistent, it just forgets the most recent writes. For Temporal, that means the server may have acknowledged a state change (a started workflow, a completed task) that is gone after recovery. We accepted that risk consciously. I wouldn’t recommend it as a default.

More importantly, a faster database treats the symptom. The cause was in our code.

Act IV: accepted is not done

Here is my best explanation of what happened. It fits everything we observed, but I didn’t have the server metrics to prove every step, so treat it as a model rather than a post-mortem.

Every start costs the database work. Accepting a start means persisting the execution, its first history events and a task that hands the first workflow task over to matching. Our importer produced these writes as fast as the server would accept them, so the database was busy mostly with new starts.

Workflow tasks pile up. Kubernetes scaled the note workers out to eight instances, and they still couldn’t keep up, so a backlog built up. The first workflow tasks of hundreds of thousands of new workflows filled the queue, adding latency for every workflow that had already been started. For comparison, the bounded version described below ran comfortably on two workers.

The real work has to queue too. When an early note workflow finally ran, it called tagFlavors and started child workflows for its attachments. Separate task queues per workflow type helped a little: a photo workflow didn’t wait behind the first tasks of note workflows. The shared activity queue didn’t help at all. A tagFlavors call from a photo workflow had to compete with the tagFlavors calls of nearly every note in the archive.

The cluster itself was configured in a standard way (512 history shards, the default in the Helm chart version we used), so the problem wasn’t in Temporal’s configuration.

Act V: wait for the result, not the acceptance

The fix changed one thing: what the importer waits for. Instead of starting everything, it keeps a sliding window of N notes in flight and starts the next one only when one of them finishes. Finishing is what handle.result() reports, and for a note it covers the whole thing, including the child workflows of its attachments, because a note workflow waits for its children before it completes.

The window itself is a few lines:

function createWindow(size: number) {
  const inFlight = new Set<Promise<void>>();
  let failures = 0;

  return {
    async acquire() {
      while (inFlight.size >= size) {
        await Promise.race(inFlight);
      }
    },
    track(handle: WorkflowHandle) {
      const done = handle.result().then(
        () => undefined,
        () => { failures++; },
      );
      inFlight.add(done);
      void done.finally(() => inFlight.delete(done));
    },
    drain: () => Promise.allSettled(inFlight),
    failures: () => failures,
  };
}

And the loop:

const slots = createWindow(WINDOW_SIZE);

for (let i = startFrom; i < notes.length; i++) {
  if (signal.aborted) throw new CancelledFailure("import cancelled");

  await slots.acquire(); // wait for a free slot first...
  const handle = await client.workflow.start(tastingNoteWorkflow, { /* as before */ }); // ...then start
  slots.track(handle);

  lastStartedIndex = i;
  heartbeat({ lastStartedIndex });
}

await slots.drain();
log.info("import finished", { failedNotes: slots.failures() });

There is also a timer that heartbeats every 5 minutes, because with a full window the loop can sit in acquire() for a long time.

Notes that fail are logged and counted, but they don’t fail the import. Each note has its own workflow history and can be retried on its own.

The result: the freeze disappeared immediately, and a similar archive was processed in about 7 hours instead of about 15.

And that was with synchronous_commit back on. For the bounded version we restored the default, so the database was back to full durability, and the improvement came from the code change alone.

How big should the window be?

I started with a window of 50, a number I pulled out of a hat. It turned out to be too small, because many note workflows finish in a few seconds.

Why so fast? Tasting notes vary wildly. One note is just “Yum!”, another is a detailed review with ingredients and a photo of the wrapper. In our pipeline, tagFlavors feeds two follow-up activities that run in parallel. For a note like “Yum!” it finds nothing to tag, so the workflow skips both follow-ups and ends almost immediately, freeing its slot in the window within seconds.

Little’s law shows how much the window size matters:

throughput = items in flight / average time in flight

We doubled it to 100.

Read backwards, the numbers hang together. 200,000 notes in about 7 hours is roughly 8 notes per second. With 100 in flight, that means about 12 seconds per note on average, attachments included. As a rough upper-bound estimate, assuming the same average time in flight, a window of 50 would have taken around 14 hours, about as long as the unbounded version. In practice it would probably have been somewhat less, because a smaller window also means less load and shorter waits.

I also ran short experiments with windows of 120, 150 and 200, and none of them looked bad. 150 looked especially good when the workflows were very short, because many notes took the early exit described above. Short workflows put little load on the rest of the system, so more of them can be in flight before anything downstream saturates. Overall, 100 and 150 gave very similar results. The best value depends on the data anyway, so we stayed with the more conservative 100.

The catch is that time in flight isn’t constant. Make the window large enough and the workers, the shared activity queues and the database saturate, each note takes longer, and you drift back towards the freeze. The right size depends on what sits downstream. That’s why we made it an environment variable.

Dark corners

The window fixed the freeze. But an activity that orchestrates hundreds of thousands of workflows for hours has a few corners worth knowing about.

Rebuilding the window after a retry

Our heartbeat stored only the index of the last started note. After a retry (a deploy, a crashed pod), the activity rebuilt the window as “the last N indices” and re-attached to those workflows.

That assumes the window always holds consecutive indices. It doesn’t. Take a window of 3. Note 0 is slow, while notes 1, 2, 3 and onwards finish quickly. By index 10 the window holds {0, 9, 10}. Rebuilt as [8, 9, 10], note 0 drops off the books:

• the importer thinks it has a free slot, so for a while more than N notes run;
• nobody waits for note 0, so drain() can return, and the import can report success, while note 0 is still running. If it fails, nobody counts it.

Nothing is lost, note 0 finishes on its own. We just stop watching it.

Storing the IDs of the in-flight workflows in the heartbeat fixes this, but with a window of 100 it felt clumsy. The alternative, which we never got around to implementing, is to ask Temporal. Every note workflow carries a search attribute with the archive ID, so on resume:

const running = client.workflow.list({
  query: `ArchiveId = "${archiveId}" AND WorkflowType = "tastingNoteWorkflow" AND ExecutionStatus = "Running"`,
});
for await (const wf of running) {
  slots.track(client.workflow.getHandle(wf.workflowId));
}
// then continue the main loop from lastStartedIndex + 1

The query doesn’t care about indices. It returns 0, 9 and 10 because those are the ones still running. The heartbeat stays tiny, and if more than N are running after a crash, the main loop’s acquire() simply waits until enough of them finish.

Visibility is eventually consistent, so the query can be off in two ways. A workflow that has just finished but still shows as running is harmless: its result resolves immediately. A workflow started just before the crash may not be indexed yet. One way to make that safe enough in practice would be to also re-attach the last few indices.

One more thing about the resume index itself. activityInfo().heartbeatDetails returns the last heartbeat that reached the server, not the last one the code called. The SDK throttles heartbeats, so when a pod dies, the most recent details may never be sent. The resumed index can lag behind real progress, and a few notes get started again. Running ones are simply re-attached thanks to USE_EXISTING, and finished ones run once more. Note processing is idempotent, so this costs some extra work, not wrong data.

Timeouts are part of correctness

The importer now lives as long as the whole import. Our options:

proxyActivities<typeof importActivities>({
  startToCloseTimeout: "24 hours",
  heartbeatTimeout: "20 minutes",
  retry: { initialInterval: "1 second", maximumAttempts: 3 },
});

• heartbeatTimeout is what detects a dead worker. Without it, a crashed pod would be noticed only when the 24-hour startToCloseTimeout runs out. Twenty minutes is a comfortable margin over the 5-minute heartbeat timer.
• Every deploy during an import costs an attempt: the running attempt dies with the pod and the activity is retried. With maximumAttempts: 3, three deploys during one long import fail it, even though nothing was wrong with the data.
• Every retry repeats the preparation: reading the whole archive into memory and preparing documents, before resuming from the heartbeat. In practice even a large archive loaded quickly, so it never caused trouble. It’s still worth watching: if that phase ever ran longer than the heartbeat timeout without heartbeating, every attempt would die in the same place.

Why an activity and not a workflow?

Experienced Temporal users will ask why the window lives in an activity. The idiomatic alternative is a parent workflow that starts note workflows as children, keeps N of them running and calls continue-as-new from time to time to keep its history small.

That version survives worker restarts for free. It moves the window state from volatile process memory into Temporal’s durable state, so there is no heartbeat bookkeeping, no rebuilding after a retry and no 24-hour activity to babysit.

We kept the activity because it was the smallest change to code that already worked, and even that took some convincing. The price is everything in the previous section: the state of the window lives in memory, and correctness after a retry depends on how carefully you rebuild it. If I were designing this again, I’d seriously consider the workflow version.

Takeaways

1. A started workflow is not a finished one. start() only tells you Temporal has accepted the work, so batching starts throttles the client, not the system. If you want to control load, wait for result().
2. New starts compete with the workflows already running. Throughput is capped anyway, but every new start adds work to the same system that is trying to finish the workflows already in flight. From the outside, that looks like a freeze.
3. A shared activity queue limits what separate workers can do. Work from every workflow type still meets on the shared queue.
4. Size the window with Little’s law, then watch downstream. Too small and you’re slow, too large and you’re back to the freeze.
5. Database tuning is a trade-off, not a fix. synchronous_commit = off helped a little, but it trades durability for speed and doesn’t touch the cause. Once the code was fixed, we turned it back on.
6. A long-running orchestrating activity makes timeouts and heartbeats part of correctness. They decide how fast a dead worker is noticed, how many deploys an import survives and where it resumes.

How it ended

With the window in place, imports stopped looking frozen and finished in about half the time.

Some things we didn’t change. The importer still reads the whole archive into memory. Each note workflow still receives the full note as its argument, so the data ends up in its history; passing an ID and loading the note in an activity would be lighter. And one workflow per note stayed. Moving away from it would have been a business decision, and not one I was in a position to make.

The practical lesson: when you hand work to Temporal, decide up front how much of it may be in flight at once.

The examples are simplified and anonymized (snacks instead of the real domain). You can run all of the code yourself, no Docker needed: temporal-polyglot-playground.

Act I: everything in TypeScript

In the beginning it was simple. Workflow and activity were both written in TypeScript, and I was handed the interface between them “from above”. It just worked. The activity checked whether a given snack is sweet, salty or spicy, and returned this type:

type SnackCheckResult = {
  flavors: Record<SnackFlavor, boolean> | null;
  caption: string | null;
};

The workflow used it like this:

function workflowLogic(snackCheckResult: SnackCheckResult) {
  const { flavors, caption } = snackCheckResult;
  if (flavors !== null) {
    return { isSweet: flavors.SWEET, caption };
  }
  return { isSweet: false, caption };
}

export async function snackWorkflow(args: SnackCheckArgs): Promise<SnackWorkflowResult> {
  const snackCheckResult = await checkSnack(args);
  return workflowLogic(snackCheckResult);
}

I had inherited this code and never paid attention to how exactly it worked. The realization came fairly quickly, though: the compiler was a validation layer that nobody was thinking about. On the activity side, TypeScript would not let you push anything into the pipe that wasn’t a valid SnackCheckResult. Temporal serialized it, the workflow deserialized it, and everyone was happy.

Act II: enter Python

Then came a natural need: the snack-detection logic would be better written by the ML people, and they work in Python. Part of the activities was rewritten. Not a single line of the workflow changed, and neither did the interface.

Except that the SnackCheckResult type stopped being a contract and became a wish. The Python side never sees it, the TypeScript compiler has no idea what actually arrives over the wire, and the type in the workflow code keeps reassuring everyone who reads it.

How easy is it to get this wrong? Plain Pydantic is enough (an illustrative example, not taken from the project):

class SnackCheckResult(BaseModel):
    flavors: dict[str, bool] | None = None
    caption: str | None = None

SnackCheckResult(caption="a cookie").model_dump(exclude_none=True)
# {'caption': 'a cookie'}   <- the "flavors" key simply isn't there

exclude_none=True is a common, reasonable habit on the Python side. From TypeScript’s point of view, though, it breaks the contract: a field that according to the type always exists (even if as null) has vanished.

We switched over to the Python activity and at first everything worked. Time to open the champagne.

Act III: a test that had nothing to do with the topic

The trouble surfaced during internal tests that were completely unrelated to this part of the system. The workflow started falling apart, and we had nothing configured that would have made it obvious why. The reason is that a plain TypeError thrown from workflow code does not fail the workflow. It fails the workflow task, which Temporal retries forever, so in practice the workflow just hangs.

Only in tests, after setting this:

const worker = await Worker.create({
  connection: testEnv.nativeConnection,
  taskQueue,
  workflowBundle,
  activities: { checkSnack },
  // By default a plain Error thrown from workflow code fails the workflow *task*, and the
  // task is retried forever: the workflow just hangs. Promote TypeError to a workflow
  // failure so the test can observe the exception instead of timing out.
  workflowFailureErrorTypes: { "*": ["TypeError"] },
});

were we able to see the actual exception: TypeError: Cannot read properties of undefined (reading 'SWEET'). That message, however, points at the place that detected the problem (the consumer), not the place that caused it (the producer).

Why did that line execute at all? Because the workflow guarded with if (flavors !== null). Since TypeScript promised flavors was either a record or null, nobody guarded against undefined. But in JavaScript, undefined !== null evaluates to true, so execution walked right into flavors.SWEET.

To get to the truth, we had to open the Temporal UI and look at the activity’s output. It turned out that there was no flavors at all in that output. Python had simply omitted the field.

Act IV: the quick fix

One of the developers decided he knew how to fix it quickly: add validation of the activity result with Zod. We already had Zod in the project, so a schema describing the same type was trivial (Zod 3, which was what we were on at the time):

flavors: z.record(SnackFlavor, z.boolean()).nullable()

It was a good idea, but I wanted to check whether the validation really catches everything Python might send. I asked for tests with payload variations: missing flavors, lowercase keys, unknown keys. And my hunch paid off, because one test failed.

To simulate Python without running Python, the tests register activities written in TypeScript that return exactly the payloads Python would produce, and I push them past the compiler with as unknown as. That’s an important detail: to simulate Python, I have to cheat the compiler. In a single-language world these bugs would be impossible.

Let’s take a realistic payload: the producer returns only the flavors it considered relevant.

missingSweet: async () =>
  ({
    flavors: { SPICY: false, SALTY: true },
    caption: "dried fish",
  }) as unknown as SnackCheckResult

Here are two tests that differ only in which Zod import path they use:

it('still loses isSweet on a partial record validated with "zod/v3"', async () => {
  const partial = await runSnackWorkflow(
    pythonCheckSnackVariants.missingSweet,
    snackWorkflowWithValidationZodV3,
  );

  expect(partial).not.toHaveProperty("isSweet");
});

it('rejects a partial record validated with "zod/v4"', async () => {
  const error = await runSnackWorkflow(
    pythonCheckSnackVariants.missingSweet,
    snackWorkflowWithValidationZodV4,
  ).catch((err: unknown) => err);

  expect(error).toBeInstanceOf(WorkflowFailedError);
  const cause = (error as WorkflowFailedError).cause;
  expect(cause).toBeInstanceOf(ApplicationFailure);
  expect(cause).toMatchObject({
    type: SNACK_CHECK_VALIDATION_ERROR,
    details: [{ invalidFields: ["flavors"] }],
  });
});

Same schema, the very same line z.record(SnackFlavor, z.boolean()). In Zod 3 the validation passes, and isSweet still disappears. In Zod 4 the result is rejected with a readable error that points at the flavors field, that is, at the producer.

The reason: with enum keys, Zod 3 treats the record as partial, while Zod 4 treats it as exhaustive. So whether validation protects you from an incomplete record depends on the import path ("zod" versus "zod/v4"). The first test deliberately documents the gap (hence “still loses”). If someone closes it one day, the test will start failing.

By the way, this mismatch is already visible in the types themselves. In Zod 3, z.infer for such a schema gives Partial<Record<SnackFlavor, boolean>>, not the Record<SnackFlavor, boolean> our type promises. So Zod was telling us outright that some keys might not exist, and we went on validating a “contract” that doesn’t allow for that. We saw the difference, but decided it didn’t matter and brushed it off. Only a test on a realistic payload showed that this “detail” hides the loss of isSweet.

We had installed validation and felt safe, but we weren’t. A false sense of security is more dangerous than none at all.

The worst part is that the naive version has three variants in which nothing visible happens:

The missing isSweet happens because undefined is dropped during JSON serialization, so the client receives { caption: "dried fish" } even though the type promises isSweet: boolean.

Act V: the question that should have come earlier

At this point I started wondering not about yet another patch, but about something different: why is this interface so complicated?

Record<SnackFlavor, boolean> makes sense as a method parameter in TypeScript. The compiler enforces completeness, and access by key is convenient. As a contract between languages, though, it is very brittle, because every language has its own idea of a missing field, null, an empty value and keys. It resembles a situation where, instead of a contract first approach (a contract described independently, like XML Schema in SOAP or JSON Schema in OpenAPI, and only then “translated” into each language), we do plain RPC on functions specific to one language and hope that the other side’s runtime will somehow adapt.

If the shape of the data itself allows “incomplete” states, you can either detect them or change the shape so that they don’t exist at all. I chose the latter:

import { z } from "zod";
import { SNACK_FLAVORS } from "../types";

const SnackFlavor = z.enum(SNACK_FLAVORS);

export const SnackCheckResultV2 = z.object({
  flavors: z.array(SnackFlavor).nullable(),
  caption: z.string().nullable(),
});
export type SnackCheckResultV2 = z.infer<typeof SnackCheckResultV2>;

The activity now returns the list of flavors it found. There is no such thing as an incomplete list: a flavor that isn’t on the list simply isn’t present. The workflow builds the full Record<SnackFlavor, boolean> locally from SNACK_FLAVORS.

The workflow itself also stops trusting what the activity promises:

async function checkSnackAndValidate(
  args: SnackCheckArgs,
  schema: ActivityResultSchema<SnackCheckResultV2>,
): Promise<SnackWorkflowResult> {
  // The activity's declared return type is not trusted: the payload is validated at runtime.
  const raw: unknown = await checkSnack(args);
  const { flavors, caption } = parseActivityResult(raw, schema);
  if (flavors === null) {
    return { isSweet: false, caption };
  }

  const flavorRecord = Object.fromEntries(
    SNACK_FLAVORS.map((flavor) => [flavor, flavors.includes(flavor)]),
  ) as Record<SnackFlavor, boolean>;

  return { isSweet: flavorRecord.SWEET, caption };
}

The const raw: unknown line expresses the whole lesson of this post: a TypeScript type is not proof, it’s a declaration. The compiler won’t let you use raw without going through validation. The record is rebuilt so that the rest of the workflow keeps working on the full type, as before.

The results are identical in Zod 3 and Zod 4:

Note spicyOnly. It is the counterpart of the incomplete record that previously led to a silent failure. Now it’s a perfectly valid payload with the correct answer. Lowercase and unknown flavors are caught because the elements of the array are validated, and the Zod import path no longer matters.

Responsibility has shifted too. Before, the producer had to remember to be complete. Now completeness follows from the shape of the data, and the consumer fills in the rest.

How to report a validation error

All of the error handling fits in a single function:

export function parseActivityResult<T>(raw: unknown, schema: ActivityResultSchema<T>): T {
  const parsed = schema.safeParse(raw);
  if (!parsed.success) {
    const fields = [
      ...new Set(parsed.error.issues.map((issue) => String(issue.path[0] ?? "<root>"))),
    ];
    // A plain ZodError would only fail the workflow task and retry it forever. A non-retryable
    // ApplicationFailure fails the workflow execution loudly instead.
    throw ApplicationFailure.nonRetryable(
      `invalid activity result: ${fields.join(", ")}`,
      SNACK_CHECK_VALIDATION_ERROR,
      { invalidFields: fields },
    );
  }
  return parsed.data;
}

Two decisions:

• ApplicationFailure.nonRetryable instead of a plain ZodError. A plain exception would again hang the workflow task in a loop, repeating the TypeError problem. Here the workflow ends with a clear, named error you can set an alert on. Retrying makes no sense, because with a broken contract every attempt returns the same thing.
• We report only the name of the top-level field. No values and no Zod messages, which may quote the received data. ApplicationFailure details end up in the workflow history and are visible in the UI, so real data shouldn’t leak there. And the answer to “which property failed” is exactly what someone with an alert in hand needs.

The ActivityResultSchema<T> interface describes only safeParse, the part common to Zod 3 and 4. That’s how one function handles both schemas.

A note on Zod and workflow determinism

Any code executed inside a Temporal workflow is subject to the golden rule of Temporal: determinism during replay.

Running Zod inside workflow logic is entirely safe because validating structural types (objects, arrays, strings, enums) is a pure computation. However, you need to be cautious with Zod features that can introduce non-determinism:

• Defaults or transforms that evaluate time or randomness (e.g., new Date(), Date.now(), or crypto.randomUUID()).
• Refinements (.refine()) or transforms (.transform()) that access external state, globals, or environments.

As long as schemas are pure validators over the raw payload, workflow replays will produce identical results every time.

Why validate at the call site instead of an interceptor?

A fair question for experienced Temporal users: why call parseActivityResult manually instead of moving validation into a workflow interceptor or a custom data converter?

You certainly can automate this via interceptors once schemas are mapped to activity types across the board. Doing it explicitly at the call site, however, kept the boundary front and center while we were redesigning the contract, made testing isolated workflows trivial, and served as a visible reminder that activity returns are untrusted wire data.

Takeaways

1. TypeScript types don’t cross the language boundary. As long as both ends of the pipe are in one language, the compiler does a job you don’t know about. Add a second language and that job disappears without warning.
2. Runtime validation at the boundary is necessary, but its mere presence proves nothing. Zod 3 and Zod 4 behave differently for the same schema. Write tests that check that validation rejects what it should reject, not only that it accepts what it should accept.
3. The most dangerous errors are silent. A hanging workflow raises an alarm. A wrong isSweet doesn’t.
4. The error has to point at the producer. The TypeError showed where we crashed. A named field in ApplicationFailure shows who broke the data.
5. Best of all, change the contract so the invalid state can’t be expressed. An array instead of a record eliminates a whole class of errors instead of detecting them.
6. For cross-language communication, think contract first. The shape of the data should be simple enough for every language to produce and check without guessing what the other runtime will do.

How it ended

The Python team appreciated the help with changing the contract, and the migration went smoothly. It wasn’t a one-step move, though, but the classic three-step approach: add a new version of the activity next to the old one, switch the workflow to call the new one, and finally remove the old one.

The practical lesson for the future: contracts at language boundaries should be built contract first, on supported standards (JSON Schema, Protobuf or OpenAPI), from which every language generates its own code. That code doesn’t have to be pretty or optimal from a purist’s point of view, because it lives at the edge of the system (in the delivery mechanism), not in the domain logic. Zod at that boundary is a cheaper defense, but not a replacement for a shared schema.

All the code and tests: github.com/jmilkiewicz/temporal-polyglot-playground. Run them with npm install && npm test.