Skip to content

When a node stops answering

Here is the tradeoff, in the first paragraph, because it is the thing to know before deploying two nodes: tapio decides a peer is gone by waiting, and waiting cannot tell a dead peer from a slow one. When it decides wrongly, it tells the watchers on this node that actors which are alive have stopped. It then stays wrong until somebody says otherwise.

This page explains what that looks like, why it is the default, and what changes it later.

How a peer is declared unreachable

Each association heartbeats every heartbeat_interval, one second by default. When nothing has arrived from a peer for unreachable_after, ten seconds by default, that association is declared unreachable and three things happen together:

  • Every ref on that peer that anything here was watching gets a Terminated, delivered on the system lane so supervision reacts at once.
  • The association is quarantined. Buffered and subsequent sends become dead letters naming the peer, and nothing tries to reconnect.
  • A PeerUnreachable event is published on the system event stream, so a service can log it, alarm on it, or decide to shut itself down.

From inside this node, that is indistinguishable from the peer having stopped, and it is meant to be: the code that handles a Terminated is the same code either way.

The part that can be wrong

A partition, a long pause, an overloaded peer and a dead peer all look identical from here. There is nothing this node can measure that separates them, because the only evidence is the absence of messages.

So both sides of a partition declare the other dead, and both are locally correct:

"""Two live nodes, a broken network, and both of them wrong about the other.

Concepts: the failure detector, `PeerUnreachable` on the system event stream,
quarantine, and `remote.reconnect` as the explicit repair.

This is the uncomfortable example, and it is here because the behaviour it
shows is the one people meet in production and do not expect. Nothing dies.
Both nodes run the whole time, both keep processing their own work, and the
only thing that breaks is the network between them. Each one then concludes
that the other is gone, tells its watchers so, and stops sending. Both
conclusions are wrong, and both are the best a single node can do.

The fix for this is a quorum: enough nodes to hold a vote, so that a minority
of a partition can find out that it is the minority and stand down. That is
clustering, and it is not in this version. What is here instead is a set of
defaults chosen so that being wrong is recoverable: fail fast, freeze the
address, and let a person or a supervisor decide when to try again.

What to watch in the output: the two blocks of node lines, printed side by
side. `home` says away is gone. `away` says home is gone. Both are still
answering their own callers while they say it. Then note that healing the
network on its own repairs nothing: the last two lines only happen because
somebody asked for them.

Run it with `uv run python -m tapio_examples.partition`.
"""

import asyncio
from datetime import timedelta

from tapio import (
    ActorSystem,
    Behavior,
    Behaviors,
    Message,
    Signal,
    Terminated,
    register_message,
)
from tapio.actor import ActorContext, ActorRef
from tapio.actor.events import Subscription
from tapio.remote.address import format_ref
from tapio.remote.failure import PeerUnreachable
from tapio.testkit import two_nodes

__all__ = ["Poke", "Poked", "main"]


@register_message()
class Poked(Message):
    """Proof that the actor on the other node is still working."""

    by: str


@register_message()
class Poke(Message):
    """A message with a reply address, to show the link working before it breaks."""

    by: str
    reply_to: ActorRef[Poked]


def steady(lines: list[str], node: str, answered: asyncio.Event) -> Behavior[Poke]:
    """Build an actor that answers, and keeps answering through everything.

    Args:
        lines: Where to record what it did.
        node: The node it runs on, for the output.
        answered: Set once it has answered something.

    Returns:
        The behavior to spawn.
    """

    async def on_poke(message: Poke) -> Behavior[Poke]:
        lines.append(f"{node}: poked by {message.by}, still working")
        message.reply_to.tell(Poked(by=node))
        answered.set()
        return Behaviors.same()

    return Behaviors.receive_message(on_poke, msg_type=Poke)


def mourner(
    remote: ActorRef[Poke], lines: list[str], node: str, bereaved: asyncio.Event
) -> Behavior[Poked]:
    """Build an actor that watches the other node's actor and reports its death.

    Args:
        remote: The actor on the other node.
        lines: Where to record what it was told.
        node: The node it runs on, for the output.
        bereaved: Set when `Terminated` arrives.

    Returns:
        The behavior to spawn.
    """

    def build(ctx: ActorContext[Poked]) -> Behavior[Poked]:
        ctx.watch(remote)

        async def on_message(message: Poked) -> Behavior[Poked]:
            return Behaviors.same()

        async def on_signal(
            ctx: ActorContext[Poked], signal: Signal
        ) -> Behavior[Poked]:
            if isinstance(signal, Terminated):
                # It is not dead. It is on the other side of a partition,
                # answering somebody else. Nothing in this signal says so, and
                # nothing could.
                lines.append(f"{node}: told that {signal.ref.path} has stopped")
                bereaved.set()
            return Behaviors.same()

        return Behaviors.receive_message(on_message, on_signal=on_signal)

    return Behaviors.setup(build)


def watch_events(system: ActorSystem, lines: list[str], node: str) -> Subscription:
    """Report every peer this node gives up on.

    Args:
        system: The node to subscribe to.
        lines: Where to record what it published.
        node: The node's name, for the output.

    Returns:
        The subscription, so the example can stop listening before the two
        systems shut down. A node going away makes its peer unreachable too,
        and that one is not news.
    """

    def record(event: PeerUnreachable) -> None:
        verdict = "quarantined" if event.quarantined else "link ended"
        lines.append(f"{node}: gave up on {event.peer}, {verdict}")

    return system.events.subscribe(PeerUnreachable, record)


async def main() -> list[str]:
    """Run the example.

    Returns:
        Both nodes' lines, home's first, so the two views can be read side by
        side.
    """
    home_lines: list[str] = []
    away_lines: list[str] = []
    home_answered, away_answered = asyncio.Event(), asyncio.Event()
    home_bereaved, away_bereaved = asyncio.Event(), asyncio.Event()

    # Short enough for an example. In production this is seconds, and it has
    # to be comfortably longer than the peer's heartbeat interval or an
    # ordinary slow moment reads as a dead node.
    async with two_nodes(
        alpha="home",
        beta="away",
        unreachable_after=timedelta(milliseconds=500),
        heartbeat_interval=timedelta(milliseconds=20),
    ) as nodes:
        here, there = nodes.alpha, nodes.beta
        listening = [
            watch_events(here, home_lines, "home"),
            watch_events(there, away_lines, "away"),
        ]

        here_actor = here.spawn(steady(home_lines, "home", home_answered), "steady")
        there_actor = there.spawn(steady(away_lines, "away", away_answered), "steady")
        to_there = await here.resolve(
            format_ref(there.address, there_actor.path), expect=Poke
        )
        to_here = await there.resolve(
            format_ref(here.address, here_actor.path), expect=Poke
        )
        here_watcher = here.spawn(
            mourner(to_there, home_lines, "home", home_bereaved), "mourner"
        )
        there_watcher = there.spawn(
            mourner(to_here, away_lines, "away", away_bereaved), "mourner"
        )

        to_there.tell(Poke(by="home", reply_to=here_watcher))
        to_here.tell(Poke(by="away", reply_to=there_watcher))
        await away_answered.wait()
        await home_answered.wait()

        # Nothing dies here. The two nodes simply stop being able to hear each
        # other, which is the case no single node can tell from the other one.
        nodes.partition()
        await home_bereaved.wait()
        await away_bereaved.wait()

        # Both are still doing their own work, each while believing the other
        # is gone. Poking locally proves it.
        home_answered.clear()
        away_answered.clear()
        here_actor.tell(Poke(by="home itself", reply_to=here_watcher))
        there_actor.tell(Poke(by="away itself", reply_to=there_watcher))
        await home_answered.wait()
        await away_answered.wait()

        # The network is fine again, and nothing reconnects. Both nodes have
        # already told their watchers that live actors are gone, so quietly
        # resuming would leave them holding contradictory beliefs with no way
        # to notice.
        nodes.heal()
        home_lines.append("home: network repaired, and still no association")

        there.remote.clear_quarantine(here.address)  # type: ignore[union-attr]
        await here.remote.reconnect(there.address)  # type: ignore[union-attr]
        home_lines.append("home: reconnected, because somebody decided to")

        for subscription in listening:
            subscription.unsubscribe()

    lines = home_lines + away_lines
    for line in lines:
        print(line)
    return lines


if __name__ == "__main__":
    asyncio.run(main())

That example prints both nodes' beliefs next to each other. Both are still serving, each thinks the other is gone, and neither can tell. This is the split-brain problem, and resolving it needs membership and a quorum, which v0.1 does not have.

Why fail fast is the default

Given that the guess can be wrong, there are two ways to be wrong: decide too early, or wait forever. tapio decides.

For the request/response and work-distribution shapes it is built for, wrongly deciding that a peer is dead costs a retry at the application level, which is recoverable. Waiting forever costs availability, which often is not. A worker that might be alive is not a worker you can hand the next job to.

The second half of the default matters as much: once wrong, tapio stays wrong in a way you can see. The quarantine does not clear itself.

Recovery is explicit

await system.remote.reconnect(peer_address) clears the quarantine and re-associates. Nothing does that on its own, even after the network is repaired, and that is the deliberate part.

Automatic re-association after a false positive is the dangerous case: the watchers here were already told Terminated for actors that are alive and carrying on. Silently resuming would leave two nodes with contradictory beliefs about who is alive and no moment at which either could notice. An explicit call means a human or a supervisor decided to accept the peer again, and the application gets to re-establish whatever it needs to.

A clustered system is the one exception, and it is an informed one. A node keeps knocking on its fellow members, quarantine and all, because a member that has not been downed is still a member and the cluster has a membership to consult where a single system has nothing. A peer that is not a member follows the rule above unchanged. See clustering.

Refs held across a quarantine are not reusable. Their uid belongs to a session that is over, so addressing after a reconnect goes through resolve again. That is also what makes a restarted peer a different peer rather than an impostor at the same address: a system mints a new uid per incarnation, and an association is bound to the uid it handshook with.

Designing for it

The thing that survives this cleanly is work you can hand to somebody else:

"""Watching an actor on another node, and rebuilding when that node dies.

Concepts: `ctx.watch` on a ref that came from `resolve`, `Terminated` arriving
because a whole system went away, and a coordinator that reacts by rebuilding
the work somewhere it can still reach.

The watch is the same call as in `death_watch`, on a ref that happens to point
at another process. That is the whole of the API difference. The difference
that matters is in what the signal means. Locally, `Terminated` is a fact: the
actor stopped. Across a link it is a conclusion: this node stopped hearing
from that one. Here the peer really did terminate, so the conclusion is right,
but nothing in the signal says which of the two it was, and nothing can.

Supervision does not cross the wire, so the coordinator is not the remote
worker's supervisor and never sees it fail. It sees it disappear. That is the
supported way to depend on an actor somewhere else: watch it, and have a plan
for the day it stops answering.

What to watch in the output: the third line. The coordinator does not retry
against the node that is gone. It starts a worker it owns and finishes the
job there, which is the difference between a failover and a stall.

Run it with `uv run python -m tapio_examples.node_failure`.
"""

import asyncio

from tapio import (
    Behavior,
    Behaviors,
    Message,
    Signal,
    Terminated,
    register_message,
)
from tapio.actor import ActorContext, ActorRef
from tapio.remote.address import format_ref
from tapio.testkit import two_nodes

__all__ = ["Assign", "Done", "Job", "main"]


@register_message()
class Done(Message):
    """A finished job, and which node finished it."""

    item: int
    where: str


@register_message()
class Job(Message):
    """A unit of work, carrying the ref its result goes back to."""

    item: int
    reply_to: ActorRef[Done]


class Assign(Message):
    """Tell the coordinator to get one job done, wherever it can.

    Not registered, because it never leaves the node that sends it. Only what
    crosses a link needs a wire key.
    """

    item: int


def worker(where: str) -> Behavior[Job]:
    """Build an actor that does the work and says where it was done.

    Args:
        where: The node's name, so the output shows which one ran it.

    Returns:
        The behavior to spawn.
    """

    async def on_job(message: Job) -> Behavior[Job]:
        message.reply_to.tell(Done(item=message.item, where=where))
        return Behaviors.same()

    return Behaviors.receive_message(on_job, msg_type=Job)


def coordinator(
    remote: ActorRef[Job],
    lines: list[str],
    finished: asyncio.Event,
    lost: asyncio.Event,
) -> Behavior[Assign | Done]:
    """Build the actor that hands out work and survives losing its worker.

    Args:
        remote: The worker on the other node, to start with.
        lines: Where to record what happened.
        finished: Set whenever a job comes back.
        lost: Set when the remote worker is gone and a local one has replaced it.

    Returns:
        The behavior to spawn.
    """

    def build(ctx: ActorContext[Assign | Done]) -> Behavior[Assign | Done]:
        # One call, and from here on this coordinator cannot be left holding a
        # ref to a node that is no longer there without knowing it.
        ctx.watch(remote)
        current = remote
        # The coordinator takes a wider protocol than the worker knows how to
        # send, so what goes in `reply_to` is an adapter ref: an `ActorRef[Done]`
        # that delivers into this actor. It is addressable like the actor
        # behind it, so a result crossing the link finds its way back.
        answers: ActorRef[Done] = ctx.message_adapter(lambda done: done, Done)

        async def on_message(message: Assign | Done) -> Behavior[Assign | Done]:
            nonlocal current
            match message:
                case Assign(item=item):
                    # An ordinary tell. Whether it crosses a link is decided
                    # by which ref this is, and this code does not ask.
                    current.tell(Job(item=item, reply_to=answers))
                case Done(item=item, where=where):
                    lines.append(f"home: job {item} done by {where}")
                    finished.set()
            return Behaviors.same()

        async def on_signal(
            ctx: ActorContext[Assign | Done], signal: Signal
        ) -> Behavior[Assign | Done]:
            nonlocal current
            if isinstance(signal, Terminated):
                lines.append("home: the away node is gone, rebuilding here")
                # A child of this actor, so it is supervised by the one that
                # depends on it. That is what could never be true of the
                # worker on the other node.
                current = ctx.spawn(worker("home"), name="worker")
                lost.set()
            return Behaviors.same()

        return Behaviors.receive_message(on_message, on_signal=on_signal)

    return Behaviors.setup(build)


async def main() -> list[str]:
    """Run the example.

    Returns:
        The lines the coordinator produced, in order.
    """
    lines: list[str] = []
    finished, lost = asyncio.Event(), asyncio.Event()

    async with two_nodes(alpha="home", beta="away") as nodes:
        here, there = nodes.alpha, nodes.beta
        hand = there.spawn(worker("away"), name="worker")
        remote = await here.resolve(format_ref(there.address, hand.path), expect=Job)
        boss = here.spawn(coordinator(remote, lines, finished, lost), name="boss")

        boss.tell(Assign(item=1))
        await finished.wait()
        finished.clear()

        # The whole node goes away, not just the actor. Every watcher of every
        # ref on it hears the same thing.
        await there.terminate()
        await lost.wait()

        boss.tell(Assign(item=2))
        await finished.wait()

    for line in lines:
        print(line)
    return lines


if __name__ == "__main__":
    asyncio.run(main())

The coordinator does not retry into the node that is gone. It rebuilds the worker somewhere that answers and finishes the job there, because the job was described by a message rather than by a location.

The shapes that struggle are the ones where a remote actor holds the only copy of something. If that is your design, either keep the authoritative copy where the writer is, or accept that a false positive costs you a rebuild.

What changes this

This is the default position for a bare pair of nodes, and two things improve on it. Both have landed as opt-ins that the defaults leave off, so turning either on does not change how a plain two-node link behaves:

  • A phi-accrual failure detector cuts the false-positive rate by treating a peer that is late as a probability rather than a deadline. It has landed for the cluster's failure detector, where phi_accrual selects it; the plain two-node association on this page still uses the fixed window described above.
  • Membership with a downing strategy makes the surviving side a fact rather than a guess. With a lease, the side that holds it keeps working and the other side stops, which is a real answer rather than two contradictory local ones. Both have landed: pass a downing strategy to Cluster and the losing side of a partition is written off while the loser downs itself. Without one, an unreachable member blocks the cluster instead of being guessed about, which is the safe default until an operator says how this cluster would rather resolve a split.

Two nodes is the worst case for every deterministic strategy and the best case for a lease, which is worth saying plainly, since two nodes is where most deployments start.

If your system cannot tolerate a false Terminated today, the honest options are to design around it as above, or to run a cluster with a lease-backed downing strategy: LeaseMajority with a Lease that lives outside the partition, so the side that holds it survives and the other side stops.