API reference

Generated from the source, so it cannot drift from what is installed.

System

One actor tree, running on the loop that created it.

Create it inside a coroutine, spawn actors under /user, and terminate it when done:

async with ActorSystem("hello") as system:
    greeter = system.spawn(greeter_behavior(), name="greeter")
    greeter.tell(Greet(whom="world"))

Leaving the async with block terminates the system, draining the tree bottom-up against a single deadline.

address property

How peers address this system, and what its refs write down.

With remoting configured this is the canonical address, which is what a peer dials and not always what the socket is bound to. Otherwise it is the system name alone. A ref from a system with remoting off says which system it belongs to and gives nowhere to dial.

blocking property

The threads this system runs blocking calls on.

Exposed so a test can assert that shutdown left none of them running. The pool is the one piece of the runtime that is not a task, so the leak invariant does not cover it for free.

dead_letters property

Where undeliverable messages go, and what to subscribe to.

Subscribing is what makes an absence testable. Without it, "the message was dropped" and "the code never ran" look the same.

events property

What this system publishes about itself, for whoever subscribes.

Runtime facts rather than traffic. Today those are PeerUnreachable and PeerReachable, which is how a service learns that a node it was talking to is beyond reach and decides whether to log it, alarm, or stop.

is_terminating property

Whether shutdown has begun.

log property

A logger tagged with the system's root path.

name property

The system name, and the authority in every path below it.

refs property

The live refs of this system, by path and incarnation uid.

Exposed for the same reason as watchers on a cell: a test has to be able to assert that the registry was emptied. An entry that outlives its actor is a leak.

remote property

This system's remoting, or None when it is switched off.

It holds the port, the associations, and the resolver behind every foreign ref. Exposed so a test or an operator can see which peers are associated, instead of inferring it from traffic.

settings property

The tunables this system runs with.

uid property

This incarnation's uid, presented to every peer in the handshake.

A system restarted on the same host and port is a different peer, and the uid is what says so. Without it, a node that restarts inside a failure detector's window looks the same as one that was slow.

__aenter__() async

Return the running system.

__aexit__(exc_type, exc, tb) async

Terminate the system on the way out, however the block ended.

__init__(name='tapio', settings=None)

Start a system and its guardians on the caller's event loop.

Parameters:

Name	Type	Description	Default
name	str	The system name, which is the authority in every actor path.	'tapio'
settings	TapioSettings | None	Tunables for this system. Read from the environment when omitted.	None

Raises:

Type	Description
RuntimeError	If called outside a running event loop. A system is built out of tasks and has nowhere to put them.
ValueError	If the name would not make a legal actor path.

__repr__()

Render the system name and whether it is still running.

as_deserialization_context()

Make this system the one refs deserialize against inside the block.

A ref's string form only means something relative to a system. The reading system has to know whether the address is its own, so it can hand back the live local ref rather than a proxy to itself. The receiving end of a link enters this for the duration of a decode. It is public so that a test or a debugging session can run Greet.model_validate_json(blob) on purpose.

Returns:

Type	Description
AbstractContextManager[None]	A context manager. Refs resolve against this system inside it, and
AbstractContextManager[None]	raise RefResolutionError outside.

deliver_frame(data, *, peer=None)

Take one frame off a link and deliver what is in it.

This is the receiving half of remoting. It is a plain method because every failure a peer can cause is decided here, with no socket involved, so a test can cover all of it by handing one system the bytes another produced.

It never raises. A bad size, a bad version, an unknown type key, a payload that will not validate, a recipient that has stopped, and a message the recipient does not accept all become dead letters on this system's stream, carrying the peer address when there is one.

Parameters:

Name	Type	Description	Default
data	bytes	One complete frame, length prefix included.	required
peer	Address | None	Where it came from, recorded on any dead letter.	None

resolve(uri, *, expect) async

Turn a ref's string form into something that can be told messages.

stock = await system.resolve(
    "tapio://inventory@inventory.svc:25520/user/stock", expect=Reserve
)
stock.tell(Reserve(sku="X-1", qty=2, reply_to=ctx.self_ref))

That is the whole user-facing surface of remote messaging: one resolve, then an ordinary ref. Nothing is dialled here. The first send through the ref creates the association, and the dial happens behind it. So this call does not wait for a peer that may be down, and a tell to one that never answers dead-letters rather than hanging. The ref is bound to the peer and not to a link, so it keeps working after a link fails.

expect declares what the actor over there accepts. It is a claim about the peer rather than knowledge of it, so it catches a mistake at this end. The check that decides runs on the receiving node, against the target's real message type.

Parameters:

Name	Type	Description	Default
uri	str	The full string form, tapio://sys@host:port/user/x#uid.	required
expect	type[T]	What the target accepts.	required

Returns:

Type	Description
ActorRef[T]	A ref. The live local one when the address is this system's own,
ActorRef[T]	since resolving your own address should not put a socket in the
ActorRef[T]	middle of a local send.

Raises:

Type	Description
RefResolutionError	If the text is not a ref, if it names another system with no host to dial, or if it names a reachable peer and this system has remoting switched off.
MessageTypeError	If expect is not a Message subclass or a union of them.

resolve_path(address, path)

Turn an address and a path into something that can be told messages.

This is all of ref resolution, and it never raises about the target. There are three answers:

• The address is this system's own and a live actor holds that path and uid. The answer is the live local ref, so replying to a reply_to that crossed a link is an ordinary local tell.
• The address is this system's own and nothing holds it. The answer is a dead-letter target. The uid is what makes this safe: a path on its own is reusable, so without it a stale ref would reach whoever holds that path now.
• The address is another system's. The answer is what the peer resolver returns, or a dead-letter target if there is no link.

Parameters:

Name	Type	Description	Default
address	Address	The system the ref names.	required
path	ActorPath	Where in that system it points.	required

Returns:

Type	Description
ActorRef[Any]	A ref. Always.

set_peer_resolver(resolve)

Install what turns another system's address into a usable ref.

The association layer calls this when remoting starts. Until then, and for any address it has no link to, a foreign ref resolves to a dead-letter target.

Parameters:

Name	Type	Description	Default
resolve	PeerResolver | None	Returns a ref for an address it can reach, and None for one it cannot.	required

spawn(behavior, name, mailbox=None)

Start a top-level actor under /user.

Parameters:

Name	Type	Description	Default
behavior	Behavior[T]	What the actor does.	required
name	str	Its name, unique among top-level actors.	required
mailbox	MailboxConfig | None	Capacity and overflow behaviour. The system's default when omitted.	None

Returns:

Type	Description
ActorRef[T]	A ref to the new actor.

Raises:

Type	Description
ActorSystemTerminating	If the system is shutting down.
ActorNameError	If a live top-level actor already has that name.

spawn_anonymous(behavior, mailbox=None)

Start a top-level actor under a generated name.

Parameters:

Name	Type	Description	Default
behavior	Behavior[T]	What the actor does.	required
mailbox	MailboxConfig | None	Capacity and overflow behaviour. The system's default when omitted.	None

Returns:

Type	Description
ActorRef[T]	A ref to the new actor.

Raises:

Type	Description
ActorSystemTerminating	If the system is shutting down.

spawn_system_actor(behavior, name, mailbox=None)

Start an actor under /system, beside remoting.

For the runtime's own extensions rather than for application actors. Clustering uses it, as remoting does: those actors are part of how the system works, they must not collide with names a user chose, and a failure in one is not the user tree's business.

Application actors belong under /user, through spawn. An actor started here does not sit under the user guardian, so it does not share the shutdown ordering or the escalation behaviour that the rest of an application relies on.

Parameters:

Name	Type	Description	Default
behavior	Behavior[T]	What the actor does.	required
name	str	Its name, unique among system actors.	required
mailbox	MailboxConfig | None	Capacity and overflow behaviour. The system's default when omitted.	None

Returns:

Type	Description
ActorRef[T]	A ref to the new actor.

Raises:

Type	Description
ActorSystemTerminating	If the system is shutting down.
ActorNameError	If a live system actor already has that name.

terminate() async

Stop every actor, bottom-up, and wait for the tree to drain.

Calling this more than once is safe, and so is being cancelled while it runs. The drain itself is a task the system owns, so a caller that gives up waiting, or is cancelled at a bad moment, does not cancel the shutdown: the tree still finishes and later callers still see it. This is the guarantee ActorCell.stop keeps for a single cell, one level up for the whole tree.

when_terminated() async

Wait until the system has finished shutting down.

This is where an escalation that reached a guardian surfaces. Nowhere else can raise it. The failing actor's exception never leaves its own receive loop, and every supervisor above it declined to take responsibility, so the last place to report it is the one the embedding service is already waiting on.

Raises:

Type	Description
BaseException	The original failure, if the system terminated because one escalated to a guardian.

Behaviors

Factories for the functional style.

A namespace rather than loose functions, so that Behaviors.same() reads the way it does in the actor libraries this borrows from.

empty() staticmethod

Handle nothing: user messages are unhandled, signals still arrive.

A signal goes to the last real behavior the actor held, so an actor that becomes empty() still runs its PostStop and still hears a Terminated from an actor it was watching.

ignore() staticmethod

Consume every user message silently, and keep taking signals.

Like empty(), except a user message is dropped without being reported as unhandled. Signals still arrive, so PostStop runs and a held resource is released.

receive(on_message, msg_type=None, *, on_signal=None) staticmethod

Handle messages with a (ctx, message) function.

Parameters:

Name	Type	Description	Default
on_message	Callable[[ActorContext[T], T], Awaitable[Behavior[T]]]	The handler.	required
msg_type	MessageType | None	What this behavior receives. Read from the handler's annotation when omitted.	None
on_signal	SignalHandler[T] | None	Called with (ctx, signal) for lifecycle signals. Without one, signals are reported as unhandled.	None

Returns:

Type	Description
Behavior[T]	The behavior.

receive_message(on_message, msg_type=None, *, on_signal=None) staticmethod

Handle messages with a (message) function, ignoring the context.

Parameters:

Name	Type	Description	Default
on_message	Callable[[T], Awaitable[Behavior[T]]]	The handler.	required
msg_type	MessageType | None	What this behavior receives. Read from the handler's annotation when omitted.	None
on_signal	SignalHandler[T] | None	Called with (ctx, signal) for lifecycle signals. Without one, signals are reported as unhandled.	None

Returns:

Type	Description
Behavior[T]	The behavior.

same() staticmethod

Keep the current behavior, with whatever state it holds.

setup(factory) staticmethod

Defer construction until the actor starts, and re-run it on restart.

Parameters:

Name	Type	Description	Default
factory	Callable[[ActorContext[T]], Behavior[T]]	Called with the context to produce the real behavior.	required

Returns:

Type	Description
Behavior[T]	The behavior.

stopped() staticmethod

Stop this actor, running its post-stop signal on the way out.

supervise(behavior) staticmethod

Govern a behavior's failures with a strategy.

Behaviors.supervise(worker()).on_failure(
    SupervisorStrategy.restart(max_restarts=3, window=timedelta(seconds=1)),
    on=ConnectionError,
)

Wrappers nest, and the outermost is consulted first. A specific exception governed by an inner wrapper must be wrapped again outside it to win. A failure that matches nothing stops the actor, which is what an unsupervised actor already does.

Supervision belongs to the actor, not to the behavior it currently holds. Switching behavior keeps the strategies, and a restart reinstates the ones the actor was spawned with.

Parameters:

Name	Type	Description	Default
behavior	Behavior[T]	What the actor does.	required

Returns:

Type	Description
Supervise[T]	A builder whose on_failure produces the supervised behavior.

unhandled() staticmethod

Report that this message was not handled, keeping the behavior.

with_stash(capacity, factory) staticmethod

Defer construction, handing the behavior a buffer to hold messages in.

Behaviors.with_stash(100, lambda stash: loading(stash))

For an actor that cannot answer yet. Put what arrives aside, then return stash.unstash_all(ready_behavior) once it can. The held messages go back to the front of the mailbox, ahead of anything that queued up since, and the buffer is left empty.

The capacity is required. A stash holds traffic the actor is not keeping up with, so an unbounded one is a memory leak. Overflow raises StashOverflowError in the actor that stashed, which is where the decision about what to do belongs.

A restart empties the buffer, because messages held by the state that just failed are not the new state's to answer. What is discarded is published as a dead letter rather than dropped.

Parameters:

Name	Type	Description	Default
capacity	int	How many messages the buffer can hold.	required
factory	Callable[[StashBuffer[T]], Behavior[T]]	Called with the buffer to produce the real behavior.	required

Returns:

Type	Description
Behavior[T]	The behavior.

with_timers(factory) staticmethod

Defer construction, handing the behavior a scheduler for its timers.

Behaviors.with_timers(
    lambda timers: poller(timers, every=timedelta(seconds=30))
)

A timer sends the actor a message on its own user lane, so a tick is ordinary traffic. It queues behind whatever is already there and never re-enters a busy handler.

The scheduler belongs to the cell, and the cell cancels every timer it holds when the actor restarts or stops. A tick from an incarnation that is gone cannot reach the one that replaced it. The factory runs again on restart, to schedule what the new incarnation needs.

Parameters:

Name	Type	Description	Default
factory	Callable[[TimerScheduler[T]], Behavior[T]]	Called with the scheduler to produce the real behavior.	required

Returns:

Type	Description
Behavior[T]	The behavior.

Bases: ReceivingBehavior[T], ABC

Base class for the class-based style, for actors that hold state.

The message type is read from the type parameter when the class is created:

class Counter(AbstractBehavior[Increment | GetCount]):
    def __init__(self, ctx: ActorContext[Increment | GetCount]) -> None:
        super().__init__(ctx)
        self._count = 0

Set msg_type as a class attribute to override that. It is needed when the parameter is a string forward reference, which cannot be resolved at class creation. Either way, a type that cannot be resolved raises BehaviorTypeError at class definition rather than at spawn.

An abstract subclass, one that leaves on_message abstract, is exempt, so intermediate bases in a hierarchy need no type of their own.

ctx property

The context this behavior was constructed with.

__init__(ctx)

Bind the behavior to its context.

__init_subclass__(**kwargs)

Resolve and freeze the subclass's message type.

on_message(message) abstractmethod async

Handle one message and return what the actor does next.

on_signal(signal) async

Handle one lifecycle signal, if this actor cares about any.

Override this to react to PostStop, PreRestart or a Terminated from a watched actor. The default reports the signal as unhandled, which is not a failure.

Parameters:

Name	Type	Description	Default
signal	Signal	The signal that arrived.	required

Returns:

Type	Description
Behavior[T]	What the actor does next.

receive(ctx, message) async

Delegate to on_message, since the context is already held.

receive_signal(ctx, signal) async

Delegate to on_signal, since the context is already held.

Bases: ABC, Generic[T]

What an actor does next.

A handler returns one of these: itself (same), a new behavior, or a terminal one (stopped).

msg_type = None class-attribute instance-attribute

The declared message type, or None for behaviors that carry none.

same(), stopped() and friends carry no type: they resolve against the type the actor already has, not independently. setup is None too, since its type is whatever the behavior it produces declares.

Bases: Enum

What a behavior that carries no handler asks the runtime to do.

These are returned as behaviors, Behaviors.same() and friends, so a handler has one return type. The runtime reads the directive back off the sentinel with directive_of, instead of comparing against private module constants.

EMPTY = 'empty' class-attribute instance-attribute

Report every user message as unhandled; signals still arrive.

IGNORE = 'ignore' class-attribute instance-attribute

Consume every user message silently; signals still arrive.

SAME = 'same' class-attribute instance-attribute

Keep the current behavior and its state.

STOPPED = 'stopped' class-attribute instance-attribute

Stop this actor.

UNHANDLED = 'unhandled' class-attribute instance-attribute

Report the message as unhandled, keeping the current behavior.

Return the directive a behavior carries, or None if it handles messages.

Context and refs

Bases: ABC, Generic[T]

The runtime handed to a behavior for the duration of a message.

Only the members the runtime can honour today are declared. Timers and stashing are deliberately not here. A behavior receives them from Behaviors.with_timers and Behaviors.with_stash, because both outlive an incarnation and belong to the cell rather than to whatever the actor happens to be doing.

It is an abstract class rather than a Protocol, because the runtime hands out one implementation and users are not expected to write their own.

log abstractmethod property

A logger that tags every record with this actor's path.

path abstractmethod property

Where this actor sits in the tree.

self_ref abstractmethod property

A ref to this actor, to hand out in messages.

Named self_ref rather than Pekko's self, because self is already the first parameter of every method that would use it.

message_adapter(adapt, msg_type=None) abstractmethod

Hand out a ref that translates another protocol into this actor's.

For talking to an actor whose reply type is not yours and should not become yours. Widening your declared message type to admit a foreign reply lets anyone send it, and puts someone else's vocabulary inside your handlers. An adapter avoids both:

replies = ctx.message_adapter(
    lambda price: PriceQuoted(cents=price.cents), msg_type=Price
)
pricing.tell(Quote(reply_to=replies))

A translated message arrives on this actor's own user lane, so it is ordinary traffic. It queues where it arrived, it never re-enters a running handler, and it is validated against the declared type like anything else.

The translation runs in this actor rather than in the sender, so a failure in it becomes this actor's supervision decision. A sender that has never heard of the adapter must not have this actor's bug raised into it.

Each call makes a new adapter, and one already handed out keeps working across a restart. The ref addresses the actor, not the incarnation that created it.

Parameters:

Name	Type	Description	Default
adapt	Callable[[U], T]	Turns an accepted message into one of this actor's own. Its parameter annotation says what it accepts.	required
msg_type	MessageType | None	What the adapter accepts, when the annotation cannot say. Required for a lambda, which carries none.	None

Returns:

Type	Description
ActorRef[U]	A ref to hand out in place of this actor's own.

Raises:

Type	Description
BehaviorTypeError	If neither msg_type nor an annotation resolves what the adapter accepts.
MessageTypeError	If what it resolves to is not a Message subclass or a union of them.

resolve(uri, *, expect) abstractmethod async

Turn a ref's string form into a ref, wherever the actor it names is.

stock = await ctx.resolve(
    "tapio://inventory@inventory.svc:25520/user/stock", expect=Reserve
)

The same call as ActorSystem.resolve, from inside an actor. An address this system owns resolves to the live local ref, so resolving your own system never puts a socket in the middle of a local send. Another system's address resolves to a ref that reaches it through an association.

Nothing waits for the peer here. The association is created and dialled behind the sends that follow, so this call does not fail because a node is down, and a tell to a peer that never answers dead-letters instead of hanging.

Parameters:

Name	Type	Description	Default
uri	str	The full string form, tapio://sys@host:port/user/x#uid.	required
expect	type[U]	What the target accepts. This is a claim about the peer, checked at this end. The receiving node checks it against the target's real message type, and that is the check that decides.	required

Returns:

Type	Description
ActorRef[U]	A ref to the actor it names.

Raises:

Type	Description
RefResolutionError	If the text is not a ref, if it names a system with no host to dial, or if it names a reachable peer while this system has remoting switched off.
MessageTypeError	If expect is not a Message subclass or a union of them.

run_blocking(fn, /, *args, **kwargs) abstractmethod async

Run a call that blocks on a thread, instead of on the loop.

rows = await ctx.run_blocking(cursor.execute, "select 1")

Every actor in a system shares one event loop, so a handler that blocks stops all of them. This moves the call to a bounded pool of threads that belongs to the system, sized by blocking_pool_size.

Two things about it are worth knowing before you rely on it.

The actor is parked for the duration. It is awaiting, so it is not reading its mailbox: messages queue up behind the call, and on a bounded mailbox the overflow strategy will fire while it waits. The loop is free, which is the point, but this actor is parked until the call returns.

The call cannot be cancelled. Python cannot interrupt a thread that is inside a C call. Cancelling the actor abandons the result and the thread keeps going, and shutdown waits for it only until the deadline. Pass whatever timeout the library you are calling offers.

Parameters:

Name	Type	Description	Default
fn	Callable[..., R]	The blocking callable.	required
*args	Any	Its positional arguments.	()
**kwargs	Any	Its keyword arguments.	{}

Returns:

Type	Description
R	Whatever fn returned.

Raises:

Type	Description
ActorSystemTerminating	If the system is shutting down, so the pool is no longer accepting work.
Exception	Whatever fn raised, re-raised here, where it becomes this actor's supervision decision like any other failure.

spawn(behavior, name, mailbox=None) abstractmethod

Start a child actor under this one.

Parameters:

Name	Type	Description	Default
behavior	Behavior[U]	What the child does. Behaviors.setup(...) is evaluated here, synchronously, so the child's message type is known before the ref is handed back.	required
name	str	The child's name, unique among this actor's live children.	required
mailbox	MailboxConfig | None	Capacity and overflow behaviour for the child. The system's default when omitted.	None

Returns:

Type	Description
ActorRef[U]	A ref to the new child.

Raises:

Type	Description
ActorNameError	If a live child already has that name.
ActorSystemTerminating	If this actor is already shutting down.
BehaviorTypeError	If the behavior declares no resolvable message type.

spawn_anonymous(behavior, mailbox=None) abstractmethod

Start a child under a generated name.

Generated names begin with $, and user-chosen names may not, so a generated name never collides with one someone picked.

Parameters:

Name	Type	Description	Default
behavior	Behavior[U]	What the child does.	required
mailbox	MailboxConfig | None	Capacity and overflow behaviour for the child. The system's default when omitted.	None

Returns:

Type	Description
ActorRef[U]	A ref to the new child.

unwatch(ref) abstractmethod

Stop being told when another actor stops.

Harmless if this actor was not watching it. It does not retract a Terminated that is already on the system lane, because by then the actor it reports on has stopped.

Parameters:

Name	Type	Description	Default
ref	ActorRef[Any]	The actor to stop watching.	required

watch(ref) abstractmethod

Ask to be sent Terminated when another actor stops.

The signal arrives on the system lane, so it is not queued behind waiting user traffic, and it arrives exactly once however many times the ref was watched. A restart produces no signal, because the actor's identity is unchanged and only its incarnation is new.

Watching a ref that has already stopped delivers Terminated at once rather than refusing, so the caller's code is the same either way. This is also why there is no "is it alive?" call: the answer would be out of date before the caller could read it.

Parameters:

Name	Type	Description	Default
ref	ActorRef[Any]	The actor to watch. Watching an actor twice is harmless.	required

Raises:

Type	Description
WatchError	If the ref has no live actor behind it, or if an actor tries to watch itself.

Bases: Generic[T]

A handle for sending messages to one actor.

A ref stays a valid handle after its actor dies. Sending to it never raises because the target is gone: the message becomes a dead letter instead. An "is it alive?" answer would be out of date as soon as you had it, which is why death watch is the supported way to ask.

The type parameter is static only. Generics are erased, so nothing checks it at runtime and ActorRef[Greeted] and ActorRef[Foo] validate the same. A type checker catches the mismatch at the call site. At runtime the check lives on the receiving actor, against its declared message type.

Using one as a Pydantic field:

• Validation of a live ref is an is-instance check and nothing more. It does not check that the target is still alive. That would be a race, since the target can die between the check and the send, and a dead target is not a schema error.
• Serialization gives the ref's full string form, including the address and the incarnation uid, which is what a peer needs to reply to it.
• Validation of that string resolves it against the system reading it. So model_dump() works anywhere, while model_validate() on the result works only inside a system's decode path or an explicit with system.as_deserialization_context(): block. A ref is a handle into a live runtime, and there is no meaningful ref without one.

address property

The address this ref writes itself down with.

On this base class it is the system name and nothing else, which is what a ref from a system with remoting switched off looks like. A peer reading it can tell which system it names, and that there is nowhere to dial. The refs a running system hands out override this with the canonical address that system advertises.

path property

Where this ref points.

__eq__(other)

Refs are equal when they address the same incarnation.

__get_pydantic_core_schema__(source_type, handler) classmethod

Make ActorRef a legal Pydantic field type.

The schema ignores the type parameter, as documented on the class.

__hash__()

Hash by path, so refs work as dict keys and set members.

__init__(path)

Bind a ref to an actor path.

__repr__()

Render as the class name and the path string.

ask(make, *, expect, timeout=None) async

Send one message and await one reply.

Parameters:

Name	Type	Description	Default
make	Callable[[ActorRef[R]], T]	Builds the request from the ref the reply should go to.	required
expect	type[R]	The reply type, which is required.	required
timeout	timedelta | None	How long to wait. The system's ask_timeout when omitted.	None

Returns:

Type	Description
R	The reply.

Raises:

Type	Description
NotImplementedError	Always, on this base class. Delivery belongs to the concrete refs a running actor system hands out.

offer(message) async

Send a message, waiting for the recipient's mailbox to have room.

Backpressure belongs to the mailbox, not to the send, so on an unbounded mailbox this is tell with an await in front of it.

Parameters:

Name	Type	Description	Default
message	T	The message to deliver.	required

Raises:

Type	Description
NotImplementedError	Always, on this base class. Delivery belongs to the concrete refs a running actor system hands out.

tell(message)

Send a message, without waiting and without blocking.

Parameters:

Name	Type	Description	Default
message	T	The message to deliver.	required

Raises:

Type	Description
NotImplementedError	Always, on this base class. Delivery belongs to the concrete refs a running actor system hands out.

watch_target()

Return what would arrange a death watch on this ref, if anything can.

None on this base class, and on any ref that is not a handle to a live actor: a dead-letter target has nothing to report the death of. A local ref answers with its cell, and a remote ref with the peer that holds the actor, so watching is one call either way.

Returns:

Type	Description
WatchTarget | None	The watch target, or None when this ref cannot be watched.

An immutable position in one actor system's tree.

The string form is tapio://system/user/greeter#42, where the fragment is the incarnation uid. A restart keeps both the path and the uid. Stopping and respawning under the same name gets a new uid, so a stale ref cannot address the new actor.

is_root property

Whether this is the system's root path.

name property

The last element of the path, or / at the root.

parent property

The enclosing path. The root is its own parent.

The uid is dropped, because it identifies an incarnation of this actor and says nothing about the parent's.

__post_init__()

Reject names that would make the string form ambiguous.

__repr__()

Render as the string form.

__str__()

Render as tapio://system/user/greeter#42.

child(name, uid=0)

Return the path of a child of this actor.

root(system) classmethod

Return the root path of the named system.

with_uid(uid)

Return this path stamped with an incarnation uid.

Ask

Send one message and await one reply.

Parameters:

Name	Type	Description	Default
target	LocalActorRef[T]	The actor to ask.	required
make	Callable[[ActorRef[R]], T]	Builds the request from the ref the reply should go to. It is a factory rather than a message because the promise does not exist until the ask begins.	required
expect	type[R]	The reply type. Required, because a promise has no cell and so no declared message type of its own. Without it, request/response would be the one delivery path with no type check.	required
timeout	timedelta | None	How long to wait. The system's ask_timeout when omitted.	None

Returns:

Type	Description
R	The reply, which is the object the responder passed.

Raises:

Type	Description
AskTimeoutError	If no reply arrived in time.
AskTargetTerminated	If the target stopped without replying, including when it had already stopped before the ask began.
AskTypeError	If a reply arrived that was not an expect.
MessageTypeError	If the request does not match the target's declared message type. That is an error about the message, so it belongs to the sender, as it does for tell.
RuntimeError	If called from a thread that is not running the system's loop.
ValidationError	If content validation is on and either the request or the reply does not satisfy its own model.

Run one ask: make a promise, send, watch, and wait for whichever lands.

Both asks run through here. A local one delivers into a cell and watches that cell. A remote one writes a frame and watches the actor on the peer. Everything between is the same, which is the point: the promise, the watch, the deadline and the cleanup should not have two implementations that can drift.

Parameters:

Name	Type	Description	Default
watched	WatchTarget	What to watch so a target that goes away fails the ask at once instead of after the full deadline.	required
deliver	Callable[[T], None]	How to send the request.	required
make	Callable[[ActorRef[R]], T]	Builds the request from the ref the reply should go to.	required
runtime	ActorRuntime	The asking system's slice, for the loop and the registry.	required
expect	type[R]	The reply type.	required
timeout	timedelta | None	How long to wait. The system's ask_timeout when omitted.	required
gone	TapioError	The error to raise if the target is already beyond reach when the ask begins. The caller builds it, because a stopped actor and an unreachable peer are different diagnoses.	required

Returns:

Type	Description
R	The reply.

Raises:

Type	Description
AskTimeoutError	If no reply arrived in time.
AskTargetTerminated	If the target stopped without replying.
AskTargetUnreachable	If the peer holding the target went out of reach.
AskTypeError	If a reply arrived that was not an expect.
RuntimeError	If called from a thread that is not running the loop.

Bases: ActorRef[R]

A ref with no actor behind it, whose tell completes one ask.

It is addressable rather than anonymous because a reply may come back over a link and has to find its way. Every promise has a path under /system/promises and stays registered there while the ask is running, so a reply_to that crossed a link resolves back to the future being awaited.

address property

The canonical address of the system the asker is running in.

future property

The reply, once there is one.

__init__(*, path, runtime, validate, expected, target)

Create a promise for one ask.

Parameters:

Name	Type	Description	Default
path	ActorPath	Where this promise is addressed, under /system/promises.	required
runtime	ActorRuntime	The system slice, for the loop and for dead letters.	required
validate	MessageValidator	The reply check, resolved exactly as a cell's is.	required
expected	str	The expected reply type, as it reads in an error.	required
target	ActorPath	The actor being asked, named in every error.	required

notify_terminated(ref)

Fail the ask because the actor it was waiting on stopped.

This is why an ask watches its target. Without it, a caller asking an actor that has already stopped waits out the full timeout for a reply that cannot arrive.

Parameters:

Name	Type	Description	Default
ref	ActorRef[Any]	A ref to the actor that stopped.	required

notify_unreachable(ref, detail)

Fail the ask because the peer holding its target went out of reach.

An actor watching would be told Terminated and could not tell the two apart. A caller can: an actor that stopped will not answer, while an actor behind a partition may be running and answering somebody else. Retrying elsewhere is reasonable for one and not the other, so the errors are different.

Parameters:

Name	Type	Description	Default
ref	ActorRef[Any]	A ref to the actor that can no longer be reached.	required
detail	str	Why the peer is considered gone.	required

offer(message) async

Reply, waiting for capacity that a promise never lacks.

A promise holds one future rather than a mailbox, so there is nothing to fill and nothing to wait for. This is tell.

Parameters:

Name	Type	Description	Default
message	R	The reply.	required

settle()

Close this promise, whatever became of the ask.

After this a reply is a dead letter. The future is not abandoned: a pending one is cancelled, and an exception that arrived just as the caller gave up is retrieved here. An unretrieved one would be reported by asyncio at collection time as if something had gone unhandled.

The promise also stops being addressable, so an ask leaves nothing in the ref registry however it ended.

tell(message)

Reply to the ask this promise stands for.

Safe to call from any thread, like every other tell. Validation does not run here, unlike every other tell. A reply of the wrong type is the asker's problem, not an exception to raise into whoever answered, so the check runs on the loop and its failure goes to the awaiting caller.

Parameters:

Name	Type	Description	Default
message	R	The reply. The first one wins. A later one is a dead letter, and so is any reply to an ask that has timed out.	required

Timers and stash

Bases: Generic[T]

The handle Behaviors.with_timers gives a behavior.

One scheduler serves every incarnation of its actor. A restart cancels the timers it holds rather than replacing the scheduler, so the behavior built by the factory schedules against the same object and nothing from the previous incarnation survives.

keys property

The keys of every timer currently running.

__init__(cell)

Bind the scheduler to the cell whose timers it owns.

__repr__()

Render the actor and the timers it currently has running.

cancel(key)

Stop a timer. Cancelling one that is not running is harmless.

A tick already on the mailbox is not retracted. By then it is a message like any other, and pulling one back out of a queue the actor is reading would be a different guarantee.

Parameters:

Name	Type	Description	Default
key	str	The timer to stop.	required

cancel_all()

Stop every timer this actor has running.

Called by the cell on restart and on termination. This is what keeps a tick from an old incarnation from reaching the one that replaced it.

is_active(key)

Whether a timer is running under this key.

Parameters:

Name	Type	Description	Default
key	str	The timer to ask about.	required

Returns:

Type	Description
bool	Whether it is running. A single timer that has already fired is
bool	not, and neither is one that was cancelled.

start_fixed_delay(key, message, interval, *, initial_delay=None)

Send a message repeatedly, waiting interval between sends.

The gap is measured from one send to the next, so an actor that falls behind does not build up a backlog of ticks. The timer just sends less often. Use this one by default. It is the only one that is safe on a bounded mailbox under load.

Parameters:

Name	Type	Description	Default
key	str	What to call this timer.	required
message	T	What to send, checked now as for start_single.	required
interval	timedelta	How long to wait between sends. Greater than zero: a repeating timer with no gap is a busy loop, not a schedule.	required
initial_delay	timedelta | None	How long to wait before the first send. The interval when omitted. Zero is fine here, and means send at once.	None

Raises:

Type	Description
MessageTypeError	If the message does not match this actor's declared message type.
ValueError	If the interval is not positive, or the initial delay is negative.
ValidationError	If content validation is on and the message does not satisfy its own model.

start_fixed_rate(key, message, interval, *, initial_delay=None)

Send a message on a schedule, keeping the long-run average rate.

Ticks are counted off a fixed schedule rather than from the last send, so time lost to a slow handler is made up. After a stall the missed ticks are sent one after another. That is the point of it, and also the risk: the burst arrives at an actor that has just shown it is not keeping up. Prefer start_fixed_delay unless something downstream is really counting the ticks.

The burst is capped at ten ticks. A longer stall drops what it missed, logs how many, and picks the schedule up from the clock, because thousands of ticks whose moment has passed are worth less to the actor than the mailbox space they take.

Parameters:

Name	Type	Description	Default
key	str	What to call this timer.	required
message	T	What to send, checked now as for start_single.	required
interval	timedelta	The scheduled gap between sends. Greater than zero: with no gap every tick is already due, so the timer would never yield and no other actor would run again.	required
initial_delay	timedelta | None	How long to wait before the first send. The interval when omitted. Zero is fine here, and means send at once.	None

Raises:

Type	Description
MessageTypeError	If the message does not match this actor's declared message type.
ValueError	If the interval is not positive, or the initial delay is negative.
ValidationError	If content validation is on and the message does not satisfy its own model.

start_single(key, message, delay)

Send one message to this actor after a delay.

Parameters:

Name	Type	Description	Default
key	str	What to call this timer, for cancelling or replacing it.	required
message	T	What to send. Checked against this actor's declared type now rather than when it fires, so a mistake surfaces in the handler that scheduled it.	required
delay	timedelta	How long to wait.	required

Raises:

Type	Description
MessageTypeError	If the message does not match this actor's declared message type.
ValueError	If the delay is negative.
ValidationError	If content validation is on and the message does not satisfy its own model.

Bases: Generic[T]

The handle Behaviors.with_stash gives a behavior.

One buffer serves every incarnation of its actor. A restart empties it rather than replacing it, because messages stashed by the state that just failed are not the new state's to answer. What is discarded is published as a dead letter, not dropped.

capacity property

How many messages this buffer can hold.

is_empty property

Whether it is holding nothing.

is_full property

Whether one more message would not fit.

size property

How many it is holding now.

__init__(capacity)

Create an empty buffer.

Parameters:

Name	Type	Description	Default
capacity	int	How many messages it can hold.	required

Raises:

Type	Description
ValueError	If the capacity is not at least one.

__repr__()

Show how full it is.

stash(message)

Put a message aside to be replayed later.

Parameters:

Name	Type	Description	Default
message	T	The message to hold. It is held as it is, so what is replayed is the object the sender passed.	required

Raises:

Type	Description
StashOverflowError	If the buffer is full. It is raised in the actor that stashed, because only that actor knows whether to drop the message, reject it, or let the failure become a supervision decision.

take_all()

Empty the buffer and return what it held, oldest first.

For the runtime, which does the replaying. Application code returns unstash_all(...) and lets the cell call this.

unstash_all(behavior)

Replay everything held, then continue as behavior.

The held messages go back to the front of the mailbox in the order they arrived, ahead of anything that has queued up since, and the buffer is left empty.

Parameters:

Name	Type	Description	Default
behavior	Behavior[T]	What the actor becomes for the replay and afterwards. Usually the state that is now ready, which is why the messages were held.	required

Returns:

Type	Description
Behavior[T]	A behavior to return from a handler.

Message adapters

Bases: ActorRef[U]

A ref that translates what it is told and delivers it to one actor.

Handed out by ActorContext.message_adapter. It behaves like any other ref. It never blocks, it is safe to use from any thread, and it stays a valid handle after its actor dies. What it cannot deliver becomes a dead letter reporting the message the sender sent.

It is not an actor. It has no mailbox, no cell and no children, so it cannot be watched or asked. Watch the actor that owns it instead.

An adapter lives until its owner stops, or until somebody calls release. Most actors want one adapter per foreign protocol, made once in setup, and never release it. An actor that makes one per request wants release, since otherwise every request leaves an entry in the system's ref registry for as long as the actor runs.

address property

The canonical address of the system the owning actor runs in.

An adapter is addressable like the actor behind it. Without this it would write itself down with no host, and a peer handed one in a reply_to would read it as a ref with nowhere to dial.

is_released property

Whether this adapter has been released and now delivers nothing.

__init__(*, cell, path, adapt, validate)

Bind an adapter to the actor it delivers into.

Parameters:

Name	Type	Description	Default
cell	ActorCell[Any]	The owning actor.	required
path	ActorPath	Where this adapter is addressed, under its owner.	required
adapt	Adapt	Translates an accepted message into the owner's own.	required
validate	MessageValidator	Checks an arriving message against the type this adapter declares it accepts.	required

__repr__()

Render the adapter, its owner, and what it translates with.

offer(message) async

Accept a message, waiting for the owner's mailbox to have room.

Parameters:

Name	Type	Description	Default
message	U	The message to translate and deliver.	required

Raises:

Type	Description
MessageTypeError	If it does not match the type this adapter accepts.
RuntimeError	If called off the system's loop, as for any offer.
ValidationError	If content validation is on and the message does not satisfy its own model.

release()

Stop this adapter, without stopping the actor behind it.

For an actor that hands out an adapter per request rather than one per protocol. Each adapter is addressable, so each is an entry in the system's ref registry, and nothing releases one on its own: they are bound to the actor rather than to an incarnation, which is what keeps a restart from turning replies into dead letters. That is the right default and the wrong one for a short-lived adapter, so this is the way out of it.

Afterwards the ref stops resolving and what is told to it becomes a dead letter, exactly as sending to a stopped actor does. Calling it twice is harmless.

tell(message)

Accept a message, to be translated and delivered to the owner.

The split is the same as any other send. An error about the message raises here, on the calling thread, because the sender wrote it. Errors about the recipient become dead letters, and so does a translation the owner never gets to run.

The translation does not happen here. It is the owner's code, so it runs in the owner. A failure in it is then the owner's supervision decision, not an exception in a caller that has never heard of this adapter.

Parameters:

Name	Type	Description	Default
message	U	The message to translate and deliver.	required

Raises:

Type	Description
MessageTypeError	If it does not match the type this adapter accepts.
MailboxFullError	If the owner's mailbox is full under OverflowStrategy.FAIL and the caller is on the system's loop.
ValidationError	If content validation is on and the message does not satisfy its own model.

Routers

Factories for routers, as Behaviors is for behaviors.

group(msg_type, *, path, role=None, strategy=None) staticmethod

Spread work over an actor published on every member of a cluster role.

proxy = ctx.spawn(Routers.group(Job, role="worker", path="/user/worker"))

Where a pool owns its routees, a group router routes to whatever actor each member of role publishes at path, and the pool follows membership: a member that joins is added, and one that is removed or goes unreachable is dropped within a convergence. An empty group holds and dead-letters rather than stopping, because the next member to arrive is what it is waiting for.

The routee on each node is addressed by its bare path, so it has to be published as a well-known name there, with system.refs.register_well_known(ref). Unlike a pool, this needs a clustered system: the routees are discovered from membership.

The message type is named rather than read off a routee, because the routees are on other nodes and there is no spawned child here to read it from.

Parameters:

Name	Type	Description	Default
msg_type	MessageType	What the routees accept, and what this router forwards.	required
path	str	The path the routee is published at on each member.	required
role	str | None	The role a member must carry to take a share. None routes to every member.	None
strategy	RoutingStrategy | None	How to choose between routees. Round-robin when omitted.	None

Returns:

Type	Description
Behavior[Any]	A behavior to spawn.

pool(size, behavior, *, strategy=None, routee_mailbox=None) staticmethod

Spawn size copies of a behavior and spread work over them.

workers = ctx.spawn(Routers.pool(8, worker()), name="workers")

The router accepts exactly what a routee accepts. It reads the type from the routees it spawned rather than being told it again, so the two cannot drift apart.

Pass a stateful routee as Behaviors.setup(...) or another factory. Every routee starts from the same object, so an already-built behavior holding state would be shared by the whole pool.

The router is the routees' parent, so their failures are supervised the ordinary way. Wrap behavior in Behaviors.supervise(...) and a routee that fails is restarted in place, with the pool unchanged. Without that it stops and leaves the pool, and when the last routee goes the router stops too.

Parameters:

Name	Type	Description	Default
size	int	How many routees to spawn. At least one.	required
behavior	Behavior[T]	What each routee does.	required
strategy	RoutingStrategy | None	How to choose between them. Round-robin when omitted.	None
routee_mailbox	MailboxConfig | None	Capacity and overflow behaviour for each routee. The system default when omitted. A bounded routee that fills up dead-letters what it cannot take, rather than failing the pool. Put backpressure on the router's own mailbox instead, where a sender can offer into it and wait.	None

Returns:

Type	Description
Behavior[T]	A behavior to spawn.

Raises:

Type	Description
ValueError	If size is below one.

Bases: Protocol

Chooses which routee a message goes to.

Called on the router's own receive loop, one message at a time. An implementation holding state therefore needs no locking, and it must not block.

select(routees, message)

Pick a routee.

Parameters:

Name	Type	Description	Default
routees	Sequence[ActorRef[T]]	The live routees, never empty. The pool shrinks as routees stop, so this is the pool as it is right now.	required
message	Message	The message being routed, for a strategy that reads it.	required

Returns:

Type	Description
ActorRef[T]	One of the routees.

Hand each message to the next routee in turn.

It keeps a counter rather than a position, so removing a dead routee shifts the rotation instead of restarting it. An actor that has just received work does not receive more straight away because the pool shrank.

__init__()

Start the rotation.

__repr__()

Render how far round the rotation has gone.

select(routees, message)

Return the next routee in the rotation.

Blocking calls

The pool that blocking calls run on, so they do not run on the loop.

One blocking call stalls every actor in the system. They share a loop, and a thread that is inside requests.get or a database driver is not running any of them. So a call that blocks goes to a thread, and the actor awaits the result like anything else.

The pool is per system and bounded. It is deliberately not asyncio.to_thread, which submits to the loop's default executor: that one is shared with every other library in the process and its size is not tapio's to choose, so blocking_pool_size could not be honoured. It is created on the first call, so a system that never blocks starts no threads at all.

Threads are the one piece of the runtime that is not a task, which makes them the one piece the leak invariant does not cover for free. The system shuts the pool down against the same deadline as the actor tree, and assert_no_leaked_threads() is the companion check.

A blocking call cannot be cancelled. Python has no way to interrupt a thread that is inside a C call or a sleep. Shutdown drops work that has not started and waits for what has, and past the deadline it says what is still running and gives up on it. That is the honest limit of running other people's blocking code, and it is why a call with no timeout of its own is a call that can outlive the system that made it.

BlockingPool

The threads one system runs blocking calls on.

is_accepting property

Whether the pool still takes work.

False once shutdown has begun, which is how a caller tells "the system is going away" from an error the call itself raised.

is_started property

Whether anything has been submitted yet.

A system that never blocks starts no threads, which is what keeps the thread-leak check meaningful for every other test in the suite.

size property

How many threads this pool may use.

threads property

The live threads this pool owns, by name.

Exposed for the same reason a cell exposes its watchers: a test has to be able to assert that shutdown left nothing running.

__init__(*, size, system)

Describe a pool without starting anything.

Parameters:

Name	Type	Description	Default
size	int	How many threads it may use.	required
system	str	The system it belongs to, which names its threads.	required

__repr__()

Render the bound and whether any threads exist yet.

shutdown(deadline, *, now) async

Drop queued work, then wait for what is running until the deadline.

Calling this twice is safe: the second call finds nothing to do.

Parameters:

Name	Type	Description	Default
deadline	float	The same clock reading the actor tree is racing.	required
now	Callable[[], float]	Reads that clock.	required

submit(loop, fn, /, *args, **kwargs)

Run a callable on a pool thread and return what to await.

Parameters:

Name	Type	Description	Default
loop	AbstractEventLoop	The system's loop, which the result is delivered on.	required
fn	Callable[P, R]	The blocking callable.	required
*args	args	Its positional arguments.	()
**kwargs	kwargs	Its keyword arguments.	{}

Returns:

Type	Description
Future[R]	A future for the call's result.

Raises:

Type	Description
RuntimeError	If the pool has been shut down. The caller turns this into an error about the system, since a blocking call during shutdown is an ordering bug in the same way a spawn is.

describe_blocking(fn)

Name a callable for a log line or an error message.

Messages and validation

Bases: BaseModel

Base class for messages: frozen, and re-validated on every delivery.

Subclassing this rather than BaseModel is a real constraint on user code, and it is what makes the delivery-time guarantee work. Pydantic defaults revalidate_instances to "never", so re-validating a plain BaseModel instance returns it untouched and re-checks no field. A library that advertised validation on delivery while inheriting that default would be shipping a no-op.

frozen=True is the other half. A message that has been sent is shared with its recipient, and changing it afterwards would be a data race that no amount of validation could catch.

Example

class Greet(Message):
    whom: str
    reply_to: ActorRef["Greeted"]

Bases: BaseSettings

Tunables for one actor system.

Every field can be set with an environment variable, upper-cased and prefixed: TAPIO_VALIDATE_ON_TELL=0, TAPIO_ASK_TIMEOUT=PT2S.

ask_timeout = timedelta(seconds=5) class-attribute instance-attribute

Default deadline for ActorRef.ask when the call does not give one.

blocking_pool_size = 16 class-attribute instance-attribute

Threads available to ctx.run_blocking.

A private, bounded pool rather than the loop's default executor, which is shared with every other library in the process and whose size tapio does not control, so a bound could not be honoured.

dead_letter_log_first = 10 class-attribute instance-attribute

Log this many dead letters in full, then switch to periodic summaries.

A dead actor in a hot send loop must not drown the log.

dead_letter_summary_interval = timedelta(seconds=60) class-attribute instance-attribute

How often to log a summary once dead_letter_log_first is spent.

default_mailbox property

The mailbox configuration a spawn gets when it asks for nothing.

default_mailbox_capacity = None class-attribute instance-attribute

User-lane capacity for new mailboxes; None means unbounded.

The system lane is always unbounded, whatever this says: a capacity limit that could refuse a stop signal would make shutdown unreliable.

default_mailbox_overflow = OverflowStrategy.FAIL class-attribute instance-attribute

What a bounded mailbox does when full, unless a spawn overrides it.

Never consulted while default_mailbox_capacity is None.

remote = None class-attribute instance-attribute

How this system is addressed from outside the process.

None means remoting is off, which is the default: a system that has not asked to be reachable is not. Refs it hands out still serialize, carrying the system name and no host, so a peer reading one can tell which system it names and that there is nowhere to dial.

shutdown_timeout = timedelta(seconds=10) class-attribute instance-attribute

One deadline for the whole tree, not a per-actor timeout.

Shutdown races a single clock, so worst-case shutdown time tracks this value rather than depth times timeout.

validate_on_tell = True class-attribute instance-attribute

Re-validate a message's contents on delivery, not just its type.

The type check is unconditional and cheap. This switch controls the expensive half, a full re-validation whose result is discarded, so the cost is measurable and tunable in one place rather than at call sites. Turning it off changes cost and nothing else: the recipient always receives the object the sender passed.

Addressing and the wire format

Where one actor system is, as far as other systems are concerned.

host and port are set together or not at all. Without them the address is unaddressable. It names a system, which is enough to tell a local ref from a foreign one, but there is nothing for a peer to dial.

host = None class-attribute instance-attribute

The canonical host peers dial, or None when remoting is off.

is_addressable property

Whether a peer could dial this address.

port = None class-attribute instance-attribute

The canonical port peers dial, or None when remoting is off.

system instance-attribute

The system name, which is also the first element of every path below it.

__post_init__()

Reject a half-written address and a name no path could hold.

__repr__()

Render as the string form.

__str__()

Render as tapio://orders@10.0.0.4:25520, or without the host part.

parse(text) classmethod

Read an address back from its string form.

Parameters:

Name	Type	Description	Default
text	str	tapio://orders or tapio://orders@10.0.0.4:25520.	required

Returns:

Type	Description
Self	The address.

Raises:

Type	Description
ValueError	If the text is not an address in that form.

Write a ref down as an address, a path and an incarnation uid.

Parameters:

Name	Type	Description	Default
address	Address	Where the ref's system is.	required
path	ActorPath	Where in that system the actor sits.	required

Returns:

Type	Description
str	The full string form, tapio://orders@10.0.0.4:25520/user/x#3.

Raises:

Type	Description
ValueError	If the path belongs to a different system than the address.

Read a ref's string form back into an address and a path.

Parameters:

Name	Type	Description	Default
text	str	The full string form, with or without the host part.	required

Returns:

Type	Description
Address	The address and the path, which still has to be resolved against a
ActorPath	system before it addresses anything.

Raises:

Type	Description
ValueError	If the text is not a ref string, or holds a name no actor path could hold.

Two registries: message types by key, and live refs by path.

Message types. A frame names its payload's type with a key, and that key is looked up in a dict. It is never an import path. Turning a dotted name that arrived on a socket into an importable object is remote code execution. An unregistered key becomes a dead letter naming the key, and nothing is imported to find out what it might have meant.

That table is append-only for the life of the process, on purpose. There is no deregister, no override and no reset, and a duplicate key raises at import rather than winning. A key is a promise about the wire: it has to mean the same type on both ends of a link, and on the node that reads a frame written before a restart. A type that could be swapped out under a key already in use turns a decoding error into a silently wrong object, which is exactly what naming types by key rather than by import path exists to prevent. Anything that wants to retire a key retires it by never sending it again.

Live refs. A path and an incarnation uid look up the ref to deliver into. Cells register when they start and deregister when they stop, so the registry holds exactly the live actors. A uid that no longer matches resolves to nothing rather than to whoever holds that path now. A system that has terminated leaves an empty registry behind, which the tests check.

Well-known names. An actor may also ask to be reachable by its bare path, with no uid. That is the opposite of the guarantee above, so it is opt-in and it exists for one situation: a peer that has to address something before it can know any uid. Bootstrapping a cluster is that situation, since a seed node is named by an address in a configuration file and nothing else. A ref that was written down always carries its uid, so nothing becomes bare by accident, and asking for a well-known name is a decision an actor makes about itself. The alias is dropped when its actor stops, in the same call that deregisters the ref, so the registry stays exactly as empty as it was before.

RefRegistry

The live refs of one system, by path and incarnation uid.

Unlike the message-type registry, this is not process-wide. Two systems in one process share nothing, so each keeps its own.

__init__()

Create an empty registry.

__len__()

How many live refs are registered, aliases not counted twice.

__repr__()

Render the size, and the aliases when there are any.

deregister(path)

Forget a path, whether or not anything was registered under it.

Any well-known name this actor held goes with it, so an alias cannot outlive the actor it names or survive into the next incarnation.

lookup(path)

Return the live ref at a path and uid, or None.

None covers both cases: nothing was ever there, or what was there has stopped and its uid will never be used again. A path with no uid finds only an actor that published itself as a well-known name.

names()

Every well-known alias currently published, without its uid.

The companion to paths, and exposed for the same reason. An alias that outlived its actor would hand a peer the next occupant of that path, which is the whole thing the incarnation uid exists to prevent, so a test has to be able to see that the alias went with the actor.

paths()

Every path currently registered, which is what a leak test reads.

Refs only. A well-known alias is a second key onto a ref that is already here, so counting it would report one actor twice. Read names for the aliases: a leak check wants both, because they are cleared by the same call and a bug in it would leave one of them behind.

register(ref)

Record a ref as the live occupant of its path and uid.

register_well_known(ref)

Also reach this actor by its bare path, whatever its incarnation.

For an actor a peer has to address before it can know any uid, which in practice means a bootstrap endpoint named in configuration. It is deliberately narrow: only actors that ask for it are reachable this way, and a ref written down elsewhere still carries its uid and still addresses one incarnation only.

Parameters:

Name	Type	Description	Default
ref	ActorRef[Any]	The actor to publish. The name is its own path without the uid, so an actor cannot claim somebody else's.	required

key_for_type(msg_type)

Return the wire key a message type was registered under, if any.

register_message(key=None)

Register a message type under the key that names it on the wire.

The default key is module.qualname, so the decorator usually takes no argument. Use the explicit form to rename or move a class without breaking a peer still running the previous version.

@register_message()
class Reserve(Message): ...

@register_message("orders.protocol.Reserve")
class ReserveV2(Message): ...

Parameters:

Name	Type	Description	Default
key	str | None	The wire key. module.qualname when omitted.	None

Returns:

Type	Description
Callable[[M], M]	The decorator, which returns the class unchanged.

Raises:

Type	Description
MessageRegistrationError	If the key is already taken. It is raised at import time, rather than letting the later class win silently.

registered_key(msg_type)

Return a message type's wire key, or say how to give it one.

Parameters:

Name	Type	Description	Default
msg_type	type[Message]	The type about to be written to a frame.	required

Returns:

Type	Description
str	The key it was registered under.

Raises:

Type	Description
MessageRegistrationError	If it was never registered.

type_for_key(key)

Return the message type registered under a wire key, if any.

The wire format: a length prefix, a JSON object, and no imports.

A frame is a 4-byte big-endian length followed by a JSON object:

{"v": 1, "to": "/user/checkout/session-7#3",
 "from": "tapio://web@10.0.0.9:25520",
 "t": "orders.protocol.Reserve",
 "p": {"sku": "X-1", "qty": 2,
       "reply_to": "tapio://web@10.0.0.9:25520/user/cart#11"}}

to carries no address, because a frame arriving on a link is addressed to the system that received it. from is the sending system, not a sending actor: a tell carries no sender, so there is no actor to name, and inventing one would be a guess. It is a diagnostic rather than a reply path. Replies go to the reply_to a message carries, which is a complete ref and is the only thing that ever addresses an actor.

from is what the sender claimed about itself, so a dead letter can report it beside the address the link was actually associated with. Those two agree on a healthy pair and disagree on a misconfigured one, which is worth being able to see.

t is a registry key and never an import path, for the reason tapio.remote.registry gives. The length prefix is checked before the body is read, so an oversized frame costs a header and a refusal instead of the memory the peer asked for.

Encoding is model_dump_json. Decoding is model_validate_json inside the reading system's deserialization context, so the contents check that validate_on_tell governs locally has already happened: a message off the wire is validated by construction, strictly, with no way to skip it.

LENGTH_PREFIX = 4 module-attribute

Bytes of big-endian length in front of every frame body.

Frame dataclass

One decoded frame, before its payload has become a message.

The payload stays as JSON text at this stage on purpose. Whether it can be built depends on the type key, and a frame naming a type this system has never heard of has to be reportable without building anything.

key instance-attribute

The payload's registry key.

payload instance-attribute

The payload, as the JSON text it arrived as.

sender instance-attribute

The sending system's canonical address, or None if the frame named none.

What the sender claimed, not what the link was associated with. A frame that arrived on one association claiming to come from another address is a misconfiguration worth being able to see rather than one to correct silently here.

to instance-attribute

The recipient, in the receiving system's own path space.

version instance-attribute

The sender's wire format version.

UndecodableFrame

Bases: Message

A frame that never became a message, so a dead letter can still report it.

Every other dead letter carries the message that was not delivered. A frame refused for its size, its version or an unknown type key has no message to carry. Reporting nothing would leave the failures a peer can cause as the only ones you cannot see.

sender = None class-attribute instance-attribute

Where the frame said it came from, when it parsed far enough.

The sending system's canonical address for a frame this system decoded, and the peer address of the association for one the transport refused before the decoder ever saw it.

size = 0 class-attribute instance-attribute

How many bytes arrived, which for a frame refused on its declared length is not the size it claimed. The claim is in the dead letter's detail.

type_key = None class-attribute instance-attribute

The registry key the frame named, when the frame parsed far enough.

decode(data, *, system, max_frame_bytes=None)

Read one complete frame, without building its payload.

Parameters:

Name	Type	Description	Default
data	bytes	The frame, length prefix included.	required
system	str	The reading system's name, which the recipient path takes.	required
max_frame_bytes	int | None	Refuse a frame declaring more than this, if given.	None

Returns:

Type	Description
Frame	The decoded frame.

Raises:

Type	Description
MessageDecodingError	If the frame is truncated, is not JSON, is not of a version this system speaks, or is missing a field.
FrameTooLargeError	If the declared body length exceeds the limit.

encode(message, *, to, sender=None, max_frame_bytes=None)

Write a message and its addressing into a length-prefixed frame.

Parameters:

Name	Type	Description	Default
message	Message	The message to send.	required
to	ActorPath	The recipient's path, including its incarnation uid.	required
sender	Address | None	The canonical address of the system sending it, recorded so a dead letter on the far side can name where the frame came from. It is not a reply path: a reply goes to the reply_to the message carries.	None
max_frame_bytes	int | None	Refuse a frame larger than this, if given.	None

Returns:

Type	Description
bytes	The complete frame, length prefix included.

Raises:

Type	Description
MessageRegistrationError	If the message's type has no wire key.
FrameTooLargeError	If the encoded frame exceeds max_frame_bytes.

format_target(path)

Write a path the way a frame carries it, with no address.

A frame arriving on a link is addressed to the system that received it, so the address would say nothing. The uid still travels, because it is what tells one incarnation of a path from the next.

Parameters:

Name	Type	Description	Default
path	ActorPath	The path to write.	required

Returns:

Type	Description
str	/user/checkout/session-7#3, or without the fragment when there is no
str	incarnation uid.

frame_length(prefix, *, max_frame_bytes=None)

Read a frame's declared body length, and refuse an oversized one.

Called with the length prefix alone, before the body is read, so a peer announcing a gigabyte costs this check instead of a gigabyte.

Parameters:

Name	Type	Description	Default
prefix	bytes	Exactly LENGTH_PREFIX bytes.	required
max_frame_bytes	int | None	The limit, if there is one.	None

Returns:

Type	Description
int	The body length in bytes.

Raises:

Type	Description
MessageDecodingError	If the prefix is not LENGTH_PREFIX bytes long.
FrameTooLargeError	If the declared length exceeds max_frame_bytes.

parse_target(system, text)

Read a path a frame carried back into a path in a named system.

Parameters:

Name	Type	Description	Default
system	str	Whose path space the text belongs to. That is the reading system for a recipient, and the sending one for a path that came back after crossing in the other direction, as a watch does.	required
text	object	The path as the frame carried it.	required

Returns:

Type	Description
ActorPath	The path.

Raises:

Type	Description
ValueError	If the text is not a path with an optional uid fragment.

receive_frame(data, *, context, dead_letters, max_frame_bytes=None, peer=None)

Take a frame off a link and deliver what is in it, or account for it.

This is the receiving half of remoting, and it never raises. Everything a peer can get wrong becomes a dead letter on this system's stream: a bad size, a bad version, an unknown type key, a payload that will not validate, a recipient that has stopped, and a message the recipient does not accept. Dead letters are not sent back. A link that just failed to deliver a message is not a link to report failures over, and a report over a working link would arrive long after the sender stopped caring.

Parameters:

Name	Type	Description	Default
data	bytes	One complete frame, length prefix included.	required
context	DeserializationContext	The system reading it, which resolves the refs inside.	required
dead_letters	DeadLetterOffice	Where anything undeliverable is accounted for.	required
max_frame_bytes	int | None	The size limit this system enforces, if any.	None
peer	Address | None	The address the frame arrived from, recorded on any dead letter so a subscriber can tell a missing actor from a missing node.	None

The ambient system a ref deserializes against.

Turning tapio://orders@10.0.0.4:25520/user/checkout#3 back into a working ref needs something a Pydantic validator cannot reach on its own: the system doing the reading. That system knows whether the address is its own, so it can hand back the live local ref rather than a proxy to itself, and it owns the association a foreign address resolves through.

So the receiving end sets an ambient context for the duration of a decode, and the ref validator reads it. Outside one there is no answer to give, and RefResolutionError says so: a ref is a handle into a live runtime, and there is no meaningful ref without one.

DeserializationContext

Bases: Protocol

What a ref needs from the system that is reading it.

address property

The reading system's own canonical address.

resolve_path(address, path)

Turn an address and a path into a ref that can be told things.

Never raises about the target. A stopped actor, a replaced incarnation, and a peer with no link all resolve to something whose tell produces a dead letter.

current_context()

Return the system refs are deserializing against, if there is one.

resolve_ref(text)

Turn a ref's string form into a live ref, against the ambient system.

Parameters:

Name	Type	Description	Default
text	str	The full string form of a ref.	required

Returns:

Type	Description
ActorRef[Any]	A ref for the reading system's own actor when the address is its own,
ActorRef[Any]	and a ref through the association for that address otherwise.

Raises:

Type	Description
RefResolutionError	If the text is not a ref string, or if no system is in scope to resolve it against.

use_context(context)

Make context the system refs deserialize against inside the block.

Parameters:

Name	Type	Description	Default
context	DeserializationContext	The system doing the reading.	required

Yields:

Type	Description
None	Nothing. The context is ambient for the duration of the block.

Bases: BaseSettings

Where this system listens, and how peers address it.

Nested under TapioSettings.remote rather than spread across it, so "is this system reachable from outside the process" is one is None check, not a handful of defaults that each look harmless on their own.

Bind and canonical are separate because they often differ. With containers, NAT or port mapping, the address a peer dials is not the one the socket is bound to. A ref always writes down the canonical one.

bind_host = '127.0.0.1' class-attribute instance-attribute

The interface to listen on. Loopback by default: a port that accepts frames naming actor paths and message types is a serious surface, and the default is set for someone who has not thought about it yet.

bind_port = 25520 class-attribute instance-attribute

The port to listen on. 0 takes whatever the OS hands out, and the canonical port then follows the one it bound, so a test or a sidecar that cannot pick a port in advance still advertises a dialable address.

canonical_host = None class-attribute instance-attribute

The host peers dial. bind_host when omitted.

canonical_port = None class-attribute instance-attribute

The port peers dial. bind_port when omitted.

handshake_timeout = timedelta(seconds=5) class-attribute instance-attribute

How long a link has to be dialled, accepted and handshaken.

One deadline for the whole opening, so a peer that accepts a connection and then says nothing costs this and not a parked task.

heartbeat_interval = timedelta(seconds=1) class-attribute instance-attribute

How often an idle association writes a heartbeat.

A link that carries traffic needs none of these; they exist so that silence can be told from a peer with nothing to say.

max_frame_bytes = 4 * 1024 * 1024 class-attribute instance-attribute

Refuse a frame larger than this, before its body is read.

outbound_capacity = 10000 class-attribute instance-attribute

Frames one association will hold for a peer that is not reading.

Backpressure against a socket, and deliberately not backpressure from the receiving actor: nothing in a fire-and-forget wire protocol can offer the latter. What overflows here goes to dead letters with the peer named.

secret = None class-attribute instance-attribute

The shared secret both ends prove they hold during the handshake.

Required to bind anywhere but loopback: a system that accepts frames naming actor paths and message types from any host that can reach the port, with nothing to prove, fails to start rather than serving strangers.

tls = None class-attribute instance-attribute

Certificates for the link, or None for plaintext.

unreachable_after = timedelta(seconds=10) class-attribute instance-attribute

How long a link may be silent before the peer is declared unreachable.

Nothing arriving for this long, heartbeats included, means the peer is gone as far as this system can tell. Every local watcher of an actor over there is told Terminated, the association is quarantined, and recovery is an explicit remote.reconnect. That verdict can be wrong: a partition, a long pause and an overloaded peer all look the same from one node, and resolving which it was needs membership and a quorum that a single system does not have. Set it well above the peer's heartbeat_interval.

Links and associations

The link: length-framed JSON over a TCP stream, with optional TLS.

A link carries two kinds of frame over the same stream, and tells them apart without parsing either. Message frames are what tapio.remote.codec writes, and they open with {"v":. Link frames are the transport's own, the handshake, the heartbeat and the death watch, and they open with {"link":. The reader can therefore pass a message frame straight on, and only parses the frames meant for itself.

Death watch travels as link frames rather than as messages because it is the runtime talking to itself. A watch is not addressed to an actor, carries no user payload, and must work whether or not the two systems have registered any message types in common.

Everything here is about bytes and sockets. What a frame means is tapio.remote.codec's job, and who it reaches is tapio.remote.association's.

ConnectionHandler = Callable[[asyncio.StreamReader, asyncio.StreamWriter], None] module-attribute

What a listening endpoint does with each accepted connection.

Synchronous on purpose. It runs as the connection is made, which is the one moment nothing can cancel, so it is where the endpoint takes ownership of the socket before it hands the reader to a task that a shutdown could cancel.

LINK_PREFIX = b'{"link":' module-attribute

What a link frame opens with, and a message frame never does.

Every link frame model below declares link as its first field, so model_dump_json emits it first and this prefix is a fact about the encoding rather than a hope about it.

FrameLink

One TCP connection, read and written a whole frame at a time.

Not thread-safe, and not meant to be. A link is read by one task and written by one actor, both on the system's own loop.

peer property

The socket address on the other end, for a log line.

The handshake establishes the dialable address of the system over there. This is only where the packets come from, which is not always the same thing, and is still what a reader wants in a log.

__aenter__() async

Return the open link.

__aexit__(exc_type, exc, tb) async

Close the link however the block ended.

__init__(reader, writer, *, max_frame_bytes)

Bind a link to an open stream pair.

Parameters:

Name	Type	Description	Default
reader	StreamReader	The read half.	required
writer	StreamWriter	The write half.	required
max_frame_bytes	int	Refuse an inbound frame declaring more than this, from its length prefix and before its body is read.	required

__repr__()

Render the socket on the other end.

close() async

Close the connection, ignoring how it ends.

A link is closed because something already went wrong or because the system is going away, and neither is improved by an error raised on the way out.

read_frame() async

Read one complete frame, prefix included.

Returns:

Type	Description
bytes	The frame, ready to be classified and handed on.

Raises:

Type	Description
FrameTooLargeError	If the declared length is over the limit. The body is not read, so a peer announcing a gigabyte costs only a header.
MessageDecodingError	If the prefix is malformed.
IncompleteReadError	If the peer closed, either cleanly between frames or part-way through one. Both mean the link is over, and the association reports them the same way.
OSError	If the connection failed.

read_link(timeout) async

Read one link frame, refusing anything else.

Parameters:

Name	Type	Description	Default
timeout	float	Seconds to wait for it.	required

Returns:

Type	Description
dict[str, Any]	The decoded object.

Raises:

Type	Description
MessageDecodingError	If what arrived was not a link frame.
TimeoutError	If nothing arrived in time.
IncompleteReadError	If the peer closed first.
OSError	If the connection failed.

write_frame(data) async

Write one complete frame and wait for the buffer to drain.

The drain is what makes a slow peer show up as a slow actor. The association waits here, its mailbox fills, and the overflow strategy decides what happens. The write buffer never grows without a limit.

Parameters:

Name	Type	Description	Default
data	bytes	One complete frame, length prefix included.	required

Raises:

Type	Description
OSError	If the connection failed.

write_link(message) async

Write one of the transport's own frames.

Parameters:

Name	Type	Description	Default
message	LinkFrame	The link frame to send.	required

Raises:

Type	Description
OSError	If the connection failed.

Heartbeat

Bases: LinkFrame

Proof that a silent peer is still there.

Written on an idle association every heartbeat_interval. The receiving end records when it arrived and does nothing else with it. What to conclude when heartbeats stop is the failure detector's decision, not the reader's.

link = 'heartbeat' class-attribute instance-attribute

The frame kind, first in the encoding so LINK_PREFIX holds.

Link

Bases: Protocol

One open connection, as everything above the transport uses it.

A protocol rather than the class, so that a test can put a link that drops, delays or swallows frames where a real one goes and exercise the failure detector without breaking anything real.

peer property

The socket address on the other end, for a log line.

close() async

Close the connection, ignoring how it ends.

read_frame() async

Read one complete frame, prefix included.

write_frame(data) async

Write one complete frame and wait for the buffer to drain.

write_link(message) async

Write one of the transport's own frames.

LinkFrame

Bases: BaseModel

Base for the transport's own frames, which no actor ever sees.

Unwatch

Bases: LinkFrame

Withdraw a watch. The pair of paths identifies which one.

link = 'unwatch' class-attribute instance-attribute

The frame kind, first in the encoding so LINK_PREFIX holds.

watchee instance-attribute

The actor being watched, in the receiving system's path space.

watcher instance-attribute

Who was watching, in the sending system's path space.

Watch

Bases: LinkFrame

Ask a peer to report when one of its actors stops.

link = 'watch' class-attribute instance-attribute

The frame kind, first in the encoding so LINK_PREFIX holds.

watchee instance-attribute

The actor to watch, in the receiving system's path space, uid included. A uid that no longer matches is answered at once, since that incarnation is already over.

watcher instance-attribute

Who is watching, in the sending system's path space. The receiver never resolves it. It is an opaque key that comes back on the answer, which is what keeps a watch from being a way to address actors on the watcher.

WatcheeTerminated

Bases: LinkFrame

Report that a watched actor has stopped.

Sent by the node that owns the actor, so it means the actor really did stop. A watcher that is told the same thing because the link went silent is being told a guess, and that one is decided locally and never travels.

link = 'terminated' class-attribute instance-attribute

The frame kind, first in the encoding so LINK_PREFIX holds.

watchee instance-attribute

The actor that stopped, in the sending system's path space.

watcher instance-attribute

Who was watching, in the receiving system's path space, as they said it.

bind(settings)

Bind and listen, synchronously, so the port is known before anything runs.

Binding here rather than inside the server task lets a system with bind_port=0 advertise a canonical address as soon as it is constructed. The first ref it hands out already names a port a peer can dial.

Parameters:

Name	Type	Description	Default
settings	RemoteSettings	Where to listen.	required

Returns:

Type	Description
socket	A listening socket, not yet accepting.

Raises:

Type	Description
OSError	If the address could not be bound.

client_ssl_context(tls)

Build the context this system uses when it dials a peer.

Parameters:

Name	Type	Description	Default
tls	TLSSettings	The certificate settings.	required

Returns:

Type	Description
SSLContext	A client context, presenting this system's own certificate so a peer
SSLContext	configured for mutual authentication can check it.

Raises:

Type	Description
OSError	If a certificate or key file cannot be read.

connect(host, port, *, max_frame_bytes, ssl_context) async

Dial a peer and return the link to it.

Parameters:

Name	Type	Description	Default
host	str	The canonical host the peer advertises.	required
port	int	Its port.	required
max_frame_bytes	int	The inbound frame limit for this link.	required
ssl_context	SSLContext | None	The client context, or None for plaintext.	required

Returns:

Type	Description
FrameLink	An open link, before any handshake.

Raises:

Type	Description
OSError	If the connection could not be made.

framed(body)

Put a length prefix in front of an encoded body.

is_link_frame(frame)

Whether a complete frame belongs to the transport rather than an actor.

Parameters:

Name	Type	Description	Default
frame	bytes	One complete frame, length prefix included.	required

Returns:

Type	Description
bool	True for the handshake and heartbeat frames this module writes.

link_body(frame)

Read a link frame's JSON object.

Parameters:

Name	Type	Description	Default
frame	bytes	One complete link frame, length prefix included.	required

Returns:

Type	Description
dict[str, Any]	The decoded object.

Raises:

Type	Description
MessageDecodingError	If the body is not a JSON object.

listen(handler, listener, *, ssl_context) async

Start accepting on an already-bound socket.

Parameters:

Name	Type	Description	Default
handler	ConnectionHandler	Called with the reader and writer of each accepted connection.	required
listener	socket	The socket returned by bind.	required
ssl_context	SSLContext | None	The server context, or None for plaintext.	required

Returns:

Type	Description
Server	The running server.

server_ssl_context(tls)

Build the context this system presents to peers that dial it.

Parameters:

Name	Type	Description	Default
tls	TLSSettings	The certificate settings.	required

Returns:

Type	Description
SSLContext	A server context, requiring a client certificate when cafile is set.

Raises:

Type	Description
OSError	If a certificate or key file cannot be read.

verify_bind_security(settings)

Refuse to listen beyond loopback with nothing for a peer to prove.

Parameters:

Name	Type	Description	Default
settings	RemoteSettings	The remoting configuration about to be used.	required

Raises:

Type	Description
InsecureRemoteConfig	If bind_host is not a loopback address and no secret is set.

The handshake: who is on the other end, and may they speak at all.

Before a single message frame crosses a link, both ends say who they are and prove they hold the shared secret. Three link frames, one round trip and a half:

server -> client   server-hello   protocol, nonce
client -> server   client-hello   name, address, uid, protocol, version, nonce, proof
server -> client   welcome        name, address, uid, version, proof

The side that was dialled says almost nothing first. A listening port answers anything that can reach it, so whatever the first frame carries is readable by a scanner for the cost of one connection. It carries a challenge and the protocol number, and the name, address, incarnation and release travel in the welcome, which is written only after the dialler has answered that challenge. Before this the server volunteered all four to anyone, which handed over the deployment's identity and the exact library release it runs.

The dialler still names itself in its answer, so the exposure is not symmetrical. It is not the same exposure either: a dialler chose that address, where a listener chose nobody. Closing the remaining half means a fourth frame and a second round trip on every connection, which is a real cost for a case that begins with dialling somewhere you did not mean to.

Both proofs are HMACs of the other side's nonce, so neither end can be replayed at the other, and a peer holding no secret cannot pass for one that does. Three things are established here:

• Protocol equality. Both ends must speak the same PROTOCOL_VERSION. A wire format that half works is worse than one that refuses, and it is cheap to check before something harder to diagnose goes wrong. The tapio version travels too, but only as a diagnostic: two nodes on different releases of the library talk to each other as long as neither release changed the wire, which is what makes a rolling deploy possible.
• The canonical address, used to address the peer and to key the association. It is what the peer advertises, not the socket it dialled from, since containers, NAT and port mapping routinely make those differ.
• The system uid, minted per incarnation. It makes a restarted peer a different peer rather than a slow one, which every later judgement about reachability rests on.

PeerIdentity dataclass

Who answered on the other end of a link.

address instance-attribute

The canonical address the peer advertises, which is what it is dialled by.

protocol instance-attribute

The wire protocol it speaks, which equals this system's or it got no further.

uid instance-attribute

The peer's incarnation uid. A new one means the old peer died.

version instance-attribute

The tapio version it runs, which may differ from this system's.

Kept for diagnostics. When two nodes disagree about something subtle, the first useful question is which releases they are running, and the answer should not require reading two deployment manifests.

accept(link, *, address, uid, secret, timeout) async

Handshake as the system that was dialled.

Parameters:

Name	Type	Description	Default
link	FrameLink	The freshly accepted connection.	required
address	Address	This system's canonical address, which the peer will dial.	required
uid	int	This system's incarnation uid.	required
secret	SecretStr | None	The shared secret, or None when nothing has to be proved.	required
timeout	float	Seconds allowed for the whole exchange.	required

Returns:

Type	Description
PeerIdentity	Who dialled in.

Raises:

Type	Description
HandshakeError	If the peer speaks a different version, fails the challenge, or sends something that is not the expected frame.
IncompleteReadError	If the peer closed first.
OSError	If the connection failed.
TimeoutError	If the peer stopped talking part-way through.

introduce(link, *, address, uid, secret, timeout) async

Handshake as the system that dialled.

Parameters:

Name	Type	Description	Default
link	FrameLink	The connection just opened.	required
address	Address	This system's canonical address, which the peer will dial to reply.	required
uid	int	This system's incarnation uid.	required
secret	SecretStr | None	The shared secret, or None when nothing has to be proved.	required
timeout	float	Seconds allowed for the whole exchange.	required

Returns:

Type	Description
PeerIdentity	Who answered.

Raises:

Type	Description
HandshakeError	If the peer speaks a different version, fails the challenge, or sends something that is not the expected frame.
IncompleteReadError	If the peer closed first.
OSError	If the connection failed.
TimeoutError	If the peer stopped talking part-way through.

An association: one link to one peer, and the actor that owns it.

Two systems associate on demand. The first message sent to an address creates the association, and every ref for that address then uses it. That is what makes "FIFO per association" a guarantee rather than a coincidence of how many connections happen to be open.

An association is an actor. Its writer is the cell's receive loop, its outbound buffer is the cell's bounded mailbox, its heartbeat is a cell timer, and its reader is one task the cell cancels when it stops. Remoting therefore adds no new rule about who owns a task, and the existing leak check covers it.

Delivery is at-most-once. No acks, no retries, no resend buffer: a frame written to a socket that then failed is lost, and it dead-letters here if the failure is visible from this side. Acks would make delivery at-least-once. That is not an improvement, only a different trade-off, and it would oblige every receiving actor to be idempotent. That belongs in the user's protocol, where they know what is safe to repeat.

An association also holds the death watches that cross it, in both directions: the local watchers of actors over there, and the local actors watched from over there. Both sets end when the association does. Watchers of a peer that went away are told Terminated, which is the one signal in the library that can be wrong, and tapio.remote.failure says why there is no better answer available to a single node.

AssociationMessage = Outbound | LinkOut | Beat | Close module-attribute

Everything an association actor accepts. None of it is user traffic.

Association

One link to one peer: the actor's state, and the reader behind it.

It is created in one of two ways, and only the start differs. Dialled, when this system sent to an address it has no link to. Adopted, when the peer dialled in and the handshake said who it was.

initiator property

Whose dial opened this link.

Both sides connecting at once is normal under load. Without a rule the pair keeps two connections, and FIFO per association stops meaning anything. The rule is address order, applied to this.

is_closing property

Whether this association has been asked to stop.

is_connected property

Whether a handshaken link is currently carrying frames.

peer property

The address on the other end, which keys this association.

peer_uid property

The peer's incarnation uid, or 0 before the handshake.

watched property

The local actors the peer is watching, one entry per watch.

watching property

The actors on the peer that something here is watching.

Exposed for the same reason a cell exposes its watchers: a test has to be able to assert that the watch was released.

__init__(*, host, peer, initiator, link=None, uid=0)

Create an association, before its actor exists.

Parameters:

Name	Type	Description	Default
host	AssociationHost	The endpoint that owns it.	required
peer	Address	The peer's canonical address, which keys this association.	required
initiator	Address	Whose address opened the link, which is how a simultaneous dial is resolved.	required
link	Link | None	An already-handshaken link, when the peer dialled in.	None
uid	int	The peer's incarnation uid, when the handshake established it.	0

__repr__()

Render the peer and whether a link is up.

adopt(link, uid)

Take over a link the peer opened, in place of the one in hand.

This is how the losing side of a simultaneous dial is resolved. The association survives. The queue, the mailbox, every ref pointing through it and every watch across it are unchanged, and only the socket underneath is swapped. Frames already written to the old link are at-most-once, like every other frame on a link that ended.

Parameters:

Name	Type	Description	Default
link	Link	The handshaken link to take over.	required
uid	int	The peer's incarnation uid, as that handshake established it.	required

behavior()

Build the actor that writes to this link and reads from it.

bind(ref)

Take the ref to the actor that owns this association.

close(detail)

Ask this association to stop, through its own behavior.

Parameters:

Name	Type	Description	Default
detail	str	Why, for the log and the dead letters that follow.	required

detach() async

Close the link when this association's actor never got to stop.

_release closes the socket on PostStop, which is the ordinary path. An association adopted so late that the endpoint's stop sweep had already passed it gets no PostStop, so its socket would be left for the garbage collector. The endpoint calls this on its own way down to close it, in the same reader-then-link order _release takes so a read in flight ends before the link does.

offer(message, frame, recipient) async

Queue a frame, waiting for room in the outbound buffer.

This is local backpressure against a socket that is not draining, and nothing more. It is not end-to-end backpressure from the actor on the other side, which no fire-and-forget wire protocol can give. Build that out of messages if you need it.

Parameters:

Name	Type	Description	Default
message	Message	The message the frame carries.	required
frame	bytes	The complete frame.	required
recipient	ActorPath	Where it was addressed.	required

report_terminated(watchee, watcher)

Tell the peer that one of this system's actors has stopped.

Parameters:

Name	Type	Description	Default
watchee	ActorPath	The actor that stopped, in this system's path space.	required
watcher	str	The peer's watcher path, exactly as the peer wrote it.	required

send(message, frame, recipient)

Queue a frame for the peer, or account for why it cannot be.

Never raises about the peer, just as a local tell never raises about a recipient. A full outbound buffer, a failed link and a stopped association are all things the sender can do nothing about, so they become dead letters naming the peer.

Parameters:

Name	Type	Description	Default
message	Message	The message the frame carries, for the dead letter.	required
frame	bytes	The complete frame.	required
recipient	ActorPath	Where it was addressed, in the peer's path space.	required

unwatch(watchee, watcher)

Withdraw a watch on an actor over there.

Harmless if there was none. It does not retract a Terminated that is already on its way, since by then it is as true as it was going to be.

Parameters:

Name	Type	Description	Default
watchee	ActorPath	The actor over there.	required
watcher	Watcher	Who was watching.	required

wait_connected(timeout) async

Wait until the link is up, for a caller that asked for it by hand.

Nothing else waits for a dial. A send queues behind one and a resolve starts none, so this exists for remote.reconnect, where a person or a supervisor decided to re-associate and wants to know whether it worked.

Parameters:

Name	Type	Description	Default
timeout	float	Seconds to wait.	required

Raises:

Type	Description
HandshakeError	If the link ended before it came up.
TimeoutError	If it had not come up in time.

watch(watchee, watcher)

Ask the peer to report when one of its actors stops.

Parameters:

Name	Type	Description	Default
watchee	ActorPath	The actor over there, uid included.	required
watcher	Watcher	What to tell when it stops, or when the peer goes out of reach, which from here look the same.	required

AssociationHost

Bases: Protocol

What an association needs from the endpoint that owns it.

address property

This system's canonical address, which peers dial and refs write.

dead_letters property

Where a frame that never left is accounted for.

dispatcher property

The loop this system runs on, and the reader task runs on.

events property

Where this system says that a peer went out of reach.

is_closing property

Whether this system is shutting down.

A link that ends because this end is going away is not news about the peer, so nothing is published about it.

settings property

How this system does remoting.

uid property

This system's incarnation uid, presented in every handshake.

close_link_later(link, peer)

Close a link nobody is going to use, in a task the endpoint holds.

deliver(frame, peer)

Hand an inbound message frame to the system that owns the recipient.

forget(association)

Drop an association that has stopped, so the next send dials afresh.

lookup(path)

Find a live local actor by path and incarnation uid, for a watch.

peer_ref(peer, path)

Build a ref to an actor on a peer, to name it in a Terminated.

quarantine(peer, detail)

Freeze an address: nothing sent, nothing dialled, until told otherwise.

wrap(link)

Put whatever sits between this system and its sockets in the way.

Nothing, in production. A test installs a wrapper here to drop, delay or swallow frames, which is how a partition is simulated without breaking anything real.

Beat

Bases: Message

A tick asking the association to prove the link is still there.

Close

Bases: Message

Ask an association to stop, because its link is over.

detail instance-attribute

What happened, for the log and for the dead letters that follow.

LinkOut

Bases: Message

One of the transport's own frames, queued behind whatever is in front.

A watch has to arrive after the messages sent before it and before the ones sent after, so it travels through the same mailbox as user traffic rather than jumping the queue. A frame that never leaves is dropped: there is no user message to account for, and a peer that cannot be written to is about to be declared unreachable anyway.

frame instance-attribute

The complete frame, length prefix included.

kind instance-attribute

Which link frame it is, for the log line if it has to be dropped.

Outbound

Bases: Carrier

One frame queued for a peer, with the message it was made from.

The frame is what travels. The payload comes along so that a frame which never leaves can report the message its sender sent, rather than this wrapper. Encoding happens at the send site, on the caller's thread, because an error about the message belongs to whoever wrote it.

frame instance-attribute

The complete frame, length prefix included.

recipient instance-attribute

Where it was addressed, in the peer's path space.

This system's remoting: the port, the associations, and the resolver.

It is constructed with an already-bound socket, so the canonical address is known before anything can serialize a ref. It starts once the system's guardians exist to hang its actors from.

address property

This system's canonical address, which peers dial and refs write.

associations property

The peers this system currently holds a link, or a dial, for.

Exposed for the same reason a cell exposes its watchers: a test has to be able to assert that the link was released.

dead_letters property

Where a frame that never left is accounted for.

dispatcher property

The loop this system runs on.

events property

Where this system says that a peer went out of reach.

is_closing property

Whether this system's remoting is shutting down.

peers property

Who decides which addresses this system may associate with.

StaticPeers until something replaces it: every address that was written down is a peer, minus the ones a detector here gave up on.

quarantined property

The peers this system has given up on and will not dial again.

An address stays here until reconnect clears it, which is the whole point: recovery from a peer declared unreachable is a decision somebody made, never something that happened while nobody was looking.

settings property

How this system does remoting.

uid property

This system's incarnation uid, presented in every handshake.

__init__(*, runtime, uid, deliver, listener)

Wire an endpoint to the system it belongs to.

Parameters:

Name	Type	Description	Default
runtime	ActorRuntime	The system slice this endpoint works through: its canonical address, its loop, its dead letters, its event stream, and the registry a frame's recipient is looked up in.	required
uid	int	This system's incarnation uid.	required
deliver	Callable[[bytes, Address], None]	Hands an inbound message frame to the system.	required
listener	socket	The socket already bound by open_listener.	required

__repr__()

Render the address and how many peers are associated.

association_for(peer)

Return the association for a peer, without dialling one.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required

Returns:

Type	Description
Association | None	The association, or None if this system holds none.

behavior()

Build the /system/remote actor: parent of every association.

clear_quarantine(peer)

Take an address off the list this system refuses to talk to.

Nothing is dialled. It says only that this system is willing to be associated with that peer again, which is what the other end of a healed partition needs before its dial can be accepted.

Each node gives up for itself, so each node relents for itself. A pair that gave up on each other is repaired by relenting on one side and dialling from the other:

beta.remote.clear_quarantine(alpha.address)
await alpha.remote.reconnect(beta.address)

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required

Returns:

Type	Description
bool	Whether it was quarantined at all.

close() async

Stop listening, and let the tree stop the links.

The associations are children of this endpoint's actor, so the sweep that reached here is already stopping them. What is left is the listener, the socket, any connection still mid-handshake, any link this endpoint refused and is still closing, and any association adopted so late that the sweep had already passed it. Nobody else owns those.

close_link_later(link, peer)

Close a link this endpoint is not going to use.

It gets its own task because closing waits for the transport, and the caller is a handshake that has nothing left to say. The task is held until it finishes, since the loop would not hold it for us.

deliver(frame, peer)

Hand an inbound message frame to the system that owns the recipient.

forget(association)

Drop an association that has stopped.

The next send to that peer creates a new one and dials again. Holding a stopped association would mean holding a link that is not there, and the only way to know whether the peer is back is to dial it.

forget_all(detail)

Close every association, as a link failure would one at a time.

For the tests that need a link to go away while the peer stays, which is the only way to show that a ref survives a failed link.

Parameters:

Name	Type	Description	Default
detail	str	Why, for the log and the dead letters that follow.	required

is_quarantined(peer)

Whether this system has given up on an address.

lookup(path)

Find a live local actor by path and incarnation uid.

Parameters:

Name	Type	Description	Default
path	ActorPath	The path a peer named, uid included.	required

Returns:

Type	Description
ActorRef[Any] | None	The live ref, or None when nothing is there. A uid whose
ActorRef[Any] | None	incarnation is over answers None, which is what stops a watch
ActorRef[Any] | None	from attaching to whoever holds that path now.

outbound(peer)

Return the association for a peer, dialling if there is none.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required

Returns:

Name	Type	Description
	Association | None	The association. Its link may still be coming up, and sends queue
	Association | None	behind it rather than waiting, so nothing blocks on a dial.
	Association | None	None once this system is shutting down, and for a peer it
refuses	Association | None	neither is a thing to dial, and what was being sent is
	Association | None	accounted for instead.

peer_ref(peer, path)

Build a ref to an actor on a peer, to name it in a Terminated.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required
path	ActorPath	Where in its tree the actor sat.	required

Returns:

Type	Description
ActorRef[Any]	A ref. It stays a valid handle after the actor it names has
ActorRef[Any]	stopped, exactly as a local one does.

quarantine(peer, detail)

Freeze an address, because this system decided the peer is gone.

Nothing is sent there and nothing is dialled, in either direction, until reconnect clears it. That is deliberate. Watchers have already been told that actors over there are gone, so a link coming quietly back would leave this system and its peer believing different things with no way to notice.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required
detail	str	Why, kept for the log and for reconnect.	required

reconnect(peer) async

Clear a quarantine and associate again, because someone decided to.

Recovery is never automatic. A peer declared unreachable may have been alive the whole time, and its watchers here were told otherwise, so resuming is a decision a person or a supervisor makes rather than something a timer does.

This repairs one end. A peer that gave up on this system at the same moment, which is what both sides of a partition do, refuses the dial until it has relented too. See clear_quarantine.

Refs held from before are not reusable: their uid belongs to a session that is over. Resolve again after this returns.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required

Raises:

Type	Description
ActorSystemTerminating	If this system is shutting down.
HandshakeError	If the peer could not be dialled, refused this system, or dropped the link before it carried anything.
TimeoutError	If the peer did not answer within handshake_timeout.

refusal(peer)

Why this system will not associate with a peer, if it will not.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required

Returns:

Type	Description
str | None	The words that explain the refusal, or None when the peer may
str | None	be dialled. They are what a dead letter carries, so a subscriber
str | None	reads why the message went nowhere rather than only that it did.

resolve_expecting(address, path, expect)

Return a ref checked against what the caller says the peer accepts.

Parameters:

Name	Type	Description	Default
address	Address	The peer's canonical address.	required
path	ActorPath	Where in its tree the actor sits.	required
expect	MessageType	What the caller declares that actor accepts.	required

Returns:

Type	Description
ActorRef[Any]	The ref.

resolve_peer(address, path)

Turn a foreign address and path into a ref that reaches it.

This is the peer resolver the system calls when a ref names another system, both at resolve and inside a decode. A ref that arrives in a message field therefore works with no setup, which is what makes a reply_to from a third system an ordinary send.

Parameters:

Name	Type	Description	Default
address	Address	The system the ref names.	required
path	ActorPath	Where in that system it points.	required

Returns:

Type	Description
ActorRef[Any] | None	A ref, or None when the address names nowhere to dial and the
ActorRef[Any] | None	system should account for the message instead.

set_link_filter(wrap)

Install what every link opened from now on passes through.

Parameters:

Name	Type	Description	Default
wrap	Callable[[FrameLink], Link] | None	Called with each new link, returning what to use in its place. None removes the filter. Links already open keep whatever they were wrapped in.	required

start(cell)

Begin accepting connections, under the endpoint's own actor.

Parameters:

Name	Type	Description	Default
cell	ActorCell[Any]	The endpoint's actor, which parents every association.	required

use_peers(peers)

Hand the question of who may be associated with to somebody else.

One system decides alone, so it decides from a table of its own. A clustered one decides from membership, where a peer is refused because the cluster downed it rather than because this node stopped hearing from it. The consequences are the same either way, which is why this replaces the answer and nothing else: an association is still refused, watchers are still told, sends still dead-letter.

Whatever was already refused is carried over, because those refusals were acted on. Watchers were told the actors over there are gone, and a peer that quietly became dialable again on a change of authority would leave two nodes believing different things.

Parameters:

Name	Type	Description	Default
peers	PeerProvider	The new authority.	required

wrap(link)

Put whatever sits between this system and its sockets in the way.

Nothing, in production: the link is returned as it came. A test installs a filter with set_link_filter to drop, delay or swallow frames, which is how a partition is simulated with no second machine and nothing real broken.

Parameters:

Name	Type	Description	Default
link	FrameLink	The link just opened or accepted.	required

Returns:

Type	Description
Link	The link the association will read and write.

Bases: ActorRef[T]

A ref to an actor on a peer system, reached through one association.

address property

The peer's canonical address, which is what this ref writes down.

__init__(path, *, outbox, validate, max_frame_bytes, runtime)

Bind a ref to a path on a peer, and to the link that reaches it.

Parameters:

Name	Type	Description	Default
path	ActorPath	Where the actor sits in the peer's tree, uid included.	required
outbox	Outbox	The association to that peer.	required
validate	MessageValidator	The sender-side check, resolved from what the caller declared the peer accepts.	required
max_frame_bytes	int	The size limit this system enforces on a frame.	required
runtime	ActorRuntime	The sending system's slice, which an ask needs: its loop, and the registry a reply finds its way back through.	required

__repr__()

Render the full string form: where this ref points is what names it.

ask(make, *, expect, timeout=None) async

Send one message across the link and await one reply.

The same call as a local ask, and the same promise behind it. The reply comes back addressed to /system/promises, which is why a promise is addressable at all. The target is watched for the duration, so an actor that stops and a peer that goes silent both fail the ask at once rather than after the full deadline.

Parameters:

Name	Type	Description	Default
make	Callable[[ActorRef[R]], T]	Builds the request from the ref the reply should go to.	required
expect	type[R]	The reply type, which is required.	required
timeout	timedelta | None	How long to wait. The system's ask_timeout when omitted.	None

Returns:

Type	Description
R	The reply, rebuilt from JSON, so it is equal to what the responder
R	sent and never the same object.

Raises:

Type	Description
AskTimeoutError	If no reply arrived in time.
AskTargetTerminated	If the actor over there stopped without replying.
AskTargetUnreachable	If the peer went out of reach, which may mean the actor is alive and unreachable rather than gone.
AskTypeError	If a reply arrived that was not an expect.
MessageTypeError	If the request does not match the type this ref was resolved with.
MessageEncodingError	If the request cannot be written to a frame.
RuntimeError	If called from a thread that is not running the system's loop.

offer(message) async

Send a message, waiting for room in the outbound buffer.

This is local backpressure against a socket that is not draining. It is not end-to-end backpressure from the receiving actor, which a fire-and-forget wire protocol cannot provide.

Parameters:

Name	Type	Description	Default
message	T	The message to deliver.	required

Raises:

Type	Description
MessageTypeError	If the message does not match the type this ref was resolved with.
MessageEncodingError	If the message cannot be written to a frame.
ValidationError	If content validation is on and the message does not satisfy its own model.

tell(message)

Send a message to the peer, without waiting and without blocking.

The type check here is against what the caller declared the peer accepts, which is a claim about the peer rather than knowledge of it. The check that decides runs on the receiving node, against the target actor's real message type, and a mismatch there dead-letters on that node. The sender's declaration and the receiver's protocol are deployed separately.

Parameters:

Name	Type	Description	Default
message	T	The message to deliver.	required

Raises:

Type	Description
MessageTypeError	If the message does not match the type this ref was resolved with.
MessageEncodingError	If the message has no wire key or no JSON representation. Raised before any I/O, since the message is the sender's.
FrameTooLargeError	If the encoded frame is over the size limit.
ValidationError	If content validation is on and the message does not satisfy its own model.

watch_target()

Return the peer, which is what a death watch on this ref goes through.

A death watch on an actor that lives on another node.

It stands where a cell stands for a local watch, so ctx.watch is one call whichever node the actor is on. What it can promise is weaker. A local cell knows whether its actor is alive; this knows only whether the peer is still being talked to, and a peer that goes silent produces the same Terminated as one whose actor really stopped.

is_alive property

Whether a watch registered now could still produce a signal.

It says nothing about the actor. A quarantined peer answers False, because nothing will be sent there and nothing will come back, so the watcher is told at once instead of waiting forever.

path property

Where the watched actor sits in the peer's tree.

__init__(outbox, path)

Bind a watch to one actor on one peer.

__repr__()

Render the actor being watched, address included.

add_watcher(watcher)

Ask the peer to report this actor's death.

remove_watcher(watcher)

Tell the peer to stop reporting it.

Bases: BaseSettings

Certificates for a link, and optionally for the peer on the other end.

The shared secret proves who a peer is. TLS keeps the conversation private. They answer different questions, so they are configured separately, and both are recommended for anything crossing a machine boundary. A secret sent in plaintext protects the handshake and nothing after it.

cafile = None class-attribute instance-attribute

The authority peers' certificates are checked against.

Set on both ends for mutual authentication: a server with this set requires a client certificate, and a client with it set verifies the server's.

certfile instance-attribute

This system's certificate, presented to every peer.

check_hostname = True class-attribute instance-attribute

Whether a dialled peer's certificate must match the host dialled.

Off only for the deployments where the canonical host is not what the certificate names, which is a thing to know about rather than to discover.

keyfile = None class-attribute instance-attribute

The private key, when it is not in certfile.

The version of the wire contract, which is not the version of the library.

Two nodes have to agree about the shape of what crosses between them: the frame layout, the fields of a handshake, and what a link frame means. That agreement is this number, and it changes when the contract changes.

It is deliberately not tapio.__version__. The package version moves for a fixed docstring, a faster mailbox and a new supervisor strategy, none of which a peer can observe. Pinning a link to it would make every release a flag day: during any rolling deploy, half the nodes would refuse the other half, and a patch release would be undeployable without stopping the fleet. So the handshake checks this number, the hellos carry the package version as a diagnostic, and 0.1.1 talks to 0.1.0 exactly as long as neither changed the wire.

Equality is still required rather than negotiated. A wire format that half matches corrupts a session instead of refusing one, and a peer speaking a protocol this node has never seen cannot be reasoned about. What changed is which number gets that treatment.

Raising it is a decision with a deployment cost attached, so it deserves a sentence in the pull request that does it. Adding an optional field to a frame does not change the contract, because a reader that does not know the field ignores it. Removing a field, changing what one means, or adding one the reader must understand does.

PROTOCOL_VERSION = 1 module-attribute

What this node speaks, on the wire and in a handshake.

It appears in every frame as v, and in both hellos, and a peer that answers with a different number is refused before anything else is read.

One contract change has happened without raising this, and it is recorded here because the rule above says it should have. The handshake stopped volunteering a system's identity to anything that could open a connection: the name, address, incarnation uid and release moved out of the server-hello and into the welcome, in the release that became v0.6.0. By the rule that is a contract change, since a reader has to know where those fields now are.

The reason first written here for leaving it at 1 was that nothing was deployed to be incompatible with. That was wrong. v0.5.0 and every release before it were already tagged and published, and they speak the old handshake while calling it protocol 1.

It stays at 1 anyway, for a reason that survives checking. A version number gates compatibility, and it can only gate a change that has not shipped yet. Raising it now would not reach v0.5.0, which is frozen at 1 and unreachable whatever this says. What it would reach is every release from v0.6.0 onwards, which all speak the same handshake as this one and interoperate today: they would start refusing the next release over a wire format that never changed. That is the flag day the paragraph above exists to prevent, and paying it to improve an error message for releases that have been unreachable since v0.6.0 is the wrong trade.

So two nodes either side of that change both say 1 and fail at the frame rather than at the number: a malformed server-hello or a malformed welcome, instead of a protocol mismatch naming both versions. If you are reading this while debugging exactly that, the answer is that one end predates v0.6.0, and the fix is to upgrade it rather than to look for a version mismatch that will never be reported. The next contract change raises the number, and that one will gate something.

Clustering

One node's membership in a cluster: how it joins, and what it sees.

address property

This node's canonical address, in the form members are named by.

gossip_rounds property

How many times this node has sent its view to a peer.

One round is one peer, so this counts sends rather than messages multiplied by members. It is here because "how long did convergence take" is a question about rounds rather than about seconds.

heartbeats_sent property

How many probes this node has sent to the members it watches.

One per watched member per round, which is what makes the traffic linear in the number of nodes rather than quadratic.

leader property

The address of the node allowed to act, as this node computes it.

management_address property

Where an operator reaches this node, or None if management is off.

The host and port actually bound, in host:port form, which is what a test that asked for port 0 reads to find the port it got.

members property

Every member that has not been downed or removed, in address order.

monitored property

The members this node watches, in address order.

Every node sorts the member addresses, finds itself, and watches the few that follow it, so every member is watched by that many others however little traffic there is between them.

self_member property

This node as the cluster sees it, or None before it has joined.

state property

What this node currently believes about the cluster.

A snapshot of a value, so reading it twice and comparing is a fair question to ask. It is this node's view and not the truth: another node may believe something newer, and convergence is the condition under which they are known to agree.

__init__(system, settings=None, *, downing=None, terminate_on_down=False, management=None)

Start this node's cluster daemon, without joining anything yet.

Parameters:

Name	Type	Description	Default
system	ActorSystem	The system to cluster. It must have remoting configured, since members address each other by their canonical addresses.	required
settings	ClusterSettings | None	How often to gossip and how patient to be. The defaults when omitted.	None
downing	DownStrategy | None	What to do about an unreachable member. Omitted leaves one blocking convergence for ever, which is the safe default until an operator says how this cluster would rather resolve a split. Pass a strategy such as KeepMajority or DownAll to have the losing side downed and, for this node, to have it down itself.	None
terminate_on_down	bool	Whether to shut the whole system down when this node downs itself, rather than leaving that to the application. A downed member may not rejoin as itself, so a service whose only reason to run was the cluster wants this. One that does other work leaves it off and awaits when_downed instead. It has no effect without a downing strategy, since nothing then downs this node.	False
management	ManagementSettings | None	A small HTTP surface an operator reaches this node on, to read its membership or to ask it to let a member leave or down one. Off when omitted, like remoting: a port that can down a member is opened on purpose, not by default. The tapio-cluster command speaks to it.	None

Raises:

Type	Description
ClusterError	If the system has remoting switched off, so it has no address other nodes could dial.
InsecureRemoteConfig	If management would answer beyond loopback with no token, the same refusal remoting makes about a secret.

__repr__()

Render this node's address and how many members it can see.

join_seed_nodes(seeds, *, timeout=None) async

Join the cluster the seeds are in, or form it if this is the first.

Every node passes the same list in the same order. A node asks each seed to let it in, and keeps asking until it sees itself in the gossip that comes back, because a join is delivered at most once like every other message. The first seed in the list, and only the first, may form a new cluster, and only after seed_form_after in which it has heard from nobody. That is the rule that stops a restart from producing a second cluster beside the first.

Parameters:

Name	Type	Description	Default
seeds	Sequence[str]	The seed addresses, in their string form, in the order every node lists them. This node's own address may be in the list and is skipped.	required
timeout	timedelta | None	How long to wait to reach Up. The join_timeout setting when omitted.	None

Returns:

Type	Description
Member	This node's member record, once the leader has accepted it.

Raises:

Type	Description
ClusterError	If the seed list is empty, or if this node has not reached Up within the timeout. Timing out does not stop the node trying: it keeps asking, and the caller decides whether to keep waiting or to give up on the process.

leave(*, timeout=None) async

Leave the cluster gracefully, and wait to be written off.

The member walks out through the lattice rather than vanishing: Leaving, then Exiting once every node has seen it, then Removed. Each step needs a converged view, so leaving takes as long as agreement takes, and every node ends up holding the same tombstone.

The system is not terminated. Ending the process is the application's decision, and doing it here would take the choice away from a node that is only leaving one cluster.

Parameters:

Name	Type	Description	Default
timeout	timedelta | None	How long to wait to reach Removed. The leave_timeout setting when omitted.	None

Raises:

Type	Description
ClusterError	If this node never joined, or if it has not been removed within the timeout.

members_with_role(role)

The members that carry a role, oldest first.

A role is what a node says it is for, fixed when it joined and part of what the cluster agreed on. Cluster-aware features filter on these: ClusterSingleton places one instance among the members of a role, and a group router from Routers.group spreads work over them.

Ordered by seniority, the same definition of "oldest" the singleton and the downing strategies use, so taking the first is taking the same member they would.

Parameters:

Name	Type	Description	Default
role	str	The role to filter on.	required

Returns:

Type	Description
Member	Every member that has not been downed or removed and carries the
...	role, oldest first.

subscribe(subscriber, *events)

Have cluster events delivered to an actor's mailbox.

cluster.subscribe(worker, MemberUp, MemberRemoved, UnreachableMember)

The events are ordinary messages, so reacting to membership is behaviour switching and supervision like everything else. The subscriber must accept the events it asks for as part of its declared message type. It hears the current membership straight away, as the events that would have carried it, so an actor that subscribes after the cluster has formed still learns who is up before it hears the next change.

A subscriber that stops is forgotten, because the daemon watches it, so unsubscribe is only for an actor that wants to keep running and stop listening. Subscribing an actor again replaces what it asked for rather than doubling its events.

Parameters:

Name	Type	Description	Default
subscriber	ActorRef[Any]	The actor the events go to.	required
events	type[ClusterEvent]	Which event types to deliver, from tapio.cluster.events. None given means every one of them.	()

unsubscribe(subscriber)

Stop delivering cluster events to an actor that is still running.

Harmless if the actor was not subscribed. An actor that stops is forgotten without this.

Parameters:

Name	Type	Description	Default
subscriber	ActorRef[Any]	The actor to forget.	required

when_downed() async

Wait until a downing strategy has made this node down itself.

A downed member may not rejoin as itself, so the usual response is to shut the system down:

await cluster.when_downed()
await system.terminate()

Ending the process is left to the application on purpose, the same way leave leaves it, so a node that is only one part of a larger service decides for itself what a downing means for the rest of it.

It never returns when downing is switched off, since nothing then downs this node, and never returns for a node that leaves gracefully: leaving walks a member out through the lattice and is not a downing.

Cluster events: what a node tells an application about membership changes.

These are ordinary messages, delivered to an ordinary actor mailbox. Reacting to the cluster is therefore behaviour switching and supervision like everything else: an actor subscribes with Cluster.subscribe, and the daemon tells it an event the moment its own view of membership changes. There is no second event bus and no callback that runs on the daemon's turn. What the subscriber does with an event, it does in its own turn, on its own mailbox.

An event is this node's view rather than the truth. It is emitted when this node's membership state moves, so two nodes may see the same change a gossip round apart. That is the same guarantee the rest of clustering gives: a value merged pairwise, eventually consistent, never voted on.

None of these cross a link. They are built from gossip that already crossed one, so they are not registered on the wire, and a peer that sent one would be answered with a dead letter naming a key nothing is listening for.

ClusterEvent

Bases: Message

What the cluster tells a subscriber about a change in membership.

A base class so that a subscriber can accept every cluster event with one declared type, and so that Cluster.subscribe with no filter can mean "all of them". It carries no fields of its own.

LeaderChanged

Bases: ClusterEvent

The node this one computes as the leader changed.

The leader is a function of a converged view, not a post somebody holds, so this is emitted when the address that function returns changes, the empty cluster's None included.

leader instance-attribute

The new leader's address, or None when there is nobody to lead.

MemberLeaving

Bases: ClusterEvent

A member began leaving gracefully, so it is on its way out.

Emitted when a member first reaches leaving (or exiting, if this node's view skipped straight to it), one or more converged rounds before the removed that follows. A crashed or downed member never reaches here: it goes to down and is only ever seen as removed. This is what lets a predecessor let go before a successor computed from the removal starts, so the two do not overlap.

member instance-attribute

The member that is leaving, as this node last saw it.

MemberRemoved

Bases: ClusterEvent

A member reached removed, so it is gone and will not return as itself.

A member that leaves gracefully and one that is downed both end here, since what a subscriber does about a member that is no longer part of the cluster is the same either way.

member instance-attribute

The member, as it was last seen before the tombstone.

MemberUp

Bases: ClusterEvent

A member reached up, so it is a full member the cluster agreed on.

member instance-attribute

The member, with its roles and the order it was accepted in.

ReachableMember

Bases: ClusterEvent

A member that was unreachable is reachable again.

Every node that reported it unreachable has retracted, so the cluster as a whole can hear it once more.

member instance-attribute

The member that came back into reach.

SelfDown

Bases: ClusterEvent

This node was downed, so its membership is over.

A downed member may not rejoin as itself. The usual response is to shut the system down and come back, if at all, as a new incarnation. This is the same fact as ClusterDowned on the system event stream, delivered to a subscriber's mailbox instead so that reacting to it is ordinary message flow.

member instance-attribute

This node's own member record, at down.

UnreachableMember

Bases: ClusterEvent

A member became unreachable: at least one node cannot hear it.

An observation rather than a decision. The member is still up, and it blocks the leader from acting until a downing strategy resolves it or it answers again. What a subscriber does about that is the subscriber's call.

member instance-attribute

The member that went out of reach.

Build a manager that runs one instance of a behavior across the cluster.

ctx.spawn(ClusterSingleton(coordinator(), name="coordinator", role="worker"))

Spawn the same manager on every node. Each subscribes to membership, and the one on the oldest member of role runs behavior as an actor named name. When that member is removed, the next oldest takes over.

The instance is spawned fresh wherever it runs, so pass a factory such as Behaviors.setup(...), not an already-built behavior holding state: state that mattered on the old host does not cross to the new one, which is the honest shape of a singleton that survives its host going away. Supervise behavior the ordinary way for failures that do not end its node.

Parameters:

Name	Type	Description	Default
behavior	Behavior[Any]	What the singleton instance does.	required
name	str	The instance's actor name, under the manager that runs it.	required
role	str | None	The role whose oldest member hosts the instance. None places it among all members, so a cluster with no roles still has one.	None

Returns:

Type	Description
Behavior[_ManagerMessage]	The manager behavior, to spawn on every node.

Build a group router over an actor published on the members of a role.

Parameters:

Name	Type	Description	Default
msg_type	MessageType	What the routees accept, and so what this router forwards. A group router cannot read this off a routee the way a pool reads it off a child it spawned, because its routees live on other nodes, so it is named here.	required
path	str	The path the routee is published at on each member, such as /user/worker. It is resolved against each member's address, so the actor there must be reachable by that bare path, which means published as a well-known name.	required
role	str | None	The role a member must carry to take a share. None routes to every member.	None
strategy	RoutingStrategy | None	How to choose between routees. Round-robin when omitted.	None

Returns:

Type	Description
Behavior[Any]	The router behavior, to spawn.

Bases: Message

One system in the cluster, as every other system sees it.

address instance-attribute

Where the system is, and what it sorts by.

key property

What identifies this member: its address and its incarnation.

rank property

Where this member's status sits in the lattice.

roles = frozenset() class-attribute instance-attribute

What it says it is for. Every cluster-aware feature filters on these.

status = MemberStatus.JOINING class-attribute instance-attribute

Where it is in its life.

uid instance-attribute

Its incarnation. A restart at the same address is a different member.

up_number = 0 class-attribute instance-attribute

The order it was accepted in, which is what "oldest member" means.

Zero until the leader accepts it, so a member that is still Joining has no place in that order yet.

__repr__()

Render the address, incarnation and status, in that order.

merge(other)

Return what two views of the same member agree on.

The higher status wins, roles are unioned, and the higher up_number wins because zero means "not yet accepted". Each of those is a join, so the whole thing is one: order does not matter, and merging twice changes nothing.

Parameters:

Name	Type	Description	Default
other	Member	The other view. It must name the same member.	required

Returns:

Type	Description
Member	The merged member.

Raises:

Type	Description
ValueError	If the two are not the same member.

with_status(status, *, up_number=0)

Return this member at a new status.

It refuses to move backwards, because the lattice is the merge rule and a transition that contradicted it would be undone by the next gossip that arrived.

Parameters:

Name	Type	Description	Default
status	MemberStatus	Where the member is moving to.	required
up_number	int	The order it was accepted in, when the leader is accepting it. Kept as it was when zero.	0

Returns:

Type	Description
Member	The member at the new status.

Raises:

Type	Description
ValueError	If the new status is below the current one.

Bases: StrEnum

Where a member is in its life, from joining to gone.

The values are ordered, low to high, in the order they are declared here, and a merge takes the higher one: see Member.rank. WeaklyUp, which Akka has between Joining and Up, is deliberately absent: it lets a node join while another member is unreachable, at the cost of a member that half the cluster has agreed on, and every feature that places something has to know about it. It is worth adding when somebody has the problem it solves.

DOWN = 'down' class-attribute instance-attribute

Declared dead. A member that reaches this may not return.

EXITING = 'exiting' class-attribute instance-attribute

Leaving, and every node has seen it, so the handoff may run.

JOINING = 'joining' class-attribute instance-attribute

Contacted a seed, not yet accepted by the leader.

LEAVING = 'leaving' class-attribute instance-attribute

A graceful exit was asked for, and has not finished.

REMOVED = 'removed' class-attribute instance-attribute

Gone, and kept only as a tombstone so gossip cannot resurrect it.

UP = 'up' class-attribute instance-attribute

A full member.

Bases: Message

One node's view of the cluster, and the value that travels between nodes.

alive property

Every member that has not been downed or removed.

converged property

Whether every member that matters has seen this exact version.

A Down or Removed member neither blocks convergence nor takes part in it. Everyone else must be reachable and must have seen the version, so one unreachable member stops the leader from acting until somebody decides what to do about it.

leader property

The node allowed to act, when the view is converged.

The first member in address order whose status is Up or Leaving. Before anybody is Up, which is every cluster's first moment, it falls back to the first member in address order: somebody has to be able to accept the first join, and picking the same somebody on every node is all this rule has to do.

Returns:

Type	Description
str | None	The leader's address, or None when there is nobody to lead.

members = () class-attribute instance-attribute

Every member known, in address order, tombstones included.

reachability = Reachability() class-attribute instance-attribute

Who currently cannot hear whom.

seen = frozenset() class-attribute instance-attribute

Which nodes are known to have seen this exact version.

It is what makes convergence observable, and it is the one field that is about the spread of the state rather than about the state. A version that changes empties it, because nobody has seen the new one yet.

unreachable property

Every node a live observer currently cannot hear.

Judged only on observations by members that are still alive, so a record left behind by a downed member neither blocks convergence nor steers a downing strategy.

version = VectorClock() class-attribute instance-attribute

What orders two views of the cluster against each other.

__repr__()

Render the members and whether this view has converged.

bumped_by(address)

Return this state as a new version, produced by one node.

Every change a node makes to the state goes through here, which is what keeps the vector clock a record of who changed what. The seen set collapses to the node that made the change, because nobody else has seen this version yet.

Parameters:

Name	Type	Description	Default
address	str	The node making the change.	required

Returns:

Type	Description
Gossip	The new state.

founding(member) classmethod

Return the state of a node that has just formed a cluster alone.

Parameters:

Name	Type	Description	Default
member	Member	The founding member, which is already Up because there was nobody else to agree with.	required

Returns:

Type	Description
Gossip	The state.

member(address)

Return the member at an address, or None.

Parameters:

Name	Type	Description	Default
address	str	The address in its string form.	required

Returns:

Type	Description
Member | None	The member. When two incarnations of one address are known, which
Member | None	happens whenever a node has restarted, the live one is returned.
Member | None	Among incarnations that are all live, or all gone, the one that
Member | None	has gone furthest through its life wins, since that is the one a
Member | None	caller is deciding about.
Member | None	The live one has to win, because a restart leaves a tombstone
Member | None	behind and tombstones are kept forever. Ranking on status alone
Member | None	would answer with the dead incarnation from the restart onwards,
Member | None	and a caller asking about the member at an address is asking about
Member | None	the one that is running.

merge(other)

Return what two views of the cluster agree on.

Members are merged pairwise by the status lattice, reachability by observation version, and the clock by per-key maximum. Each of those is a join, so this is one: the same three laws hold for the whole state.

The seen set is the one field that is not simply joined, because it describes who has seen a version rather than what the state is. It is carried across only for the version it belongs to: kept when both sides are at that version, taken from whichever side is newer, and emptied when the merge produces a version neither side had seen.

Parameters:

Name	Type	Description	Default
other	Gossip	The other view.	required

Returns:

Type	Description
Gossip	The merged view.

observing(observer, observed, status)

Return this state with one node's opinion of another recorded.

The version is not touched, like with_member, so a caller making a change calls bumped_by as well.

Parameters:

Name	Type	Description	Default
observer	str	Who is watching, which is always the node recording it.	required
observed	str	Who is being watched.	required
status	ReachabilityStatus	What the observer now believes.	required

Returns:

Type	Description
Gossip	The new state.

primaries()

Return the primary member at every address, in one pass.

The same choice member makes for one address, made for all of them at once: the live incarnation wins over a tombstone, and among incarnations that are all live or all gone the one furthest through its life wins. It exists so a caller that needs every address does not call member in a loop, which is quadratic in the membership because each call rescans it.

Returns:

Type	Description
dict[str, Member]	The primary member keyed by address. Empty when there are no
dict[str, Member]	members.

seen_by(address)

Return this state recorded as seen by one more node.

Parameters:

Name	Type	Description	Default
address	str	The node that has now seen it.	required

Returns:

Type	Description
Gossip	The state. This one is unchanged.

with_member(member)

Return this state with a member added or replaced.

The version is not touched, so a caller that is making a change calls bumped_by as well. Keeping the two apart lets the leader apply several transitions and bump once.

Parameters:

Name	Type	Description	Default
member	Member	The member to record.	required

Returns:

Type	Description
Gossip	The new state.

Return the state after the leader has moved every member it may.

A pure function of a converged view, which is what makes it testable without a cluster: the caller decides whether the leader may act, and this decides what acting means. Each member moves exactly one step, and the caller bumps the version afterwards, so a leaving member walks Leaving to Exiting to Removed across separate converged rounds. That is what gives a handoff somewhere to happen, and it is why the steps are not collapsed even though the leader could see all three at once.

A member that reaches Removed is kept as a tombstone rather than dropped. Dropping it would let a peer holding an older view put it back by merging the record in again, since a merge unions the members it is given. Its reachability observations are kept for the same reason and for the same mechanism: they are ignored because their observer is no longer a live member, not deleted, since deleting them is not a join and a merge would undo it.

Parameters:

Name	Type	Description	Default
state	Gossip	The converged view.	required

Returns:

Type	Description
Gossip	The state after this round of transitions, or the same value when
Gossip	there was nothing to do.

Bases: Message

A counter per node, and the partial order they induce.

Nodes are named by their address string, so two systems that restart at the same host and port share a counter. That is deliberate at this level: the incarnation lives in the member record, and the clock only has to order the gossip states a node produced.

counters = Field(default_factory=dict) class-attribute instance-attribute

How many times each node has changed the state it gossips.

A node that has changed nothing is absent rather than present at zero. The two would mean the same thing and compare differently, and a merge whose result depended on which of them a peer happened to send would not be the same function run twice.

__repr__()

Render the counters in node order.

compare(other)

Say how this clock stands to another.

Parameters:

Name	Type	Description	Default
other	VectorClock	The clock to compare against.	required

Returns:

Type	Description
Ordering	The ordering. CONCURRENT when each has seen something the other
Ordering	has not, which is the case that forces a merge instead of a
Ordering	choice.

empty() classmethod

Return the clock of a node that has said nothing yet.

increment(node)

Return this clock with one node's counter moved on by one.

Parameters:

Name	Type	Description	Default
node	str	The node making the change, as its address string.	required

Returns:

Type	Description
VectorClock	The new clock. This one is unchanged.

merge(other)

Return the clock that has seen everything both of these have.

The per-key maximum, which is a join: commutative, associative and idempotent, so gossip may arrive in any order, twice, or out of order, and every node still computes the same clock.

Parameters:

Name	Type	Description	Default
other	VectorClock	The clock to merge with.	required

Returns:

Type	Description
VectorClock	The merged clock.

Bases: StrEnum

How two vector clocks stand to each other.

AFTER = 'after' class-attribute instance-attribute

The first happened after the second.

BEFORE = 'before' class-attribute instance-attribute

The first happened before the second, which knows everything it knows.

CONCURRENT = 'concurrent' class-attribute instance-attribute

Each has seen something the other has not, so neither one wins.

SAME = 'same' class-attribute instance-attribute

Neither has seen anything the other has not.

Bases: Message

Every current observation, as one mergeable value.

records = () class-attribute instance-attribute

One record per observer and observed pair, in a canonical order.

unreachable property

Every node that at least one observer currently cannot hear.

__repr__()

Render how many observations there are and who is unreachable.

empty() classmethod

Return the table in which everyone can hear everyone.

is_reachable(address, observers=None)

Whether every observer that has an opinion can hear this node.

One observer is enough to make a node unreachable, because a node that half the cluster cannot hear is not a node the cluster can converge with.

Parameters:

Name	Type	Description	Default
address	str	The node in question.	required
observers	frozenset[str] | None	The observers whose opinion still counts, or None to count every observer. A caller passes the live members here so that a record left by a member since downed no longer pins a healthy node unreachable, which is a claim nobody living is making and nobody can retract.	None

Returns:

Type	Description
bool	Whether nobody who still counts reports it unreachable.

merge(other)

Return the table that has seen every observation both of these have.

Per pair, the higher version wins. Versions tie only when two nodes made up different records for the same pair and version, which the protocol does not do, so the tie is broken by preferring UNREACHABLE: it keeps the merge a join, and it fails towards blocking convergence rather than towards pretending everything is fine.

Parameters:

Name	Type	Description	Default
other	Reachability	The table to merge with.	required

Returns:

Type	Description
Reachability	The merged table.

observing(observer, observed, status)

Return this table with one observation replaced.

The version is taken from the record being replaced and moved on by one, so the new observation beats the old one on every node it reaches and the order it arrives in does not matter.

Parameters:

Name	Type	Description	Default
observer	str	Who is watching.	required
observed	str	Who is being watched.	required
status	ReachabilityStatus	What the observer now believes.	required

Returns:

Type	Description
Reachability	The new table. This one is unchanged.

says(observer, observed)

What one node currently says about another.

Reachable when it has never said anything, since an observation is only recorded once there is something to report.

Parameters:

Name	Type	Description	Default
observer	str	Who is watching.	required
observed	str	Who is being watched.	required

Returns:

Type	Description
ReachabilityStatus	That observer's current opinion, and nobody else's.

unreachable_among(observers)

Every node an observer that still counts currently cannot hear.

The observer-filtered counterpart of unreachable. A record left by a member since downed is skipped, so a dead node's stale claim neither blocks convergence nor steers a downing strategy.

Parameters:

Name	Type	Description	Default
observers	frozenset[str]	The observers whose opinion still counts, which is the live members.	required

Returns:

Type	Description
frozenset[str]	The observed nodes at least one live observer cannot hear.

Bases: Message

One node's current observation about one other node.

observed instance-attribute

Who is being watched.

observer instance-attribute

Who is watching.

pair property

The two nodes this record is about, observer first.

status instance-attribute

What the observer currently believes.

version = 1 class-attribute instance-attribute

The observer's own counter for this pair, so a later view wins.

__repr__()

Render observer, observed and belief, which is the whole record.

Bases: StrEnum

What one node currently believes about another's reachability.

REACHABLE = 'reachable' class-attribute instance-attribute

Frames are arriving often enough to believe it.

UNREACHABLE = 'unreachable' class-attribute instance-attribute

They have stopped, and this observer has given up waiting.

The members one node watches, and what it currently believes about them.

It holds no timers and reads no clock: the daemon passes the time in, the same way it passes the membership in. That keeps the whole of "who is watched and what does silence mean" testable without a cluster.

peers property

The members this node watches, in address order.

__init__(*, address, size, detector)

Describe how one node watches its share of the cluster.

Parameters:

Name	Type	Description	Default
address	str	This node's address, which is its place on the ring.	required
size	int	How many peers to watch.	required
detector	Callable[[float], FailureDetector]	Builds a detector for a peer that has just been picked up, given the time it was picked up at. Injected so that phi-accrual can replace the fixed window without this class knowing.	required

__repr__()

Render how many peers are watched and which of them look gone.

follow(members, now)

Take up the ring the current membership implies.

Parameters:

Name	Type	Description	Default
members	Iterable[Member]	The live members.	required
now	float	The current time, which a peer picked up now is credited with. A member this node has never probed is not a silent one.	required

Returns:

Type	Description
str	The peers this node has just stopped watching, so the caller can
...	take back whatever it said about them. A claim left behind by a
tuple[str, ...]	node that no longer watches the member would block convergence
tuple[str, ...]	with nothing left to retract it.

heard(peer, at)

Record that a peer answered this node's probe.

Parameters:

Name	Type	Description	Default
peer	str	Who answered.	required
at	float	When, on the loop's monotonic clock.	required

Returns:

Type	Description
bool	Whether this node watches that peer. An answer from a peer it does
bool	not watch is late or is somebody else's business, and either way
bool	there is nothing to record.

link_lost(peer)

Record that the transport has given up on the link to a peer.

Remembered for any peer, not only a watched one, so that a peer this node picks up later starts from what the transport already knows.

Parameters:

Name	Type	Description	Default
peer	str	The peer in question.	required

Returns:

Type	Description
bool	Whether this node watches that peer.

link_open(peer)

Record that a link to a peer is up again.

The link coming back is what retracts the transport's verdict, and nothing else can: this node's own probe never asked the transport anything.

It does not count as an answer. A completed handshake proves a process is accepting connections, and this node is asking whether the daemon behind it is still replying. Feeding the detector here would let a peer whose links churn faster than the window stay reachable for ever without answering once, which is the failure this monitor exists to catch. The next probe settles it, one round later.

Parameters:

Name	Type	Description	Default
peer	str	The peer in question.	required

Returns:

Type	Description
bool	Whether this node watches that peer.

verdicts(now)

Say what this node believes about every peer it watches.

Parameters:

Name	Type	Description	Default
now	float	The current time, on the loop's monotonic clock.	required

Returns:

Type	Description
dict[str, ReachabilityStatus]	One belief per watched peer.

Return the members one node watches, by their place on the ring.

A pure function of the membership, so every node works out the same ring from the same view and nobody has to be told who watches whom.

Parameters:

Name	Type	Description	Default
address	str	The watching node's address.	required
members	Iterable[Member]	The members to arrange, which are the live ones. A member with two incarnations counts once, since the ring is over addresses and both records name the same place on it.	required
count	int	How many to watch. More than there are peers means all of them.	required

Returns:

Type	Description
str	The addresses to watch, in ring order starting after this node. Empty
...	when this node is not a member of the view it was given, which is
tuple[str, ...]	every node's first moment.

Bases: Protocol

What a cluster does when some members are unreachable.

Most implementations are a pure function of the view, so that the two sides of a partition reach an agreeing verdict from mirror-image inputs without a message passing between them. LeaseMajority is the one that reaches outside the view, which is why deciding is asynchronous.

decide(state) async

Return the addresses to down, given a view with unreachable members.

Parameters:

Name	Type	Description	Default
state	Gossip	This node's view. Called when its reachability shows at least one unreachable member, and treated as stable: the daemon waits for the split to settle before asking, so that a passing blip is not downed.	required

Returns:

Type	Description
frozenset[str]	The addresses to move to Down. Empty when nothing should be
frozenset[str]	downed. It contains this node's own address exactly when this node
frozenset[str]	is on the side that loses, which is how self-down is told from
frozenset[str]	downing a peer.

Down every member, so a split heals by the whole cluster restarting.

The one strategy that is always safe, because it keeps nothing: there is no surviving side that another surviving side could contradict. It trades the most availability for it, since a partition that a smarter strategy would have ridden out on the majority takes the majority down as well. It is the honest default when the operator cannot promise the cluster its size or its shape, and it is what the counting strategies fall back to when their own rule cannot name a single winner.

__repr__()

Render the class name; there is no state to show.

decide(state) async

Down everyone, or nobody when there is no split to resolve.

Parameters:

Name	Type	Description	Default
state	Gossip	This node's view.	required

Returns:

Type	Description
frozenset[str]	Every live address when a member is unreachable, and the empty set
frozenset[str]	otherwise.

Keep the larger side and down the smaller, tie broken by lowest address.

The usual choice for a cluster whose size drifts, since it needs to be told no number: it counts what it can see. Its safety rests on there being one majority, which holds as long as a split produces two parts. A split into three parts can leave every part a minority, and then this downs them all, which is DownAll arrived at by counting.

A tie, two sides of equal size, is broken by keeping the side that holds the lowest address. Both sides compute the same lowest address from the same membership, so both agree which side that is, which is the same reason the leader is the lowest address: a total order over the bytes both sides hold needs no round to settle.

role = None class-attribute instance-attribute

The role to count, or None to count every member.

__repr__()

Render the role, which is the whole of the configuration.

decide(state) async

Keep the majority side, downing the minority.

Parameters:

Name	Type	Description	Default
state	Gossip	This node's view.	required

Returns:

Type	Description
frozenset[str]	The minority side's addresses, or this side's own when this side is
frozenset[str]	the minority. Every live address when neither side counts anybody,
frozenset[str]	since a role that names nobody leaves no side to keep.

Keep a side only if it still holds a fixed number of members.

The operator names the number, and a side survives when it can count that many and the other side cannot. It is exact where KeepMajority is relative, which is what makes it safe for a cluster that changes size only when an operator says so: the quorum is set to more than half the largest the cluster is allowed to reach, and then two sides can never both hold it.

If the cluster outgrows that promise both sides can reach the quorum at once, and keeping either would be keeping a side the other contradicts, so this downs everything instead. The same happens when neither side reaches the quorum. Naming a number the cluster then exceeds is the one way to misconfigure this, and downing all is how it fails when that happens: loudly and toward stopping, not quietly toward a split brain.

role = None class-attribute instance-attribute

The role the quorum is counted over, or None to count every member.

size instance-attribute

How many members a side must hold to be kept. At least one.

__post_init__()

Refuse a quorum of nothing, which no side could fail to reach.

Raises:

Type	Description
ValueError	If the size is below one.

__repr__()

Render the quorum size and the role it is counted over.

decide(state) async

Keep the side that alone reaches the quorum, downing the rest.

Parameters:

Name	Type	Description	Default
state	Gossip	This node's view.	required

Returns:

Type	Description
frozenset[str]	The side that falls short, or every live address when both sides
frozenset[str]	reach the quorum or neither does.

Keep the side that holds the oldest member, and down the other.

The oldest member is the one accepted first, by its up_number, and there is exactly one, so both sides agree which side holds it and one of them is kept. It suits a cluster with a member that matters more than the others, a singleton's usual home, since keeping the oldest keeps that member's side running through a split.

Its weakness is the oldest member being cut off on its own. Then keeping the oldest's side keeps one node and downs the rest, which is the split brain resolver behaving worse than doing nothing. Setting down_if_alone guards that case: when the oldest is the only member on its side, it downs itself instead, and the larger side lives. Both sides see the same single node alone against the same larger group, so both still agree.

down_if_alone = False class-attribute instance-attribute

Whether an oldest member cut off on its own downs itself rather than the rest.

role = None class-attribute instance-attribute

The role the oldest is chosen among, or None to choose among every member.

__repr__()

Render whether a lone oldest yields, and the role it is chosen among.

decide(state) async

Keep the oldest member's side, unless it is alone and told to yield.

Parameters:

Name	Type	Description	Default
state	Gossip	This node's view.	required

Returns:

Type	Description
frozenset[str]	The side without the oldest member. This side's own addresses when
frozenset[str]	this side lacks the oldest, or when the oldest is alone and
frozenset[str]	down_if_alone is set. Every live address when a role names nobody
frozenset[str]	to be oldest.

Keep the side that takes an outside lease, so an even split still has one winner.

The strategy for a split the count cannot decide. Two equal halves have no majority, and every deterministic rule keeps both or neither, so this hands the decision to a Lease that only one side can hold. The side that takes it survives and downs the other; the side that cannot downs itself. Because the lease admits one owner, exactly one side lives, which is the guarantee no view-only rule can make about an even split.

Every node names its own side to the lease, by the lowest address on it, so all of a side asks with one owner and the lease's re-entrancy lets them all hold it or all fail together. The two sides name different owners, so the lease keeps them apart.

Whichever side reaches the lease first wins it, so a partition of a running cluster can leave the smaller side standing if it got there first. That is safe, since only one side ever lives, but it is not the most available outcome. Preferring the majority by making the minority wait before it reaches for the lease is a refinement this does not make yet.

The winning side keeps the lease and never gives it back, so the lease must be one that expires on its own; see Lease for why a never-expiring lease cannot resolve a second, later split.

lease instance-attribute

The outside lock the sides race for. It must live outside the partition.

__repr__()

Render the lease, which is the whole of the configuration.

decide(state) async

Keep this side if it can take the lease, and down it if it cannot.

Parameters:

Name	Type	Description	Default
state	Gossip	This node's view.	required

Returns:

Type	Description
frozenset[str]	The unreachable side's addresses when this side takes the lease, and
frozenset[str]	this side's own when it cannot.

Bases: Protocol

An outside lock that at most one owner holds, for breaking a tie.

A split with no majority, an even one, cannot be resolved from the view: it is symmetric, so any rule read from its shape keeps both halves or neither. A lease breaks the symmetry from outside. Both sides try to take the same lease, it lets only one owner hold it, and the side that holds it survives while the side that cannot down themselves.

For this to mean anything the lease has to live somewhere both sides can still reach when they cannot reach each other, which is to say outside the partition: a row in a database, a Kubernetes lease, a key in etcd. A lease held inside the split, LocalLease being the extreme case of one held in a single process, cannot arbitrate a partition it is on one side of, so it is for tests and for systems that share a process rather than for a real cluster.

An owner is a string, and the whole cluster resolving one split uses one name for it, so a side either all takes the lease or all fails to. Acquiring is therefore re-entrant: it succeeds when the lease is free or already held by this owner, and fails only when another owner holds it.

tapio holds the lease for the life of the decision and never releases it: the winning side keeps it so that a node arriving late reads the same winner. A real lease must therefore expire on its own, the way an etcd or Kubernetes lease does once its holder stops renewing it. A lease that never expires cannot arbitrate a later, independently composed split, because the earlier winner still holds it under a name the new sides do not use, so both new sides fail to acquire and down themselves. LocalLease is that never-expiring case, which is one more reason it is for tests only.

acquire(owner) async

Try to hold the lease for an owner.

Parameters:

Name	Type	Description	Default
owner	str	Who is asking, which for downing is the identity of a whole side rather than one node, so that a side agrees with itself.	required

Returns:

Type	Description
bool	Whether the lease is held by this owner now. True when it was free
bool	or already this owner's, False when another owner holds it.

A lease held in one process, for tests and for systems that share one.

It cannot arbitrate a real partition, because a partition splits processes and this lease is on one side of that split. What it does arbitrate is several systems in a single process, which is exactly the shape a test of LeaseMajority takes: every node holds the same object, so the lease mediates between them the way an outside one would mediate between machines. In production a lease reaches an outside service instead, and this class is not that.

__init__()

Start with the lease free.

__repr__()

Render who holds the lease, if anyone.

acquire(owner) async

Take the lease for an owner, if it is free or already theirs.

Parameters:

Name	Type	Description	Default
owner	str	Who is asking.	required

Returns:

Type	Description
bool	Whether this owner holds it now.

What cluster nodes say to each other, and what a node says to itself.

The five that cross a link share a base class, so one resolved ref can carry all of them, and they are registered so a peer can name them on the wire. The ticks are the daemon's own: they never leave the process, they are not registered, and a peer that invented one would be answered with a dead letter naming a key nothing is listening for.

Nothing here is acknowledged. A Join that is lost is sent again on the next retry, and a gossip round that is lost is superseded by the next one. That is the same at-most-once delivery every other message in tapio gets, and gossip is the one protocol shaped to need nothing more.

ClusterMessage = WireMessage | Seeds | Subscribe | Unsubscribe | Down | Tick | JoinTick | FormTick | HeartbeatTick | LinkChanged module-attribute

Everything the cluster daemon accepts, its own ticks included.

ClusterDowned

Bases: Message

Published on the system event stream when this node downs itself.

A downing strategy decided this node is on the side of a partition that loses, so the node marked itself Down. A Down member may not return, so the process cannot rejoin as itself: the honest response is to shut the system down, and to come back, if at all, as a new incarnation. Subscribe to this, or await Cluster.when_downed, to do that.

It never leaves the process. What the rest of the cluster learns is that the member is Down, which travels as ordinary gossip.

address instance-attribute

This node's canonical address, the one that was downed.

detail instance-attribute

Why, in words, for the log and for whoever shuts the node down.

Down

Bases: Message

Ask this node to down a member, because an operator decided it is gone.

Local, like Seeds: it arrives from this node's own management endpoint rather than across a link, so it is not registered on the wire. It carries the address explicitly because an operator downs another member far more often than the node it is speaking to.

Downing is ordinarily a strategy's decision about which side of a split survives. This is the operator's escape hatch: the manual verdict for a member a strategy will not reach, either because none is configured or because the member is unreachable to everyone and no strategy fires without a split. It moves the member up the lattice to Down exactly as a strategy would, so gossip carries the decision and the downed member hears it and shuts itself down. A Down cannot be taken back, which is the whole reason it is a member's last honest status before Removed.

address instance-attribute

The member to down.

FormTick

Bases: Message

The moment the first seed may form a cluster, if it has heard nothing.

GossipEnvelope

Bases: WireMessage

One node's whole view of the cluster, sent to one other node.

gossip instance-attribute

What the sender believes.

sender instance-attribute

Who sent it, so the receiver can answer with a newer view.

Heartbeat

Bases: WireMessage

Ask a member whether it is still answering.

Sent every round to the members this node watches, and to nobody else, so the traffic is bounded by how many peers a node watches rather than by how many members there are.

sender instance-attribute

Who is asking, so the answer knows where to go.

HeartbeatReply

Bases: WireMessage

Answer a member that asked whether this node is still answering.

Nothing is carried back but the answerer's address. What the watcher is measuring is the arrival, and the arrival is the whole of the evidence.

sender instance-attribute

Who answered, which is the member being watched.

HeartbeatTick

Bases: Message

Probe the members this node watches, and judge the ones that went quiet.

Join

Bases: WireMessage

Ask a member to let this node into the cluster.

Sent to every seed until this node sees itself in the gossip that comes back. A node that is not itself a member ignores it, which is what stops two nodes that started together from admitting each other into two different clusters.

member instance-attribute

The joining node, as it describes itself: address, incarnation, roles.

JoinTick

Bases: Message

Ask the seeds again, because joining is retried rather than acknowledged.

Leave

Bases: WireMessage

Ask the cluster to let a member go gracefully.

Ordinarily a node asks about itself, but the address is carried explicitly because an operator tool may ask about another one, and because what acts on it is the leader rather than the member named.

address instance-attribute

The member that is to leave.

LinkChanged

Bases: Message

What the transport saw about a peer, on its way into membership.

Remoting publishes its verdicts on the system's event stream, and a subscriber runs wherever the publisher happens to be. This carries the verdict into the daemon's mailbox instead, so the state is changed by the actor that owns it, in its own turn, like every other change.

It never leaves the process: what the cluster does with the observation travels as ordinary gossip.

peer instance-attribute

The peer the transport reached a verdict about.

reachable instance-attribute

Whether a link to it is open, as the transport last saw.

Seeds

Bases: Message

Tell this node's daemon which seeds to ask, and start it asking.

Seeding is a message rather than a setter because the timers it starts belong to the actor. Reaching in from outside to start them would be changing an actor's state from another task, which is the one thing an actor system exists to make unnecessary.

addresses instance-attribute

The seeds, in the order every node lists them.

At least one. The daemon reads addresses[0] to decide whether it is the first seed, which is the node allowed to form a cluster alone, so an empty list has no answer to that question and used to raise inside the receive loop instead of where the message was built.

Subscribe

Bases: Message

Ask the daemon to deliver cluster events to an actor's mailbox.

Local only, like Seeds: it carries a ref into the daemon rather than crossing a link, so it is not registered on the wire. The daemon replays the current membership to the new subscriber as events straight away, so an actor that subscribes after the cluster has formed still learns who is up, and then hears each change as it happens.

events = () class-attribute instance-attribute

Which events to deliver. Empty means every one of them.

subscriber instance-attribute

Where the events go. The daemon watches it and forgets it when it stops.

Tick

Bases: Message

Gossip to one peer, and act if this node leads a converged view.

Unsubscribe

Bases: Message

Ask the daemon to stop delivering cluster events to an actor.

Harmless if the actor was not subscribed. A subscriber that stops is forgotten without this, because the daemon watches every subscriber, so this is for an actor that wants to keep running and stop listening.

subscriber instance-attribute

The actor to forget.

WireMessage

Bases: Message

What one cluster node may send another.

A base class rather than a union, so that a node resolves one ref per peer and sends every kind of cluster message through it. It carries no fields of its own and nothing declares a field of this type: a field annotated with a base class is re-validated as that base and loses everything the subclass added.

One node's membership state, and the actor that keeps it moving.

address property

This node's canonical address, in its string form.

downed property

Set once this node has downed itself, for the application to wait on.

heartbeats property

How many probes this node has sent, which a test counts.

One per watched member per round, so this is what shows the traffic is bounded by the ring rather than by the size of the cluster.

joined property

Whether this node appears in the membership it holds.

monitored property

The members this node watches, in address order.

rounds property

How many gossip rounds this node has sent, which a test counts.

self_member property

This node as the cluster sees it, or None before it has joined.

state property

What this node currently believes about the cluster.

__init__(*, address, uid, refs, events, settings, relent, linked, strategy=None, choose=random.choice)

Describe a node's cluster daemon, before its actor exists.

Parameters:

Name	Type	Description	Default
address	str	This node's canonical address, in its string form.	required
uid	int	This system's incarnation uid, which is half of what identifies the member.	required
refs	RefRegistry	This system's ref registry, where the daemon publishes its well-known name.	required
events	EventStream	This system's event stream, where remoting says that a peer went out of reach or came back, and where this daemon says that it has downed itself.	required
settings	ClusterSettings	How often to gossip, and how patient to be.	required
relent	Callable[[str], None]	Tells remoting to stop refusing a peer it gave up on. A member that has not been downed is still a member, so this node keeps knocking rather than waiting to be told the quarantine is over.	required
linked	Callable[[str], bool]	Says whether remoting already holds an association with a peer. It is asked before answering a heartbeat from an address membership does not know, so that answering reuses a link that exists rather than opening one to wherever the message said.	required
strategy	DownStrategy | None	What to do about an unreachable member. None leaves an unreachable member blocking convergence for ever, which is the safe default for a node that has not been told how its cluster would rather resolve a split.	None
choose	Callable[[Sequence[str]], str]	Picks the peer to gossip to this round. Injected so a test can make a round deterministic; the default is uniformly at random, which is what keeps gossip traffic linear in the number of nodes.	choice

__repr__()

Render this node's address and what it believes.

behavior()

Build the daemon actor.

Bases: BaseSettings

How this node gossips, and how patient it is while joining.

Passed to Cluster rather than nested in TapioSettings, because a cluster is something an application starts and hands a list of seeds to. Remoting has to be configured before the system exists, since the port settles the canonical address; joining a cluster is an action taken afterwards.

down_after = timedelta(seconds=7) class-attribute instance-attribute

How long an unreachable split must hold still before a strategy acts.

Downing cannot be taken back, so it waits for the split to settle: the set of unreachable members has to stay the same for this long before the leader downs anybody. A shorter disturbance, a pause or a link that flaps, is ridden out rather than resolved. Only consulted when a downing strategy is configured, and measured from when the split was first seen, so the whole wait before a member is downed is this on top of unreachable_after. Set it above the few gossip rounds a real partition takes to be seen the same way across a side.

gossip_interval = timedelta(seconds=1) class-attribute instance-attribute

How often this node sends its view to one other member.

One peer per round, chosen at random, which is what keeps the traffic linear in the number of nodes rather than quadratic.

heartbeat_interval = timedelta(seconds=1) class-attribute instance-attribute

How often this node asks each member it watches whether it is answering.

Separate from the link heartbeat in RemoteSettings: that one keeps a connection warm and judges the connection, and this one judges a member, including one this node would otherwise never send anything to.

join_retry_interval = timedelta(seconds=1) class-attribute instance-attribute

How often an unjoined node asks the seeds to let it in again.

Joining is at-most-once like every other send, so it is retried rather than acknowledged. The retries stop as soon as the node sees itself in the gossip it receives.

join_timeout = timedelta(seconds=30) class-attribute instance-attribute

How long join_seed_nodes waits to see this node reach Up.

leave_timeout = timedelta(seconds=30) class-attribute instance-attribute

How long leave waits to see this node reach Removed everywhere.

monitored_peers = 5 class-attribute instance-attribute

How many other members this node watches, by their place on the ring.

Every node sorts the member addresses, finds itself, and watches the few that follow it. So every member is watched by this many others whatever the traffic does, and the heartbeat traffic stays linear in the number of nodes. All-to-all monitoring is quadratic, and it is what makes naive implementations fall over at a few dozen nodes.

phi_acceptable_pause = timedelta(seconds=3) class-attribute instance-attribute

How much silence to tolerate on top of a member's learned rhythm.

Rides out a scheduler or garbage-collection pause without relearning it as the normal interval. Set it to cover the longest hiccup that is not a failure. Only consulted when phi_accrual is set.

phi_accrual = False class-attribute instance-attribute

Judge a watched member with a phi-accrual detector, not a fixed window.

A fixed window (unreachable_after) has to be set well above heartbeat_interval or a slow moment reads as death, and that slack is latency a real failure waits out. Phi-accrual learns the spread of a member's answer times instead, and suspects it on a scale that means the same confidence whether the link is fast and steady or slow and jittery. Off by default so behaviour does not change under anyone until they ask for it; when on, unreachable_after is not consulted and phi_threshold and phi_acceptable_pause take over.

phi_threshold = 8.0 class-attribute instance-attribute

How much suspicion is enough to call a member unreachable.

A log-scale value: 8 is roughly a one-in-a-hundred-million chance the member is merely slow rather than gone. Higher is more patient and less likely to be wrong, at the cost of noticing a real death later. Only consulted when phi_accrual is set.

roles = frozenset() class-attribute instance-attribute

What this node says it is for. Every cluster-aware feature filters on these, and they are fixed for the life of the member: a role is part of what the rest of the cluster agreed on when it accepted the node.

seed_form_after = timedelta(seconds=5) class-attribute instance-attribute

How long the first seed waits before forming a cluster on its own.

Only the first node in the seed list may do this, and only if it has heard from nobody at all in that time. That is the rule that stops a restart from producing a second cluster that never meets the first, so this has to stay comfortably longer than the time it takes a running seed to answer a join with gossip.

unreachable_after = timedelta(seconds=5) class-attribute instance-attribute

How long a watched member may go without answering before it is called unreachable.

An unreachable member blocks convergence and is not written off: deciding to stop waiting for it is downing, and downing is a separate decision with strategies of its own. Set this well above heartbeat_interval, since a fixed window has no opinion about how variable the network is.

Cluster management

The tapio-cluster command: read a cluster, and move a member.

An operator points this at the management port of any one node, and asks that node what it believes or asks it to act. There are three things to ask:

tapio-cluster status
tapio-cluster leave tapio://orders@10.0.0.2:2551
tapio-cluster down  tapio://orders@10.0.0.3:2551

It speaks HTTP to ManagementSettings, or HTTPS when the node is configured for TLS and the command is given --tls (or a --cafile or --client-cert, which imply it). The only thing it needs at runtime is a typer for its own argument parsing and the standard library for the requests, and the same calls are a curl away for anyone who would rather script them. What it reports is one node's view, which is the truth once the cluster has converged and that node's best guess until then, the same caveat every gossip-based answer carries.

A leave or a down is answered the moment the node has been asked, not once the member has gone: the decision travels as gossip, so the command prints that the node accepted it and the next status shows it taking effect.

down(ctx, address)

Down a member an operator judges gone.

leave(ctx, address)

Ask a member to leave the cluster gracefully.

status(ctx)

Show the members, the leader, and reachability.

A node's operator surface: the port, and what answers on it.

__init__(*, listener, snapshot, members, daemon, token, tls, address)

Describe a node's management endpoint, before its actor exists.

Parameters:

Name	Type	Description	Default
listener	socket	The socket bound by open_management_listener, already listening so the port is settled.	required
snapshot	Callable[[], Mapping[str, Any]]	Reads what this node believes about the cluster, as the plain data a JSON response is built from. Called between the daemon's turns, so it must not await.	required
members	Callable[[], frozenset[str]]	The addresses this node currently holds a live member record for, read the same way and used to answer a leave or a down for a member the node does not know with a 404.	required
daemon	ActorRef[ClusterMessage]	This node's cluster daemon, where a leave or a down is sent.	required
token	SecretStr | None	The bearer token an operator must present, or None to ask for nothing.	required
tls	TLSSettings | None	Certificates for the port, or None for plaintext HTTP. When set the port speaks HTTPS, and requires a client certificate too when the settings carry a cafile.	required
address	str	This node's canonical address, for the log.	required

__repr__()

Render the address this endpoint answers for.

behavior()

Build the /system/cluster-management actor.

Bind the port an operator reaches this node on, before anything runs.

Bound here rather than inside the serving task so a node asked for port 0 knows the port it got before it hands anyone a way to reach it, the same reason remoting binds at construction.

Parameters:

Name	Type	Description	Default
settings	ManagementSettings	How this node answers operators.	required

Returns:

Type	Description
socket	A listening socket, not yet accepting.

Raises:

Type	Description
InsecureRemoteConfig	If it would answer beyond loopback with no token.
OSError	If the address could not be bound.

Bases: BaseSettings

Where a node answers an operator, and what proves the operator is one.

Passed to Cluster alongside the cluster, the same way ClusterSettings is: managing a cluster is something an operator does to a system that already exists, not part of how the system is addressed. A node with these open runs a small HTTP surface that reads its membership and asks it to let a member leave or down one. The tapio-cluster command speaks to it.

Off unless it is configured, like remoting, since a port that can down a member is a serious surface. When it is on, it binds loopback by default, for the same reason remoting does: the default is set for someone who has not thought about who can reach it yet.

bind_host = '127.0.0.1' class-attribute instance-attribute

The interface to answer operators on. Loopback by default: this port can down a member, so reaching it from another host is a decision to make on purpose rather than a default to inherit.

bind_port = 25530 class-attribute instance-attribute

The port to answer operators on. 0 takes whatever the OS hands out, which is what a test binds so no two of them argue over a number.

tls = None class-attribute instance-attribute

Certificates for the management port, or None for plaintext HTTP.

The same TLSSettings remoting uses. With certfile and keyfile, the port speaks HTTPS and an operator's client verifies it. Add cafile and the port also requires a client certificate signed by that authority, which authenticates the operator the way a token does: mutual TLS is therefore a second way to satisfy the bind-beyond- loopback rule, where a bearer token alone travels in a header an eavesdropper on a plaintext link would read.

token = None class-attribute instance-attribute

The bearer token an operator presents, or None to ask for nothing.

One of the two ways to prove an operator is one, the other being a client certificate under tls. At least one is required to bind anywhere but loopback: a port that can down a member, reachable from any host with nothing to prove, fails to start rather than serving strangers. On loopback it is optional, since reaching the port at all already means being on the machine. Compared in constant time, and presented as Authorization: Bearer <token>.

Remote spawning

Register a behavior a peer can ask a spawner to start.

@remote_behavior("worker")
def worker(args: WorkerArgs) -> Behavior[Job]: ...

The factory takes exactly one arguments model and returns the behavior to start. Wrap that behavior in Behaviors.supervise(...) here if it should be supervised: supervision happens entirely on the node that runs the actor, so this is the only place it can be declared.

Arguments models may not carry an ActorRef. They are validated on the peer after the factory key has been checked, which is deliberately outside the decode that resolves refs, so a ref in them has nothing to resolve against. Send the new actor a message instead: the requester holds Spawned.ref and can tell it whatever it needs to know, and refs in that message resolve normally.

Parameters:

Name	Type	Description	Default
key	str | None	What a frame names this factory by. The function's name when omitted. Unlike a message key, the default is not qualified by module: a spawner's allowlist is written out by hand and reads better short.	None
args	type[Message] | None	The arguments model, when the annotation cannot say. Read from the factory's single parameter when omitted.	None

Returns:

Type	Description
Callable[[F], F]	The decorator, which returns the function unchanged.

Raises:

Type	Description
BehaviorRegistrationError	If the key is already taken, or if the arguments model cannot be resolved. Both are raised at import time: a duplicate key would otherwise decide itself by import order, and a factory whose arguments cannot be built is one no peer could ever call.

Build the actor that starts other actors on this node, on request.

system.spawn(spawner(offers=["worker"]), name="spawner")

It is an ordinary actor. Nothing about it is special to the runtime: the request is a message, the answer is a message, and the actors it starts are its own children, supervised by it and never by whoever asked. Give it a path a peer can name, since a peer reaches it by resolve like anything else.

Parameters:

Name	Type	Description	Default
offers	Iterable[str]	The factory keys this spawner will start. Nothing else, whether or not it is registered.	required

Returns:

Type	Description
Behavior[Spawn]	The behavior to spawn.

Raises:

Type	Description
BehaviorRegistrationError	If a key is offered that no @remote_behavior registered. The allowlist is checked here rather than when a peer asks, so a typo in it fails where it was written.

Bases: Message

Ask a spawner to start an actor on its own node.

args = {} class-attribute instance-attribute

What to call the factory with. Pass the arguments model itself.

factory instance-attribute

The key of the factory to start, as registered on the peer.

name = None class-attribute instance-attribute

What to call the actor. Generated by the peer when omitted.

A generated name cannot collide, so a name is worth asking for only when something else has to be able to find the actor by path.

reply_to instance-attribute

Where the answer goes. ask fills this in.

Bases: Message

What a spawner answers, either way.

Both answers share a base so that one ask can receive either: expect=SpawnReply accepts a refusal as an ordinary reply rather than failing the ask with AskTypeError. A refusal is news the requester can act on, not a broken protocol.

Bases: SpawnReply

An actor was started, and this is the handle to it.

factory instance-attribute

The key it was started from.

name instance-attribute

The name it was given, generated when the request named none.

ref instance-attribute

A ref to the new actor.

Typed as Any because the type parameter cannot cross a socket: a frame carries a path and an address, and nothing that could reconstruct a type argument. Assign it to an ActorRef[YourProtocol] and the claim is checked where every claim about a peer is checked, on the receiving node, against the actor's real message type.

Bases: SpawnReply

No actor was started, and this is why.

detail instance-attribute

What happened, in a sentence, for a log or an error message.

factory instance-attribute

The key that was asked for.

reason instance-attribute

One of the constants in SpawnFailure.

Why a spawner refused.

String constants in a namespace rather than an enum, for the reason a dead letter's reason is one: the set grows, and a peer running an older version has to be able to read a reason it has never seen instead of failing to decode the reply.

FACTORY_FAILED = 'factory-failed' class-attribute instance-attribute

The factory raised while building the behavior. The spawner replies rather than failing, because it is the parent of every actor it has started and one bad request must not stop the rest of them.

INVALID_ARGS = 'invalid-args' class-attribute instance-attribute

The arguments did not validate against the model the factory declared. The sender's idea of what a factory takes is a claim about the peer, and this is the check that decides.

NAME_REFUSED = 'name-refused' class-attribute instance-attribute

The requested name is already taken by a live child of that spawner, or is not a name an actor can have.

NOT_ALLOWED = 'not-allowed' class-attribute instance-attribute

The key is registered on the peer, and that spawner does not offer it. A spawner's allowlist is the whole of what it will start, because a spawner that starts anything registered is a capability handed to whoever can reach the port.

TERMINATING = 'terminating' class-attribute instance-attribute

The peer, or the spawner itself, is shutting down.

UNKNOWN_FACTORY = 'unknown-factory' class-attribute instance-attribute

The peer has no factory registered under that key. Almost always version skew: both nodes have to be running the same code, since what crosses the wire is a key and never the behavior itself.

Bases: Message

The arguments of a factory that needs none.

Every factory takes exactly one arguments model, so that the rule is one sentence long rather than two with a special case. A factory that needs nothing declares this one.

One behavior a peer can ask for, by key.

args_type instance-attribute

The model the arguments are validated into, before build sees them.

build instance-attribute

Called with the arguments model to produce the behavior.

key instance-attribute

What a frame names it by.

arguments(args)

Build the arguments model from what arrived.

Parameters:

Name	Type	Description	Default
args	dict[str, Any]	The JSON object the request carried.	required

Returns:

Type	Description
Message	The model this factory declared.

Raises:

Type	Description
ValidationError	If the object does not satisfy it.

Return the factory registered under a key, if this node has one.

Parameters:

Name	Type	Description	Default
key	str	The key a request named.	required

Returns:

Type	Description
RemoteFactory | None	The factory, or None. None is the answer a peer running different
RemoteFactory | None	code produces, and it is reported rather than guessed at: nothing is
RemoteFactory | None	imported to find out what the key might have meant.

Reachability

Deciding that a peer is gone, and admitting what that decision is worth.

This is the part of remoting that cannot be made to feel local. A partition, a long pause and a peer that died all look the same from one node: the frames stop. So a system has to guess, and every guess it makes can be wrong.

The guess is split in two, so that each half can be replaced on its own:

• A FailureDetector says whether a peer still looks alive from here. Today that is a fixed timeout. Clustering replaces it with phi-accrual, which reads the same interface.
• A DownDecider says what to do about it. Today it says yes, alone, immediately. Clustering replaces it with a strategy over converged membership, so that a minority partition stops itself rather than both halves declaring the other dead.

Writing the decider as an interface for a function that currently returns a constant is the whole point. The association asks rather than deciding inline, so the day the answer stops being a constant, nothing above it changes.

The verdict can be false and there is no fix for that inside one node. Both sides of a partition will declare the other dead and both will be locally correct. Resolving it needs membership and a quorum, which a single system does not have, so it fails fast and stays failed instead. For request/response and work distribution that is the better trade: wrongly deciding a peer is dead costs a retry, which is recoverable, and waiting forever costs availability, which often is not.

DeadlineDetector

A fixed timeout: alive until nothing has arrived for long enough.

The simplest detector that can work, and an honest one to start from. It has no opinion about how variable a network is, so the timeout has to be set well above the peer's heartbeat interval or a slow moment reads as a dead peer. Phi-accrual, which learns the distribution instead of being told a number, fits behind the same interface.

last_heard property

When something last arrived, on the loop's clock.

__init__(*, unreachable_after, started_at)

Start the clock on a peer that has just been heard from.

Parameters:

Name	Type	Description	Default
unreachable_after	float	Seconds of silence that mean the peer is gone.	required
started_at	float	When the link came up, which counts as being heard from: a peer that has just handshaken is not a silent one.	required

__repr__()

Render the window, which is the whole of the configuration.

heartbeat(at)

Record that something arrived from the peer.

is_available(now)

Whether the peer has been heard from inside the window.

DownAlone

Decide alone, immediately, and always yes.

One node cannot do better. It has no membership to consult and no quorum to be part of, so "wait and see" would only mean waiting, and waiting is what the detector already did. Clustering replaces this with a strategy that knows how many nodes there are and which side of a partition it is on, and that is the entire difference.

__repr__()

Render the class name; there is no state to show.

decide(peer) async

Give up on the peer.

Parameters:

Name	Type	Description	Default
peer	Address	The peer in question.	required

Returns:

Type	Description
DownDecision	A decision to down it, naming this system as the only voter.

DownDecider

Bases: Protocol

What a system does when a peer stops looking alive.

decide(peer) async

Decide whether a peer that has gone quiet should be given up on.

Parameters:

Name	Type	Description	Default
peer	Address	The peer in question.	required

Returns:

Type	Description
DownDecision	The decision.

DownDecision

Bases: Message

What to do about a peer the detector has given up on.

detail instance-attribute

Why, for the log, the event and the errors that follow.

down instance-attribute

Whether to treat the peer as gone.

FailureDetector

Bases: Protocol

Whether a peer still looks alive, judged from the frames it sends.

heartbeat(at)

Record that something arrived from the peer.

Parameters:

Name	Type	Description	Default
at	float	When, on the system loop's monotonic clock.	required

is_available(now)

Whether the peer still looks alive.

Parameters:

Name	Type	Description	Default
now	float	The current time, on the same clock.	required

Returns:

Type	Description
bool	Whether frames are still arriving often enough to believe it.

PeerReachable

Bases: Message

Published when a link to a peer comes up and can carry traffic.

The counterpart to PeerUnreachable, and the only thing that can retract one. It says a link is open, which is a weaker claim than the actors over there being the ones you remember: a peer that restarted comes back with a new incarnation uid, and uid is how a subscriber tells that apart from a link that merely reconnected.

It is published every time a link opens, including the first, so a subscriber that only cares about recovery has to know whether it had written the peer off. Publishing only after an unreachability would need the association to remember a verdict that closed it, and the association that comes back is a new one.

peer instance-attribute

The peer's canonical address.

uid instance-attribute

The incarnation on the other end, as the handshake established it.

PeerUnreachable

Bases: Message

Published when a peer can no longer be reached through its association.

Subscribe to it on system.events to log, alarm, or shut a service down. It says nothing about whether the actors over there are running: a peer that terminated cleanly and one behind a partition produce the same event.

detail instance-attribute

What happened, in words.

peer instance-attribute

The peer's canonical address.

quarantined instance-attribute

Whether the address is now frozen.

True after a detector gave up on a silent peer: nothing will be sent there and nothing dialled until remote.reconnect says so. False when the link merely ended, in which case the next send dials again.

uid instance-attribute

The incarnation that was associated, or 0 if the link never came up.

PhiAccrualDetector

Alive until the peer's silence is longer than its own history explains.

A fixed window has to be set well above the heartbeat interval or a slow moment reads as death, and that slack is latency a real failure waits out. This detector learns the spread of a peer's arrival times instead of being told a number, and reports a suspicion level phi that rises smoothly as silence outruns what the peer's own timing led it to expect. phi is on a log scale: phi around 1 is roughly a one-in-ten chance the next beat is merely late, phi around 2 about one in a hundred, and so on. One threshold therefore means the same confidence whether the link is fast and steady or slow and jittery, which is the whole reason to prefer it over a window.

It reads the same interface as DeadlineDetector, so RingMonitor and the association hold one without learning which.

The estimate is seeded before any real interval is seen, so a peer picked up a moment ago is not suspected for never having answered a probe that has not been sent yet. One sample has no spread, so the seed is two synthetic intervals around a first estimate.

__init__(*, started_at, threshold, acceptable_pause, first_interval_estimate, max_samples=200, min_std_deviation=0.05)

Start suspecting a peer that has just been heard from.

Parameters:

Name	Type	Description	Default
started_at	float	When the link came up, which counts as being heard from, exactly as DeadlineDetector treats it.	required
threshold	float	The phi at which the peer is called unreachable. Higher is more patient and less likely to be wrong, at the cost of noticing a real death later.	required
acceptable_pause	float	Seconds of extra silence tolerated on top of the learned mean, to ride out a scheduler or garbage-collection hiccup without relearning it as normal.	required
first_interval_estimate	float	The interval assumed before any real one is seen, so a peer is not suspected during its first rounds. The probe interval is the natural value.	required
max_samples	int	How many recent intervals shape the estimate. Older ones fall out, so the detector tracks a network whose timing drifts.	200
min_std_deviation	float	A floor on the spread, in seconds. A peer that answers like a metronome would otherwise be suspected on a pause of milliseconds, which is too sharp to be safe.	0.05

__repr__()

Render the threshold, which is the knob a reader reaches for first.

heartbeat(at)

Record that something arrived, and learn the interval since the last.

A zero or negative interval is not learned: two arrivals credited to the same instant say nothing about the peer's rhythm, and a clock that went backwards must not poison the estimate.

Parameters:

Name	Type	Description	Default
at	float	When, on the system loop's monotonic clock.	required

is_available(now)

Whether suspicion is still below the threshold.

Parameters:

Name	Type	Description	Default
now	float	The current time, on the loop's monotonic clock.	required

Returns:

Type	Description
bool	Whether the peer still looks alive.

phi(now)

The suspicion that the peer is gone, given how long it has been quiet.

Parameters:

Name	Type	Description	Default
now	float	The current time, on the loop's monotonic clock.	required

Returns:

Type	Description
float	A value that is near zero while beats arrive on time and climbs
float	without bound as silence outlasts the learned interval.

Which peers this system will talk to, and which it has given up on.

Remoting resolves a peer from an address somebody wrote down: a string in a configuration file, or a ref that arrived in a message. Every such address is dialable, and the only reason to refuse one is that this system decided the peer is gone (quarantine, see failure). That is what StaticPeers implements, and it is the whole answer for a system that has no membership to consult.

Clustering answers the same question from membership instead: a member that the cluster has downed is refused, and it is refused for a reason every node agrees on rather than one this node reached alone. The consequences are identical either way, which is why there is one lookup and not two: watchers have been told the actors over there are gone, sends dead-letter, and nothing is dialled again until somebody says so. Only the voter changes.

So the endpoint asks rather than checking a table of its own. A refusal carries the words that explain it, because they end up in a log line and in the dead letter for every message that was on its way there, and "quarantined" on its own tells a reader nothing about which of the two decided it.

PeerProvider

Bases: Protocol

Which peers a system may associate with, and why not when it may not.

give_up(peer, detail)

Refuse a peer from now on.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required
detail	str	Why, kept for the log, the dead letters, and whoever asks later what happened.	required

refusal(peer)

Say whether this system refuses to be associated with a peer.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required

Returns:

Type	Description
str | None	Why the peer is refused, in words, or None if it is not
str | None	refused and may be dialled.

refusals()

List every peer this system refuses, and why it refuses each.

Returns:

Type	Description
Mapping[Address, str]	A snapshot, so a caller may read it while the answer changes.

relent(peer)

Stop refusing a peer.

Parameters:

Name	Type	Description	Default
peer	Address	The peer's canonical address.	required

Returns:

Type	Description
str | None	Why it was refused, or None if it was not refused at all.

StaticPeers

Every address is a peer, until this system gives up on one.

The right answer for a pair of systems that were told about each other rather than joined into a cluster. There is no membership to consult, so the only peers that are refused are the ones a failure detector here gave up on, and the only way back is for somebody to say so.

__init__()

Start with nothing refused, since nothing has failed yet.

__repr__()

Render how many peers are refused, which is all the state there is.

give_up(peer, detail)

Refuse a peer from now on, recording why.

refusal(peer)

Say why this system refuses a peer, or None if it does not.

refusals()

List every refused peer with the words that explain it.

relent(peer)

Stop refusing a peer, returning why it was refused.

What one system publishes about itself, and who is listening.

One per system. Publishing is synchronous and runs on the system's loop, so a handler must not block and must not raise. One that raises is logged and the rest still run: a bad subscriber cannot break the runtime event that reached it.

total property

How many events this stream has published.

__init__()

Create an empty stream.

__iter__()

Iterate the event types currently subscribed to, for tests to check.

__repr__()

Show the running totals.

publish(event)

Hand an event to every subscriber that asked for its type.

Parameters:

Name	Type	Description	Default
event	Message	What happened.	required

subscribe(event_type, handler)

Register a handler for one event type.

Parameters:

Name	Type	Description	Default
event_type	type[E]	What to be told about. Subclasses count.	required
handler	Callable[[E], None]	Called with each matching event.	required

Returns:

Type	Description
Subscription	A handle for unsubscribing, usable as a context manager.

A handle for undoing one subscribe call.

__enter__()

Return the subscription, for use as a context manager.

__exit__(*exc)

Unsubscribe on the way out of the block.

__init__(cancel)

Bind the subscription to the stream that made it.

unsubscribe()

Stop receiving events. Calling this twice is harmless.

Supervision

One decision, plus the limits that apply when it is restart.

Build one with the classmethods rather than the constructor, so that a strategy reads as the decision it makes:

Behaviors.supervise(worker()).on_failure(
    SupervisorStrategy.restart(max_restarts=3, window=timedelta(seconds=1)),
    on=ConnectionError,
)

backoff = None class-attribute instance-attribute

How long to wait before each restart, or None to restart at once.

decision instance-attribute

What to do with the failed actor.

max_restarts = None class-attribute instance-attribute

How many restarts are allowed inside window, or None for no limit.

Exceeding it stops the actor: the failure is not transient after all, and an actor restarting forever is a bug that never gets reported.

window = None class-attribute instance-attribute

The span max_restarts is counted over, or None for all time.

__post_init__()

Reject limits that only make sense on a restart strategy.

Raises:

Type	Description
ValueError	If restart limits are set on a strategy that does not restart, or if max_restarts is not positive.

__repr__()

Render as the factory call that produces it.

escalate() classmethod

Stop the actor and hand the failure to its parent.

restart(*, max_restarts=None, window=None, backoff=None) classmethod

Rebuild the actor from the behavior it was spawned with.

Parameters:

Name	Type	Description	Default
max_restarts	int | None	How many restarts to allow within window. The actor is stopped once that is exceeded.	None
window	timedelta | None	The span restarts are counted over. Without one, the count runs for the life of the actor.	None
backoff	Backoff | None	How long to wait before each restart. Without one the restart is immediate. That is right for a fault that clears instantly, and wrong for anything involving a dependency.	None

Returns:

Type	Description
SupervisorStrategy	The strategy.

resume() classmethod

Keep the actor and its state, and move on to the next message.

stop() classmethod

Stop the actor. This is what an unsupervised actor already does.

Bases: Enum

What supervision does with a failed actor.

ESCALATE = 'escalate' class-attribute instance-attribute

Stop the actor and hand the failure to its parent as the parent's own.

The parent then takes its own decision, so a subtree can be restarted by the actor that knows how to rebuild it rather than by the one that broke.

RESTART = 'restart' class-attribute instance-attribute

Rebuild the actor from the behavior it was spawned with.

Children are stopped and respawned by the re-run setup, the mailbox survives, and watchers hear nothing: the actor's identity is unchanged and only its incarnation is new.

RESUME = 'resume' class-attribute instance-attribute

Keep the actor, its behavior, and its state, and take the next message.

The failed message is gone. This is the right choice when the failure is about the message rather than about the actor.

STOP = 'stop' class-attribute instance-attribute

Stop the actor. Its watchers get Terminated like any other stop.

Exponential backoff with jitter, for restarts that should not thrash.

A dependency that just refused a connection will usually refuse the next one too. Restarting immediately burns the whole restart window in a millisecond, and stops the actor for a fault that would have cleared on its own. Waiting, and waiting longer each time, is what makes a restart limit mean "this is not getting better" instead of "this failed fast".

max_backoff instance-attribute

The ceiling the doubling stops at.

min_backoff instance-attribute

How long to wait before the first restart.

random_factor = 0.2 class-attribute instance-attribute

How much jitter to add, as a fraction of the delay.

0.2 means up to twenty percent longer. Jitter matters when a shared dependency fails: without it, every actor that noticed at the same moment retries at the same moment, forever.

__post_init__()

Reject a backoff that could not produce a sensible delay.

Raises:

Type	Description
ValueError	If either bound is negative, if the maximum is below the minimum, or if the random factor is not in [0, 1].

delay(restart, *, jitter)

How long to wait before the given restart.

This is a pure function, and the jitter is an argument rather than a call to random. A test can then assert the exact schedule, and the randomness stays at the one call site that wants it.

Parameters:

Name	Type	Description	Default
restart	int	Which restart this is, counting from one.	required
jitter	float	A value in [0, 1), scaled by random_factor.	required

Returns:

Type	Description
float	The delay in seconds.

Bases: Generic[T]

The half-built result of Behaviors.supervise, awaiting a strategy.

It takes two calls rather than one because which failures are governed and what to do about them are separate choices. Keeping them apart is what makes a nested supervision stack readable.

__init__(behavior)

Bind the behavior whose failures are about to be governed.

__repr__()

Render the behavior still waiting for its strategy.

on_failure(strategy, *, on=Exception)

Apply a strategy to the failures this actor raises.

Parameters:

Name	Type	Description	Default
strategy	SupervisorStrategy	What to do when a matching failure happens.	required
on	type[Exception] | tuple[type[Exception], ...]	Which exceptions it governs. Anything else falls through to the next wrapper out, and to stop if none matches.	Exception

Returns:

Type	Description
Behavior[T]	The supervised behavior, to spawn or to wrap again.

Mailbox and signals

A queue with a system lane and a user lane, drained in that order.

config property

How this mailbox handles capacity.

is_full property

Whether the user lane is at capacity.

A closed mailbox is never full. Senders waiting in offer have to be able to finish, and the cell discards what they enqueue rather than leaving them stuck.

system_size property

How many signals are queued.

user_size property

How many user messages are queued.

waiting_senders property

How many senders are parked in offer waiting for capacity.

__init__(config=None)

Create an empty mailbox.

Parameters:

Name	Type	Description	Default
config	MailboxConfig | None	Capacity and overflow behaviour. Unbounded when omitted.	None

__len__()

Total queued envelopes across both lanes.

__repr__()

Show the depth of each lane.

close()

Wake every blocked sender, so a stopped actor strands nobody.

Their offer resumes and enqueues into a mailbox nobody is reading, and the cell dead-letters what is left. Called from the termination sequence, so no sender waits for a slot that will never come.

get() async

Take the next envelope, waiting if the mailbox is empty.

Returns:

Type	Description
Envelope	The next signal if any is queued, otherwise the next user message.

Raises:

Type	Description
RuntimeError	If a second reader is already waiting. A single waiter is what makes the wakeup above safe. A mailbox has exactly one consumer, its own cell, so we check this rather than assume it.

get_system() async

Take the next signal, ignoring the user lane entirely.

This covers the one state where an actor is absent rather than idle. A cell backing off before a restart stops taking user messages, so its mailbox keeps filling, but a stop signal must still reach it within the shutdown deadline.

Returns:

Type	Description
Signal	The next signal.

Raises:

Type	Description
RuntimeError	If a second reader is already waiting, as for get.

offer(message) async

Append to the user lane, waiting for capacity instead of dropping.

Senders wait on individual futures and are woken one per freed slot, in arrival order. This is not symmetric with get, on purpose. There is one consumer but there can be many senders, so an Event would wake all of them for one slot and give it to whichever the scheduler happened to pick.

Parameters:

Name	Type	Description	Default
message	Message	The message to enqueue.	required

Raises:

Type	Description
CancelledError	If the wait is cancelled, having first removed this sender's own future so nothing is left behind.

put(message)

Append to the user lane, applying the overflow strategy if full.

Parameters:

Name	Type	Description	Default
message	Message	The message to enqueue.	required

Returns:

Type	Description
Message | None	The message that was displaced and should go to dead letters, or
Message | None	None when nothing was displaced. The caller owns the sink, so
Message | None	the mailbox decides what is dropped and never where it goes.

Raises:

Type	Description
MailboxFullError	If the lane is full under OverflowStrategy.FAIL.

put_front(message)

Put a message back at the head of the user lane.

This is used for unstashing, which replays a message rather than sending a new one. These messages were accepted once already, so they go back ahead of everything that queued up while the actor was not ready for them. Capacity is not checked, for the same reason the system lane has no capacity: a replay that could be refused would make unstash_all a best effort rather than a guarantee, and the messages have nowhere else to go.

put_system(signal)

Append to the system lane.

The system lane is unbounded whatever the user lane's capacity is. A limit that could refuse a stop signal would make shutdown unreliable.

take_pending()

Pop one queued user message, or None when the lane is empty.

The cell's termination sequence calls this to account for undelivered messages instead of discarding them with the mailbox.

Bases: Signal

The actor has stopped and will handle no further message.

Best effort. An actor cancelled at the shutdown deadline while stuck in a handler may never see it. Resources are released here, so treat it as "usually runs" rather than a guarantee.

Bases: Signal

The actor is about to be restarted, and this incarnation is ending.

Delivered to the incarnation that failed, before its children are stopped and before the original behavior is evaluated again. PostStop does not follow, because a restart is not a stop. Releasing resources twice would be as wrong as never releasing them.

Bases: Signal

A watched actor has stopped, for any reason including failure.

Delivered to everyone who called ctx.watch on it, exactly once, on the system lane. A restart does not produce one, because the ref, path and uid are unchanged and only the incarnation behind them is new.

ref instance-attribute

The actor that stopped.

The two ends of a death watch: the protocols, and the book an actor keeps.

Death watch started as a relationship between two cells, and the maps that hold it were typed that way. Two features broke that assumption.

Ask broke the watcher end. A promise ref watches the actor it is waiting on, so that a target which stops fails the caller at once rather than after the full timeout, and a promise ref has no cell. Watcher is the two things a watched actor actually uses, so both kinds of watcher fit in one map.

Remoting broke the watched end. An actor on a peer has no cell here either, and watching it means sending a frame and waiting for one back. WatchTarget is what the watching cell actually uses, so a local actor and an actor on another node are registered the same way.

DeathWatch is the book each actor keeps of both: who is watching it, and who it is watching. It lives here rather than in the cell because it is entirely about the two protocols above and needs nothing else from an actor. The cell still is the watcher and the target, so it keeps the protocol methods and delegates the bookkeeping to this.

DeathWatch

Both sides of one actor's death watches.

Both directions are kept because both can leak. An entry in the watchers map outliving the actor it names holds that actor's ref forever, and an entry in the watched map means this actor is still registered on something that will call it after it has stopped. Death watch exists to save users from writing that bookkeeping, so it cannot get it wrong itself.

watchers property

Who has asked to be told when this actor stops.

__init__()

Start with no watch registered in either direction.

add_watcher(watcher)

Register something to be told when this actor stops.

Keyed by path, so watching twice still delivers exactly one signal.

Parameters:

Name	Type	Description	Default
watcher	Watcher	What to tell.	required

release(watcher, ref)

Tell the watchers, and leave nothing behind in the watched.

Parameters:

Name	Type	Description	Default
watcher	Watcher	The stopping actor, as its targets registered it.	required
ref	ActorRef[Any]	A ref to the stopping actor, which its watchers are told about.	required

remove_watcher(watcher)

Deregister a watcher. Harmless if it was not registered.

Parameters:

Name	Type	Description	Default
watcher	Watcher	What to stop telling.	required

stop_watching(path)

Forget a watch this actor holds, and say what it was on.

Parameters:

Name	Type	Description	Default
path	ActorPath	The watched actor.	required

Returns:

Type	Description
WatchTarget | None	What was being watched, so the caller can deregister from it, or
WatchTarget | None	None if this actor was not watching it.

watching(target)

Record that this actor is watching another.

Parameters:

Name	Type	Description	Default
target	WatchTarget	What is being watched. It has already been told.	required

WatchTarget

Bases: Protocol

Something a death watch can be registered on.

is_alive property

Whether a watch registered now could still produce a signal.

False means the answer is already known, so the watcher is told at once rather than waiting for a signal that will never come. For an actor on a peer it means the peer is beyond reach, not that the actor is known to have stopped.

path property

Where the watched actor sits, which is the key it is held under.

add_watcher(watcher)

Register something to be told when this actor stops.

remove_watcher(watcher)

Deregister a watcher.

Watcher

Bases: Protocol

Something that can be registered for another actor's death.

path property

Where this watcher sits, which is the key it is held under.

Watchers are keyed by path so that watching twice still delivers exactly one signal.

notify_terminated(ref)

Take delivery of a watched actor's death.

Called on the system's loop, from the watched actor's own termination sequence, so it must not block and must not raise.

Parameters:

Name	Type	Description	Default
ref	ActorRef[Any]	A ref to the actor that stopped.	required

notify_unreachable(ref, detail)

Take delivery of a watched actor becoming unreachable.

The actor was on a peer, and the link to that peer ended or went silent. Whether the actor itself is alive cannot be known from here, which is the difference from notify_terminated.

An actor watching another sees no difference: both arrive as Terminated, because a supervisor that had to tell them apart could do nothing useful with the answer. An ask does tell them apart, since the caller may want to retry somewhere else.

Parameters:

Name	Type	Description	Default
ref	ActorRef[Any]	A ref to the actor that can no longer be reached.	required
detail	str	Why the peer is considered gone, for the error and the log.	required

Bases: Signal

A child failed and its supervision decision was to escalate.

The parent's cell treats this as its own failure and takes its own decision. Escalation is therefore ordinary message flow, which makes it ordered, observable and testable. An exception injected across a task boundary would have no defined order against the parent's in-flight message.

Handled by the runtime, never by a behavior's signal handler.

error instance-attribute

What it failed with, carried unchanged up the chain.

ref instance-attribute

The child that failed.

Test support

Bases: Generic[T]

An actor that records what it is sent, for a test to assert on.

__test__ = False class-attribute instance-attribute

Keeps pytest from collecting this as a test class.

The name begins with Test, which is pytest's rule for a class holding tests, and a class with a constructor cannot be collected. Without this flag, importing a probe into a test module is a collection warning in somebody else's project.

path property

Where this probe sits in the tree.

pending property

How many messages are recorded and not yet taken.

ref property

The ref to hand to the code under test.

__init__(system, msg_type, *, name=None, mailbox=None)

Start a probe as a top-level actor in a running system.

Parameters:

Name	Type	Description	Default
system	ActorSystem	The system to spawn it in.	required
msg_type	MessageType	What it accepts. A probe validates on delivery like any other actor, so a message of the wrong type dead-letters here too, rather than being quietly recorded.	required
name	str | None	Its actor name. Generated when omitted, so several probes in one test need no names at all.	None
mailbox	MailboxConfig | None	Capacity and overflow behaviour, for a test about backpressure. Unbounded when omitted.	None

__repr__()

Render where the probe sits and how much it is holding.

expect_message(expected, timeout=None) async

Take the next message and assert it is the one expected.

Equality rather than identity, so this reads the same for a message that crossed a link as for one that did not. A local tell does deliver the very object that was sent, and receive is there for a test that wants to say so.

Parameters:

Name	Type	Description	Default
expected	T	What should arrive.	required
timeout	timedelta | None	How long to wait. DEFAULT_TIMEOUT when omitted.	None

Returns:

Type	Description
T	The message that arrived.

Raises:

Type	Description
AssertionError	If nothing arrived in time, or something else did.

expect_message_of(msg_type, timeout=None) async

Take the next message and assert what kind it is.

For when the contents are not known in advance, which is most replies carrying an id or a timestamp.

Parameters:

Name	Type	Description	Default
msg_type	type[M]	The type it should be.	required
timeout	timedelta | None	How long to wait. DEFAULT_TIMEOUT when omitted.	None

Returns:

Type	Description
M	The message, narrowed to that type.

Raises:

Type	Description
AssertionError	If nothing arrived in time, or the wrong type did.

expect_no_message(within=None) async

Assert that nothing arrives for a while.

This is the one wait a passing test actually spends, so the window is short by default. What it catches is a message sent immediately, which is the mistake that happens.

Parameters:

Name	Type	Description	Default
within	timedelta | None	How long to watch. NO_MESSAGE_WINDOW when omitted.	None

Raises:

Type	Description
AssertionError	If a message arrived.

expect_terminated(target, timeout=None) async

Assert that a watched actor stopped.

Parameters:

Name	Type	Description	Default
target	ActorRef[Any]	The actor that should have stopped. It has to have been passed to watch first.	required
timeout	timedelta | None	How long to wait. DEFAULT_TIMEOUT when omitted.	None

Returns:

Type	Description
Terminated	The signal that arrived.

Raises:

Type	Description
AssertionError	If no signal arrived in time, or one arrived about a different actor.

receive(timeout=None) async

Take the next message, waiting for one if none has arrived.

Parameters:

Name	Type	Description	Default
timeout	timedelta | None	How long to wait. DEFAULT_TIMEOUT when omitted.	None

Returns:

Type	Description
T	The message.

Raises:

Type	Description
AssertionError	If nothing arrived in time.

stop()

Stop the probe, as an ordinary actor stops.

Rarely needed: the system's shutdown stops it with everything else. Useful for a test where the probe itself has to stop, so that whoever was watching it hears about that.

tell(message)

Send this probe a message, as anything else would.

Parameters:

Name	Type	Description	Default
message	T	The message to record.	required

watch(target)

Watch another actor, so its stopping can be expected.

The probe is an ordinary watcher, so this works on a ref that points at another node exactly as it does locally.

Parameters:

Name	Type	Description	Default
target	ActorRef[Any]	The actor to watch.	required

Bases: Generic[T]

One behavior, run by hand, with everything it did written down.

behavior property

What the actor would handle the next message with.

children property

Just the spawns, which is what most tests are asking about.

ctx property

The context the behavior is run with.

effects property

Everything the behavior asked its context to do, in order.

is_stopped property

Whether the behavior has returned Behaviors.stopped().

self_inbox property

What the behavior has sent to itself, in order.

self_ref property

A ref to the behavior under test, which records rather than delivers.

Hand it to the code under test as a reply_to, then read self_inbox.

supervision property

The strategies wrapped around this behavior, outermost first.

The kit does not apply them: a handler that raises raises into the test. What can be asserted here is that the behavior declared what it meant to declare.

__init__(behavior, *, msg_type=None, name='test', system='test', settings=None)

Prepare a behavior for running, evaluating any setup at once.

Parameters:

Name	Type	Description	Default
behavior	Behavior[T]	What to test. Behaviors.setup(...) is evaluated here, as a cell would evaluate it on spawn.	required
msg_type	MessageType | None	What it accepts, when the behavior itself cannot say. Read from the behavior otherwise.	None
name	str	The actor name this behavior thinks it has.	'test'
system	str	The system name its path sits in.	'test'
settings	TapioSettings | None	Tunables, for a test about validate_on_tell. Read from the environment when omitted.	None

Raises:

Type	Description
BehaviorTypeError	If the message type cannot be resolved, exactly as a spawn would fail.
TapioError	If the behavior needs a cell to exist at all, which with_timers and with_stash do.

__repr__()

Render the current behavior and whether it is still running.

child(name)

Return one spawn by name.

Parameters:

Name	Type	Description	Default
name	str	The child's name.	required

Returns:

Type	Description
Spawned	The recorded spawn, including the ref and what has been sent to it.

Raises:

Type	Description
AssertionError	If nothing by that name was spawned.

run(message) async

Handle one message, and become whatever it returned.

Parameters:

Name	Type	Description	Default
message	T	The message to handle. It is validated against the declared message type first, exactly as delivery would.	required

Returns:

Type	Description
Behavior[T]	What the behavior returned, before it was resolved against the
Behavior[T]	current one.

Raises:

Type	Description
MessageTypeError	If the message is not of the declared type.
AssertionError	If the behavior has already stopped.
Exception	Whatever the handler raised. The kit does not supervise, so a failure surfaces in the test.

signal(signal) async

Deliver one lifecycle signal, and become whatever it returned.

Parameters:

Name	Type	Description	Default
signal	Signal	The signal to deliver.	required

Returns:

Type	Description
Behavior[T]	What the behavior returned.

Raises:

Type	Description
AssertionError	If the behavior has already stopped.

Bases: Effect

The behavior started a child.

behavior = behavior instance-attribute

The behavior it was started with, to assert on or to test in turn.

inbox = ref.inbox instance-attribute

What the behavior has sent to that child so far.

mailbox = mailbox instance-attribute

The mailbox configuration it was given, or None for the default.

name = name instance-attribute

The child's name. Generated names begin with $.

ref = ref instance-attribute

The ref the behavior received back.

__eq__(other)

Equal to another spawn of the same name, and to that name.

Comparing against the bare name is what makes the common assertion read well: assert kit.effects == ("worker",). Comparing the behavior objects would test identity of a closure, which says nothing.

__hash__()

Hash by name, to match equality.

__init__(name, behavior, mailbox, ref)

Record one spawn, and the ref that was handed back for it.

__repr__()

Render the name and what it was started with.

Bases: Effect

The behavior started or stopped watching another actor.

ref = ref instance-attribute

The actor it was about.

watching = watching instance-attribute

True for a watch, False for an unwatch.

__eq__(other)

Equal when the same actor was watched, or unwatched, either way.

__hash__()

Hash by path and direction, to match equality.

__init__(ref, *, watching)

Record one watch or unwatch.

__repr__()

Render which way round it was, and about whom.

Bases: ActorRef[T]

A ref that writes messages down instead of delivering them.

It is what a behavior under test sends to. There is no cell behind it, so it cannot be watched and cannot be asked: those need a running system and a TestProbe.

inbox = [] instance-attribute

Everything sent to this ref, in order, as the objects sent.

__init__(path)

Create an empty recording ref at a path.

__repr__()

Render the path and how much it is holding.

offer(message) async

Record a message. Nothing here has a mailbox to be full.

Parameters:

Name	Type	Description	Default
message	T	What was sent.	required

tell(message)

Record a message.

Parameters:

Name	Type	Description	Default
message	T	What was sent. It is kept as it is, so an assertion can compare by identity as well as by equality.	required

The pytest plugin: fixtures for tests that need a running system.

Installed with tapio and registered through the pytest11 entry point, so there is nothing to add to conftest.py and nothing to import. Ask for a fixture by name and it is there.

async def test_the_greeter(actor_system, make_probe):
    probe = make_probe(Greeted)
    greeter = actor_system.spawn(greeter_behavior(), "greeter")

    greeter.tell(Greet(whom="world", reply_to=probe.ref))

    await probe.expect_message(Greeted(whom="world"))

Every fixture here terminates what it started, and actor_system asserts on the way out that the test left no task and no thread behind. That check is the reason the fixtures exist at all: a system a test forgot to terminate keeps a port and a thread pool, and the failure lands in whichever test runs next.

The fixtures are async, so the plugin needs an asyncio test runner. pytest-asyncio in auto mode is what tapio's own suite uses.

ProbeFactory = Callable[..., TestProbe[Any]] module-attribute

Makes a probe in the test's system, given what it should accept.

actor_system(tapio_settings) async

A running system, terminated however the test ends.

Parameters:

Name	Type	Description	Default
tapio_settings	TapioSettings	What to build it with. Override tapio_settings to change them, including switching remoting on.	required

Yields:

Type	Description
AsyncIterator[ActorSystem]	The system.

Raises:

Type	Description
AssertionError	If the test leaves a task or a thread behind, which means something outlived the system it belonged to.

make_probe(actor_system)

Make probes in the test's system.

replies = make_probe(Greeted)
named = make_probe(Greeted, name="replies")

A factory rather than a probe, because a probe declares what it accepts and only the test knows that.

Parameters:

Name	Type	Description	Default
actor_system	ActorSystem	The system the probes are spawned in.	required

Returns:

Type	Description
ProbeFactory	A callable taking the message type, and optionally a name and a
ProbeFactory	mailbox configuration.

tapio_settings()

Settings for the test's system, with the environment left out.

Override this fixture to change them. Reading TAPIO_ variables is deliberately switched off: a developer's environment should not be able to change what a test is asserting.

Returns:

Type	Description
TapioSettings	The settings the actor_system fixture uses.

Assertions that a block of work left nothing running behind it.

The runtime does not use task groups, because supervision needs semantics a task group cannot express. The price is that "no orphaned tasks" is an invariant the library holds rather than one the language enforces. These helpers turn it into an assertion, and every lifecycle test wraps itself in one.

assert_no_leaked_tasks()

Fail if the block leaves an unfinished task behind.

Tasks already running when the block opens are ignored, so this nests inside a test runner that has tasks of its own.

Yields:

Type	Description
None	Nothing. The check runs when the block exits.

Raises:

Type	Description
AssertionError	If a task created inside the block is still pending.

assert_no_leaked_threads()

Fail if the block leaves a thread behind.

The companion to the task check, for the blocking-call pool. A terminated system must leave no live threads either, and the pool is the one piece of the runtime that is not a task.

Yields:

Type	Description
None	Nothing. The check runs when the block exits.

Raises:

Type	Description
AssertionError	If a thread started inside the block is still alive.

Two nodes in one process, and a way to break the link between them.

Everything interesting about remoting happens when the network misbehaves, and none of it can be tested by waiting. A partition is not a slow link: it is a link that carries nothing and reports nothing, which is exactly what a test cannot arrange by hand without a second machine.

So the fault sits inside the link. A wrapper goes between an association and its socket, and it drops, delays or swallows frames on command. Nothing real is broken, nothing is unplugged, and the system under test cannot tell the difference: frames stop arriving, the failure detector gives up, and the quarantine that follows is the production one.

async with two_nodes() as nodes:
    worker = nodes.beta.spawn(work(), "worker")
    ...
    nodes.partition()      # both directions, both nodes
    ...                    # watchers get Terminated, sends dead-letter
    nodes.heal()           # the packets flow again, and nothing re-associates
    await nodes.alpha.remote.reconnect(nodes.beta.address)

The pair runs in one process on loopback ports the OS picks, so a test needs no orchestration and no port nobody else is using.

LinkFaults

What is wrong with one system's links, and how wrong.

One of these covers every link a system opens or accepts after it is installed, because a partition is a property of the network rather than of one connection. The settings are read at each frame, so a link already open is affected the moment they change.

partitioned property

Whether frames are currently going nowhere in either direction.

__init__()

Start with links that behave.

__repr__()

Render what is currently wrong.

allow_write() async

Decide what happens to a frame on its way out, and count it.

Returns:

Type	Description
bool	Whether the frame should be written. A False means it is lost,
bool	and nothing anywhere is told: that is what makes it a fault worth
bool	injecting rather than an error worth handling.

delay(seconds)

Hold every frame this system writes for a while before sending it.

Parameters:

Name	Type	Description	Default
seconds	float	How long. 0 stops delaying.	required

drop(frames)

Swallow the next few frames this system writes.

Parameters:

Name	Type	Description	Default
frames	int	How many to lose. Link frames count: a lost heartbeat is one of the things worth being able to arrange.	required

heal()

Let frames through again.

Nothing re-associates by itself. A system that gave up on a peer stays given up on until remote.reconnect says otherwise, which is what this is for testing.

partition()

Cut this system off: nothing it writes leaves, nothing it reads arrives.

Nothing raises and nothing is told, which is the point. A partition looks exactly like a peer that has stopped talking, and telling those apart from one node is the problem remoting cannot solve.

A socket closing on the other side is silenced too. A peer that gives up first really does close its connection, and a FIN arriving through a partition would hand this side a piece of news the network was supposed to be swallowing.

wait_healed() async

Wait until frames are allowed through again.

wrap(link)

Put a faulty link in front of a real one.

Parameters:

Name	Type	Description	Default
link	FrameLink	The link just opened or accepted.	required

Returns:

Type	Description
Link	The link the association will use instead.

TwoNodes

Two systems on loopback, and the network between them.

alpha = alpha instance-attribute

One system.

alpha_faults = alpha_faults instance-attribute

What is wrong with alpha's links.

beta = beta instance-attribute

The other.

beta_faults = beta_faults instance-attribute

What is wrong with beta's links.

__init__(alpha, beta, alpha_faults, beta_faults)

Hold the pair and the faults on each side.

__repr__()

Render both systems and the state of the network.

heal()

Let the packets flow again, which on its own repairs nothing.

A node that gave up on its peer stays given up on. remote.reconnect is the repair, and it is explicit because a false alarm has already told watchers that live actors are gone.

partition()

Cut both nodes off from each other, with both still running.

Each side breaks its own links, so neither hears the other and neither has died. That is the case worth testing: both will declare the other unreachable, and both will be locally correct.

link_faults(system)

Install fault injection on every link a system opens from now on.

Parameters:

Name	Type	Description	Default
system	ActorSystem	The system to break links for. Call it before any traffic, so that every link is wrapped and a partition covers all of them.	required

Returns:

Type	Description
LinkFaults	The controls.

Raises:

Type	Description
TapioError	If the system has remoting switched off, in which case it has no links to break.

two_nodes(*, alpha='alpha', beta='beta', unreachable_after=timedelta(milliseconds=300), heartbeat_interval=timedelta(milliseconds=20)) async

Start two systems that can reach each other, and stop them afterwards.

Both listen on loopback ports the OS picks, so nothing has to agree on a number in advance, and both are terminated however the block ends.

The default timings are far shorter than production ones, because a test that waits ten seconds to see a quarantine is a test nobody runs. They are still a heartbeat interval well inside a detector window, which is the only relationship between the two that matters.

Parameters:

Name	Type	Description	Default
alpha	str	The first system's name.	'alpha'
beta	str	The second system's name.	'beta'
unreachable_after	timedelta	How long silence lasts before a peer is given up on.	timedelta(milliseconds=300)
heartbeat_interval	timedelta	How often an idle link says it is still there.	timedelta(milliseconds=20)

Yields:

Type	Description
AsyncIterator[TwoNodes]	The pair, and the controls for breaking the network between them.

Errors

The tapio error hierarchy.

Every error tapio raises derives from TapioError, so a caller can catch the whole library with one clause. Where an error has an obvious builtin counterpart it inherits from that too: MessageTypeError is a TypeError and AskTimeoutError is a TimeoutError, so existing except clauses keep working.

No tapio error inherits from ValueError. Pydantic turns a ValueError raised inside a validator into a ValidationError, which would bury the message. Raising something else lets it propagate intact.

ActorNameError

Bases: TapioError

A child could not be given the name it asked for.

Names are unique among an actor's live children, because the path they form is the actor's identity in logs and in every error message.

ActorSystemTerminating

Bases: TapioError, RuntimeError

An operation was attempted on an actor or system that is shutting down.

AskTargetTerminated

Bases: TapioError

The target of an ask stopped before it replied.

AskTargetUnreachable

Bases: TapioError

The peer holding the target of an ask became unreachable.

Different from AskTargetTerminated on purpose. That one says an actor stopped, which is a fact. This one says a link went silent, which is a judgement that can be wrong: the actor may be alive on the other side of a partition. Both fail the ask at once rather than after the full timeout, and which one arrived tells the caller whether retrying elsewhere makes sense.

AskTimeoutError

Bases: TapioError, TimeoutError

No reply arrived within the ask timeout.

AskTypeError

Bases: TapioError, TypeError

A reply to an ask did not match the expected reply type.

BehaviorRegistrationError

Bases: TapioError

A behavior could not be offered to peers, or was never registered.

Raised at import time for a duplicate factory key and for a factory whose arguments model cannot be resolved, since a factory no peer could call is a bug where it is written. Raised at construction for a spawner offering a key nothing registered, which is almost always a typo in the allowlist.

BehaviorTypeError

Bases: TapioError, TypeError

A behavior's message type could not be resolved.

Raised when there is neither an explicit msg_type nor a readable annotation. A behavior with no message type is never spawned, because silently skipping the type check is what the check exists to prevent.

ClusterError

Bases: TapioError

Clustering was asked for something it cannot do.

Raised where the caller can act: a system with no address to be dialled at, a join with no seeds to ask, or a join or a leave that did not finish in the time allowed. The last of those does not stop the node trying, so catching it is a decision about how long to wait rather than about whether the cluster is broken.

FrameTooLargeError

Bases: MessageEncodingError

A frame exceeded the configured size limit.

On the way out it raises at the send site. On the way in, the declared length is checked before the body is read, so the frame costs a header and a refusal instead of the memory it asked for.

HandshakeError

Bases: TapioError

A link was refused before it carried a single message.

The causes are a version this system does not speak, a secret that did not match, or a peer that stopped talking part-way through. The connection is closed, the reason is logged, and no further frames are read. A wire format that half works is worse than one that refuses.

InsecureRemoteConfig

Bases: TapioError

Remoting was configured to listen beyond loopback with nothing to prove.

Raised at system construction, so a deployment that would accept frames from anything that can reach the port fails to start. The error names both settings involved: bind somewhere else, or set a secret.

MailboxFullError

Bases: TapioError

A bounded mailbox with the Fail overflow strategy was full.

Raised in the sender, because only the sender knows whether to retry, drop the message, or escalate.

MessageDecodingError

Bases: TapioError

A frame could not be read.

Never raised into application code. The receiving end turns it into a dead letter naming what was wrong, because the failure belongs to a peer and there is no local caller to tell.

MessageEncodingError

Bases: TapioError

A message could not be written to the wire.

Raised at the send site, because the message belongs to the sender. An error about it is the sender's to catch, as it is for a local tell.

MessageRegistrationError

Bases: TapioError

A message type could not be registered, or was never registered.

Raised at import time for a duplicate wire key, because two classes sharing one would decode as whichever imported last. Raised at encode time for a type with no key, because a key is never an import path and a peer could not rebuild an unregistered type.

MessageTypeError

Bases: TapioError, TypeError

A message does not match the recipient's declared message type.

Also raised when a declared message type is not a tapio.Message subclass, because re-validation on a plain BaseModel silently does nothing.

RefResolutionError

Bases: TapioError

A ref could not be rebuilt from its string form.

Raised when no system is in scope to resolve it against, and when the string is not a ref at all. model_dump() on a model holding a ref works anywhere. Feeding the result back to model_validate() works only inside a system's decode path or an explicit with system.as_deserialization_context(): block. A ref is a handle into a live runtime, and there is no meaningful ref without one.

StashOverflowError

Bases: TapioError

A stash was full and one more message was put aside.

Raised in the actor that stashed, because only it knows whether to drop the message, reject it, or let the failure become a supervision decision. A stash is bounded for the same reason a mailbox can be: it holds traffic the actor is not keeping up with.

TapioError

Bases: Exception

Base class for every error raised by tapio.

WatchError

Bases: TapioError

A ref could not be watched.

Raised for a ref with no live cell behind it, and for an actor watching itself. The second would promise a signal that cannot be delivered: once the actor has stopped, nobody is left to read its mailbox.