Architecture

4.1Names and the control connection

A name is 1 to 245 characters of [A-Za-z0-9_.-], not starting with . or - (names.isValid). Both sides turn it into the same rendezvous address: on Linux, the abstract AF_UNIX socket fastipc-<euid>-<name> (no file on disk, and private to the user through the uid); on Windows, the pipe \\.\pipe\fastipc-<session>-<name>; on macOS, a socket file in the user's private directory, claimed by flocking a lock file beside it. A name too long for the address keeps its first 32 characters (8 on macOS), followed by ~ and 32 hex digits of its SHA-256. A valid name can't contain ~, so a shortened name never collides with a real one.

Only two 64-byte frames ever cross the control connection (wire_protocol.ControlStreamSegment and wire_protocol.ControlStreamReady): the server's SEGMENT, with the ring capacity, the segment size and a random 16-byte session id, and the client's READY. On Linux and macOS, SEGMENT also carries the segment's file descriptor as SCM_RIGHTS ancillary data; on Windows the client opens the segment by its session id instead. After READY, nothing more is sent. From then on the control connection's only job is to close when a side goes away.

4.2The handshake, on the caller's thread

fipc_stream_create (create in stream/server.zig) does nothing but claim the name (Listener.claimName); no background thread starts. fipc_stream_accept (accept) does all of the setup on the calling thread, in a loop of single steps around two waits:

1. Wait for a client (acceptClient → control.acceptClient → platform.acceptWithin), Then, without waiting: check for a fork and the peer's user (platform.peer), create the segment and connection object (newSetup), and send SEGMENT. The setup is now pending.
2. Wait for READY (awaitReady → platform.readWithin), collecting bytes into pending. Once READY is complete, the hand-over (handOver) moves the connected handle into the connection, publishes its session and the CONNECTED phase, starts its watcher (endpoint.startAccepted) and returns the connection.

Resumable. A timeout can only end a call inside a wait, and pending survives it, so the next call picks up where the last one stopped. A server polling with a timeout of 0 admits a client within two calls. Every path out of pending other than the hand-over goes through dropPending. While the last connection returned is still open (client_open), accept returns INVALID immediately: one client at a time.

On the other side, fipc_stream_connect (connect → handshake in stream/client.zig) retries until its deadline. Each attempt runs connectOnce (if nobody is listening or the server is busy, it backs off from 1 ms, doubling up to 100 ms, and tries again), then awaitSegment, then attach (map and check, §4.3), and finally sends READY. Since each side blocks on the other, a client and its server must run on different threads.

4.3The segment

A segment is align_up(640 + 2 × capacity, 4096) bytes: a 640-byte header (wire_protocol.SegmentHeader), then the server-to-client ring (s2c), then the client-to-server ring (c2s). The capacity is a power of two from 1 KiB to 2 GiB.

Creation (segment.create, server side). On Linux: memfd_create, ftruncate, then fallocate of the whole size, so a full /dev/shm fails now with NO_MEMORY instead of later with SIGBUS; then seals, so nobody can resize it. On macOS: shm_open under a random name, unlinked at once. On Windows: a pagefile-backed section named after the session id, plus four named auto-reset events. The kernel zeroes the memory; the server writes the header and only then offers the segment.

Attach (segment.attach, client side) checks everything before use: the seals and exact size on Linux, the object's type and size on macOS, then the header's magic FIPCSEG1, capacity, header size and session id against what SEGMENT said. The client writes nothing to the segment before READY. After that, neither side ever reads capacity from shared memory again: each keeps a private copy (Hot.view_capacity), so the peer can't change it underneath them.

4.4The ring and its frames

head and tail aren't offsets but monotonic byte counts: head counts every byte the writer has ever published, tail every byte the reader has consumed. They only wrap at 2⁶⁴. A position's offset in the ring is pos mod capacity (a mask, since the capacity is a power of two), and the ring holds head − tail bytes. Counting this way avoids the classic ambiguity of offset-based rings, where equal offsets could mean either empty or full.

A ring holds frames: a 16-byte header (wire_protocol.FrameHeader: flags, frag_len, total_len) followed by a piece of a message, padded to 16 bytes. A frame never wraps: if the next frame doesn't fit before the ring's end, the writer fills the end with a PAD frame and starts at offset 0. So every piece is contiguous and 16-byte aligned, which is what lets the zero-copy calls hand out a plain pointer into the ring.

A message longer than one piece (at most capacity − 64 bytes, fipc_max_piece) travels as several frames: the first carries START and the total length, the last carries END. A receive checks the total length against the caller's buffer before consuming anything, so TOO_LARGE leaves the message intact. It then copies the pieces out one by one, publishing tail after each so the writer can reuse the room. That way any message size fits through any ring size, with no allocation. If a receive stops halfway through a message, the rest of it becomes orphans (pieces without START), which the next receive skips. If a writer stops mid-message, it simply begins its next message with START, which tells the reader to drop the unfinished one.

inline fn plan(space: Space, capacity: u32, frame: u64) Plan {
    if (space.free > capacity or space.contiguous % wire_protocol.frame_align != 0) return .corrupt;
    if (space.contiguous < frame) return .pad;
    if (space.free < frame) return .wait;
    return .reserve;
}

Pieces of 64 bytes or more go through copyLong (core/copy.zig), which copies in 32-byte vectors with aligned stores, so the speed doesn't depend on whichever memcpy the library happens to link against.

4.5Indices, kept indices and cache lines

A cache line is the unit the CPU moves between cores: 64 bytes on x86-64 (whose prefetcher also fetches lines in adjacent pairs), 128 bytes on Apple Silicon. When two cores write different variables that happen to share a line, the line bounces between them on every write even though no data is actually shared. That's false sharing, and the layout avoids it both inside the segment and inside each process.

In the segment, each index sits on 128 bytes of its own, with the sleep flag of the side that waits for it (§4.3), so a publish (store the index, swap the flag) touches one line, and on either architecture no other side's store takes it.

In the process, the connection object starts with three 128-byte lines laid out by hand (std.atomic.cache_line), checked by a comptime block in endpoint.zig:

So one thread can send while another receives on the same connection, and neither writes a line the other writes. The view is published once, before CONNECTED: first its fields (monotonic), then view_session and view_published (release). Every data call starts by loading view_published (acquire).

Kept indices. A reader remembers the last head it loaded (head_seen) and only loads the shared one again once it has used up everything before it; a writer does the same with tail (tail_seen). Since indices only grow, a kept copy can only lag the shared one, never run ahead of it: nothing unpublished is read and nothing unconsumed is overwritten. Meanwhile each index's line stays in its writer's cache instead of crossing between cores on every message. That alone measured at about 2 to 4 times the small-message rate.

Waiting for an eighth. A writer that finds the ring full spins until an eighth of the ring is free (writer_spin_share), not just enough for its frame, and then fills that room without loading tail again.

4.6Waits and wake-ups

A side only waits when it has nothing to do: a reader for data, a writer for room. It spins first (up to 4,096 CPU pause hints, spin_iters), then sleeps in the kernel in slices of at most 100 ms. Neither side makes a system call until one of them has given up spinning.

Sleeping uses a sleep flag next to the index, and every change to a flag is an atomic swap:

pub inline fn publishHead(hot: *const Hot, ring: Ring, head: u64) void {
    ring.indices.writer.head.store(head, .release);
    wakeIfAsleep(hot, sleeper(ring, .data));
}

inline fn wakeIfAsleep(hot: *const Hot, who: Sleeper) void {
    if (who.flag.swap(0, .acq_rel) == 1) wake(hot, who.flag, who.object);
}

The waiter must set its flag with a swap, not a plain store. On x86 a store can sit in the store buffer while a later load runs ahead of it (store buffering); the waiter could then miss a publish and sleep a whole slice with data waiting. A litmus test in ring_wait.zig races the real protocol against that weaker variant: the real one must never lose a wake-up, and the weak one must lose some on the test machine, or the test is skipped, since a zero there would prove nothing. A writer's publish costs one plain store and one locked swap; the waiter's swaps are all on its slow path. The reader's publish of tail only looks at the writer's flag, and swaps it when it is set: the room's wake is inexact, and the reader makes it exact on its slow path, before it sleeps (docs/protocol.md §6.1). A writer waits for room only on a full ring, and a locked swap for every message a reader consumes would cost x86 a quarter of its small-message rate.

Spinning politely. Every time a reader looks at head, it pulls that line away from the writer, which has to win it back for its next publish. So a reader that is only receiving (it has sent nothing since its last wait, per SendLine.head_sent) looks once, then lets stream_pace pause hints go by before it looks again (spin): 32 on x86-64 Linux, 64 on Windows, 512 on aarch64, each tuned on real hardware and kept in the stream_paces table. A reader waiting for a reply looks after every hint.

Waking your own waiters. fipc_cancel and the peer's end use the same rule: publish the change in requests or phase, then swap this side's own two flags to 0 (wakeOwnWaiters). The peer's sleepers are never touched.

4.7Zero-copy

Zero-copy send. sendAcquire reserves one frame at the head and records it on SendLine; the application writes straight into the ring; sendCommit then writes the header (START | END) and publishes. A zero-copy message is always a single piece, so it can be at most fipc_max_piece bytes.

Zero-copy receive. recvAcquire returns a one-piece message in place and records it on RecvLine; recvRelease (or the next receive) consumes it exactly once, and only if it is still the next frame.

4.8The watcher: noticing a dead peer without heartbeats

Once a connection is up, its watcher (control.runWatcher, a task on the connection's Io) does exactly one thing: it reads from the control connection. Nothing is ever sent there again, so the read only returns when the peer's end closes, and the kernel closes a process's sockets and pipes whenever it exits, crashes included. So there is no heartbeat, no shared clock and no timeout: end-of-file is the death notice. Whatever the read returns (unless this side's own close stopped it), the watcher closes the connection, publishes PEER_DEAD and wakes this side's ring waiters.

From then on, sends return DISCONNECTED immediately, but receives first deliver every message the peer finished sending: a waiting receive takes one more look at the ring after it sees the end (checkStop in ring_wait.zig). The trade-off: a hung peer is never detected, and a crashed one only once its process has fully exited.

Close (endpoint.close → teardown) follows the stop rule: set requests.shutdown; unblock the watcher's read with platform.unblock (on Linux and macOS, a shutdown of the socket, which ends a blocked read without a signal); wait for the watcher with platform.endTask (on Windows this cancels the pipe read through Io); then clear the view and unmap, under the endpoint's mutex. The library never installs a signal handler and never sends a signal.

4.9The process: the shared Io, handles, fork and keep-loaded

The shared Io. All watchers in a process run on a single std.Io.Threaded, reference-counted by the library's objects (process_state.acquire, release): the first object creates it and the last one released joins its threads. Objects from the native Zig API share it too.

Handles. Every OS handle an object holds sits in a named slot of its Handles (listening, cancel, connected, spare, segment, received, and a broadcast's, a queue's or a service's poller and poller_signal), opened and closed under the process-wide handle lock, so leaks and fork can be reasoned about one slot at a time.

fork (Linux, macOS). A child created by fork() inherits every descriptor but none of the threads. pthread_atfork handlers take the reference-count lock and the handle lock before the fork. In the child, afterForkInChild walks the fork registry of every Handles, closes each descriptor (only closes: a shutdown would end the parent's connection too), clears each view_published and marks the object as forked. Calls on an inherited object return INVALID, and the first one logs why. One handle can't be created under the handle lock, the socket that accept returns, so control.acceptClient compares the fork count (forkCount) before and after the accept and drops a client accepted across a fork.

Keep-loaded. Fork handlers can't be unregistered, so if the library were unloaded, the next fork would jump into unmapped code. To prevent that, keepLoaded reopens the library's own image with RTLD_NOLOAD | RTLD_NODELETE when the handlers are first registered, and a later dlclose leaves it in place.

4.10Trusting nothing in shared memory

The peer runs as the same user, so it's trusted not to be malicious, but not trusted to be correct. Corrupt content makes a call fail with INVALID; it never causes a crash, a hang or an access outside the mapping. Concretely:

• A ring holding more than its capacity (head − tail > capacity) is corrupt for both sides (ring_wait.corrupt, and Space.plan's .corrupt). Without that check, a tail past the head would make every wait return "ready" and send a receive walking the ring forever.
• A head off a 16-byte boundary is corrupt for its writer (Space.plan).
• A frame that would end past the ring's end is corrupt for its reader and left where it is (Ring.fits). Its 16-byte header has already been read by then, at most 15 bytes past the ring's end, but that memory is still mapped: the other ring or the segment's last page follows.
• A piece that doesn't continue its message is consumed and its message dropped (continues), and so is a START whose total length is no longer than its own piece (skipToMessage): better to lose one message than wedge the ring.
• total_len needs no bound: the reader never allocates, so an absurd length is just TOO_LARGE.
• The capacity is never read from shared memory after the attach.

5.1Roles and objects

• The owner is the publisher's process. fipc_broadcast_create (server.create) claims the name as a stream server does (Admission.init) and builds the whole Channel up front, so nothing grows as subscribers join: the segment with every slot, the Roster, one member record per slot (so the poller never allocates), the publisher's private record of payload positions (payload_positions, a sixteenth of the capacity) and the poller task, which admits joiners from then on (§5.6). It returns the publisher's connection. There is no separate listener: the publisher's connection owns the channel, and closing it ends the broadcast and frees the name at once.
• The publisher is an Endpoint of view kind broadcast_publisher, with no control connection and no watcher. Its state, writer.Writer, belongs to the sending thread. Its sends only ever wait for gated subscribers, and never return DISCONNECTED.
• A subscriber is an Endpoint of view kind broadcast_subscriber, with its own mapping, its receive state (reader.Reader) and a watcher, just like a stream connection.
• A member (server.Member) is the owner's record of one slot: while a subscriber holds it, that subscriber's control connection and what the poller has read from it. Member records only change under the lock of the channel's Owner (core/owner.zig, shared with queues and services), which the data path never takes.
• The poller (pollMembers), one task per broadcast, admits joiners and watches every member's control connection, so a broadcast costs one thread no matter how many subscribers join. platform.Poller is level-triggered epoll on Linux and kqueue on macOS. On Windows it uses ReadFileEx reads whose completion routines run during the poller thread's alertable waits; not a completion port, because Io runs its own routines on the same pipes.
• The member count (Channel.joined, read by fipc_broadcast_wait_for_members) can be read or waited on from any thread. Each change bumps a process-local word (CountWaiters.changes) and wakes its futex waiters, which only sleep while the word still holds the value they loaded before checking the count.
• Dispatch. The data calls (core/calls.zig) load view_kind once and switch on it: a stream call goes to stream/data_path.zig, a broadcast call to the publisher's or the subscriber's side. A broadcast message is always a single piece, and a broadcast's fipc_max_piece is a quarter of the capacity.

5.2The broadcast's segment: two regions

The layout (segment.describe) puts everything only the publisher writes first and everything subscribers also write after it, each region starting on a 64 KiB boundary (Windows' mapping granularity, and a multiple of every page size). Once attached, a subscriber makes the first region read-only in its own view (platform.protectReadOnly), so a stray write in a subscriber's process faults there instead of corrupting every subscriber's messages. Lines here are 128 bytes, the size of an Apple Silicon line and of x86's adjacent-line pair, so the words only the publisher writes, and each subscriber's cursor, never share a line with a word someone else writes. The words both sides write (armed_any with wake_gen, writer_sleeps, the two bitmaps) get lines of their own.

What a buggy subscriber can do. As with a stream's peer (§4.10), subscribers run as the publisher's user and are trusted not to be malicious, but not to be correct. Any subscriber can write all of region B: every slot's cursor, the wake words and the bitmaps. So a buggy one can disrupt delivery to the others: lost messages, or messages overwritten before a gated subscriber read them; sends that stall or time out; late or spurious wake-ups; even a gated subscriber unsubscribed because its cursor was set back while it slept, making it look stalled. What it can't do is crash anyone. The publisher relies on its own records (§5.5), and every subscriber bounds every index and checks every cell before copying, so a bad value makes the reading call fail with INVALID, never access memory outside the segment.

5.3A publish, step by step

Message i (counting from 0) uses cell i mod cell_count, where cell_count = capacity / 64, and its payload sits in the data ring at payload_position, a monotonic byte count like a stream ring's indices. Writer.send is reserve, a copy, then publish:

1. Place (payload_ring.place): right after the previous payload, its length rounded up to 64 bytes. If it won't fit before the ring's end, it goes at the start instead; a payload never wraps.
2. Room: check the next index against cell_limit and the payload's end against data_limit. Past either one, wait for gated subscribers (room, §5.5). With no gated subscribers, both limits simply advance a lap at a time.
3. Frontier: if the payload ends beyond the last announced frontier, store a new frontier (the payload's end, rounded up to the next sixteenth of the ring) and next_mark (this message's index), then a store fence. The publisher only ever writes data below a frontier it has already announced.
4. Copy the payload into the data ring.
5. Cell (writeCell). The cell is a seqlock: a sequence word marks it "being written" and then "complete", so a reader can detect a torn read. Store seq = (i+1) | busy, a store fence, the fields, then seq = i+1 with release.
6. Record the position in the owner's own payload_positions.
7. Publish next = i+1 (sequentially consistent), then load armed_any. Only if some subscriber has armed for a wake-up does the publisher do anything more (wakeSubscribers, §5.7).

pub fn writeCell(cell: *wire_protocol.BroadcastCell, index: u64, at: u64, len: u32) void {
    const sequence = index + 1;
    cell.seq.store(sequence | busy, .monotonic);
    fence.storeFence();
    cell.payload_position.store(at, .monotonic);
    cell.len.store(len, .monotonic);
    cell.flags.store(0, .monotonic);
    cell.seq.store(sequence, .release);
}

Because seq is the index plus one, 0 means "never written", so a freshly zeroed segment is already a valid empty broadcast and nothing needs to initialize the cells. Per message, the publisher pays for one copy, the cell's seqlock (a store fence, which is a compiler barrier on x86-64 and dmb ishst on aarch64, plus a release store), one store into its own record, one sequentially consistent store of next and one load of armed_any: roughly what a stream ring's publish costs. The frontier's store and fence only happen once per sixteenth of the ring. None of it depends on how many subscribers there are.

5.4A lossy read, and how a lap is detected

A lossy subscriber at position p never writes anything the publisher reads. It copies each message out, then proves the copy wasn't overwritten:

1. Load the cell's seq (acquire) and classify it (cellState): equal to p+1 means this is the message (here); larger, ignoring the busy bit, means the publisher has already reused the cell (lapped); anything else means not yet (pending).
2. readCell: load payload_position and len, a load fence, then seq again. If seq changed, the publisher started rewriting the cell in the meantime.
3. Check the fields (validPiece: 1 to max_piece bytes, inside the ring), then copy (seqlockCopy).
4. A load fence, then load frontier. Accept the copy only if frontier − payload_position ≤ capacity.

const piece = readCell(cell, expected_sequence) orelse return .torn;
if (!reader.layout.validPiece(piece.payload_position, piece.len)) return .corrupt;
len.* = piece.len;
if (piece.len > buf.len) return error.TooLarge;
seqlockCopy(buf[0..piece.len], reader.layout.bytes(piece.payload_position, piece.len));
// After the copy's loads: a frontier past a ring from the payload means the writer may have overwritten part
// of what was copied
fence.loadFence();
if (reader.layout.header.frontier.load(.monotonic) -% piece.payload_position > reader.layout.capacity) return .torn;

After a lap, the receive returns OVERRUN once, and resync jumps the subscriber to next_mark: the publisher's index when it announced its current frontier, clamped to the published messages after the subscriber's position. That lands it on a recent message straight away, rather than catching up a lap at a time. The receive reports how many messages were skipped, always at least 1, since the torn one counts. The copy deliberately reads memory the publisher may be writing at that very moment; that's how a seqlock works, and it is the one race the library tells ThreadSanitizer to ignore (seqlockCopy).

5.5A gated read, and how room is computed

A gated subscriber owns a cursor in its slot: the index of the first message it hasn't consumed. The publisher never writes cell i until every gated cursor has reached at least i + 1 − cell_count, and never writes payload bytes a full ring past a cursor's message. So a gated subscriber can read its message in place (nextInPlace, then either copy it for recv or hand out the pointer for recvAcquire) and store its cursor with release when it's done (publishCursor).

Room, from the publisher's side. The publisher keeps free_index, an index every gated cursor is known to have reached, and derives two limits from it: cells below free_index + cell_count, and data below payloadPositionOf(free_index) + capacity. Most sends compare against those two numbers and touch nothing else. When a send passes one, room computes need, the oldest message that has to be consumed to make room, and sweeps the cursors of every slot set in the gated bitmap. If the sweep finds a cursor below need, it stops at that laggard: the publisher waits for it (waitForLaggard), and the next pass resumes from there (Scan). Scan.prepare sets up each pass: it keeps the cursors already seen only while they still matter for need, and loads the publisher's next, which caps free_index, before the scan's first bitmap load (§5.6 explains why). With no gated subscribers, a sweep reads one line once per lap.

The publisher trusts its own records. A subscriber's process could map the segment writable and change anything in region B, or even a cell. So the publisher reads payload positions from payload_positions, in its own memory, never from the cells, and it only waits for and wakes slots its Roster holds, whatever bits the segment shows (nextGatedSlot ANDs the segment's gated word with the roster's). A corrupt cursor can only hold back or release that subscriber's own messages.

Stalls. Every wait for room also runs checkStalls. A gated subscriber whose cursor hasn't moved for the broadcast's max_stall_ms, counted from the first send it held back (time spent in sends that timed out adds up, in Stall), is reported once to on_stall (server.onStall), which sends it UNSUBSCRIBE (§5.8). Several stuck subscribers are handled together, each on its own clock, and the publisher's sleep is cut short when the next stall falls due.

5.6Joining and leaving: slots and generations

A broadcast's handshake has the same shape as a stream's, with different frames (wire_protocol.ControlBroadcastOffer, wire_protocol.ControlBroadcastJoin, wire_protocol.ControlBroadcastSlot: OFFER, JOIN, WELCOME, and later UNSUBSCRIBE and UNSUBSCRIBED). The owner runs its side on its poller (pollMembers, using the steps in core/admission.zig), never on an application thread, so a process can join its own broadcast from any thread, including the one that created it. No step blocks: each either completes at once or leaves the joiner for the poller's next wait. At most 8 joiners can be between their accept and their JOIN at any time:

1. Listen while a slot is free, a joiner record is free and no accept back-off is running (Admission.listenWhile, Poller.listen). Otherwise stop listening (Poller.unlisten), and joiners wait in the backlog, or for a pipe instance, within their own timeouts. The poller's wait only times out at a joiner's deadline or the end of a back-off, so an idle broadcast causes no wake-ups at all. To start and stop listening, Linux adds the listening socket to its epoll set and removes it; macOS adds and deletes an EVFILT_READ; Windows issues an overlapped ConnectNamedPipe whose event joins the poller's alertable wait, and cancels and awaits it.
2. Accept the client without waiting (Admission.accept → control.acceptClient: the fork check and the peer's user, then on Windows a fresh pipe instance) into a free joiner record, with its JOIN due within 2 s. Claim a slot (server.claimSlot → Roster.claim): the next free one, round robin, so a just-freed slot is the last to be reused. Its generation goes up by one and its cursor is set to 0. Send OFFER with the slot and generation, and watch the joiner's connection alongside the members'.
3. Read JOIN (onJoinerReady, frames.checkJoin: the slot and generation must be echoed back, the role must be valid, and a gated role is only accepted if the broadcast allows gated subscribers). The connection moves into the slot's member record (Admission.handOver). Then startMember runs under the broadcast's lock: for a gated joiner, mark the slot gated in the roster; start the subscriber (broadcast.startSlot, below); count the member; send WELCOME, then watch the connection (Poller.watch). Because the poller is level-triggered, it still reports an end that happened before the watch, on its own thread, after the member has joined. So a subscriber that has been told it joined is always watched and always counted.
4. Drop a joiner that disconnects, sends anything other than its JOIN, or sends nothing for 2 s after its accept (dropJoiner frees its slot and closes its connection); a genuine joiner sees the end and tries again within its own timeout. So a joiner that stalls or sends garbage holds up neither the other joiners nor any member's end, and ties up one of the 8 records and one slot for 2 s at most.

The generation keeps frames and stall reports about a slot's previous subscriber from being mistaken for ones about its current subscriber.

The start. Before counting the subscriber or sending WELCOME, the owner fixes where the subscriber starts, by storing the published next as the slot's cursor. For a gated joiner it stores it twice, the double store (broadcast.startSlot):

const cursor = &layout.slots[slot].cursor;
if (gated) {
    cursor.store(layout.header.next.load(.seq_cst), .monotonic);
    setBit(&layout.shared.gated, slot, .seq_cst);
}
cursor.store(layout.header.next.load(.seq_cst), .release);

Why twice? A sweep that sees the new gated bit reads a cursor at or before the start. A sweep that doesn't see the bit loaded its bitmap words before the bit was set, and the next it caps free_index with was loaded earlier still, so it is at most the start. Either way, the publisher can't overwrite the message the new subscriber starts at. Why the owner, not the subscriber? The publisher's count (fipc_broadcast_wait_for_members) includes the subscriber before WELCOME is sent, and the subscriber's process may take a while to read it. If the subscriber picked its own start from next after WELCOME, it would miss whatever the publisher sent in between. The owner's start is at or before every message sent after the count includes the subscriber (the count's increment is a release, and its load an acquire).

The joiner (client.join → client.attemptJoin) attaches and checks the segment, makes region A read-only, sends JOIN and reads WELCOME (if it reads an end instead, the owner dropped it, and it starts over), then begins at its slot's cursor (Reader.init).

Fixing the start at admission has two consequences. A gated joiner holds the publisher back from the moment it's admitted, like any gated subscriber, max_stall_ms included; one that dies right after WELCOME does so until the poller sees its end. And a lossy joiner of a busy broadcast whose process is slow to start reading may find the publisher already a lap ahead: its first receive then returns FIPC_OVERRUN, for messages sent after it was counted.

Leaving. When a gated subscriber closes, it clears its gated bit and wakes the publisher (Reader.leaveChannel), under the guard and only if its subscription is still running. Its watcher then closes the control connection, and the owner's poller sees the end and frees the slot (Roster.release). If the owner's read fails in any other way, or an unexpected frame arrives, the subscriber may still be running with the slot mapped, so the slot is retired instead (owner.MemberEnd, from endOf; Roster.retire): held by nobody and given to nobody until the broadcast ends.

5.7The sleeping words and the wake-ups

A subscriber with nothing to read (Reader.wait) spins, then sleeps. The spin's pace adapts (Reader.spin): if its first look found a message, the subscriber is trailing a busy publisher, so the next spin waits twice as long before looking, up to the platform's stream pace. Before sleeping, announceSleep runs: on Linux and macOS, set the sleeping flag (used by its own interrupt, below); read wake_gen; arm the slot (its bit in the armed bitmap, then its word's bit in armed_any); if gated, also store the cursor and check writer_sleeps; then check next, the cancel request, the phase and ended one last time, and sleep only if none of them changed. All of these accesses are sequentially consistent.

The publisher's wake (wakeSubscribers) runs after it stores next, and only if armed_any isn't 0: swap armed_any to 0, swap each armed word it points to to 0, keep only the bits of slots its roster holds, and wake those.

No wake is lost. The argument (docs/protocol.md §9.5) is the stream ring's, generalized to a bitmap, and it depends on all these accesses falling into one total order. A subscriber's own interrupt (a cancel, or the end of its subscription) wakes every sleeper of the broadcast on Linux and macOS, so it only does so if its receive actually announced a sleep (the sleeping flag) and the publisher hasn't already woken everyone on close. Otherwise N subscribers noticing the publisher's end would wake every sleeper N times.

The publisher waiting for room sleeps on writer_sleeps, the way a stream writer sleeps on its flag, in slices starting at 1 ms and doubling up to 10 ms (room_slice_ms). On every receive, a gated subscriber stores its cursor with release and checks the flag with a relaxed load, because the exact pair would cost a locked instruction per message. So the publisher can miss a cursor move that races with its own announcement; the 10 ms slice caps that delay. Before it goes to sleep itself, though, the subscriber does use the exact, sequentially consistent pair.

5.8Ends: the segment guard, unsubscribe and revoke

A gated subscriber writes to its slot (its cursor) and its gated bit. Once its subscription ends, the owner may hand the slot to the next joiner, so the departing subscriber must have stopped writing first. The segment guard guarantees that:

• A gated call enters (Reader.enterSegment): store in_segment = 1, then load the phase, both sequentially consistent. Only a call that saw CONNECTED may write the slot (live). It keeps the guard while it sleeps, so its mapping stays valid while it waits, and leaves on return.
• The watcher that ends the subscription (client.endSubscription) runs Reader.drop: store the new phase, then load in_segment, both sequentially consistent. It then wakes any sleeping receive and waits, in 1 ms naps, until no call is inside. Only then does it close the connection (which tells the owner) or revoke the mapping.

The two halves form the classic store-then-load pair (a Dekker pair): at least one side must see the other's store. Either the call sees the new phase and writes nothing, or the watcher sees the call inside and waits for it to leave.

if (!reader.gated) return;
reader.in_segment.store(1, .seq_cst);
const phase = endpoint.hot.phase.load(.seq_cst);
if (phase == .unsubscribed) {
    reader.in_segment.store(0, .release);
    return error.Unsubscribed;
}
reader.live = phase == .connected;

Unsubscribe (gated subscribers only). A stall report sends UNSUBSCRIBE, and the subscriber's watcher runs client.unsubscribe: it ends the subscription as unsubscribed through the guard, then revokes the whole mapping (platform.revoke: PROT_NONE, or PAGE_NOACCESS on Windows) under the endpoint mutex, so a message the application still holds in place faults instead of silently showing whatever the publisher writes there next. Finally it answers UNSUBSCRIBED, which frees the slot. From then on every receive returns UNSUBSCRIBED. A subscriber whose whole process is paused can't answer, so the publisher's sends time out until it resumes or exits.

The publisher's close (server.closePublisher) ends the broadcast. First it stores ended = 1 (after its last message) and wakes every sleeper once, so each subscriber drains what was published and reports DISCONNECTED without waiting on its control connection. Then it stops the poller and waits for it to finish, never cancelling it through Io. On Windows the poller's last step cancels every read it has in flight, plus the listening instance's connect, and runs their completion routines (Poller.drain), so none is left queued to the thread. Finally it closes the joiners' connections and the listening handle, which frees the name at once, and then every member's connection.

If the publisher's process dies instead, ended is never set. A gated subscriber can then no longer tell whether the publisher is still writing, so it stops trusting in-place reads and copies and checks each remaining message the way a lossy subscriber does (recvChecked), up to where its subscription ended (drain_end).

5.9Built to be killed

A broadcast spans many processes, and any of them can die at any instruction. Here is what happens in each case:

• A subscriber dies: the kernel closes its control connection, the owner's poller sees the end, frees the slot, clears its gated bit and wakes the publisher. No timeout is involved.
• A joiner dies mid-join: before sending JOIN, the owner reads an end where JOIN should be and frees the slot. After JOIN it is already a member, so its end frees the slot and clears its gated bit (set before WELCOME); until the poller sees that end, a gated joiner holds the publisher back.
• A subscriber writes garbage: every index it can influence is bounded (max_index, 2⁶³), every cell checked before a copy, every bit filtered through the owner's roster; the publisher's positions come from its own memory. A bad value makes the reading call fail with INVALID, never access memory outside the segment.
• A subscriber stops but doesn't die: a gated one is unsubscribed after max_stall_ms and its view revoked; a lossy one simply gets lapped.
• The publisher dies: each subscriber's watcher reads the end, drains what was published and reports DISCONNECTED.

The tests cover this with peers that crash at every step of the handshake and of a join (core/test_hooks.zig, in the slow tier), checked message streams to more subscriber processes than there are CPUs, and the chaos harness (devtool test chaos), which kills, pauses and restarts publishers and subscribers in a loop and checks for leaks.

6.1Roles and objects

• The owner is the process that calls fipc_queue_create (server.create). It claims the name as a broadcast owner does and builds the Queue up front: the whole segment with every slot, one roster per role (core/roster.zig: which slots are held, their generations and, on Windows, their wake events), one member record per slot, and the poller task, which admits joiners from then on. The Queue is the handle the call returns (fipc_queue_t), and the handle is the queue: closing it (fipc_queue_close) ends the queue for every member and frees the name.
• The owner isn't a member. To send or receive, it joins its own queue with fipc_queue_join, by name, from any thread, exactly as another process would, and its poller admits it like any other joiner. Since admission runs on the library's thread, even a single-threaded process can create a queue and then join it.
• A producer and a consumer are Endpoints of view kinds queue_producer and queue_consumer, each with its own mapping, its client.Member (its client.State, and a producer.Producer or a consumer.Consumer) and a watcher, just like a subscriber.
• A member (server.Member) is the owner's record of one slot: while a producer or consumer holds it, that member's control connection and the slot's generation. One poller per queue (pollMembers, on the same platform.Poller as a broadcast) admits joiners and watches every member's connection, so a queue costs one thread no matter how many members join. The lock of the queue's Owner (core/owner.zig, the type a broadcast and a service use too) guards the rosters and member records, and the data path never takes it.

6.2The queue's segment: a ring per producer

The layout (segment.describe) starts with a header, one 128-byte line each for the description, ended, the ready bitmap (bit p: "ring p may have a message") and the idle bitmap (bit c: "consumer c is parked"). Next comes one 128-byte slot per consumer, holding its park flag sleeps and its claim. Then one region per ring: a control block (tail, head and the room flag room_sleeps, each on its own line), the cells (one per 64 bytes of capacity) and the data ring. The producers' rings come first, then the return ring, ring number max_producers, whose producer is the owner. Consumers find it through bit max_producers of the same ready word, so the message path pays nothing extra for it; that's also why a queue has at most 63 producers. Every region is a whole number of pages. Nothing is read-only in any view, because a consumer claims a message by writing to its ring's cell.

A cell's state names its message. Message i of a ring uses cell i mod cell_count, and the cell's state word holds the message's index plus one in its top 56 bits, the claiming consumer in bits 2 to 7, and a phase in bits 0 and 1: published, claimed or done. Indices count a ring's messages over its whole life, across every producer that has held its slot, so a state never comes back once it has been left: a compare-and-swap expecting message h's state can never succeed on another message's (no ABA problem). A cell holds only the library's bookkeeping; the payload lives in the producer's data ring, rounded up to 64 bytes, and never wraps.

pub const QueueCellState = packed struct(u64) {
    phase: QueuePhase,
    claimer: u6,
    tag: u56,
};
pub const QueuePhase = enum(u2) { none = 0, published = 1, claimed = 2, done = 3 };

6.3A send and a claim, step by step

The producer (Producer.send: reserve, a copy, publish) writes only its own ring:

1. Room: the next cell and enough data-ring space for the payload must be free (fits); if not, it reclaims, then waits for room.
2. Copy the payload, write the cell's payload fields, then the state published(i) with release.
3. Publish tail = i + 1 (sequentially consistent), then check ready: if the ring's bit was clear (consumers had seen the ring empty), set it and wake one idle consumer. Finally, check ended.

A consumer (Consumer.recv → take) looks through the rings whose ready bit is set, starting after the ring it last took from, so consumers spread out across the rings:

1. Load tail, then head (validRing: they must be at most a ring of cells apart and below the index bound, or the ring is treated as corrupt and skipped), then the state of the cell at head.
2. If the buffer is shorter than the largest possible message, check the cell's len first: a message that doesn't fit returns TOO_LARGE with its length, and stays in the queue for any consumer.
3. Claim: record the intent in its own slot (claim, release), then compare-and-swap the state from published(head) to claimed(head, me). Exactly one consumer wins, and the winner advances head with another compare-and-swap. A consumer that finds a cell already claimed or done helps out by moving head past it. Claims always follow head, and head only passes a message once it's claimed.
4. Copy the message (or hold it in place until fipc_recv_release), then store done with release, and wake the producer if it is sleeping for room.

Reclaim. The producer frees the space of its consecutive done messages up to head, using its own record of payload positions (never the cells), and stops at the first message not yet done. So a message held in place holds back its producer's space, but not the other messages: consumers keep taking those until the producer is a full ring ahead of the held one.

6.4Ready bits, parks and one wake at a time

Two handshakes, with every access sequentially consistent, make sure no work is ever stranded:

• The ready bit. A consumer that finds a ring empty, or takes its last message, clears the ring's bit and then looks at the ring again, setting the bit back if a message has arrived. Together with the producer's store of tail and load of ready, that's a store-then-load pair on each side: either the consumer sees the new message, or the producer sees the bit clear and sets it. Either way, a ring with a message ends up with its bit set by someone who then claims the message or wakes a consumer.
• The park. A consumer with nothing to take spins, then swaps its sleeps flag to 1, sets its idle bit, and checks ready, its cancel request, its phase and ended; it only sleeps if none of them changed. A waker takes one idle bit (with an atomic AND), swaps that consumer's sleeps to 0, and wakes it if the swap returned 1.

A claim that leaves more messages behind wakes one more idle consumer (chaining); that's for parallelism only, not correctness. A producer waiting for room sleeps on room_sleeps in slices from 1 ms doubling to 10 ms, like the broadcast's publisher: a consumer's done is a release store and its check of the flag a relaxed load, so a producer can miss a done that races with its announcement. The slice caps that delay; the exact pair would cost a locked instruction per message.

6.5Joining, slots and draining

The owner admits joiners on its poller just as a broadcast owner does (§5.6: no step blocks, at most 8 joiners in setup, each dropped if no JOIN arrives within 2 s of its accept), as long as a slot of either role is free. It sends OFFER (the capacity, both maximums and the segment) and reads JOIN, which carries the joiner's role; this is the first the owner learns of it. It then claims a free slot for that role, round robin, with a new generation, resets the slot's words, counts the member and sends WELCOME with the slot. If no slot of that role is free, it drops the joiner, which retries within its own timeout. A new producer starts at its ring's tail, which must equal head.

A producer's slot is only reused once its ring has settled (ringSettled): every message the previous producer sent has been claimed (head = tail), and none is still held by a consumer (no consumer's claim points to a cell that is still claimed). Until then the slot is draining; the poller rechecks it before listening, every 5 ms while no slot is free. The next producer continues the ring's indices, so nothing its predecessor left behind can be mistaken for one of its own messages.

6.6Ends: the return ring, and the queue's end

• A member's end. Nothing travels on a member's connection after WELCOME, and the owner never closes a live member's connection while the queue runs, so when that connection ends, the member can no longer be writing to the segment. For a consumer, the owner reads its claim; if that cell still holds claimed(index, c) (heldClaim), it returns the message. It copies the message into the return ring without waiting, publishes it there as a producer would, and only then swaps the original to done and moves head past it, so the producer can't reuse the original's space until the copy exists. If the return ring is full, the claim stays, and so does the consumer's slot, until a later tick of the poller finds room. A held message from the return ring itself moves straight into the owner's scratch FIFO (up to capacity bytes, allocated as it fills), and later ticks publish it back from there. Together the return ring and the FIFO never hold more than the capacity, so no copy ever waits on a consumer that has ended. When a producer ends, its slot starts draining. A member that sends an unexpected frame, or whose connection fails some other way, might still be running, so its slot is retired for the life of the queue and its connection kept open.
• The queue's end. Closing the queue (fipc_queue_close, server.close) stops the poller, publishes whatever returns fit, stores ended, closes the joiners' connections and the listening handle (freeing the name at once), and then every member's connection. If the owner's process dies instead, each member's watcher reads the end of its connection and stores ended itself. From then on producers return DISCONNECTED, and consumers take what's left, in passes whose loads are sequentially consistent, before returning DISCONNECTED too. A send that returned OK is always taken by a draining consumer: the producer stores tail and then loads ended, the consumer loads ended and then tail, so at least one sees the other. A send that returned DISCONNECTED right as the queue ended may still have been delivered.
• A member's own failure (an unexpected frame from the owner) makes it return DISCONNECTED immediately, without setting ended: the queue carries on for everyone else. The member's watcher keeps the connection open until the member is closed, so the owner only frees the slot once the member can no longer write.

The tests cover this with a checksummed stress test of producers and consumers on threads (including more threads than cores); litmus tests of the claim, the ready bits, the park, the end and cell reuse; producer, consumer and owner processes killed at each step of a join (in the slow tier); the chaos harness's queue-* scenarios, which kill and pause owners, and producers and consumers holding messages, and end every cycle by accounting for every message sent (none lost but the one a killed consumer held, none delivered twice); and the fuzz harness's forged owners, forged joiners and corrupt members.

7.1Roles and objects

• The owner is the process that calls fipc_service_create (server.create). It claims the name as a queue's owner does and builds the Service up front: the whole segment, a ring for every client and a slot for every server, the owner's record of its members (core/members.zig: a roster per role, the draining client slots, one record per slot, the count of each role), and the poller task, which admits joiners from then on. The Service is the handle the call returns (fipc_service_t), and, as with a queue, the handle is the service: closing it ends the service for every member and frees the name.
• The owner isn't a member. To call or to serve, it joins its own service with fipc_service_join (FIPC_CLIENT or FIPC_SERVER), by name, from any thread, and its poller admits it like any other joiner. Since a reply never waits, one thread can create a service, join it as a client and as a server, and send, receive the request, reply and receive the reply in a row.
• A client and a server are Endpoints of view kinds service_client and service_server, each with its own mapping, its client.State, a watcher, and its side of the data path on cache lines of its own: a client's caller.Caller (its requests, a Sender of the ring set, and its Replies), a server's responder.Responder (a Taker of the ring set, the request it holds, and its replying thread's state). No new data call exists: a client's fipc_send is a request and its fipc_recv the next reply; a server's fipc_recv is the next request of any client and its fipc_send the reply. fipc_max_piece is the service's max_message, the owner's choice, from 1 byte to a quarter of the capacity, for requests and replies alike.
• Shared with the queue. The rings, the cells, the bitmaps, the claim, the park and the owner's member bookkeeping are the work queue's code, generic over a shape: a kind's wire types, roles, limits and wake names. core/ring_set.zig's RingSet, core/sender.zig, core/taker.zig and core/members.zig each have two instances, the queue's and the service's, and the service's differences are switched at compile time (replies, a taker's drains), so the queue's code is unchanged by them.

7.2The service's segment: a ring per client

The layout (segment.describe) is a queue's with the clients in the producers' place and no return ring: a header (the description, ended, ready with bit c for client c's ring, idle with bit s for a parked server: a 128-byte line each), one 128-byte slot per server (its park flag sleeps and its intent claim, a queue consumer's slot), then one region per client: its ring's control block, its cells (one per 64 bytes of capacity) and its data ring. Unlike a queue's, the cells are spread: request i's cell is i's low bits rotated left by 2, so consecutive requests' 32-byte cells are 128 bytes apart and a client publishing its next request never takes the line of the one a server is answering. The control block has one more line than a queue's: besides tail, head and room_sleeps, the client's receiving thread sleeps on reply_sleeps. One bit per client in one 64-bit ready word is why a service has at most 64 clients, and one per server in idle why it has at most 64 servers. Nothing is read-only in any view: a server claims in the cells and writes its reply into its client's data ring.

A request's room holds its reply. The client reserves R, the service's max_message rounded up to 64 bytes, and then the request's length rounded up to 64, in one piece that never wraps: [reply room][request]. The cell names the request's position, and the reply's room is the R bytes just before it. A cell is a queue's cell (its first 24 bytes are the same, so the claim is the same code) with two more words that the server writes before its done: the reply's length and the outcome, replied or abandoned.

pub const ServiceCell = extern struct {
    state: Atomic(u64), // a QueueCellState: published, claimed, done
    payload_position: Atomic(u64), // the request's; its reply's room is just before it
    len: Atomic(u32), // the request's length
    reply_len: Atomic(u32), // the reply's, written by its server before the done
    outcome: Atomic(u32), // a ServiceOutcome: none, replied, abandoned
    reserved: u32,
};

The segment is round4K(512 + 128 × max_servers) + max_clients × (round4K(512 + capacity / 2) + capacity) bytes, created whole at the create: 12 KiB for a 4 KiB service of one client and one server, about 6.3 MiB for 64 KiB rings, 64 clients and 8 servers. The magic is FIPCSVC1, the frames are of kind 3, and the roles, 5 and 6, go on from the queue's, so no kind's joiner is taken for another's. Its trust model is a queue's (§6.2): a buggy member can disturb the others' requests and replies, but every word that indexes or sizes anything is checked where it is read. A server checks a request's position and length before it writes anything, so a corrupt client can make it write only inside that client's own data ring; a client reads a reply only at the position in its own record, never at one a cell names.

7.3A request and its reply, step by step

1. Send (fipc_send, or fipc_send_acquire and fipc_send_commit for a request written in place). The client's sending thread is the ring set's sender, a queue producer's code: it reserves R plus the request, waiting for room if it must (the only wait on the request's path), writes the request at the reservation's start plus R, and records the reservation's start in its own position record. Then the cell's fields (with reply_len and outcome 0), published(i) with release, tail = i + 1 sequentially consistent, the ready bit and a wake, and last the look at ended.
2. Take (a server's fipc_recv or fipc_recv_acquire). The server's receiving thread is the ring set's taker: the pass over the clients' rings, the look at len before a claim into a short buffer, the intent, the compare-and-swap to claimed(i, s) and the move of head are a queue consumer's (§6.3), so servers take each client's requests in the order it sent them. A claimed request whose position or length doesn't describe a request of the ring is a corrupt peer's: it is answered as abandoned, and the receive returns INVALID. Otherwise the server holds it, one at a time: a receive while it holds one returns INVALID and changes nothing, and fipc_recv_release does nothing on a server. The held request is published with release and read with acquire, so a server may receive on one thread and reply on another.
3. Reply (a server's fipc_send, or fipc_send_acquire and fipc_send_commit, which hand out the reply room itself). The reply, 1 to max_message bytes, goes into the room at payload_position − R; then answer stores reply_len and outcome = replied, then done(i, s) sequentially consistent, then swaps the ring's reply_sleeps to 0 and wakes the client if it read 1; last the server loads ended. After the done it touches nothing of the request. A reply never waits, and a reply to a client that has left returns OK: the server's connection is to the service, not to that client. A request held in place stays valid until the reply, since its room is the client's until the client consumed the answer.
4. Receive (a client's fipc_recv or fipc_recv_acquire). The client's receiving thread keeps next, its count of the answers it consumed, and waits for the done of request next, whichever server stored it. Replied, with a length of 1 to max_message: the reply is at the start of the request's reservation, which its own position record gives. Abandoned: the receive returns FIPC_ABANDONED, with a length of 0. Anything else is a corrupt server's answer: INVALID. Each consumes the answer by storing next + 1 with release; a reply held in place is consumed at its release or the next receive. A receive that times out, or a buffer too short (TOO_LARGE), consumes nothing: the next receive returns that answer.

Reclaim. The sending thread frees a request's room, its reply's included, only below the smaller of head and next, which it loads with acquire: so a request's room is reused only after the receiving thread has read the reply out of it. A client that sends without receiving waits for room its own replies hold: with one thread, that send times out. Several requests may be in flight at once, as many as the ring has room for, and the servers work on them in parallel.

7.4Waits and wake-ups

• Ready bits, parks and one wake at a time are the work queue's (§6.4), with the servers in the consumers' place: a client that publishes onto a ring that looked empty sets its bit and wakes one idle server, and a server parks on its slot's sleeps only after it set its idle bit and saw no ready bit.
• The reply's wake is exact, since it is a synchronous call's hot path. A server stores the done (sequentially consistent), then swaps the client's reply_sleeps to 0 and wakes it if it read 1. The client, with no answer yet, swaps the flag to 1, then loads its request's state, its requests, its phase and ended (all sequentially consistent), and sleeps a slice at most only if none changed. Either the client sees the done, or the server's swap sees its 1. Several servers may swap the flag, since they answer a client's requests in any order; a wake for a later request only makes the client look and sleep again. The owner's abandon wakes the client the same way. Each reply costs the server two locked instructions on x86-64, the done and the swap.
• The room's wake is inexact, as a queue producer's. A client waiting for room swaps room_sleeps to 1, reclaims again and sleeps in slices from 1 ms doubling to 10 ms. Only its own receiving thread frees room, so only it wakes the sending thread, after it consumed an answer, with a relaxed look at the flag; the slice bounds a missed wake. Room waits are rare, and the exact pair would cost a locked instruction per reply.
• Own interrupts (a cancel, the end) are stored first, then the member swaps its own flags to 0 (a client's two, a server's one) and wakes each that read 1.

7.5Joining, slots and draining

The owner admits joiners on its poller exactly as a queue's owner does (§6.5): no step blocks, at most 8 joiners are in setup, each is dropped if its JOIN hasn't come 2 s after its accept, and the role arrives in JOIN, which claims a free slot of that role round robin, with a new generation and its words reset (a server slot's sleeps, claim and idle bit; a client slot's two flags). OFFER carries the capacity, max_message, both maximums and the segment. A new client starts at its ring's tail, which must equal head, and its count of answers starts there too; a server starts at once.

A client's slot is reused only once its ring has settled (ringSettled, the queue's): every request its last client sent has been claimed or dropped (head = tail), and no server still holds one of them. Until then the slot is draining, and the poller rechecks it before it listens, every 5 ms while no slot is free. A server's slot is free as soon as its server's end has been handled.

7.6Ends: abandon, drop, and the service's end

• A server's end. As in a queue, the end of a member's connection means that member writes the segment no more. For a server, the owner reads its claim; if that cell still holds claimed(i, s) (heldClaim), it abandons the request (abandonHeld): reply_len 0 and outcome = abandoned, then the compare-and-swap from claimed(i, s) to done(i, s), sequentially consistent, then the client's wake. The client's receive of that answer returns FIPC_ABANDONED, and its next receive goes on to its next request. A server that stored its done just before it ended keeps its reply, since the compare-and-swap then fails; a reply half written is never shown, since the done comes after it. Whatever the server held, the owner then wakes the client its intent names (a server that ended between its done and its wake left it asleep), clears the server's idle bit and frees its slot. There is no return ring, and nothing is sent again: a request may or may not have been carried out by a server that ended, and only the client can tell whether repeating it is safe.
• A server's close abandons the request it still holds itself, as the owner would, unless the service has ended.
• A client's end. Nobody waits for its answers any more, so the owner drops its requests no server took (dropUnclaimed): each from published to done, by the servers' own compare-and-swaps, so it races their claims safely. The requests servers already hold are answered as usual, into the ring, and their replies return OK. The slot drains until they are (§7.5).
• Retired slots. A member whose connection fails otherwise, or which sends a frame, may still be running: its slot is retired, as in a queue. A server retired for a frame stays watched, and its request is abandoned at its connection's end, since its client receives no later answer before that one; the owner can't abandon it at the retirement, since the server may still be writing its reply. Nothing of a retired client is dropped.
• The service's end. Closing the service (fipc_service_close, server.close) stops the poller, stores ended, closes the joiners' connections and the listening handle (freeing the name at once), and then every member's connection; if the owner's process dies instead, each member's watcher stores ended itself. From then on clients' sends and servers' receives return DISCONNECTED: servers don't drain (their taker's drains is off), since no owner would be left to abandon a crashed server's request. A client receives the answers done before the end, in order, then DISCONNECTED at its first request without one, and that DISCONNECTED is final. A reply that returned OK is received by a client that receives: the server stores its done and then loads ended, the client loads ended and then the state, all sequentially consistent, so at least one sees the other's store. A reply that returned DISCONNECTED just as the service ended may still have reached its client.

The tests cover this with a stress test of clients and servers on threads whose requests carry their client, their sequence number and a checksum (every client must get each reply once, in its requests' order); litmus tests of the reply's wake, the end's pair and the abandon; client, server and owner processes killed or crashed at each step of a join, inside a reply before and after its done, and holding a request (in the slow tier); the chaos harness's service-* scenarios, which kill and pause owners, clients and servers and end every cycle by accounting for every request (each answered once, by its reply or, only for a killed server's held request, as abandoned); and the fuzz harness's forged owners, forged joiners, and corrupt clients and servers.