Architecture

Overview

EdgeSync is a thin layer on top of libfossil that adds NATS-based sync, an embedded NATS server, optional iroh QUIC tunneling, and a bidirectional messaging channel. The leaf agent is the only required process; the bridge is optional.

Module layout

EdgeSync is a Go workspace (go.work) with four modules:

• Root (./) — hosts cmd/edgesync/ (the unified CLI binary), sim/ (integration simulator with real NATS + TCP fault proxy), and dst/ shims.
• leaf/ — the leaf agent. leaf/agent/ has the daemon (agent.go), config (config.go), the embedded NATS mesh (nats_mesh.go), HTTP serving (serve_http.go), and NATS-side sync listener (serve_nats.go). leaf/agent/notify/ is the messaging subsystem.
• bridge/ — the NATS-to-HTTP translator. bridge/bridge/ defines Config, New(), Start(), Stop(). Used only when interoperating with an unmodified upstream Fossil server.
• dst/ — deterministic simulation tests. SimNetwork (bridge mode), PeerNetwork (leaf-to-leaf). Shares simio/ abstractions with libfossil.

The CLI in cmd/edgesync/ embeds libfossil’s cli.RepoCmd (38 Fossil-compatible subcommands) plus EdgeSync-specific commands: sync start, sync now, bridge serve, notify {init,send,ask,watch,threads,log,status}, doctor.

Sync wire protocol

EdgeSync reuses Fossil’s /xfer card protocol verbatim — same blobs, same igot/gimme/file/cfile cards, same UV (unversioned) sync. The only thing EdgeSync changes is the transport:

This is why bridge mode works: each side is just xfer cards in a different envelope. The bridge re-frames between an HTTP body and a NATS message without re-parsing the cards themselves.

Embedded NATS server

Every leaf agent runs its own NATS server in-process. The role flag selects topology:

• peer — full server, accepts and initiates connections to other peers
• hub — full server, accepts inbound leaf connections (NATS leafnode protocol)
• leaf — outbound only, connects to a hub

This means a developer can run two leaves on localhost with no broker installed. In production, one VPS typically runs as hub and edge devices run as leaf. See the Browser WASM key findings for the embedded-server constraints (notably DontListen + nats.InProcessServer for in-browser leaves).

Iroh transport

Iroh is opt-in. When --iroh is set, the leaf agent:

1. Loads or generates a node key (--iroh-key)
2. Establishes QUIC streams to peers from --iroh-peer tickets
3. Multiplexes NATS leafnode protocol over those streams

This lets two leaves behind separate NATs talk to each other without a relay server’s intermediation in steady state (relays are only used for hole-punching).

Observability

The leaf agent and sim tests emit traces, metrics, and logs via OpenTelemetry. Telemetry is optional — when OTEL_EXPORTER_OTLP_ENDPOINT is unset, the OTel observer is nil and the no-op observer (zero-cost) is used. Secrets for OTel export are managed via Doppler.

The observer pattern is libfossil’s: implement SyncObserver and CheckoutObserver, hand the implementation to the agent at construction time. EdgeSync’s leaf/telemetry/observer.go provides an OTel-backed implementation.

Determinism

The dst/ module runs many leaf agents under a deterministic event loop with a seeded PRNG and a simio.SimClock. BUGGIFY (probabilistic fault injection) gates failure paths in production code so the simulator can exercise them without changing the production code path. See libfossil’s testing docs for the seed sweep workflow — EdgeSync inherits it directly.