In an earlier post we looked at Varlink as a small experiment. At the time it was mostly a “hello world” exercise: define an interface, open a Unix socket, call a method, return some JSON. That was useful, but it was still a lab setup.
Since then we have started to move the idea into the node layer of DownToZero. Two projects are especially interesting here: dtz-node-stats and dtz-node-volume. The first one exposes information about a worker node, including power state and hardware basics. The second one controls a FUSE filesystem that mounts a DTZ volume into the local machine.
This post is not a source-code walkthrough. Some of the surrounding repository code is internal infrastructure and not something we want to publish as an implementation guide. But the shape of the system, the interfaces, and the pattern are worth sharing, because Varlink and Zlink are a very good fit for this class of problem.
The short version is this: we want small local services on a node to be discoverable, scriptable, and boring to operate. Varlink gives us the protocol. Zlink gives us a Rust implementation that fits our async services. FUSE gives us a way to present remote or object-backed storage as a normal filesystem. Put together, this lets higher-level orchestration code say “mount this volume for this context” without knowing how the filesystem driver talks to storage.
DownToZero nodes are not just anonymous Linux boxes. They are part of an energy-aware platform. We care about where power comes from, how much a node consumes, whether a machine is running on battery or solar, and what workloads should be placed there.
At the same time, node-level features should not all end up inside one giant daemon. Metrics collection, power handling, volume mounting, container scheduling, update logic, and local maintenance tasks all have different lifecycles. Some of them should restart independently. Some of them need special permissions. Some should remain small enough that they can be understood and tested in isolation.
The problem is coordination. Once these pieces are separate processes, they need a clean way to talk to each other. A REST server on localhost would work, but it brings HTTP routing, ports, TLS questions, and usually more surface area than we need. A custom Unix socket protocol would also work, but then every service invents its own framing, error model, and debug story. D-Bus is established, but we wanted something simpler for service-specific machine APIs.
Varlink sits nicely in the middle. It uses sockets, has a simple JSON-based protocol, supports introspection, and is easy to call from command-line tools. For Rust services, Zlink gives us typed method handling while keeping the wire format easy to inspect.
The result is a local API surface that feels more like:
varlinkctl call /tmp/dtz.varlink rocks.dtz.Node.GetInformation {}
and less like designing a new control plane from scratch.
dtz-node-stats is the simpler of the two projects. Its job is to collect and expose node-level facts: CPU cores, memory, uptime-related metrics, power source, and power consumption. It can integrate with a power meter, for example a Tasmota device, or fall back to configured consumption estimates.
The interesting part for this post is the Varlink interface. The service listens on a Unix socket and exposes a few focused methods:
rocks.dtz.Node.GetInformation() -> node information
rocks.dtz.Node.GetPowerState() -> current energy state
rocks.dtz.Node.Shutdown() -> graceful shutdown
org.varlink.service.GetInfo() -> standard Varlink introspection
Calling GetInformation returns stable node facts such as the node ID, the configured power source, CPU cores, and memory size. Calling GetPowerState returns the current energy state and cumulative consumption in watt-hours.
This is already used by two worker roles on our nodes.
The sync-worker calls GetInformation during startup and uses the returned node ID as its local identity. It also uses the reported CPU and memory values when it announces itself to the rest of the platform. That means the worker does not need a second configuration file for basic capacity information, and it does not need to invent its own idea of what machine it is running on. The node service is the local source of truth.
The async-worker uses the same pattern, but for a slightly different purpose. It reads the node information once on startup, keeps it with the worker’s runtime state, and then uses the node ID when handling requests, reporting health, and tagging metrics. If you later look at latency or request counters, the node label comes from the same Varlink-backed identity that the rest of the local services see.
That is the point of keeping this interface small. dtz-node-stats does not need to know what kind of work each process performs. It only has to answer a few node-level questions consistently. The workers can then use those answers for registration, routing hints, metrics, and operational visibility without copying host-detection logic into every service.
The important design choice is that this is not a metrics endpoint pretending to be a control API. Prometheus can still receive metrics. Logs can still go to journald. But if another local process needs to ask the node service a question, it gets a small typed method instead of scraping text output or importing implementation code.
The Shutdown method is also a good example of why local IPC is useful. During updates, a new process may find an existing socket. Instead of blindly failing or deleting the socket file, the startup path can check whether a service is active and ask it to shut down cleanly. If the socket is stale, it can remove it. If the service is alive, it can request a graceful stop. This sounds small, but it removes a lot of operational awkwardness from edge machines.
dtz-node-volume is where the pattern becomes more interesting. The service mounts a DTZ-backed volume as a local filesystem. From the perspective of software running on the node, the result is just a directory under ./dtz/.... Files can be created, read, written, listed, and removed using normal POSIX operations.
Under the hood, the filesystem is implemented with FUSE. FUSE is a powerful boundary: the kernel forwards filesystem requests to a userspace process, and that process decides how to satisfy them. That means the storage backend can be an object store, an API, a cache, a remote service, or a combination of those. Applications do not need to know. They just call open, read, write, readdir, getattr, and so on.
The missing piece is control. A FUSE filesystem is useful after it is mounted, but something still needs to decide when to mount it, for which context, at which path, and when to unmount it again. We do not want every caller to know the FUSE setup details. We also do not want to expose the storage implementation directly to every part of the node.
So dtz-node-volume exposes a Varlink interface:
rocks.dtz.Volume.GetInfo() -> active mounts
rocks.dtz.Volume.Mount(context, volume) -> mountpoint and status
rocks.dtz.Volume.Umount(context, volume) -> acknowledgement
org.varlink.service.GetInfo() -> standard Varlink introspection
The interface is intentionally small. Mounting a volume is a local control action. The caller provides the context and the volume ID. The service turns that into a mountpoint like:
/dtz/{context_id}/{volume_id}
The response is equally small:
{
"mountpoint": "/dtz/context-.../volume-...",
"status": "mounted"
}
From there, the caller does not talk Varlink for every file read. That would be the wrong abstraction. Varlink controls the lifecycle of the mount. FUSE handles the filesystem operations. This separation is the main point.
It is tempting to build everything as an RPC API. For volumes, that would mean methods like ReadFile, WriteFile, ListDirectory, DeletePath, and dozens more. That approach works, but then every consumer needs a custom client. Existing tools like cp, rsync, editors, shells, and build systems cannot use it directly.
FUSE lets us avoid that. The data plane remains the filesystem interface that Linux programs already understand. The control plane becomes a tiny Varlink service.
For the volume lifecycle, the sync-worker owns local workload preparation. When a workload declares a volume, the sync-worker asks the volume service whether it is already mounted and calls Mount if needed. It then bind-mounts the resulting host path into the workload. After that, the workload does normal file IO; it does not know about Varlink or the object store.
A sequence view is a better fit than a topology diagram here, because the interesting part is the order of operations:
sequenceDiagram
participant W as sync-worker
participant V as dtz-node-volume
participant F as FUSE session
participant C as container workload
participant K as Linux VFS
participant S as DTZ storage backend
W->>V: GetInfo(context, volume)
alt volume is not mounted
W->>V: Mount(context, volume)
V->>F: create FUSE mount at /dtz/context/volume
F-->>V: mount ready
V-->>W: mountpoint + status
else volume is already mounted
V-->>W: existing mountpoint + status
end
W->>C: start workload with bind mount
C->>K: normal file IO
K->>F: FUSE requests
F->>S: object operations
S-->>F: object data and metadata
F-->>K: filesystem reply
K-->>C: read/write result
W->>V: Umount when lifecycle policy says so
This keeps the API small and the user experience normal. The sync-worker can mount the volume just before a workload starts. After that, the workload uses a directory. When the workload is finished, the same worker role can unmount it or leave it available for reuse, depending on the lifecycle policy.
For us, that is exactly the kind of interface we want on a node. It is explicit enough to automate, but it does not force every workload to learn a DTZ-specific storage API.
There are many ways to map a filesystem onto an object store. We are using a layout that separates namespace metadata from file data.
In simplified terms:
fs/ filesystem namespace and metadata
data/ larger file chunks
Each file or directory has a metadata object under the fs/ prefix. That metadata contains the POSIX-ish attributes we need: mode, owner IDs, size, modification time, directory flag, and optional extended attributes. Small file contents can be inlined directly into the metadata object. Larger files use a data reference, and their bytes are stored as chunks under data/.
This has a few useful properties.
Directory listings only need to inspect the namespace. They do not need to filter through file chunks. Renames can be cheap because the metadata object can move while the large data chunks stay where they are. Sparse files are possible because unwritten chunks do not need to exist. Small files avoid extra round trips because their content lives in metadata.
Again, the exact implementation details are not the interesting part to publish. The important architectural point is that object storage and POSIX filesystems have different strengths. The FUSE layer translates between them. Varlink does not try to be part of that translation; it only starts, stops, and reports on the mount.
One detail we added to the volume design is extended attribute support. This can look like a niche filesystem feature, but it matters once you want the mounted directory to behave like a real filesystem instead of a toy demo.
Extended attributes are used for user metadata, capabilities, security labels, desktop metadata, and various application-specific hints. If we ignore them completely, many things still work, but the abstraction leaks. If we preserve them in metadata, the FUSE mount becomes much more compatible with normal Linux expectations.
The practical design is straightforward: metadata can contain an optional map of xattr names to base64-encoded values. FUSE operations like getxattr, setxattr, listxattr, and removexattr become read-modify-write operations on the metadata object.
There are tradeoffs. Updating xattrs rewrites metadata. Concurrent metadata updates need careful thought. Very large xattrs would be a bad idea. But the pattern is still a good match for the kind of metadata we expect around mounted volumes.
The GetInfo method on dtz-node-volume returns the currently active mounts. This includes the context, volume ID, mountpoint, status, and the last observed access timestamp.
That last field is small but useful. A node-level orchestrator can ask:
varlinkctl call /tmp/dtz-volume.varlink rocks.dtz.Volume.GetInfo {}
and get a current view of what the volume service believes is mounted. This is useful for debugging, cleanup, and later for more energy-aware behavior. If a mount has not been accessed for a while, maybe it can be unmounted. If a workload starts, maybe the mount should be created just in time. The Varlink interface gives us a place to expose that state without coupling callers to the FUSE implementation.
We are building most of this node-side code in Rust, so the Rust ergonomics matter. Zlink lets us keep the service implementation typed while still speaking the Varlink protocol. Method calls deserialize into Rust enums and structs. Replies serialize back into the expected JSON shape. Standard Varlink introspection is just another method exposed by the service.
That gives us a pleasant balance. The interface remains inspectable from the outside:
varlinkctl call /tmp/dtz-volume.varlink org.varlink.service.GetInfo {}
but the service code does not become a pile of stringly typed request handling.
It also makes local tooling easy. During development, a shell command can mount a volume, list mounts, or ask the node for its power state. During operations, systemd can manage the service process and the Unix socket path can remain local to the node. We do not need a public network listener for these actions.
One of the reasons we chose Varlink is not only that the protocol itself is standardized. The tooling around it is standardized as well. In practice, this matters just as much as the interface definition.
With varlinkctl, every service immediately becomes usable from a shell:
varlinkctl call /tmp/dtz.varlink rocks.dtz.Node.GetInformation {}
varlinkctl call /tmp/dtz-volume.varlink rocks.dtz.Volume.GetInfo {}
That means maintenance work does not require a special debug binary or a small one-off client. If a node behaves strangely, we can SSH into it and ask the local services what they know. If an update script needs to check whether a volume is mounted, it can call the Varlink service directly. If a systemd unit or cron-style maintenance job needs a graceful shutdown, it can use the same public method as the Rust clients.
This is a big difference from an internal library API. A library is only convenient for programs written in the same language and released on the same cadence. A Varlink socket with varlinkctl is convenient for Rust code, shell scripts, maintenance playbooks, and manual debugging. For node infrastructure, that flexibility is valuable.
The immediate use case is practical: mount DTZ volumes on worker nodes and expose node state to other local services. But the pattern is bigger than these two projects.
A local service can expose a tiny Varlink control API while doing something more complex internally. That “something” might be a FUSE filesystem, a metrics collector, a hardware controller, a scheduler helper, or an updater. The caller gets a stable method interface. The service keeps its implementation private.
For DownToZero this is especially useful because we are trying to make infrastructure smaller and more energy-aware. A node can expose its capabilities locally. Higher-level services can make decisions based on power state, available resources, and mounted storage. Workloads can still interact with normal files and directories.
This also keeps responsibilities clear:
dtz-node-stats reports what the node is and how it is powered.dtz-node-volume controls when DTZ volumes appear as local filesystems.This is still evolving. The interfaces are intentionally small because it is easier to grow a local API than to shrink one after multiple services depend on it. The volume service is also the place where we expect the most learning. Filesystems have many edge cases, and object stores do not magically become POSIX filesystems just because a FUSE layer exists.
But the direction feels right. We can expose useful behavior without publishing internal service logic. We can debug with standard tools. We can let normal Linux programs use mounted storage. And we can keep node-level coordination local instead of turning every internal action into a platform API.
The most important lesson so far is that Varlink and FUSE complement each other well. Varlink is not trying to move file data. FUSE is not trying to be a service discovery protocol. Together, they create a clean shape: call a small local method to create the filesystem, then use the filesystem like any other directory.
That is a pattern we will keep using in the DownToZero node stack.