Volumes: storage fileshare creates

Since v0.21.0 the admin API can create the storage a share serves — a ZFS dataset, a btrfs subvolume, an XFS or ext4 directory under a project quota — sized, owned and tracked, and serve a share from it. Linux only. Ceph is not supported yet.

Creating that storage needs CAP_SYS_ADMIN (quotactl, the btrfs quota ioctls, /dev/zfs, mount(2)). fileshare serve parses five network protocols for strangers and holds no privilege, so the part that does is a second role of the same binary, fileshare provisioner, behind a unix socket and a small, closed protocol:

                         admin gRPC (unix socket 0600 + peer uid, or mTLS)
  operator ─────────────────────────────────────────────▶  fileshare serve
                                                             (unprivileged)
                                                                  │
                     provision gRPC (unix socket, SO_PEERCRED = fileshare's uid)
                                                                  ▼
                                                    fileshare provisioner
                                                    (CAP_SYS_ADMIN, CAP_CHOWN, CAP_FOWNER)
                                                                  │
                                    go-fsctl: zfs · btrfs · projquota (ioctls, no CLI)

One binary rather than two daemons: one version to deploy, one protocol definition. The storage itself is done by go-fsctl (pure Go, no zfs or btrfs command). The design and what changed while building it are in the fileshare repository’s docs/volumes.md.

Both processes, configured

fileshare serve, as the user fileshare (uid 990):

# /etc/fileshare/fileshare.hcl
admin {
  listen          = "unix:///run/fileshare/admin.sock"
  state_file      = "/var/lib/fileshare/shares.json"
  source_roots    = ["/srv/fileshare/volumes"]
  provisioner     = "unix:///run/fileshare-provisioner/provisioner.sock"
  provisioner_uid = 0          # who must answer on that socket; 0 is the default
  allowed_uids    = [990]      # who may call this API
}

A volume’s path must lie under source_roots, like any other source.

fileshare provisioner, as root:

# /etc/fileshare-provisioner.hcl
provisioner {
  listen     = "unix:///run/fileshare-provisioner/provisioner.sock"
  client_uid = 990
  group      = "fileshare"
  max_volume = "10T"
  state_file = "/var/lib/fileshare-provisioner/volumes.json"

  parent "tank" {
    zfs  = "tank/fileshare"
    root = "/srv/fileshare/volumes/tank"
  }
  parent "fast" { btrfs = "/srv/fileshare/volumes/fast" }
  parent "plain" {
    xfs         = "/srv/fileshare/volumes/plain"
    project_ids = "100000-199999"
  }
}

The provisioner’s file holds nothing but its provisioner block: fileshare’s own file handed to it by mistake is refused rather than half-read.

A volume is always <parent root>/<name>: dataset tank/fileshare/<name> mounted at /srv/fileshare/volumes/tank/<name> (with refquota), a btrfs subvolume limited by its qgroup, an XFS or ext4 directory with a project id from the range and a hard block limit. XFS and ext4 need the prjquota mount option (ext4 also the project,quota features).

The systemd pair

The provisioner:

# fileshare-provisioner.service
[Service]
ExecStart=/usr/local/bin/fileshare provisioner -c /etc/fileshare-provisioner.hcl
RuntimeDirectory=fileshare-provisioner
RuntimeDirectoryMode=0755
StateDirectory=fileshare-provisioner
StateDirectoryMode=0700
CapabilityBoundingSet=CAP_SYS_ADMIN CAP_CHOWN CAP_FOWNER
# Run as root, or as a dedicated user with:
# AmbientCapabilities=CAP_SYS_ADMIN CAP_CHOWN CAP_FOWNER
NoNewPrivileges=yes
SystemCallFilter=@system-service @mount quotactl quotactl_fd
SystemCallArchitectures=native
LockPersonality=yes
RestrictRealtime=yes
# NOT ProtectSystem= nor ReadWritePaths=: they give the service its own
# mount namespace, and the ZFS volumes it mounts would be seen by nobody
# else. And no Landlock: a Landlock-restricted thread cannot mount(2).

The server, with no capability at all:

# fileshare.service
[Unit]
Requires=fileshare-provisioner.service
After=fileshare-provisioner.service

[Service]
User=fileshare
Group=fileshare
ExecStart=/usr/local/bin/fileshare serve -c /etc/fileshare/fileshare.hcl
RuntimeDirectory=fileshare
StateDirectory=fileshare
# No capability at all: CAP_SYS_RESOURCE would let it past an ext4 quota,
# and fileshare refuses to serve volumes with it.
CapabilityBoundingSet=
AmbientCapabilities=
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/srv/fileshare/volumes /var/lib/fileshare

ProtectSystem= is fine for the server: it is the provisioner that mounts, and mounts propagate into the server’s namespace, not out of it, so a ZFS volume mounted after the server started is visible to it.

The admin API’s volume calls

fileshare serve relays these to the provisioner (admin.proto):

DeleteShare never deletes a volume. “The shares that use it” are the shares made from it and any share — of the configuration files too, served, disabled or unavailable — whose image or directory lies inside it.

A name matches ^[a-z0-9][a-z0-9_-]{0,62}$, and a request never carries a path: it names a parent from the provisioner’s own configuration.

With reflection = true in the admin block, grpcurl can make the calls. The socket is 0600 and allowed_uids above names only fileshare’s uid, so run it as that user:

S=/run/fileshare/admin.sock
A=fileshare.admin.v1.AdminService

sudo -u fileshare grpcurl -plaintext -unix $S $A/ListParents

# a 32 GiB volume on the XFS parent
sudo -u fileshare grpcurl -plaintext -unix $S \
  -d '{"parent": "plain", "name": "projects", "quota_bytes": "34359738368"}' \
  $A/CreateVolume

# a share served from it
sudo -u fileshare grpcurl -plaintext -unix $S \
  -d '{"name": "projects",
       "volume": {"parent": "plain", "name": "projects"},
       "grants": [{"subject": {"group": "staff"}, "access": "ACCESS_WRITE"}]}' \
  $A/CreateShare

# grow it to 64 GiB
sudo -u fileshare grpcurl -plaintext -unix $S \
  -d '{"parent": "plain", "name": "projects", "quota_bytes": "68719476736"}' \
  $A/ResizeVolume

Status codes

The provisioner’s codes come back as it gave them: INVALID_ARGUMENT (a name outside the grammar, an unknown parent, a quota of 0), OUT_OF_RANGE (above max_volume), RESOURCE_EXHAUSTED (the quota, or a resize’s growth, is more than the parent has free now; or the project-id range is used up), ALREADY_EXISTS, NOT_FOUND, FAILED_PRECONDITION, UNIMPLEMENTED.

Three are fileshare’s own:

• no provisioner configured: FAILED_PRECONDITION;
• the provisioner refuses this server’s uid: FAILED_PRECONDITION too — it is a deployment mistake (its client_uid is wrong), not something the admin caller may not do;
• the provisioner does not answer: UNAVAILABLE.

What is checked before a volume is served

fileshare asks the provisioner where the volume is, and then checks, itself:

• the path is clean, absolute and — resolved — under source_roots;
• it is a directory;
• statfs reports the kind’s filesystem (ZFS_SUPER_MAGIC, BTRFS_SUPER_MAGIC, XFS_SUPER_MAGIC, EXT4_SUPER_MAGIC);
• ZFS: it is a mount point — its device is not its parent’s. An unmounted dataset’s directory would take writes under no quota. (Which dataset is mounted there is not verified: the provisioner’s answer does not say.)
• btrfs: it is a subvolume’s root — a plain directory under the parent has no qgroup;
• XFS and ext4: it carries a non-zero project id that its new files inherit — a directory outside every project is served unbounded.

The state file keeps the volume’s name, not its path, and every start asks and checks again. A volume that is gone, fails a check, or whose provisioner does not answer within 10 s leaves its share defined and not served — said at the start, by fileshare check, and in the share’s unavailable field — while the rest of the server starts. EnableShare tries it again.

A full share

XFS says a full project with ENOSPC; ext4, btrfs and ZFS say EDQUOT. Since v0.21.1 every protocol answers “full”:

In v0.21.0, SMB answered STATUS_ACCESS_DENIED and NFS answered a quota with NFS3ERR_NOSPC.

This applies to every directory share, not only to volumes — see the upgrade note.

The size a client sees

Every protocol asks the server for a share’s size at each query: NFS FSSTAT, SMB FileFsFullSizeInformation and WebDAV’s RFC 4331 quota properties.

• ZFS, XFS and ext4: the size is what statfs says inside the volume, which the kernel answers with the quota (refquota for ZFS, the project quota for XFS and ext4).
• btrfs (since fileshare v0.23.0): btrfs’s statfs ignores qgroups and reports the whole filesystem, so the server uses the provisioner’s numbers instead. The size is the volume’s quota, and the free space is the quota less the qgroup’s referenced bytes, or the filesystem’s own free space when that is less. The server asks the provisioner in the background, at most once every 5 seconds while clients are asking, and answers from the last numbers in the meantime. btrfs updates a qgroup’s count when it commits a transaction (every 30 seconds by default), so the free space a client sees can lag behind writes by that long. Before v0.23.0, a btrfs volume showed the whole filesystem; the quota held all the same.

Used space comes from the provisioner (used_bytes): reading an XFS or ext4 project’s usage needs CAP_SYS_ADMIN, which fileshare does not have.

What the provisioner promises

• One uid, read from the kernel. The peer is identified with SO_PEERCRED when it connects, and anybody but client_uid — root included — is PERMISSION_DENIED before a byte of the request is decoded.
• A closed set of verbs. A method outside its service, or a message that does not decode or carries a field it does not define, gets no answer: the connection is closed.
• It never touches what it did not create. ZFS: the dataset’s own fileshare:volume property — set on the dataset, since user properties are inherited and an inherited mark is not its own. btrfs: the subvolume id and uuid recorded in state_file. XFS/ext4: a project id inside the parent’s range and recorded. Checked before every resize, snapshot and delete.
• Delete refuses data and snapshots unless destroy_data is set, and is logged before it happens.

Measured

In fileshare’s CI, as root on real pools and loop-backed filesystems, with fileshare serve running as nobody and driven through the admin API: writes over WebDAV and over SFTP stop at a 32 MiB volume’s quota on all four kinds.

Not yet

• Ceph. CephFS quotas and RBD are a later phase.