S3 — a share is a bucket
serve "s3" { addr = "0.0.0.0:9000" }Or over HTTPS, since v0.9.0, with tls = true and a
tls block:
serve "s3" {
addr = "0.0.0.0:9000"
tls = true
}The fifth protocol, and the one that makes an image reachable from anything
that speaks object storage — a backup tool, a data pipeline, rclone, a
browser’s fetch(). Nothing mounts.
GET / the shares this person may use, as buckets
GET /photos?list-type=2 the files in that share, as keys
GET /photos/holiday.jpg the file itself, Range and allA share is a bucket
Served over the same per-user tree SFTP uses: a filesystem whose top level is the shares this person may use. So a share alice may not use is not a bucket alice can see, and the access rules are applied in one place rather than copied into a second protocol.
A directory inside a share is a prefix, not an object
Which is what S3 clients already believe — they display prefixes ending in
the delimiter as folders. GET of a directory answers NoSuchKey, because
answering 200 would make a client think it can fetch the bytes of a folder.
An access key is a user; the secret key is their password
export AWS_ACCESS_KEY_ID=alice
export AWS_SECRET_ACCESS_KEY=$(cat alice.pw)
aws --endpoint-url http://attic.example.org:9000 s3 lsSigV4 proves possession of the secret by computing an HMAC from it — the password never crosses the wire, exactly as with NTLMv2.
So the directory must HOLD the password, not merely check it
A source that can only verify a password — an LDAP bind, a bcrypt column — can answer WebDAV and cannot answer S3.
And not an NT hash either: that is the MD4 SMB can work from, and an HMAC
cannot be computed from it. An identity holding only an NT hash serves
SMB and not S3 — near neighbours rather than the same answer.
check prints the column.
⛔ There is no Password() accessor anywhere in the program: the password is
reached only through Identity.Derive, which takes a derivation and returns
its result. For S3 the derivation is the identity, because SigV4’s key
schedule lives in go-filesystems/s3; the secret goes to that signature check
for that one user and nowhere else.
What is served
ListBuckets, HeadBucket, ListObjectsV2 with prefix, delimiter, max-keys
and continuation, HeadObject, and GetObject with Range. SigV4 in both
forms, header and presigned.
The object API itself is
go-filesystems/s3.
What is refused, and why
Writes answer 403
PUT, POST and DELETE are refused with AccessDenied (“this export is
read-only”). fileshare keeps the S3 server read-only because a share a person
may only read has to refuse at the same place SFTP does, and that is not
wired yet; go-filesystems/s3 v0.3.0, which this release is built on, refuses
every write anyway. Refusing beats a half-written object.
A bearer token is not accepted here
OIDC works over WebDAV and nowhere else. S3 authenticates with a SigV4 signature — an HMAC over the request, with no field a bearer token fits in.
The usual route is STS AssumeRoleWithWebIdentity, which exchanges an OIDC
token for temporary credentials the client then signs with. That is a
different mechanism, and it is not implemented.
Multipart upload, versioning, ACLs, policies, lifecycle rules, replication and
tagging are not implemented. Setting any of them, and starting a multipart
upload, is a write, and answers AccessDenied like any other. A GET that
asks for one of them (?versioning, ?acl, ?tagging…) is not told apart:
the query is ignored and the bucket’s listing or the object is returned.
NotImplemented answers a listing other than list-type=2, any method but
GET on /, and a method no operation here uses.
Two limits worth knowing before pointing it at something large
GetObject reads the whole object into memory. The driver interface offers
ReadFile and nothing else — no opener, no reader — so there is no streaming
read to use. A bound, 256 MiB, answers EntityTooLarge rather than taking
the process down.
LastModified is the epoch. filesystem.Stat carries Mode, Size and
Inode, and no modification time at all. The field is not optional in the S3
schema, so something must be sent; the epoch is obviously not a real time,
where time.Now() would look like a file that just changed and make every sync
tool copy everything, every run.
Created on by tannevaled
Modified on by tannevaled