For years, needing to self-host an S3 storage service had a default answer: minio. Unfortunately, the company decided to abandon the open source project and archive the git repository, shifting it to fully closed source. Now administrators need a new S3 storage system, and luckily there are many good open source options to choose from.
Biased decision summary
If you are in a hurry, here is a short summary of when to pick which:
Serve a directory over simplified S3 (e.g. for backups): rclone.
Single node to medium (30-50nodes) clusters: Garage for fully automated simple storage, RustFS if you need the full S3 API spec.
Medium - Large k8s clusters: CubeFS.
Medium - Large bare-metal clusters: CEPH.
Swift typically loses to CEPH in modern deployments because of features and ecosystem tooling, SeaweedFS has significant manual maintenance overhead and has some potential issues making it a riskier choice than garage/RustFS.
Of course this selection guide is biased, and your priorities may differ. The following paragraphs discuss each option in more detail so you can make your own conclusion.
rclone
"I want a directory to speak basic S3 for a few minutes"
This is a bit of an unusual contender, but if all you need is a plain backup target for restic or plain s3 up-/download without clustering, rclone s3 serving might be what you are looking for. It is ultra-lightweight with decent performance and lets you serve a single directory over S3 protocol on the fly. Simply run:
rclone serve s3 --auth-key myuser,mypass --addr 0.0.0.0:8080 /path/to/dirNow /path/to/dir is served on port 8080 secured by access key myuser and secret key mypass. Every subdirectory inside /path/to/dir is available as a bucket through the S3 endpoint, although without enforceable quotas. Note that created objects must be valid POSIX filenames since no conversion is done, check the [rclone documentation](https://rclone.org/commands/rclone_serve_s3/) for details.
Especially as a translation unit between a storage / NAS server and (backup) tooling that expect S3 endpoints (restic, Barman, PgBackRest, velero, longhorn, ...), spinning up a temporary endpoint this way can be genuinely all you need and be reasonable for a backup or syncing workflow. That said, there is no way to add scaling / clustering, serverside encryption or even bucket quotas, so it does not replace a real storage service to back production apps or workloads.
Garage
Minimal S3 API subset, tiny hardware requirements, fully automated, simple to run.
Single node - Medium clusters (30-50 nodes)
If you are looking for the least effort selfhosted S3 option, this is probably it. Garage focuses on simplicity, both in terms of architecture and administrative overhead. It only implements the most common subset of the S3 API spec that 99% of applications actually use, leaving out less popular ones (most notably bucket tagging / replication, object locking / versioning / retention, IAM/ACL/policies and legacy APIs). Check their docs for a full list.
It replaces IAM and access control APIs with simple access key permissions per bucket, and offers replication settings for all data implicitly instead of per-bucket.
This simplicity allows it to run nearly fully automated in most environments, requiring virtually no operator oversight or maintenance. Necessary rebalancing (after adding nodes to the cluster) is done automatically. There is no erasure coding support, so replication typically triples necessary storage (1GiB objects = 3GiB space across disks), but in turn makes recovery so simple it requires no manual steps beyond removing/replacing the dead node at all.
It is designed to cause minimal friction, dealing well with dynamically growing clusters using unevenly sized nodes and disks over time.
Think of it as an object database on cruise control that happens to be S3 compatible rather than a traditional S3 service.
RustFS
Complete S3 API, low hardware requirements, mostly automated, builtin web UI.
Single node - Medium clusters (30-50 nodes)
If you are looking for a direct MinIO replacement, RustFS has you covered. It has all the features you are used to, including erasure coding, clustering support and a builtin web UI, plus tiered storage, server-side encryption and support for KMS and SSO.
Erasure coding means only using only ~1.3x physical disk storage overhead (average, varies by durability settings), but also consumes more hardware resources during recovery. It is mostly automated, but requires some manual operations, for example rebalancing when adding or recovering nodes, and prefers mostly evenly sized nodes and disks for cluster use.
CubeFS
Shared storage service for S3, HDFS and POSIX access, significant hardware overhead, mostly automated when run in kubernetes, builtin web UI
Medium - Large (kubernetes) clusters
Built from the ground up as a cloud-native storage service, CubeFS works best when running inside kubernetes, giving it access to strong automation and failover tools to remove most manual operator tasks from daily operations.
It offers S3 storage as well as direct filesystem or HDFS access and a builtin web UI. It composed of multiple services and separates metadata from storage volumes, allowing it to deal well with many (>= billions) of tiny files at scale, but requiring more careful planning before deployment.
While it can run standalone, it works best inside kubernetes, adding to its self-healing and high-availability strengths without adding more manual administrative tasks, but keep in mind that its scale-out architecture makes it much heavier on cpu/memory ressources than previous entries.
Swift
Old-school complete S3 service, large hardware overhead, complex architecture and maintenance, lots of manual maintenance, often skipped in favor of CEPH
Large - Very large clusters, mostly within OpenStack environments
The old-school object storage service, although losing more and more relevance over time. It is by no means bad, but simply cannot keep up with the ease of use or flexibility of modern alternatives: Other services either offer more than simple S3 APIs or are much easier to deploy and maintain - often both.
Swift was built for the openstack ecosystem, designed as the object storage backbone to cloud clusters starting at a petabyte-scale. It still performs exceptionally well as an S3-only service today, but remains much more complex and manually involved than previous entries, without added benefit. It requires evenly sized nodes and disks, recovery can put significant strain on hardware resources and performance is often bound to memory availability. It is extremely mature and well-understood, but has difficulty competing in modern environments.
Ceph
Enterprise class block- object- and filesystem storage, huge hardware requirements, very complex architecture and manual maintenance overhead, scales infinitely
Large - Extremely large clusters (especially when using block storage for k8s/vms)
The undisputed king of storage. While it is easily the most complex in terms of architecture, deployment planning and hardware requirements, it is also the single most well understood storage backbone today, powering anything from private VM clusters to public cloud vendor object storage offerings.
The steep learning curve and complex maintenance enable a highly flexible and performant service that can grow without limits and offer multiple types of storage, from virtual disks to objects.
Newer versions have become somewhat easier to manage, and projects like rook.io aim to automate it further using kubernetes capabilities, but it remains the most complex contender in this list by a large margin. If size is what you need, CEPH is the best bet, especially if you need more than object storage alone.
Seaweedfs
Support for S3, HDFS and FUSE mounting, medium hardware requirements, but much more restricted than alterantives and large manual maintenance overhead
Medium - Large clusters, but hard sell against garage/RustFS or CubeFS
Built for small to large clusters with manageable resource overhead, designed to circumvent the many small files problem at scale, exporting storage through many protocols like S3, HDFS or POSIX filers, able to run in the cloud and on bare metal. It sounds great on paper, but comes with hefty downsides.
First and foremost, it puts a lot of manual maintenance burden on operators. Tasks like vacuuming, rebalancing and recovery are all entirely manual and require human execution and oversight. Automation and data safety features exist - but only for the closed source enterprise version. To top it off, it is fully owned and steered by a single company - just like MinIO was, until it was suddenly taken closed source without warning.
It is difficult to recommend SeaweedFS despite its very nice sounding features and seemingly mature implementation, because it is the only S3 storage in this list that has these problems; no other one has even one, let alone all three.
The risk this introduces is almost never worth the features it provides over other, more free and open alternatives with better automation.