SILO 20261109: lifecycle, Object Lock and batch authorization

SILO Server 20261109 brings the maintenance work since 20261026 into one release: safer multi-pool lifecycle and tier cleanup, two-phase native restore, stronger Object Lock checks, and batch jobs bound to the submitter’s current permissions. It embeds Console 2.7.0, mcli 20261109 and silo-pkg 3.16.0.

GitHub release · Full changelog · Upgrade guide · Downloads

Components

The server package version is 20261109000000.0.0. KES remains the separate, optional 20261026 release. Updating a standalone Console or client does not replace components embedded in an older Server binary. The maintained PGSTY stack is the supported combination; upstream MinIO/MC compatibility remains best effort.

Storage and recovery

• Lifecycle across pools: evaluate expiry under the object lock using authoritative version history and protection from every pool. Refuse incomplete histories and protected whole-key deletions; exact-key deletion no longer reaches child keys. Decommission and rebalance move versions that cannot safely expire.
• Remote tiers: retain remote data while another pool still references it. Exclude free versions from ordinary version history, expire restored copies without deleting the underlying object, and account only for actual removals.
• Native restore: download tier data outside the object write lock, then validate ownership before publishing it. Failed or canceled attempts try marker cleanup for 100 ms before bounded background retries, avoiding a one-minute synchronous lock wait.
• Locking and conditional writes: align single-pool mutations with their erasure-set lock, fix CopyObject lock ordering and starvation, and evaluate multi-pool multipart initiation conditions against the current object. Ordinary bulk deletion groups can run concurrently across sets.
• Reads and healing: return retryable errors for known partial disk views, classify metadata checksum failures for healing, preserve data during heal dry runs, and remove empty directories after failed writes.

Object Lock, IAM and jobs

S3 DeleteObjects now checks physical version removals inside the same lock used for retention and legal-hold updates, including protection held by another pool’s copy. Directory null versions are protected as well. Replicated purges honor the target’s Object Lock; a source GOVERNANCE bypass does not grant a bypass at the target, so such replication can remain failed and retry.

Batch expiration and key rotation enforce the authenticated submitter’s local data permissions during execution. Revoked or changed identities end the job; unavailable IAM or authorization dependencies pause it with a recorded reason and bounded retry. Startup discovery retries storage outages, completed jobs remain completed, and replication jobs can resume their scans. Use mcli batch status to distinguish completion, refusal, failure and pause. External authorization plugins must return a successful HTTP status with an explicit boolean decision (result or result.allow); absent/null decisions, trailing JSON and responses over 1 MiB are rejected, and each call is bounded by one minute. Verify custom plugins before upgrading.

IAM edits operate on persisted records so stale caches cannot restore old credentials or deleted identities. The release also fixes long-DN LDAP login on erasure sets, embedded Console root login with LDAP enabled, and webhook target reloads across peers. Console 2.7.0 adds ten UI languages and fixes usage-worker TLS/budgets, plain-HTTP folder-size and key-copy actions, and malformed uploads.

Upgrade and rollback

1. Stop batch submissions before changing versions. Drain or cancel all running, pending and paused jobs. Unfinished legacy jobs require resubmission after every node upgrades; completed jobs do not. New submitter-bound manifests and reports use formats that 20261026 rejects. Do not downgrade with unfinished new jobs or expect old executors to resume them. Follow the batch checklist.
2. Drain incompatible single-pool layouts. If the sets do not all share the same hosts (normally 18 or more hosts with automatic layout), stop every old process before starting new ones. Old and new lock domains do not safely support rolling upgrades for this layout.
3. Pause restores during mixed versions. New nodes return 503 to POST ?restore while any peer is old or unreachable. Full multi-pool lifecycle and bulk Object Lock protection requires all serving nodes to be upgraded.
4. Verify real S3 access after restarting. ready remains a process/configured-dependency probe. Even cluster can succeed while a node reconnects remote drives. Use bounded signed writes and readback through every node before resuming applications; preserve peer bootstrap and Kubernetes publishNotReadyAddresses. See health checks.
5. Pin the Helm image. Chart 7.0.4 defaults to this release. helm upgrade --reuse-values retains the stored tag, so explicitly set image.tag=RELEASE.2026-11-09T00-00-00Z.

Costs and limits

Safety checks deliberately read more metadata and may refuse work that cannot be proven safe. Multipart initiation can hold its object lock longer; multi-pool expiry reads and reconciles history. Object Lock bucket version purges in a bulk request now use the single-object path sequentially, so large purge batches take longer. Ordinary buckets and marker creation retain the bulk path. In a bounded Linux ARM64 two-pool check, the average per-round median for 1,000 removable locked-bucket versions increased from about 202 ms to 310 ms. This is an intentional safety cost, not a production throughput estimate. Steady HEAD/GET medians changed by about +1.5%/+3.4%; the earlier macOS read slowdown did not reproduce in the Linux steady-state read check. Durable-write timings varied even when the same binary was compared with itself. On the same data directories and key set, three alternating rounds measured about +2.1% for 4 KiB overwrite and −8% for 1 MiB overwrite/new PUT medians; tmpfs and OS-call traces showed no matching request-path work increase. These measurements do not promise unchanged write tail latency on production storage.

Lifecycle-created deletions are not replicated. Known limitations remain in multipart-upload listing (#79), multi-site delete-marker purge (#217), replayed bucket creation (#241), and tag-only authorization returning 403 when an object cannot be read (#293). The pinned upstream SDK can drop conditional headers at multipart completion, so mcli put --if-not-exists is not a guarantee against overwrites throughout a multipart upload. These limits are not resolved by upgrading.

Release acceptance covers the exact source’s Go/race and packaging gates, two-pool and four-node runtime checks, S3/client/Console compatibility, and upgrade/rollback of the exercised object and IAM data. Local performance measurements use Linux ARM64 in a VM with directory-backed drives; they do not establish production disk or cross-host capacity, nor do these bounded checks constitute a long-duration soak test. Verify the workload and topology used by your deployment.