Ana içeriğe geç

Architecture

Tenarium is a single Go binary. It serves the REST API and the admin UI, keeps state in PostgreSQL (or in memory for development), and drives plugins.

request ─▶ authenticate ─▶ authorize ─▶ validate ─▶ hooks ─▶ ┌ transaction ──────────────┐
│ write state │
│ append event (outbox) │
└───────────────────────────┘
│ commit, NOTIFY
┌──────────────────────────────────┴─────────────┐
▼ ▼
subscription "core/provisioning" subscription "kafka/publish" …
own cursor · retries · DLQ · lease own cursor · retries · DLQ · lease

Domain model​

Tenant: id (UUIDv7, so IDs sort by creation time), slug (a DNS label, unique among tenants that are not deleted, immutable), name, optional parent_id, status, plan, labels, settings (free-form JSON, optionally validated by a JSON Schema), feature and limit overrides, and version for optimistic locking.

Tenant status follows a state machine:

create ─▶ provisioning ─fail─▶ failed
│ ▲ │
│ └────retry─────┘
▼ ok
active ◀─resume── suspended
│ └──suspend──────▲
▼ delete (from any live status)
deleting ─▶ deleted

Without provisioners a tenant is active immediately and is deleted at once.

Hierarchy. A tenant may have a parent, which is set at creation and never changes. You cannot create a child under a deleting or deleted parent, or delete a tenant that still has live children (both checks lock the parent row, so they hold under concurrency). A child created without a plan inherits its parent's plan. Tenant-scoped administrators (see auth) reach their tenant and all of its descendants.

Plan: an ID, a name, features (map[string]bool) and limits (map[string]int64). A tenant's entitlements are its plan's features and limits (or those of a pinned plan revision), plus add-ons and feature rollouts, with the tenant's own overrides applied on top (see plans and placement). A plan that live tenants still use cannot be deleted. With tenants.require_known_plans set, tenants can only reference existing plans.

Member: an email, a role (from tenants.member_roles), and an optional user_id linking an identity. A member without user_id is invited; setting it activates the member. The members limit caps the number of members per tenant.

Usage: named counters per tenant (api_calls, storage_gb, …). RecordUsage adds a delta atomically and refuses increments that would exceed the tenant's limit for that metric (HTTP 429, plus a tenant.quota_exceeded event). Only active tenants can consume quota; negative deltas are always allowed. Counters clamp at zero.

Events​

Every state change appends a domain event to the event log in the same transaction as the change itself: a transactional outbox. A crash therefore never leaves a change without its event, or an event without its change.

EventWhen
tenant.created, tenant.updated, tenant.plan_changedcreate / update (plan changes emit both)
tenant.provisioning, tenant.provisioned, tenant.provisioning_failedprovisioning
tenant.suspended, tenant.resumedsuspension
tenant.deleting, tenant.deleted, tenant.deprovisioning_faileddeletion
tenant.member_added, tenant.member_updated, tenant.member_removedmembers
tenant.region_changeda region move
tenant.purgedthe data of a deleted tenant was purged
tenant.quota_exceeded, tenant.usage_threshold_reached, tenant.usage_period_closedusage
tenant.action_scheduled, tenant.action_canceled, tenant.action_failedscheduled actions
tenant.maintenance_started, tenant.maintenance_endedmaintenance windows
rollout.updated, rollout.deletedfeature rollouts
plan.created, plan.updated, plan.deletedplans
user.*, domain.*, apikey.*, sso.*, role.*, group.*, webhook.*, oauth_client.*, signing_key.*users and tenant configuration

The complete list, with a JSON Schema for each type's data, is the event schema registry.

Events are CloudEvents 1.0 with the extension attributes tenantid, actor (who caused it: {authenticator}:{subject}, {authenticator}:{subject};impersonated-by={authenticator}:{subject} during impersonation, plugin:<instance>, system:scheduler or system), sequence, and the W3C traceparent/tracestate of the request that caused them. Tenant events carry the new tenant and the previous state in data.

Ordering without gaps​

Readers consume the log by sequence number ("everything after N"). A naive bigserial outbox breaks this: transaction A takes sequence 10, transaction B takes 11 and commits first; a reader sees 11, moves its cursor past 10, and never sees A's event. Tenarium takes a transaction-scoped advisory lock right before drawing sequence numbers, so writers commit in sequence order and no reader can observe a hole that is later filled. The lock covers only the end of write transactions; Tenarium is a control plane, so its write rate is modest. The store conformance suite (internal/store/storetest) checks this with concurrent writers against both stores.

Delivery​

The dispatcher runs one worker per subscription. A subscription is a plugin's event handler with a stable name ({plugin-instance}/{name}) and its own durable cursor. Delivery is:

  • At least once and ordered per subscription. Handlers must be idempotent; use the event ID.

  • Isolated. A failing subscriber retries with exponential backoff and jitter. After MaxAttempts the event goes to the dead-letter queue and the cursor moves on. Other subscribers are not affected. Dead letters can be inspected, replayed and discarded through the API or the UI.

  • Prompt. Commits wake workers in the same process directly, and other replicas through PostgreSQL LISTEN/NOTIFY, so polling (events.poll_interval) is only a fallback.

  • Traceable. Each event stores the W3C trace context of the request that caused it. A delivery attempt is its own span, linked to the producing span rather than parented by it, because delivery is asynchronous and may repeat (see tracing).

A new subscription starts at the beginning of the log (it replays history) unless it sets StartAtEnd. A time range of stored events can be redelivered to one subscription without moving its cursor (POST /v1/events/replay, see replaying a time range).

Provisioning is a subscription too​

The core subscription core/provisioning reacts to tenant.created, tenant.provisioning (retry) and tenant.deleting. It runs the provisioners in configuration order (each step is retried a few times) and then marks the tenant active, or failed with the reason. Deprovisioning runs in reverse order with the same per-step retries. A tenant is only reported deleted once its resources are gone; if a step keeps failing, the tenant becomes failed with the reason (tenant.deprovisioning_failed), and deleting it again retries. Failing rather than retrying forever keeps one stuck tenant from blocking provisioning for everyone else. Because provisioning is driven by the outbox, a process that dies halfway simply has the event redelivered, and the idempotent provisioners run again.

Hooks​

Hooks run synchronously before a change is stored, in priority order. before_create and before_update hooks may modify the proposed tenant (for example to add default labels); member add and update hooks may change the role and user ID. Other hooks can only allow or reject. A hook rejects with sdk.Reject(...), which the API returns as 422 with the reason. After hooks run, Tenarium restores identity and lifecycle fields (ID, slug, parent, status, version, timestamps) and validates the result again, so hooks cannot get around the state machine.

Hooks run outside the transaction because they may be slow. The write that follows is guarded by the version that was read, so a concurrent change makes the operation fail with a version conflict rather than overwrite it.

Running several replicas​

All replicas serve the API. For event delivery, each subscription is guarded by a lease in the database: one replica holds it and renews it every lease_ttl/3. If that replica stops, another takes the lease within lease_ttl and continues from the shared cursor. A replica that loses its lease (for example after a long GC pause) cancels its in-flight delivery, so two replicas do not process the same subscription at the same time.

Plugins​

See plugins. In short, a plugin implements sdk.Plugin and, during Init, registers any of: event subscriptions, hooks, provisioners, HTTP routes, authenticators and authorizers. Registrations are fixed at startup. A plugin whose Init fails stops startup, and nothing it registered before failing is kept. Panics in plugin code are recovered and treated as errors.

Stores​

store.Store has two implementations: memory for development and tests, and postgres for production. Both pass the same conformance suite. The PostgreSQL store applies its embedded migrations at startup under an advisory lock, so several replicas can start at once.

Package map​

PackageResponsibility
pkg/sdkPublic plugin contract: types, interfaces, registry
internal/tenantDomain services, validation, state machine, provisioning runner
internal/eventsDispatcher: cursors, leases, retries, DLQ, replay
internal/store/{memory,postgres}Persistence and outbox
internal/pluginPlugin loading, registrar, hook chain, scoping
internal/authzTenant-scope rule and authorizer combination
internal/apiHTTP handlers, authentication chain, idempotency, metrics
internal/uiEmbedded admin UI
internal/identityToken signing keys, JWTs, OIDC discovery
internal/ssoPer-tenant OIDC and SAML sign-in
internal/oidcproviderTenarium as an OpenID provider (/oauth)
internal/scimSCIM 2.0 service provider per tenant
internal/webhooksSelf-service tenant webhooks
internal/eventschemaJSON Schemas of every event type
internal/secrets, internal/store/encryptedEnvelope encryption of stored secrets
internal/certsACME certificates for platform and tenant hostnames
internal/netguardSSRF guard for tenant-chosen URLs
pkg/serverConfiguration and wiring