Operations¶

This page covers the platform once it is installed and validated: backing up and restoring, renewing the certificates, updating a service, adding one, and knowing what has to be looked at every day. It does not replay the installation, which lives in the Step by step, nor the diagnosis of a failure, which lives in Troubleshooting.

This is the last page of the guide. After it, the operator is alone with the instance: everything that follows is written to be reread six months later, without the context of the installation.

Backups and restore¶

The Vault unsealing procedure does not exist yet. It is blocking.

Vault reseals itself at every complete restart, and the seal state is specific to each process: the three pods are unsealed one by one, each with three of the five keys. As long as that is not done, no ExternalSecret synchronises, and nothing else in the platform signals it: the conformance recipe only sees it indirectly, and only if an ExternalSecret attempts a resynchronisation in the meantime.

This guide carries no written and tested unsealing or Vault backup procedure. That is a known gap. It has to be written and exercised at the customer's site before going live, and it has to answer four questions: who holds the unseal keys, how many people hold them, where they are kept, and who is reachable at night. Without that procedure, a complete restart of the cluster stops the platform until a person holding three of the five keys logs in.

What to back up, and at what rhythm¶

Object	Backup	Restore	Frequency
Vault store data (Raft storage)	`vault operator raft snapshot save`, from the workstation and authenticated: the complete command and how to observe it are just under this table	`vault operator raft snapshot restore`, through the same authenticated path, on an initialised and unsealed Vault	daily, and before any intervention on the store
Unseal keys and root token	this is not a command: it is the file produced by `vault operator init`, to be kept outside the repository and outside the cluster	without three of the five keys, no restoration is possible, the Raft snapshot being encrypted	once at installation, then at every key change
Keycloak database (realm, clients, users)	`pg_dump -h <hote> -U <utilisateur> -Fc <base> > keycloak-$(date +%F).dump`	`pg_restore -h <hote> -U <utilisateur> -d <base> --clean keycloak-<date>.dump`	daily, and imperatively before any modification of the realm
Persistent volumes of the cluster	through the storage controller: declare an external backup target (NFS or S3), then a recurring task on each volume	restore the volume from the target, then reattach the `PersistentVolumeClaim` to the restored volume	daily towards the external target
Rendered manifests and customer sheet `scripts/install.env`	copy outside the operator's workstation, in the same place as the unseal keys	put the file back in place and replay `render-config.sh`	at every modification

Backing up Vault, in practice¶

vault operator raft snapshot save hits an authenticated endpoint of the Vault API. A kubectl exec inside the container is therefore not enough: secrets/vault.yaml places no token there, and the command exits on missing client token. The contrast with vault operator unseal, which the step by step does launch through kubectl exec, is not an inconsistency: unsealing, for its part, is not authenticated.

The path that works is the one of step 6.2 of the Step by step, a port-forward towards the internal Service followed by an authentication:

kubectl port-forward -n vault svc/vault 8200:8200 &
export VAULT_ADDR=http://127.0.0.1:8200
vault login                     # token asked for interactively, never passed as an argument
vault operator raft snapshot save "vault-$(date +%F).snap"

vault login without an argument asks for the token and does not display it. Passing it as an argument would leave it in the shell history and in the process table of the machine.

Observing that the backup succeeded. Without that observation, a daily task can produce empty files for months, and this is only discovered on the day of the restoration:

ls -l "vault-$(date +%F).snap"
vault operator raft snapshot inspect "vault-$(date +%F).snap"

The file has a non-zero size, and inspect returns the Raft index and the count of the entries it contains. That is the only signal available: the presence of the file proves nothing, and a truncated archive makes inspect fail while ls shows it.

The automated task is provided: secrets/vault-backup.yaml. A daily CronJob takes the snapshot, verifies it with inspect, drops it into a MinIO bucket outside the cluster (TV_MINIO_ENDPOINT from the customer sheet), and prunes those older than thirty days. It carries no token: it authenticates through the vault-backup ServiceAccount against the Kubernetes role of the same name, which scripts/build-vault.sh configure creates with a policy that opens only that endpoint:

path "sys/storage/raft/snapshot" {
  capabilities = ["read", "sudo"]
}

The sudo capability is not optional on this endpoint. On the MinIO side, a bucket and an account that writes only into it, whose keys go into Vault under prod/vault-backup (access-key, secret-key) through vault kv patch: the ExternalSecret in the same file brings them down into the vault namespace. The SauvegardeVaultEnRetard rule fires after thirty-six hours without a success, and also as long as the first success has not happened.

Restoring takes the same authenticated path, on an initialised and unsealed Vault:

vault operator raft snapshot restore "vault-<date>.snap"

A snapshot is encrypted with the keys of the instance that produced it. The keys that will unseal Vault after the restoration are therefore those of the original instance, not those of the Vault into which it is restored. That is what makes the snapshot plus keys pair inseparable, and what makes the next line of the table something other than a formality.

Four points of vigilance, each already responsible for an unusable backup elsewhere:

A Vault snapshot without the unseal keys is worth nothing. It is encrypted with them. Keep the keys elsewhere than the snapshots, and both elsewhere than the cluster.
The Keycloak realm is created by hand, never recreated by a manifest. Losing that database means recreating one by one every OIDC client, every Valid Redirect URI and every Web Origin. It is the backup whose absence costs the most, and the only one whose content exists nowhere else in the repository.
A volume snapshot that stays on the same disks protects nothing. Without a declared external backup target, the snapshot disappears along with the node it was meant to cover.
scripts/install.env is excluded from git and only exists on the operator's workstation. Losing it forces every value of the instance to be derived again.

The Keycloak database is not provided by this repository

The Keycloak deployment hard-codes its JDBC URL towards a keycloak-db-rw Service on port 5432, which no manifest of this repository creates, and this seventh endpoint does not figure among the six external data endpoints of the customer sheet. The database is therefore provided by the customer, and the backup command depends on what they provide: pg_dump through the Service if the database is reached by that name, or the mechanism proper to the Postgres operator that runs it, if it is run inside the cluster. Settle that point with the customer at installation time, not at the time of the first incident. In both cases, pg_dump remains the portable fallback, and it is the one the table above gives.

The storage controller is not provided by this repository either

No manifest of this repository installs a storage controller: core/platform-access/ carries only the Ingress of the Longhorn interface, not its installation. The name longhorn nevertheless appears in the customer sheet, as the default value of TV_STORAGE_CLASS: that is the name of the default StorageClass that preflight.sh expects to find already laid down on the cluster, not an installation the guide would perform. Volume backup is therefore configured in the controller chosen by the customer. If it is Longhorn: its console also allows volumes to be deleted and has no authentication by default. Put the basic-auth of Observability in place before opening it to anyone.

What a backup does not prove¶

A restoration that has never been executed is a hypothesis, not a backup. Exercise it at least once before going live, on a disposable namespace or cluster, and note the duration measured: that is the only recovery-time figure that will be available on the day the question comes up for real. The conformance recipe of Validation exercises neither the backups nor their restoration, and does not claim to.

Certificate renewal¶

In nominal operation, there is nothing to do. Let's Encrypt issues for ninety days and cert-manager renews at two thirds of the lifetime, so around the sixtieth day.

The manual check¶

kubectl get certificate -A
openssl s_client -connect ${TV_VIP}:443 -servername hub.${TV_DOMAIN} </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -enddate -ext subjectAltName

It is the second command that counts. It reads what Traefik actually presents on the wire, not what the cluster believes it holds. A Certificate object in READY=True and an expired certificate served to browsers are two perfectly compatible states. The date to watch is the one returned by -enddate, not the READY column.

The symptom of a blocked renewal¶

There is none, for up to sixty days. That is what makes this failure peculiar: the cause and the symptom are separated by two months. Nothing in the logs, nothing in the events, nothing in the dashboards, until the day the certificate expires and every host becomes unreachable at once, with no browser workaround possible if ${TV_DOMAIN} is on the HSTS preload list.

Three causes cover almost every case:

The two DNS-01 arguments were erased from the cert-manager Deployment, typically by a kubectl apply of the upstream manifest, which does not carry them. Symptom, visible only by inspecting the challenge: Could not determine the zone for "_acme-challenge.${TV_DOMAIN}.": ... SERVFAIL.
The Cloudflare API token was revoked, expired, or lost its scopes. It needs Zone:DNS:Edit and Zone:Zone:Read on the customer's zone.
cert-manager older than v1.16.4 with Cloudflare. The challenge passes, then the TXT cleanup fails with DELETE "/zones//dns_records/<id>", error 7003, and the Order stays pending with a valid challenge. Cloudflare no longer returns zone_id in its records; older versions depended on it. Upgrading is enough, the pending cleanup goes through at the rollout: core/cert-manager/README.md.

After fixing the cause, force a new issuance without waiting for the exponential backoff:

kubectl -n traefik-ingress delete certificaterequest \
  -l cert-manager.io/certificate-name=wildcard-tradeval-app
kubectl get certificate -n traefik-ingress -w

The detail of the diagnosis is at Troubleshooting, entry 1.

What signals it now¶

Four rules cover certificates on the reference instance, in the certificats group of observability/prometheus-values.yaml: Certificate not Ready, renewal more than seven days late, expiry less than seven days away, and disappearance of the certmanager_* metrics. The last one exists because the other three read metrics collected through annotations that the cert-manager chart places on its own pod: a version bump that moves them would leave the first three silent without breaking anything visible. The manual check above remains useful before going live, it is no longer the only safety net.

Updating a service¶

The common case¶

kubectl set image deployment/<nom> <conteneur>=${TV_REGISTRY}/<image>:<tag> -n <zone>
kubectl rollout status deployment/<nom> -n <zone>

Rolling back: kubectl rollout undo deployment/<nom> -n <zone>.

What has to be seen. A rollout status that returns successfully rolled out is not enough: it says that the deployment finished, not that the new content is running.

kubectl get pods -n <zone> -l app=<nom> \
  -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}'
kubectl get pods -n <zone> -l app=<nom>

The image actually running is the expected one, and the pods are 1/1 and stay that way over several minutes.

An identical tag makes the update unverifiable and the rollback impossible

Most of the application images of this repository carry the latest tag. With an unchanged tag, rollout status succeeds without any proof existing that the new content was pulled, and rollout undo brings back the same reference, hence nothing. Publish every version under a distinct tag, or force imagePullPolicy: Always knowing that the rollback will remain impossible.

Changing a configuration value¶

The value is changed in Vault, never in the Kubernetes Secret, which is rewritten at the next synchronisation. Reaching Vault from the workstation first requires a port-forward and an authentication, as at step 6.2 of the Step by step: the Vault interface has no public DNS record and the vault client does not know how to talk to a Kubernetes Service on its own.

kubectl port-forward -n vault svc/vault 8200:8200 &
export VAULT_ADDR=http://127.0.0.1:8200
vault login
vault kv patch -mount=secrets prod/<service> CLE=valeur
kubectl rollout restart deployment/<nom> -n <zone>

patch, never put on a shared path. And the restart is not optional: environment variables are frozen at container startup. Without it, the Secret carries the new value and the running pods keep serving with the old one, without any error signalling it.

The particular case of the portals¶

The NEXT_PUBLIC_* variables of the three Next.js portals are resolved at the moment the image is built. No modification of a Deployment, of a Secret or of an environment variable reaches them, and a rollback restores an image that carries the old value.

Operational consequence: any change of ${TV_DOMAIN} or of a public URL requires rebuilding the three portal images before the deployment, not after. What is observed if this is forgotten: the pods are Running, Ready, with no error, and only the requests issued from the browser go to the old address. The complete diagnosis is at Troubleshooting, entry 8.

Adding a service¶

Five things to write, none of them optional. The first four are manifests, the last one is what prevents the new service from becoming invisible to the person who will operate the platform after you.

#	To create	Where	What it carries, and the trap
1	`Deployment`	`apps/api/<service>/` or `apps/web/<portail>/`	image on `${TV_REGISTRY}`, `imagePullSecrets: acr-pull-secret` (which no manifest creates, it is placed by hand), `envFrom` on the `Secret` produced by the `ExternalSecret`, and distinct readiness and liveness probes, each on a direct path that does not redirect and does not authenticate
2	`Service`	same directory	`ClusterIP`, with the exact name the `Ingress` refers to
3	`Ingress`	same directory	host `<service>.${TV_DOMAIN}`, without `spec.tls`: the certificate comes from the default `TLSStore` that serves the wildcard. An `Ingress` that declares its own `spec.tls` forces the wildcard Secret, and its private key, to be duplicated into the namespace
4	`ExternalSecret`	same directory, a single file	`ClusterSecretStore/vault-store`, `dataFrom: extract` on `prod/<service>`, `creationPolicy: Owner`, `deletionPolicy: Retain`, `refreshInterval: 1m`. A `kubectl apply` on a directory applies in alphabetical order and the last definition silently wins: two files for the same object is a trap, not a redundancy
5	The entry in the configuration matrix	Configuration	service, namespace, Vault path, runtime. This is the table one rereads before writing into Vault

Nothing to add to the validation recipe, and that is deliberate. scripts/validate-install.sh carries no hand-written list of hosts: check 8 derives its own from the repository's Ingress manifests, minus the hosts routed by the tunnel. Laying down the Ingress of line 3 is therefore enough to bring the new host into the recipe. The number of ok lines of check 8 increases by one: that is what proves the addition took effect.

A sixth thing existed until 2026-09-10: mounting an internal CA ConfigMap, whose key and environment variable depended on the runtime. It disappeared with the internal CA. A new service no longer has anything to mount in order to trust *.${TV_DOMAIN}: the Let's Encrypt chain is already in the default store of its image.

Two decisions that are not files¶

Public or internal? Internal by default: no public DNS record, no tunnel route. Three names only are public. Since the internal TLD was abandoned, this partition rests on nothing but that discipline, and its only automatic safeguard is check 8 of the recipe, hence line 5 of the table above. Making a host public is a decision, not a consequence: it is taken explicitly, and it adds a DNS record and a tunnel route, never one without the other.

Do the resource limits cover the peak? A CPU limit does not cause an error, it causes slowness, invisible in the state of the Kubernetes objects. For any bursty compute load (image optimisation, PDF generation, compression, cryptography), the limit must cover the peak, not the average; the requests remain the lever for the reservation. And never an autoscaler on memory for a garbage-collected runtime (.NET, JVM, Node): it climbs to its maximum whatever the real load.

Verifying after the addition¶

kubectl get externalsecret <nom> -n <zone>            # SecretSynced
kubectl get pods -n <zone> -l app=<nom>               # 1/1, stable
kubectl get endpoints <service> -n <zone>             # the pod IP figures there
dig +short <service>.${TV_DOMAIN} @1.1.1.1                 # no answer if the host is internal
curl -sSI https://<service>.${TV_DOMAIN}                   # from a workstation that resolves to the VIP

The endpoint is the check that is most often missing: a Running pod whose IP does not appear in the endpoints is a pod that Traefik does not see, and the host will answer 404.

What to monitor day to day¶

Two versioned dashboards carry the essentials, both to be imported by hand into Grafana, described in Observability: tradeval-red answers on throughput, error rate and latency per service, from the single Traefik source; tradeval-ressources answers the question of whether the cluster is the limiting factor.

What those two dashboards do not show has to be looked at some other way. The table below lists only things that, on this platform, are signalled nowhere else.

What	Where to read it	What must raise an alarm	Why it is only visible here
Latency per service	`tradeval-red` dashboard	a latency that climbs without the error rate moving	a CPU limit set too low produces neither an error, nor a restart, nor an event: only latency moves
Pods `Running` but not `Ready`	`kubectl get pods -A`	any `0/1` pod for more than a few minutes	no classic dashboard shows it, they count `Running` pods. That is the defect that left a portal in 404 for three hours
Expiry of the wildcard certificate	the `openssl` command above	fewer than thirty days left	a blocked renewal is only visible at the next renewal, up to sixty days after its cause
Seal state of Vault	`kubectl exec -n vault vault-0 -- vault status`	`Sealed: true`	nothing else signals it; the recipe only sees it indirectly, through the `ExternalSecret`, and only if they resynchronise
State of the `ExternalSecret`	`kubectl get externalsecret -A`	anything other than `SecretSynced`	`deletionPolicy: Retain` turns the disappearance of a key on the Vault side into a silent drift: the `Secret` outlives its source, and the gap is only visible there
Restarts of the apiservers and of Traefik	`kubectl get pods -n kube-system -o wide`	any `RESTARTS` counter that climbs	before the saturation incident, the apiservers had accumulated several restarts over two months, in a supposedly normal regime
Memory of the control-planes	`kubectl top nodes`	beyond 80 %	at 2 GiB they were running at more than 100 % in normal operation, which is visible in no application symptom until saturation
Internal hosts not published	check 8 of Validation	any internal host that resolves publicly	the only automatic safeguard of the public/internal partition

One line of this table falls silent if the metrics source is mute

kubectl top nodes and the HPA values come from metrics-server, laid down at step 10 of the Step by step. If it no longer serves metrics, kubectl top returns nothing and the HPAs display <unknown>: the "Memory of the control-planes" line above then monitors nothing at all, and the operator does not notice it, since its absence of result looks like an absence of problem. Check it at the same time as the rest, diagnosis at Troubleshooting, entry 13.

Thirty-two rules are in place, and the recipient is a real mailbox

Every line of this table now has a rule backing it: restarts, pods not ready, node memory, latency, 5xx, scrape targets gone. The manual check keeps its value before going live, it is no longer the only safety net. Alerts leave through the customer's SMTP to TV_ACME_EMAIL, once the recipient is written into the receiver (Observability page). Check once, after the installation, that an email really arrives: the simplest is the command on the Observability page, amtool alert add, or to read the Alertmanager log (kubectl -n monitoring logs sts/prometheus-alertmanager): a repeated notify retry there points at the SMTP, and nothing else will report it.

Before going live: what remains to be settled with the customer¶

Outbound mail is checked end to end, for each of the three senders

trade-notification-svc, Keycloak and Alertmanager each send with their own configuration (steps 11 and 14 of the step by step, Observability page). A misconfigured SMTP produces no error visible to the user: a registration that does not send its email, a password reset that never arrives, an alert that does not leave. Before going live, trigger a real send through each path and see it land in a mailbox, then check at the provider that the sender domain is verified (DKIM, SPF): without that, emails leave and end up as spam.

Three other points have to be closed before declaring the instance live, and none of them is covered by the conformance recipe:

The Vault unsealing and backup procedure, written and exercised, with the four questions of the admonition at the top of the page.
The restoration, exercised at least once, measured duration noted.
The authentication of the operating interfaces. Grafana carries its built-in login and Vault its token authentication; the other three, Prometheus, Alertmanager and the storage console, are exposed without the slightest password, and the last one allows volumes to be deleted. The basic-auth recipe is in Observability.