Strategy¶
The previous page fixes the six layers and their dependencies. This one explains why that order cannot be permuted, what a single command applies and what is applied by hand, the place of ArgoCD, two pairs of mechanisms that are easy to confuse (liveness and readiness probes, HPA and metrics-server), and how to roll back after an update.
Why the order of the six layers cannot be permuted¶
Four concrete dependencies, each of them already observed or already verified in this repository, justify the order fixed on the previous page.
- Secrets before certificates, even though nothing hints at it. The
ClusterIssuerof the DNS-01 challenge resolves its Cloudflare token through theExternalSecretcloudflare-dns-token, fed by Vault through theClusterSecretStorevault-store. Issuingcerts/before that path works leaves theClusterIssuerwithout a token: the DNS-01 challenge never resolves, the wildcard certificate staysFalseindefinitely, and nothing tells the operator that the cause is a link one layer further away. None of this makes the order circular: Vault (secrets/vault.yaml, namespacevault) is covered bykubectl kustomize ., hence available from the very first command, and its Servicevault.vault.svc:8200resolves through the standard internal DNS of Kubernetes, depending neither on the customised CoreDNS nor on any certificate. It is that route, not an Ingress, that is used to unseal Vault and to write the Cloudflare token into it. - The certificate before the Ingresses. The Traefik TLSStore references the wildcard
certificate issued by cert-manager. An Ingress applied before that certificate is
Readystill starts, but Traefik can only serve it a default certificate, not the one for the customer's domain. - Secrets before applications. External Secrets Operator injects the connection credentials (application database, SMTP, MinIO) into the Kubernetes Secrets consumed by the .NET pods and the portals. An application started before those Secrets exist restarts in a loop with no explicit message on the Kubernetes side, the same symptom as an unreachable external data endpoint.
- CoreDNS before the .NET pods. The resolution of
*.${TV_DOMAIN}by CoreDNS conditions the resolution of the name for the .NET services. The trap already hit on this repository, a CNAME answer that glibc does not follow, replays itself if a .NET pod starts before CoreDNS is in place. TLS trust no longer depends on anything: since the internal CA was removed on 2026-09-10,*.${TV_DOMAIN}is signed by Let's Encrypt, already present in the default store of every image.
Placing the Cloudflare token by hand does not exempt you from populating Vault
It is possible to create the cloudflare-dns-token Secret by hand, without going through
Vault or ESO, in order to issue the wildcard without waiting. That shortcut does not hold
over time: as soon as ESO starts, the ExternalSecret of the same name takes that Secret
over. If the corresponding Vault path has not been populated beforehand, the token placed
by hand disappears at that moment, with no message to signal it. The failure then shows up
only at the next renewal of the certificate, weeks later, far from its real cause. That is
why the recommended order remains Vault and ESO first, never the other way round.
What a single command covers, what is applied per directory¶
The repository is not entirely kustomize, for a historical reason: the apps, ingress,
externalsecrets, keycloak and mailpit modules were added after the base layer, without being
reintegrated into the root kustomize. kubectl kustomize . therefore only covers that base
layer: namespace, network, cert-manager, cluster-internal data layer, one observability stack,
Vault, Traefik, ArgoCD. Everything else is applied per directory. The inventory below is
ordered by layer, it is not an execution order and it is not the installation procedure: it
names what the base command alone leaves aside. Playing it from top to bottom would fail, if
only because kube-vip and the Cloudflare token must come before certs/. The exact commands,
their order and their prerequisites live in the Step by step.
| Layer | What the root kustomize does not lay down | Effect of the omission |
|---|---|---|
| 2, secrets | secrets/eso/vault-store.yaml |
no ExternalSecret synchronises |
| 3, certificates | core/cert-manager/cloudflare-dns-token.yaml, then certs/ |
the ClusterIssuer has no token, the wildcard is never issued |
| 4, network and ingress | core/kube-vip/ |
no LoadBalancer service obtains an address |
| 4 | core/traefik/traefik-rbac.yaml, core/traefik/traefik-ingressclass.yaml |
Traefik runs and reads no Ingress, hence serves no route, with no visible error |
| 4 | core/traefik/tlsstore-default.yaml |
Traefik serves its self-signed certificate on every host |
| 4 | core/coredns/coredns-configmap.yaml |
*.${TV_DOMAIN} does not resolve from the pods |
| 4 | the two named Ingresses of core/platform-access/, core/lan-dns/ |
no access to the operating interfaces, no LAN or VPN resolution |
| 4 | core/metrics-server/metrics-server.yaml |
kubectl top silent, ten HPA inert at <unknown> |
| 4 | core/monitoring/node-exporter.yaml, which step 10 of the step by step adds outside its main block |
no host metric (CPU, memory, disk of the nodes) reaches Prometheus; the tradeval-ressources dashboard stays empty on those series |
| 6, applications | apps/api/, apps/keycloak/, apps/web/ |
neither Keycloak, nor microservices, nor portals |
| 6 | the four files of core/cloudflare/, in order |
no public exposure |
The node-exporter line is the only one in the table that the base layer did not lay down and
that no inherited manifest called for: it is step 10 of the
Step by step that adds
it, with its command and its verification, outside its main block so that this status stays
visible. Do not confuse it with metrics-server: both serve metrics, but only
metrics-server feeds kubectl top and the HPA, and only node-exporter feeds Prometheus.
core/vpn/ does not appear in this table, and that is not an oversight: what that directory
contains is not applied by the procedure. VPN access goes through a gateway VM outside the
cluster, the option retained by the repository; wireguard-incluster.yaml is option B, its
header carries Non appliqué, and it mounts a Secret that no applied manifest creates. Step 13
of the Step by step.
An operator who does not know about this split risks a silent omission: applying only
kubectl kustomize . leaves the cluster without a VIP of the customer's own, without a
wildcard, without a customised CoreDNS, without CA trust, without a ClusterSecretStore hence
without a single synchronised secret, without LAN or VPN resolution, without active HPA,
without Keycloak, without microservices or portals, and without public exposure, with no error
to signal it at the time of the apply.
Two observability stacks, only one actually wired up
The root kustomize deploys a Prometheus, Grafana, Kibana stack in the observability
namespace. It runs, but nothing connects to it: the stack actually used to operate the
instance (Prometheus, Grafana, Loki, Alloy) is installed separately by Helm, in the
monitoring namespace, outside the scope of kubectl kustomize .. An operator who applies
the root kustomize gets the first stack and sees no useful metric go through it, with no
message to explain why. The Observability page details which of the two to install and how.
The place of ArgoCD¶
The installation remains driven by the kubectl apply commands of this page and of the
previous one, run by hand. ArgoCD comes after it: installed by Helm from platform/argocd/, it
then follows the main branch of this repository for the application directories (apps/),
and application updates go through git. The base layers (Vault, External Secrets, network)
stay applied by hand. See platform/README.md.
Liveness and readiness probes, why they are distinct¶
A readiness probe (readinessProbe) removes a pod from the Service as long as it does not
answer, without restarting it. A liveness probe (livenessProbe) restarts the pod if it fails
for too long. Confusing the two, or pointing either of them at an unsuitable path,
produces very different symptoms: a pod that restarts in a loop with no application reason in
one case, a pod that is Running but unreachable for hours in the other, with no classic
dashboard signalling it since it counts Running pods.
It is exactly that second symptom that occurred on the reference instance: a portal stayed
unavailable for several hours, without restarting and without any error appearing on the
Kubernetes side, because its readiness probe followed a redirect towards an authentication
page whose rendering time exceeded the delay granted to the probe. The lesson to retain: a
probe path must never redirect, nor authenticate, nor depend on an external service. On the
ten deployments of this repository that expose an application HTTP route, the readiness and
liveness probes are already separated, each on a direct path (/health and /health/ready on
the .NET side, /api/health on the Next.js portals side): an operator who adds a service will
follow that same scheme rather than point a probe at a page that can redirect.
The second half of the rule, the delay granted to the probe, is not respected everywhere in
the delivered repository, however: the readiness probe of hub-portail does not set
timeoutSeconds and therefore takes the default of one second, the very value that the
incident incriminates. To be fixed before go-live, detail at
Troubleshooting, entry 11.
HPA and metrics-server¶
The ten HPA of this repository, applied with the rest of apps/api/ and apps/web/, react to nothing as long
as metrics-server is not installed: they stay inert, and signal it only by an <unknown>
value in kubectl get hpa. The manifest is
core/metrics-server/metrics-server.yaml, applied at layer 4 in the block of commands above:
it does not live in kubectl kustomize ., and an operator who does not apply it ends up with
ten inert autoscalers with no message to indicate it. Installing metrics-server wakes up all
the existing HPA at once, with their configuration as it is written in the repository, with no
running-in period. That is what happened during the reference installation: the HPA climbed to
their maximum within a few minutes, saturating the CPU of the cluster and blocking every
deployment under way. The mechanism, the delivered values and the way to check that they have
not drifted are at
Troubleshooting, entry 10;
the lesson to retain is that the CPU and memory requests values, and the maxReplicas
bounds, must be confronted with real usage before installing metrics-server, not after.
Updating a service and rolling back¶
Rolling back an application deployment uses kubectl rollout undo, as for any standard
Kubernetes Deployment.
That mechanism is enough for the .NET microservices, whose configuration is read when the
container starts. It is not enough for the Next.js portals: their NEXT_PUBLIC_* variables are
frozen at the moment the image is built, not at startup. If the rollback follows a domain
change, going back to the previous image restores code that still calls the old
${TV_DOMAIN}, even after the rollback. In that specific case, a complete rollback requires
rebuilding the portal image with the right variables, not merely changing the deployed version.