Step by step¶
The previous pages fixed the six layers, their dependencies and why that order does not permute. This page executes: every step carries its command, its check, and the symptom that shows up if it is chained too early. A check that merely records that a command answered, without recording that the step produced its effect, is deliberately left out: it would turn an immediate failure into a failure discovered several steps further on, with no visible link to its cause. This page does not go back over the why already detailed in Architecture and Strategy, except for the one point that was written nowhere before this page: the exact bootstrap order between secrets and certificates, which breaks an apparent circular dependency. This page does that first, before the commands, because departing from it blocks the installation in a way that is not easy to diagnose.
Before starting¶
- The customer sheet is filled in: Parameters.
scripts/preflight.sh --env scripts/install.env --skip-clusterpasses: Prerequisites.- The six virtual machines exist on the hypervisor side, but none of them has a Talos configuration applied yet.
Why secrets come before certificates¶
At first sight, secrets and certificates reference each other. The Let's Encrypt
ClusterIssuer (certs/letsencrypt-issuer.yaml) resolves its Cloudflare token through the
cloudflare-dns-token Secret, produced in the cert-manager namespace by an ExternalSecret
(core/cert-manager/cloudflare-dns-token.yaml) that targets the vault-store
ClusterSecretStore. And the Vault web interface is itself exposed under the wildcard domain
(core/platform-access/vault.yaml), an Ingress that can only serve an already issued
certificate. Issuing the certificate therefore seems to require Vault, which seems to require
the certificate.
The loop does not close, because Vault never needs that Ingress in order to work.
secrets/eso/vault-store.yaml reaches Vault through its internal Service, in the clear,
inside the cluster:
That Service, vault.vault.svc:8200 in the vault namespace, resolves through the standard
internal DNS of Kubernetes as soon as the CNI is in place, depending neither on the customised
CoreDNS of layer 4, nor on the wildcard certificate, nor on any Ingress. It is that Service
the operator reaches directly from their workstation, through kubectl port-forward or
kubectl exec, in order to unseal Vault and write into it the Cloudflare token the
ClusterIssuer is waiting for. The Ingress of the platform-access page only adds more
comfortable web access to an already working UI, once the certificate is issued: it is a
prerequisite of nothing.
The order that follows from this, detailed step by step below: Talos, the base layer, storage,
the service VIP, Vault and its population, the ClusterSecretStore, and only at that point can
the Cloudflare Secret materialise and the certificates be issued. The rest of the network
layer, the data layer, the application images, Keycloak and the applications come next, in that
order because each of them consumes something the previous step produces.
The manual fallback exists, and so does its trap
It is possible to place the cloudflare-dns-token Secret by hand
(kubectl create secret generic cloudflare-dns-token -n cert-manager --from-literal=api-token=...)
in order to issue the wildcard before Vault has been populated. That works on the first
pass. But the ExternalSecret of the same name takes over as soon as the
ClusterSecretStore is ready, with creationPolicy: Owner: it overwrites the Secret at
every synchronisation. If the Vault path prod/cloudflare-dns has never been populated
by then, the token placed by hand disappears with no error signalling it at the time. The
failure only shows up at the next renewal of the certificate, weeks later, far from its
cause. Place the token by hand only as a transitory test, never as a final state.
1. Bootstrap the Talos cluster¶
| Command | talosctl gen secrets -o secrets.yaml, then talosctl gen config tradeval-cluster https://${TV_CP_ENDPOINT}:6443 --with-secrets secrets.yaml, filling in ${TV_POD_SUBNET} and ${TV_SVC_SUBNET} in the generated config; talosctl apply-config --insecure -n <control-plane> --file controlplane.yaml on each control-plane, worker.yaml on each worker; talosctl bootstrap -n <un control-plane> |
| Check | kubectl get nodes: as many nodes in Ready as ${TV_NODE_COUNT}, six on the reference sizing |
| Symptom on failure | Nodes in NotReady, or talosctl not answering: check install.disk (it must designate the system disk, distinct from the Longhorn disk) and the extensions of the worker profile. Full detail: talos/README.md section "Régénérer / réinstaller un cluster depuis zéro" and section PROD |
Generate a set of Talos secrets specific to this instance: never reuse the one from another
installation. ${TV_CP_ENDPOINT} lives in that machine configuration and nowhere else, out of
reach of render-config.sh. ${TV_POD_SUBNET} and ${TV_SVC_SUBNET} live there too, but
not only there: render-config.sh substitutes them into
core/vpn/peer-officer.template.conf, where the tunnel advertises those two ranges to the
agents. The values entered here and those of the sheet must therefore be the same, failing
which the tunnel comes up, the agents connect and reach nothing behind it, with no error
(Parameters).
The CNI: Canal, and a single cni block to fill in
This repository used to lay down core/networking/calico.yaml on top of the Talos default
Flannel, without anyone having observed what the coexistence produced. The reference
instance measured it on 2026-09-08: the Calico laid down this way wired no pod, but it
allocated address blocks and installed routes competing with those of Flannel, one of
them more specific, which would have made the pods of a whole range unreachable.
That file has been removed from the repository.
The CNI retained is Canal, core/networking/canal.yaml: Flannel for the network, Calico
for the NetworkPolicies. It requires two things, in this order: apply the machine
configuration, whose cni block is already set to none, so that Talos does not install its
own Flannel, then apply the manifest. Never the other way round: the canal DaemonSet
embeds its own flanneld, and two daemons would fight over the same subnet lease.
The VXLAN port in the manifest is aligned with the Talos one, 4789: do not change it without
checking what the Talos Flannel uses, two nodes on different ports do not talk to each other.
2. Apply the customer sheet to the manifests¶
| Check | Symptom on failure |
|---|---|
git diff --stat lists dozens of modified files |
Empty diff: the sheet still carries the original values of the repository, render-config.sh found nothing to substitute |
This step must precede any manifest application: the following layers read files already rewritten with the domain, the VIP and the other values of the sheet.
3. Lay down the base layer¶
| Check | Symptom on failure |
|---|---|
kubectl get pods -n vault: three pods vault-0, vault-1, vault-2 created (not Running yet, for want of storage: see the next step); kubectl get pods -n argocd: one pod starting up |
Admission errors that name a precise resource: fix that resource before replaying the command, rather than replaying blindly |
for z in web internal api auth; do kubectl get pods -n $z; done: no pod, the four zones have just been created and stay empty |
A pod already present here: the cluster is not new, or a command from a previous installation has been replayed. Deal with it before continuing, it would skew the reading of step 13, "Applications and exposure" |
This command applies kubectl apply -k .: the namespaces, the network (Calico), the
cluster-internal data layer (Redis, Kafka, Elasticsearch), the inherited observability stack,
cert-manager, Vault (the StatefulSet, not its content yet), Traefik and ArgoCD. It also lays down
the Traefik LoadBalancer Service, with its ${TV_VIP} address already pinned: that is the one
step 5, "Publish the service VIP: kube-vip", will have advertised.
It creates the four zones and lays down no object in them: the eleven application
workloads arrive at step 13, "Applications and exposure", and not before. A CrashLoopBackOff
or an ImagePullBackOff in one of these zones is therefore never expected, neither at this step
nor at another, and is never let through on the grounds that a dependency might still be missing
(Troubleshooting).
The four zones¶
The application workloads do not share a single namespace: they are spread over four trust
zones. Everywhere this guide writes -n <zone>, this table is where the value is read.
| Zone | Workloads | Exposure |
|---|---|---|
web |
hub-portail, portal-portail |
public, through the tunnel |
internal |
internal-portail |
VPN only |
api |
the seven trade-*-svc |
internal and VPN |
auth |
keycloak and its database keycloak-db |
public, auth.${TV_DOMAIN} |
Two practical consequences, both a source of error if ignored:
- A Secret is a namespaced object.
acr-pull-secretmust exist in all four zones, see step 12, "Images and registry secrets". - A short Service name only resolves within its own zone. A call that crosses a boundary is
written in full,
<service>.<zone>.svc.cluster.local. The URLs concerned live in Vault, not in the manifests.
Cleanup on an already installed cluster: the nginx controller
Does not concern a fresh installation, where there is nothing to do. This repository long
laid down, through its base layer, an nginx-ingress-controller Deployment with two
replicas in the ingress-nginx namespace, in addition to Traefik. It served no purpose: the
file creates no Service, so nothing routes to it; it creates no IngressClass; and the
seventeen Ingress objects of the repository all carry ingressClassName: traefik. It was
not without effect for all that, since it tolerates the control-plane taints and reserves
2 x 256 MiB, the very property that got the kube-apiserver processes of the reference
instance killed
(Troubleshooting, entry 12).
kustomization.yaml no longer pulls it in, and since kubectl apply -k has no --prune,
that removal deletes nothing of what is already running. On a cluster installed before that
change, delete it once:
cert-manager comes from the base layer, at v1.16.5
core/cert-manager/ carries the upstream v1.16.5 manifest as is, plus a kustomize patch
adding the two DNS-01 arguments. They are not optional: by default cert-manager checks the
challenge propagation against CoreDNS, which answers every *.${TV_DOMAIN} with Traefik.
The check would query Traefik instead of a public resolver and fail every time. Verify after
the apply:
kubectl get deploy cert-manager -n cert-manager -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}{.spec.template.spec.containers[0].args}{"\n"}'
v1.16.5 image, and --dns01-recursive-nameservers-only among the arguments.
Do not go below v1.16.4: Cloudflare no longer returns zone_id in its DNS records, and
earlier versions fail to delete the challenge TXT, which leaves the Order stuck after a
validation that did succeed. Details and upgrade procedure: core/cert-manager/README.md.
On a cluster installed before 2026-09-15 from the repository's old cert-manager.yaml, the
Deployment carries a different spec.selector (app: cert-manager), an immutable field:
the apply fails with field is immutable. Delete that Deployment and the old manifest's
service account first, then apply again:
4. Provision storage¶
Vault (step 3, "Lay down the base layer") and everything that follows depend on a default
StorageClass. No manifest of this repository installs a storage controller: grep -rn
storageClassName over the whole repository returns nothing, and
core/platform-access/longhorn.yaml is only the Ingress of its interface, not its installation.
The three volumeClaimTemplates of secrets/vault.yaml stay Pending indefinitely as long as
that point is not dealt with.
Installing the storage controller
docs/guide/02-prerequis.md documents the Talos extension iscsi-tools and the
/var/lib/longhorn mount point of the workers, consistently with TV_STORAGE_CLASS=longhorn
by default in the customer sheet. Longhorn itself comes from the upstream Helm repository,
with the repository's values:
helm repo add longhorn https://charts.longhorn.io
helm repo update
helm upgrade --install longhorn longhorn/longhorn -n longhorn-system --create-namespace \
--version 1.12.0 -f core/longhorn/longhorn-values.yaml
csi.kubeletRootDir is mandatory under Talos: if it
does not match the kubelet's real directory, the CSI plugin starts normally and no volume
mounts at all, the error showing on the pods that mount a PVC and not on Longhorn.
defaultSettings.defaultReplicaCount is 2 and not 3, because the three storage-carrying
workers have unbalanced disks.
| Check | Symptom on failure |
|---|---|
PREFLIGHT : tous les controles passent., in particular check 7, default StorageClass |
aucune StorageClass par defaut, les PVC resteront Pending: the storage controller is not installed or has not laid down its default StorageClass yet; StorageClass par defaut = X, attendue = longhorn: another StorageClass has taken the default role, to be fixed before continuing |
Check 8, couche de donnees, still shows its six ok lines |
<NOM> <hote>:<port> injoignable: that endpoint has stopped answering since the run of page 02, which already exercised it |
kubectl get pods -n vault: the three pods Running, not merely created |
Still Pending: the PVCs found no volume, check the default StorageClass again |
This is the first time in this procedure that preflight.sh runs without --skip-cluster.
That flag skips exactly the two sections that require a cluster, reachability (6) and the
default StorageClass (7): this run therefore only adds those two. The six endpoints of the data
layer, check 8, were already exercised by the run of page 02.
5. Publish the service VIP: kube-vip¶
vip_interface carries the network card name of the reference instance, to be fixed before applying
core/kube-vip/kube-vip-ds.yaml pins vip_interface to ens34. That is the network card
name of the VMs of the reference instance, not a universal value: a different hypervisor, a
different Talos version or a different card model gives ens192, eth0 or something else.
No variable of the customer sheet parameterises it and render-config.sh does not touch it.
With a name that matches nothing, the pods start, stay Running, and never publish an
address. Read the real name on a node before applying, then fix the manifest if it differs:
ID column of that output carries the interface names of the node. The same name must be
used in the vip block of the Talos machine configuration, if there is one.
Open question: the service account of this DaemonSet
core/kube-vip/kube-vip-ds.yaml declares serviceAccountName: kube-vip, but no manifest of
this repository creates that ServiceAccount nor the RBAC rights that go with it (reading and
updating Service, Endpoints, and Lease objects for leader election). If the pods stay
in a creation error for want of that service account, provision it by hand before
continuing, with the standard rights documented by the kube-vip project for its service mode
(svc_enable). Not verified in this repository: to be confirmed before a customer
installation, or to be fixed in the repository upstream of this procedure.
| Check | Symptom on failure |
|---|---|
kubectl get pods -n kube-system -l app.kubernetes.io/name=kube-vip-ds: one pod per control-plane node, Running |
Pods absent or in a creation error: the kube-vip ServiceAccount is missing, see the open question above |
An address really advertised: kubectl get svc -n traefik-ingress traefik carries ${TV_VIP} as EXTERNAL-IP |
<pending>: the pods run but kube-vip advertises nothing. The two causes are in the two boxes above, in that order of likelihood: vip_interface does not name the real interface of the nodes, or the kube-vip ServiceAccount does not have the rights to update the status of Service objects. kubectl logs -n kube-system -l app.kubernetes.io/name=kube-vip-ds tells the two apart |
This DaemonSet is distinct from the kube-vip of Talos: it advertises addresses for
LoadBalancer services (the Traefik VIP, the LAN DNS, the WireGuard concentrator), and not the
control-plane endpoint, which its manifest disables explicitly through cp_enable: "false".
What kube-vip does here, and what it does not do
This DaemonSet assigns no address. It is the manager of kube-vip: it advertises over ARP an address already carried
by a Service, it does not choose one. Assignment would be the role of
kube-vip-cloud-provider, a separate Deployment that reads the kubevip ConfigMap of the
kube-system namespace; this repository installs it nowhere, and that ConfigMap is
therefore read by no one.
Consequence for the operator, and it holds beyond this step: ${TV_VIP_RANGE} is a range to
be reserved and documented, not a range that is handed out. Every LoadBalancer Service
pins its own address, and the three of the repository already do so, in two different forms:
spec.loadBalancerIP for core/traefik/traefik.yaml, the kube-vip.io/loadbalancerIPs
annotation for core/lan-dns/lan-dns.yaml, and both at once for
core/vpn/wireguard-incluster.yaml, which is not applied by this procedure. A Service added
later only needs one of the two, the annotation being the form kube-vip documents today,
spec.loadBalancerIP being deprecated on the Kubernetes side. A LoadBalancer Service added
later with no pinned address will stay <pending> indefinitely, and no message will say why.
What is observed here is therefore not an assignment, it is an advertisement, and it bears on a Service that already exists: the Traefik one, laid down by the base layer at step 3, "Lay down the base layer", with its address pinned. Nothing between this step and step 10, "The rest of the network layer, ingress and internal DNS", creates it or modifies it, and that is exactly what that later step reads again five steps further on. The check line above therefore brings forward by five steps the discovery of the two failures the boxes above describe, and there is no reason to wait.
Do not manufacture another check here. Creating a test LoadBalancer Service, with no pinned
address, would never return anything, for want of assignment: that would be a symptom
invented on a healthy installation.
6. Install and bootstrap Vault¶
Vault exists and its pods are Running (step 4, "Provision storage"), but Vault has neither
content nor a neighbour able to query it. Three things are missing: the External Secrets Operator
controller itself, the initialisation of Vault, and its configuration to talk to Kubernetes.
6.1. External Secrets Operator¶
Installing the ESO controller
No manifest of this repository deploys the controller itself: it comes from the upstream Helm repository, with the repository's values.
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm upgrade --install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace --version 2.5.0 \
-f secrets/eso/external-secrets-values.yaml
installCRDs: true is not optional: without the ExternalSecret and ClusterSecretStore
CRDs, the chart installs and the apiserver rejects every manifest of the repository that
uses them.
Check: kubectl get pods -n external-secrets: the controller pods Running.
6.2. Initialise and unseal Vault¶
Still with no Ingress, through a port-forward to the internal Service. Write the result of the
initialisation outside the git clone: that file contains five unseal keys and a root token, and
a slightly too broad git add . is enough to pour them into the history of the repository.
kubectl port-forward -n vault svc/vault 8200:8200 &
export VAULT_ADDR=http://127.0.0.1:8200
vault operator init -key-shares=5 -key-threshold=3 > "$HOME/vault-init-${TV_DOMAIN}.txt" # outside the git clone, never committed
Unseal vault-0 with three of the five keys, then log in:
vault operator unseal <cle-1>
vault operator unseal <cle-2>
vault operator unseal <cle-3>
vault login # with the root token
The seal state is specific to each process: vault-1 and vault-2 join the raft thanks to the
retry_join already configured in secrets/vault.yaml, but stay sealed as long as this is not
done explicitly, with the same keys:
kubectl exec -n vault vault-1 -- vault operator unseal <cle-1>
kubectl exec -n vault vault-1 -- vault operator unseal <cle-2>
kubectl exec -n vault vault-1 -- vault operator unseal <cle-3>
# repeat for vault-2
| Check | Symptom on failure |
|---|---|
vault status (through the port-forward): Sealed: false, HA Mode: active on one of the three pods |
Sealed: true persisting on a pod: wrong unseal keys, or pod restarted since (the seal comes back at every restart of the process) |
6.3. Mount the vault and Kubernetes authentication¶
secrets/eso/vault-store.yaml expects a KV v2 mount named secrets and a Kubernetes
authentication method mounted on kubernetes: those two names are not arbitrary, they must
match exactly.
The rights to validate Kubernetes tokens
Configuring the Kubernetes authentication of Vault requires that the service account used to
query the API (the review token, token_reviewer_jwt) carry the right to create
TokenReview objects, through a binding to the system:auth-delegator ClusterRole.
secrets/vault.yaml ships that ClusterRoleBinding for vault-sa: applying the file is
enough, there is nothing to provision by hand.
Without it, vault write auth/kubernetes/config is accepted without error but the validation
of a token presented by ESO then fails with Forbidden. The symptom shows on the status of
the ExternalSecret, never on Vault, which makes the cause hard to track down.
Create the eso policy and role that vault-store.yaml references, restricted to reading the
prod/* paths (the prod prefix is the one all the ExternalSecret objects of the
repository carry, whatever the real environment of the customer, it is not a value to adapt):
vault policy write eso-read - <<'EOF'
path "secrets/data/prod/*" {
capabilities = ["read"]
}
path "secrets/metadata/prod/*" {
capabilities = ["list"]
}
EOF
vault write auth/kubernetes/role/eso \
bound_service_account_names=external-secrets \
bound_service_account_namespaces=external-secrets \
policies=eso-read \
ttl=1h
6.4. Write the bootstrap secret¶
The only secret strictly needed to unblock the immediate continuation of the procedure is the Cloudflare token, on its own path, separate from any other secret:
The token must carry Zone:DNS:Edit and Zone:Zone:Read, restricted to the ${TV_DOMAIN}
zone. put is suitable here because that path is dedicated to that single key. On any path
shared by several keys, use vault kv patch, never put, which would replace the entire
content of the path.
| Check | Symptom on failure |
|---|---|
vault kv get -mount=secrets prod/cloudflare-dns returns the api-token key |
Path absent: the previous command failed silently, or the secrets mount does not exist yet |
7. Close the loop: the ClusterSecretStore¶
| Check | Symptom on failure |
|---|---|
kubectl get clustersecretstore vault-store: READY=True |
READY=False: kubectl describe clustersecretstore vault-store carries the cause in the message of the Ready condition, and the three causes are not fixed in the same place. permission denied on a path: the eso-read policy or the eso role of the previous step. TokenReview or Forbidden: the RBAC binding of open question 6.3. connection refused or Vault is sealed: Vault has resealed itself, go back to step 6.2, "Initialise and unseal Vault" |
This is where the circular dependency closes in the right direction: from this step onwards,
any ExternalSecret of the repository can read Vault, without any Ingress or certificate
having been needed to get there.
8. Materialise the Cloudflare token in cert-manager¶
| Check | Symptom on failure |
|---|---|
kubectl get externalsecret cloudflare-dns-token -n cert-manager: SecretSynced; kubectl get secret cloudflare-dns-token -n cert-manager exists |
SecretSyncedError: the Vault path prod/cloudflare-dns is absent or misspelled, or the ClusterSecretStore is not Ready yet |
Wait until the Secret exists before continuing: the ClusterIssuer objects of the next step
reference it by its name, not through an explicit Kubernetes dependency, nothing stops them
from starting before it and silently failing to resolve the DNS-01 challenge.
9. Issue the certificates¶
| Check | Symptom on failure |
|---|---|
kubectl get certificate -A: READY=True on the wildcard certificate |
READY=False persisting: Cloudflare token absent or badly scoped (step 6.4, "Write the bootstrap secret"), or the two DNS-01 arguments missing on cert-manager (step 3, "Lay down the base layer"). kubectl describe certificate wildcard-tradeval-app -n traefik-ingress details the failing challenge |
Point of no return: do not chain on to the rest until this is Ready
If ${TV_DOMAIN} ends in .app, the domain is on the HSTS preload list of the browsers. As
long as that certificate is not Ready, no host of the platform is reachable over HTTPS, and
no browser workaround exists: the "continue anyway" button does not exist on that kind of
domain. There is no point applying the rest of the procedure before that line turns to
True.
10. The rest of the network layer, ingress and internal DNS¶
kubectl apply -f core/traefik/tlsstore-default.yaml
kubectl apply -f core/traefik/traefik-rbac.yaml
kubectl apply -f core/traefik/traefik-ingressclass.yaml
kubectl apply -f core/coredns/coredns-configmap.yaml
kubectl rollout restart deployment/coredns -n kube-system
kubectl apply -f core/lan-dns/
kubectl apply -f core/metrics-server/metrics-server.yaml
kubectl apply -f core/platform-access/longhorn.yaml \
-f core/platform-access/vault.yaml
traefik-rbac.yaml and traefik-ingressclass.yaml are covered neither by the base layer nor by
any other command of this procedure: without the first, Traefik runs but can read no Ingress,
for want of the rights to list that resource, and therefore serves no route, with no error
showing it other than through layer 4 of the events of the pod.
core/platform-access/ is applied through named files, and not as a block: of its five files,
grafana.yaml, prometheus.yaml and alertmanager.yaml carry namespace: monitoring. No
manifest of this repository creates that namespace, and no page of this guide installs the stack
that carries it
(Observability).
Applied as a block, the directory returns three times
Error from server (NotFound): namespaces "monitoring" not found and kubectl exits 1, after
having applied the two other files all the same: the operator sees three errors, believes the
step failed, and replays a command that will fail identically. The two files named above target
namespaces that exist at this stage, longhorn-system, created by the helm install of step 4,
"Provision storage", and vault, created by the base layer. The three Ingresses of the monitoring stack are
applied once that stack is installed, which this guide does not cover.
| Check | Symptom on failure |
|---|---|
kubectl get ingress -n longhorn-system and kubectl get ingress -n vault: each returns one Ingress, longhorn.${TV_DOMAIN} and vault.${TV_DOMAIN} |
No resources found on either of the two: step 4, "Provision storage", did not create the longhorn-system namespace, or the command above was not run on the two named files. A kubectl apply -f core/platform-access/ as a block is not the cause: it lays down those two Ingresses before failing on the three others |
kubectl get svc -n traefik-ingress: EXTERNAL-IP equal to ${TV_VIP} |
<pending>: the VIP is not in the kube-vip range, or step 5, "Publish the service VIP: kube-vip", did not complete |
A route really served: curl -o /dev/null -w '%{http_code}\n' https://vault.${TV_DOMAIN}/ from a workstation that resolves that host to the VIP: 200, 302 or 307 |
EXTERNAL-IP correct but connection refused, or still the default certificate of Traefik: traefik-rbac.yaml has not been applied, Traefik sees no Ingress |
From a glibc pod, with no hostAliases: getent hosts hub.${TV_DOMAIN} answers with an A address, that of ${TV_TRAEFIK_CLUSTERIP} |
No answer, or a CNAME chain that glibc does not follow: the CoreDNS rule has not been applied yet, does not respect the answer auto format, or has already been overwritten by Talos (see the warning further down) |
kubectl get apiservice v1beta1.metrics.k8s.io: AVAILABLE=True, then kubectl top nodes returns figures for all the nodes |
NotFound, or AVAILABLE=False, or a silent kubectl top nodes: metrics-server has been applied without serving metrics. The kubectl apply succeeded, that is not what is observed here. Diagnosis at Troubleshooting, entry 13 |
The LAN resolver forwards to two hard-coded public resolvers
core/lan-dns/lan-dns.yaml ends its Corefile with forward . 1.1.1.1 8.8.8.8. That resolver
serves the LAN and the VPN, and those two addresses are its only upstream for everything that
is not *.${TV_DOMAIN}. No variable of the sheet parameterises them. Two consequences to be
settled with the customer: if their network filters outbound port 53, that resolver no longer
answers for the rest of the internet, and the workstations that take it as their main DNS
lose all resolution; and if their policy mandates their own resolvers, that line is to be
replaced by hand in the manifest. It is the same outbound constraint on port 53 as the two
DNS-01 resolvers of cert-manager, and it is validated at the same moment
(Prerequisites).
Talos used to rewrite CoreDNS: settled on 2026-09-10, but the order matters
The coredns ConfigMap carried the config.k8s.io/owning-inventory annotation, which Talos
considers its own: a control-plane restart, a Talos upgrade or a talosctl apply-config
could overwrite it and make the resolution of *.${TV_DOMAIN} disappear with no warning.
Talos no longer lays down CoreDNS: talos/patches/coredns-disabled.yaml carries
cluster.coreDNS.disabled: true, and core/coredns/coredns-stack.yaml takes over. The full
procedure is in core/coredns/README.md.
One thing not to invert on a fresh installation: remove the inventory annotation from the
six objects BEFORE applying the Talos patch. Talos prunes based on that annotation; in this
order the switch cuts nothing, and in the other it deletes the kube-dns Service, hence the
DNS of the whole cluster, until it is recreated with the same ClusterIP.
This step also laid down, until 2026-09-10, two trust-manager Bundles that distributed an
internal CA to the four zones. The CA has been removed: *.${TV_DOMAIN} is signed by Let's
Encrypt, whose chain is already in the default store of every image. There is therefore nothing
left to distribute, and nothing to mount in the workloads.
One manifest is deliberately kept out of the block above: core/monitoring/node-exporter.yaml,
the source of the host metrics of the Observability page. The base layer
does not lay it down, and it belongs to no inherited command of the procedure: this step adds it,
below, because this is the logical moment, at the same time as metrics-server. Without it, the
host metrics, CPU, memory and disk of the nodes, do not come up, and nothing signals it: the
tradeval-ressources dashboard simply shows empty series.
| Check | Symptom on failure |
|---|---|
kubectl get pods -n node-exporter: one pod per node, Running |
Namespace non-existent: the command above has not been run, and no other one of the procedure lays down that manifest |
metrics-server wakes up the ten HPA of the repository all at once
That line of the block above is not innocuous. As long as metrics-server is not there, the
ten HPA of the repository show <unknown> and do nothing; as soon as it serves metrics, they
all run, with the configuration as it is written and with no run-in period. On the reference
instance, that switch saturated the CPU of the cluster in a few minutes. The values shipped
carry the fix from that incident (maxReplicas: 3, 6 for hub-portail, cpu metric
alone), but they were calibrated on a cluster of three workers with 4 vCPU. Before applying
that manifest on a different sizing, compare the requests and the maxReplicas against real
usage, as Observability asks. Reading the result:
kubectl get hpa -A -o custom-columns=\
'NS:.metadata.namespace,NOM:.metadata.name,MAX:.spec.maxReplicas,METRIQUES:.spec.metrics[*].resource.name'
MAX at 3 everywhere except hub-portail at 6, and METRIQUES at cpu alone. A
memory metric or a much higher MAX is a divergence, not the state as shipped, and must
not be allowed to meet metrics-server.
11. Data layer¶
The six external endpoints (application database, Kafka, Redis, Elasticsearch, MinIO, SMTP) were
provisioned and checked reachable on the Prerequisites page. This step writes
their coordinates into Vault, on the prod/<service> paths that each application
ExternalSecret will consume at step 13, "Applications and exposure":
patch, never put: those paths carry several keys, and put would erase the ones that do not
appear in the command.
SMTP is consumed by three senders, and the three are configured separately:
trade-notification-svc reads SMTP__HOST, SMTP__PORT, SMTP__USESSL, SMTP__USERNAME,
SMTP__PASSWORD and SMTP__FROMEMAIL from prod/notification; Alertmanager reads its
password from prod/alertmanager, key smtp_password, the rest coming from the values file
(Observability page); Keycloak is set in the realm, at step 14,
"Keycloak and its realm".
A transactional provider (Resend, SES, SendGrid) only accepts the EXACT sender domain it has
verified: noreply@notifications.${TV_DOMAIN} passes when notifications.${TV_DOMAIN} is
the name carrying the DKIM and SPF records, and noreply@${TV_DOMAIN} is then refused with a
550, at send time only, after a successful authentication.
| Check | Symptom on failure |
|---|---|
vault kv get -mount=secrets prod/<service> does return the keys that have just been written, for each service concerned |
Path absent or key missing on re-reading: the command of the previous line failed or targeted the wrong path. That check is done on the spot, without waiting for the application pods to exist |
Open question: the exact name of the keys expected by each service
This repository ships the ExternalSecret objects (the name of the Vault path, the name of
the Kubernetes Secret produced), but not the list of the variables each microservice reads in
that Secret: those names live in the application code, outside the scope of this repository.
Confirm with the vendor the exact keys expected by each service before writing into Vault: a
missing key reproduces the symptom already documented on CONTENT_SERVICE_URL, a service
that starts normally and only fails in use.
Open question: the Keycloak database
apps/keycloak/keycloak.yaml hard-codes
jdbc:postgresql://keycloak-db-rw:5432/app as KC_DB_URL. That Service is created by no
manifest of this repository, and that seventh endpoint does not appear among the six external
data variables of the customer sheet. Without it, Keycloak never starts at step 13,
"Applications and exposure", and the gap only shows up at step 14, "Keycloak and its realm",
when trying to reach a realm that does not exist. Provision a Postgres database reachable
under that name and that port before that step (through a Kubernetes Service carrying that
exact name if the database lives outside the cluster, or through the operator that already
runs it, depending on what the customer provides), then write its credentials into Vault, at
the path the keycloak-db-external-secret ExternalSecret expects:
12. Images and registry secrets¶
${TV_REGISTRY} already carries the eleven application images, published with their exact names
and tags (Parameters page). Eleven manifests of this repository, the seven
APIs, the three portals and Keycloak, reference an imagePullSecrets named acr-pull-secret.
No manifest creates it: without it, the eleven pods go into ImagePullBackOff from the next
step onwards, however correct the registry and the tags may be.
A Secret is a namespaced object, and the workloads are spread over four zones. It must
therefore be created four times, with exactly the same content. This is the one step of the
guide where forgetting a zone shows up late: the other three start normally, and only the
fourth goes into ImagePullBackOff.
Open question: this step has never been played against a customer registry
The pull described below, the table of error strings and the announced compatibility of
--for=jsonpath are reasoned from the manifests of this repository and from the documented
behaviour of kubectl, not observed on a real registry. Not verified in this repository: to
be confirmed before a customer installation. A registry message that the table does not
foresee therefore does not say that the installation is abnormal, it says that the table is
incomplete. In that case, the Pulled line remains the only criterion that concludes, and
the exact text of the message is reported to the vendor rather than interpreted on site.
for z in web internal api auth; do
kubectl create secret docker-registry acr-pull-secret -n "$z" \
--docker-server=${TV_REGISTRY} \
--docker-username=<registry-user> \
--docker-password=<registry-password-or-token>
done
| Check | Symptom on failure |
|---|---|
The events of the throwaway pod below carry a Pulled line naming the image pulled from ${TV_REGISTRY} |
A Failed to pull image line: unauthorized or denied, the user or the password are not right; not found or manifest unknown, the image has not been published on ${TV_REGISTRY} or not under that tag; no basic auth credentials, the Secret has not been attached, either because it is missing from the zone, or because the --overrides was badly copied. No Pulling line at all, with a Pending pod: that pod has not been scheduled, and the registry is not at fault |
That the Secret exists says nothing about what it contains: a mistyped --docker-server, an
expired password or a token with no read right all produce a perfectly well formed Secret, and
eleven ImagePullBackOff one step further on. The only check that concludes is a pull:
kubectl run acr-pull-check -n api --restart=Never \
--image="${TV_REGISTRY}/trade-hub-ui-prod:latest" \
--overrides='{"spec":{"imagePullSecrets":[{"name":"acr-pull-secret"}]}}'
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/acr-pull-check -n api --timeout=180s
kubectl describe pod acr-pull-check -n api | sed -n '/Events/,$p'
kubectl delete pod acr-pull-check -n api --ignore-not-found
Three remarks on the form of those commands, each for a precise reason.
--overrides is not decorative: kubectl run attaches no imagePullSecrets of its own, and
without it the pull would take a completely different route than that of the eleven workloads.
It is the describe that answers, not the wait. The wait only bounds the waiting: it
exits with timed out waiting for the condition on a refused pull just as much as on a
container that was pulled correctly but does not start. Its exit code therefore does not tell
the two apart, and the messages of the registry only live in the events of the pod. Run the
describe in every case, including when the wait succeeds.
An empty list of events is not read in the table above: it means the describe arrived too
early, before the kubelet had written anything at all. That is the case if the wait failed
immediately, for example because the kubectl version of the workstation refuses the
--for=jsonpath syntax (accepted since 1.23). Run the describe alone again after a few
seconds, before concluding anything at all; do not read it as the absence of a Pulling line,
which is a diagnosis and not a silence.
No entrypoint is replaced, deliberately: the eleven images do not all embed a shell nor the same
command, and a --command forces you to guess. The counterpart is that the container really
starts, without its configuration Secret: it will in all likelihood exit in error. Under
--restart=Never that gives a pod in phase Failed, container Terminated, reason Error;
never a CrashLoopBackOff, which presupposes a restart policy that this pod does not have.
CreateContainerError remains possible, if the image carries no usable entrypoint.
Those states do not matter here; they come after the Pulled line, and it is that line, and
that line alone, that this step observes. Replace the image reference with any of the eleven published on
${TV_REGISTRY}, with its exact tag; the tags are heterogeneous and one image carries none
(Parameters). Delete the pod in every case: it has no --rm, and it
does not have one because a pod that never pulled its image would not be deleted by that flag
anyway.
13. Applications and exposure¶
kubectl apply -R -f apps/api/ -f apps/keycloak/ -f apps/web/
kubectl apply -f core/cloudflare/namespace.yaml
The three directories named above lay down the eleven application workloads, and those alone.
apps/ is not targeted as a block, and that is not a writing shortcut: apps/mailpit/ holds
a mail collector reserved for acceptance runs, which captures what the platform sends and
delivers none of it. It is never applied at a customer site: the platform sends through the
customer's SMTP from the first start, with the TV_SMTP_HOST details written into Vault at
step 11, "Data layer". That point is settled
with the customer before going live
(Operations).
core/cloudflare/namespace.yaml is applied on its own, and before the rest of its directory.
kubectl apply -f core/cloudflare/ visits the files in lexical order, which places
namespace.yaml last, after the three objects that live inside it: on a fresh installation, the
command returns three times namespaces "cloudflare" not found and kubectl exits 1. The three
other files are applied further down, once the tunnel is created and its credentials written into
Vault, failing which the ExternalSecret has nothing to read and the cloudflared pods stay in
ContainerCreating.
The VPN is not laid down by this procedure, and that is not an oversight
core/vpn/wireguard-incluster.yaml exists and could be tempting: do not apply it. Its first
four lines carry Non appliqué and option B ; option A = VM passerelle. The option chosen
by this repository is the gateway VM, outside the cluster, which leaves the cluster with no
inbound port at all, in the same spirit as the choice of the Cloudflare tunnel. This guide does
not describe it: it names it, and stops there.
Applied all the same, that manifest does not start. It mounts a wireguard-config Secret
that no applied manifest creates: it only exists in wg-server-secret.example.yaml and in
externalsecret-wireguard.example.yaml, both of them examples. The pod stays in
ContainerCreating indefinitely. An operator who nevertheless chooses option B provides that
Secret themselves, following core/vpn/README.md, and then leaves this procedure.
Never apply a whole directory without checking its content
A file whose name carries example or template before its extension contains placeholder
values, by convention of this repository: it is never meant to be applied as it is.
core/vpn/ contains two files of that kind, wg-server-secret.example.yaml and
externalsecret-wireguard.example.yaml. kubectl apply -f core/vpn/ without naming the file
applies them too: the operator ends up with a WireGuard Secret carrying fake keys, or an
ExternalSecret pointed at a Vault path that does not exist, with no error signalling it.
That is one more reason not to target core/vpn/ at all at this step. The same risk holds
for any other command of this page that applies a whole directory: of the directories
targeted elsewhere in this procedure, only certs/, core/kube-vip/ and core/lan-dns/ are
applied as a block, and none of the three contains a file of that kind; core/platform-access/,
core/cloudflare/ and the subdirectories of apps/ are applied through named files or named
directories. That observation dates from the writing of
this page; one command redoes it in one pass, on the repository as it stands:
find apps certs secrets/eso core/kube-vip core/lan-dns core/platform-access \
core/cloudflare core/vpn \
\( -name '*example*.yaml' -o -name '*template*.conf' \)
core/vpn/. The pattern matches example and
template wherever they sit in the name, and not only the *.example.yaml suffix: a fourth
file, switchover-example.yaml, fell outside that suffix and escaped the check, which
concluded "nothing else" having verified nothing but a naming convention. That file went away
with the internal CA, the wide pattern stays. Any other line
names a directory that must no longer be applied as a block.
The Cloudflare tunnel is created through the API, never through cloudflared tunnel login: that
command drops on the machine an origin certificate that grants full powers over the zone. The
details, token permissions, edge WAF rules and DNS records, are in core/cloudflare/README.md.
Three actions come before applying the three remaining files, and their order matters:
- Create the tunnel through the API, under the
${TV_CF_TUNNEL_NAME}name of the customer sheet, with a token carrying theAccount -> Cloudflare One Connector: cloudflared -> Editpermission, to be revoked straight afterwards. The response carries the UUID of the tunnel and acredentials_filefield usable as it is. - Write those credentials into Vault, at the
prod/cloudflaredpath thatcore/cloudflare/externalsecret-cloudflared.yamlreads:vault kv put -mount=secrets prod/cloudflared credentials.json=@credentials.json. - Carry the UUID over by hand into
core/cloudflare/cloudflared-config.yaml, at thetunnelkey, which still carries the UUID of the original tunnel: no substitution touches it (Parameters).
Only then, the three remaining files of the directory:
kubectl apply -f core/cloudflare/externalsecret-cloudflared.yaml
kubectl apply -f core/cloudflare/cloudflared-config.yaml
kubectl apply -f core/cloudflare/cloudflared-deployment.yaml
The three DNS records are published last, once the edge rules are in place: as soon as the DNS exists, the service is reachable from the internet.
| Check | Symptom on failure |
|---|---|
kubectl get pods -n internal: the eleven application workloads are there, and none of them is in CrashLoopBackOff or in ImagePullBackOff. That namespace was empty until now (step 3, "Lay down the base layer"): any failing pod therefore comes from this command |
ImagePullBackOff: acr-pull-secret missing (step 12, "Images and registry secrets"), image absent from the ${TV_REGISTRY} registry, or tag that does not match what was published. Twelve workloads instead of eleven: apps/ was applied as a block and apps/mailpit/ went through with it |
kubectl get ns cloudflare exists, and kubectl get externalsecret cloudflared-credentials -n cloudflare: SecretSynced |
namespaces "cloudflare" not found: the directory was applied as a block and the lexical order laid the namespace down last. SecretSyncedError: the prod/cloudflared path was not written into Vault (action 2 above), or the eso Vault role does not allow that path |
kubectl get deploy cloudflared -n cloudflare: READY 2/2, and kubectl -n cloudflare logs deploy/cloudflared contains Registered tunnel connection |
Pods in ContainerCreating: the cloudflared-credentials Secret does not exist, the volume does not mount. Pods Running without that line: the tunnel UUID was not carried over into cloudflared-config.yaml (action 3), and the pods attach to the original tunnel, or the HTTPS egress to the Cloudflare network is filtered |
Three public hosts, not one more
Publish only three DNS records: hub, portal and auth, and open only the corresponding
tunnel routes. Since the internal TLD was abandoned, the partitioning between what is public
and what is internal rests on nothing but that operating discipline. A record published for
an internal host exposes it to the internet, with no automatic safeguard other than check 8
of scripts/validate-install.sh.
14. Keycloak and its realm¶
The Keycloak pod has just started, at the previous step. The creation of the realm is not
automated in this repository: it is done through the administration console, at
https://auth.${TV_DOMAIN}.
No administrator account exists yet, and the logs say nothing about it
This Keycloak deployment sets neither KEYCLOAK_ADMIN nor KC_BOOTSTRAP_ADMIN_*. On recent
versions of Keycloak, the absence of those variables produces no temporary account and writes
no credentials into the logs of the pod: without the step below, the administration console
stays unreachable and the creation of the realm is blocked, not merely awkward.
Once logged in: create the ${TV_KC_REALM} realm, the clients of each application, their Valid
Redirect URIs and their Web Origins. Then the realm's mail, Realm settings > Email: without
it, password reset and address verification fail silently. Host and credentials are those of
TV_SMTP_HOST, the sender is on the domain verified by the provider (step 11, "Data layer"). On port 465,
tick "Enable SSL" and leave "Enable StartTLS" unticked; on 587, the reverse. Both ticked
together make the connection fail: the session is already encrypted when Keycloak attempts
the STARTTLS. The "Test connection" button sends to the address of the logged-in user, who
must therefore have a real one; a 550 ... not authorized to send emails from ... failure
points at the sender, not at the password.
| Check | Symptom on failure |
|---|---|
curl -sS https://auth.${TV_DOMAIN}/realms/${TV_KC_REALM}/.well-known/openid-configuration: the issuer field is exactly https://auth.${TV_DOMAIN}/realms/${TV_KC_REALM} |
issuer field different from the expected one: KC_HOSTNAME does not carry the domain of the sheet, either because the sheet applied at step 2, "Apply the customer sheet to the manifests", carried another one, or because Keycloak was reapplied from a manifest that had not been rendered; go back to step 2, "Apply the customer sheet to the manifests", then reapply Keycloak as at step 13, "Applications and exposure". issuer field absent from the answer: the ${TV_KC_REALM} realm does not exist yet, or its name differs from the sheet |
This check only covers the public path
The curl above starts from the workstation of the operator and therefore goes through
Traefik, which adds X-Forwarded-Proto: https. It reads the issuer advertised on that
path, nothing more: a correct issuer field does not say that the internal path, the one
the .NET services take to validate tokens, advertises the same value. The defect where the
two paths advertise two different issuers therefore passes this check, exactly as it passes
check 6 of the conformance recipe, and shows up elsewhere: IDX10205 in the logs and 401 on
every API call. The only test that concludes compares the two paths, it is described at
entry 4 of the Troubleshooting.
End-to-end verification¶
Each KO line corresponds to one of the checks detailed above, in order: nodes ready,
certificate issued, Traefik on the VIP, CoreDNS resolution, ExternalSecret objects
synchronised, OIDC issuer, public hosts answering, internal hosts not resolved publicly. A KO
line sends you back to the step of this page that produces it, not to a generic repair.