Troubleshooting

This page is ordered by observable symptom, never by component. An operator who is debugging does not know that the failure comes from CoreDNS: what they know is that they see a 401, a page that does not load, a pod that stays 0/1. The entry has to be findable from what they have in front of them.

Every entry carries five blocks, in this order:

Block	What it gives
Symptom	what is seen, and from where it is seen
The false lead	the wrong attribution this symptom prompts, and what rules it out
Cause	the actual mechanism
Fix	the commands, in order
Fixed when	the effect that has to be observed to conclude. Not "the command went through"

The entries below describe failures that really happened on the reference instance. The false lead is not a rhetorical precaution: it is the path the diagnosis actually took the first time, and the one an operator who sees the same thing will take again.

Finding your entry

What you see	Entry
No host answers, the browser refuses any way around it	1
Everything worked, then nothing from inside the cluster, after a restart	2
The .NET APIs answer 401, the portals work	3
IDX10205 in the logs, 401 after a successful login	4
An ExternalSecret in SecretSyncedError, or a pod that cannot find its Secret	5
Article images do not show, and not on every workstation	6
The exchange rates section is empty, and goes empty again after every restart	7
The portals still call the old domain after a rename	8
The hub images take tens of seconds to show	9
The pod count explodes, scheduling locks up	10
A host returns 404 intermittently while the pod answers	11
kubectl becomes unreachable while the applications keep serving	12
kubectl top returns nothing, the HPAs stay at <unknown>	13

Before working through an entry, ask the question that sorts half of them: is the symptom visible from a workstation, from inside the cluster, from the internet, or from all three? A fault visible from a workstation but not from a pod, or the other way round, does not have the same family of causes.

1. No host answers, and the browser offers no way around it

Symptom. Every host, public and internal alike, is unreachable over HTTPS. The browser shows a security error of the kind Did Not Connect: Potential Security Issue - this website requires a secure connection, with no button to proceed anyway. The behaviour is identical on every workstation and in every browser. On the command line, curl exits with a TLS error or returns the code 000.

The false lead. Looking at DNS or at Traefik. The name resolves, port 443 accepts the connection, Traefik serves: it simply serves its internal self-signed certificate. Then trying to "proceed anyway" in the browser, which does not exist if ${TV_DOMAIN} is on the HSTS preload list embedded in browsers, which is the case for every .app domain. The failure is therefore not a degradation with a way around it, it is a total outage.

Cause. The wildcard certificate is not Ready. Traefik's TLSStore default has nothing to present and falls back to the self-signed certificate. Upstream of that, two causes cover almost every case:

• the cloudflare-dns-token Secret is missing from the cert-manager namespace, or the token does not have the Zone:DNS:Edit and Zone:Zone:Read scopes on the customer's zone;
• the two DNS-01 arguments are missing from the cert-manager Deployment. They are not in the upstream manifests: a kubectl apply of the upstream Deployment silently wipes them. Without them, cert-manager checks challenge propagation by querying CoreDNS, which sends all of *.${TV_DOMAIN} to Traefik. The message in the challenge is then Could not determine the zone for "_acme-challenge.${TV_DOMAIN}.": ... SERVFAIL.

Fix.

kubectl get certificate -A
kubectl describe certificate wildcard-tradeval-app -n traefik-ingress
kubectl get secret cloudflare-dns-token -n cert-manager
kubectl get deploy cert-manager -n cert-manager \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'

The output of the last command must carry --dns01-recursive-nameservers-only and --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53. If they are missing, put them back with the same patch as at step 3 of the Step by step:

kubectl patch deploy cert-manager -n cert-manager --type=json -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--dns01-recursive-nameservers-only"},
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53"}
]'

This patch appends the two arguments to the end of the list: running it again on a Deployment that already carries them writes them in twice. Re-read the argument list before patching, with the command above. Then force a new attempt, without waiting for cert-manager's exponential backoff:

kubectl -n traefik-ingress delete certificaterequest \
  -l cert-manager.io/certificate-name=wildcard-tradeval-app

These object names do not carry the customer's domain

wildcard-tradeval-app and its Secret wildcard-tradeval-app-tls are Kubernetes resource names, not names derived from ${TV_DOMAIN}: the parameterisation of the customer sheet does not rewrite them. On an instance whose domain is something else entirely, the certificate still carries those names. Do not look for an object named after the customer's domain, it does not exist.

Fixed when. Three observations, in this order:

kubectl get certificate wildcard-tradeval-app -n traefik-ingress
kubectl get secret wildcard-tradeval-app-tls -n traefik-ingress
openssl s_client -connect ${TV_VIP}:443 -servername hub.${TV_DOMAIN} </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -enddate -ext subjectAltName

READY=True, the Secret exists, and above all the third command returns a Let's Encrypt issuer, an expiry date more than thirty days out and a *.${TV_DOMAIN} SAN. As long as it returns Traefik's default issuer, nothing is fixed, whatever the Certificate object says.

A browser that failed once keeps the failure in memory

A browser that hit the error has memorised HSTS for that host. It will go on refusing the connection after the repair. Use "Forget About This Site" in Firefox, or test from a private window, or from a browser that has never opened the URL. Without that, you wrongly conclude that the fix did nothing, and you go off looking somewhere else.

2. Internal resolution of *.${TV_DOMAIN} disappeared after a restart

Symptom. Everything worked. After a control-plane restart, a Talos upgrade or a talosctl apply-config, the .NET services go back to 401, while the portals keep serving pages. From a pod, getent hosts auth.${TV_DOMAIN} returns nothing, or returns a public address instead of ${TV_TRAEFIK_CLUSTERIP}.

The false lead. Believing it is a certificate or a Keycloak problem. Nothing says "DNS": the .NET service logs an authentication failure, not a resolution failure. And the platform stays perfectly reachable from a workstation for the whole outage, since a workstation queries the LAN resolver, not CoreDNS. The failure therefore looks like it only touches certain services, and only from the inside.

Cause. The rewrite stop ... answer auto block, which sends *.${TV_DOMAIN} to the ingress Service, has disappeared from the Corefile. Until 2026-09-10 the usual cause was Talos, which owned the ConfigMap through its config.k8s.io/owning-inventory annotation and rewrote it. That is no longer the case: Talos no longer lays down CoreDNS. So look instead at an unlucky kubectl apply or a regression in core/coredns/coredns-stack.yaml.

Fix.

kubectl get configmap coredns -n kube-system -o yaml | grep -A4 'rewrite stop'
kubectl apply -f core/coredns/coredns-configmap.yaml
kubectl rollout restart deployment/coredns -n kube-system

Fixed when. From a glibc pod, and not from busybox, whose getent applet does not exist:

kubectl run dns-check --rm -i --restart=Never --image=debian:stable-slim \
  -- getent hosts auth.${TV_DOMAIN}
kubectl run dns-check --rm -i --restart=Never --image=debian:stable-slim \
  -- getent hosts kubernetes.default.svc.cluster.local

The first must return an A address equal to ${TV_TRAEFIK_CLUSTERIP}. The second is the non-regression check: a malformed Corefile breaks all cluster resolution, and this second call is the only one that shows it. The pods recover on their own as soon as resolution answers; if 401s persist beyond a few minutes on a .NET service, restart it, since its OIDC client caches the discovery document.

This failure no longer comes back after a Talos operation, since 2026-09-10

It replayed on every control-plane reboot, version bump or talosctl apply-config, because Talos owned the ConfigMap. That is no longer the case: cluster.coreDNS.disabled: true sits in talos/controlplane.yaml and in the configuration of the three nodes, and core/coredns/coredns-stack.yaml carries the six objects. Reapplying the repository's machine configuration is therefore safe. The check that proves it: talosctl get manifests no longer returns any coredns entry. The repair above stays valid for the other causes, an unlucky kubectl apply or a regression in the versioned file, except that it now plays out on coredns-stack.yaml. Two Prometheus rules in the dns group report the disappearance of the rewrite block within five minutes.

3. The .NET services answer 401 on every call, the portals work

Symptom. Every authenticated call to the .NET APIs fails with 401. The Next.js portals work, users log in to Keycloak without trouble. Keycloak is healthy: 1/1, no restart.

The false lead. Suspecting Keycloak, or the OIDC configuration of the APIs. Two things feed the error. First, /api/auth/profile returns 401 without a token, which is the normal behaviour: the symptom looks like an API misconfiguration. Second, every endpoint tested in isolation answers 200. Checking each one separately proves nothing here, and wastes time.

Cause. CoreDNS returns a CNAME chain instead of an A record. glibc, the C library of the .NET containers, refuses to follow it: the TLS and OIDC call never leaves. musl, used by the Alpine and Node images, is less strict and resolves anyway. Hence a failure that looks like it only touches certain services, and the fact that the Node portals work is what points at it.

A second mechanism used to exist and produced the same 401: trust in an internal CA, handed to the workloads by trust-manager as ConfigMaps mounted through SSL_CERT_FILE. It was removed on 2026-09-10 together with the CA itself, *.${TV_DOMAIN} being signed by Let's Encrypt, present in the default store of every image. A cluster installed from this repository has no internal CA, no trust-manager and none of those ConfigMaps; on an older cluster, a kubectl get bundles that still answers something flags a leftover to remove, not a cause to treat.

Fix. Check the resolution from a .NET pod, not from the workstation:

kubectl -n api exec deploy/trade-valuation-svc -- getent hosts auth.${TV_DOMAIN}

If getent does not return ${TV_TRAEFIK_CLUSTERIP}, it is DNS: handle entry 2.

Fixed when. The logs of the .NET service are of no use here: they log an authentication failure, never its cause. The two useful observations are:

• getent hosts auth.${TV_DOMAIN} from a .NET pod returns ${TV_TRAEFIK_CLUSTERIP};
• a real journey: a login followed by an API call from a portal returns something other than a 401.

A 401 that persists while those two points are true is no longer this fault. Move on to entry 4.

4. Every authenticated call fails with IDX10205

Symptom. IDX10205 in the logs of the .NET services, and 401 on every API call right after a successful login. The user logs in, obtains a token, and that token is refused everywhere.

The false lead. Believing Keycloak is down. It is perfectly healthy: 1/1, no restart, and the only error line in its logs is an unrelated client disconnection (ClosedChannelException). The discovery document answers 200 when called from the workstation, and it answers 200 when called from the cluster. The two answers simply do not agree with each other, and no test that takes them separately can show it.

Cause. KC_HOSTNAME carries the short form (auth.${TV_DOMAIN}) instead of the full URL form. Keycloak accepts both, and they do not behave the same:

Form of KC_HOSTNAME	Behaviour
auth.${TV_DOMAIN}	the host name is pinned, the scheme is inferred from each request
https://auth.${TV_DOMAIN}	scheme, host and port pinned, invariant issuer

With KC_PROXY_HEADERS=xforwarded, the inference relies on X-Forwarded-Proto. Traefik adds it, a direct call to the keycloak Service does not have it. The same server therefore announces https://... on the public path and http://... on the internal path. The tokens handed to browsers carry iss as https; the APIs, which validate against the internal document, expect http; the iss does not match and everything is rejected.

Second cause, and this is the one an operator will actually meet: after a domain rename, the Vault keys that carry the issuer (AUTH_KEYCLOAK_ISSUER, NEXTAUTH_URL, NEXTAUTH_URL_INTERNAL, the OIDC authority of the .NET services) were not changed at the same time as KC_HOSTNAME. Those keys change together or not at all.

The short form is not reachable on a fresh installation

apps/keycloak/keycloak.yaml already pins KC_HOSTNAME to the full URL form, and render-config.sh only substitutes the domain in it. The first cause above is therefore the history of the fault, not its most likely way in: it only becomes reachable again if someone shortens that value by hand. On a conformant installation, check the Vault keys of the second cause first. The mechanics of the first are still worth knowing, because they are what explains why comparing the two paths is the only test that concludes.

Fix. The test that settles it compares the two access paths to the same endpoint:

curl -sS https://auth.${TV_DOMAIN}/realms/${TV_KC_REALM}/.well-known/openid-configuration \
  | grep -o '"issuer":"[^"]*"'

kubectl -n auth port-forward svc/keycloak 8080:80 &
curl -sS http://127.0.0.1:8080/realms/${TV_KC_REALM}/.well-known/openid-configuration \
  | grep -o '"issuer":"[^"]*"'

The port-forward adds no X-Forwarded-Proto header: it is exactly the path the APIs take. If the two outputs differ, fix KC_HOSTNAME to the full URL form in apps/keycloak/keycloak.yaml, re-apply, then restart the .NET services:

for z in web internal api auth; do kubectl rollout restart deployment -n "$z"; done

That restart is not optional: the .NET OIDC client caches the discovery document, with a default refresh of the order of twelve to twenty-four hours. Without it, the APIs go on validating against the old issuer long after the fix, and the symptom looks untouched.

Fixed when. The two commands above return exactly the same string, https://auth.${TV_DOMAIN}/realms/${TV_KC_REALM}, and a real journey of a login followed by an API call returns something other than a 401.

The conformance recipe does not detect this fault

Check 6 of Validation compares the issuer announced on the public path to the expected one. On this precise fault, the public path announces exactly the right value: the recipe passes while the APIs, which query the internal path, receive the other one. Comparing the two paths is the only test that concludes.

5. An ExternalSecret stays in SecretSyncedError

Symptom. kubectl get externalsecret -A shows one or more objects in something other than SecretSynced. Downstream, the pods that consume the Secret concerned stay in CreateContainerConfigError if it does not exist at all, or start normally with stale values if it still exists.

The false lead. Two of them, equally expensive. The first is to conclude that External Secrets Operator is broken, when a single Vault path is: the number of objects in error is the diagnosis, read the list before touching anything. The second is to fix the Kubernetes Secret by hand, through kubectl edit or kubectl create secret: creationPolicy: Owner rewrites it at the next synchronisation, a minute later, without saying anything. The only write point is Vault.

Cause. It is read in the shape of the output:

What kubectl get externalsecret -A shows	Cause	Where to confirm
All in error	the ClusterSecretStore is not ready, or Vault re-sealed itself after a restart	kubectl get clustersecretstore vault-store must read READY=True; kubectl exec -n vault vault-0 -- vault status must return Sealed: false
One or two named objects	the Vault path is missing or misspelled	kubectl describe externalsecret <name> -n <ns> names the path; vault kv get -mount=secrets prod/<service> says whether it exists
No object returned at all	ESO is not installed, its CRDs are missing
All SecretSynced, but a service misbehaves	an enumerated data: instead of dataFrom: extract: a forgotten key is silent, the pod starts and only fails in use	compare the keys of the produced Secret with those of the Vault path

Fix. Depending on the cause: unseal Vault (page Operations), create the missing Vault path with vault kv patch, or rewrite the ExternalSecret in the mandatory form (dataFrom: - extract:, creationPolicy: Owner, deletionPolicy: Retain). In every case, finish with a restart of the Deployments concerned: environment variables are frozen at container start, and a corrected Secret does not reach a pod that is already running.

kubectl rollout restart deployment/<name> -n <zone>

vault kv put on a shared path breaks a service with no error

put replaces the whole content of the path: any key absent from the command disappears, and therefore disappears from the Secret at the next synchronisation. The ExternalSecret stays SecretSynced, since the path still exists: it has merely got thinner. deletionPolicy: Retain does not protect against this case. The running pods keep running on their frozen variables, and the failure appears at the next restart, decoupled from the command that caused it. On any prod/<service> path, it is vault kv patch.

Fixed when.

kubectl get externalsecret -A
kubectl get secret <name> -n <ns> -o jsonpath='{.data}' | tr ',' '\n' | cut -d'"' -f2
kubectl get pods -n <zone> -l app=<name>

Every object in SecretSynced, and the target Secret really carries all the expected key names, and the pod that consumes it is 1/1 after restart. A SecretSynced on its own does not prove that the Secret carries the right keys: it is the count of key names that proves it. The command above only prints names, never values: never replace it with a -o yaml.

6. Article images do not show, 403 depending on the workstation

Symptom. The news images show on one workstation and not on another. The page itself is served normally. In the content-service logs: You are not allowed to perform this action., several dozen times a day.

The false lead. Going after the workstation, because the difference between workstations is real and obvious. It is a consequence, not the cause. The reflex that puts things back the right way round: a symptom that is identical on hub and on portal points at the source, that is the data or the content-service; a symptom that differs by workstation or by browser points at the resolution of *.${TV_DOMAIN} on the workstation. Here the two combine, and it is the source that has to be fixed.

Cause. SERVER_URL (or PAYLOAD_PUBLIC_SERVER_URL) is set on the content-service. Payload then returns absolute media URLs. The browser's <img> tag calls the content-service directly, with no API key and no cookie, and access to the media requires an authorisation: an anonymous read gets a 403. Absent, Payload emits relative URLs, which the browser resolves from the portal's origin; they therefore reach the /api/media proxy of hub and of portal, which adds the API key server-side.

Fix.

kubectl exec -n api deploy/trade-content-service-svc -- \
  env | grep -E 'SERVER_URL|PAYLOAD_PUBLIC'
vault kv patch -mount=secrets prod/content-service SERVER_URL=
kubectl rollout restart deployment/trade-content-service-svc -n api

The variable must stay empty, never filled in. patch and not put: this path carries all the other keys of the service.

Fixed when.

• On the new pod, kubectl exec ... env | grep -E 'SERVER_URL|PAYLOAD_PUBLIC' returns nothing, or returns an empty value. The environment of a pod that has already started is frozen: measuring on the previous pod proves nothing.
• In the browser's network inspector, on a page that carries images, the media requests go to https://hub.${TV_DOMAIN}/api/media/... and no longer to https://content.${TV_DOMAIN}/..., and they return 200.
• The 403 counter stops going up. That is checkable, and it is exactly what has to be looked for in the logs:

kubectl logs -n api deploy/trade-content-service-svc --since=10m \
  | grep -c 'not allowed to perform this action'

Expected: 0.

SERVER_URL is used elsewhere too

It feeds the links in emails, the preview and certain absolute links of the Payload administration console. Check those three uses after the removal. If one of them breaks, the fix is in the application, on the portals side, never the return of the variable: putting it back brings the 403s back unchanged.

7. The exchange rates section is empty, and goes empty again at every restart

Symptom. The exchange rates section is empty on the hub and on the portal, at the same time. In the database, every rate carries exchangeStatus = "expired" while its validTo is in the future, that is while they are still valid. Characteristic signature: expiredBy is NULL, which means a system write and not a human action. And the symptom comes back after every pod restart, including after a repair of the data.

The false lead. Two of them. The first is to repair the data and consider the matter closed: as long as the application fix is not deployed, every restart re-corrupts the rates, and a repair made before that deployment is wiped at the next restart. The second is to look in the logs: the seed's catch hides the uniqueness error under a message along the lines of "already exists, skipping". Nothing appears in the logs. It is that silence that makes the failure expensive.

Cause. PAYLOAD_AUTO_SEED is set. The full seed then runs at every pod start, with no guard and no lock. The chain: the seed recreates a rate with the same validFrom for each currency; a hook expires the existing overlapping rate, in a separate transaction that commits immediately; the create then fails on the unique index and is rolled back, but the expiry is already committed. What is left is an expired row with no replacement, repeated across every currency. An HPA that starts several pods amplifies the corruption, each pod launching its own seed.

Fix.

The safe state is the absence of the key, not an empty value. And vault kv patch only knows how to add or replace a key, never to remove one: removal goes through a deliberate rewrite of the path, from its content re-read just before.

kubectl exec -n api deploy/trade-content-service-svc -- env | grep PAYLOAD_AUTO_SEED
vault kv get -format=json -mount=secrets prod/content-service | jq '.data.data' > cs.json
# remove the PAYLOAD_AUTO_SEED line from cs.json, and nothing else
vault kv put -mount=secrets prod/content-service @cs.json
vault kv get -mount=secrets prod/content-service
rm -f cs.json
kubectl rollout restart deployment/trade-content-service-svc -n api

This is the only legitimate put on a shared path, and it is only legitimate because the file carries all the other keys, re-read from the path itself one command earlier. The vault kv get that follows the put is not decorative: it observes that the other keys are still there. The intermediate file contains secret values in the clear: delete it, and never write it into the git clone.

Then repair the data, and only the rows expired by the system. expired_by_id IS NULL tells them apart from an expiry decided by an administrator, for instance the withdrawal of a wrong rate, which must on no account be put back into service. Run the read-only query first, then the same WHERE as an UPDATE, inside a transaction:

SELECT id, currency_id, valid_from, valid_to FROM exchange_rates
WHERE exchange_status = 'expired' AND expired_by_id IS NULL
  AND valid_from <= NOW() AND valid_to >= NOW()
  AND NOT EXISTS (
    SELECT 1 FROM exchange_rates e2
    WHERE e2.currency_id = exchange_rates.currency_id
      AND e2.administration_id = exchange_rates.administration_id
      AND e2.exchange_status = 'active' AND e2.id <> exchange_rates.id
      AND e2.valid_from <= NOW() AND e2.valid_to >= NOW());

Fixed when. The line Running auto-seed... no longer appears at start-up. It is the only positive signal the logs give on this fault:

kubectl logs -n api deploy/trade-content-service-svc --since=10m | grep -i 'auto-seed'

Expected: no output. Then the decisive check, the only one that proves the fix and not the repair: restart the pod a second time and check that the section stays populated on both portals. A data repair followed by a restart with no corruption is the proof; the repair on its own proves nothing.

8. The portals still call the old domain after a rename

Symptom. After a change of ${TV_DOMAIN}, with the customer sheet replayed and the manifests re-applied, the portal pages load, but the calls issued by the browser go to the old domain: failed requests in the browser console, a login that redirects to the old auth host. On the Kubernetes side, nothing: the pods are Running, Ready, with no restart and no error in the logs.

The false lead. Redeploying, running a kubectl rollout restart, changing an environment variable on the Deployment, patching the value in Vault. None of these actions has the slightest effect, and the absence of effect sends you looking elsewhere when the cause is identified from the first attempt.

Cause. The NEXT_PUBLIC_* variables are resolved at the moment the image is built, never at container start. Nothing at runtime can change them. Corollary: a kubectl rollout undo does not repair anything either, it restores an image that also carries the old domain.

Fix. Rebuild the three portal images with the right values, publish them to ${TV_REGISTRY} under the expected names and tags, then redeploy. That is the only fix. In the same move, check the values that do live at runtime and only need a restart: AUTH_KEYCLOAK_ISSUER, NEXTAUTH_URL, NEXTAUTH_URL_INTERNAL, the OIDC authority of the .NET services, the *_API_BASE_URL, CONTENT_SERVICE_URL, ALLOWED_ORIGINS. And outside the cluster: the Valid Redirect URIs and Web Origins of the Keycloak realm, as well as the DNS resolution of the workstations.

Fixed when. Two observations, neither of which goes through the state of the pods:

kubectl get deploy hub-portail -n web \
  -o jsonpath='{.spec.template.spec.containers[0].image}'; echo

The tag that is read must be different from the one before. Most application images carry latest: with an identical tag, nothing proves that the new content was pulled, and the pod can serve the old image indefinitely. Use a distinct tag per version, or force imagePullPolicy: Always.

Then, in the browser, with the page reloaded without cache: the network inspector shows no request to the old domain, and a complete login journey succeeds. A Running and Ready pod proves nothing here, and that is exactly why this fault survives a redeployment.

9. The hub images take tens of seconds to show

Symptom. On the hub home page, an image can take up to some thirty seconds before the first byte, internally as much as publicly. The static files of the same page come back in a few tens of milliseconds. No restart, no OOMKilled, no Kubernetes event, no error in the logs. The pod is Running and Ready for the whole duration of the problem.

The false lead. Suspecting the ingress, a WAF or the Cloudflare tunnel, especially if one of them was turned on the same day. The witness that rules them all out in a single measurement: a static file from the same host answers in 25 ms. Comparing an instrumented path with a witness path avoids looking in the wrong place. Second false lead, the network: curl -w shows time_starttransfer almost equal to time_total, so time spent before the first byte, so computation and not transfer.

Cause. A CPU ceiling that is too low for a bursty compute load. The portal resizes and re-encodes its images on the fly (/_next/image); with limits.cpu: 250m, the cgroup spreads over four times as much time a job that needs a whole core for a few hundred milliseconds. The container is not killed, it is slowed down. Unlike a memory limit, which kills the container and leaves a trace (OOMKilled, exit 137), CPU throttling is invisible in the state of the Kubernetes objects. Three factors amplify it: a source file of several mebibytes, eight variants requested in several formats depending on the browser, and a cache that is ephemeral and per-replica, emptied at every rollout.

Fix. Raise the portal's CPU limit to 2000m and its memory limit to 512Mi, the AVIF encoding buffers being greedy. Leave the requests unchanged: nothing more is then reserved on the node. Do it on all three portals, not just the one that was reported: same runtime, same latent fault, and a portal behind authentication does not show its slowness from the outside, one more reason not to wait for a user to report it. The underlying fix is in the application: make the source files lighter, a 1920 px WebP of equivalent quality weighing of the order of 200 KB against several mebibytes.

Fixed when. Two measurements from the same command, one before the change, the other after, both cold, that is right after a rollout restart that empties the cache:

kubectl rollout restart deployment/hub-portail -n web
kubectl rollout status deployment/hub-portail -n web
curl -sS -o /dev/null -w 'starttransfer=%{time_starttransfer} total=%{time_total}\n' \
  'https://hub.${TV_DOMAIN}/_next/image?url=%2Fimages%2Fhero-slide-1.png&w=1920&q=75'

It is the ratio between the two measurements that concludes, not an absolute value. Going from 250m to 2000m lifts a spreading factor of the same order: an unchanged time, or one that only gains a third, means the ceiling has not moved. Two reasons to transpose no figure: the customer's hardware is unknown, and curl does not announce Accept: image/avif, so it does not receive the AVIF variant, the most expensive one and the one a browser asks for. The curl measurement is a relative indicator, never the time the user sees. A second call, with a warm cache, proves nothing: it answers fast in both cases.

Complementary check, the one that proves the new limit is really in force: during the request,

kubectl top pod -n web -l app=hub-portail

must show a consumption above 250m. If it caps at exactly 250m, the modified manifest has not been applied, whatever the latency observed.

No dashboard and no alert would have reported this failure. Monitor latency, not only availability: it is the tradeval-red dashboard of Observability that carries it.

10. The pod count explodes and scheduling locks up

Symptom. Shortly after installing metrics-server, the pod count jumps, worker CPU climbs to 98-99 %, pods stay in Pending with 0/6 nodes are available: 3 Insufficient cpu, updates in progress lock up, and intermittent 404s appear on the public hosts.

The false lead. Concluding that metrics-server broke something, and uninstalling it. It broke nothing: it made an erroneous configuration that had been dormant executable. Until then the HPAs displayed <unknown> and had never done anything, so their configuration had never been confronted with reality. Uninstalling it hides the fault again instead of fixing it, and the next operator will find it intact.

Cause. An HPA based on memory, with a request lower than the usage at rest, scales up indefinitely. The .NET services consume markedly more than their request, so they are structurally above target: adding replicas does not reduce the memory per pod of a garbage collected runtime, which occupies whatever it is given. The system climbs to maxReplicas whatever the real load. Aggravating factor, a very high maxReplicas on a cluster of three workers.

Fix. Start by situating what you are looking at: the ten HPAs shipped already carry the fix for this incident. All are at maxReplicas: 3, except the one for hub-portail at 6, all have only a cpu metric, no memory metric, and every manifest carries the explanation at the place where the value is read.

kubectl get hpa -A -o custom-columns=\
'NS:.metadata.namespace,NOM:.metadata.name,MAX:.spec.maxReplicas,METRIQUES:.spec.metrics[*].resource.name'

What is read in that output decides what comes next:

What the output shows	What it means
MAX at 3, or 6 for hub-portail, and METRIQUES at cpu alone	the shipped state. The problem is elsewhere, this is not that incident
A memory metric, or a much higher MAX	a divergence, not the shipped state. Someone modified the HPA in the cluster, or an older manifest was re-applied

In the second case, re-apply the manifests of the repository, which carry the right values. If the divergence comes from the repository itself, fix it in both: otherwise a later kubectl apply re-applies the old values. General rule to keep beyond this case: never enable a memory HPA on a garbage collected runtime (.NET, JVM, Node) without having checked that the request reflects the usage at rest.

Fixed when.

kubectl get hpa -A
kubectl get pods -A --field-selector=status.phase=Pending
kubectl top nodes

The TARGETS column no longer carries a memory metric, REPLICAS is back to nominal and stays there over several minutes, no pod is Pending, and worker CPU has come back down.

A quiet HPA is not necessarily a repaired HPA

An HPA that displays <unknown> in TARGETS is not stabilised, it is blind: the metrics source does not answer it. kubectl top nodes must return values. A cluster that is calm because its autoscalers see nothing is exactly the state that preceded this incident, and it is a failure in its own right: entry 13.

11. A pod answers but Kubernetes declares it unavailable, and the host returns 404

Symptom. An ingress host returns 404 intermittently, internally as much as through the tunnel. The corresponding pod is Running but 0/1, sometimes for hours, without ever restarting. From inside the container, the health path answers 200.

The false lead. Suspecting Cloudflare or the ingress, since the symptom is a 404 on a route. The identical behaviour internally and through the tunnel rules out the tunnel in a single measurement. Second false lead, more stubborn: measuring the health path from inside the container, observing 200, and concluding that the pod is healthy. It is. That is not the path the probe queries.

Cause. The readiness probe points at a path covered by the authentication middleware. That path answers with a redirect to the login page, and the kubelet follows HTTP redirects: the probe therefore measures the render time of the login page, which varies and is regularly above the timeoutSeconds granted. The pod loses its readiness, it is removed from the Service, and Traefik answers 404 for want of an endpoint. A probe that fails one time in five produces a pod that oscillates, far harder to diagnose than a clean outage.

The revealing signal is not in the pod logs, which say nothing: it is in the kubelet events.

kubectl describe pod <name> -n <zone> | sed -n '/Events/,$p'

Look for a line Readiness probe failed: Get "http://<ip>:3000/api/auth/signin?callbackUrl=%2Fhealth": context deadline exceeded. The redirected URL that appears in the message is the complete diagnosis: the probe is not querying what you think.

Fix. Point the readiness and startup probes at a path that answers 200 without redirecting (/api/health on the Next.js portals, /health and /health/ready on the .NET side), and raise timeoutSeconds to 5 s for a server-rendered Next.js application. A probe path must never redirect, nor authenticate, nor depend on an external service: otherwise it does not measure the health of the pod, but that of the chain it crosses.

One shipped manifest does not yet respect the second half of this rule

The three portals do point at /api/health, which does not redirect: the first half of the fault is fixed everywhere. The second is not. apps/web/portal/portal-portail.yaml and apps/web/internal/internal-portail.yaml carry timeoutSeconds: 5 on their readiness probe; apps/web/hub/hub-portail.yaml carries none, so it takes the Kubernetes default, 1 second, exactly the value the incident blames, for a server render measured at 40-80 ms with peaks at 2.3 s. hub-portail is the only public portal without authentication, so the one whose flapping is immediately visible from the outside. Align its probe to 5 s before going into service.

Fixed when. An intermittent fault is not observed in a single call.

kubectl get endpoints <service> -n <zone>
kubectl get pods -n <zone> -l app=<name> -w    # watch for at least two minutes
for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{http_code} ' https://portal.${TV_DOMAIN}/
done; echo

The pod IP is indeed in the endpoints, and it is what was missing when Traefik answered 404. The pod stays 1/1 for the whole observation, without oscillating. And the twenty requests return the same success code twenty times, without a single 404 and without a single 000: that last one is not a response code, it is curl that did not get through, and twenty 000 are therefore not twenty identical codes. This is the only place in the guide where curl deliberately stays silent, so that the twenty codes remain readable: faced with a 000, replaying a single request as curl -sS -o /dev/null -w '%{http_code}\n' https://portal.${TV_DOMAIN}/ gives the reason the loop was keeping quiet. Finally, kubectl describe pod no longer shows any recent Readiness probe failed event.

12. kubectl becomes unreachable and the hosts drop intermittently

Symptom. The ingress hosts become unreachable intermittently, then kubectl stops answering completely while the applications keep serving. The kube-apiserver of two of the three control-planes are in CrashLoopBackOff.

The false lead. Believing the cluster lost. The data plane survives the loss of the API: the workloads keep serving throughout. And diagnosis stays possible by aiming directly at the surviving control-plane:

kubectl --server=https://<ip-of-the-surviving-control-plane>:6443 get pods -A

This is the command to know before needing it: with a kubeconfig endpoint pointed at a machine that is down, everything looks dead.

Cause. Turning on a level 2 WAF (Coraza) on the Traefik Ingresses, through three distinct mechanisms that chained together:

1. An invalid middleware breaks routing, it does not let it through. Mistyped, the plugin does not merely sit inactive: the routers that reference it become inoperative and the public hosts drop. A misconfigured WAF is a failure, not an absence of protection.
2. Memory consumption is driven by concurrency, the WASM host instantiating one module per concurrent request. A single instance measured at 762 MiB against a baseline of about 80 MiB.
3. Raising Traefik's memory limit while it tolerates the control-plane taints saturates the control-planes, whose kube-apiserver is then killed. That is the mechanism that turns an ingress failure into the loss of the API.

Fix. The level 2 WAF is not viable in this configuration and has been removed from every Ingress: do not turn it back on. Traefik must never be schedulable on a control-plane: pin it to the workers with a nodeSelector and remove the tolerations. Keep the public Ingresses on the websecure entrypoint alone, with port 80 redirecting globally to 443. And size the control-planes at 4 GiB: at 2 GiB, they were running at 100-117 % memory usage under normal load, with six or seven apiserver restarts accumulated over sixty-seven days.

Fixed when.

kubectl get pods -n kube-system -l component=kube-apiserver -o wide
kubectl get pods -n traefik-ingress -o wide
kubectl top nodes

The three apiservers Running with a RESTARTS counter stable over several minutes: a counter that keeps climbing means the nodes are still saturated. No Traefik pod on a control-plane node. And control-plane memory well under 100 %. Last observation, the simplest one: kubectl get nodes answers from the normal kubeconfig endpoint, and no longer only when aiming at a specific machine.

Note the three control-plane addresses before needing them

If the kubeconfig endpoint designates a single machine, losing that machine removes all administration access, including for diagnosis. Record the addresses of the three control-planes in the customer's operations file, next to the Vault unseal keys. That information cannot be found again once kubectl no longer answers.

13. kubectl top returns nothing and the HPAs stay at <unknown>

Symptom. kubectl get hpa -A shows <unknown> in the TARGETS column on every HPA, and kubectl top nodes returns error: Metrics API not available or returns nothing. Nothing else complains: the application pods run, the hosts answer, and the conformance recipe passes, since it does not look at the metrics source.

The false lead. Concluding that this is the normal state, on the grounds that "the HPAs are inert as long as metrics-server is not installed". That was true before the installation. The guide lays down core/metrics-server/metrics-server.yaml at step 10 of the Step by step: after a conformant installation, <unknown> means the source is not serving metrics, not that it is absent by design. The second false lead is to live with it, because nothing breaks as long as you look for nothing.

Cause. Two of them, and two commands separate them. The v1beta1.metrics.k8s.io APIService is declared in the same file as the Deployment, and that file is part neither of the base layer nor of any other kubectl apply of the procedure: its absence and its presence therefore say the same thing as those of the pod.

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get pods -n kube-system -l k8s-app=metrics-server

What the two commands return	Cause
Error from server (NotFound) on the apiservice, and no pod	1. Step 10 was not played in full, the manifest was never applied. This is the most frequent case, that manifest not being part of the base layer
The apiservice exists with AVAILABLE=False, with a pod at 0/1 or pending	2. The pod is not serving metrics. FailedDiscoveryCheck on the apiservice is the consequence of that state, not a third lead: it stays False as long as the pod is not 1/1

On cause 2, the known pattern on Talos is the refusal to validate the kubelet serving certificates, which carry no IP SAN: x509: cannot validate certificate for <address> because it doesn't contain any IP SANs. The manifest of the repository already carries the --kubelet-insecure-tls flag that lifts this case; that message in the logs therefore means that the Deployment in place is not the one from the repository, typically because an upstream manifest was applied over it.

What this failure costs, and why it is classified here. It silences four checkpoints given as decisive elsewhere in this guide: the kubectl top pod of entry 9, the only check that proves a new CPU limit is in force; the kubectl top nodes of entries 10 and 12; and the "Control-plane memory" line of the daily monitoring table of Operations. An operator who believes they are monitoring is monitoring nothing. It also immobilises the ten HPAs of the repository, without the slightest message.

Fix.

kubectl logs -n kube-system -l k8s-app=metrics-server --tail=50
kubectl apply -f core/metrics-server/metrics-server.yaml

The same kubectl apply repairs both causes: it lays down the missing Deployment in the first case, and in the second it restores the flag that the upstream manifest does not carry. On cause 1, the logs have nothing to say, there is no pod; it is on cause 2 that they serve, and the line to look for there is the one that contains IP SANs.

Fixed when.

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl top nodes
kubectl get hpa -A

AVAILABLE=True on the apiservice, kubectl top nodes returns figures for every node, and the TARGETS column of the HPAs carries a percentage instead of <unknown>. Allow up to a minute after the pod starts: the first collection window is not instantaneous. A kubectl top nodes that returns only some of the nodes is not repaired, it is partially blind.

Do not make the metrics available without having read entry 10

Making them available wakes the ten HPAs of the repository all at once, with their configuration as it is written, with no run-in period. That is what saturated the CPU of the reference cluster in a few minutes. The shipped values already carry the fix for that incident (entry 10): check that they have not been modified since before going on.

14. Two Longhorn DaemonSets restart endlessly on Talos

longhorn-iscsi-installation and longhorn-preflight-installer stay in Init:CrashLoopBackOff, with a restart counter climbing indefinitely. On this cluster it had reached 24,000 per pod, over 85 days.

kubectl logs -n longhorn-system <iscsi-pod> -c iscsi-installation --tail=20

The message is unambiguous:

nsenter: can't execute 'apk': No such file or directory
nsenter: can't execute 'rc-update': No such file or directory
nsenter: can't execute 'service': No such file or directory

These two DaemonSets install open-iscsi on the host with Alpine's package manager. Talos is immutable: it has no apk, no rc-update, no service. They can structurally never succeed, and no setting will help them.

They are also unnecessary. The workers ship the Talos iscsi-tools extension (Prerequisites), and Longhorn confirms it itself:

kubectl get nodes.longhorn.io -n longhorn-system -o custom-columns=\
'NODE:.metadata.name,PACKAGES:.status.conditions[?(@.type=="RequiredPackages")].status'

RequiredPackages=True means the prerequisite is already satisfied.

The fix is to delete them:

kubectl delete daemonset longhorn-iscsi-installation -n longhorn-system
kubectl delete daemonset longhorn-preflight-installer -n longhorn-system

Check before deleting that they carry no Helm label (app.kubernetes.io/managed-by): applied by hand, nothing recreates them. Managed by a release, they must instead be disabled in the values, otherwise they come back at the next upgrade.

What is not a failure

Two anomalies appear on a conformant installation and do not have to be fixed. Mistaking them for a fault wastes time, and fixing them "just in case" creates a real one.

What is seen	Why it is normal
An empty Grafana, with no datasource, in the observability namespace	Legacy stack, laid down by the base layer, which receives nothing. The one that carries the data is the monitoring stack (Observability)

The list stops there, and that is deliberate. Two symptoms in particular are not on it and must never join it:

• An ImagePullBackOff in one of the four application zones is never normal. It means that acr-pull-secret was not created, that the image was not published to ${TV_REGISTRY}, that its tag does not match what was published, or that outbound HTTPS on port 443 is filtered from the nodes (Prerequisites), a cause no check of preflight.sh tests. The gesture that separates this fourth cause from the three others is kubectl get pods -A | grep -E 'ImagePullBackOff|ErrImagePull': the first three only touch the four application zones, whereas a closed port 443 also blocks the upstream images of the base layer, in kube-system, traefik-ingress or cert-manager. The kubectl describe pod confirms it, a timeout or an unresolved name on the registry host rather than a 401 or a not found. No pod of that namespace is supposed to fail to pull its image.
• <unknown> in kubectl get hpa is not normal either after a conformant installation: it is a failure, and it silences four checks of these two pages. See entry 13.

Three operating interfaces exposed without authentication, on the other hand, are not a benign anomaly: it is a known exposure, to be handled before going into service, according to the recipe of Observability.

Collecting the material before asking for help

Before any restart, and in this order. A rollout restart run too early destroys the evidence of the fault you are trying to have analysed.

scripts/validate-install.sh --env scripts/install.env > diag-validate.txt 2>&1
kubectl get events --sort-by=.lastTimestamp -A     > diag-events.txt
kubectl get pods -A -o wide                        > diag-pods.txt
kubectl get certificate,order,challenge -A         > diag-certs.txt
kubectl get externalsecret -A                      > diag-secrets.txt
kubectl get clustersecretstore vault-store -o wide >> diag-secrets.txt
kubectl describe pod <pod-concerned> -n <ns>       > diag-describe.txt
kubectl logs deploy/<service-concerned> -n <ns> --tail=500 --timestamps > diag-logs.txt
kubectl logs deploy/<service-concerned> -n <ns> --previous --tail=500 > diag-logs-precedent.txt 2>/dev/null

Four precautions that decide whether what you send can be used:

• Events first. kubectl get events only carries one hour of history by default. It is the first thing to collect, and the only one that contains the probe message of entry 11.
• --previous is the only way to read the logs of a container that has already restarted. After a second restart, they are lost too.
• No secret value. Never attach the output of vault kv get, nor a kubectl get secret -o yaml, nor the file produced by vault operator init. Send the key names:

  kubectl get secret <name> -n <ns> -o jsonpath='{.data}' | tr ',' '\n' | cut -d'"' -f2
  

• Say from where the symptom is visible: from a workstation, from inside the cluster, from the internet, or from all three. Half the entries of this page are told apart on that single piece of information.

Finally, attach in the clear what is not secret and without which nothing is interpretable: ${TV_DOMAIN}, ${TV_VIP}, ${TV_KC_REALM}, the date and time of the last change made on the cluster, and what has been tried since.