Observability¶
This page only covers monitoring: metric collection, log collection, dashboards, access to the
operating interfaces. Alerts are covered on the Operations page: on the reference instance,
thirty-four rules and the Alertmanager routing live in observability/prometheus-values.yaml,
and emails leave through the customer's SMTP. Nor does this page cover the conformance recipe
after installation, which lives on the Validation page.
Two monitoring stacks, only one to deploy¶
This is the main trap for a third-party installer.
| Stack | Origin | Role in a fresh installation |
|---|---|---|
monitoring |
Helm charts, applied by hand, outside the root kustomize | this is the one that gets deployed and consumed |
observability |
observability/*.yaml manifests, pulled in by the root kustomize |
legacy. An installer does not deploy it |
An operator who applies kubectl kustomize . (step 3 of the
Step by step) gets the second stack, in the observability namespace: a
Prometheus, a Grafana and a Kibana that start on their default configuration, with no ConfigMap,
no --config.file argument, no provisioned datasource. It runs, it answers, and it receives no
metric and no log from the platform. An operator who opens that Grafana without knowing this
page wrongly concludes that monitoring is broken, when they are looking at the wrong Grafana:
the one that actually carries the data, the two versioned dashboards and the intended
authentication is the one of the monitoring stack.
Decision for every customer installation: deploy and operate the monitoring stack only.
The observability stack laid down by the root kustomize can be deleted without risk once the
installation is finished (kubectl delete namespace observability) or left inert; in both
cases, never rely on it and never import the dashboards of this page into it. Kibana falls
under the same arbitration: its Deployment carries no ELASTICSEARCH_HOSTS variable, so
nothing connects it to Elasticsearch. The log path actually used goes through Alloy and Loki,
described further down.
Installing the four Helm releases of the monitoring stack
None of these releases comes from the root kustomize. Each has its values file in
observability/, complete and dated; helm upgrade replaces the whole set of values, so
those files are the source of truth, not helm get values. Alertmanager is not a fifth
release: it is a subchart of Prometheus, and the prometheus-alertmanager Service comes
from there.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
kubectl apply -f observability/alertmanager-smtp.yaml
helm upgrade --install prometheus prometheus-community/prometheus -n monitoring --create-namespace \
--version 29.14.0 -f observability/prometheus-values.yaml
helm upgrade --install grafana grafana/grafana -n monitoring \
--version 10.5.15 -f observability/grafana-values.yaml
helm upgrade --install loki grafana/loki -n monitoring \
--version 7.0.0 -f observability/loki-values.yaml
helm upgrade --install alloy grafana/alloy -n monitoring \
--version 1.10.0 -f observability/alloy-values.yaml
kubectl apply -f observability/scrape/services-metriques.yaml
| Release | Chart | What the values file carries |
|---|---|---|
prometheus |
prometheus-community/prometheus 29.14.0 |
the thirty-four alert rules, the Alertmanager routing, the chart's node-exporter disabled |
grafana |
grafana/grafana 10.5.15 |
data source, dashboard provisioning |
loki |
grafana/loki 7.0.0 |
SingleBinary mode, memcached bounds, labels |
alloy |
grafana/alloy 1.10.0 |
log collection into Loki |
observability/alertmanager-smtp.yaml comes BEFORE the prometheus release, and not out of
habit: the values file mounts the alertmanager-smtp Secret into the Alertmanager pod, which
reads its SMTP password there (smtp_auth_password_file). If the Secret does not exist, the
pod stays in ContainerCreating on a volume that cannot be found, and no alert leaves. The
Vault path is prod/alertmanager, key smtp_password, written at step 11 of the step by
step; the server, the sender and the recipient are in the values file, in clear, since
nothing there is secret. The server is TV_SMTP_HOST, which render-config.sh substitutes
there too. The login (smtp_auth_username) and the sender (smtp_from) stay those of the
reference instance: replace them by hand with the customer's before the helm upgrade, the
sender on the domain the relay has verified. The recipient (to: under
default-receiver) is a mailbox of the reference instance whose domain alone
render-config.sh rewrites: put ${TV_ACME_EMAIL} there by hand, in the same pass.
Check the full path once the release is in
place, without waiting for a real outage:
kubectl -n monitoring exec sts/prometheus-alertmanager -- \
amtool alert add TestCourriel severity=warning --alertmanager.url=http://localhost:9093
[FIRING:1] TestCourriel email arrives within a minute (group_wait: 30s). If it does
not, the log of the same pod carries the cause, notify retry with the SMTP code, and
nothing else will. The test alert expires on its own after five minutes.
observability/scrape/services-metriques.yaml adds the targets Prometheus does not discover
on its own: external-secrets, Longhorn, the Keycloak Postgres database, cloudflared and Vault. Without it,
the secrets, stockage, coffre, base-keycloak and exposition rule groups have no series to read and
stay silent.
Components and versions¶
| Component | Namespace | Manifest or chart | Role |
|---|---|---|---|
| Prometheus | monitoring |
chart prometheus-29.14.0, observability/prometheus-values.yaml; Service prometheus-server:80 |
metrics and alert rules |
| Alertmanager | monitoring |
Prometheus subchart, same values file; Service prometheus-alertmanager:9093 |
routing of the thirty-four rules |
| Grafana | monitoring |
chart grafana-10.5.15, observability/grafana-values.yaml; Service grafana:80 |
rendering |
| Loki | monitoring |
chart loki-7.0.0, observability/loki-values.yaml, SingleBinary mode |
logs |
| Alloy | monitoring |
chart grafana/alloy, observability/alloy-values.yaml, DaemonSet |
reading pod logs |
| node-exporter | node-exporter |
core/monitoring/node-exporter.yaml, node-exporter:v1.11.1, port 9100; outside the base layer, laid down by step 10 of the Step by step |
host metrics |
| metrics-server | kube-system |
core/metrics-server/metrics-server.yaml, v0.7.2, --metric-resolution=15s |
kubectl top and HPA, not a monitoring source |
metrics-server feeds kubectl top and the HPA, never an alert or a dashboard: its latency of
15 to 30 seconds makes it blind to fast failures.
Traefik exposes its own Prometheus metrics on its administration entrypoint: that single source covers the whole HTTP layer of the platform, the three portals, Keycloak and the seven business APIs, with no application instrumentation. It is the source of the RED dashboard described further down.
The prometheus.yaml file versioned in observability/ (image prom/prometheus:v2.54.1, PVC
prometheus-data of 50Gi) belongs to the observability stack, not to the monitoring stack:
it is not a reference sizing for the instance actually operated, only the default configuration
of a Prometheus that receives nothing.
Parameters to set per customer¶
The two Loki parameters below can only be set once the loki release has been created: see the
open question further up.
| Parameter | Reference value | What to do for a new customer |
|---|---|---|
| Prometheus retention, PVC size | not fixed in this repository | open question, see the box above: these values are set at the moment the monitoring stack is installed, to be agreed with the customer according to the contractual retention duration and the available storage |
| Bounds of the Loki memcached caches | chunksCache.allocatedMemory: 1024, resultsCache.allocatedMemory: 512 (MB), in observability/loki-values.yaml |
these values were calibrated on the memory of the workers of the reference instance, after a chart default at 8192 MB monopolised 91% of the memory of one node; recalibrate them downwards or upwards according to the real memory of the customer's workers before the first helm upgrade, not after |
| Loki push endpoint | http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push |
fixed for every customer: it is a cluster-internal Service address, independent of ${TV_DOMAIN} |
The Loki values file must always carry the entirety of the wanted values: helm upgrade
replaces the set provided, a partial file resets everything else to the chart defaults, among
them the cache at 8192 MB already responsible for an incident. The same goes for
observability/alloy-values.yaml.
Loki labels, and why they must stay stable¶
The Kubernetes metadata arrives in Alloy as internal labels prefixed with __, which are never
persisted if they are not explicitly copied over. Without that copy, Loki only receives the
pod label, which contains the hash of the ReplicaSet and changes at every recreation: the logs
of a service then drop out at every deployment, and the name of the new pod has to be
re-selected by hand.
observability/alloy-values.yaml therefore copies over four stable labels, through a
discovery.relabel step that is not to be removed:
| Label | Source | Role |
|---|---|---|
namespace |
__meta_kubernetes_namespace |
partitioning |
app |
pod label app.kubernetes.io/name, otherwise app |
selecting the service, invariant across deployments |
container |
__meta_kubernetes_pod_container_name |
sorting the containers within one pod |
pod |
__meta_kubernetes_pod_name |
isolating one specific replica |
The reference query targets the service, never the pod:
The Alloy config is managed by Helm: a kubectl edit or a direct kubectl apply on its
ConfigMap is overwritten at the next helm upgrade. The only source of truth is
observability/alloy-values.yaml.
Importing the dashboards¶
Grafana loads no dashboard at startup on this instance: no sidecar goes looking for files, the
path /etc/grafana/provisioning is mounted by nothing. The import is manual, for each of the
two files versioned in the repository (red-par-service.json, UID tradeval-red;
ressources.json, UID tradeval-ressources):
Dashboards -> New -> Import -> Upload JSON file, then choose the Prometheus datasource.
Both files declare an __inputs block: the dialog box therefore asks for the datasource at
every import, nothing is hard-coded. The tradeval-red dashboard answers throughput, error
rate and latency per service, on the Traefik source alone. The tradeval-ressources dashboard
answers the question of whether the cluster is the limiting factor, with a selection per
namespace. The files of the repository are authoritative: a modification made in the Grafana
interface only exists in its SQLite database and disappears at the next import.
Checking that the stack really receives data¶
The import succeeds identically whether the chosen datasource receives metrics or not: Grafana
displays "Dashboard imported" just as well if the monitoring stack works, if it does not
exist yet (see the open question further up) or if the datasource retained by mistake is that
of the observability stack, which receives nothing. A successful import therefore proves
nothing on its own: the three checks below verify that the data arrives, not merely that the
gesture was performed.
The two datasources are both called "Prometheus" in the Grafana import dialog; only their URL
distinguishes them. The right one targets the Service prometheus-server in the monitoring
namespace; the one of the legacy stack targets the Service prometheus in the observability
namespace. It is that Service name, in the URL of the datasource, that settles it.
| Check | Symptom on failure |
|---|---|
The Prometheus targets are up, not merely declared: kubectl -n monitoring port-forward svc/prometheus-server 9090:80 & then curl -sS 'http://localhost:9090/api/v1/query?query=up' returns at least two series at the value 1, one of them for Traefik |
No series, or a command that fails: the port-forward targets the wrong stack (observability has no prometheus-server Service), or Traefik does not yet carry its scrape annotations |
A control LogQL query returns lines, after having caused traffic on the service queried: kubectl -n api rollout restart deploy/trade-classification-svc, then kubectl -n monitoring port-forward svc/loki-gateway 3100:80 & and curl -sS -G http://localhost:3100/loki/api/v1/query_range --data-urlencode 'query={namespace="api", app="trade-classification-svc"}' |
Empty response: the alloy release does not exist (see the open question further up), the labels are not copied over, or the port-forward targets a Loki that does not exist |
The tradeval-red dashboard displays a non-zero throughput on trade-classification-svc, after having sent about twenty requests that really go through Traefik: for i in $(seq 1 20); do curl -sS -o /dev/null https://classification.${TV_DOMAIN}/; done, from a workstation that resolves that host to the VIP, or from a pod of the cluster. A pod restart is not enough: that dashboard only reads Traefik metrics, never produced by a simple rollout restart |
Throughput that stays at zero despite the requests sent, or "No data": the datasource retained at import carries the URL of the Service prometheus in the observability namespace (the legacy stack), not that of the Service prometheus-server in monitoring (the only one to choose); or the requests did not reach Traefik, host badly resolved or TLSStore not ready yet |
Access to the interfaces and basic-auth¶
core/platform-access/ contains five Traefik Ingresses: the three of the monitoring stack,
Longhorn, and the Vault UI. Step 10 of the
Step by step lays down
only two of them, Longhorn and Vault, which it applies through named files: the three others
target the monitoring namespace, which no manifest of the repository creates, and are only
applied once the monitoring stack is installed, which this guide does not cover. All of them
carry the same entrypoint and TLS annotations, without a spec.tls of their own: the certificate
comes from the default TLSStore that serves the wildcard *.${TV_DOMAIN}. This section only
covers the first four; the Vault UI, already presented in the step by step, has its own token
authentication and a distinct risk profile.
| Host | Backend | Authentication |
|---|---|---|
grafana.${TV_DOMAIN} |
monitoring:grafana:80 |
built-in Grafana login |
prometheus.${TV_DOMAIN} |
monitoring:prometheus-server:80 |
Traefik basic-auth, core/platform-access/auth.yaml |
alertmanager.${TV_DOMAIN} |
monitoring:prometheus-alertmanager:9093 |
idem |
longhorn.${TV_DOMAIN} |
longhorn-system:longhorn-frontend:80 |
idem |
The basic-auth is laid down by core/platform-access/auth.yaml
since 2026-09-15: one ExternalSecret and one dashboard-auth Middleware per namespace
concerned, monitoring and longhorn-system, and the
traefik.ingress.kubernetes.io/router.middlewares: <namespace>-dashboard-auth@kubernetescrd
annotation on the three Ingresses. The htpasswd file lives in Vault, prod/platform-access,
key users, one line per account in bcrypt format. Nothing is created by hand: put the key in
Vault, apply the file.
htpasswd -nbB admin '<password>' | vault kv patch -mount=secrets prod/platform-access users=-
kubectl apply -f core/platform-access/auth.yaml
Without htpasswd on the workstation, any httpd:2.4-alpine container carries one. Traefik
accepts bcrypt, MD5-APR1 and SHA1; openssl passwd -6 produces SHA-512, which it does not read.
Two other options exist, in place of basic-auth: restricting by source IP with an ipAllowList
Middleware, or going through an SSO (Traefik forward-auth to Keycloak) for a protection
homogeneous with the rest of the platform.
Never publish a public DNS record for those four hosts
The basic-auth protects against an unfortunate click on the internal network, not against the Internet. The Longhorn console allows volumes to be deleted. The real protection of those hosts remains never being reachable except through the internal network or the VPN: no public record, no route in the Cloudflare tunnel.
Without metrics-server the HPA stay inert; do not replay the historical incident on them
The ten HPA of this repository stay inert as long as metrics-server is not installed,
and signal it only by an <unknown> value in kubectl get hpa. On the reference instance,
installing metrics-server one day woke up, all at once, HPA that were set on a memory
target and a maxReplicas of 50, saturating the CPU of the cluster within a few minutes.
That configuration has been fixed since: the HPA versioned today no longer carry a memory
metric and cap at 3 replicas, 6 for hub-portail. Do not conclude from this that installing
metrics-server is risk-free on a cluster of a different sizing, or after
adding a new service: confront the requests values and the maxReplicas bounds with real
usage before installing metrics-server, rather than inheriting the figures already
present in the repository without checking them again.