Multi-Cluster Networking: mTLS with Gateway API

The Cross-Cluster Requirement

We run multiple Kubernetes clusters: local (Kind), test (Scaleway), and production (Scaleway). Each cluster has Prometheus scraping local metrics.

For centralised monitoring, our management cluster runs Grafana and Mimir. Grafana needs to query Prometheus in each cluster. This is cross-cluster networking, and it needs to be secure.

Requirements:

• Grafana queries Prometheus in test/production clusters
• Connections must be encrypted (TLS)
• Connections must be mutually authenticated (client certs)
• No public exposure of Prometheus endpoints
• Manageable certificate lifecycle

Gateway API with cert-manager solves this cleanly.

The Gateway API Pattern

We use separate Gateways for public and internal traffic:

Public Gateway: Serves application traffic, uses Let's Encrypt certificates, no client authentication required.

Internal Gateway: Serves cluster-to-cluster traffic, requires mutual TLS (both server and client certs), not publicly accessible.

This separation means internal endpoints aren't accidentally exposed. Different Gateway, different TLS configuration, different security posture.

Implementation with cert-manager

Step 1: Create cluster CA

Each cluster has a cert-manager ClusterIssuer for internal certificates:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-secret

The CA certificate is generated once and stored in internal-ca-secret. This CA issues certificates for both servers (Prometheus endpoints) and clients (Grafana).

Step 2: Server certificate for Prometheus

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: prometheus-server-cert
  namespace: monitoring
spec:
  secretName: prometheus-tls
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  dnsNames:
  - prometheus.monitoring.svc.cluster.local
  - prometheus.example.com

cert-manager creates prometheus-tls secret with certificate and key.

Step 3: Client certificate for Grafana

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: grafana-client-cert
  namespace: monitoring
spec:
  secretName: grafana-client-tls
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  commonName: grafana-client
  usages:
  - client auth

Step 4: Gateway configuration for mTLS

This is where Gateway API shows its value:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: internal-gateway
  namespace: monitoring
spec:
  gatewayClassName: envoy-gateway
  listeners:
  - name: prometheus-mtls
    protocol: HTTPS
    port: 9090
    hostname: "prometheus.example.com"
    tls:
      mode: Terminate
      certificateRefs:
      - name: prometheus-tls
        kind: Secret

For client validation, we use Envoy Gateway's BackendTLSPolicy (Gateway API extension):

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: mtls-validation
  namespace: monitoring
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: internal-gateway
  tls:
    clientValidation:
      caCertificateRefs:
      - name: internal-ca-secret
        group: ""
        kind: Secret

This configuration:

• Terminates TLS with the server certificate
• Requires client certificate for authentication
• Validates client cert against the cluster CA

Step 5: HTTPRoute to Prometheus

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: prometheus-route
  namespace: monitoring
spec:
  parentRefs:
  - name: internal-gateway
    namespace: monitoring
  hostnames:
  - "prometheus.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: prometheus
      port: 9090

ReferenceGrants for Cross-Namespace Access

Gateway API enforces security: a Gateway in namespace A can't reference a Secret in namespace B without explicit permission.

Problem: Gateway is in monitoring namespace. CA certificate might be in cert-manager namespace.

Solution: ReferenceGrant

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-gateway-to-cert-manager
  namespace: cert-manager
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: monitoring
  to:
  - group: ""
    kind: Secret

This grants the Gateway in monitoring namespace permission to reference Secrets in cert-manager namespace. Explicit, auditable, secure.

What We Learned

Separation is cleaner than complex policies: We initially tried to use one Gateway with different TLS policies per route. This was complicated and error-prone. Separate Gateways—one public, one internal—is simpler.

ReferenceGrants add friction (intentionally): Having to explicitly grant cross-namespace access feels like extra configuration. But it prevents accidental exposure. The friction is security.

Certificate rotation is automatic: cert-manager handles renewal. Certificates rotate before expiry without manual intervention. This is better than static certificates that require runbooks.

Debugging TLS is still hard: When mTLS doesn't work, error messages are often cryptic. We added detailed logging and tested thoroughly in test environment before production.

Gateway API abstracts the proxy: We run Envoy Gateway, but the HTTPRoute definitions don't depend on Envoy specifics (except TLS validation configuration). If we need to switch to Istio or Cilium later, most configuration remains unchanged.

Sources:

• Gateway API documentation
• cert-manager documentation
• Envoy Gateway mTLS guide

#gatewayapi #mtls #networking #kubernetes #security