Deploy Cofide SPIRE (multiple clusters)

A trust zone can span multiple Kubernetes clusters. Workloads in each cluster share the same SPIFFE trust domain and can authenticate to each other without any federation configuration.

One cluster hosts the Cofide SPIRE server; all other clusters run SPIRE agents only.

Note

This guide works whether the trust zone has one Kubernetes cluster today or several. If you’re confident this trust zone will only ever run on a single cluster, Deploy Cofide SPIRE (single cluster) is a slightly simpler setup. If you’ve already deployed a trust zone using that guide and now need to add a cluster, see Migrate a single-cluster trust zone to multiple clusters.

How it works

Connect stores the topology as three API resources: the TrustZone (representing the trust domain), one Cluster entry per Kubernetes cluster, and a TrustZoneServer record that identifies which cluster hosts the Cofide SPIRE server. The Cofide SPIRE server automatically reconfigures its Kubernetes node attestor as clusters are added or removed. No manual SPIRE server configuration is required.

Prerequisites

• A Cofide Connect account.
• A Kubernetes cluster to host the Cofide SPIRE server (this can be bare-metal, VM or a cloud provider such as EKS or GKE).
• Kubeconfig access to each cluster you want to add.
• Decide on the authentication option for the server cluster.
• The Cofide Helm charts repository added (helm repo add cofide https://charts.cofide.dev).
• cofidectl installed (see Install and configure CLI).

Create the trust zone

Create the trust zone Connect API resource.

With cofidectl:

cofidectl trust-zone add \
  <your-trust-zone-name> \
  --trust-domain <your-trust-domain>

For additional configuration options, see cofidectl trust-zone add -h.

With Terraform:

resource "cofide_connect_trust_zone" "example" {
  name         = "<your-trust-zone-name>"
  trust_domain = "<your-trust-domain>"
}

Configure API resources

With the trust zone created above, add two clusters: the first hosts the Cofide SPIRE server, and the second runs SPIRE agents only. The Cofide SPIRE server uses OIDC for authentication with Connect.

Using Terraform (recommended)

A multi-cluster trust zone requires three further Terraform resources, in addition to the cofide_connect_trust_zone resource created above. See the Terraform reference for provider setup, and the Terraform Registry provider docs for the full resource argument reference.

# The cluster that hosts the SPIRE server
resource "cofide_connect_cluster" "server" {
  name            = "<your-server-cluster-name>"
  trust_zone_id   = cofide_connect_trust_zone.example.id
  profile         = "kubernetes"
  oidc_issuer_url = "https://<your-server-cluster-oidc-issuer>"

  trust_provider = {
    kind = "kubernetes"
    k8s_psat_config = {
      enabled = true
      allowed_service_accounts = [
        { namespace = "spire-system", service_account_name = "spire-agent" }
      ]
    }
  }
}

# Designates which cluster hosts the SPIRE server for this trust zone
resource "cofide_connect_trust_zone_server" "example" {
  trust_zone_id              = cofide_connect_trust_zone.example.id
  cluster_id                 = cofide_connect_cluster.server.id
  kubernetes_namespace       = "spire-server"
  kubernetes_service_account = "spire-server"

  connect_k8s_psat_config = {
    audiences                   = ["spire-server"]
    spire_server_spiffe_id_path = "/cofide-spire-server"
  }
}

output "trust_zone_server_id" {
  value = cofide_connect_trust_zone_server.example.id
}

# Additional cluster - runs SPIRE agents only
resource "cofide_connect_cluster" "workload" {
  name            = "<your-additional-cluster-name>"
  trust_zone_id   = cofide_connect_trust_zone.example.id
  profile         = "kubernetes"
  external_server = true
  oidc_issuer_url = "https://<your-additional-cluster-oidc-issuer>"

  trust_provider = {
    kind = "kubernetes"
    k8s_psat_config = {
      enabled = true
      allowed_service_accounts = [
        { namespace = "spire-system", service_account_name = "spire-agent" }
      ]
      api_server_url             = "https://<your-additional-cluster-api-server>"
      spire_server_audience      = "<unique-audience-for-additional-cluster>"
    }
  }
}

Repeat the cofide_connect_cluster block with external_server = true for each additional cluster.

Using cofidectl

# Add the cluster that will host the SPIRE server
cofidectl cluster add <your-server-cluster-name> \
  --trust-zone <your-trust-zone-name> \
  --kubernetes-oidc-issuer https://<your-server-cluster-oidc-issuer> \
  --psat-config '{"enabled":true,"allowed_service_accounts":["spire-system/spire-agent"]}'

# Configure the trust zone server within Connect
cofidectl trust-zone-server add \
  --trust-zone <your-trust-zone-name> \
  --cluster <your-server-cluster-name> \
  --kubernetes-namespace spire-server \
  --kubernetes-service-account spire-server \
  --connect-psat-audiences spire-server \
  --connect-psat-spiffe-id-path /cofide-spire-server

# Add each additional cluster (agent only)
cofidectl cluster add <your-additional-cluster-name> \
  --trust-zone <your-trust-zone-name> \
  --kubernetes-oidc-issuer https://<your-additional-cluster-oidc-issuer> \
  --external-server \
  --psat-config file://<path-to-psat-config.json>

The --external-server flag marks the cluster as one that does not host the SPIRE server. The --psat-config JSON for each additional cluster must include the fields that allow the SPIRE server to call that cluster’s Kubernetes API for node attestation:

{
  "enabled": true,
  "allowed_service_accounts": ["spire-system/spire-agent"],
  "api_server_url": "https://<your-additional-cluster-api-server>",
  "spire_server_audience": "<unique-audience-for-additional-cluster>"
}

Repeat the cluster add and PSAT config steps for each additional cluster.

Deploy Cofide SPIRE

Generate Helm values targeting the server cluster:

cofidectl trust-zone helm values <your-trust-zone-name> \
  --cluster <your-server-cluster-name> \
  --output-file spire-server-values.yaml

SPIRE agents in workload clusters connect to the SPIRE server over the network, so the server must be reachable from outside the cluster it runs in. The server cluster also requires values to enable cross-cluster node attestation and to expose the server. Save the following to spire-server-extra-values.yaml, substituting a different exposure mechanism (NodePort, ingress, etc.) if LoadBalancer is not available in your environment:

spire-server:
  nodeAttestor:
    connectK8sPSAT:
      enabled: true
    k8sPSAT:
      enabled: false
    externalK8sPSAT:
      enabled: false
  service:
    type: LoadBalancer

Install the chart, substituting the TrustZoneServer ID from the trust-zone-server add output above, or terraform output -raw trust_zone_server_id if you used Terraform.

helm install spire cofide/spire \
  --version <version> \
  --kube-context <your-server-cluster-context> \
  --namespace spire-mgmt \
  --create-namespace \
  --values spire-server-values.yaml \
  --values spire-server-extra-values.yaml \
  --set "spire-server.dataStore.connect.trustZoneServerID=<trust-zone-server-id>" \
  --wait

Add additional clusters

If your trust zone doesn’t have a TrustZoneServer record yet (for example, it was originally deployed as single-cluster), see Migrate a single-cluster trust zone to multiple clusters first.

Configure kube-apiserver JWT trust

The SPIRE server calls each workload cluster’s Kubernetes API to perform Kubernetes node attestation. It authenticates using a JWT-SVID issued by Connect, so the workload cluster’s kube-apiserver must be configured to trust Connect as a JWT issuer before agents can attest.

How you configure this depends on your Kubernetes distribution. For clusters that support structured authentication, add an entry to the JWT section of your AuthenticationConfiguration:

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
    url: <connect-oidc-issuer-url-for-trust-zone>
    certificateAuthority: <base64-encoded-ca-cert> # CA for Connect's trust bundle store; omit if using web PKI
    audiences:
    - <your-additional-cluster-name>
  claimMappings:
    username:
      claim: sub
      prefix: "spire:" # namespaces the username to avoid conflicts with other cluster users

If you don’t already know <trust-zone-id>, look it up by trust zone name with cofidectl:

cofidectl connect api call \
  --service proto.connect.trust_zone_service.v1alpha1.TrustZoneService \
  --rpc ListTrustZones \
  --data '{"filter": {"name": "<your-trust-zone-name>"}}' | jq -r '.trustZones[0].id'

Get the <connect-oidc-issuer-url-for-trust-zone> value with cofidectl:

cofidectl connect api call \
  --service proto.connect.trust_zone_service.v1alpha1.TrustZoneService \
  --rpc GetTrustZone \
  --data '{"trust_zone_id": "<trust-zone-id>"}' | jq '.trustZone.jwtIssuer'

If you used Terraform, it is the jwt_issuer attribute of the cofide_connect_trust_zone resource.

The audiences value must match the spire_server_audience you set in the cluster’s PSAT config, which defaults to the cluster name.

Grant RBAC to the SPIRE server

The SPIRE server needs permission to create token reviews, read nodes cluster-wide, and read pods in the namespace the SPIRE agent runs in (typically spire-system). After the kube-apiserver JWT trust is configured, the SPIRE server’s SPIFFE JWT-SVID is presented as the username spire:spiffe://<your-trust-domain>/cofide-spire-server (the spire: prefix comes from the claim mapping above). Apply the following RBAC to the workload cluster:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: spire-token-reviewer
rules:
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: spire-token-reviewer-binding
subjects:
- kind: User
  name: "spire:spiffe://<your-trust-domain>/cofide-spire-server"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: spire-token-reviewer
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: spire-server-pod-reader
  namespace: spire-system
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: spire-server-pod-reader-binding
  namespace: spire-system
subjects:
- kind: User
  name: "spire:spiffe://<your-trust-domain>/cofide-spire-server"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: spire-server-pod-reader
  apiGroup: rbac.authorization.k8s.io

Deploy SPIRE agent

Generate Helm values for each additional cluster:

cofidectl trust-zone helm values <your-trust-zone-name> \
  --cluster <your-additional-cluster-name> \
  --output-file spire-agent-values.yaml

Because the cluster has external_server = true, the generated values configure the SPIRE agent only. Install on each additional cluster, passing the server’s external address so agents can reach it and the bundle URL so agents can fetch the trust bundle on startup:

helm install spire cofide/spire \
  --version <version> \
  --kube-context <your-additional-cluster-context> \
  --namespace spire-mgmt \
  --create-namespace \
  --values spire-agent-values.yaml \
  --set "spire-agent.server.address=<server-external-address>" \
  --set "spire-agent.trustBundleURL=<bundle-url-for-trust-zone>" \
  --wait

Get the <bundle-url-for-trust-zone> value with cofidectl, using the same <trust-zone-id> as above:

cofidectl connect api call \
  --service proto.connect.trust_zone_service.v1alpha1.TrustZoneService \
  --rpc GetTrustZone \
  --data '{"trust_zone_id": "<trust-zone-id>"}' | jq '.trustZone.bundleEndpointUrl'

If you used Terraform, it is the bundle_endpoint_url attribute of the cofide_connect_trust_zone resource.

Adding a cluster later works the same way as adding one during initial setup: create its Cluster resource as shown in Configure API resources, then repeat the steps above for it. The SPIRE server detects the new cluster via Connect and reconfigures automatically; no server restart or manual configuration change within SPIRE is required.

See also

• Concepts: Trust Zone and Cluster
• Deploy Cofide SPIRE (single cluster)
• Migrate a single-cluster trust zone to multiple clusters
• Authentication methods