1.1 - Installation
Installation guide for the capsule-proxy
Capsule Proxy provides filtered access to Kubernetes resources for Capsule users.
The default configuration requires a working Capsule installation. To use only
global grants for authenticated non-Capsule users, Capsule is optional with
--require-capsule-user=false; see User scope.
The capsule-proxy can be deployed in standalone mode, e.g. running as a pod bridging any Kubernetes client to the APIs server. Optionally, it can be deployed as a sidecar container in the backend of a dashboard.
We officially only support the installation of Capsule using the Helm chart. The chart itself handles the Installation/Upgrade of needed CustomResourceDefinitions. The following Artifacthub repository are official:
Perform the following steps to install the capsule Operator:
Add repository:
helm repo add projectcapsule https://projectcapsule.github.io/charts
Install Capsule-Proxy:
helm install capsule-proxy projectcapsule/capsule-proxy -n capsule-system --create-namespace
or (OCI)
helm install capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy -n capsule-system --create-namespace
Show the status:
helm status capsule-proxy -n capsule-system
Upgrade the Chart
helm upgrade capsule-proxy projectcapsule/capsule-proxy -n capsule-system
or (OCI)
helm upgrade capsule-proxy oci://ghcr.io/projectcapsule/charts/capsule-proxy --version 0.13.0
Uninstall the Chart
helm uninstall capsule-proxy -n capsule-system
GitOps
There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource.
ArgoCD
Visit the ArgoCD Integration for more options to integrate Capsule with ArgoCD.
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: capsule
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: system
source:
repoURL: ghcr.io/projectcapsule/charts
targetRevision: 0.15.2
chart: capsule
helm:
valuesObject:
...
proxy:
enabled: true
webhooks:
enabled: true
certManager:
generateCertificates: true
options:
generateCertificates: false
oidcUsernameClaim: "email"
extraArgs:
- "--feature-gates=ProxyClusterScoped=true"
serviceMonitor:
enabled: true
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
destination:
server: https://kubernetes.default.svc
namespace: capsule-system
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
- RespectIgnoreDifferences=true
retry:
limit: 5
backoff:
duration: 5s
factor: 2
maxDuration: 3m
---
apiVersion: v1
kind: Secret
metadata:
name: capsule-repo
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
url: ghcr.io/projectcapsule/charts
name: capsule
project: system
type: helm
enableOCI: "true"
FluxCD
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: capsule
namespace: flux-system
spec:
serviceAccountName: kustomize-controller
targetNamespace: "capsule-system"
interval: 10m
releaseName: "capsule"
chart:
spec:
chart: capsule
version: "0.15.2"
sourceRef:
kind: HelmRepository
name: capsule
interval: 24h
install:
createNamespace: true
upgrade:
remediation:
remediateLastFailure: true
driftDetection:
mode: enabled
values:
proxy:
enabled: true
webhooks:
enabled: true
certManager:
generateCertificates: true
options:
generateCertificates: false
oidcUsernameClaim: "email"
extraArgs:
- "--feature-gates=ProxyClusterScoped=true"
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: capsule
namespace: flux-system
spec:
type: "oci"
interval: 12h0m0s
url: oci://ghcr.io/projectcapsule/charts
Considerations
Considerations when deploying capsule-proxy
User Scope
By default, --require-capsule-user=true: subjects referenced by ProxySetting,
GlobalProxySettings, or reflected RoleBindings receive proxy grants only when
the caller is a recognized Capsule identity. Referencing a subject in a rule or
RoleBinding does not by itself make that subject a Capsule user. Other callers
are forwarded using their own identity and Kubernetes RBAC.
Configure eligible users and groups in
CapsuleConfiguration.spec.users. The
proxy reads the configuration’s reconciled user and group status. ServiceAccounts
that own a tenant, including promoted ServiceAccounts, are also recognized.
--capsule-configuration-name selects the configuration and defaults to default.
Include all authenticated identities
For callers whose resolved groups include system:authenticated, one option is
to treat that whole group as Capsule users.
Merge the following fields into the selected CapsuleConfiguration, replacing
platform-admins with your administrator group:
apiVersion: capsule.clastix.io/v1beta2
kind: CapsuleConfiguration
metadata:
name: default
spec:
users:
- kind: Group
name: system:authenticated
ignoreUserWithGroups:
- platform-admins
system:authenticated includes authenticated users and ServiceAccounts. This
makes them eligible for Capsule access features; visibility still depends on
tenant ownership, tenant delegation, reflected RBAC, or matching global grants.
It does not grant every authenticated identity access to every tenant.
This configuration also broadens Capsule’s admission scope; use a narrower group
when only selected identities should be managed by Capsule.
The proxy matches the groups supplied by its authentication provider. Kubernetes
TokenReview supplies system:authenticated for authenticated bearer tokens.
Direct client-certificate authentication uses the certificate’s Organization
fields and does not automatically add that group. For those clients, also include
a matching certificate group or explicit user in spec.users.
Configure administrator exclusions on the proxy as well. With Helm:
options:
ignoredUserGroups:
- platform-admins
ignoredUsernames:
- platform-admin
The corresponding flags are --ignored-user-group=platform-admins and
--ignored-username=platform-admin; use your actual administrator identities.
Ignored callers bypass proxy filtering and retain their native Kubernetes RBAC
permissions. These options do not grant administrator permissions themselves.
Capsule’s ignoreUserWithGroups controls Capsule admission handling and does not
replace the proxy’s ignored-identity options.
RoleBinding reflection
For Capsule identities that receive access through additional RoleBindings,
enable --enable-reflector=true (options.roleBindingReflector: true with Helm).
Reflection requires Capsule integration and caching, so keep
--disable-caching=false, its default.
Put the reflection label on the additional RoleBinding template in the Tenant’s
specification, together with the subjects that should receive access:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: team-a
spec:
owners:
- kind: User
name: tenant-owner
clusterRoles:
- admin
additionalRoleBindings:
- clusterRoleName: view
labels:
reflection.proxy.projectcapsule.dev/enabled: "true"
subjects:
- kind: Group
name: team-a-readers
apiGroup: rbac.authorization.k8s.io
Capsule copies this label onto the generated RoleBindings; a label on the
Tenant’s own metadata is insufficient. Under the default identity gate,
team-a-readers members must also be recognized Capsule users, for example
through the system:authenticated configuration above.
For cluster-wide lists of namespaced resources, the reflector checks the
binding’s label, subjects, and the referenced role’s LIST rules for the requested
API group and resource. Eligible bindings add visibility for the associated
tenant, including its sibling namespaces. Requests for resources within a
specific namespace continue to use Kubernetes RBAC. See
Reflection for details and the separate Namespace
visibility behavior.
Global grants for non-Capsule users
To allow authenticated subjects to use matching GlobalProxySettings without
making them Capsule users, set
--require-capsule-user=false and --feature-gates=ProxyClusterScoped=true.
With Helm:
options:
requireCapsuleUser: false
extraArgs:
- --feature-gates=ProxyClusterScoped=true
Ignored identities keep their native forwarding behavior. For non-Capsule
callers, a global rule must match the subject, API group, resource, and operation;
other requests use caller impersonation and Kubernetes RBAC. A matching rule
whose selector selects no objects returns an empty list. Global grants do not
enable watches, mutations, or access to resources inside selected namespaces.
Global-only requests use the generic cluster-resource handlers and do not inherit
tenant ownership or reflected namespace access.
All conditions within a selector must match; separate matching rules grant the
union of their selections. GET accepts any complete matching rule. LIST can
combine identical rules or rules that differ only in one positive label value
set, preserving their other conditions. Other rule combinations return a
Kubernetes BadRequest rather than broadening access across label combinations.
Capsule is optional in this mode. If the selected CapsuleConfiguration is absent
at startup, the proxy skips Capsule configuration reconciliation, tenant indexes,
tenant delegation, and RBAC reflection. Namespace access then uses the generic
cluster-resource handlers. Install the proxy’s CRDs and RBAC as usual. When the
CapsuleConfiguration exists, the existing tenant-aware behavior remains enabled.
Availability is checked at startup; restart the proxy when Capsule or the
selected configuration is installed or removed. Unexpected discovery or read errors
still fail startup.
Scalability
For large clusters you might need to consider adjusting values for the Capsule controller.
QPS/Burst
In order to handle a large number of tenants and resources, you may need to increase the QPS and Burst values for the Capsule-Proxy. This avoids the Proxy being throttled by the Kubernetes API server (Client Rate limited). You can set the following values in the Helm chart:
options:
# -- QPS to use for interacting with Kubernetes API Server.
clientConnectionQPS: 200
# -- Burst to use for interacting with kubernetes API Server.
clientConnectionBurst: 400
API Priority and Fairness (APF)
With APF enabled, the Capsule controller will be subject to the APF configuration of the cluster. If you are running a large cluster with many users/tenants, you may need to adjust the APF configuration to ensure that the Capsule controller has sufficient resources to operate effectively. For more information on APF, see Kubernetes API Priority and Fairness.
We provide a built-in APF configuration for the Capsule-Proxy, which provides API priority for all LIST operations and especially for subjectaccessreviews and tokenreviews. This configuration is applied automatically when you install Capsule-Proxy. To enable the built-in APF configuration, set the following value in the Helm chart:
apiPriorityAndFairness:
# -- Change to `true` if you want to insulate the API calls made by Capsule admission controller activities.
# This will help ensure Capsule stability in busy clusters.
# Ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/
enabled: true
# -- Only the first matching FlowSchema for a given request matters. If multiple FlowSchemas match a single inbound request, it will be assigned based on the one with the highest matchingPrecedence.
# Ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/#flowschema
matchingPrecedence: 900
# -- Priority level configuration.
# The block is directly forwarded into the priorityLevelConfiguration, so you can use whatever specification you want.
# ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/#prioritylevelconfiguration
priorityLevelConfigurationSpec:
type: Limited
limited:
nominalConcurrencyShares: 100
limitResponse:
type: Queue
queuing:
queues: 64
handSize: 6
queueLengthLimit: 100
Ignore Groups
When tokens have a lot of groups, you can try to exclude groups which may never be used for authorization. This can be done by setting options.impersonationGroupRegexp and options.ignoredImpersonationGroups in the Helm chart:
options:
# -- Regular expression to match the groups which are considered for impersonation
impersonationGroupRegexp: "kubernetes:.*"
# -- Names of the groups which are not used for impersonation (considered after impersonation-group-regexp)
ignoredImpersonationGroups:
- "specific:ignored"
This results in less SSAR requests to the Kubernetes API server and less memory usage for the Capsule-Proxy. This is especially useful when using OIDC providers which provide a lot of groups, but only a few are used for authorization.
Exposure
Depending on your environment, you can expose the capsule-proxy by:
Gateway API (Recommended)Ingress (Recommended)NodePort ServiceLoadBalance ServiceHostPortHostNetwork
Gateway API
If you are using a Gateway API compliant Ingress Controller, you must first make the decision, how TLS is terminiated or rather what’s possible in your environment. We have two potential options.
Backend Termination (Recommended)
This is where the TLS termination is executed by the capsule-proxy, meaning that the Ingress Controller will just forward the encrypted traffic to the capsule-proxy, which will decrypt it and forward it to the Kubernetes API Server. In this way, the client certificate authentication will be preserved and reversed to the upstream.
- On your Gateway, add a listener which allows to forward the encrypted traffic to the capsule-proxy (Pass-Through TLS):
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: service-gateway
namespace: solar-system
spec:
gatewayClassName: default
listeners:
- allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
kubernetes.io/metadata.name: capsule-system
hostname: api.cluster-name.company.com
name: https-capsule-proxy
port: 443
protocol: TLS
tls:
mode: Passthrough
- Install a
TLSRoute resource to forward the encrypted traffic to the capsule-proxy:
extraManifests:
- apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata:
name: capsule-proxy-tls-route
namespace: capsule-system
spec:
parentRefs:
- name: service-gateway
namespace: solar-system
sectionName: https-capsule-proxy
hostnames:
- api.cluster-name.company.com
rules:
- backendRefs:
- name: capsule-proxy
port: 9001
Gateway Termination
When the Gateway is terminating we must ensure, that users use the corresponding CA/Serving Certificate in their Kubeconfigs. You must ensure that the CA certificate of the Gateway is distributed to the users, so they can use it to verify the identity of the capsule-proxy. In this way, the client certificate authentication will be withdrawn and not reversed to the upstream.
- Just create a listener on the Gateway and Reference the TLS certificate. The following example uses the cert-manager integration:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
annotations:
cert-manager.io/cluster-issuer: cluster-service-issuer
cert-manager.io/private-key-algorithm: RSA
cert-manager.io/private-key-size: "4096"
name: service-gateway
namespace: solar-system
spec:
gatewayClassName: default
listeners:
- allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
kubernetes.io/metadata.name: capsule-system
hostname: api.cluster-name.company.com
name: https-capsule-proxy
port: 443
protocol: HTTPS
tls:
certificateRefs:
- group: ""
kind: Secret
name: capsule-proxy-tls
mode: Terminate
Ingress
When using an Ingress Controller, you can expose the capsule-proxy through an Ingress resource. The Ingress Controller will handle the TLS termination and forward the requests to the capsule-proxy. The capsule-proxy will then forward the requests to the Kubernetes API Server.
+-----------+ +-----------+ +-----------+
kubectl ------>|:443 |--------->|:9001 |-------->|:6443 |
+-----------+ +-----------+ +-----------+
ingress-controller capsule-proxy kube-apiserver
You can use the Ingress Values provided in the Helm chart to configure the Ingress resource for the capsule-proxy:
ingress:
enabled: true
className: "nginx" # or your ingress class name
hosts:
- host: capsule-proxy.company.com
paths:
- "/"
User Authentication
The capsule-proxy intercepts all the requests from the kubectl client directed to the APIs Server. Users using a TLS client-based authentication with a certificate and key can talk with the API Server since it can forward client certificates to the Kubernetes APIs server. You can configure the capsule-proxy to use multiple authentication methods, for example, you can prefer Bearer Token authentication over TLS client-based authentication or use Forwarded Client Certificate Authentication (XFCC) if supported by your Ingress Controller. The following sections describe the supported authentication methods.
options:
authPreferredTypes: "BearerToken,TLSCertificate,XForwardedClientCert"
Bearer Token Authentication
Bearer Token authentication is supported by default, meaning that users providing tokens are always able to reach the APIs Server. You can configure the capsule-proxy to prefer Bearer Token authentication over TLS client-based authentication:
options:
authPreferredTypes: "BearerToken"
Client Certificate Authentication
It is possible to protect the capsule-proxy using a certificate provided by Let’s Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream. For such cases you may want to rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server or consider using the Forwarded Client Certificate Authentication (XFCC) if supported by your Ingress Controller.
options:
authPreferredTypes: "TLSCertificate"
Forwarded Client Certificate Authentication (XFCC)
It is possible to protect the capsule-proxy using a certificate provided by Let’s Encrypt. Keep in mind that, in this way, the TLS termination will be executed by the Ingress Controller, meaning that the authentication based on the client certificate will be withdrawn and not reversed to the upstream.
If your prerequisite is exposing capsule-proxy using an Ingress, you must rely on the token-based authentication, for example, OIDC or Bearer tokens. Users providing tokens are always able to reach the APIs Server.
options:
authPreferredTypes: "XForwardedClientCert"
By default the HTTP-Header used for the client certificate is X-Forwarded-Client-Cert, but it can be customized using the --xfcc-header-name argument:
options:
authPreferredTypes: "XForwardedClientCert"
extraArgs:
- "--xfcc-header-name=X-My-Custom-Client-Cert"
Trusted Sources
CIDR ranges of trusted proxies allowed to send forwarded client certificate headers:
options:
trustedProxyCidrs:
- "10.0.0.0/8"
- "127.0.0.1/32"
Certificate Management
By default, Capsule delegates its certificate management to cert-manager. This is the recommended way to manage the TLS certificates for Capsule.This relates to certifiacates for the proxy and the admissions server. However, you can also use a job to generate self-signed certificates and store them in a Kubernetes Secret:
options:
generateCertificates: true
certManager:
generateCertificates: false
Distribute CA within the Cluster
The capsule-proxy requires the CA certificate to be distributed to the clients. The CA certificate is stored in a Secret named capsule-proxy in the capsule-system namespace, by default. In most cases the distribution of this secret is required for other clients within the cluster (e.g. the Tekton Dashboard). If you are using Ingress or any other endpoints for all the clients, this step is probably not required.
Here’s an example of how to distribute the CA certificate to the namespace tekton-pipelines by using kubectl and jq:
kubectl get secret capsule-proxy -n capsule-system -o json \
| jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
| kubectl apply -n tekton-pipelines -f -
This can be used for development purposes, but it’s not recommended for production environments. Here are solutions to distribute the CA certificate, which might be useful for production environments:
External Secrets (ESO)
How to distribute the CA certificate using External Secrets Operator (ESO). In the following example we have Headlamp running in a different namespace and we want to distribute the CA certificate to it.
First allow ServiceAccount headlamp to read the Secret capsule-proxy in the capsule-system namespace:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: headlamp:capsule-proxy-ca-reader
namespace: capsule-system
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["capsule-proxy"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: headlamp:capsule-proxy-ca-reader
namespace: capsule-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: headlamp:capsule-proxy-ca-reader
subjects:
- kind: ServiceAccount
name: headlamp
namespace: dashboard-system
Create ExternalSecret to sync the Secret capsule-proxy from the capsule-system namespace to the dashboard-system namespace:
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: capsule-proxy-ca
namespace: dashboard-system
spec:
provider:
kubernetes:
remoteNamespace: capsule-system
server:
url: https://kubernetes.default.svc
caProvider:
type: ConfigMap
name: kube-root-ca.crt
key: ca.crt
auth:
serviceAccount:
name: headlamp
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: capsule-proxy-ca
namespace: dashboard-system
spec:
refreshInterval: 1h
secretStoreRef:
kind: SecretStore
name: capsule-proxy-ca
target:
name: capsule-proxy-ca
creationPolicy: Owner
data:
- secretKey: ca.crt
remoteRef:
key: capsule-proxy
property: ca.crt
HTTP Support
NOTE: kubectl will not work against a http server.
Capsule proxy supports https and http, although the latter is not recommended, we understand that it can be useful for some use cases (i.e. development, working behind a TLS-terminated reverse proxy and so on). As the default behaviour is to work with https, we need to use the flag –enable-ssl=false if we want to work under http.
After having the capsule-proxy working under http, requests must provide authentication using an allowed Bearer Token.
For example:
TOKEN=<type your TOKEN>
curl -H "Authorization: Bearer $TOKEN" http://localhost:9001/api/v1/namespaces
Metrics
Starting from the v0.3.0 release, Capsule Proxy exposes Prometheus metrics available at http://0.0.0.0:8080/metrics.
The offered metrics are related to the internal controller-manager code base, such as work queue and REST client requests, and the Go runtime ones.
Along with these, metrics capsule_proxy_response_time_seconds and capsule_proxy_requests_total have been introduced and are specific to the Capsule Proxy code-base and functionalities.
capsule_proxy_response_time_seconds offers a bucket representation of the HTTP request duration. The available variables for these metrics are the following ones:
path: the HTTP path of every single request that Capsule Proxy passes to the upstream
capsule_proxy_requests_total counts the global requests that Capsule Proxy is passing to the upstream with the following labels.
path: the HTTP path of every single request that Capsule Proxy passes to the upstream
status: the HTTP status code of the request
1.2 - Controller Options
Configure the Capsule Proxy Controller
You can customize the Capsule Proxy with the following configurations. See
User scope for identity eligibility, administrator
exclusions, RoleBinding reflection, and grants for non-Capsule users.
Controller Options
You can provide additional options via the helm chart:
options:
extraArgs:
- --disable-caching=true
Options are also available as dedicated configuration values:
# Controller Options
options:
# -- Set the listening port of the capsule-proxy
listeningPort: 9001
# -- Set leader election to true if you are running n-replicas
leaderElection: false
# -- Set the log verbosity of the capsule-proxy with a value from 1 to 10
logLevel: 4
# -- Name of the CapsuleConfiguration custom resource used by Capsule, required to identify the user groups
capsuleConfigurationName: default
# -- Require Capsule identities for proxy grants. Set false to allow authenticated non-Capsule callers to use matching GlobalProxySettings; Capsule is optional. Requires ProxyClusterScoped for global grants.
requireCapsuleUser: true
# -- Define which groups must be ignored while proxying requests
ignoredUserGroups: []
# -- Define which usernames must be ignored while proxying requests
ignoredUsernames: []
# -- Names of the groups which are not used for impersonation (considered after impersonation-group-regexp)
ignoredImpersonationGroups: []
# -- Regular expression to match the groups which are considered for impersonation
impersonationGroupRegexp: ""
# -- Specify if capsule-proxy will use SSL
oidcUsernameClaim: preferred_username
# -- Specify if capsule-proxy will use SSL
enableSSL: true
# -- Set the directory, where SSL certificate and keyfile will be located
SSLDirectory: /opt/capsule-proxy
# -- Set the name of SSL certificate file
SSLCertFileName: tls.crt
# -- Set the name of SSL key file
SSLKeyFileName: tls.key
# -- Specify if capsule-proxy will generate self-signed SSL certificates
generateCertificates: false
# -- Specify additional subject alternative names for the self-signed SSL
additionalSANs: []
# -- Specify an override for the Secret containing the certificate for SSL. Default value is empty and referring to the generated certificate.
certificateVolumeName: ""
# -- Set the role bindings reflector resync period, a local cache to store mappings between users and their namespaces. [Use a lower value in case of flaky etcd server connections.](https://github.com/projectcapsule/capsule-proxy/issues/174)
rolebindingsResyncPeriod: 10h
# -- Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector.
disableCaching: false
# -- Enable RoleBinding reflection. Namespaced resource LIST grants require the reflection.proxy.projectcapsule.dev/enabled label.
roleBindingReflector: false
# -- Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate, XForwardedClientCert]
authPreferredTypes: "BearerToken,TLSCertificate"
# -- QPS to use for interacting with Kubernetes API Server.
clientConnectionQPS: 20
# -- Burst to use for interacting with kubernetes API Server.
clientConnectionBurst: 30
# -- Enable Pprof for profiling
pprof: false
# -- CIDR ranges of trusted proxies allowed to make requests to the proxy
trustedProxyCidrs: []
The following options are available for the Capsule Proxy Controller:
--allowed-paths strings URL paths which are not inspected by capsule-proxy (still require valid authentication) (default [/api,/apis,/version])
--auth-preferred-types string Authentication types to be used for requests. Possible Auth Types: [BearerToken, TLSCertificate, XForwardedClientCert]
First match is used and can be specified multiple times as comma separated values or by using the flag multiple times. (default "[TLSCertificate,BearerToken]")
--capsule-configuration-name string Name of the CapsuleConfiguration used to retrieve the Capsule user groups names (default "default")
--client-connection-burst int32 Burst to use for interacting with kubernetes apiserver. (default 30)
--client-connection-qps float32 QPS to use for interacting with kubernetes apiserver. (default 20)
--disable-caching Disable the go-client caching to hit directly the Kubernetes API Server, it disables any local caching as the rolebinding reflector (default: false)
--enable-leader-election Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
--enable-pprof Enables Pprof endpoint for profiling (not recommend in production)
--enable-reflector Enable reflection for RoleBindings labelled reflection.proxy.projectcapsule.dev/enabled=true
--enable-ssl Enable the bind on HTTPS for secure communication (default: true) (default true)
--feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
ProxyAllNamespaced=true|false (ALPHA - default=false)
ProxyClusterScoped=true|false (ALPHA - default=false)
SkipImpersonationReview=true|false (ALPHA - default=false)
--ignored-impersonation-group strings Names of the groups which are not used for impersonation (considered after impersonation-group-regexp)
--ignored-user-group strings Names of the groups which requests must be ignored and proxy-passed to the upstream server
--ignored-username strings Usernames whose requests must be ignored and proxy-passed to the upstream server
--impersonation-group-regexp string Regular expression to match the groups which are considered for impersonation
--listening-port uint HTTP port the proxy listens to (default: 9001) (default 9001)
--metrics-addr string The address the metric endpoint binds to. (default ":8080")
--oidc-username-claim string The OIDC field name used to identify the user (default: preferred_username) (default "preferred_username")
--require-capsule-user Require Capsule identity membership for proxy grants; when false, authenticated non-Capsule callers may use matching GlobalProxySettings and Capsule is optional (default true)
--rolebindings-resync-period duration Resync period for rolebindings reflector (default 10h0m0s)
--ssl-cert-path string Path to the TLS certificate (default: /opt/capsule-proxy/tls.crt)
--ssl-key-path string Path to the TLS certificate key (default: /opt/capsule-proxy/tls.key)
--trusted-proxy-cidrs strings CIDR ranges of trusted proxies allowed to make requests to the proxy
--webhook-port int The port the webhook server binds to. (default 9443)
--xfcc-header-name string Name of the header inspected for forwarded client certificates (default "X-Forwarded-Client-Cert")
--zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
--zap-encoder encoder Zap log encoding (one of 'json' or 'console')
--zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic'or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
--zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
--zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
User scope and reflection
By default, options.requireCapsuleUser: true (--require-capsule-user=true)
requires callers to be recognized Capsule identities before receiving proxy
grants. Configure eligible users and groups in the CapsuleConfiguration selected
by options.capsuleConfigurationName (--capsule-configuration-name). See
User scope for the
CapsuleConfiguration example, including system:authenticated and administrator
exclusions.
For Capsule users, the following Helm values enable RoleBinding reflection and
exclude your administrators from proxy filtering:
options:
capsuleConfigurationName: default
requireCapsuleUser: true
ignoredUserGroups:
- platform-admins
ignoredUsernames:
- platform-admin
disableCaching: false
roleBindingReflector: true
Replace the example administrator group and username with your actual identities.
The corresponding exclusion flags are --ignored-user-group and
--ignored-username. Ignored identities retain their native Kubernetes RBAC
permissions; these settings do not grant administrator access.
options.roleBindingReflector: true sets --enable-reflector=true.
Reflection requires Capsule integration and caching, so keep
options.disableCaching: false (--disable-caching=false). For namespaced
resource LIST grants, also set
reflection.proxy.projectcapsule.dev/enabled: "true" on the Tenant’s
spec.additionalRoleBindings[].labels. See
RoleBinding reflection
for the Tenant example and access boundaries.
Global grants for non-Capsule users
To allow authenticated callers to receive matching GlobalProxySettings grants
without becoming Capsule users, use these Helm values:
options:
requireCapsuleUser: false
extraArgs:
- --feature-gates=ProxyClusterScoped=true
These values set --require-capsule-user=false and
--feature-gates=ProxyClusterScoped=true. A global rule must still match the
caller, resource, operation, and selected objects. This does not grant tenant
delegation or reflected access to non-Capsule callers. Capsule is optional in
this mode; see
Global grants for non-Capsule users
for startup behavior and supported operations.
Feature Gates
Feature Gates are a set of key/value pairs that can be used to enable or disable certain features of the Capsule Proxy. The following feature gates are available:
| Feature Gate | Default Value | Description |
|---|
SkipImpersonationReview | false | SkipImpersonationReview allows to skip the impersonation review for all requests containing impersonation headers (user and groups). DANGER: Enabling this flag allows any user to impersonate as any user or group essentially bypassing any authorization. Only use this option in trusted environments where authorization/authentication is offloaded to external systems. |
ProxyClusterScoped | false | ProxyClusterScoped allows to proxy all clusterScoped objects for all tenant users. These can be defined via ProxySettings |
2 - ProxySettings
Configure proxy settings for your tenants
The configuration for the Proxy is also declarative via CRDs. This allows both Administrators and Tenant Owners to create flexible rules.
GlobalProxySettings
GlobalProxySettings grants selected subjects read access to labelled, cluster-scoped resources through Capsule Proxy. A subject does not have to own a Tenant, which makes this resource useful for shared infrastructure and project-wide views of resources such as PersistentVolumes, StorageClasses, ResourcePools, and Capsule global quota resources.
The ProxyClusterScoped feature gate must be enabled on Capsule Proxy:
options:
extraArgs:
- --feature-gates=ProxyClusterScoped=true
These permissions exist only when a request is sent through Capsule Proxy. GlobalProxySettings does not create Kubernetes ClusterRole or ClusterRoleBinding objects, so the same subject may still receive Forbidden when connecting directly to the Kubernetes API server.
Subjects and resources
Every entry in spec.rules contains one or more subjects and clusterResources. Supported subject kinds are User, Group, and ServiceAccount. By default, callers must also be recognized Capsule identities. See User scope for eligibility and the --require-capsule-user=false option for global grants to authenticated non-Capsule users.
apiGroups and resources use plural resource names and follow the same general conventions as Kubernetes ClusterRole rules:
- Use an empty string (
"") for the core Kubernetes API group. - Use the API group name, such as
capsule.clastix.io, for grouped APIs. - Use
"*" to match every discovered API group or resource. - Only cluster-scoped resources can be selected. For example,
persistentvolumes are supported, while namespaced persistentvolumeclaims are not.
Supported operations
GlobalProxySettings supports only List and Get. Mutation operations such as Create, Update, Patch, and Delete must be granted with native Kubernetes RBAC and are not added by Capsule Proxy.
The effective mappings are:
Configured operations | Effective Kubernetes verbs | Request shape |
|---|
| Omitted | list, get | Collection and named-resource reads |
[List] | list, get | Collection and named-resource reads |
[Get] | get | Named-resource reads only |
[List, Get] | list, get | Collection and named-resource reads |
List also includes Get for backward compatibility. Older v1beta1 configurations could only specify List, while Capsule Proxy already intercepted both collection and named-resource routes. Existing rules therefore do not need to be rewritten to retain named-resource access.
Although both operations use the HTTP GET method, Kubernetes treats them as different verbs:
GET /api/v1/persistentvolumes is the Kubernetes list verb.GET /api/v1/persistentvolumes/<name> is the Kubernetes get verb.GET /apis/capsule.clastix.io/v1beta2/tenantowners is list.GET /apis/capsule.clastix.io/v1beta2/tenantowners/<name> is get.
For a collection request, Capsule Proxy combines the applicable selectors and adds them to the request as a label selector. For a named-resource request, the proxy first resolves the object and grants access only when its labels match an applicable selector. A rule never grants access to objects outside its selector.
Default LIST and GET example
When operations is omitted, both collection and named-resource reads are enabled. The following rule lets Alice and members of oidc:org:platform list solar PersistentVolumes and fetch a single matching PersistentVolume by name:
apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
name: solar-proxy-settings
spec:
rules:
- subjects:
- kind: User
name: alice
- kind: Group
name: oidc:org:platform
clusterResources:
- apiGroups:
- ""
resources:
- persistentvolumes
selector:
matchLabels:
capsule.clastix.io/tenant: solar
For example, both of these requests are covered:
kubectl get persistentvolumes
kubectl get persistentvolume pvc-0574499c-f01b-4a53-85f1-ddb002de9cae
The PersistentVolume itself must contain the selected label:
metadata:
labels:
capsule.clastix.io/tenant: solar
GET-only example
Use Get without List when a subject may retrieve a known object but must not enumerate the collection:
apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
name: solar-proxy-settings
spec:
rules:
- subjects:
- kind: User
name: alice
clusterResources:
- apiGroups:
- capsule.clastix.io
resources:
- tenantowners
operations:
- Get
selector:
matchLabels:
projectcapsule.dev/tenant: solar
This adds permission for kubectl get tenantowner alice when that object matches the selector, but it does not add permission for kubectl get tenantowners. Native Kubernetes RBAC may still grant additional access independently.
Multiple resources and selectors
A rule can select several resources, and a GlobalProxySettings can contain several rules. The following legacy-style List declarations enable both LIST and GET because of the compatibility mapping described above:
apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
name: global-proxy-settings
spec:
rules:
- subjects:
- kind: User
name: alice
clusterResources:
- apiGroups:
- capsule.clastix.io
resources:
- resourcepools
- globalcustomquotas
- globalresourcequotas
- tenantowners
operations:
- List
selector:
matchLabels:
projectcapsule.dev/tenant: green
- apiGroups:
- kyverno.io
resources:
- "*"
operations:
- List
- Get
selector:
matchLabels:
app.kubernetes.io/type: dev
ProxySetting
A ProxySetting is created in a namespace of a tenant, if it’s not in a namespace of a tenant it’s not regarded as valid. With the ProxySetting Tenant Owners can further improve the experience for their fellow tenant users.
apiVersion: capsule.clastix.io/v1beta1
kind: ProxySetting
metadata:
name: solar-proxy
namespace: solar-prod
spec:
subjects:
- kind: User
name: alice
proxySettings:
- kind: IngressClasses
operations:
- List
Primitives
This will be refactored
Namespaces are treated specially. A users can list the namespaces they own, but they cannot list all the namespaces in the cluster. You can’t define additional selectors.
The proxy setting kind is an enum accepting the supported resources:
- Nodes: Based on the NodeSelector and the Scheduling Expressions nodes can be listed
- StorageClasses: Perform actions on the allowed StorageClasses for the tenant
- IngressClasses: Perform actions on the allowed IngressClasses for the tenant
- PriorityClasses: Perform actions on the allowed PriorityClasses for the tenant
PriorityClasses
- RuntimeClasses: Perform actions on the allowed RuntimeClasses for the tenant
- PersistentVolumes: Perform actions on the PersistentVolumes owned by the tenant
The following operations belong to the legacy proxySettings field shown in this section. They are separate from the GlobalProxySettings clusterResources.operations mappings documented above.
Each Resource kind can be granted with several verbs, such as:
Special routes for kubectl describe
When issuing a kubectl describe node, some other endpoints are put in place:
api/v1/pods?fieldSelector=spec.nodeName%3D{name}/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/{name}
These are mandatory to retrieve the list of the running Pods on the required node and provide info about its lease status.
3 - Reflection
Leverage the RoleBinding Reflector to grant LIST capabilities to non-owners across namespaces.
Namespace RoleBinding reflection allows users and groups to list every Namespace in which they are referenced by a RoleBinding. The user does not need to be a Tenant Owner.
Enable the reflector in the Helm values:
options:
roleBindingReflector: true
Alternatively, pass the corresponding command-line argument:
Namespaces
The functionality of allowing access to other Namespaces is achieved by enabling the RoleBinding Reflector, which allows the Capsule Proxy to list the namespaces where a rolebinding mentions a user. First to make use of this feature you must enable it for the proxy:
For example, the following Role and RoleBinding allow alice to list Pods in the green-test Namespace. Because alice is referenced by the RoleBinding, Namespace reflection also allows her to see green-test when listing Namespaces through Capsule Proxy:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: green-test
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: alice-pod-reader
namespace: green-test
subjects:
- kind: User
name: alice
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Namespace reflection supports the following subject kinds:
The RoleBinding does not require a special label for Namespace reflection. All RoleBindings mentioning the requesting subject are considered. The label reflection.proxy.projectcapsule.dev/enabled: “true” is only required when reflecting thereferenced Role or ClusterRole permissions to other namespaced resource LIST requests.
GlobalProxySettings
The same behavior can be achieved by using GlobalProxySettings to enable further listing of Namespace resources. Note that this must use the label "kubernetes.io/metadata.name". This provides the user alice the ability to list the Namespaces green-test and green-prod without being Tenant Owner or having any other permissions on the Tenant:
apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
name: green-tenant-proxy-settings
spec:
rules:
- subjects:
- kind: User
name: alice
clusterResources:
- apiGroups:
- ""
resources:
- "namespaces"
selector:
matchExpressions:
- key: "kubernetes.io/metadata.name"
operator: In
values:
- green-test
- green-prod
Namespaced Items
For all namespaced items it’s possible to grant users LIST permissions within any Tenant namespace. They don’t have to be a Tenant Owner or anything. This is useful for example for operators, where they might also want to see all pods but are not directly owner on any Tenant. Reflection is resolved based on RoleBindings and the associated Roles and ClusterRoles (recommended).
The Role or ClusterRole must provide LIST permissions to the allowed resource(s). The RoleBindings considered for reflection must always use the label reflection.proxy.projectcapsule.dev/enabled: "true" to be considered for reflection. The following example shows how to grant a user LIST permissions on all pods within any Tenant namespace:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: custom:pod-viewer
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
reflection.proxy.projectcapsule.dev/enabled: "true"
name: custom-pod-viewer
namespace: green-test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: custom:pod-viewer
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: alice
This grants the user alice the ability to LIST all pods within any Tenant's namespace. The RoleBinding must be created in each namespace where you want to grant this permission, and it must reference a Role or ClusterRole that has the appropriate permissions.
With this ClusterRole you can grant LIST permission to all namespaced resources from the core API:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: custom:proxy-viewer
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
reflection.proxy.projectcapsule.dev/enabled: "true"
name: custom-viewer
namespace: green-test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: custom:proxy-viewer
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: alice
Provision RoleBindings
Provisioning the RoleBindings via Tenant Rules is the best way to ensure that the RoleBindings are created in all namespaces of a Tenant and also in any new namespace created within the Tenant. The following example shows how to provision a RoleBinding for the user joe in all namespaces of the solar tenant:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
rules:
- permissions:
bindings:
- clusterRoleName: 'custom:proxy-viewer'
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: joe
labels:
reflection.proxy.projectcapsule.dev/enabled: "true"
By default, any user can add further RoleBindings with the reflection.proxy.projectcapsule.dev/enabled: "true" label to any RoleBinding. To prevent that, you can deny it with another Enforcement rule:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
rules:
- permissions:
bindings:
- clusterRoleName: 'custom:proxy-viewer'
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: joe
labels:
reflection.proxy.projectcapsule.dev/enabled: "true"
enforce:
action: deny
metadata:
- apiGroups:
- "rbac.authorization.k8s.io/v1"
kinds:
- "RoleBinding"
labels:
reflection.proxy.projectcapsule.dev/enabled:
values:
- exp: ".*"
4 - Gangplank
Capsule Integration with Gangplank
Gangplank is a web application that allows users to authenticate with an OIDC provider and configure their kubectl configuration file with the OpenID Connect Tokens. Gangplank is based on Gangway, which is no longer maintained.
Prerequisites
For Authentication you will need a Confidential OIDC client configured in your OIDC provider, such as Keycloak, Dex, or Google Cloud Identity. By default the Kubernetes API only validates tokens against a Public OIDC client, so you will need to configure your OIDC provider to allow the Gangplank client to issue tokens. You must make use of the Kubernetes Authentication Configuration, which allows to define multiple audiences (clients). This way we can issue tokens for a gangplank client, which is Confidential, and a kubernetes client, which is Public. The Kubernetes API will validate the tokens against both clients. The Config might look like this:
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://keycloak/realms/realm-name
audiences:
- kubernetes
- gangplank
audienceMatchPolicy: MatchAny # This one is important
claimMappings:
username:
claim: 'email'
prefix: ""
groups:
claim: 'groups'
prefix: ""
Read More
Integration
We provide the option to install Gangplank alongside the Capsule Proxy. This allows users to authenticate with their OIDC provider and configure their kubectl configuration file with the OpenID Connect Tokens, which are valid for the Capsule Proxy Ingress. This way users can access the Kubernetes API through the Capsule Proxy without having to worry about the authentication and token management. To install gangplank, you must enable it:
Gangplank won’t just work out of the box. You will need to provide some configuration values, which are required for gangplank to work properly. These values are:
GANGPLANK_CONFIG_AUTHORIZE_URL: https://keycloak/realms/realm-name/protocol/openid-connect/authGANGPLANK_CONFIG_TOKEN_URL: https://keycloak/realms/realm-name/protocol/openid-connect/tokenGANGPLANK_CONFIG_REDIRECT_URL: https://gangplank.example.com/callbackGANGPLANK_CONFIG_CLIENT_ID: gangplankGANGPLANK_CONFIG_CLIENT_SECRET: <SECRET>GANGPLANK_CONFIG_USERNAME_CLAIM: The JWT claim to use as the username. (we use email in the authentication config above, so this should also be email)GANGPLANK_CONFIG_APISERVER_URL: The URL Capsule Proxy Ingress. Since the users probably want to access the Kubernetes API from outside the cluster, you should use the Capsule Proxy Ingress URL here.
When using the Helm chart, you can set these values in the values.yaml file:
gangplank:
enabled: true
config:
clusterName: "tenant-cluster"
apiServerURL: "https://capsule-proxy.company.com:443"
scopes: ["openid", "profile", "email", "groups", "offline_access"]
redirectURL: "https://gangplank.company.com/callback"
usernameClaim: "email"
clientID: "gangplank"
authorizeURL: "https://keycloak/realms/realm-name/protocol/openid-connect/auth"
tokenURL: "https://keycloak/realms/realm-name/protocol/openid-connect/token"
# Mount The Client Secret as Environment Variables (GANGPLANK_CONFIG_CLIENT_SECRET)
envFrom:
- secretRef:
name: gangplank-secrets
Now the only thing left to do is to change the CA certificate which is provided. By default the CA certificate is set to the Kubernetes API server CA certificate, which is not valid for the Capsule Proxy Ingress. For this we can simply override the CA certificate in the Helm chart. You can do this by creating a Kubernetes Secret with the CA certificate and mounting it as a volume in the Gangplank deployment.
gangplank:
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: token-ca
volumes:
- name: token-ca
projected:
sources:
- serviceAccountToken:
path: token
- secret:
name: proxy-ingress-tls
items:
- key: tls.crt
path: ca.crt
Note: In this example we used the tls.crt key of the proxy-ingress-tls secret. This is a classic Cert-Manager TLS secret, which contains only the Certificate and Key for the Capsule Proxy Ingress. However the Certificate contains the CA certificate as well (Certificate Chain), so we can use it to verify the Capsule Proxy Ingress. If you use a different secret, make sure to adjust the key accordingly.
If you directly use the CA certificate generated by cert-manager, your values will look like this:
gangplank:
enabled: true
config:
clusterCAPath: "/capsule-proxy/ca.crt"
volumeMounts:
- mountPath: /capsule-proxy/
name: proxy-ca
volumes:
- name: proxy-ca
projected:
sources:
- secret:
name: capsule-proxy
items:
- key: ca.crt
path: ca.crt