Why Kubernetes Secrets Are Not Enough for Enterprise Infrastructure

Kubernetes has become the default environment for running many backend services, so it is natural that teams start with native Kubernetes primitives. Secrets are one of those primitives. They are convenient, familiar, and easy to integrate into a deployment model.

The problem starts when a Kubernetes Secret is treated as a complete secrets management system.

A Secret is primarily a delivery mechanism. It can make sensitive data available to a workload, but it does not provide a full lifecycle model. It does not automatically define ownership. It does not decide how long a credential should remain valid. It does not rotate external passwords by itself. It does not provide a universal policy layer across platforms. And it does not give you the same lease, revocation, or audit model that a dedicated secrets management system can provide.

There is also a basic security detail that is often underestimated: base64 is not encryption. Without proper encryption at rest, access to the Kubernetes API or etcd can become access to the secret value itself.

That does not mean Kubernetes Secrets cannot be used in production. They can. But they should be used deliberately: with encryption at rest, strict RBAC, namespace isolation, careful pod access controls, and a clear understanding of what they do and do not solve.

In an enterprise environment, the core question is not “where do we store the secret?” The better question is: “how do we manage access to this secret throughout its lifecycle?”

The IngressNightmare vulnerability chain is a useful example because it shows how secrets exposure often happens through architecture, not through the Secret object alone.

The important point is not simply that a vulnerability existed in an ingress component. The deeper issue is what happens when a component with broad permissions becomes compromised.

Ingress controllers often need access to TLS materials because they terminate or route traffic for multiple services. If such a component has wide access to Secrets across namespaces, the blast radius becomes large. A single vulnerable component can turn into access to many unrelated secrets.

This is exactly why least privilege matters in Kubernetes.

Infrastructure components are often granted broad permissions because it is operationally convenient. Broad access reduces friction during setup and avoids edge-case failures. But these permissions become dangerous when the component is compromised.

First, do not treat cluster-level access as harmless just because it belongs to an infrastructure component. Infrastructure components are also software, and software can have vulnerabilities.

Second, design secret access around the smallest practical blast radius. A service, controller, or workload should only be able to access the secrets required for its own responsibility. Anything broader needs a clear justification, monitoring, and a plan for reducing that scope over time.

The same mistakes appear repeatedly in Kubernetes environments.

The first mistake is placing secrets too close to code and build artifacts. A database password or API token may start as a temporary configuration value, then end up in an application config, Docker image, CI log, registry, or environment variable without a real lifecycle. After that, the credential becomes hard to find, hard to rotate, and hard to prove safe.

The second mistake is reusing one identity across multiple services. A team copies a working deployment template, reuses the same service account, and gradually that account becomes a shared credential for several workloads. This may look harmless at first, but it increases the blast radius of every incident. If one workload is compromised, the attacker inherits access that was never intended for that workload alone.

The third mistake is relying on long-lived credentials. Static external keys, database passwords, old service tokens, and certificates without automated renewal can remain valid for months or years. Even when Kubernetes itself supports more modern token patterns, older integrations often continue to depend on static credentials.

These problems usually do not create visible incidents immediately. They accumulate as security debt. Then one vulnerability, one leaked token, or one expired certificate turns that debt into an infrastructure failure.

A centralized model can work when the infrastructure is small.

One team issues certificates. Other teams submit requests. The process is controlled from one place. Everyone knows who owns the system.

This model breaks down when the number of services grows into the hundreds or thousands. Every internal API, database platform, Kubernetes service, PaaS environment, and data processing platform needs its own certificates. Each renewal, parameter change, replacement, and incident creates another operation for the central team.

At that point, the central team becomes a bottleneck.

Product teams wait for certificates. Infrastructure engineers spend time on repetitive operations. Manual work increases the chance of mistakes. And the risk of an expired certificate grows with every service that depends on the same request-driven process.

A better model is service-oriented PKI.

In this model, the central platform or security team still owns the trust chain, policies, templates, and guardrails. But service-owning teams get a standardized way to manage certificate lifecycle within those rules. They should not need to manually request every routine certificate operation. The safe path should be automated, observable, and easy enough that teams do not need to invent workarounds.

The ownership model is just as important as the technical implementation. Certificate lifecycle has to become part of service ownership, not a queue owned entirely by a central team.

The goal is not simply to issue a certificate inside Kubernetes. The goal is to design a model that survives real enterprise infrastructure: different clusters, different platform teams, different maturity levels, different network constraints, and different deployment patterns.

One option is to use a cluster-level controller. This can work well in many scenarios, especially when certificates are managed as Kubernetes resources and stored in Kubernetes Secrets.

But this is not the only possible model.

For heterogeneous environments, it can be useful to move certificate issuance and renewal logic closer to the workload. Depending on the platform, this can be implemented through an entrypoint wrapper, an init container, or a sidecar. The component authenticates the workload, requests a certificate from an internal CA, writes the certificate where the application expects it, and renews it without manual involvement.

This approach is not a universal replacement for cert-manager. It solves a different ownership problem.

If all certificate lifecycle logic is tied to a single cluster-level controller, then the Kubernetes platform team becomes responsible for operating that controller everywhere. That can be too rigid for an organization where clusters evolve differently.

A workload-adjacent model gives more flexibility. Kubernetes provides the execution environment, while certificate lifecycle remains part of the broader PKI service model. This makes it easier to roll out gradually, adapt to different platforms, and keep responsibility understandable for service teams.

In the architecture discussed here, the internal CA was based on step-ca. Open-source CA systems are useful because they can be adapted to internal workflows, but they also require real engineering ownership. At enterprise scale, a CA is not just a tool you install. It becomes an internal platform product with policies, templates, integrations, monitoring, testing, and operational support.

Vault can act as a control plane for access to secrets and credentials.

For some data, Vault may store static key-value secrets. For other systems, it can issue dynamic credentials with TTLs, leases, and revocation. This distinction is important. A static password copied into a configuration file can live for years. A dynamic credential is expected to expire, be renewed, or be replaced.

Instead of asking “where is this password stored?”, the system can ask: who requested the credential, under which policy, for which service, for how long, and what happens when the lease is revoked?

That model creates a better audit trail and reduces the time window in which a leaked credential remains useful.

But there is a nuance: using an external secrets manager does not automatically remove secrets from etcd. If the integration simply synchronizes external secrets into Kubernetes Secret objects, then a copy still exists in the Kubernetes API data store. That copy still needs encryption at rest, RBAC protection, namespace isolation, and operational discipline.

To reduce dependence on etcd, the delivery model matters. Runtime injection, Vault Agent, CSI-based delivery, or similar mechanisms can help avoid leaving long-lived copies inside Kubernetes Secrets. The right choice depends on the workload, platform maturity, operational constraints, and failure modes.

The broader architectural shift is this: secret management is not only storage. It is access management, lifecycle management, policy enforcement, and blast-radius control.

The target of the certificate automation initiative was to automate the lifecycle of approximately 95% of internal TLS certificates in scope. The work covered roughly 2,000 certificates across key platforms, including PaaS, DBaaS, and compute platforms used for analytics and machine learning workloads.

The most important effect was reducing the risk of incidents caused by expired certificates.

An expired certificate rarely looks like a small isolated issue in a large infrastructure. It can break service-to-service communication, stop internal platforms, degrade user-facing workflows, and create business impact. The larger the infrastructure, the harder it becomes to rely on manual reminders and request queues.

The second effect was a change in ownership.

Service-owning teams gained a controlled way to manage certificate lifecycle through standardized tools. The central team still controlled policies, trust chains, and security requirements, but it no longer had to act as the manual operator for every routine certificate operation.

The third effect was scalability.

Large companies do not have one perfectly uniform infrastructure. Platforms grow organically. Some clusters are newer, some are older. Some teams have strong Kubernetes maturity, others operate mixed environments. A certificate automation model has to work in this reality, not only in an ideal target architecture.

That is why the implementation strategy matters. It should allow gradual rollout, support different environments, and keep the ownership model understandable.

The effectiveness of a secrets or certificate management platform should not be measured by the presence of a specific tool alone.

Having Vault, step-ca, cert-manager, or any other component does not automatically mean the architecture is working. The better metrics are operational.

The first metric is the share of manual operations. If a team still needs to create a ticket, wait for approval, manually copy files, and coordinate renewal through chat messages, the main risk has not been removed.

The second metric is issuance and rotation speed. A safe path should be fast enough that teams prefer using it. If getting a credential takes days, engineers will eventually search for shortcuts.

The third metric is automation coverage. For certificate management, high coverage matters because forgotten exceptions are often where incidents happen. A few old certificates outside the common process can become the next outage.

The fourth metric is auditability. The platform should be able to answer who received a certificate or secret, when it was issued, under which policy, and for which service.

The fifth metric is blast radius. If one workload is compromised, the impact should stay close to that workload’s responsibility. If one service account opens access to unrelated namespaces, databases, or platforms, the access model is too broad.

For certificates, lifetime and renewal behavior are also important. Revocation is not always instant in practice because it depends on CRLs, OCSP behavior, and client implementation. In many internal environments, short-lived automatically renewed certificates are more practical than relying only on manual revocation after an incident.

The first recommendation is to design security as part of the platform from the beginning.

It is much harder to retrofit a secrets management model after the infrastructure has already grown to hundreds of services. By that point, teams have created exceptions, copied templates, reused service accounts, and built integrations around assumptions that are difficult to unwind.

The second recommendation is not to treat Kubernetes Secrets as a complete enterprise secrets platform. They are useful, but they need to be supported by encryption at rest, least-privilege RBAC, auditability, rotation, and often an external secrets manager or PKI platform.

The third recommendation is to apply least privilege early. Infrastructure components should not receive broad access “just in case.” Temporary access exceptions have a way of becoming permanent.

The fourth recommendation is to design around ownership. It is not enough to ask which controller or secret manager to install. Teams also need to decide who owns the policy, who owns the integration, who responds to failures, who explains the model to service teams, and how the system behaves across different generations of infrastructure.

The final recommendation is automation.

Manual certificate and token management eventually fails in predictable ways: an expired certificate, a forgotten key, an over-permissioned service account, or a credential that nobody knows how to rotate safely.

Kubernetes security is not a single feature. It is an architectural discipline. The earlier a company builds secrets and certificate lifecycle management into the platform, the less it pays later in incidents, manual work, and operational risk.