Secrets Management in the Cloud: Vault, Secrets Manager, and Kubernetes Secrets Without Leaks

Secrets management is not about choosing “Vault or Secrets Manager”; it is about controlling the entire secret lifecycle: creation, storage, delivery to the application, copies being made, rotation, revocation, and auditing.

A secret may be stored correctly but still leak after it is delivered: through CI logs, Helm values, a Docker image layer, an environment variable, a crash dump, kubectl describe, an etcd backup, or overly broad service account permissions. Therefore, you need to protect not only the storage system, but the secret’s entire path.

The tools serve different roles:

●        Vault — source of truth, access control, dynamic and short-lived secrets, detailed audit logging;

●        Cloud secrets manager — a managed secret store with IAM, KMS, rotation, and cloud auditing;

●        Kubernetes Secrets — a mechanism for delivering values within a cluster, but not a full-fledged secrets management strategy;

●        External Secrets Operator — a synchronizer that transfers a secret from an external source into Kubernetes, but does not eliminate Kubernetes-layer risks.

The core principle is that a secret must have a single managed source of truth, and all copies must have a defined lifetime, limited permissions, and a revocation path. If a secret ends up in Git, a CI log, a Docker layer, or a Helm release, it must be treated as compromised: fixing the file is not enough; rotation is required, along with verification that the old access no longer works.

What Secrets Management Is and Why It Matters

After the TL;DR, it is worth starting with the terminology. Secrets Management — refers to the management of sensitive technical values that provide access to systems, data, and infrastructure. This is not just about passwords: a secret can be any string, file, certificate, or key that an application, user, or pipeline uses to access a resource.

To avoid treating secrets as just passwords, it is useful to divide them into several groups:

The purpose of secrets management is not simply to remove a password from the code. A secret must be stored in a controlled location, have a clearly defined owner (the person responsible for it), be provided only to verified applications and users, have a limited scope, be updated regularly, be quickly revocable, and leave an audit trail.

Without such a model, secrets end up being treated like ordinary strings. They are copied into .env files, sent in chats, placed in CI/CD variables without rotation, added to Helm values, passed through environment variables, written to logs, or accidentally stored inside a Docker image. A team may think that “passwords are no longer in the code,” but in practice they remain in a dozen other places.

The consequences usually do not appear immediately. A leaked token can provide access to a database, a cloud account, a production API, or an internal service. If a secret is long-lived and is not tied to a specific role, environment, or workload, the blast radius increases: a single key opens access to more systems than the application actually needs.

Secret Owner

An important and often overlooked consideration is defining the secret owner. This is the person responsible for the secret: where it is used, how it is used, what it is used for, and for managing its lifecycle and rotation. Without an assigned owner, or when ownership is unclear (for example, assigned broadly to the DevOps team), a situation can eventually arise where something is missed during scheduled or unscheduled secret rotation and the application loses the access it requires.

The Secret Lifecycle: From Creation to Audit

In one of the previous chapters, we established the key point: a secret is not just a password or token that needs to be “put somewhere.” It is a value that moves through different environments, tools, and processes. That is why, going forward, it is important to look not only at where a secret is stored, but at its entire lifecycle.

If this lifecycle is not defined, any secrets manager can easily become a “secure place where a password was stored.” But leaks often happen not during storage, but before the value is written to Vault or a cloud secrets manager—or after it has already been issued to an application. A secret should therefore be treated as an object in motion: at each stage, the participants, permissions, copies, and risk model change.

First, it is worth examining the path up to the point where a secret is issued to an application:

The risk does not end after issuance. The secret enters the application process, is used in connections, and may appear in memory, logs, dumps, or diagnostics. The second part of the lifecycle is therefore no longer about “where to store it,” but about “how to use, update, revoke, and verify access securely”:

If only storage is controlled, the design remains fragile. A secrets manager reduces risk, but it does not automatically address issuance, copies in Kubernetes and CI/CD, consumer updates, or revocation of old values.

For example, a database password may be created by a pipeline and written to a cloud secrets manager; then an application receives access through an IAM role, the value appears in a pod as a file, and during rotation it is replaced and the old value is revoked. In this scenario, what matters is not the name of the tool, but how well the transitions between stages are controlled: who created the secret, who requested it, where a copy appeared, when it was updated, and whether it can be proven that the access was legitimate.

This is why tools should be compared not as competing “safes,” but as different layers of the same route.

One is responsible for the source of truth and access policies, another for synchronization, a third for delivering the value into Kubernetes, while the method used to pass it into a pod determines where the next copy of the secret will appear. The next chapter separates these roles and shows where the responsibility of Vault or a cloud secrets manager ends and where the risks of the Kubernetes layer begin.

Tool roles: storage, synchronization, and delivery are distinct layers

Kubernetes Secrets: in-cluster delivery

Kubernetes Secrets are useful when an application needs a standard in-cluster object: a file in a volume, a configuration value, or integration with a controller. However, a Kubernetes Secret should not automatically be treated as a complete secrets management strategy.

By default, base64 is encoding, not encryption. Without additional configuration, a secret is stored in the Kubernetes API and typically in etcd, and its security depends on etcd encryption, RBAC, service accounts, namespaces, pod access, backups, and Kubernetes auditing.

The main risk with Kubernetes Secrets is the broad read-access scope. If a role, operator, controller, or service account has access to secrets in a namespace, an external store no longer helps: a copy of the value exists inside the cluster and must be protected separately.

Vault: Source of Truth and Issuance Control

Vault is strongest where strict access control, short-lived or dynamic secrets, revocation, granular policies, and detailed auditing are required. Its model can be described as auth → policy → token → access: a workload authenticates, is assigned a policy, then receives a token with limited permissions, and only then gains access to a specific secret.

Vault’s advantage is controlled issuance. It can do more than store a static password: it can issue temporary credentials, limit their lifetime, revoke access, and record actions in audit logs.

However, Vault requires operational work: high availability, unsealing or auto-unsealing, backups, upgrades, policies, monitoring, and readiness to investigate incidents. If the team does not have the resources to maintain it, Vault may become not a security enhancement but a new critical platform without sufficient support.

Cloud secrets manager: a managed source of truth

A cloud secrets manager reduces some of the operational burden because it runs as a provider-managed service. In AWS, this may be a combination of Secrets Manager, KMS, IAM, resource policies, CloudTrail, and CloudWatch. In Google Cloud and Azure, Google Secret Manager and Azure Key Vault play a similar role.

This approach works well when the infrastructure primarily runs in a single cloud and the access model can be expressed through IAM, KMS/HSM, resource policies, and cloud auditing. The team gets a managed secret store, built-in integration with cloud services, and fewer tasks related to operating the secrets platform itself.

The main risk is dependence on the cloud permissions model. If IAM roles are too broad, accounts are poorly separated, and resource policies are configured “just in case,” a cloud secrets manager will not protect against legitimate but excessive access.

External Secrets Operator: synchronization, not protection in itself

External Secrets Operator is not a standalone secret store. Its role is to retrieve a value from an external source, such as Vault or a cloud secrets manager, and create or update a Kubernetes Secret in the required namespace.

This is useful: secrets do not need to be stored in Git or manually copied into the cluster. However, ESO does not eliminate risks at the Kubernetes layer. After synchronization, the value still exists as a Kubernetes Secret, which means it remains subject to RBAC, service accounts, etcd, pod access, auditing, and cluster backups.

For sensitive use cases, ESO permissions should be minimal: it should read only the required secrets from the external source and write only to a limited set of namespaces. Otherwise, the synchronizer becomes a mechanism for automatically replicating sensitive data across the cluster.

How these roles fit into a single architecture

In a working architecture, tools usually do not replace one another; they are combined into a chain:

1. Source of truth — Vault or a cloud secrets manager stores the original secret and manages access, rotation, and auditing.
   
   
2. Workload identity — an application, pod, CI/CD job, or service account proves that it is authorized to request the secret.
   
   
3. Synchronization or delivery — the secret is either requested directly or delivered via External Secrets Operator, Secrets Store CSI Driver, or another delivery mechanism.
   
   
4. Local copy — a Secret appears in Kubernetes, a file in a volume, an environment variable, or another way of passing the value to the process.
   
   

EU Cloud Infrastructure You Control

Run production workloads on dedicated resources across EU data centres. Transparent pricing, no hidden costs. Full control over compute, storage, and networking.

Explore Infrastructure



Predictable Virtual Machines for Production

Dedicated vCPUs, consistent performance, no noisy neighbours. Deploy and scale VMs in minutes. Full root access, fixed pricing.

View VM Plans



Up to €22,000 to Migrate Your Workloads

Cut OPEX when migrating to EU infrastructure. Get migration support, cost assessment, and infrastructure credits with no upfront commitment.

Apply for Bonus



Kubernetes with Full Control

Deploy production-ready clusters with control over node configuration, storage, and networking. Predictable costs and direct access to infrastructure.

Deploy Cluster


5. Application Use — the application connects to a database, API, or external service and must not write the secret to logs, dumps, or traces.
   
   
6. Rotation and revocation — the new value is delivered to consumers, the old one stops working, and the events are recorded in the audit log.

This leads to a practical conclusion: after a secret is passed to the next layer, it must be protected as a separate instance. A Kubernetes Secret cannot be considered “automatically secure” simply because the original is stored in Vault or a cloud secrets manager.

You can then move on to choosing an approach for each scenario. The architecture will differ for a small project, a Kubernetes cluster, a regulated workload, and CI/CD: in some cases a cloud secrets manager is sufficient, in others Vault is required, and in others Kubernetes Secrets and External Secrets Operator remain only the delivery layer.

How to choose an approach: a small project, a Kubernetes cluster, adjustable load, and CI/CD

Small project

For a small project, it usually makes more sense to start with a cloud secrets manager as the source of truth. If the application runs in a single cloud, access can be defined through IAM, while secrets are protected with KMS, access policies, auditing, and rotation of critical values.

In this scenario, you should avoid running your own Vault without an operations team. Vault may be more technically advanced and flexible, but it adds operational overhead: high availability, upgrades, policies, backups, monitoring, and incident response.

A minimal setup for a small project is: separate secrets by environment, least-privilege access, access logging, rotation of important keys, and a prohibition on storing secrets in Git, images, and CI logs. If there is a Kubernetes cluster, Kubernetes Secrets can be used as the delivery layer, and External Secrets Operator as a way to synchronize them from an external source.

Kubernetes Cluster

For a Kubernetes cluster, it is safer to use an external source of truth: Vault or a cloud secrets manager. In this model, a Kubernetes Secret remains a local copy that the application needs inside the cluster, but it should not be the only place where the secret is managed.

Key considerations here include etcd encryption, strict RBAC, separate service accounts, namespace isolation, Kubernetes API auditing, and prohibiting the storage of secrets in Git/Helm. External Secrets Operator or Secrets Store CSI Driver can be used for delivery, but they do not eliminate Kubernetes-layer risks.

Vault is needed when there are multiple clusters, access policies are complex, short-lived credentials are required, or a unified access model is needed across different environments. A cloud secrets manager may be sufficient if the cluster is tied to a single cloud and permissions can be conveniently defined through IAM.

Regulated workload

For a regulated workload, storage and delivery are not the only concerns; control must also be provable: who received the secret, under which policy, for how long, where a copy appeared, and when the old value stopped working.

This usually requires centralized management, detailed auditing, a KMS or HSM, separation of duties, short secret lifetimes, strict policies, and regular revocation. Vault is often justified when dynamic secrets, team or tenant isolation, fine-grained policies, and a unified access model outside a single cloud are required.

A cloud secrets manager may also be sufficient if the requirements are covered by cloud-native auditing, KMS/HSM, IAM, and access policies. However, in regulated scenarios, Kubernetes Secrets should remain only a controlled delivery point, not the primary store.

It is important not to overstate the role of External Secrets Operator here. It can be useful as a transport layer: retrieving a value from Vault, AWS Secrets Manager, Google Secret Manager, or Azure Key Vault and creating a Kubernetes Secret in the required namespace. However, ESO does not, by itself, demonstrate audit compliance or manage the lifetime of the copy. The operator’s permissions must be minimal: only read access to specific secrets in the external source and write access to a limited set of namespaces.

CI/CD

In CI/CD, the main risk is long-lived keys stored in pipeline settings. A better approach is to use credentials scoped to a specific job, separate dev/stage/prod environments, OIDC or workload identity, short-lived tokens, and automatic revocation after the job completes.

If a pipeline needs access to a cloud environment, database, or API, it is better not to store a permanent key in CI/CD variables. A more secure model is to grant temporary access for the duration of the task and record it in the audit trail.

Vault is needed when pipelines must obtain dynamic credentials for databases, cloud environments, and internal APIs. A cloud secrets manager is sufficient if CI/CD is integrated with cloud IAM and OIDC without permanent keys. Kubernetes Secrets are usually not a source for CI/CD, but they may appear at the deployment stage when secrets are delivered to the cluster through ESO or another mechanism.

The conclusion is simple: the choice is determined not by the size of the project or by the “most secure” tool, but by manageability. You need to consider least privilege, auditing, revocation speed, blast radius, and whether there is a team capable of maintaining the chosen model.

Where secrets actually leak: Git, CI/CD logs, images, Helm values, and environment variables

In the previous chapters, we separated the roles of the tools and chose an approach for each scenario. However, even a sound architecture built around Vault or a cloud secrets manager does not guarantee that a secret will not end up in a less secure location. Once a value is issued, it starts to live in pipelines, manifests, images, pods, logs, and diagnostics.

A typical example: a password is stored in AWS Secrets Manager, the pipeline retrieves it before deployment, and then accidentally prints it to the log because of echo, set -x, shell command tracing, or a curl error. The source is protected, but the copy has already become a plain string in CI/CD.

For infrastructure reviews, it is easier to check these locations as the route taken by copies of the secret:

●        Git. Removing the secret from the current file does not solve the problem. The value may remain in the commit history, forks, repository mirrors, and developers' local clones.

●        CI/CD logs. Secrets surface through debug output, command tracing, variable dumps, deployment errors, and overly verbose pipeline messages.

●        Container image layers. A secret in a Dockerfile, build args, or a temporary file may remain in a previous layer, even if the file was deleted later.

●        Helm values. Values files, templates, and release metadata can become a hidden store of secrets accessible through release history and deployment tools.

●        Environment variables. They are convenient for startup, but may end up in process dumps, crash reports, debug endpoints, and diagnostic utilities.

Kubernetes adds its own risk model on top of this. A secret can be exposed through kubectl logs, kubectl describe, the ability to exec into a pod, etcd backups, overly broad service accounts, and excessive permissions to read secrets in a namespace. This is no longer a Vault or cloud secrets manager problem, but a problem with copies after the secret has been issued.

The practical rule is simple: if a secret appears in Git, a CI/CD log, a Docker layer, a Helm release, or an environment variable, it should be considered compromised. Fixing the file or the pipeline is not enough. You need to revoke the old value, rotate it, update consumers, and verify that the previous access no longer works.

This is where the entire logic of secrets management comes together. Security is determined not only by where the original secret is stored, but also by how well its copies, delivery, use, rotation, revocation, and audit traces are controlled.

Conclusion

Secrets management works only when the secret’s entire path is controlled: from creation and storage to delivery to the application, copies, rotation, revocation, and auditing. Vault, cloud secrets managers, Kubernetes Secrets, and External Secrets Operator cover different parts of this path, but no single tool automatically protects all the other layers.

The practical test is simple: a secret must have an owner, a managed source of truth, a limited scope, a clear lifetime, and a verifiable revocation path. If a value ends up in Git, a CI/CD log, a Docker layer, a Helm release, or an environment variable, it must be considered exposed: fixing the file is not enough; the old access must be revoked and verified.

FAQ

Can Kubernetes Secrets be used on their own?

Yes, but only in limited scenarios: with etcd encryption enabled, strict RBAC, separate service accounts, access control for pods, and audit logging configured. A Kubernetes Secret is useful as an in-cluster delivery mechanism, but it does not replace a complete source of truth with rotation, revocation, and centralized auditing.

When do you need Vault, and when is a cloud secrets manager sufficient?

Vault is justified when you need dynamic secrets, short-lived access credentials, unified policies across different environments, and granular control over secret issuance. A cloud secrets manager is usually sufficient when the infrastructure runs in a single cloud and the requirements are covered by IAM, KMS, rotation, and cloud audit logging.

Does External Secrets Operator make Kubernetes Secrets more secure?

Not by itself. ESO synchronizes data from an external source into a Kubernetes Secret and helps keep secrets out of Git, but after synchronization the value still exists inside Kubernetes. This means it must be protected through RBAC, namespaces, service accounts, etcd encryption, auditing, and pod access controls.

Which is safer for a pod: an environment variable or a file in a volume?

Both options carry risks. Environment variables are more likely to end up in dumps, diagnostic output, and debugging tools. A file in a volume is easier to control through file system permissions and updates, but it is still a copy of the secret inside the pod. The choice depends on the application and the rotation model.

What should you do if a secret ends up in Git or a CI log?

Treat the secret as compromised. Revoke or rotate the value, update the consumers, verify that the old access no longer works, and address the cause of the leak: debug output, echo, set -x, insecure Helm values, build artifacts, or the absence of secret scanning.

How can you reduce the impact of a leak?

Reduce the blast radius: grant access only to a specific role, workload, and environment; separate dev, stage, and prod; use short-lived tokens; restrict service accounts and IAM roles; enable auditing; and regularly verify rotation. The shorter a secret’s lifetime and the narrower its scope, the easier it is to contain an incident.

Sources

1. OWASP Secrets Management Cheat Sheet

2. Kubernetes Documentation: Secrets

3. HashiCorp Vault Documentation: How Vault works

4. AWS Secrets Manager Documentation