Table of Contents
- Scanning Answers the Wrong Question
- The Configuration That Actually Matters
- Why Privileged Containers Are Not Contained
- Secrets in Images and Environments
- Network Policy Is Default-Allow
- Admission Control Beats Detection
- Runtime Detection Signals
- A Prioritised Hardening Order
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: Image scanning finds known vulnerabilities in packages. Container compromises overwhelmingly exploit runtime configuration — excessive privileges, mounted sockets, permissive network policy, and over-broad service account tokens. These are invisible to scanners.
Scanning Answers the Wrong Question
Your registry scans every image. The pipeline blocks builds with critical findings. The dashboard is green.
None of that tells you whether a container runs as root, whether it can escape to the host, what it can reach on the network, or what cloud permissions its service account holds. Those are the properties that determine what a compromise becomes, and they live in deployment manifests rather than in image layers.
The distinction is worth being precise about. Image scanning answers “does this image contain packages with known vulnerabilities?” That is a real and useful question. Container security requires also answering “if this process is compromised, what can it do?” — and the second question is where the consequential exposure sits.
This is not an argument against scanning. It is an argument that scanning is one control addressing one class of problem, and that treating it as container security leaves the larger surface unexamined. A container running a fully patched image as root with the host filesystem mounted is a scanning success and a security failure.
The Configuration That Actually Matters
The settings that determine blast radius, in rough order of severity:
Running as root. The default in many base images. A compromise inside the container starts with root in the container’s namespace, which makes several escape techniques and privilege escalation paths available that a non-root process cannot use.
Privileged mode. Disables essentially all container isolation. A privileged container is functionally a process on the host with a different filesystem view. Escape is trivial by design.
Host namespace sharing. Host network, host PID, or host IPC each remove a boundary. Host network access means the container reaches everything the node reaches, including cloud metadata endpoints and other pods regardless of network policy.
Mounted container runtime socket. A container with the runtime socket mounted can create other containers, including privileged ones. This is equivalent to root on the node and appears in monitoring tools and CI runners regularly.
Writable host path mounts. Mounting host directories, particularly system paths, provides a direct route to modifying the node.
Added capabilities. Individual Linux capabilities that permit specific privileged operations. Some are close to full privilege in effect.
Over-broad service account tokens. Automatically mounted tokens with permissions to read secrets or create workloads convert a container compromise into cluster access.
No resource limits. One container can exhaust node resources, which is availability impact rather than confidentiality and matters operationally.
Every item in that list is a field in a deployment manifest. None are detectable by inspecting an image.
Why Privileged Containers Are Not Contained
Worth understanding mechanically, because the term “container” implies isolation that privileged mode removes entirely.
Container isolation is constructed from kernel namespaces, control groups, capability restrictions, and mandatory access control. Privileged mode grants all capabilities, removes device restrictions, and disables the mandatory access control profile. What remains is a process sharing the host kernel with a different filesystem root — which is not a security boundary.
From a privileged container, an attacker can access host devices directly, mount the host filesystem, load kernel modules, and interact with the container runtime. Documented escape techniques are short and reliable.
The reason privileged containers persist is that they make certain workloads work without further thought — monitoring agents, storage drivers, and CI runners frequently request them because it resolves permission errors. Most of these needs are satisfiable with specific capabilities rather than full privilege, and the narrowing requires someone to determine which capabilities are genuinely needed.
The same reasoning applies to the runtime socket. A build system that needs to build images inside a container has alternatives that do not involve socket access — rootless builders exist specifically for this. Mounting the socket because it is the path of least resistance grants node-level control to whatever runs in that container.
Secrets in Images and Environments
Secret handling in containers fails in a few consistent ways.
Secrets baked into image layers. Even when removed in a later layer, they remain in the earlier one and are extractable from the image. Build arguments containing credentials end up in the image history.
Secrets in environment variables. Visible in the process environment, in orchestrator API responses, in crash dumps, in debugging output, and to anything that can read the pod specification. Environment variables are a convenient and leaky mechanism.
Secrets in the manifest. Committed to version control, visible to anyone with repository read access, which is usually a much broader group than should hold production credentials.
Base64 as protection. Orchestrator secret objects are frequently encoded rather than encrypted, and encoding is not encryption. Without encryption at rest enabled for the backing store, these are plaintext.
Better mechanisms: mount secrets as files from a secret manager rather than injecting environment variables, so they are not in the process environment. Use workload identity so the platform attests to the workload and issues short-lived credentials with no static secret at all. Enable encryption at rest for the orchestrator’s backing store. And scan images for embedded secrets in the pipeline, which catches the accidental cases.
Workload identity is the meaningful improvement, because it removes the secret rather than storing it more carefully.
Network Policy Is Default-Allow
A property that surprises teams: in most orchestrator installations, every pod can reach every other pod by default. There is no network isolation until you create policies establishing it.
This means a compromise in a low-sensitivity workload — a marketing site, a batch job — can reach the database, the internal API, and the administrative interface. The flat network problem that segmentation solved in traditional infrastructure reappears inside the cluster.
Establishing isolation requires a default-deny policy per namespace, then explicit allow rules for legitimate flows. The work is enumerating those flows, which is genuinely tedious and frequently reveals connections nobody documented.
Two additional points. Egress policy matters as much as ingress: a compromised pod with unrestricted outbound access can exfiltrate data and reach cloud metadata endpoints. And network policy requires a network plugin that enforces it — policies applied where the plugin does not support them are accepted silently and do nothing, which is a failure mode worth verifying explicitly.
Admission Control Beats Detection
The most effective container security control is refusing to run misconfigured workloads.
Admission control evaluates every workload against policy before it is created. A deployment requesting privileged mode, root user, or a host path mount is rejected at submission rather than detected afterwards. This eliminates the window between deployment and remediation, and it puts the feedback in front of the engineer who can fix it.
A baseline policy set that most clusters can adopt:
deny privileged: true
deny hostNetwork / hostPID / hostIPC
deny hostPath volume mounts (except a narrow allowlist)
deny runAsUser: 0
deny allowPrivilegeEscalation: true
deny mounting the container runtime socket
deny capabilities beyond a minimal set
require resource requests and limits
require readOnlyRootFilesystem where feasible
require automountServiceAccountToken: false unless neededImplementation advice from teams that have done this: start in audit mode, which reports violations without blocking, and measure how much existing workload would fail. That number is usually large. Then enforce on new namespaces while granting time-bounded exceptions for existing workloads, tracked as remediation items rather than permanent carve-outs.
The last line of that policy deserves emphasis. Service account tokens are mounted automatically by default, and most workloads never call the orchestrator API. Disabling automatic mounting removes a cluster access path from every pod that does not need it, which is nearly all of them.
Runtime Detection Signals
Prevention is incomplete, so detection matters. Behaviours worth alerting on:
| Signal | Why it matters |
|---|---|
| Shell spawned inside a running container | Containers should not receive interactive shells in production |
| Package manager execution at runtime | Images should be immutable; installation indicates compromise |
| Unexpected outbound connection | Reaching a destination not in the expected set |
| Write to an unexpected filesystem path | Read-only root filesystems make this loud |
| Access to the cloud metadata endpoint | Frequently credential theft |
| New process not present in the image | Strong signal of injected code |
| Privilege escalation attempt | Capability use inconsistent with the workload |
The first two are unusually high-value because they have near-zero legitimate occurrence in production. A shell in a production container is either an engineer bypassing process or an attacker, and both warrant knowing about.
The general principle that makes runtime detection tractable: containers should be immutable and predictable. A container runs one known process set, writes to known paths, and connects to known destinations. That predictability makes deviation detectable in a way that general-purpose host monitoring cannot match.
A Prioritised Hardening Order
Ordered by risk reduction per unit of effort:
Disable automatic service account token mounting where not needed. One field, removes cluster API access from most pods.
Enforce non-root execution. Requires image changes in some cases and eliminates a large class of escalation.
Reject privileged and host namespace workloads via admission control. Audit first, then enforce.
Block runtime socket mounts. Equivalent to node root and rarely genuinely necessary.
Default-deny network policy per namespace, then allow known flows. Highest effort in this list and the largest reduction in lateral movement.
Read-only root filesystems with explicit writable volumes. Makes filesystem write detection meaningful.
Move secrets out of environment variables to mounted files or workload identity.
Add runtime detection for shell spawning, package manager use, and metadata endpoint access.
Continue image scanning. Still useful for known vulnerabilities and no longer the whole programme.
Note that scanning appears last rather than absent. It addresses a real problem and it is the control most organisations already have, which is why the items above it represent the available improvement.
Common Pitfalls
Treating image scanning as container security. It examines packages, not runtime privileges.
Privileged mode as a shortcut for permission errors. Grants far more than the specific capability needed.
Mounting the runtime socket. Equivalent to root on the node.
Assuming pods are network-isolated. Default is allow-all between pods.
Secrets in environment variables. Visible in far more places than expected.
Applying network policies without a supporting plugin. Accepted silently and enforced not at all.
Enforcing admission policy without an audit phase. Breaks existing workloads and loses organisational support.
Conclusion
Container security is mostly configuration security. The image contents matter and the deployment manifest determines what a compromise becomes.
The highest-return work is admission control that rejects privileged mode, host namespace sharing, root execution, and runtime socket mounts — enforced at submission so misconfiguration cannot reach the cluster. Alongside that, disabling automatic service account token mounting removes cluster API access from the majority of pods that never use it.
Then establish network isolation, because the default is that every pod reaches every other pod, which reproduces the flat network problem inside the cluster. Move secrets out of environment variables toward workload identity. And add runtime detection for the behaviours with no legitimate production occurrence — a shell in a container, a package manager running, a request to the metadata endpoint.
Keep scanning. Recognise that a clean scan describes the packages and says nothing about the privileges.
Frequently Asked Questions
Is image scanning still worth doing? Yes. It addresses known vulnerabilities in dependencies, which is a real risk. It simply does not address runtime configuration, which is where most container compromises originate.
How can workloads that genuinely need privileges be handled? Determine the specific capabilities required and grant those rather than full privilege. Where full privilege is unavoidable, isolate the workload on dedicated nodes with restricted network access and elevated monitoring.
Are containers a security boundary? Weaker than virtual machines, since the kernel is shared. Adequate for isolating workloads of similar trust level. For genuinely untrusted code, stronger isolation — dedicated nodes, sandboxed runtimes, or virtual machines — is appropriate.
What does read-only root filesystem break? Applications writing temporary files to unexpected locations. The fix is explicit writable volumes for the paths that need them, which also makes any other write attempt a clear signal.
Is a service mesh a security control? It provides mutual TLS and identity-based authorisation between services, which is genuinely valuable. It also adds substantial operational complexity and resource overhead. Network policy delivers a large share of the isolation benefit at lower cost.
How should distroless or minimal base images be evaluated? They reduce attack surface meaningfully — fewer packages means fewer vulnerabilities and no shell for an attacker to use. They also complicate debugging, which is a real trade-off and generally worth accepting for production images.
Where should the hardening effort begin? Admission control in audit mode. It costs little, breaks nothing, and produces an accurate picture of how much of your existing workload is misconfigured. That inventory determines everything else.
