Part 1 of 4: Understanding how AWS's minimal Linux distribution leverages UEFI Secure Boot and dm-verity to create a cryptographically verified system from hardware to application
As attackers grow more sophisticated - from DDoS attacks repeatedly downing BankID just recently, to more advanced intrusions like TietoEvry that destroy systems - we face a crucial dilemma. How do we secure platforms for education, healthcare, and vital public services without sacrificing modern development practices that will keep Europe competitive?
At Molnett, we're building a European serverless platform where developers simply deploy containers while still getting strong hardware-backed VM isolation on hardware we directly control. But this architecture is only as secure as its foundation. We evaluated numerous operating systems for our infrastructure, searching for one that could withstand both today's threats and tomorrow's compliance requirements. We quickly zoomed in on Bottlerocket not just for what it offered out of the box, but for its fundamental architecture that allowed us to extend it for our specific security needs. AWS's Bottlerocket Linux demonstrates that we can have both: a security posture fit for high compliance workloads and also fully enable modern development practices.
This four-part series will explore why Bottlerocket's security model made it the clear choice and how we've enhanced it for for our specific use-case. Firstly, we'll examine how Bottlerocket uses UEFI Secure Boot and dm-verity to create a foundation where compromise is cryptographically detectable (and will fail secure). In later posts we will cover our TPM contributions to Bottlerocket, runtime security, and Bottlerocket's atomic updates.
Understanding UEFI Secure Boot
Secure Boot is a UEFI specification that ensures only trusted software can execute during the boot process. It's fundamentally a public key infrastructure (PKI) system built into your computer's firmware that validates digital signatures on boot components before allowing them to run.
Secure Boot's power comes from its active enforcement model. Unlike security measures that simply log violations, Secure Boot halts the boot process immediately if any verification fails. Each component is verified in real-time before it executes, creating a chain of responsibility where verified components become responsible for verifying the next stage. This creates hardware-rooted trust, with UEFI firmware providing the initial verification anchor that everything else builds upon.
The Trust Anchors
The secure boot chain begins with several key databases stored in UEFI firmware. The Platform Key (PK) serves as the root of trust. Below that, the Key Exchange Key (KEK) is used to sign updates to the signature databases, most systems come preloaded with Microsoft's KEK and therefore also their db and dbx. The Signature Database (db) contains certificates and hashes of trusted boot components, while the Forbidden Signatures Database (dbx) maintains a revocation list of known-bad components that should never execute.
When your system boots, the UEFI firmware will only execute boot loaders that are either directly signed by a certificate in the db database or that present valid signatures chaining back to these trust anchors.
Bottlerocket's Self-Signed Approach
While most Linux distributions support both Microsoft-signed shims (this allows you to run in secure boot mode with the standard root of trust) and self-signed options, Bottlerocket takes a more focused approach: it only supports self-signed bootloader components.
When you build Bottlerocket using twoliter (their build tool), it generates a complete set of secure boot keys that you must install in your UEFI firmware if you want to activate Secure Boot. If you install AWS's distributed OS images, it relies on their secure boot trust chain. This architectural decision provides exclusive trust—only code signed with your (or AWS's) specific keys can boot. There are no shared trust roots, which means compromised third-party keys cannot be used to boot unauthorized code on your systems. You maintain complete control over the entire signing chain from hardware to kernel, and the reduced attack surface means fewer entities in the trust chain and therefore fewer potential compromise points.
The trade-off is operational complexity - you must securely manage your own signing keys and install them in every system's UEFI firmware if you build your own Bottlerocket variant. For organizations that need complete control over their boot chain, this trade-off is worth it.
The Multi-Layered Verification Architecture
Let's trace through each layer of Bottlerocket's boot verification:
Layer 1: UEFI to Shim
The boot process begins with the UEFI firmware verifying Bottlerocket's custom-signed shim bootloader. Because you've installed Bottlerocket's certificate in the UEFI db database, only this specific shim can execute.
Layer 2: Shim's Verification Protocols
Bottlerocket's shim contains an embedded vendor certificate and, crucially, installs UEFI verification protocols that persist after shim hands off control. As explained in Rod Smith's detailed analysis, shim doesn't just verify and load GRUB - it hooks into the UEFI LoadImage() and StartImage() protocols to provide ongoing verification services.
This design, which you can verify by examining Bottlerocket's signing script, works as follows:
- The shim itself is signed with the db certificate (the one you install in UEFI)
- When shim runs, it installs hooks into the UEFI LoadImage() protocol, intercepting all subsequent image loads
- These protocol hooks remain active even after shim transfers control to GRUB
- When GRUB later calls LoadImage() to load the kernel, shim's hooks verify the kernel's signature using the embedded vendor certificate
This protocol-based approach provides several advantages: it maintains compatibility with existing bootloaders that don't understand PE/COFF signatures, centralizes all signature verification logic in one place, and allows the verification mechanism to be updated by simply updating shim without modifying other boot components.
Layer 3: GRUB Configuration Verification
GRUB itself contains an embedded GPG public key that it uses to verify its configuration file (grub.cfg). This is crucial because the configuration contains critical security parameters: kernel command line parameters that control system behavior, the dm-verity root hash that ensures filesystem integrity, and various boot-time security policies. By cryptographically signing this configuration, Bottlerocket prevents attackers from modifying boot parameters even if they gain write access to the boot partition.
Layer 4: Kernel Verification via Shim Protocols
When GRUB needs to load the kernel it looks like this: GRUB calls the standard UEFI LoadImage() and StartImage() functions, which now use shim's installed protocols to verify signatures. The kernel, signed with the same vendor certificate embedded in shim, passes this protocol-based verification before execution. This design allows GRUB to maintain PE/COFF signature verification without understanding the signature format itself.
Layer 5: Direct Root Mount - A Critical Design Decision
Here's where Bottlerocket makes a crucial architectural departure from traditional Linux distributions: it doesn't use an initrd or initramfs.
Most distributions use an initial squashfs-based ramdisk that contains a minimal userspace environment, handles complex boot scenarios like encrypted disks and network boot, and eventually performs a "root pivot" to switch to the real root filesystem. This creates security challenges—the initrd runs in a privileged context and must be trusted, the root pivot process itself is a potential attack vector, and the additional complexity means more potential vulnerabilities.
The Encryption vs. Integrity Trade-off
Traditional Linux distribution security implementations rely on encrypted root partitions to protect against offline attacks (when an attacker removes the disk and attaches it to another system). However, this approach has significant limitations:
- Requires initrd: Decrypting the root partition necessitates an initrd with decryption tools, adding complexity and attack surface
- No runtime protection: Once decrypted and mounted, the filesystem is mutable—malware can modify system files freely
Bottlerocket takes a different approach with dm-verity and a read-only filesystem using EROFS, which provides superior security properties:
- Offline attack protection: While not encrypted, any tampering is cryptographically detectable - attackers can read but cannot modify without detection (we will discuss dm-verity more later)
- Runtime attack protection: The filesystem remains immutable and verified throughout operation—even root cannot modify system files
- No initrd needed: Direct mounting eliminates an entire layer of complexity and potential vulnerabilities
This demonstrates an important security principle: integrity protection with immutability often provides stronger security guarantees than encryption with mutability. An attacker who gains root on a traditional encrypted system can still install persistent malware. On Bottlerocket, even root cannot modify the verified filesystem.
Bottlerocket eliminates the initrd entirely. The kernel directly mounts the dm-verity-protected root filesystem, with all necessary drivers compiled in. This provides a simpler attack surface with no intermediate userspace to compromise, eliminates privilege transitions since there's no root pivot and therefore no opportunity for privilege escalation attacks, enables immediate verification as the root filesystem is verified from the first userspace instruction, and reduces complexity—fewer moving parts means fewer failure modes.
Layer 6: dm-verity Root Filesystem Protection
With the root filesystem directly mounted, Bottlerocket protects it using dm-verity, a kernel feature providing transparent integrity checking for block devices.
How dm-verity Works
dm-verity creates a Merkle tree (hash tree) of the entire filesystem at the block device level (so below the filesystem). First, every 4KB block of the filesystem is hashed. These hashes are then arranged in a tree structure where each parent node contains the hash of its children. The tree ultimately produces a single root hash that represents the entire filesystem. During runtime operation, every block read from disk is verified against this hash tree.
Any modification to any file changes the root hash, making tampering immediately detectable.
Bottlerocket stores the hash tree in a separate partition and embeds the root hash in the kernel command line. During boot, the kernel command line (containing the root hash) is protected by GRUB's GPG signature. The kernel then sets up dm-verity using this root hash, and from that point forward, every filesystem block read is verified against the hash tree. Any corruption or tampering causes an immediate kernel panic, preventing the system from running in a compromised state.
This creates a powerful property: once the dm-verity root filesystem is mounted, everything in that filesystem is inherently trusted and will continue to be trusted as it verifies it on every read.
Layer 7: Trusted Userspace
Because the entire root filesystem is verified through dm-verity, when systemd starts, we can trust it completely. All binaries, libraries, and configuration files have been cryptographically verified through the chain we've established.
The Complete Chain of Trust
Let's visualize the complete verification flow:
The complete signing process is implemented in Bottlerocket's build tooling, where you can examine the exact certificates and signing operations used.
Security Properties and Implications
This architecture provides several critical security properties:
Immutability
The combination of secure boot and dm-verity creates a system where the base OS cannot be modified without detection. Any change to system files, no matter how small, breaks the verification chain and prevents the system from booting.
Cryptographic Verification
Every component undergoes cryptographic verification before execution. Unlike security systems that rely on heuristics, behavioral analysis, or "trust on first use" policies, Bottlerocket's approach is deterministic—either the cryptographic signatures match, or the system doesn't boot.
Self-Contained Trust
By establishing and maintaining its own complete trust chain rather than relying on external trust roots like Microsoft's certificates, Bottlerocket ensures that your security posture isn't dependent on third parties. A compromise of Microsoft's signing infrastructure, for example, would have no impact on Bottlerocket systems using self-signed keys.
Fail-Secure Design
The system is designed to fail securely. If any verification step fails, the boot process halts completely. There's no "fallback to insecure mode," no "skip verification" option, and no way to bypass the security checks. This prevents scenarios where systems might run in a degraded security state without operators realizing it.
Supply Chain Attack Resistance
Consider a sophisticated attacker who compromises a container registry and injects malicious code into your base images. On a traditional system, this code might escape the container, modify the host OS, and establish persistence that survives reboots. On Bottlerocket, the immutable and verified base OS prevents any such modifications. The malicious code cannot alter system files, cannot install rootkits, and cannot survive a reboot. The system always returns to its known-good, cryptographically verified state.
Challenges and Trade-offs
While powerful, this approach comes with considerations:
Key Management
Organizations must develop robust processes for cryptographic key management. This includes securely generating and storing signing keys in hardware security modules or equivalent systems, establishing procedures for installing certificates in UEFI firmware across your fleet, creating key rotation strategies that don't disrupt operations, and planning for key compromise scenarios with clear revocation procedures.
Bottlerocket's build system supports both local key storage and AWS KMS for production signing operations, as shown in the signing implementation. At Molnett, we're investing heavily in automating these processes to make secure boot operationally viable at scale.
Operational Changes
The immutable root filesystem represents a fundamental shift in how systems are managed. Traditional approaches like SSH-ing into a system to install packages or modify configuration files simply don't work. Instead, all configuration must be done through Bottlerocket's API, user data at before first boot, or configuration files in designated writable partitions. System updates involve replacing the entire OS image rather than patching individual packages. While this requires rethinking operational procedures, it also eliminates entire classes of configuration drift and "snowflake server" problems.
Hardware Requirements
Bottlerocket's secure boot requires modern hardware with UEFI firmware - legacy BIOS systems are not supported. The hardware must also support custom secure boot keys, as some locked-down systems only accept manufacturer keys. Additionally, the UEFI implementation must properly support the secure boot specification, as some early or buggy implementations can cause issues.
Conclusion
Bottlerocket's secure boot implementation creates an unbreakable chain of trust from hardware to userspace. By combining UEFI Secure Boot with dm-verity and eliminating complex boot transitions, it establishes a foundation where the base operating system is extremely difficult to compromise and would require multiple vulnerabilities in key security components to compromise it without detection.
This foundation is critical for Molnett's security. While we also spend considerable effort ensuring our customers are isolated from each other, Bottlerocket ensures the platform underneath remains trustworthy regardless of what runs on top.
The secure foundation that Bottlerocket provides gave us confidence to build our sovereign cloud platform. But we needed more - hardware-backed encryption and compliance evidence. In the next post, we'll explore how we provided Bottlerocket upstream with Trusted Platform Module (TPM) capabilities for measured boot and PCR Signing Policies, showing how this solid foundation can be enhanced to meet the most demanding requirements.
This is part 1 of a 4-part series on Bottlerocket's security architecture and how we at Molnett uses it. Next: Extending Secure Boot with TPM and Measured Boot.