Turbocharging Nginx on Oracle Cloud ARM: Zero-Copy HTTPS with Kernel TLS (kTLS)
If you run high-concurrency web applications, media assets, or content management systems like WordPress on Oracle Cloud Infrastructure (OCI) Ampere Altra (ARM64) instances, you are already benefiting from high core counts and efficient compute.
However, serving static assets over HTTPS standardly incurs an invisible system tax: redundant memory copying and excessive CPU context switches.
By offloading symmetric TLS record processing directly to the Linux kernel via Kernel TLS (kTLS), you unlock true zero-copy sendfile() over HTTPS, slashing CPU overhead and maximizing throughput on Ubuntu 24.04 LTS.
The Hidden Bottleneck of User-Space HTTPS
In a traditional Nginx setup serving encrypted traffic:
[Disk / Page Cache] ──(Copy 1)──> [User Space: Nginx / OpenSSL]
│
(Encrypts Data)
│
[Network Interface (NIC)] <──(Copy 2)── [Kernel Socket Buffer]- The kernel reads requested files from disk or page cache into user-space memory buffers.
- OpenSSL in user space encrypts the payload block by block.
- Encrypted buffers are copied across the user/kernel space boundary into socket buffers.
- The network stack finally transmits the packets over the wire.
Because OpenSSL processes symmetric encryption in user space, the Linux kernel's high-performance sendfile() syscall is disabled for HTTPS. Every static asset—images, cached HTML, CSS, JavaScript, and binaries—triggers double memory copies and CPU cache invalidations.
How kTLS Solves the Problem
Kernel TLS cleanly separates the control plane from the data plane:
- Handshake (Control Plane): Remains entirely in user space. OpenSSL handles certificate validation, key exchange, and session negotiation.
- Data Plane (Encryption/Decryption): Once session keys are derived, OpenSSL passes them to the kernel via socket options (
setsockopt(..., SOL_TLS, ...)). The socket switches to thetlsUpper Layer Protocol (ULP).
[Disk / Page Cache] ──(In-Kernel Zero-Copy sendfile)──> [Kernel Crypto: ARM CE] ──> [NIC]With kTLS active, Nginx issues a standard sendfile() syscall over HTTPS. The kernel reads directly from page cache, applies symmetric encryption (AES-GCM or ChaCha20-Poly1305) in-kernel using ARMv8 Neoverse N1 Cryptographic Extension instructions, and streams packets directly to the network interface. Zero user-space memory copies.
Prerequisites on Ubuntu 24.04 LTS (ARM64)
- OS: Ubuntu 24.04 LTS (Kernel 6.8+ with
CONFIG_TLS=menabled by default) - Hardware: OCI Ampere Altra (ARMv8.2+ with hardware-accelerated AES/SHA Cryptographic Extensions)
- Software: Nginx with OpenSSL 3.x
Step-by-Step Implementation & Common Pitfalls
1. Load and Persist the Kernel Module
The tls module is included in Ubuntu 24.04 but must be loaded:
# Load module immediately
sudo modprobe tls
# Persist module across reboots
echo "tls" | sudo tee /etc/modules-load.d/ktls.confVerify that the module is loaded:
lsmod | grep tls2. The Nginx Configuration Pitfall
Many online guides suggest adding this directive to /etc/nginx/nginx.conf:
ssl_conf_cmd Options KTLS;On standard Ubuntu 24.04 builds, running nginx -t will often throw an error:
[emerg] unknown directive "ssl_conf_cmd" in /etc/nginx/nginx.conf
nginx: configuration file /etc/nginx/nginx.conf test failedThe Fix: Do not place ssl_conf_cmd in Nginx. Instead, configure OpenSSL 3.x system-wide so all Nginx worker processes automatically use kTLS.
3. Enable kTLS via OpenSSL 3.x Configuration
Open /etc/ssl/openssl.cnf:
sudo nano /etc/ssl/openssl.cnfStep A: Locate the existing [openssl_init] block (around line 43) and add ssl_conf = ssl_sect:
[openssl_init]
providers = provider_sect
ssl_conf = ssl_sectStep B: Scroll to the very bottom of the file and append the SSL section:
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
CipherString = DEFAULT:@SECLEVEL=2
Options = KTLSSave and exit the file.
4. Configure Nginx and Restart
Ensure your /etc/nginx/nginx.conf has sendfile and modern TLS protocols enabled:
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
# ... remaining server configurations ...
}Test configuration and restart the service:
sudo nginx -t && sudo systemctl restart nginxVerifying kTLS in Production
Generate test HTTPS traffic against your server:
curl -k -o /dev/null https://127.0.0.1/1. Check Kernel TLS Statistics
Inspect the kernel's real-time TLS statistics:
cat /proc/net/tls_statKey counters to watch:
TlsTxSw: Increments with every TLS record encrypted via software crypto routines in the kernel.TlsCurrTxSw: Shows currently active kernel TLS transmit sockets.
2. Inspect Active Socket Offload
Run ss to verify that active connections use the TLS Upper Layer Protocol:
ss -ti '( sport = :443 or dport = :443 )'Look for ulp:tls in the socket options output.
Performance Comparison
| Metric | Standard TLS (User Space) | Kernel TLS (kTLS Zero-Copy) |
|---|---|---|
| Data Path | 2 Buffer Copies (Kernel $\rightarrow$ User $\rightarrow$ Kernel) | 0 Copies (Direct Page Cache $\rightarrow$ NIC) |
| sendfile() Syscall | Disabled for HTTPS | Fully Supported |
| CPU Overhead | High %usr (User space encryption) |
Low %usr, shifts cleanly to %sys |
| Throughput | Memory bandwidth bound | Line-rate network bound |
Summary
Enabling kTLS on Oracle Cloud ARM instances is a zero-cost optimization that yields substantial returns for static file delivery. By fixing the OpenSSL 3.x system configuration on Ubuntu 24.04, you eliminate user-space copy bottlenecks and allow the ARM Neoverse crypto engine to process encrypted traffic at native line rates.
