Turbocharging Nginx on Oracle Cloud ARM: Zero-Copy HTTPS with Kernel TLS (kTLS)

If you run high-concurrency web applications, media assets, or content management systems like WordPress on Oracle Cloud Infrastructure (OCI) Ampere Altra (ARM64) instances, you are already benefiting from high core counts and efficient compute.

However, serving static assets over HTTPS standardly incurs an invisible system tax: redundant memory copying and excessive CPU context switches.

By offloading symmetric TLS record processing directly to the Linux kernel via Kernel TLS (kTLS), you unlock true zero-copy sendfile() over HTTPS, slashing CPU overhead and maximizing throughput on Ubuntu 24.04 LTS.

---

The Hidden Bottleneck of User-Space HTTPS

In a traditional Nginx setup serving encrypted traffic:

[Disk / Page Cache] ──(Copy 1)──> [User Space: Nginx / OpenSSL]
                                           │
                                    (Encrypts Data)
                                           │
[Network Interface (NIC)] <──(Copy 2)── [Kernel Socket Buffer]

1. The kernel reads requested files from disk or page cache into user-space memory buffers.
2. OpenSSL in user space encrypts the payload block by block.
3. Encrypted buffers are copied across the user/kernel space boundary into socket buffers.
4. The network stack finally transmits the packets over the wire.

Because OpenSSL processes symmetric encryption in user space, the Linux kernel's high-performance sendfile() syscall is disabled for HTTPS. Every static asset—images, cached HTML, CSS, JavaScript, and binaries—triggers double memory copies and CPU cache invalidations.

---

How kTLS Solves the Problem

Kernel TLS cleanly separates the control plane from the data plane:

• Handshake (Control Plane): Remains entirely in user space. OpenSSL handles certificate validation, key exchange, and session negotiation.
• Data Plane (Encryption/Decryption): Once session keys are derived, OpenSSL passes them to the kernel via socket options (setsockopt(..., SOL_TLS, ...)). The socket switches to the tls Upper Layer Protocol (ULP).

[Disk / Page Cache] ──(In-Kernel Zero-Copy sendfile)──> [Kernel Crypto: ARM CE] ──> [NIC]

With kTLS active, Nginx issues a standard sendfile() syscall over HTTPS. The kernel reads directly from page cache, applies symmetric encryption (AES-GCM or ChaCha20-Poly1305) in-kernel using ARMv8 Neoverse N1 Cryptographic Extension instructions, and streams packets directly to the network interface. Zero user-space memory copies.

---

Prerequisites on Ubuntu 24.04 LTS (ARM64)

• OS: Ubuntu 24.04 LTS (Kernel 6.8+ with CONFIG_TLS=m enabled by default)
• Hardware: OCI Ampere Altra (ARMv8.2+ with hardware-accelerated AES/SHA Cryptographic Extensions)
• Software: Nginx with OpenSSL 3.x

---

Step-by-Step Implementation & Common Pitfalls

1. Load and Persist the Kernel Module

The tls module is included in Ubuntu 24.04 but must be loaded:

# Load module immediately
sudo modprobe tls

# Persist module across reboots
echo "tls" | sudo tee /etc/modules-load.d/ktls.conf

Verify that the module is loaded:

lsmod | grep tls

2. The Nginx Configuration Pitfall

Many online guides suggest adding this directive to /etc/nginx/nginx.conf:

ssl_conf_cmd Options KTLS;

On standard Ubuntu 24.04 builds, running nginx -t will often throw an error:

[emerg] unknown directive "ssl_conf_cmd" in /etc/nginx/nginx.conf
nginx: configuration file /etc/nginx/nginx.conf test failed

The Fix: Do not place ssl_conf_cmd in Nginx. Instead, configure OpenSSL 3.x system-wide so all Nginx worker processes automatically use kTLS.

3. Enable kTLS via OpenSSL 3.x Configuration

Open /etc/ssl/openssl.cnf:

sudo nano /etc/ssl/openssl.cnf

Step A: Locate the existing [openssl_init] block (around line 43) and add ssl_conf = ssl_sect:

[openssl_init]
providers = provider_sect
ssl_conf = ssl_sect

Step B: Scroll to the very bottom of the file and append the SSL section:

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
CipherString = DEFAULT:@SECLEVEL=2
Options = KTLS

Save and exit the file.

4. Configure Nginx and Restart

Ensure your /etc/nginx/nginx.conf has sendfile and modern TLS protocols enabled:

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    # ... remaining server configurations ...
}

Test configuration and restart the service:

sudo nginx -t && sudo systemctl restart nginx

---

Verifying kTLS in Production

Generate test HTTPS traffic against your server:

curl -k -o /dev/null https://127.0.0.1/

1. Check Kernel TLS Statistics

Inspect the kernel's real-time TLS statistics:

cat /proc/net/tls_stat

Key counters to watch:

• TlsTxSw: Increments with every TLS record encrypted via software crypto routines in the kernel.
• TlsCurrTxSw: Shows currently active kernel TLS transmit sockets.

2. Inspect Active Socket Offload

Run ss to verify that active connections use the TLS Upper Layer Protocol:

ss -ti '( sport = :443 or dport = :443 )'

Look for ulp:tls in the socket options output.

---

Performance Comparison

Metric	Standard TLS (User Space)	Kernel TLS (kTLS Zero-Copy)
Data Path	2 Buffer Copies (Kernel $\rightarrow$ User $\rightarrow$ Kernel)	0 Copies (Direct Page Cache $\rightarrow$ NIC)
sendfile() Syscall	Disabled for HTTPS	Fully Supported
CPU Overhead	High %usr (User space encryption)	Low %usr, shifts cleanly to %sys
Throughput	Memory bandwidth bound	Line-rate network bound

---

Summary

Enabling kTLS on Oracle Cloud ARM instances is a zero-cost optimization that yields substantial returns for static file delivery. By fixing the OpenSSL 3.x system configuration on Ubuntu 24.04, you eliminate user-space copy bottlenecks and allow the ARM Neoverse crypto engine to process encrypted traffic at native line rates.