Why ML-DSA Signing is Different

ML-DSA implements the Fiat-Shamir with Aborts construction, which uses rejection sampling as a core mechanism - a design choice rooted in lattice-based cryptography’s unique mathematical properties. Here’s what that means practically:

The Numbers: Rejection Probability and Expected Attempts

Here’s where benchmarking gets interesting. The acceptance probability - the chance a signing attempt succeeds on the first try - varies by ML-DSA parameter set:

What this tells us:

• ML-DSA-44 (compact variant) succeeds about 1 in 4 times per attempt
• ML-DSA-65 (NIST Category 3, most deployed) expects roughly 5 attempts on average
• ML-DSA-87 (highest security) actually has the best acceptance probability, requiring ~3.9 attempts

These aren’t guesses - they’re mathematically derived from the algorithm’s structure and parameters defined in FIPS-204 using Equation 5 from Li32, assuming a random bit generator (RBG) as specified in Section 3.6.1.

Understanding the Distribution

The expected number of attempts is only part of the story. Due to the geometric distribution of the rejection-sampling loop, we need to understand the “tail” of the distribution-what happens in worst-case scenarios.

Practical Implications for Constrained Devices

For battery-powered IoT devices and embedded systems, this variability matters significantly:

Best Practices for Benchmarking ML-DSA Signing

If you’re benchmarking ML-DSA implementations on constrained devices, don’t fall into the trap of reporting a single timing number. Here’s what to measure:

Comparing to Traditional Signatures

To illustrate why rejection sampling benchmarking is different, consider RSA or ECDSA:

• Signing time is deterministic: You can measure a single 2048-bit RSA signature and get the same runtime within microseconds every time
• Energy consumption is predictable: An ECDSA-P256 signature consumes nearly identical energy regardless of message or key
• Performance metrics are straightforward: Report a single timing number; it accurately represents all signing operations

The choice of metric dramatically affects system design. Budget for average-case and 1% of your operations will timeout. Budget for 99% case and you’re over-provisioning resources by 4-5x.

Conclusion

The rejection sampling in ML-DSA’s signing operations is a carefully engineered security feature, not a limitation. It’s fundamental to how lattice-based signatures provide provable security against known attacks. But it does require a thoughtfully different approach to performance evaluation than you might expect from traditional signature algorithms. It is worth to note that:

• Performance is probabilistic, not deterministic. A single timing measurement is meaningless. Instead, you need to understand the distribution of signing times.
• The expected overhead is manageable. Averaging 4-5 iterations for ML-DSA-65 is reasonable. The core signing operation (one iteration) executes in acceptable time on modern embedded hardware.
• You can predict and measure it precisely. Using the geometric distribution model and FIPS-204 parameters, you now have the mathematical framework to estimate signing time distributions without extensive benchmarking.
• System design must account for variability. Real-time systems, battery-powered devices, and time-sensitive protocols need to budget for 99th percentile cases, not average-case performance.
• Signing only. The mechanism applies only to the signing operation. This abort/retry mechanism mechanism doesn’t apply to key generation and verification.

Context: An Older Platform and Brownfield Devices

SmartFusion2 is not a new platform. It has been deployed in real products for years, often in long-lived industrial, infrastructure, and security-sensitive systems. This matters, because a large part of its relevance today comes from brownfield deployments, not greenfield designs.

In engineering terms, brownfield devices are systems that:

• Are already deployed or close to deployment
• Have fixed hardware constraints
• Cannot be redesigned freely without high cost or risk
• Must be extended, maintained, or upgraded in place

This is in contrast to greenfield designs, where hardware, software, and tooling choices can be made from scratch. For brownfield devices, the problem is rarely “design the perfect system.” Instead, it is:

• How to add new functionality without changing hardware
• How to modernise software workflows on top of legacy platforms
• How to introduce new security mechanisms without destabilising a proven system

SmartFusion2 fits squarely into this category. Many teams encounter it not because they would choose it today, but because it is already part of an existing product or certification boundary.

The approach described in this article is shaped by that reality: it assumes fixed hardware, aging tooling, and long product lifetimes, and focuses on making such systems workable and productive rather than ideal.

Philosophy: Software First, Hardware Fixed

The guiding idea behind this setup is simple:

Fix the hardware early, keep it minimal, and let software move fast.

SmartFusion2 allows deep hardware customisation, but recompiling FPGA designs is slow, license-gated, and unnecessary for early development. By freezing a small, known-good MSS configuration and distributing it as a ready-to-flash image, software developers can iterate without touching Libero at all.

A Minimal and Repeatable Development Platform

The evaluation setup is intentionally designed to remove friction during early development. All interaction with the board is handled through a single USB connection. The kit exposes a built-in FlashPro programmer, so no external probes or adapters are required to establish a usable development environment.

In practice, the setup reduces to three fixed elements:

• One USB cable used for both programming and UART console access
• A jumper configuration that allows flashing of both the FPGA fabric and firmware
• A known, static DIP-switch configuration

Once configured, the board can remain in this state for the entire development cycle.

On the hardware side, the FPGA design is deliberately minimal. Only the components required to make the MSS usable are enabled: a Cortex-M3 clocked at 166 MHz, APB buses running at full core speed, a single UART for console output, GPIO-mapped LEDs for visible execution state, and one GPIO routed as an external trigger for measurement and debugging. There is no custom logic, no accelerators, and no unused peripherals. The result is a predictable execution environment that behaves identically on every boot.

The FPGA design is developed using Libero SoC, Microchip’s integrated FPGA design environment for SmartFusion2. Libero is used to configure the FPGA fabric, MSS peripherals, clocks, and pin assignments, and to generate the final FPGA bitstream. It is a comprehensive but heavyweight toolchain, typically operated by hardware teams, and it requires licenses, long build times, and detailed device-level knowledge. Libero produces both the FPGA bitstream and the associated firmware artifacts, which can then be used to build a BSP. Firmware engineers can subsequently continue software development in SoftConsole IDE or, for more low-level workflows, directly edit the code (e.g., in vim) and build it using a GCC-based ARM toolchain.

To keep the workflow software-centric, the FPGA can be programmed using FlashPro Express rather than a full Libero project. The hardware is delivered as a pre-built programming job, which avoids licensing requirements and lengthy synthesis or place-and-route steps. Every developer works against an identical hardware configuration, and flashing the FPGA becomes a one-time operation that takes seconds (well, maybe longer…).

The firmware follows the same philosophy. It does only what is necessary to confirm that the platform is alive: initialise clocks and GPIOs, bring up a UART console, and provide a visible heartbeat via an LED. If UART output is visible and the LED toggles, the system is ready. Anything beyond that belongs in application code, not in bring-up firmware.

Two firmware build modes are supported: a debug configuration that runs from SRAM for fast iteration, and a release configuration that runs from on-chip non-volatile memory for deployment-like testing. This split keeps development efficient without sacrificing realism.

Enabling UART

Using SmartFusion2 from Linux works well in practice, but one detail regularly trips people up: UART access via the on-board FTDI device. Microchip’s documentation is very complete for Windows, but Linux workflows are less well covered (even though Microchip support is excellent). The issue is not the hardware, but how Linux binds drivers to the FTDI interfaces by default.

The evaluation board exposes a multi-interface FT4232H USB device. From a hardware perspective, several virtual serial channels are available. From the Linux kernel’s perspective, however, only one of those interfaces is automatically bound to the ftdi_sio driver, while the remaining three are not.

The FT4232H device connected to the SmartFusion2 micro-USB port sets up four virtual ports. Under Linux, the root device is listed as:

Bus 003 Device 005: ID 1514:2008 Actel Embedded FlashPro5

and each individual interface as

/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 4: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 4: Dev 5, If 1, Class=Vendor Specific Class, Driver=, 480M
    |__ Port 4: Dev 5, If 2, Class=Vendor Specific Class, Driver=ftdi_sio, 480M
    |__ Port 4: Dev 5, If 3, Class=Vendor Specific Class, Driver=, 480M

By default, Linux binds ftdi_sio only to interface If 2. In practice, the remaining interfaces can also be bound to ftdi_sio. You can force the driver to match them as follows:

echo 1514 2008 | sudo tee /sys/bus/usb-serial/drivers/ftdi_sio/new_id

This command causes the kernel to rebind the remaining FTDI interfaces, which can be verified in dmesg. After that, UART access is available via the /dev/ttyUSBx device corresponding to interface If 3 (the last FTDI interface).

(Credit goes to a colleague who helped identify this behaviour.)

Flashing Firmware from the Linux Command Line

Flashing firmware from the command line is not officially supported by the standard toolchain; Microchip recommends using SoftConsole. The GUI relies on OpenOCD (bundled with SoftConsole) and GDB, both of which connect to the on-chip debugger via USB.

A command-line workflow is essential for automation, CI, and reproducible Linux-based development. While it is possible to reverse-engineer the OpenOCD invocations used by the GUI, there is a cleaner alternative.

The programmer is supported by the pyOCD toolset, which provides a practical and well-maintained solution for flashing and debugging SmartFusion2 devices using an external probe.

SmartFusion2 evaluation boards expose a standard RVI debug header, which allows the use of common ARM debug probes such as:

• Keil ULINK (CMSIS‑DAP)
• Other CMSIS‑DAP compliant probes
• SEGGER J‑LINK

A key drawback of using an external programmer is that jumper J8 must be moved to position 2–3 in order to program the FPGA bitstream using FlashPro 4 or 5.

pyOCD supports these probes out of the box and provides a clean, scriptable interface suitable for automation.

In practice, the workflow looks like this:

• Connect the external probe to the RVI header
• Install pyOCD on the host system
• Install the CMSIS device pack for the SmartFusion2 target
• Flash binaries directly from the command line

Once set up, flashing a firmware image becomes a single command, which integrates naturally into Makefiles, CMake builds, or CI pipelines. This avoids reliance on vendor GUIs while remaining robust and repeatable.

In practice, this looks as follows:

• HW configuration: The user connects ULINK2 into the RVI port, connects pins 1-2 of jumper J8 and J31 (as below).
• SW configuration: Once pyOCD is installed, the user needs to download a CMSIS package for M2S090 board. No special udev rules are required.

 pyocd pack install m2s090
Downloading packs (press Control-C to cancel):
    Microsemi.M2Sxxx.1.0.65
Downloading descriptors (001/001)

> pyocd list -p
  #   Probe/Board                           Unique ID   Target
----------------------------------------------------------------
  0   Keil Software Keil ULINK2 CMSIS-DAP   V0010M9E    n/a

With this setup, flashing becomes straightforward:

pyocd flash --target m2s090 app/hello.bin
pyocd reset --target m2s090 -m hw

The reset command resets the board, and the -m hw option ensures that the device does not enter debug mode.

For measurement and analysis, UART is often not the most convenient interface.

As an alternative, SEGGER J-LINK or CMSIS-DAP programmers can be used. The RTT interface provided by J-LINK enables fast data transfer between the device and the host, which can be useful for collecting data for constant-time analysis, serving as an alternative to UART.

In addition, the board includes a Trace Port Interface Unit (TPIU) supporting ITM and ETM (Instruction Trace Module / Embedded Trace Module), which can also be used as an alternative to RTT.

Using usbip for Remote Access

In shared lab environments, it is often useful to access SmartFusion2 boards remotely—for example, from CI servers or developer machines without physical USB access. It also helps avoid accidents, such as spilling coffee on expensive hardware.

In such cases, usbip can be used to export a USB-attached debug probe (and, if needed, the FlashPro interface) from a remote host and reattach it on the client machine over the network.

This enables:

• Remote firmware flashing using pyOCD
• Centralised lab hardware shared across multiple users
• Integration of physical boards into CI systems

Configuration is very simple: * Server side (machine with the USB device physically plugged in)

sudo modprobe usbip_core usbip_host
sudo usbipd -D
sudo usbip list -l
sudo usbip bind -b <BUSID>

The usbip list -l command lists available BUSID values, which can be exported over IP using usbip bind -b.

• Client side:

sudo modprobe usbip_core vhci_hcd
sudo usbip list -r <SERVER_IP>
sudo usbip attach -r <SERVER_IP> -b <BUSID>

Similarly, usbip list -r lists the USB devices exported by the remote server.

As a practical note, usbip uses TCP port 324, so make sure this port is allowed through the firewall.

When combined with pyOCD and stable UART device naming (via udev), usbip allows SmartFusion2 boards to be treated much like network‑attached test equipment. This setup is particularly useful for regression testing, automated measurements, and long‑running experiments.

As with any USB‑over‑IP solution, latency and reliability depend on the network, but for flashing and debug control the approach works well in practice.

Conclusion

SmartFusion2 can feel intimidating at first, especially if approached from an FPGA-centric mindset. Treated instead as a microcontroller platform with a fixed hardware personality, it becomes much easier to work with.

Despite being an older platform, SmartFusion2 remains highly capable. The MSS provides 80 KB of SRAM (or 64 KB when error correction is enabled), which may sound restrictive by modern standards—but in practice it is sufficient for serious cryptography. In our work, we were able to fit both ML-DSA and ML-KEM comfortably, using significantly less. The resulting implementations run reliably and with respectable performance for a Cortex-M3-class device.

This does, however, require care. On Cortex-M3, certain long multiplication instructions are not constant-time, which means developers must be deliberate when implementing cryptographic arithmetic, especially in security-sensitive contexts. Understanding where the microarchitecture leaks and how to work around it is essential.

That topic deserves a deeper discussion of its own and is a story for another blog.

By keeping hardware minimal, firmware simple, and tooling predictable, the platform becomes stable enough to fade into the background. Once the basics are solid, complexity can be added where it actually matters. Starting simple is what makes that possible.

QAT software stack

The diagram below shows how applications reach QAT hardware through OpenSSL, the QAT Engine, and Intel’s software libraries:

• Application: Consumes cryptographic services (e.g. TLS termination, IPsec, disk encryption, PKI, secure messaging).
• OpenSSL: Exposes standard crypto APIs; can dispatch operations to software or QAT hardware transparently.
• QAT Engine: OpenSSL engine plugin that offloads supported primitives (RSA, ECDSA, DH, AES-GCM, ChaCha20-Poly1305, SHA variants) to QAT hardware when available.
• Intel IPP: Highly optimized CPU software primitives (SIMD, microarchitecture-tuned) for symmetric and asymmetric cryptography when hardware offload is not used.
• Multi-Buffer Crypto (IPP-crypto): Batches multiple independent crypto jobs (e.g. parallel RSA, AES-GCM streams) to improve core utilization—useful for high-concurrency servers.
• Intel QAT Driver: Kernel + user-space interface to QAT devices. Two branches exist:
  • In-tree (QATlib): Aligned with kernel development model; standardized feature management.
  • Out-of-tree: Broader feature set for some legacy or extended hardware. Driver utilities (e.g. adf_ctl) configure and monitor accelerator instances. Driver families 1.x and 2.x support different hardware generations.
• QAT Hardware: Integrated (selected Xeon SKUs) or discrete PCIe accelerators providing queues for crypto and compression services.
• QATlib: User-space library exposing a stable API to submit crypto jobs to QAT hardware or to fall back on software paths.

Key Points

• Applications typically call OpenSSL, which can use the QAT Engine to offload to hardware or fall back to software (IPP, IPP-crypto).
• QAT Engine is designed specifically for hardware acceleration.
• Intel IPP and Multi-buffer Crypto provide CPU-based optimizations when hardware acceleration is unavailable or unnecessary.
• Multi-buffer Crypto boosts performance by parallelizing cryptographic operations across multiple data buffers, which is ideal for high-concurrency servers.

Cryptographic support

The list of algorithms supported by QAT is documented in the official QAT documentation. Support varies by hardware generation; the driver detects availability. If an algorithm is unsupported, its context initialization returns CPA_STATUS_UNSUPPORTED. Numeric algorithm IDs are defined in the driver sources here.

For this study, the relevant hashing algorithms provided by the installed hardware are:

• SHA2-256
• SHA2-512
• SHA3-256

(SHAKE is not supported on this device.)

     4. Compute the string K as follows:
        for ( i = 0; i < p; i = i + 1 ) {
          tmp = x[i]
          for ( j = 0; j < 2^w - 1; j = j + 1 ) {
            tmp = H(I || u32str(q) || u16str(i) || u8str(j) || tmp)
          }
          y[i] = tmp
        }
        K = H(I || u32str(q) || u16str(D_PBLC) || y[0] || ... || y[p-1])

Technological Fit for QAT

To realize these benefits, applications must integrate QAT asynchronously. Offloading compute-heavy primitives frees CPU cycles and improves throughput, particularly in high-concurrency environments.

A key enabler is asynchronous execution. In the past, Intel invested in OpenSSL by implementing ASYNC_JOB infrastructure. This functionality is based on reactor pattern, which we describe below.

QAT ecosystem: IPP and Multi-buffer Crypto

This part describes in more detail the software components used in the QAT ecosystem.

Reuse within a cryptographic software stack

When integrating QAT (or any accelerator) into a cryptographic software stack, three patterns are common:

1. Integration into the core crypto stack
2. Separate companion component alongside the core stack
3. Plugin-based integration for open-source ecosystems

Conclusions

The quantitative study presented in this post was conducted using QAT hardware connected via PCIe. While the host machine used is relatively powerful, the PCIe communication introduces latency during data transfers.

Intel’s 4th Generation Xeon Scalable processors, released in 2023, represent a significant architectural advancement. These processors feature integrated QAT acceleration engines and support for the CXL 1.1 (Compute Express Link) standard. QAT performance on these CPUs may differ significantly from our current results. Preliminary analysis suggests that CXL offers a more efficient communication model between CPUs and cryptographic accelerators, making it a better fit for such workloads. Even with CXL, unless latency drops substantially, the underlying pattern is likely unchanged: QAT is effective for bulk workloads but not for the many small, sequential hash invocations typical in PQ signature schemes.

Why Start the Migration?

You don’t want to be caught off guard when quantum computers become capable of breaking current cryptographic standards. The migration process is complex and time-consuming, often taking several years to complete. Organizations must first evaluate when and how to begin, considering factors such as:

1. Data lifetime – how long the information you protect needs to remain confidential.
2. Migration complexity – how much effort is required across systems, hardware, and vendors.
3. Quantum threat timeline – how soon a CRQC could become practical.

Even without immediate existential risk, planning and testing today ensures you aren’t forced into a rushed, high-cost migration tomorrow. More complicated systems, such as old, small embedded devices, may take years to update, replace, or ideally redesign with PQC in mind.

Starting this transition early is definitely a good idea, because migration can be complex and time-consuming. But without panic and feeling pressured to start immediately. Be deliberate when selecting products and suppliers - this technology is still maturing and not yet fully commoditized, which means higher costs and potential risks. Open-source solutions can help mitigate some of these challenges, though they come with their own uncertainties. Proprietary options may provide stronger support and stability but can also create dependency on specific vendors and ecosystems. In short, try hard to avoid the “sell-now, forget-later” mindset - this remains a developing field with trade-offs, uncertainties, and costs that must be carefully balanced.

Key Exchange in TLS: The Most Urgent Step

TLS key exchange is the top priority for PQC migration. If session keys are negotiated using quantum-vulnerable algorithms, future quantum computers could decrypt recorded traffic. To mitigate this, PQ/T hybrid key exchange is recommended during the transition. These approaches combine at least one post-quantum and one classical algorithm, so security is maintained as long as one remains unbroken. Hybrid KEMs are also easier to roll out than post-quantum signatures, since they are ephemeral and not linked to long-term identity.

As PQC matures, hybrid KEMs will become less important, and eventually only the post-quantum algorithm will be needed.

The IETF began work on standardizing post-quantum key exchange for TLS in 2019, after a key workshop at Mozilla’s Mountain View offices and early Google-led experiments. This resulted in a framework for new key exchange methods, and the TLS Post-Quantum Experiment showed hybrid KEMs in action.

The first widely adopted Internet draft for hybrid key exchange in TLS is now close to completion. This draft and its extension have become the de facto standard for modern TLS, with implementations already available in OpenSSL, NGINX, and AWS’s AWS-LC library, and are already widely deployed. Jan Schuman from Akamai has a great post on sites already using PQC.

TLS is not the only protocol that needs quantum-safe key exchange. For this reason, there is also an effort to standardize a more generic approach to hybrid KEM constructions for broader use.

Digital Signatures: Important, But Less Urgent

Digital signatures remain a vital component of cryptographic protocols, ensuring authenticity, integrity, and non-repudiation. As such, post-quantum digital signature schemes are necessary for a future-proof internet infrastructure. However, the urgency to deploy them is relatively lower compared to key encapsulation mechanisms (KEMs).

Unlike key exchange, authentication cannot be broken retrospectively, meaning quantum-safe signatures are only needed once cryptanalytically relevant quantum computers become available. As a result, the migration to post-quantum digital signatures is less time-sensitive than for KEMs, allowing for a more deliberate and carefully planned transition. Since post-quantum signature schemes often involve larger keys and signatures, greater computational overhead, and increased implementation complexity, their deployment may incur higher costs - reinforcing the importance of keeping the migration - —as simple and efficient as possible.

Determining whether and when to adopt PQC certificates or PQ/T hybrid schemes may depend on several factors, such as:

• Frequency and duration of system upgrades
• Operational flexibility to enable or disable algorithms

Deployments with limited flexibility (e.g., embedded systems) benefit significantly from PQ/T hybrid signatures. This approach mitigates the risks associated with delays in transitioning to PQC and provides an immediate safeguard against zero-day vulnerabilities.

While hybrid constructs may seem plausible for long-term security, they also introduce complexity, potential performance overhead, and long-term implications:

• The number of possible hybrid combinations leads to interoperability challenges and increased implementation burden.
• If one scheme is compromised, forgery is only a concern while the corresponding public key remains trusted.
• Long-term protection through hybrids may be limited in practice due to standard key management practices.

There is another risk related to the potential misuse of PQ/T hybrid signatures. Consider this: a deployment may use hybrid signatures to facilitate migration, resulting in a mix of devices - some aware of PQ schemes and some not. Devices unaware of PQ schemes may continue to validate only the traditional signature, while those aware of PQ schemes may validate both signatures. A deployment might continue this approach even after the traditional algorithm has been broken. While this may simplify operations by avoiding re-provisioning of trust anchors, it introduces a significant risk. A CRQC could forge the broken traditional signature component over a message, then combine it with the valid post-quantum component to produce a new composite signature that verifies successfully. This underscores the critical need to retire hybrid certificates containing broken algorithms once CRQCs become available (and always validate both components of a hybrid signature).

The IETF has many experts working on this topic. For example, draft-ietf-lamps-pq-composite-sigs describes how to create and verify composite signatures that combine a post-quantum signature with a classical signature.

Nevertheless, hybrid signatures remain complicated and may not be suitable for all scenarios. Fortunately, they are only needed once quantum computers are capable of breaking current signature algorithms. So, we still have some time to make the authentication migration as smooth as possible.

Infrastructure Costs: What to Expect

This topic is both important and frequently overlooked. Migrating to post-quantum cryptography (PQC) often requires significant updates to existing infrastructure. Careful planning and budgeting are essential, as costs can arise in multiple areas.

The first step is discovery - building a comprehensive inventory of all cryptographic assets and where they are used. This process may require specialized tools and can itself be resource-intensive.

Another major factor is whether updates can be delivered through software patches or require new hardware. For some systems, particularly constrained or niche devices, supporting PQC may require custom development or even physical replacement. Engaging with vendors early is critical to understand available options and associated costs.

Key infrastructure components that will need attention include:

• Network protocols – TLS, SSH, and QUIC must be adapted to handle larger PQC artifacts. While PQC KEMs such as ML-KEM often perform competitively in handshakes, their larger message sizes can increase bandwidth use, add round trips, or introduce latency.

• Message processing – PQC signature algorithms typically process entire messages rather than digests. This can hurt performance in systems like HSMs that rely on streaming data, unless applications adopt pre-hashing or streaming-friendly designs.

• PKI systems – Certificate Authorities (CAs), certificate formats, and trust anchors must all evolve to support PQC. Hybrid certificate formats can ease transition but also add complexity and operational overhead.

• Constrained devices – Long-lived systems (e.g., satellites, industrial controllers, smart meters) are especially difficult to update. Limited memory and compute resources may force costly redesigns or replacements, and in-field updates can be logistically challenging.

• Hybrid approaches, while helpful for resilience during the transition, can add two layers of cost: first to support dual algorithms (certificates, key management, validation), and later to migrate again once hybrids are no longer needed. In some environments, this two-step process is more expensive than planning a direct migration to PQC at the right moment.

• Training and Awareness: Ensuring that staff are knowledgeable about PQC concepts, like KPIs provided by PQC implementations and impact on performance, familiar with tradeoffs of PQ/T hybrid schemes and their implications on migration process are essential. This may involve training programs, workshops, or hiring specialized personnel, all of which contribute to the overall cost.

Long-term savings come from embracing cryptographic agility: designing systems that can switch algorithms without major architectural changes. This reduces the cost of future transitions - but achieving true agility requires upfront investment in both design and standardization.

Final Thoughts

Migrating to post-quantum cryptography is not a single upgrade — it is a long-term process. As mentioned at the beginning of this article, starting this transition early is definitely a good idea.

It is worth recalling that the first IETF draft proposing a PQ/T hybrid key exchange for TLS was published back in 2019 (I agree, it wasn’t serious proposal at the time), nevertheless we are now in 2025, and only recently have PQ/T hybrid standards been finalized by the IETF and started to see adoption by major browsers. That is six years for a single - and relatively simple - use case: key exchange in TLS.

When it comes to digital signatures, the migration will be significantly more complex and time-consuming. Based on current timelines and projections for quantum computing, cryptographically relevant quantum computers are expected to emerge by the end of the next decade. This leaves less than ten years to finalize standards and initiate large-scale migration. Not in TLS, but everywhere. Given that such migration efforts typically span several years, it is clear that we are already behind schedule.

The situation today, however, is quite different from 2019: the first PQC algorithms have been standardized, and there is far greater motivation and momentum to move forward. Nevertheless, much remains to be done. The key is to plan for agility and actively align with emerging standards. Going back to the beginning of this article: don’t wait until the last minute, but also don’t rush into unproven solutions. Balance risk, cost, and operational complexity carefully.

The transition will be uneven: some systems will adopt hybrids, some will wait for pure PQC. Constrained devices may require tailored strategies and highly optimized implementations to match the performance and resource utilization of traditional algorithms. But the direction is clear: a future-proof Internet must stay safe!

Ongoing work in the IETF focuses on general guidance for migration to PQC as well as guidance for constrained devices. Feel free to join the discussions in the PQUIP Working Group.

The lattice and related hard-problems

The picture below visualizes a lattice as points in a two-dimensional space. A lattice is defined by the origin \(O\) and base vectors \(\{b_1,b_2\}\). Every point on the lattice is represented as a linear combination of the base vectors, for example \(V=−2b_1+b_2\).

There are two classical NP-hard problems in lattice-based cryptography:

• Shortest Vector Problem (SVP): Given a lattice, to find the shortest non-zero vector in the lattice. In the graph, the vector \(s\) is the shortest one. The SVP problem is NP-hard only under some assumptions.
• Closest Vector Problem (CVP): Given a lattice and a vector \(V\) (not necessarily in the lattice), to find the closest vector to \(V\). For example, the closest vector to t is \(z\).

In the graph above, solving SVP and CVP would be quite simple. However, the lattices used in cryptography have higher dimensions, say above 1000, as well as non-orthogonal vector basis. It is believed that in these instances, the problems are hard enough to solve, even for future quantum computers.

Coppersmith and Shamir, proved in “Lattice Attacks on NTRU” (EUROCRYPT’97) that NTRU problem, can be reduced to SVP.

Rings

NTRU operates in a ring of polynomials of degree \(N\). The degree of a polynomial is the highest exponent of its variable. For example, \(x^7+6x3+11x^2\) has degree of 7. One can add polynomials in the ring in the usual way, by simply adding theirs coefficients modulo some integer. In NTRU this integer is called as \(q\). Polynomials can also be multiplied (obviously), and the result of a multiplication is always a polynomial of degree less than \(N\). It basically means that exponents of the resulting polynomial are added to modulo \(N\). For example:

In other words, polynomial ring arithmetic is very similar to modular arithmetic, but instead of working in a “set of numbers” less than \(N\), one works in a set of polynomials with a degree less than \(N\).

NTRU scheme: basic idea

To instantiate the NTRU cryptosystem, the following domain parameters must be chosen:

• \(N\) - degree of the polynomial ring, in NTRU the principal objects are polynomials of degree \(N−1\).
• \(p\) - small modulus, used during key generation and decryption for reducing message coefficients.
• \(q\) - large modulus, used during algorithm execution for reducing coefficients of the polynomials.

First, we generate a pair of public and private keys. To do that, two polynomials \(f\) and \(g\) are chosen from the ring in a way that their randomly generated coefficients are much smaller than \(q\). Then key generation computes two inverses of the polynomial:

The values \(f\) and \(f_p\) make up the private key. The public key \(pk\) is computed, as follows: The \(f_q\) is not part of any key, however it must remain secret.

It might be the case that after choosing \(f\), the inverses modulo \(p\) and \(q\) do not exist. In this case, the algorithm has to start from the beginning and generate another \(f\). That’s unfortunate because calculating the inverse of a polynomial is a costly operation. The recent instantiations of some NTRU schemes (like NTRU-HRSS) are design in a way to ensure those inverses always exist. Which makes key generation faster and more reliable.

The encryption of a message \(m\) proceeds as follows. First, the message \(m\) is converted to a ring element \(pt\) (there exists an algorithm for performing this conversion in both directions). During encryption, NTRU randomly chooses one polynomial \(b\) called \(blinder\). The goal of the blinder is to generate different ciphertexts per encryption. Thus, the ciphertext \(ct\) is obtained as:

Decryption looks a bit more complicated, but it can also be easily understood. It uses both the secret value \(f\) and \(f_p\). To recover the plaintext as:

Taking all that was described above, evaluation done during decryption is something like:

After obtaining \(pt\), the message \(m\) is recovered by inverting the conversion function.

The underlying hard assumption is that given two polynomials: \(f\) and \(g\) whose coefficients are short compared to the modulus \(q\), it is difficult to distinguish \(pk={f g}\) from a random element in the ring. It means that it’s hard to find \(f\) and \(g\) given only public key \(pk\).

Concrete schemes

The original scheme has a long (over 20 years) history. Since then it has been changed multiple times, as a response to account for cryptanalytic advances. Several variants of concrete KEM and signature schemes based on NTRU were proposed during NIST PQC standardization. In the case of KEM, two candidates NTRUEncrypt and NTRU-HRSS-KEM been merged together and end up as a scheme called … well, “NTRU”. The scheme is fairly easy to implement in constant-time with is characterized by performance, allowing it to be used in the production environment. The NTRU-based signature scheme Falcon also came to the last round of the standardization. It’s characterized by very fast execution and relatively small key sizes. Nevertheless, performance efficient, constant-time implementation can be quite complicated. It also seems, patent situation for NTRU schemes is much clearer than in case of other candidates.

How does the UUM detector work?

Techniques used for detecting Use of Uninitialized Memory, implemented in tools like Valgrind’s Memcheck or LLVM’s Memory Sanitizers, has been developed over the years and currently are quite advanced. At high level both Memcheck and MSan are similiar.

The main difference between Memcheck and Memory Sanitizer is that in the case of Valgrind binary instrumentation is done at startup. Memcheck uses the Valgrind framework for instrumenting already compiled binary. In contrast, Memory Sanitizer instruments code at the compilation stage and leverages mechanisms implemented in LLVM, like its intermediate representation. Memcheck combines the detection of UUM and memory addressability bugs in a single tool. In the case of LLVM, these are implemented in two different tools - Memory Sanitizer (MSan) for detecting UUMs and a sibling tool called AddressSanitizer (ASan) for addressability bugs. There are down and upsides to both approaches. The approach taken by MSan allows for execution to be magnitude faster and have almost no startup penalty.

Except for those differences, at a high level, the internals of both Memcheck and MSan are similar. UUM detector tracks the state of every bit of memory or register used by the program. It uses a concept called shadow memory (Valgrind calls shadow memory a VBITS’s), which stores information on whether each bit of memory was properly initialized.

It allows using uninitialized memory if it is “safe” to do so. For example, copying it from one place to the other is not a problem. It reports a problem only when the execution of a program depends on an uninitialized state. For example, when branching, dereferencing a pointer or using uninitialized memory as for array indexing. That’s exactly what we need to test constant-time functions. Uninitialized memory can be propagated to other variables in the code (i.e. copying). To track the propagation, UUMs implement propagation of the shadow memory - when the uninitialized value is used as an operand of a “safe” operation, its state is propagated to the result of that operation. The new shadow value is computed based on the values of the operands and their shadow values.

Let’s summarize those concepts by analysing a concrete example. Function on the left side adds 1 to an argument n and returns the result. On the right side, we see the state of the shadow memory and the current values of variables. Uninitialized memory is marked red and initialized is blue. In this example, the function is called with argument n=6. The 12 least significant bits of an argument n are uninitialized.

The function creates on a stack temporary variable b and assigns 1 to it. Variable is properly initialized, hence its shadow memory is marked blue. The second variable r is initially uninitialized. In addition, the UUM detector looks at shadow bits of both operands and calculates a new shadow value for variable r. Adding uninitialized to initialized results in uninitialized. So, the shadow memory for r is the same as for the argument n (4 most significant bits have initialized the rest is not). In addition, the operation doesn’t change program flow it doesn’t trigger the UUM.

It should also be noticed that resulting shadow memory depends, on the operation being done as well as the state of shadow memory of both operands.

Origin tracking helps to understand potential problems. It assigns an ID to each variable and serves as an identifier that created uninitialized bits in shadow memory. Once UUM is triggered, the detector uses those IDs to backtrack the origin of uninitialized values and print them in the final report. MSan implements more advanced origin tracking, its report shows lines of code where uninitialized memory was created as well as places at which it was propagated before UUM was triggered (see the result of running “toy example” above). In the case of MemorySanitizer, it is enabled by providing -fsanitize-memory-track-origins= flag to the compiler, in case of Valgrind it is --track-origins=yes option.

Toy example

A constant-time table lookup is an important tool in secure, cryptographic implementations. For instance, it is used for the implementation of elliptic curves-based schemes, like ECDH - widely deployed on the Internet and used by HTTPS connection key exchange schemes. In that system, the (rational) point on the elliptic curve is multiplied by a scalar. A scalar is used as a secret key and hence it must be protected against leaks. The optimized version of such multiplication may use, so-called window technique. In this case, to speed up computation, an algorithm starts with pre-computing small multiplies of a point (like , for fixed window size ) and stores them in some table. Then, scalar-by-point multiplication consists of slicing the binary representation of a scalar into equal -bit long pieces, iterate over such split and use those pieces for point multiplication (see here for a detailed description of a technique). Most importantly at each iteration, an algorithm gets a small multiply of a point for from the table. That is a time variable operation - as the value may be loaded from different locations (CPU register, cache or RAM). An attacker can then try to guess secret scalar by exploiting those time differences. Hence, lookups done be secure implementations need to be implemented in a constant-time manner.

The toy example shows how to use LLVM’s Memory Sanitizer to detect whether table lookup is constant-time. Program starts with initializing table pow2 with powers of two in a range . Then it reads from the command line to the variable secret, uses it to get from the table and returns the result.

In the first step, we want to ensure that MSan triggers UUM whenever program execution depends on the value of secret variable, if it does - UUM must be triggered. Fortunately, the LLVM’s MemorySanitizer API offers such a possibility, the __msan_allocated_memory marks address ranges as containing undefined or defined data, exactly what we need. All the MemorySanitizer API functions are located in the msan_interface.h header, which needs to be included in the code. Let’s look at the example below:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>

#include <sanitizer/msan_interface.h>

#define POW2_NUM 64
static uint64_t pow2[POW2_NUM];

static inline uint64_t select_n(uint64_t n) {
    return pow2[n];
}

int main(int argc, const char* argv[]) {
    uint64_t ret, secret;
    // Initialize a table with powers of 2
    for (size_t i=0; i<64; i++) {
        pow2[i] = 1ULL << i;
    }

    secret = atoi(argv[1]);

    // Denote "secret" variable as uninitialized
    __msan_allocated_memory(&secret,sizeof(secret));
    // Time dependent operation possible load from cache or memory
    ret = select_n(secret);

    // Denote memory as defined to eliminate false possitive, due
    // to non constant-time implementation of printf
    __msan_unpoison(&secret, sizeof(secret));
    // Denote also 'ret' in case shadow bits were propagated
    __msan_unpoison(&ret, sizeof(ret));
    printf("2^%lu = %lu\n", secret, ret);
}

In that code, the select_n function performs memory lookup in a table pow2. Just before the function is called, 64-bits (8 bytes) of memory storing secret is denoted as uninitialized. That’s done by __msan_allocated_memory function. Then program calls the select_n function and at this point, UUM should be triggered. To avoid reporting a false positive error (caused by printf), the secret and ret are marked as defined, just after select_n returns. The result of the run is as expected:

> clang -g -fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer test.c
> ./a.out 47
==1307703==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55f795fe2a1c in select_n /home/kris/test.c:11:12
    #1 0x55f795fe269f in main /home/kris/test.c:26:11
    #2 0x7fc798ac8b24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)
    #3 0x55f795f6114d in _start (/home/kris/a.out+0x2014d)

  Uninitialized value was stored to memory at
    #0 0x55f795fe299e in select_n /home/kris/test.c:10
    #1 0x55f795fe269f in main /home/kris/test.c:26:11
    #2 0x7fc798ac8b24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)

  Memory was marked as uninitialized
    #0 0x55f795fbd1bb in __msan_allocated_memory (/home/kris/a.out+0x7c1bb)
    #1 0x55f795fe2647 in main /home/kris/test.c:24:5
    #2 0x7fc798ac8b24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/kris/test.c:11:12 in select_n
Exiting

Execution correctly triggers UUM at line 11, which is precisely where table lookup is done. Sanitizer also reports some information about origin of the problem. Generation of that information is enabled by -fsanitize-memory-track-origins=2 flag and proves to be quite useful during designing functions with constant-time execution.

Detection works that’s excellent. Let’s try to use it now on a code that’s constant time and see if UUM is not triggered. Following implementation is functionally equivalent to select_n, but now table lookup is done in constant time. Namely, it always goes thru all the elements of the table. The function calculates a mask variable, which sets all the bits only when element n is processed. Then thanks to logical & value is copied to the variable ret.

static inline uint64_t const_select_n(uint64_t n) {
    uint64_t mask, sign, i, ret = 0;
    sign = 1ULL << (63 - n);
    // Always iterate over all elements
    for (i=0; i<POW2_NUM; i++, sign<<=1) {
        // Arithmetical shift right propagates MSB if
        // set. Thanks to 'sign' set above, this is
        // done only once during whole iteration.
        mask = ((int64_t)sign) >> 63;
        // With correctly set mask only one value
        // is assigned to 'a' variable
        ret |= pow2[i] & mask;
    }
    return ret;
}

We can now swap the call to select_n with a call to const_select_n. Such implementation doesn’t trigger UUM anymore, as execution doesn’t depend on uninitialized data - the program always reads the whole table. On the flip side, the implementation of const_select_n is much more complicated to analyse, so tools are needed.

> clang -g -fsanitize=memory -fsanitize-memory-track-origins=2 -fno-omit-frame-pointer test.c
> ./a.out 47
2^47 = 140737488355328
>

Utility called ct_check.h

Both, the Memcheck and Memory Sanitizer provide programmatic API, that can be used to design constant-time code. The ct_check provides a unified API for using both of those tools. A flag is used at compile time to control which tool to use. At the development stage, I use both tools- MSan is faster and gives more information in the final report, checks by Memcheck are more granular. Such a wrapper allows writing code only once and hence it is quite useful.

The ct_check.h exposes following functions:

With that set of functions, I’ve used tests implemented by A. Lagnley to ensure the correctness of MSan and ctgrind are the same. Implementation of those tests with ct_check.h is here, but indeed, results are the same.

Applying ct_check.h to the existing implementation

Instead of toy-code, let’s now take an existing, modern cryptographic implementation, which was vulnerable to timing attacks and see if Memory Sanitizer can detect a problem in vulnerable code. Quantum-safe cryptographic implementations are currently my main focus, so I’ll apply it to one of Key Encapsulation Mechanism (KEM) submitted to NIST for post-quantum standardization. All the code presented below comes from PQC library available on Github (branch called blog/frodo_constant_time_issue).

A KEM is defined by 3 algorithms. A key generation returning pair of public and private keys, encapsulation algorithm which uses the public key to return shared secret in plain form and in encrypted form as ciphertext. Finally, decapsulation algorithm, that takes ciphertext and secret key as an input and returns shared secret, which then can be used for symmetric encryption (i.e. in TLS). To avoid leaking the secret key, the decapsulation function must ensure that the operation done on the private key is constant-time. This problem has been reported in FrodoKEM and exploited in recent paper. In that work, the authors propose (section 3) a generic side-channel technique that can be applied to recover the secret key of (LWE-based) KEM. Then (in section 4) describes how to use that technique to recover the FrodoKEM key. I highly recommend the paper (or video) to anybody interested in secure cryptographic implementations.

The following, variable time, implementation allowed attack to succeed.

// https://github.com/kriskwiatkowski/pqc/blob/e57a8915834e08998f1a93f3d111cfaf3fcd94a7/src/kem/frodo/frodokem640shake/clean/kem.c#L229
int PQCLEAN_FRODOKEM640SHAKE_CLEAN_crypto_kem_dec(uint8_t *ss, const uint8_t *ct, const uint8_t *sk) {
    ...
    if (memcmp(Bp, BBp, 2*PARAMS_N*PARAMS_NBAR) == 0 &&
        memcmp(C, CC, 2*PARAMS_NBAR*PARAMS_NBAR) == 0) {
        // Load k' to do ss = F(ct || k')
        memcpy(Fin_k, kprime, CRYPTO_BYTES);
    } else {
        // Load s to do ss = F(ct || s)
        // This branch is executed when a malicious ciphertext is decapsulated
        // and is necessary for security. Note that the known answer tests
        // will not exercise this line of code but it should not be removed.
        memcpy(Fin_k, sk_s, CRYPTO_BYTES);
    }

The ciphertext is a concatenation of two parts ciphertext = Bp || C. The decapsulation function, implemented by FrodoKEM, uses a secret key to decrypt the ciphertext, encrypt it again and compares a result with ciphertext received (see Fujisaki-Okamoto transform). That is what is being done in the code above. Values, BBp and CC represent ciphertext that was recomputed during decapsulation. Those values are compared with received ciphertext. If the comparison succeeds, the shared secret sk_kis returned, otherwise the function returns some random value.

There are two problems related to variable time execution:

1. comparison uses memcmp: this function is not constant-time - it fails as soon as it detects the first difference
2. it is used in short-circuit evaluation: in case first memcmp returns a value different than 0, second memcmp is not called. Hence that’s also not constant-time behaviour.

The first issue is already enough to recover the private key. Let’s see if Memory Sanitizer will help to design constant-time implementation. I’m using PQC library, which integrates both variable and constant time decapsulation in FrodoKEM/640. Let’s start with a unit test:

// Uses GTEST and C++
TEST(Frodo, CtDecaps) {

    // Get descriptor of an algorithm
    const pqc_ctx_t *p = pqc_kem_alg_by_id(PQC_ALG_KEM_FRODOKEM640SHAKE);

    // Initialize buffers for KEM output
    std::vector<uint8_t> sk(pqc_private_key_bsz(p));
    std::vector<uint8_t> pk(pqc_public_key_bsz(p));
    std::vector<uint8_t> ct(pqc_ciphertext_bsz(p));
    std::vector<uint8_t> ss(pqc_shared_secret_bsz(p));
    bool res;

    // Generate key pair and perform encapsulation
    ASSERT_TRUE(pqc_keygen(p, pk.data(), sk.data()));
    ASSERT_TRUE(pqc_kem_encapsulate(p, ct.data(), ss.data(), pk.data()));

    // Mark secret material as uninitialized, so that variable time implementation causes UUM.
    // First 16 bytes is a shared secret, then next 9616 is just a public key, and then next
    // 10240 is another part of secret material (a secret matrix S used by FrodoKEM). Both
    // shared secret and matrix S not leak, but it is OK to do variable-time operations on
    // public key.
    ct_poison(sk.data(), 16);
    ct_poison((unsigned char*)sk.data()+16+9616, 2*640*8);

    // Decapsulate
    res = pqc_kem_decapsulate(p, ss.data(), ct.data(), sk.data());

    // Purify res to allow non-ct check by ASSERT_TRUE
    ct_purify(&res, 1);
    ASSERT_TRUE(res);
}

The test is compiled with flags enabling Memory Sanitizer and origin tracking. When run, it correctly triggers UUM as expected.

./ut --gtest_filter="Frodo.CtDecaps"
Running main() from /home/kris/repos/pqc/3rd/gtest/googletest/src/gtest_main.cc
Note: Google Test filter = Frodo.CtDecaps
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from Frodo
[ RUN      ] Frodo.CtDecaps
Uninitialized bytes in MemcmpInterceptorCommon at offset 0 inside [0x7ffc94382040, 10240)
==3099896==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x559140afe8cd in memcmp (build.msan.debug/ut+0xd08cd)
    #1 0x5591410d5020 in PQCLEAN_FRODOKEM640SHAKE_CLEAN_crypto_kem_dec kem.c:233:9
    #2 0x559140e75c5b in pqc_kem_decapsulate pqapi.c:112:13
    #3 0x559140b30983 in Frodo_CtDecaps_Test::TestBody() test/ut.cpp:148:11
    #4 0x559140de55a4 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (build.msan.debug/ut+0x3b75a4)
    ...
  Uninitialized value was stored to memory at
    #0 0x5591410dec98 in PQCLEAN_FRODOKEM640SHAKE_CLEAN_key_decode /home/kris/repos/pqc/src/kem/frodo/frodokem640shake/clean/util.c:123:18
    #1 0x5591410d44e0 in PQCLEAN_FRODOKEM640SHAKE_CLEAN_crypto_kem_dec /home/kris/repos/pqc/src/kem/frodo/frodokem640shake/clean/kem.c:184:5
    ...

Runs as expected. In this case, Memory Sanitizer proves itself to be useful for the detection of code that’s not constant-time. It would find a bug in FrodoKEM if it was used.

In this case, ct_check.h has detected that uninitialized memory is used at line 229 (call to memcmp). It also gives a lot of additional output, due to origin tracking enabled (I have removed most of it). Now, to make the code constant-time, we must swap usage of memcmp with the implementation that compares bytes in constant-time. Implementation of such function looks like this:

// Compares in constant time two byte arrays of size 'n'
uint8_t ct_memcmp(const void *a, const void *b, size_t n) {
    const uint8_t *pa = (uint8_t *) a, *pb = (uint8_t *) b;
    uint8_t r = 0;
    // XOR bytes in 'a' with corresponding bytes in 'b'. If
    // all bytes are equal, 'r' will be == 0.
    while (n--) { r |= *pa++ ^ *pb++; }
    // Set most significant bit to 1 only if r!=0, otherwise
    // r stays == 0
    r   = (r >> 1) - r;
    r >>= 7;
    // return last byte - 0 means a==b
    return r;
}

After swapping memcmp with ct_memcmp and running with Memory Sanitizer, UUM has not triggered anymore. And in this case, that’s BAD. The first problem is fixed, but the second problem is not - code is still not constant-time. We can verify that by running the same code in Valgrind (thanks to ct_check.h).

> valgrind --tool=memcheck ./ut --gtest_filter="Frodo.CtDecaps"
==3096880== Memcheck, a memory error detector
==3096880== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3096880== Using Valgrind-3.17.0 and LibVEX; rerun with -h for copyright info
==3096880== Command: ./ut --gtest_filter=*CtDecaps*
==3096880==
Running main() from /home/kris/repos/pqc/3rd/gtest/googletest/src/gtest_main.cc
Note: Google Test filter = *CtDecaps*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from Frodo
[ RUN      ] Frodo.CtDecaps
==3096880== Conditional jump or move depends on uninitialised value(s)
==3096880==    at 0x1B721E: PQCLEAN_FRODOKEM640SHAKE_CLEAN_crypto_kem_dec (kem.c:244)
==3096880==    by 0x179C7D: pqc_kem_decapsulate (pqapi.c:112)
==3096880==    by 0x11A4AD: Frodo_CtDecaps_Test::TestBody() (ut.cpp:148)
...

Ok, Memcheck correctly reports an error. Interesting, but why?

Shadow memory propagation: MSan vs Valgrind

It took me some time to understand why it doesn’t work. It turns out that rules for shadow memory propagation are different in Memcheck and LLVM’s Memory Sanitizer. After analysing FrodoKEM, I’ve found out that the root cause boils down to the following example:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include "common/ct_check.h"

int main(int argc, const char* argv[]) {
    // Store 1 bit of first argument provided at command line
    uint16_t sign = ((uint16_t)atoi(argv[1])) & 1;
    uint16_t s;
    // Use ct_poison and logical AND to mark least significant bit
    // of 2-byte value as uninitialized.
    ct_poison(&sign, 2);
    sign = sign & 1;
    // Print shadow memory - should produce output 01 00.
    // It means that only least significant bit is uninitialized,
    // the rest is properly initialized.
    printf("Shadow memory: \n");
    ct_print_shadow(&sign, 2);
    // On Intel, negation is two's complement operation. So depending
    // on value of 'sign' (0 or 1), shadow propagation may be needed.
    s = (-sign); // Same as 's = ~sign + 1'
    ct_print_shadow(&s, 2);
    // Take a branch depending on uninitialized value. Should trigger UUM.
    s >>= 15;
    ct_print_shadow(&s, 2);
    if (s==0) {
        printf("Branch A taken\n");
    } else {
        printf("Branch B taken\n");
    }
}

The program reads input from the command line and stores it in 16-bit value sign. It marks the least significant bit of sign as uninitialized and then negates the value of sign. Negation on Intel platform is done by Two’s complement operation, hence it is equal to s = ~sign + 1. So, if sign == 0, then ~sign == 0xFFFF and adding value 1 will cause all the bits to flip, hence uninitialized state should be propagated to all the bits (similar operation is done by frodo_sample_n function in FrodoKEM).

Finally, a branch is taken depending on the value of a most significant bit of s. An important point to notice here is that - execution path of the program depends on uninitialized data, so UUM should be triggered. Let’s run, first Memory Sanitizer.

> clang -g -O0 -DPQC_USE_CTSANITIZER -fsanitize=memory -fno-omit-frame-pointer -fno-optimize-sibling-calls test.c
> ./a.out 0
Shadow memory:
01 00
01 00
00 00
Branch A taken
> ./a.out 1
Branch B taken

UUM happy, nothing reported. The ct_print_shadow shows the state of shadow memory - the second line shows that only the least significant bit is marked uninitialized, so after the right (logical) shift, all bits must be properly initialized. Now Memcheck:

> clang -g -O0 -DPQC_USE_CTGRIND -fno-omit-frame-pointer -fno-optimize-sibling-calls test.c
> valgrind --tool=memcheck ./a.out 0
==3156952== Memcheck, a memory error detector
==3156952== Command: ./a.out 0
Shadow memory:
01 00
FF FF
01 00
==3156952== Conditional jump or move depends on uninitialised value(s)
==3156952==    at 0x109239: main (test.c:26)
Branch A taken

As expected UUM is not happy. It is clear that shadow propagation rules are different - memcheck propagates shadow memory following complement’s two operation and Memory Sanitizer uses some less strict rules.

Initially, I thought that’s a bug (reported in GH#1430, GH#1424, GH#1427), but MSan maintainers from Google made me realize (thanks!) that it is a design decision which allows to faster execution. Indeed, looking at shadow propagation rules, described in “MemorySanitizer: fast detector of uninitialized memory use in C++” (see chapter 3.3.1), it seems shadow memory propagation following carry propagation is not implemented for efficiency reasons.

Speed

The graph below shows execution time difference when running FrodoKEM decapsulation without any instrumentation, with compile-time instrumentation Memory Sanitizer and then runtime instrumentation done by Valgrind’s Memcheck. The origin tracking is pretty expensive, so separated results are shown for a run with those enabled and disabled.

The control run shows that decapsulation takes around 3000 ms. With origin tracking disabled, the Memory Sanitizer seems to be 5 times slower comparing to the control run. Memcheck is 25 times slower. Then, with origin tracking enabled, Memory Sanitizer incurs 9x slowdown, in the case of Valgrind it is 59x - that’s a big difference.

Execution of a code instrumented at compile time is much faster. As described in the earlier section, Memcheck does more granular checks, so slower execution is expected. Nevertheless, the difference is significant. It should be noted that results do not include setup time. It means they are slightly biased as the whole runtime instrumentation done by Valgrind is not included in those results.

Conclusion, limitations and future direction

Memory Sanitizer, in some rare cases, is not going to discover uses of uninitialized data. This negatively impacts checking constant-time implementations. But still, it is pretty good at a job. My CI runs a build with Memory Sanitizer anyway, so adding extra checks has zero costs. Nevertheless, at the development stage, I use both, Memcheck and MSan the additional assurance provided by Memcheck is needed. I think it would be useful to have a possibility of controlling rules for shadow memory propagation in Memory Sanitizer. I.e. a compilation flag to use for choosing between more performant or more granular checks.

But what I would like to see in the future is a type system and better integration with a build system. Imagine that all the variables in the code that are used to store sensitive data, could use some kind of “secure” type (or annotation). Then, by introducing special build configuration, we could tell build system to instrument the code in a way that data using “secure” type is automatically marked uninitialized. In case of non constant-time access and with proper unit-tests, UUM detector would automatically report errors. I think it is an interesting feature for modern programming language like Rust, which seems to occupy a space of secure implementations.

Comming back to current state of art, it is also worth to mention that Memory Sanitizer has some additional limitations: * it is supported by Linux/x86_64, NetBSD and FreeBSD only * requires to instrument all memory accesses in the program. This includes standard C++ library (i.e. used by gtest). Instructions here describe how to instrument and integrate libc++ into a project.

Finally, side-channel attacks are much more complicated and there is no single tool which will be able to detect them. But from the other hand problems like the one in FrodoKEM, described above, are pretty basic. Automatic detection of such bugs is possible and and should be done by tools, so that Cryptography Engineers can spent time on more interesting things.

Sources

I’ll get NGINX sources, change it’s the build configuration, re-compile the server and rebuild deb package. To get sources, we have to add the NGINX repositories to the /etc/apt/sources/list.

The following two lines go to the end of a file:

deb https://nginx.org/packages/mainline/debian buster nginx
deb-src https://nginx.org/packages/mainline/debian buster nginx

Then add NGINX public key for verification and download the sources:

sudo wget https://nginx.org/keys/nginx_signing.key
sudo apt-key add nginx_signing.key
sudo apt-get update
sudo apt-get upgrade
sudo apt-get build-dep nginx
sudo apt-get source nginx

Sources should now be downloaded to the nginx directory.

Link NGINX with the BoringSSL

My current setup of HTTP server uses bssl. The choice comes from the fact that it’s simpler, documentation is clearer and it contains more modern features (or those I’m intersted in).

To link NGINX with BoringSSL, one needs to copy the sources to nginx/debian/modules and compile it.

cd nginx/debian/modules/ &&
git clone https://github.com/google/boringssl
mkdir -p boringssl/build && cd boringssl/build
cmake .. && make -j 8

Following step instructs NGINX to use BoringSSL instead of OpenSSL (used by default). To do that, one needs to modify the rules file nginx/debian/rules.

config.status.nginx: config.env.nginx
    cd $(BUILDDIR_nginx) && \
    CFLAGS="" ./configure {...} --with-stream_ssl_preread_module \
    --with-cc-opt="-I$(CURDIR)/debian/modules/boringssl/include $(CFLAGS) -Wno-ignored-qualifiers" \
    --with-ld-opt="-L$(CURDIR)/debian/modules/boringssl/build/ssl \
    -L$(CURDIR)/debian/modules/boringssl/build/crypto""

Example above shows how to modify a “release” target, but if needed, debug target can be modified in exact same way. Notice that -Wno-ignored-qualifiers has been added. That’s because BoringSSL throws compilation warnings, which become errors in the NGINX build.

Adding QUIC support

The QUIC protocol specified by RFC9000 opens new possibilities. Something I would definitely like to try. Clone the newest version and overwrite whatever is currently provided by NGINX.

hg clone -b quic https://hg.nginx.org/nginx-quic
rsync -r nginx-quic/ nginx

Again, the rules file needs to be modified to enable the support. Add --with-http_v3_module --with-http_quic_module --with-stream_quic_module to config.env.nginx and config.env.nginx_debug targets (somewhere after --with-stream_ssl_preread_module).

Package creation

Following commands re-create debian package with NGINX, which the can be installed by dpkg. This two step procedure requires, first to modify nginx/debian/changelog file and add information about changes done to the package. Add something like:

nginx (1.21.0-2~buster) buster; urgency=low

  * 1.21.0-1 adds quic

 -- Kris Kwiatkowski <kris@amongbytes.com>  Tue, 12 Jun 2021 16:01:22 +0300

Next step is to build a package. Build process will use GPG key to sign the package. To specify a key I use -kkris@amongbytes.com which identify secret key used for signing.

To start the build use following command:

sudo dpkg-buildpackage -b -kkris@amongbytes.com

NGINX configuration

Following instructions enable post-quantum in TLS and add support for QUIC protocol (unfortunatelly, PQ in QUIC is not supported).

1. Post-Quantum support: BoringSSL supports post-quantum key exchange. It can be enabled only in TLS v1.3 and uses a variant of NTRU-HRSS mixed with X25519, called CECPQ2 (detailed description here). To enable that support, following line needs to be added to nginx.conf

    ssl_protocols TLSv1.2 TLSv1.3;      # Enable both TLS 1.3 and 1.2
    ssl_ecdh_curve CECPQ2:X25519:P-256; # Enable PQ key exchange

It is important to add CECPQ2 as a first on that list as well as add some classical key exchange algorithm for backward compatibility. This server supports post-quantum key exchange.

2. QUIC support: for HTTP/3 over QUIC add following changes to the virtual server config:

server{
    listen 443 http3 quic reuseport;
    listen 443 ssl http2;

    quic_retry on;
    ssl_early_data on;

    http3_max_field_size 5000;
    http3_max_table_capacity 50;
    http3_max_blocked_streams 30;
    http3_max_concurrent_pushes 30;
    http3_push 10;
    http3_push_preload on;

    add_header alt-svc '$quic=":443"; ma=3600';

The add_header alt-svc is need to make sure that the web browser will know that your server supported http/3. Other settings also need to setup in order for the NGINX QUIC not to produce 404 error on your file assets.

Security

But firstly, why do I think secure storage provides enough security?

The TEE that I’m claims compliance with GlobalPlatform API. Looking at the GP requirements in this specification (see 2.2.2), the basic requirement regarding secure storage are to:

• obviously, encrypt the data (provide confidentiality as well as integrity)
• be bound to a device, this one is important. It means that sensitive data can be accessed only by those applications which are running on a particular device and in the particular TEE (there may be multiple TEEs on the same device).
• have an ability to hide sensitive keys form the TEE process running in the TEE
• allow access to the data only by the TEE application which has created it (btw: TA=Trusted Application, an application running in the TEE).

In my context, it means that VPN private key is stored encrypted and can be used only by a single device. The secure storage can be copied to a different device, but as it is bound to a particular one, it can’t be decrypted there. Key can’t be accessed by malicious TA installed on the same device thanks to access separation. Finally, the TA that owns the key doesn’t have access to the sensitive data, so in case of a bug in the TA, the key doesn’t leak. It may leak in case of bug in the TEE, but in this case, the whole system is probably already compromised.

The spec gives hope for a decent level of security. Looking at implementation details, the Key Manager is a component implemented in OP-TEE, which ensures confidentiality and integrity of the data (see implementation details of secure storage). To provide device binding it uses Hardware Unique Key (HUK), which is defined as symmetric secret key stored in a piece of hardware (often in the SoC itself) of the device and is globally unique. OP-TEE uses it to derive, a key called SSK, which is then used to provide device binding. SSK is created at boot-time and stored in secure memory (never stored on disk):

The SSK is then used to derive TSK key which is unique per TA installed in the TEE. This provides a possibility to allow access to the data only for TA which owns it. Finally, there is a FEK, randomly generated key used for file encryption.

An Important part of this whole story, but just implementation detail: the OP-TEE, as on GitHub, doesn’t actually try to use HUK. Retrieval of the HUK is specific to the SoC and needs to be implemented during integration with the concrete device/platform. Namely, there is a function called tee_otp_get_hw_unique_key, which must be filled with proper code for HUK retrieval. Similarly, to provide secure storage, the “chip ID” needs to be also retrieved, this is done by tee_otp_get_die_id which also needs to be filled with proper code. Currently, OP-TEE uses the stream of 0 bytes, as HUK.

Finally, the secure storage kept in normal world OS filesystem (/data/tee by default on linux). This subsystem uses AES/128. My ultimate goal is to have quantum-resistant TEE and AES/128 is too small to be resist quantum attacks (because of Grover’s algorithm), hence migration to 256-bit symmetric key is needed.

TLS client authentication

The X.509 certificates are used to authenticate a client to a VPN server. In this authentication method, a client sends a certificate and a proof for possession of the private key that corresponds to that certificate. In this case, the private key never leaves TEE, hence the primary functionality of an application running in the TEE, is to create a proof when requested.

Looking at the TLS level (TLSv1.3), the client authentication starts with a server requesting it in TLS Server Hello (4.3.2. of RFC 8446). In response, the client produces a proof by creating following signature:

The client uses the same algorithm as the one used when signing X.509 certificate and a private key, to create a signature. Signature is created over a concatenation of strings defined in the RFC (section 4.4.3) and a TLS transcript hash (section 4.4.1). Both, the X.509 certificate and proof are sent back to the server for verification.

Secure world

The Trusted Application is mostly copied from the previous project. In the current state, it is assumed that the key is loaded to TEE at some initial point, and then it is used when Normal World requests signing. An alternative implementation, could create a private key during the first boot and use it to create CSR, which is then signed by the CA and returned to the device. It’s a more complicated process, but this way, one can ensure that the client’s private key never existed anywhere else but on the device.

The demo TA comes with a simple key management app which can be used to install or remove keys from the device. It is also a good place to see how communication from Normal World to Secure World is implemented. Assuming, the TEE is running on the device, and tee-supplicant with Linux driver is loaded in the Normal World (see here for setup), an application can use GlobalPlatform API to send/receive requests to/from TEE. The code would look somehow like that:

    // TEE context
    TEEC_Context ctx;
    // Session with the TA
    TEEC_Session sess;
    // Operation context
    TEEC_Operation op;
    // ID of an app in the TEE
    TEEC_UUID uuid = TA_UUID;

    // Initialize a context connecting us to the TEE
    TEEC_InitializeContext(NULL, &ctx);
    // Open a session to the TA identified by uuid
    TEEC_OpenSession(&ctx, &sess, &uuid,
        TEEC_LOGIN_PUBLIC, NULL, NULL, &err_origin);

    // Initialize operation context 'op' (see github)
    // ...

    // Send command to the TA running in TEE
    TEEC_InvokeCommand(&sess, TA_INSTALL_KEYS, &op, &err_origin);

After opening a session with the TEE on a line 13, the application sets op context, by providing input arguments and setting buffers for the output. Then call to TEEC_InvokeCommand will trigger communication with the TEE. During this process, TA signature verification is done the TA is started. The entry point to the TA is a function called TA_InvokeCommandEntryPoint.

TEE_Result TA_InvokeCommandEntryPoint(void __maybe_unused *sess_ctx,
            uint32_t cmd_id,
            uint32_t param_types, TEE_Param params[4]) {
    (void)&sess_ctx; /* Unused parameter */
    switch (cmd_id) {
    case TA_INSTALL_KEYS:
        return install_key(param_types, params);
    case TA_SIGN_ECC:
        return sign_ecdsa(param_types, params);
    case TA_GET_PUB_KEY:
        return get_public_key(param_types, params);
        ...
    }
}

The TA is instructed by providing cmd_id to run specific logic, like key installation, signing or returning public key (the reason for which is described in next section). When installing the key, the TA will copy private and public key attributes to temporary transient_object and then create a file on persistent storage containing those attributes. The key is identified by key_id received from Normal World.

// Puts the key to the storage
static TEE_Result install_key(uint32_t param_types, TEE_Param params[4]) {
    //...
    TEE_ObjectHandle transient_obj = TEE_HANDLE_NULL;
    // ...
    TEE_AllocateTransientObject(TEE_TYPE_ECDSA_KEYPAIR,
            ecc->x.sz * 8, &transient_obj);
    ATTR_REF(cnt, TEE_ATTR_ECC_PRIVATE_VALUE, ecc->scalar);
    ATTR_REF(cnt, TEE_ATTR_ECC_PUBLIC_VALUE_X, ecc->x);
    ATTR_REF(cnt, TEE_ATTR_ECC_PUBLIC_VALUE_Y, ecc->y);
    TEE_InitValueAttribute(&attrs[cnt++], TEE_ATTR_ECC_CURVE,ecc->curve_id, 0);
    TEE_PopulateTransientObject(transient_obj, attrs, cnt);

    ret = TEE_CreatePersistentObject(
        TEE_STORAGE_PRIVATE,
        key_id, 32,
        TEE_DATA_FLAG_ACCESS_WRITE,
        transient_obj,
        NULL, 0, &persistant_obj);
    // ...
}

When signing, the TA will initialize key_handle - the handler to the key, it’s done by calling TEE_OpenPersistentObject with the key_id. Then, key_handle is used when setting up an operation identified by op (line 13) and finally used for signing (line 14). One should notice, that private key material stays in the TEE, it is never revealed to the TA.

// Performs ECDSA signing with a key from secure storage
static TEE_Result sign_ecds (uint32_t param_types, TEE_Param params[4]) {
TEE_OperationHandle op = TEE_HANDLE_NULL;
TEE_ObjectHandle key_handle;

TEE_OpenPersistentObject(
    TEE_STORAGE_PRIVATE,
    key_id, 32,
    TEE_DATA_FLAG_ACCESS_READ, &key_handle);

// perform ECDSA sigining
TEE_AllocateOperation(&op, TEE_ALG_ECDSA_P256, TEE_MODE_SIGN, 256);
TEE_SetOperationKey(op, key_handle);
TEE_AsymmetricSignDigest(op, NULL, 0,
    params[1].memref.buffer, params[1].memref.size,
    params[2].memref.buffer, &params[2].memref.size);
LOG_RET(ret);

}

The demo code (here) supports ECDSA/p256 only but can be easily extended to provide support for all the schemes used by TLS v1.3.

OpenSSL engine for OP-TEE

One of the goals for this project was the ease the integration with the TLS layer. It should be possible to provide whole functionality as a plugin loaded to any modern version of OpenSSL, code modifications. OpenSSL provides the possibility to extend functionalities by implementing, so-called, ENGINE API. The dynamically loadable library may implement some cryptographic operations (like signing, verification, key generation) and register it by calling ENGINE’s API. When processing a cryptographic operation the OpenSSL uses custom implementation if provided. The general architecture and guide to build OpenSSL engines can be found in an excellent paper called Start your ENGINEs: dynamically loadable contemporary crypto.

In case of engine for OP-TEE, the code structure looks briefly like:

static int OPTEE_ENG_bind(ENGINE *e, const char *id) {
    // ... some initialization code ...

    // Set name and ID of an engine
    ENGINE_set_id(e, OPTEE_ENG_ENGINE_ID);
    ENGINE_set_name(e, OPTEE_ENG_ENGINE_NAME);
    // Call OPTEE_ENG_load_private_key to load the private key
    ENGINE_set_load_privkey_function(e, OPTEE_ENG_load_private_key));
    // Register callback for signing
    ENGINE_set_pkey_meths(e, OPTEE_ENG_pkey_meths);
}
static int OPTEE_ENG_pkey_meths(ENGINE *e, EVP_PKEY_METHOD **pmeth,
    const int **nids, int nid) {
    // Use EVP_PKEY_meth_copy to copy all the callbacks to new_meth
    EVP_PKEY_METHOD *new_meth = EVP_PKEY_meth_new(EVP_PKEY_EC, 0);
    EVP_PKEY_meth_copy(new_meth, EVP_PKEY_meth_find(EVP_PKEY_EC));
    // Set new callback for signing
    EVP_PKEY_meth_set_sign(new_meth, 0, OPTEE_ENG_evp_cb_sign);
    // Return new EVP_PKEY_METHOD struture
    *pmeth = new_meth;
    return 1;
}

// Tell the OpenSSL to call OPTEE_ENG_bind when plugin is loaded
IMPLEMENT_DYNAMIC_BIND_FN(OPTEE_ENG_bind)
IMPLEMENT_DYNAMIC_CHECK_FN()

The OP-TEE engine adds to the OpenSSL with 2 following custom implementations. The OPTEE_ENG_load_private_key extends the functionality of theENGINE_load_private_key function. The former is an ENGINE API function used by the OpenVPN to load private keys. The custom implementation, provided by the optee_eng, checks if a key with the given ID exists in the TEE. It returns initialized EVP_PKEY object, used by the OpenSSL for message signing, during TLS session establishment. Contrary to standard implementation, EVP_PKEY object returned by optee_eng doesn’t store the private key material instead, it keeps an ID corresponding to the private key.

The second functionality is implemented by OPTEE_ENG_evp_cb_sign. This function gets invoked when signing is requested for a key returned by OPTEE_ENG_load_private_key. The EVP_PKEY contains a list of function pointers, implementing singing, verification, key generation, etc. This callback is assigned to a pointer for message signing. Implementation of this function, calls TA in the TEE with an ID of a key and a message to sign. Then control is transferred to sign_ecdsa function implemented by the TA, which initializes handle to the key and calls TEE OS to perform performs ECDSA/p256 signing.

The IMPLEMENT_DYNAMIC_BIND_FN macro binds everything together. It defines an entry point of an engine - a first function that gets executed when the library is loaded to the OpenSSL (OPTEE_ENG_bind in this case). The function sets an identifier and name of an engine and uses ENGINE API to assign the callbacks (line 8 and 18 in the code listing above).

Side note: In case of the private key, the OpenSSL v1.1.1 requires that EVP_PKEY structure contains a public part of a key, otherwise loading of the certificate fails and TLS client won’t be able to initialize the connection. In this program, the public part is stored also in the TEE.

Ok, so dynamic engine provides implementation, but OpenSSL needs to somehow know how to load such a library. Following configuration can be added to the OpenSSL’s config file (/etc/ssl/openssl.cnf on my Linux), so that framework knows where to find the dynamic library when requesting engine load by ID `` in this case.

# Additional content of openssl.cnf

[default_conf]
engines = engine_section

[engine_section]
optee = optee_section

[optee_section]
engine_id = optee
dynamic_path = "/opt/liboptee_eng.so"
init = 1

Let’s try, if it works. On qemu emulating ARMv8 machine I now get:

qemu> openssl
OpenSSL> engine -c -v optee
(optee) OpTEE OpenSSL ENGINE.
 [id-ecPublicKey]

Seems engine can be loaded correctly. Now, when OpenSSL tries to sign a message it needs to do a call to TEE (which is an SMC call to switch CPU into the secure world), get a key from secure storage and return the signature. Also, a crypto operation is now not done by OpenSSL, but by crypto library provided by the OP-TEE OS (in this case it is a fork of LibTomCrypt). All in all, there is a cost of all that dance. Measuring this difference will give some idea and also is a good way to check if the whole flow works correctly. That is done by speed.cc located in the project’s repository. Benchmark runs 2 functions, the SignREE performs signing, fully in the Normal World by calling pure OpenSSL implementation and SignTEE uses optee_eng for singing. I’ve got the following results when running it on HiKey960 (ARM Cortex-A73).

The operation works correctly - the benchmarking code loads optee_eng ENGINE into vanilla OpenSSL and uses only EVP_API. Nevertheless, the slowdown is significant. At this point I need to say, that I haven’t done any more investigation, hence I’m not sure where the slow down comes from exactly. I’ve run release version of the software and used similar settings for the board as described here. I’m pretty sure optimization level in OpenSSL is much better than in LibTomCrypt, hence there is probably lots of room for improvement.

Side note: the benchmark expects to find in TEE a key with a name bench_key. It must be inserted by using key management app optee_keymgnt put bench_key <PEM_file_with_a_key>.

Plugging to the OpenVPN

At this point integration with the OpenVPN is very easy. The only requirement is a version 2.5 of the software (which includes this change). That change adds the possibility to use OpenSSL ENGINE to load private key, what’s needed here.

There is a trick that needs to be used here to configure OpenVPN correctly. So, the configuration file specifies has a key parameter which specifies the name of the file with the private key, corresponding to the certificate provided by cert parameter. In case of optee_eng, this is a name of the key stored in the TEE (this name is provided to ENGINE_load_private_key as key_id argument). Additionally, file with the same name must exist in the OpenVPN configuration directory. The OpenVPN will try to use the engine to load a key, only if loading from the file fails. So the file needs to be empty, to make sure the load of a key fails. The configuration needs to also specify engine parameter, to instruct OpenSSL to use the optee_eng. Whole configuration file as used on the client can be found here.

Setting-up OP-TEE image, building & running

The code of the solution is available on github. It was tested against OP-TEE 3.11, running in QEMU and on HiKey960 development board. To build and play with the solution, one requires first to build the OP-TEE itself (instructions here).

To compile the solution:

git clone https://github.com/henrydcase/optee_eng
cd optee_eng
git submodule init && git submodule update
mkdir build && cd build
cmake -DOPTEE_BUILD_DIR=<OPTEE location> -DPLATFORM=qemu ..
make
make install

The <OPTEE location> is a root directory for OPTEE. The -DPLATFORM specifies a platform for which solution should be built. I’ll use QEMU in this example. The make install command will copy all needed files to the OP-TEE’s build directory.

OP-TEE uses buildroot to create Normal World OS, where examples can be run. By default, the OpenVPN is not enabled. It can be done by applying 2 patches from optee_eng repo:

cd optee_eng
patch -p1 -d <OPTEE location>/buildroot < optee-patches/0001-openvpn-2.4.9-to-2.5.0.patch
patch -p1 -d <OPTEE location>/build < 0002_build_enable_openvpn.patch
cd <OPTEE location>/build
make run

To connect to the VPN, we need a server. The repository contains configuration for server and client, as well as a set of X.509 certificates (to regenerate certificates the create_cert.sh can be used). The command below configures and starts OpenVPN server on the host machine.

> cd optee_eng
> sudo openvpn --cd cfg --config openvpn_srv.conf
2021-02-06 23:39:53 OpenVPN 2.5.0 [git:makepkg/a73072d8f780e888+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Nov  6 2020
2021-02-06 23:39:53 library versions: OpenSSL 1.1.1h  22 Sep 2020, LZO 2.10
2021-02-06 23:39:53 TUN/TAP device tun0 opened
2021-02-06 23:39:53 net_iface_mtu_set: mtu 1500 for tun0
2021-02-06 23:39:53 net_iface_up: set tun0 up
2021-02-06 23:39:53 net_addr_v4_add: 172.16.0.1/16 dev tun0
2021-02-06 23:39:53 UDPv4 link local (bound): [AF_INET][undef]:1194
2021-02-06 23:39:53 UDPv4 link remote: [AF_UNSPEC]
2021-02-06 23:39:53 Initialization Sequence Completed

Once QEMU is started and the user is log-in as root in NWd terminal, the tee-supplicant -d needs to be started. The supplicant makes it possible to communicate from Normal World to Secure World. Then next thing to do is to, is to insert a client key into TEE and start VPN.

> optee_keymgnt put vpn.testlab.com /etc/openvpn/certs/client.key
> rm /etc/openvpn/certs/client.key
> openvpn --cd /etc/openvpn/ --config client.conf
2021-02-09 00:27:27 Initializing OpenSSL support for engine 'optee'
2021-02-09 00:27:27 OpenSSL: error:0909006C:PEM routines:get_name:no start line
2021-02-09 00:27:27 PEM_read_bio failed, now trying engine method to load private key
2021-02-09 00:27:27 TCP/UDP: Preserving recently used remote address: [AF_INET]172.16.0.1:1194
...
2021-02-09 00:27:28 [vpn.testlab.com] Peer Connection Initiated with [AF_INET]172.16.0.1:1194
...

The second terminal displays logs from TEE OS running in parallel to Linux. One should see the following traces there:

# When inserting a key to the TEE
I/TA: New key [F671A1B757] registered
# During TLS handshake
I/TA: Sign for a key ID [F671A1B757] requested
I/TA: Message signed with key ID [F671A1B757]

At this point the VPN tunnel should be correctly created.

Conclusion

Hopefully, this example shows how to utilize ARM TrustZone from OpenSSL to secure private keys for OpenVPN. Ideas similar to implemented by optee_eng can be used with any software using OpenSSL - the same engine can be used by Nginx, ssh-agent or strongSwan on both client and server-side. The solution is fully “pluggable”, it doesn’t require any modification to existing software. It’s worth to notice that such isolation of private keys from internet-facing applications, may help to avoid security incidents. For example, it would be enough to use optee_eng to avoid hearthbleed, as the private key is not stored in the process running OpenSSL library.

As an improvement to this idea, one could think of using PKCS#11 standard for communication with TEE. It wasn’t done here for 2 reasons - PKCS#11 would require TA implementing the standard, which is not finished yet (but ongoing). The other reason is that my ultimate goal (which wasn’t presented here) is to use post-quantum cryptography. Those new schemes are not yet incorporated properly into PKCS#11 standard.

Finally, upcoming OpenSSL 3.0 removes support for ENGINE API completely. Instead, there is a new concept called providers. Hence, implementation of optee_eng for the upcoming version of OpenSSL will look probably slightly different. But from one hand OpenVPN doesn’t support this new version yet and from the other hand, it doesn’t seem 3.0 provides yet similar functionality for loading private keys.

Solution proposed

For brevity I’m assuming server uses only TLS 1.3 as specified in [RFC8446], but solution can be adapted to any version of TLS.

The idea is to perform TLS session signing inside Trusted Execution Environment. The traffic-private-key will be accessible only to TEE. Additionally, solution ensures that key is stored in encrypted form in trusted storage. The storage is bound to the physical machine and hence copy of the storage can’t be used on some different machine.

The solution as implemented in the PoC and described below is based on ARM TrustZone and it uses open sourced TEE called OP-TEE (see [OP-TEE]), sources of OP-TEE are stored on github (see [OP-TEE-SRC]). OP-TEE was driven by the fact that author is quite familiar with environment nevertheless it can be implemented with other TEEs which provide device bound trusted storage. Author is convinced that Intel SGX with Asylo would be better choice here. Solution makes also heavy use of BoringSSL for handling with TLS traffic.

Points below describe implementation in more details:

1. Key provisioning server

   It is assumed that machine is initially provisioned with a software which acts as a server for traffic-private-key provisioning.

   In order to install traffic-private-key on a machine, operator connects to key provisioning server and sends the traffic-private-key to be installed on the machine. This operation is done over TLS connection which uses client authentication. Possition of some form of TLS provisioning is required by the operator. Key provisioning server must be able to verify provisioning key, hence verification-provisioning-key is also preinstalleld.

   After sucessuful TLS authentication, operator sends a pair of traffic-private-key and domain name for which the key must be used. This pair is installed on secure storage which accessible from TEE only. TEE ensures traffic-provisioning-key can’t be read from outside of TEE.

2. TLS session signing

   Solution uses BoringSSL to offload TLS traffic. BoringSSL API gives a possibility to register a function which is called during TLS handshake, when server needs to sign a session with traffic-private-key.

   It means that there are no modifications needed to BoringSSL in order to use it for signing TLS session with traffic-private-key stored in TEE.

   The code which registers signing operation looks like this:

   void signing_operation(message_to_sign, domain_name, *signature) {
       ... calls TEE for signing ...
   }
   
   SSL_PRIVATE_KEY_METHOD private_key_methods {
       .sign = signing_operation
       .decrypt = ...
       .complete = ...
   };
   SSL_CTX_set_private_key_method(SSL_CTX, &private_key_methods)

   TLS server calls signing_operation function when TLS session needs to be signed. This function passes message_to_sign and domain_name to the TEE. While in the TEE, the domain_name is used as an index in order to retrieve right traffic-private-key (many domains can be handled by the server). TEE performs signing and signature is returned to the BoringSSL. BoringSSL continues TLS handshake as normal.

3. Key can’t be used on another machine.

   Trusted storage in OP-TEE is bound to the physical device. It means even if the storage is coppied to another device, it won’t be possible to decrypt stored data.

   In more detail, OP-TEE implements GlobalPlatform Trusted Storage API. Device binding is one of the requirements for trusted storage. In order to make it possible each device needs to come with preinstalled Hardware Unique Key (HUK).

   More details about trusted storage can be found on in OP-TEE documentation (see [OP-TEE-STORAGE]).

   It must be mentioned, that in order to use trusted storage, SoC specific customization is needed (see comment in orange at the bottom of [OP-TEE-STORAGE]).

PoC implementation

As mentioned before, implementation uses OP-TEE as a base for TEE. Example was tested with OP-TEE running inside QEMU emulating ARMv8.

PoC is composed of:

• admin_cli: Client used for installing the private keys inside TEE. This component is used instead of key provisioning server as such server was not implemented in PoC.

• server: It is a TLS offloading server. Server listens on 127.0.0.1:443 and uses BoringSSL to accept and handle TLS connection. Server implements function callback, which calls TEE when private key operation needs to be done. Only ECDSA/P256 sining is currently supported.

• ta: Trusted application running inside TEE. The application is responsible for processing requests from admin_cli, which is storing the keys on trusted storage and deleting them if requested. As well as processing signing requests from the server.

The section called “Example of usage” explains how to use the software in details.

Links

• [PoC code] https://git.amongbytes.com/kris/TLS-delegation-to-TEE
• [RFC8446] https://tools.ietf.org/html/rfc8446
• [OP-TEE] https://www.op-tee.org/
• [OP-TEE-SRC] https://github.com/OP-TEE
• [OP-TEE-STORAGE] https://optee.readthedocs.io/architecture/secure_storage.html
• [BORING-BUILD] https://github.com/google/boringssl/blob/master/BUILDING.md

Requirements

The solution needs to be implemented in C and have following functionalities * Possibility to connect to I2C slave * Send encrypted data * Receive and decrypt data * Possibility to check connection status

The code

The code itself is here. To compilie with gcc simply download and make.

> modprobe i2c-dev
> modprobe i2c-stub chip_addr=0x03

> i2cdetect -l
i2c-1   i2c         i915 gmbus dpc                      I2C adapter
i2c-2   i2c         i915 gmbus dpd                      I2C adapter
...
i2c-8   smbus       SMBus stub driver                   SMBus adapter <-- this one
...

> i2cdump -y 8 0x03
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

> ./bin/main -s 8 && echo $?
0

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 1f f8 47 8e 7f 24 1d 2b 47 ca 64 be ce 0a 3f bd    ??G??$?+G?d?????
10: 08 1c 05 87 b0 31 6c 85 46 94 6f c8 9e 49 dd b2    ?????1l?F?o??I??
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

[root@cryptoden final]# ./bin/main -r 8
RECEIVED DATA:
HELLO WORLD!!!

> i2cdump -y 8 0x03 b > dump
> cat dump
[root@cryptoden ~]# cat dump
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 1f f8 47 8e 7f 24 1d 2b 47 ca 64 be ce 0a 3f bd    ??G??$?+G?d?????
10: 08 1c 05 87 b0 31 6c 85 46 94 6f c8 9e 49 dd b3    ?????1l?F?o??I??
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

## Here I modify 32-nd byte from b3 to b2 and load data to i2c-stub
> i2c-stub-from-dump 0x03 dump
256 byte values written to 8-0003

## Trying to read
> ./bin/main -r 8
[i2c_recv() src/i2c.c:165] Error occured when decrypting
[i2c_recv() src/i2c.c:172] ERROR: can't receive encrypted data
[test_receive() src/main.c:88] Error occured when receiving data

> ./bin/main -t 8 && echo $?
0

Additional links

• https://robot-electronics.co.uk/i2c-tutorial
• http://alokprasad7.blogspot.com/2018/01/fake-i2c-device-i2c-stub.html
• https://electronicayciencia.github.io/wPi_soft_i2c/
• https://learn.sparkfun.com/tutorials/i2c/all
• https://electronicayciencia.github.io/wPi_soft_i2c/

Step 1: SIDH support in TLS

With tls-tris it is possible to perform SIDH key exchange. In order to link application (Caddy in this case) with TLS tris one needs to swap TLS implementation in standard library and recompile Go from source. All this is implementated in the Makefile that comes with the library. Binaries are placed in tls-tris/_dev/GOROOT folder, the GOROOT needs to point to this folder.

Let’s first download needed sources:

# Create some workspace
WORKSPACE=/tmp/workspace/
mkdir -p ${WORKSPACE}

# Get Go 1.10 sources (if not already done)
cd ${WORKSPACE} 
wget https://dl.google.com/go/go1.10.4.linux-amd64.tar.gz -O - | tar -xz

# Clone tls-tris 
git clone https://github.com/cloudflare/tls-tris

Next step is to build Go with TLSv1.3 and SIDH support. This is done automatically and makefile implements all needed steps. The build-all target will basically:

• Swap standard library TLS with tls-tris
• Download SIDH crypto library and vendor it to ${WORKSPACE}/go/src/vendor directory. This way SIDH is available as it would be a part of standard library.

# By specifying GOROOT_ENV makefile knows where to look for Go sources
cd tls-tris; GOROOT_ENV=${WORKSPACE}/go make -f _dev/Makefile build-all

Finally GOROOT needs to be adjusted:

export GOROOT=${WORKSPACE}/tls-tris/_dev/GOROOT/linux_amd64

Step 2: Patching Caddy

By default Caddy supports TLS up to version 1.2. Thanks to steps above, TLSv1.3 and Hybrid SIDH/503-X25519 key exchange are ready to use. Nevertheless, some code changes to Caddy are needed in order to benefit from those new features.

Let’s start by cloning the sources:

mkdir -p /tmp/gopath
export GOPATH=/tmp/gopath
go get github.com/mholt/caddy/caddy 
go get github.com/caddyserver/builds
cd $GOPATH/src/github.com/mholt/caddy

Now we need to add 2 lines of code in order to: 1. Enable TLS v1.3: In file caddytls/config.go Caddy keeps a map of supported TLS protocols. The TLSv.1.3 needs to be added to this map. 2. Enable SIDH: In the same file Caddy also keeps map of supported curves. The scheme called tls.HybridSIDHp503Curve25519 needs to be added to this map.

The whole diff should look something like that:

> git diff
diff --git a/caddytls/config.go b/caddytls/config.go
index 8cf61e4..fc7510d 100644
--- a/caddytls/config.go
+++ b/caddytls/config.go
@@ -583,6 +583,7 @@ var SupportedProtocols = map[string]uint16{
        "tls1.0": tls.VersionTLS10,
        "tls1.1": tls.VersionTLS11,
        "tls1.2": tls.VersionTLS12,
+       "tls1.3": tls.VersionTLS13,
 }
 
 // GetSupportedProtocolName returns the protocol name
@@ -682,6 +683,7 @@ var supportedCurvesMap = map[string]tls.CurveID{
        "P256":   tls.CurveP256,
        "P384":   tls.CurveP384,
        "P521":   tls.CurveP521,
+       "SIDH/503-X25519": tls.HybridSIDHp503Curve25519,
 }
 
 // List of all the curves we want to use by default.

With those changes applied, Caddy can be built:

# Once again, just to make sure right Go version is used
export GOROOT=${WORKSPACE}/tls-tris/_dev/GOROOT/linux_amd64

# And let's build caddy
cd $GOPATH/src/github.com/mholt/caddy/caddy
go run build.go

Caddy configuration and server bring up

At this point hybrid post quantum key exchange should be supported by Caddy. Obviusly that’s not a default configuration, so one needs to tell Caddy to use it. Caddy is configuring by providing Caddyfile. In this file max version of the protocol is set to “tls1.3”. Also elliptic curve preferences are changed by specifying “SIDH/503-X25519” as key exchange algorithm. My minimal Caddyfile looks like this:

localhost:2015 {
        tls self_signed
        tls {
                protocols tls1.2 tls1.3
                curves X25519 P256 "SIDH/503-X25519"
        }
        log stdout
        proxy / http://www.amongbytes.com
}

This file is placed in the same folder where “caddy” binary lives, namely $GOPATH/src/github.com/mholt/caddy/caddy/Caddyfile. It will basically open a port 2015 on localhost and forward all traffic to http://www.amongbytes.com. The self_signed certificate will be used in this test configuration. Let’s start it:

> cd $GOPATH/src/github.com/mholt/caddy/caddy
> ./caddy
Activating privacy features... done.
https://localhost:2015

Looks like it’s working. Now it would be good to actually test if post quantum key exchange is working. There are probably not too many browsers supporting SIDH/503-X25519 (if any). Nevertheless tls-tris contains a patch for boringssl which adds SIDH and is used for interoperability testing. We can reuse it to test our setup.

cd ${WORKSPACE}

# Patch for BoringSSL adding SIDH/P503-X25519 support
wget https://raw.githubusercontent.com/cloudflare/tls-tris/master/_dev/boring/sidh_ff433815b51c34496bb6bea13e73e29e5c278238.patch

# Clone and checkout BoringSSL. I'm using specific commit as I want to make sure patch applies correctly
git clone https://boringssl.googlesource.com/boringssl
cd boringssl
git fetch && git checkout ff433815b51c34496bb6bea13e73e29e5c278238 
patch -p1 < ../sidh_ff433815b51c34496bb6bea13e73e29e5c278238.patch

# When building, make sure EXP_SIDH is defined as it enables SIDH
mkdir build && cd build; cmake -DEXP_SIDH=1 -GNinja .. && ninja

Assuming server is running we can now check if quantum resistant TLS handshake works.

> ./tool/bssl client -curves x25519sidh503 -connect localhost:2015                                         
Connecting to [::1]:2015
Connected.
  Version: TLSv1.3
  Resumed session: no
  Cipher: TLS_AES_128_GCM_SHA256
  ECDHE curve: x25519sidh503
  Signature algorithm: ecdsa_secp256r1_sha256
  Secure renegotiation: yes
  Extended master secret: yes
  Next protocol negotiated: 
  ALPN protocol: 
  OCSP staple: no
  SCT list: no
  Early data: no
  Cert subject: O = Caddy Self-Signed
  Cert issuer: O = Caddy Self-Signed

BoringSSL reports that ECHDE curve used for key exchange was “x25519sidh503”. It seems post quantum key exchange works just all right!

Conclusion and next steps

I’m neither an expert nor big fan of Golang. Nevertheless, I think it is great language for performing experiments - especially those related to networking. The setup presented here is used for experiments related to post-quantum cryptographic primitive implementation. I’m using it both on ARM and Intel and thanks to Golang’s build tools, compilation process is very simple, fast and quite easy to perform.

The SIDH looks interesting as a quantum resistant replacement for ECDH. Nevertheless, KEM construction providing IND-CCA2 security sounds to me like something worth trying. In the next steps I will be experimenting with other algorithms - my non exhaustive list contains Round5 presenting very interesting results, something NTRU based, Kyber and CSIDH. Also at further step I’ll try to tackle quantum-resistant signature schemes.

Testing application

It’s a implementation of simple C-based test application, which compiles and links against library under test and run on ARMv8 platform running Android operating system. The app is composed of client and server. As I’m only interested in client side of the TLS end, we fix the server to always use same library (it’s based on BoringSSL). Server is configured to support only TLSv1.2 (as 1.3 is not supported by mbedTLS, yet [16]). In order to start a server, user provides an argument which specifies cetificate type to be used (RSA, ECDSA or EdDSA based). Once run it always enforces same cipher suite to be used - for example in case of RSA it will be ECDHE key agreement with RSA signature and AES/256 in GCM AEAD mode.

Client application is the one which I want to benchmark. I have implemented one which uses mbedTLS API and links with this library and similar one for BoringSSL. Client always establishes TCP connection in blocking mode (simplicity). It implements 3 different tests:

• Handshake : during this test client opens TCP connection and performs many handshake without closing the connection. Performance of this test depends on key type used for certificate signing and symmetric key agreement algorithm (as well as elliptic curve used), hence this test is performed multiple times, once for each certificate type
• Write: clients opens TCP connection and sends few hundred megabytes of data. This test is done mostly to assess performance of symmetric encryption
• Read: clients opens TCP connection and sends a request to the server which sends back few hundred megabytes of data. This test is done mostly to assess performance of symmetric decryption

Preparation step

Following script is used to set-up platform for benchmarking. Most important step is to fix CPU frequency so that it is not auto-regulated by things like EAS [11].

# Number of CPUs on the board
NUM_CPU=8
# CPU scaling governor.
GOVERNOR=userspace
# Requested CPU frequency
MAX_FREQ=1200000


adb root
adb remount
# Prevent system from suspending
adb shell "echo temporary > /sys/power/wake_lock"
# Probably useful only on qcom, but anyway...
adb shell stop thermal-engine
adb shell stop mpdecision

for ID in `seq 0 $((NUM_CPU-1))`
do
adb shell "echo 1 > /sys/devices/system/cpu/cpu${ID}/online"
adb shell "echo ${GOVERNOR} > /sys/devices/system/cpu/cpu${ID}/cpufreq/scaling_governor"
adb shell "echo ${MAX_FREQ} > /sys/devices/system/cpu/cpu${ID}/cpufreq/scaling_setspeed"
done

for ID in `seq 0 $((NUM_CPU-1))`
do
adb shell "cat /sys/devices/system/cpu/cpu${ID}/online"
adb shell "cat /sys/devices/system/cpu/cpu${ID}/cpufreq/scaling_governor"
adb shell "cat /sys/devices/system/cpu/cpu${ID}/cpufreq/scaling_cur_freq"
done

Binary size reduction

Notes on size reduction

• Something that wasn’t tried is a Link Time Optimization feature which may provide binary with reduced size (see [3], [4] and [5]).
  • It might be interesting to see how different results will be when using this features instead/with section indexing
• I’ve calculated also size of shared libraries for boring ssl - libcrypto.so: 1072KB; libssl.so: 276KB
• mbedTLS doesn’t implement hardware acceleration, so performance won’t be as good as for BoringSSL. I wonder if it would make sense to take exremly small SSL implementation from mbedTLS and use crypto from BoringSSL.

Performance comparison comparison

Out of scope / left for further analysis:

Few points there were not checked:

• 32bit (armeabi-v7a) code may be smaller and still run on ARM64. Thumb mode (variable-length instruction set) will produce even more compact code. Thumb mode is default setting in NDK
• Something I havn’t checked is a power consumption, which is important in case of mobile application. It’s not complicated but requires specific hardware (see [18] and [19]). I assumed that thing which executes in smaller amount of time will consume less. But this assumption should be verified, as it’s probably not always true.
• Implementing hardware acceleration mbedTLS is obvious improvement which should be considered. See here for more details. It is also highly time consuming task.
• mbedTLS supports so called “alternative” implementation. One idea on using it would be to swap existing implementation of ECC with either smaller or faster implementation (for smaller implementation I would recomend uECC, which can be as small as 4KB). Other option could be to use small SSL implementation from mbedTLS and fast crypto implementation from BoringSSL or NaCL [17].
• mbedTLS has a configuration option called MBEDTLS_SSL_MAX_CONTENT_LEN which determines the size of internal I/O buffer. Playing with this value may help improve performance or reduce size.
• Performance of Poly-ChaCha on ARMv7

Conclusion

My preference goes to BoringSSL for following reasons:

• It offers much better performance on ARM
• It offers more features like TLSv1.3 and Curve25519
• It compiles to binary size which is reasonable. Smallest possible resulting library is 3 times bigger than the one based on mbedTLS, overall result is just 350KB. The difference between smallest possible mbedTLS based client and BoringSSL one is just 248KB. Let say the library will be linked to each and every application on the phone. Assuming user has has 100 apps on a phone, the difference in size is 24MB, which nowadays is negligible. Also ccording to report by Statista [14], on average users have 27 apps instaled on the phone (which is less an argument and more interesting information).
• BoringSSL is a default TLS library on Android and is a Google product. It means that there is a lot of intrest to make even more secure and fast.
• Recently BoringSSL received formally verified implementation of Curve25519 and P-256 (see [15])

It seems both libraries have very different design goals. mbedTLS is made for resource constrained embedded systems, which face challanges in terms of memory availability. Embedded platforms often do not exceed 256KB of RAM, often don’t have memory management units and cannot support virtual memory, as a result dynamic allocation is avoided. I believe for such systems mbedTLS is unbeatable and a great choice.

BoringSSL doesn’t seem to have similar design goal. It seems to be designed for devices which offer more RAM, storage space and in general have much different profile than resource constrained embedded systems. Mobile devices offer all those features and it would be huge mistake not make use of it.

When thinking about software design, there is great difference between aiming for “reasonably small” and “smallest possible bianry size” - those are basically two different goals.

Finally

I would like to thank Ron E. from mbedTLS team for all the answers for my questions.

UPDATE: Recently one of my co-workers has implemented performance improvement for ARMv64. It is small change which give good speedup - see more details (here)[https://github.com/ARMmbed/mbedtls/pull/1964].

Footnotes

• [0] Android NDK: reducing binary sizes: https://blog.algolia.com/android-ndk-how-to-reduce-libs-size/
• [1] to check: https://stackoverflow.com/questions/6771905/how-to-decrease-the-size-of-generated-binaries
• [2] C/C++ reducing size http://ptspts.blogspot.co.uk/2013/12/how-to-make-smaller-c-and-c-binaries.html
• [3] “Link time optimization” in https://www.iecc.com/linker/linker11.html
• [4] LTO GCC: https://gcc.gnu.org/onlinedocs/gccint/LTO-Overview.html
• [5] LTO LLVM: https://llvm.org/docs/LinkTimeOptimization.html
• [6] https://github.com/ARMmbed/mbedtls/pull/1424
• [7] “Link time garbage collection” in https://www.iecc.com/linker/linker11.html
• [8] https://github.com/android-ndk/ndk/issues/436
• [9] https://tls.mbed.org/kb/how-to/reduce-mbedtls-memory-and-storage-footprint
• [10] https://github.com/ARMmbed/mbedtls/issues/941
• [11] https://wiki.linaro.org/WorkingGroups/PowerManagement/Resources/EAS
• [12] https://boringssl.googlesource.com/boringssl/+/HEAD/BUILDING.md
• [13] https://blog.cloudflare.com/introducing-0-rtt/
• [14] https://www.apptentive.com/blog/2017/06/22/how-many-mobile-apps-are-actually-used/
• [15] https://boringssl.googlesource.com/boringssl/+/HEAD/third_party/fiat/
• [16] https://tls.mbed.org/discussions/feature-request/any-plans-for-tls-1-3-support
• [17] https://eprint.iacr.org/2018/354/20180418:202819
• [18] https://source.android.com/devices/tech/power/component
• [19] https://developer.arm.com/products/software-development-tools/ds-5-development-studio/streamline/arm-energy-probe

Configuration file

OpenSSL uses configuration file in order to store information required during certificate creation. Configuraiton file contains things like organization name, address, location, internet address, default hash algorithm used to produce signatures, etc.

Name of both - my example CA and an organization for which server certificate will be created - is called “Cert Testing Organization” with an address www.cert_testing.com.

Here below configuration file used in this example. Copy & paste it to file openssl.cnf:

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = .
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/root.key
certificate       = $dir/root.pem

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/intermediate.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 9999
preserve          = no
policy            = policy_loose

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the `ca` man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name (full name)
localityName                    = Locality Name (eg, city)
organizationalUnitName          = Organizational Unit Name (eg, section)
commonName                      = Common Name

stateOrProvinceName_default     = PACA
countryName_default             = FR
localityName_default            = Cagnes sur Mer
organizationalUnitName_default  = Cert Testing Organization
commonName_default              = Cert Testing Organization
commonName_max                  = 64

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier        = hash
authorityKeyIdentifier      = keyid:always,issuer
basicConstraints            = critical, CA:true
keyUsage                    = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier        = hash
authorityKeyIdentifier      = keyid:always,issuer
basicConstraints            = critical, CA:true, pathlen:0
keyUsage                    = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints        = CA:FALSE
nsCertType              = client, email
nsComment               = 'Cert Testing Intermediate - Client'
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid,issuer
keyUsage                = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage        = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints        = CA:FALSE
nsCertType              = server
nsComment               = 'Cert Testing Intermediate - Server'
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid,issuer:always
keyUsage                = critical, digitalSignature, keyEncipherment
extendedKeyUsage        = serverAuth
subjectAltName          = @alt_names

[ client_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints        = CA:FALSE
nsCertType              = client, email
nsComment               = 'Cert Testing EE - Client'
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid,issuer
keyUsage                = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage        = clientAuth, emailProtection

[ crl_ext ]
# Extension for CRLs (`man x509v3_config`).
authorityKeyIdentifier  = keyid:always

[ ocsp ]
# Extension for OCSP signing certificates (`man ocsp`).
basicConstraints        = CA:FALSE
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid,issuer
keyUsage                = critical, digitalSignature
extendedKeyUsage        = critical, OCSPSigning

[alt_names]
DNS.1   = *.cert_testing.com
IP.1    = 127.0.0.1

Preparation

We will need some directories where output of cert generation will be stored:

mkdir -p private
mkdir -p certs
mkdir -p csr

CA cert creation

1. CA private key

   First step is to create private key of CA cert. Root cert will use RSA keypair with key length of 4096 bits.

   OpenSSL will ask for pasword - provide test123.

    openssl genrsa -aes256 -out private/ca.key 4096

or in case of ECDSA certificates:

    openssl ecparam -name prime256v1 -genkey -noout -out private/ca.key
    openssl ec -in private/ca.key -out private/ca.key -aes256

Here second line (encrypting ca.key) is needed only for rest of the article to be copy-paste'able.

2. Create CA cert

   This command will use a key created above and create self-signed CA certificate. Certificate will be valid for 9999 days.

   Provide password test123 and hit enter on everything else. openssl will use values defined in openssl.cnf.

     openssl req -config openssl.cnf \
        -extensions v3_ca -new -x509 -days 9999 \
        -key private/ca.key \
        -out certs/ca.cert

One interesting option to notice is ``-extensions v3_ca`` - it is reference to the section with the same name in ``openssl.cnf``. This section tells the ``openssl`` that created certificate must be a CA cert (``CA:true``).

Server cert creation

In this example, certificate signing is done in 3 steps.

• Create server certificate private key
• Create certificate singing request
• Sign the request with CA private key

So let’s do it.

1. Server’s private key (I skip intermediate certs creation for the brevity).

   • RSA/2048 with e=3, for fast verification

        openssl genpkey -algorithm RSA \
            -pkeyopt rsa_keygen_bits:2048 \
            -pkeyopt rsa_keygen_pubexp:3 \
            -out private/rsa_2048.key

* ECDSA/P-256

        openssl genpkey -algorithm EC \
            -pkeyopt ec_paramgen_curve:P-256 \
            -pkeyopt ec_param_enc:named_curve \
            -out private/ecdsa_p256.key

* EdDSA/25519 (supported by newer version of ``openssl`` and in TLS 1.3 only)

        openssl genpkey -algorithm Ed25519 \
            -out private/ed25519.key

2. Create certificate signing request - intermediary step

   • RSA

        openssl req -config openssl.cnf -new \
          -sha256 \
          -passin pass:test123 \
          -key private/rsa_2048.key \
          -out csr/rsa_2048.csr \
          -days 9999

   • ECDSA

        openssl req -config openssl.cnf -new \
          -sha256 \
          -passin pass:test123 \
          -key private/ecdsa_p256.key  \
          -out csr/ecdsa_p256.csr \
          -days 9999

   • EdDSA

        openssl req -config openssl.cnf -new \
          -passin pass:test123 \
          -key private/ed25519.key  \
          -out csr/ed25519.csr \
          -days 9999

3. Create server cert

   Finally we can create set of server certificates.

   • RSA

        openssl x509 \
          -extfile openssl.cnf \
          -extensions server_cert -sha256 -req  \
          -CA certs/ca.cert -CAkey private/ca.key -CAcreateserial \
          -passin pass:test123 \
          -in csr/rsa_2048.csr \
          -out certs/rsa_2048.cert \
          -days 9999

   • ECDSA

        openssl x509 \
          -extfile openssl.cnf \
          -extensions server_cert -sha256 -req  \
          -CA certs/ca.cert -CAkey private/ca.key -CAcreateserial \
          -passin pass:test123 \
          -in csr/ecdsa_p256.csr \
          -out certs/ecdsa_256.cert \
          -days 9999

   • EdDSA

        openssl x509 \
          -extfile openssl.cnf \
          -extensions server_cert -req  \
          -passin pass:test123 \
          -CA certs/ca.cert -CAkey private/ca.key -CAcreateserial \
          -passin pass:test123 \
          -in csr/ed25519.csr \
          -out certs/ed25519.cert \
          -days 9999

It is currently believed that all private keys created above provide similar attack resistance, which is comparable to 128-bit symmetric cipher. Nevertheless, it’s worth to notice that byte size of those keys are much different.

Client cert creation

Commands below will create client private key and certificate that can be used for mutual TLS (client authentication). Procedure is similar to creating server certificate, so I’ll do it only for ECDSA.

1. Client’s private key

openssl genpkey -algorithm EC \
          -pkeyopt ec_paramgen_curve:P-256 \
          -pkeyopt ec_param_enc:named_curve \
          -out private/cli_ecdsa_p256.key

2. Create certificate signing request - intermediary step

openssl req -config openssl.cnf -new \
          -sha256 \
          -passin pass:test123 \
          -key private/cli_ecdsa_p256.key  \
          -out csr/cli_ecdsa_p256.csr \
          -subj "/O=Cert Testing ORG/CN=Client Cert"

3. Create client cert

openssl x509 \
          -extfile openssl.cnf \
          -extensions client_cert \
          -req  \
          -CA certs/ca.cert \
          -CAkey private/ca.key \
          -CAcreateserial \
          -in csr/cli_ecdsa_p256.csr \
          -passin pass:test123 \
          -out certs/cli_ecdsa_p256.cert \
          -days 9999

Verification

In order to verify server certificate against CA following command can be used.

> openssl verify -CAfile certs/ca.cert certs/ecdsa_256.cert
certs/ecdsa_256.cert: OK

That’s it, I hope it helps, but most of all I hope I won’t have to look for this stuff ever again.

Also

Thank you to @mattcaswell from OpenSSL team, for helping to figure out how to create EdDSA certs.

Method 1: Using ndk-build

This method follows Android’ic way of doing things:

• Create required directories

  mkdir -p hello_world/jni
  mkdir -p hello_world/libs

In the jni directory create

• Android.mk

    LOCAL_PATH := $(call my-dir)
    include $(CLEAR_VARS}
    # give module name
    LOCAL_MODULE := hello_world
    # list your C files to compile
    LOCAL_SRC_FILES := main.c
    include $(BUILD_EXECUTABLE)

• Copy/create main.c to jni directory
• Go to jni directory, call ndk-build. Compilation result should be in hello_world/libs/armeabi/hello_world

Method 2: Makefile

With the second (and my prefered) way you have better control over files being compiled, compiler settings, etc. There is also no “magic” that ndk-build provides.

Following Makefile uses clang from NDK 16b in order to compile a file for Android with API version 27 and for ARMv8 CPU. The makefile can be used a template.

# Change this to whereever you keep NDK
NDK            = /opt/android-ndk
SRCDIR         = .
OBJDIR         = .
DBG           ?= 0

# Debug/Release configuration
ifeq ($(DBG),1)
MODE_FLAGS     = -DDEBUG -g -O0
else
MODE_FLAGS     = -Os -fdata-sections -ffunction-sections
endif

## NDK configuration (clang)

# NDK Version
NDK_TARGETVER  = 27

# Target arch - here aarch64 for android
NDK_TARGETARCH = aarch64-linux-android

# Target CPU (ARMv8)
NDK_TARGETSHORTARCH = arm64

# Toolchain version
NDK_TOOLVER  = 4.9

# Architecture of a machine that does cross compilation
NDK_HOSTARCH = linux-x86_64

# Set needed preprocessor symbols
NDK_TOOLS    = $(NDK)/toolchains/llvm/prebuilt/$(NDK_HOSTARCH)/bin
NDK_SYSROOT  = $(NDK)/sysroot
NDK_TOOL     = $(NDK_TOOLS)/clang
NDK_LIBS     = $(NDK)/toolchains/$(NDK_TARGETARCH)-$(NDK_TOOLVER)/prebuilt/linux-x86_64/lib/gcc/$(NDK_TARGETARCH)/4.9.x
NDK_INCLUDES = -I$(NDK)/sysroot/usr/include \
               -I$(NDK)/sysroot/usr/include/$(NDK_TARGETARCH)
NDK_SYSROOT  = $(NDK)/platforms/android-$(NDK_TARGETVER)/arch-$(NDK_TARGETSHORTARCH)

# Options common to compiler and linker
OPT          = $(MODE_FLAGS) \
               -std=c99 \
               -fPIE \
               -Wall \
               -target $(NDK_TARGETARCH)

# Compiler options
CFLAGS       = $(OPT) \
               $(NDK_INCLUDES)

# Linker options
LDFLAGS      = $(OPT) \
               $(MODE_FLAGS) \
               -pie \
               --sysroot=$(NDK_SYSROOT) \
               -B $(ANDROID_NDK)/toolchains/$(NDK_TARGETARCH)-$(NDK_TOOLVER)/prebuilt/linux-x86_64/$(NDK_TARGETARCH)/bin \
               -L$(NDK_LIBS)

all:
    $(NDK_TOOL) -c $(SRCDIR)/main.c -o $(OBJDIR)/main.o $(CFLAGS)
    $(NDK_TOOL) -o main $(OBJDIR)/main.o $(LDFLAGS)

adb-prepare:
    adb root
    adb remount

push: adb-prepare
    adb push main /data/app/

run: adb-prepare push
    adb shell /data/app/main

Copy this file to same directory as main.c and try

make all
make run

This should compile the file, push it to target and run (if target is connected).

hdc@cryptoden 23:49 > ~/example 
> make run   
adb root
adb remount
remount succeeded
adb push main /data/app/
main: 1 file pushed. 0.7 MB/s (6000 bytes in 0.008s)
adb shell /data/app/main
Hello World

Creating initrd

Best is to start on some actions that need to be done on raspberry. We need to install mkinitcpio and create initram file.

pacman -S mkinitcpio
cp /etc/mkinitcpio.conf ~/mkinitcpio.ripi.conf
vi ~/mkinitcpio.ripi.conf

Make sure that in the configuration file you have HOOKS and MODULES variables changed as below:

MODULES="dm_mod hid usbhid usbcore"
HOOKS="base udev autodetect modconf block filesystems keyboard encrypt fsck"

In MODULES most important is dm_mod and in HOOKS encrypt. Also order is very important in HOOKS. Once done generate new init-ram.

mkinitcpio -k `uname -r` -c ~/mkinitcpio.ripi.conf -g /boot/initrd-crypt

Creating encrypted volume

This must be done on PC. Insert SD card, mount root partition and copy it’s content to some temporary location. Don’t forget trailing / after temporary_location, it is important.

mount /dev/mmcblk0p2 /media
mkdir /temporary_location
rsync --progress -axv /media /temporary_location/

Next step is to create encrypted volume, format it and copy back root partition content:

cryptsetup luksFormat /dev/mmcblk0p2
cryptsetup luksOpen /dev/mmcblk0p2 root-raspberry
mkfs.ext4 /dev/mapper/root-raspberry
mount /dev/mapper/root-raspberry /mnt
rsync --progress -axv /temporary_location/ /mnt

Modification in /etc/fstab, /mnt/boot/config.txt and /mnt/boot/cmdline.txt file

Watch out here - many sources on internet says that you need to specify and address on which initram is loaded (something like initramfs initrd-crypt 0x0a000000, in config.txt). This doesn’t work with kernel 4.1. It’s enough to specify name of the init-ram file in config.txt and cmdline.txt

• /mnt/etc/fstab: Change device that mounts on /. File must have following entry (remove entry that starts with /dev/mmcblk0p2)

  /dev/mapper/root / ext4 defaults,discard,commit=120 0 1

• /mnt/boot/config.txt: Set initramfs. This file needs to have following line

  initramfs initrd-crypt

• /mnt/boot/cmdline.txt: Add following kernel command line arguments:

  cryptdevice=/dev/mmcblk0p2:root:allow-discards root=/dev/mapper/root rootwait rootfstype=ext4 initrd=initrd-crypt

Unmount and close crypto device:

sync
unmount /mnt
cryptsetup luksClose root-raspberry

Now you can put back SD card to raspberry and boot device. It should ask for password while booting.

Password on USB key

Raspberry can also read a password directly from file on USB key while booting. In order to do it, create a file with password:

dd if=/dev/urandom of=/mnt/sdb1/ripi.txt
cryptsetup luksAddKey /dev/mmcblk0p2 /mnt/sdb1/ripi.txt

And add following entry to cmdline.txt

cryptkey=/dev/disk/by-uuid/ABCD-EFGH:vfat:/ripi.txt

Where value for ABCD-EFGH you get by running blkid on partition of USB key that contains password:

blkid /dev/sdb1
/dev/sda: UUID="ABCD-EFGH" TYPE="vfat"

Interesting links

1. https://www.pavelkogan.com/2014/05/23/luks-full-disk-encryption/
2. https://outflux.net/blog/archives/2017/08/30/grub-and-luks/