drivers: crypto: qcom: CRYPTO0 CE cipher and AEAD support

[Amirreza Zarrabi (qc-azarrabi)]

This series adds AES-ECB, AES-CBC, and AES-GCM hardware acceleration for the Qualcomm CRYPTO0 crypto engine on Lemans SoCs, and prepares the existing HWKM driver to support the CRYPTO0 key manager slave.

Hardware overview

The CRYPTO0 crypto engine (CE) is a general-purpose AES accelerator present on Qualcomm SoCs. It exposes two data path interfaces:

• Register mode: the host writes plaintext and reads ciphertext through 32-bit MMIO DATA_IN/DATA_OUT FIFO registers, polling STATUS for completion. This is what the driver uses.
• BAM mode: a Bus Access Manager (DMA engine) feeds descriptors to the CE, allowing zero-copy transfers. BAM support is not implemented in this series. The hardware is capable of it, but BAM requires a separate descriptor protocol and interrupt infrastructure that is out of scope here. The register-mode path is sufficient for the OP-TEE use cases (tadb, TA cryptographic operations).

The CE supports AES-128 and AES-256. AES-192 is not implemented in hardware; the GCM driver falls back transparently to the software implementation for 24-byte keys.

For GCM, 96-bit nonces follow the standard J0 = nonce || 0x00000001 construction. For non-96-bit nonces (up to 128 bits), J0 is derived by feeding the nonce through the CE GHASH auth engine directly, avoiding any software GF(2^128) multiply.

The FIFO loop is byte-stream oriented: each 32-bit FIFO word is scattered/gathered byte-by-byte so payloads of any length are handled without alignment constraints on the caller. All polling loops are bounded by a 1-second timeout to avoid hanging the core on a wedged engine.

Series structure

Patches 1-2 prepare the existing HWKM driver. The register map is made group-relative and reusable across instances, and the CRYPTO0 KM slave is enabled so that keys can be provisioned into CRYPTO0 hardware key slots.

Patches 3-4 add the shared CE helper layer (register accessors, FIFO polling, config builders) and register the AES-ECB and AES-CBC drvcrypt cipher providers.

Patches 5-6 add the AES-GCM drvcrypt authenc provider and enable it on Lemans.

Testing

xtest 4003 (AES cipher operations) and xtest 4005 (AES-GCM) pass on Lemans hardware. The GCM provider is also exercised by the REE filesystem (tadb) which uses AES-GCM as its default authenticated encryption algorithm for TA storage.

[Harshal Dev (harshaldev27)]

Hi Amir, Post GPCE init, should we also derive and transfer the L3 keys to the GPCE slave after generating and wrapping with a TP_KEY?

[Amirreza Zarrabi (qc-azarrabi)]

Yes, I have them for separate PR. There is no use for them in cipher operation; keeping this PR to the minimum.

[Harshal Dev (harshaldev27)]

I just noticed this, we are setting CMD_DONE field to 1 and the entire IRQ_STATUS register to 0 here.. I think we should use io_write32_off_field? I see QTEE doing the same.

[Amirreza Zarrabi (qc-azarrabi)]

Not according to IPCAT - IRQ_STATUS is W1TC field. If QTEE does that it has a bug and should be fixed.

[Harshal Dev (harshaldev27)]

Hi Amir, each field of the IRQ_STATUS register is a W1TC field, including CMD_DONE. But I notice that we are setting every other field to 0 except CMD_DONE here, whereas in QTEE code we are explicitly setting only CMD_DONE to 1 to clear that bit and not touching the others. The programming guide for HWKM says similar:

7.	Clear The CMD_DONE Status bit (W1TC)
a.	Write KM_BANKN_IRQ_STATUS. CMD_DONE == 1

But I think both approaches might be fine as long as writing 0 to the other fields doesn't affect them at all. 😄

[Selvam Sathappan (zelvam95)]

AI agent's feedback to make AES-192 keep working:

🔴 core/drivers/crypto/qcom/cipher/cipher.c:62-76, cipher/ecb.c:40-41, cipher/cbc.c:40-41 — AES-192 ECB/CBC now hard-fails; no software fallback (functional regression)

/* cipher.c: do_allocate() succeeds for the algorithm, key size unknown here */
case TEE_ALG_AES_ECB_NOPAD:
	return ce_aes_ecb_allocate(ctx);   /* always TEE_SUCCESS */
...
/* ecb.c / cbc.c: key size is only seen at init(), and 24 is rejected */
if (key1_len != 16 && key1_len != 32)
	return TEE_ERROR_BAD_PARAMETERS;

The drvcrypt software fallback in crypto_cipher_alloc_ctx()
(core/crypto/crypto.c:114-175) only falls back to the built-in AES
implementation when drvcrypt_cipher_alloc_ctx() returns
TEE_ERROR_NOT_IMPLEMENTED, and that decision is made at alloc time,
keyed on the algorithm — not the key size. For TEE_ALG_AES_ECB_NOPAD /
TEE_ALG_AES_CBC_NOPAD, do_allocate() unconditionally returns
TEE_SUCCESS (the key length is not yet known), so the CE context is
installed and the software path becomes unreachable. The 24-byte key is not
seen until ecb_init() / cbc_init(), which reject it with
TEE_ERROR_BAD_PARAMETERS.

Net effect: once this driver is enabled, AES-192-ECB and AES-192-CBC stop
working — a TA (or TEE_AllocateOperation with a 192-bit key) that worked
on the pure-software build now fails at init. AES-192 is a mandated GP
key size, so this is a behavioural regression, not merely an unsupported
optimisation.

Note this is exactly the case the AEAD side handles correctly: gcm_init()
(authenc/gcm.c:712) detects key_len == 24 and transparently routes to a
software GCM context (gcm_init_sw() → crypto_aes_gcm_alloc_ctx()). The
cipher providers should mirror that: either fall back to a software
AES-ECB/CBC context for 192-bit keys, or (simpler) keep a sw_ctx in
ce_ecb_ctx / ce_cbc_ctx as GCM does. As written the two providers are
inconsistent with the GCM sibling in the same PR.

Suggest: handle key_len == 24 in ecb_init/cbc_init by delegating to
crypto_aes_ecb_alloc_ctx() / crypto_aes_cbc_alloc_ctx() the way
gcm_init_sw() does, so previously-working AES-192 keeps working.

[Selvam Sathappan (zelvam95)]

🟡 core/drivers/crypto/qcom/hwkm.c:246-247 vs 293-311 — initialized = true is set before crypto0_base is populated; a gpce_init() failure leaves a half-valid context that can panic

/* hwkm_init(): */
hwkm_ctx.base = base;
hwkm_ctx.initialized = true;        /* line 247 */
...
/* gpce_init() runs AFTER and is the one that sets crypto0_base: */
hwkm_ctx.crypto0_base = base;       /* line 290 */

hwkm_get_context() gates purely on initialized
(hwkm.c:53-56), and hwkm_init() sets initialized = true at line 247 —
before gpce_init() runs. If gpce_init() then fails (its
phys_to_virt() returns NULL, or its BIST check trips → returns
TEE_ERROR_GENERIC at lines 259-269), hwkm_driver_init() returns the
error, but hwkm_ctx is left with initialized == true and
crypto0_base == 0.

Downstream, ce_get_base() (ce/ce.c:35-41) does:

struct hwkm_drv_ctx *ctx = hwkm_get_context();
assert(ctx && ctx->crypto0_base);       /* ctx != NULL, crypto0_base == 0 */
return ctx->crypto0_base + CE_REG_OFFSET;

hwkm_get_context() now returns non-NULL (because initialized is true),
so the guard passes the NULL check but crypto0_base is 0 — the assert
fires (panic on a debug build), and on a release build with asserts
compiled out it computes 0 + CE_REG_OFFSET and dereferences an unmapped
address. The cipher/authenc driver_init_late() registration callbacks
(cipher.c:87, authenc.c:119) also call ce_get_base() directly, so
this is reachable during boot, not only from a TA.

Suggest: set hwkm_ctx.initialized = true only at the very end of
hwkm_driver_init(), after both hwkm_init() and gpce_init() have
succeeded — or have hwkm_get_context() also require crypto0_base before
returning non-NULL. As-is the "initialized" flag doesn't actually imply the
CE base is usable.

[Selvam Sathappan (zelvam95)]

🟡 core/drivers/crypto/qcom/ce/ce.c:333-347 / 473-478 — FIFO port index i is reset to 0 per polling iteration but DIN_SIZE_AVAIL reflects the whole FIFO; confirm multi-word bursts land on the right ports

In both ce_aes_xfer() and aead_transfer(), each poll iteration reads the
available word count and then writes/reads that many words in an inner loop,
using a local i that starts at 0 every outer iteration:

unsigned int i = 0;
avail = ce_din_avail(s2);
while (avail && din_done < din_words) {
	...
	ce_fifo_write(base, i, w);   /* CE_DATA_IN(i % CE_FIFO_PORTS) */
	i++;
	...
}

ce_fifo_write()/ce_fifo_read() select the port as i % CE_FIFO_PORTS
(4 ports, ce_regs.h:24-32). Because i restarts at 0 on every outer poll
iteration, the port sequence depends on how the available-word count happens
to split across iterations. If the hardware requires that consecutive words
of a transfer go to strictly increasing/rotating ports across the whole
transfer (not per burst), a transfer that pauses mid-way (FIFO briefly full)
could resume writing at port 0 rather than the next expected port. If the
port index is purely a convenience alias (any DATA_IN word appends to the
same FIFO regardless of n), then this is fine. This should be confirmed
against the CE FIFO spec — it doesn't manifest in a single-burst transfer
(which is the common test case) but would corrupt data on any transfer that
takes more than one poll iteration. Worth a comment either way, since the
i % CE_FIFO_PORTS aliasing intent is not obvious.

[Selvam Sathappan (zelvam95)]

Hi Amirreza Zarrabi (@qc-azarrabi),

I tried the AI Agentic framework to review this PR & it gave some comments. Can you check if they're valid/make sense/are appropriate? (would help finetune the agentic framework 🗡️)

On that note -> The ce.c -> If its crypto engine that we're referring here -> Is this a particular version of GPCE/crypto engine thats present in these targets? Should we call it ce5.c or something more specific thats actually supported on these targets?

[Copilot]

🟡 Not ready to approve

There are correctness issues in newly added CE/HWKM code paths (key-slot access control coverage and CE helper length-safety) plus an internal register-field documentation inconsistency that should be resolved before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Adds Qualcomm CRYPTO0 Crypto Engine (CE) acceleration and extends the existing Qualcomm HWKM driver so keys can be provisioned for use by CRYPTO0 on Lemans (Hoya) SoCs.

Changes:

• Extend HWKM to support a CRYPTO0 “slave” instance and route transactions to the appropriate BANK0 FIFO.
• Add shared CRYPTO0 CE helper layer plus drvcrypt cipher providers (AES-ECB/AES-CBC).
• Add drvcrypt authenc provider for AES-GCM (with AES-192 software fallback) and enable CE algorithms on Lemans.

File summaries

File	Description
core/drivers/crypto/qcom/sub.mk	Build integration for CE, cipher, and authenc submodules under CFG_QCOM_CE* flags.
core/drivers/crypto/qcom/include/hwkm.h	Add CRYPTO0 destination and extend HWKM context with CRYPTO0 MMIO base.
core/drivers/crypto/qcom/include/hwkm_regs.h	Refactor HWKM register offsets to be group-relative and add master/CRYPTO0 base offsets.
core/drivers/crypto/qcom/hwkm.c	Map CRYPTO0 MMIO, init CRYPTO0 HWKM instance, and configure access control for CRYPTO0 key slots.
core/drivers/crypto/qcom/hwkm_transaction.c	Generalize FIFO transaction submission and add CRYPTO0 transaction path.
core/drivers/crypto/qcom/crypto.mk	Define CE feature flags and force-enable appropriate drvcrypt layers based on enabled algorithms.
core/drivers/crypto/qcom/cipher/sub.mk	Build rules for CE cipher provider and per-mode sources.
core/drivers/crypto/qcom/cipher/cipher.c	drvcrypt cipher dispatcher registering ECB/CBC implementations.
core/drivers/crypto/qcom/cipher/ecb.c	AES-ECB context implementation using CE helpers.
core/drivers/crypto/qcom/cipher/cbc.c	AES-CBC context implementation using CE helpers (IV readback between updates).
core/drivers/crypto/qcom/cipher/algorithms.h	Algorithm allocation prototypes for enabled cipher modes.
core/drivers/crypto/qcom/ce/sub.mk	Build rules for shared CE helper layer.
core/drivers/crypto/qcom/ce/include/ce.h	Public CE helper API for cipher and AEAD operations.
core/drivers/crypto/qcom/ce/include/ce_regs.h	CRYPTO0 CE register/bitfield definitions used by the helper layer.
core/drivers/crypto/qcom/ce/ce.c	CE helper implementation (register programming, FIFO polling, cipher + AEAD transfers).
core/drivers/crypto/qcom/authenc/sub.mk	Build rules for CE authenc provider and AES-GCM implementation.
core/drivers/crypto/qcom/authenc/authenc.c	drvcrypt authenc dispatcher registering AES-GCM implementation.
core/drivers/crypto/qcom/authenc/gcm.c	AES-GCM context using CE (with software fallback for AES-192).
core/drivers/crypto/qcom/authenc/algorithms.h	Algorithm allocation prototypes for enabled authenc modes.
core/arch/arm/plat-qcom/hoya/lemans/target.mk	Enable CE AES-ECB/CBC/GCM on Lemans by default.
core/arch/arm/plat-qcom/hoya/arch_config.h	Define CRYPTO0 MMIO base/size for HWKM/CE mapping.

Review details

Suppressed comments (1)

core/drivers/crypto/qcom/ce/ce.c:359

• ce_aes_xfer() copies a full 32-bit word to @out for every rounded-up dout_word. If len is not a multiple of 4 this writes past the end of @out on the final iteration. Copy only the remaining bytes for the last word.

				w = ce_fifo_read(base, i);
				memcpy(out + dout_done * sizeof(uint32_t),
				       &w, sizeof(uint32_t));
				dout_done++;

• Files reviewed: 21/21 changed files
• Comments generated: 3
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Jorge A. Ramirez-Ortiz (ldts)]

Amirreza Zarrabi (@qc-azarrabi) this is probably the last hurdle to get this merged. do you mind having a look?

[Amirreza Zarrabi (qc-azarrabi)]

Amirreza Zarrabi (Amirreza Zarrabi (@qc-azarrabi)) this is probably the last hurdle to get this merged. do you mind having a look?

Sorry for the late reply -- was closing some other task, I am on it :).

[Amirreza Zarrabi (qc-azarrabi)]

Sent to upstream.