What Confidential Inference Actually Protects
Standard enterprise security controls encrypt model weights and datasets when they are stored on disk, and encrypt traffic as prompts and responses move across networks. Neither control protects that same data while it sits in memory and is being processed by a CPU or GPU. During inference, model weights are loaded into memory, prompts are decrypted for processing, and intermediate activations are computed in plaintext. On standard infrastructure, a cloud provider, hypervisor operator, or co-tenant with sufficient privilege could theoretically access this data during that window.
Confidential inference addresses this gap using trusted execution environments: hardware-enforced isolated regions where code and data are encrypted in memory and shielded from the host operating system, hypervisor, and other tenants. The Confidential Computing Consortium defines this as protecting data in use, distinct from the at-rest and in-transit protections most enterprises already have in place. For LLM workloads specifically, this means model weights, the prompts submitted to the model, and the outputs generated can remain encrypted from the perspective of the underlying infrastructure operator throughout the inference request.
The Hardware Layer: CPU and GPU TEEs
Confidential inference for LLMs depends on two categories of hardware working together. CPU-based TEEs, such as AMD SEV-SNP and Intel TDX, create isolated virtual machines with encrypted memory. AMD SEV-SNP adds memory integrity protection beyond earlier SEV versions, addressing certain remapping and replay attacks. Intel TDX creates hardware-isolated trust domains with memory encryption and integrity checks managed by a CPU-resident trusted module, keeping VM state isolated from the hypervisor.
Because LLM inference is GPU-accelerated, CPU TEEs alone are insufficient. NVIDIA's H100 GPUs support a confidential computing mode that encrypts GPU memory and isolates compute from the host OS and hypervisor, and this mode is designed to extend the CPU TEE's trust boundary onto the GPU. In practice, this means an architecture needs a confidential CPU VM and a confidential GPU instance operating together, with a coordinated trust boundary spanning both, to prevent model weights or intermediate activations from being exposed to the host system at any point in the pipeline.
Cloud Platform Options
Major cloud providers now offer confidential computing options that support this architecture. Microsoft Azure provides confidential VMs built on AMD SEV-SNP and Intel TDX, and pairs these with confidential GPU VM options that combine NVIDIA H100 confidential computing with confidential CPU instances for end-to-end protection of AI workloads. Google Cloud offers Confidential VMs using AMD SEV and Intel TDX technologies, extending encryption specifically to memory in use.
Availability of confidential GPU instances varies by provider and region, and enterprise architects should verify current support for their specific GPU generation and target regions directly with the provider before committing to an architecture, since this changes as hardware and platform offerings evolve.
Tradeoffs Architects Need to Evaluate
Confidential inference introduces real architectural constraints alongside its protections. Multi-GPU and multi-node inference, common for larger models, adds complexity because confidential computing guarantees must extend across interconnected GPUs and distributed memory, and current confidential GPU offerings are more mature for single-GPU or single-node deployments. Multi-tenant GPU sharing models can also conflict with the isolation guarantees confidential computing requires, which is why some cloud offerings require dedicated confidential GPU instances rather than shared ones.
Existing inference serving stacks and MLOps pipelines may require modification, vendor-specific SDKs, or updated orchestration support to run inside confidential VM or enclave boundaries without breaking existing workflows. Performance overhead from memory encryption and attestation checks is workload and hardware dependent; enterprises should benchmark their own model sizes and batch configurations rather than assume a fixed cost, since no consistent overhead figure is broadly established across generations and providers.
What TEEs Do Not Cover
A trusted execution environment protects the confidentiality and integrity of computation inside its boundary. It does not, on its own, determine who is authorized to submit prompts, what data an agent or user should be permitted to send to the model, or how outputs should be logged and audited for compliance purposes. Attestation policy management, deciding which TEE configurations and firmware versions are acceptable and rejecting outdated ones, is an operational governance function that sits alongside the hardware guarantee, not inside it.
Key management is a similar dependency: keys must be released to the enclave only after successful attestation, and organizations need a process for handling attestation failures during autoscaling, migration, or failover without silently falling back to an unprotected path. These are runtime governance and access control functions that TEEs assume are handled elsewhere. Enterprises evaluating confidential inference should treat hardware-based confidentiality and runtime policy enforcement, including agent identity, permissions, and audit logging over inference requests, as complementary layers rather than substitutes for one another.