Request a Demo
Join a 30 minute demo with a Cloudian expert.
NIXL (NVIDIA Inference Xfer Library) is an open-source, hardware- and storage-agnostic data movement library. It provides a unified API to transfer data quickly between GPU memory, CPU memory, and local or networked storage. It is built with a modular plug-in architecture to minimize overhead and support high-performance backends.
Rather than forcing an inference framework to implement separate transfer logic for every hardware or storage backend, NIXL provides a modular abstraction layer. It provides multiple transfer backends that support different transport and storage technologies, including RDMA-based networking, GPUDirect Storage, NVMe, and cloud or distributed storage.
You can download the NIXL library from the official GitHub repo.
Core NIXL use cases:
How NIXL Works
This is part of a series of articles about AI infrastructure
In this article:
Distributed LLM inference depends on moving tensors and KV caches efficiently across GPUs, servers, and storage tiers. NIXL provides a unified data-movement layer with high-bandwidth, low-latency, asynchronous transfers, helping inference frameworks reduce data-transfer overhead and use compute and memory resources more effectively.
NIXL uses transfer agents, registered memory or storage regions, exchanged metadata, and pluggable backends to move data between inference components. The application identifies the source and destination resources, while NIXL handles the transfer path through the appropriate backend.
This design allows the same inference framework to move data across GPU memory, CPU memory, storage, and remote systems without implementing separate transfer logic for every transport technology.
Before data can be transferred, NIXL needs to know which memory or storage regions are available to a transfer agent. Applications register resources such as GPU buffers, host memory, files, or storage-backed regions together with the information required to access them.
Registration gives the selected backend the metadata it needs to perform efficient transfers. For GPU and host memory, this can include addresses, sizes, device information, and registration handles used by technologies such as RDMA or GPU-aware transports.
By registering resources ahead of time, NIXL can avoid repeatedly discovering or preparing the same memory regions for every transfer.
When two transfer agents need to move data between them, they exchange metadata describing the registered source and destination resources.
This metadata can include information such as memory addresses, device identifiers, access keys, storage locations, or other backend-specific details required to reach the remote resource. The transfer agent receiving the metadata can then construct a transfer without needing direct knowledge of how the remote application’s memory is organized internally.
Separating metadata exchange from the data transfer itself also helps NIXL support distributed inference architectures in which workers may be running on different servers.
NIXL supports a plugin-based backend architecture. Different backends handle specific memory types, hardware configurations, and transport technologies, enabling NIXL to operate across diverse inference environments. These include GPU-to-GPU communication over high-speed networks, transfers between GPU memory and NVMe storage, and access to distributed file systems or object storage.
NIXL can automatically select an appropriate backend based on the source and destination memory types and the backends available to the transfer agents. This allows inference frameworks to use a consistent API without implementing separate transfer logic for each hardware configuration or storage technology.
Key NIXL transfer backends include:
| Backend | Supported Hardware / Technology | Primary Use Case |
| UCX | NVIDIA GPUs, CPU memory, InfiniBand, RoCE, TCP | High-speed GPU-to-GPU and CPU memory transfers across servers |
| Libfabric | OFI-compatible networking, including AWS EFA, InfiniBand verbs, TCP | Distributed inference across supported high-performance network fabrics |
| GPUDirect Storage (GDS) | NVIDIA GPUs, compatible NVMe storage and file systems | Direct GPU-to-storage transfers without CPU bounce buffers |
| GDS-MT | Same hardware as GDS, using multithreaded I/O | Parallel GPU-to-storage transfers with higher I/O concurrency |
| POSIX | CPU memory, Linux file systems, local and network storage | Standard file transfers without requiring GPUDirect Storage |
| Mooncake | GPU and CPU memory using Mooncake Transfer Engine | Distributed KV cache transfers in Mooncake-based environments |
| DOCA GPUNetIO | NVIDIA GPUs and compatible NVIDIA networking hardware | GPU-initiated networking using GPUDirect Async |
| Object (S3) | S3-compatible object storage, including S3/RDMA | KV cache offloading and retrieval from object storage |
| HF3FS | 3FS distributed file systems | High-performance file transfers in distributed storage environments |
| Azure Blob | Microsoft Azure Blob Storage | Object storage transfers in Azure-based deployments |
| INFINIA | DDN INFINIA storage, CPU and GPU memory | Asynchronous transfers to distributed object storage |
| GUSLI | Supported block storage devices | Block-level data transfers between storage and CPU memory |
This backend flexibility allows organizations to deploy NIXL across different infrastructure configurations without redesigning their inference applications. For example, GPU clusters connected through InfiniBand can use UCX for distributed KV cache transfers, while systems with compatible NVMe storage can use GDS for cache offloading. Cloud deployments can also use object storage backends to support external cache tiers.
Backend availability depends on installed plugins, compatible hardware, drivers, and supporting libraries. GDS-MT is also a compatibility name being phased out in favor of GDS configured with mode=mt.
Once the source, destination, and transfer path are defined, NIXL can initiate the transfer asynchronously. The application provides one or more source and destination regions, and the backend begins moving the requested data.
Asynchronous operation is particularly important for LLM inference because large KV cache transfers can otherwise stall GPU work. By separating transfer submission from completion, the inference runtime can overlap communication with other useful operations where the workload allows it.
This is useful for tasks such as transferring KV state from a prefill worker to a decode worker or prefetching offloaded cache blocks before they are required.
After initiating a transfer, the application does not need to synchronously wait for the entire operation to finish. Instead, NIXL exposes mechanisms for checking transfer status and determining when the requested data is ready.
The inference scheduler can continue processing other work while periodically checking completion or integrating transfer progress into its own event loop. Once the operation completes, the destination worker can safely use the transferred data.
This non-blocking model helps prevent communication from unnecessarily idling accelerators and is important for high-throughput inference systems that may have many KV cache transfers in flight simultaneously.
KV cache transfer is one of the most important NIXL use cases for LLM inference. Prefill workers can generate key and value tensors and transfer them to decode workers, allowing the decode phase to begin without recomputing the original prompt.
NIXL can also support KV cache movement between GPUs, CPU memory, and storage tiers. This makes it useful for disaggregated serving, prefix reuse, and offloading architectures where cached context may need to move independently of the GPU that originally generated it.
Because KV caches can become very large for long contexts and high concurrency, high-bandwidth asynchronous transfers are especially valuable for preventing cache movement from becoming a bottleneck.
NIXL can transfer memory and storage regions that contain model weights, so inference frameworks can use it as part of workflows that move model data between storage, CPU memory, and accelerators.
For example, weights stored in files or other storage systems may be transferred toward GPU memory using an appropriate NIXL backend. This can be useful in model loading, dynamic deployment, or environments where accelerators need to obtain model state from shared infrastructure.
NIXL itself does not interpret tensors as model weights. It transfers the underlying data buffers, while the inference framework determines how those buffers are used.
NIXL directly supports both GPU VRAM and CPU DRAM as memory types. Applications can register regions from either memory space and use supported backends to transfer data locally or between machines.
This enables GPU-to-GPU, GPU-to-CPU, CPU-to-GPU, and CPU-to-CPU transfer patterns depending on the available backend. Technologies such as UCX, Libfabric, and GPU-aware networking can be used underneath NIXL for high-performance memory transfers.
These generic buffers can contain KV tensors, activations, intermediate inference state, model parameters, or other data managed by the inference framework.
NIXL also treats files and block storage as transferable resource types. File-backed regions can represent data on local NVMe drives, remote filesystems, or parallel storage systems, while block storage provides access to lower-level storage devices.
Backends such as GPUDirect Storage can move data directly between GPU memory and supported filesystems, reducing the need to route every transfer through CPU memory. Other backends provide conventional CPU-to-file or block-level I/O.
This makes file and block storage useful for larger inference datasets, model state, and KV cache tiers that cannot economically remain in GPU or CPU memory.
NIXL includes support for object storage, including S3-compatible systems. Its object-storage backend can move data between local memory and objects identified by bucket and object key.
The backend supports standard S3 access as well as optimized approaches such as S3 CRT and S3-over-RDMA in supported environments. This makes object storage a potential capacity tier for inference data that must persist beyond individual GPU or server memory.
For KV cache workloads, object storage is generally more appropriate for cold or less latency-sensitive data than actively used cache blocks. Its main advantage is scalability: large cache or model datasets can be retained independently of individual inference nodes and retrieved when reuse is more efficient than recomputation.
In disaggregated inference, the prefill and decode phases can run on different workers. The prefill worker processes the prompt and generates the KV cache, while the decode worker uses that cache to generate output tokens.
NIXL can transfer KV cache blocks directly between these workers using high-speed GPU or network transports. This allows the decode worker to reuse the precomputed attention state instead of processing the entire prompt again.
The approach is particularly important for long prompts, where recomputing prefill can be expensive. Faster KV movement can reduce handoff latency between workers and make independent scaling of prefill and decode resources more practical.
Long-context requests can generate very large KV caches that consume significant GPU memory. NIXL allows inference systems to move less active KV data from GPU HBM into larger tiers such as CPU memory, local SSDs, distributed filesystems, or object storage.
When the cached context is needed again, the relevant data can be transferred back toward the GPU instead of recomputing it from the original prompt. This can expand effective cache capacity and preserve expensive prefill work.
This approach is especially useful when the total working set of reusable context is much larger than available GPU memory. The inference system can keep hot cache blocks near compute while using NIXL to manage transfers to and from lower-cost storage tiers.
NIXL can also move model weights between storage, CPU memory, and GPU workers. This can reduce the time required to bring new inference workers online or redistribute model state across a cluster.
For example, weights can be transferred from shared storage to newly provisioned GPU nodes during startup. NIXL can also support workflows where model weights need to be moved as part of resharding or changing the parallelization layout of an inference deployment.
Because the same transfer layer can work across memory and storage types, inference frameworks do not need separate data-movement implementations for GPU memory, files, or object storage.
LLM traffic can vary significantly over time, requiring inference clusters to add, remove, or reconfigure workers. NIXL is designed to support dynamic environments where transfer agents can join or leave while the overall inference system continues operating.
When new GPU workers are added, model state or KV data can be transferred to them using the same high-performance data plane. Existing cache state can also be reused across workers instead of being tied permanently to the node where it was originally created.
This supports more elastic inference architectures in which compute capacity can scale independently according to demand. It is particularly useful for distributed systems that dynamically rebalance prefill, decode, or other inference workloads across available accelerator resources.
Related content: Read our article about the main types of AI workloads
Hot KV cache data should remain in GPU memory whenever possible because attention operations can access it with the lowest latency. Active sequences, frequently reused prefixes, and cache blocks likely to be needed again soon are usually the highest-priority candidates for GPU residency.
Use NIXL primarily to move colder data to CPU memory, NVMe, or other storage tiers rather than constantly transferring active cache blocks back and forth. Excessive promotion and eviction can consume interconnect bandwidth and increase inference latency.
Cache-aware scheduling and eviction policies can help identify which blocks should remain near compute and which can be offloaded with minimal performance impact.
NIXL supports multiple transfer backends because different resource types require different data paths. Select the backend that best matches the source, destination, and performance characteristics of each transfer.
For example, high-speed memory-to-memory transfers may use RDMA- or UCX-based backends, while GPU-to-storage transfers may benefit from GPUDirect Storage. File, block, and object storage can require different plugins depending on the underlying infrastructure.
Avoid treating all tiers as interchangeable. Benchmark the actual backend and hardware combination because latency, throughput, registration overhead, and supported transfer paths can differ substantially between environments.
Cross-node KV cache movement can involve very large amounts of data, particularly for long-context requests or disaggregated prefill and decode architectures. Slow networking can cause the destination GPU to wait for cache data and eliminate much of the benefit of separating inference stages.
Use high-bandwidth, low-latency networking where large GPU-to-GPU or host-to-host transfers are expected. RDMA-capable fabrics and GPU-aware networking can reduce CPU involvement and improve effective transfer performance.
Network topology also matters. When possible, schedule related prefill and decode workers so that frequently exchanged data travels over the shortest and highest-bandwidth path available.
When KV cache working sets exceed practical GPU and CPU memory capacity, storage can provide a much larger secondary tier. However, retrieving large cache blocks from storage is only beneficial when the storage subsystem can move them fast enough to outperform recomputation.
Use storage infrastructure with sufficient throughput, IOPS, and parallelism for the expected number of simultaneous cache transfers. Local NVMe can provide a fast intermediate tier, while distributed or object storage can support much larger shared cache pools.
Consider the full path from storage to the GPU, including filesystem, network, CPU memory, PCIe, and accelerator interconnects. A bottleneck at any point can limit the practical value of a high-performance storage backend.
Related content: Read our guide to avoiding storage bottlenecks in AI environments
Inference compute and KV cache capacity do not always need to scale together. Adding GPUs solely to gain more cache memory can be expensive, especially when the workload is constrained by context storage rather than arithmetic throughput.
Use NIXL to connect inference workers to independently scalable CPU memory, NVMe, distributed storage, or object-storage tiers. This allows GPU capacity to scale according to compute demand while cache infrastructure scales according to context size, reuse, and retention requirements.
Separating these resources can also improve elasticity. Prefill and decode workers can be added or removed as traffic changes while reusable KV data remains available through shared cache infrastructure rather than being tied permanently to individual GPUs.
Related content: Learn more in our detailed guide to AI storage
Cloudian HyperStore with RDMA for S3-compatible storage is a scalable, cost-effective, high-performance object storage platform built for the demands of AI and ML workloads. It supports direct RDMA data transfers between processing nodes and storage nodes. This keeps GPUs fully utilized while reducing CPU overhead. HyperStore acts as a unified data lake for training and inference, so you don’t need a separate high-performance file storage layer or costly data migrations.
Key capabilities of Cloudian HyperStore with RDMA for S3:
Learn more about Cloudian HyperStore with RDMA for S3-compatible storage
See Cloudian first in Google Search