NVMe over TCP

NVMe is a high-performance interface and protocol originally designed for direct-attached PCIe SSDs. While NVMe greatly improved local storage throughput and latency compared to older protocols like AHCI, it was initially limited by PCIe's short reach (within a server or a rack). NVMe-oF was introduced to extend NVMe across network fabrics, enabling remote access to NVMe devices with minimal added latency. The NVMe-oF 1.0 specification was released in mid-2016 by NVM Express, supporting transports such as RDMA (e.g. RoCE, InfiniBand) and later Fibre Channel. NVMe/TCP encapsulates NVMe commands and data inside TCP packets over Ethernet, avoiding the need for proprietary adapters or lossless networks. By 2018–2019 the NVMe/TCP specification was finalized under the NVMe-oF 1.1 standard, bringing NVMe-oF to any IP-based data center network. EE Times described NVMe/TCP as a means to address the "data tsunami" from AI and IoT by allowing tens of thousands of NVMe drives to appear local to hosts over Ethernet. The ratified NVMe/TCP 1.0 specification was published in 2021. ==Principles of operation==

Architecture NVMe over TCP operates within the NVMe-oF framework using a client–server architecture consisting of NVMe initiators (hosts) and NVMe targets (subsystems). Targets expose NVMe namespaces (storage volumes) over the network, allowing hosts to access remote NVMe controllers similarly to local storage devices. Small data transfers may be included directly within the capsules, while larger transfers use separate TCP segments. For write commands, the host transmits data segments following the command capsule; for read commands, the target sends data segments back to the host. NVMe/TCP relies on TCP's reliable, in-order delivery rather than remote DMA, eliminating the need for specialized hardware but slightly increasing CPU overhead and latency compared to RDMA transports. Although NVMe/TCP generally introduces additional latency, typically tens of microseconds compared to RDMA, performance optimizations such as kernel bypass and hardware offloads can reduce this impact. NVMe/TCP supports high throughput and scalability in standard IP networks without requiring lossless fabrics. ==Applications==

NVMe over TCP is frequently used in cloud and hyperscale data centers, supporting resource disaggregation and software-defined storage architectures. NVMe/TCP enables pools of NVMe SSDs to be attached and shared over standard Ethernet networks, allowing storage capacity to scale independently of computing resources. In enterprise environments, NVMe/TCP is adopted to modernize SANs. The protocol is sometimes compared to iSCSI due to its use of IP networks, though specifically adapted for NVMe-based storage performance. ==Implementations==

Multiple implementations of NVMe over TCP exist in both hardware and software forms. In the open-source domain, NVMe/TCP support was added to the Linux kernel in version 5.0, released in early 2019, which includes initiator (client) and target (server) drivers. As a result, modern Linux distributions can function as NVMe/TCP hosts or targets using built-in kernel modules. The introduction of NVMe/TCP in Linux simplified configurations compared to previous RDMA-based NVMe-oF implementations. ==Standardisation==

The NVMe over TCP protocol is defined and maintained by NVM Express, which oversees the NVMe family of specifications. NVMe/TCP was developed as an extension of the NVMe over Fabrics (NVMe-oF) specification. The initial NVMe-oF 1.0 specification, ratified in 2016, initially supported RDMA transports, and provided a foundation for later Fibre Channel integration. ==References==