NVMe Queues Explained: Submission Queues, Completion Queues, and Queue Depth

NVMe uses queues in host memory to exchange work with a storage controller. A Submission Queue (SQ) is a circular buffer where host software places commands. After adding commands, the host updates that queue’s tail doorbell so the controller knows new entries are available. The controller fetches those commands and may execute them in an order that differs from the order in which they were fetched.

A Completion Queue (CQ) is another circular buffer in host memory. The controller posts completion entries there after commands finish. Each completion identifies the originating Submission Queue and the command identifier assigned by host software, allowing the host to match a completion to the request that produced it. This queue architecture is part of the NVMe protocol; it should not be confused with the flash channels, dies or internal scheduling structures inside an SSD controller.

For a Submission Queue, host software is the producer and the controller is the consumer. The host writes one or more 64-byte command entries into available slots and advances the SQ tail doorbell. As the controller consumes entries, completion information includes the Submission Queue head value that tells host software which SQ entries have become reusable.

For a Completion Queue, the roles reverse: the controller produces completion entries and host software consumes them. An NVMe completion entry is 16 bytes and includes the Submission Queue identifier, command identifier and completion status. Multiple Submission Queues can be associated with one Completion Queue, so an SQ and CQ are not necessarily a permanent one-to-one pair.

Every NVMe controller uses an Admin Submission Queue and Admin Completion Queue for controller-management work. Windows documents these as queue identifier 0. Administrative commands include operations such as creating and deleting I/O queues, identifying the controller or namespace, and getting log pages.

Normal storage reads and writes use I/O queues rather than turning the Admin Queue into the data path. The host negotiates how many I/O Submission and Completion Queues the controller will provide, then creates the queues through Admin commands. The NVMe architecture permits many I/O queues so software can distribute work without forcing every CPU context through one shared command queue.

Queue count describes how many queues exist. Queue depth describes outstanding command concurrency within the workload or queue being discussed. A system can therefore have several I/O queues while a particular workload keeps only one command outstanding in one of them, or it can drive a queue with many outstanding requests.

The NVMe specification allows very large theoretical queue structures: I/O queues can be created with up to 64 Ki entries subject to the controller’s reported limit, and the architecture can support tens of thousands of I/O queues. Those protocol ceilings are capability limits, not evidence that a desktop application normally fills thousands of queues or maintains an extreme queue depth.

When a storage benchmark reports QD1, QD4, QD32 or another queue depth, it is describing how much request concurrency the test is presenting. Increasing queue depth gives the storage stack more outstanding work that may be scheduled in parallel. Whether that improves throughput depends on the drive, request size, read/write mix, access pattern, thread model, cache state, NAND state, firmware and other workload conditions.

That is why a high-QD benchmark result should not be read as a universal recommendation to run applications at high queue depth. A latency-sensitive workload with little concurrency asks a different question from a synthetic throughput test designed to keep a controller busy. Compare results at workload settings that resemble the behavior you actually care about.

Latency is the time a request takes through the relevant measurement boundary. IOPS counts completed I/O operations per unit of time. Throughput measures data transferred per unit of time. Queue depth describes outstanding concurrency. These quantities influence one another under load, but none is interchangeable with another.

For example, a deeper queue can allow more operations to overlap and raise IOPS while individual requests spend longer waiting or being serviced. Large sequential requests can produce high transfer throughput with a very different IOPS profile from small random requests. A benchmark needs its request size, access pattern, read/write mix, queue depth and thread settings before its headline number has useful context.

NVMe exposes a scalable command interface, but an SSD still has a finite controller, NAND layout, cache policy, firmware scheduler, thermal envelope and PCIe transport. Having room for many outstanding commands does not prove that the flash can execute all of them simultaneously. Internal NAND channels and dies are implementation details behind the NVMe command interface.

The PCIe link is another separate boundary. A faster or wider link can raise the transport ceiling, especially for large transfers, but queue depth does not create additional PCIe lanes or link rate. Conversely, a workload can be latency-limited or flash-limited without saturating the link. Treat protocol concurrency, internal media parallelism and transport bandwidth as three different layers.

One of NVMe’s architectural advantages is that host software can establish multiple I/O queues and map work across processor contexts instead of funneling all I/O through a single shared queue. NVM Express describes this as a scalable queuing mechanism, and Microsoft’s NVMe queue-utilization testing includes scenarios where queues are mapped across processors.

That scalability matters most when software has enough independent work to use it. It does not mean that creating more queues makes one isolated 4 KiB read complete proportionally faster. Single-request latency still depends on the full path through software, controller, media and any caching involved.

Start by identifying request size, sequential versus random access, read/write mix, queue depth, thread count and test duration. Then ask what layer is likely being stressed: low-concurrency latency, controller/media parallelism, sustained NAND behavior, cache behavior or PCIe transfer bandwidth. Results from different settings answer different questions even when they use the same SSD.

Queue depth is therefore best treated as part of the workload definition. NVMe submission and completion queues make concurrent command handling efficient and scalable, but the queue number on a benchmark does not by itself describe application responsiveness, flash architecture or the speed of the PCIe link. Those claims require their own measurements and evidence.