PCI Express 6.0 Specification Finalized: x16 Slots to Reach 128GBps
by Ryan Smith on January 11, 2022 12:00 PM EST
This morning the PCI Special Interest Group (PCI-SIG) is releasing the much-awaited final (1.0) specification for PCI Express 6.0. The next generation of the ubiquitous bus is once again doubling the data rate of a PCIe lane, bringing it to 8GB/second in each direction – and far, far higher for multi-lane configurations. With the final version of the specification now sorted and approved, the group expects the first commercial hardware to hit the market in 12-18 months, which in practice means it should start showing up in servers in 2023.
First announced in the summer of 2019, PCI Express 6.0 is, as the name implies, the immediate follow-up to the current-generation PCIe 5.0 specification. Having made it their goal to keep doubling PCIe bandwidth roughly every 3 years, the PCI-SIG almost immediately set about work on PCIe 6.0 once the 5.0 specification was completed, looking at ways to once again double the bandwidth of PCIe. The product of those development efforts is the new PCIe 6.0 spec, and while the group has missed their original goal of a late 2021 release by mere weeks, today they are announcing that the specification has been finalized and is being released to the group’s members.
As always, the creation of an even faster version of PCIe technology has been driven by the insatiable bandwidth needs of the industry. The amount of data being moved by graphics cards, accelerators, network cards, SSDs, and other PCIe devices only continues to increase, and thus so must bus speeds to keep these devices fed. As with past versions of the standard, the immediate demand for the faster specification comes from server operators, whom are already regularly using large amounts of high-speed hardware. But in due time the technology should filter down to consumer devices (i.e. PCs) as well.
By doubling the speed of a PCIe link, PCIe 6.0 is an across-the-board doubling of bandwidth rates. X1 links move from 4GB/second/direction to 8GB/second/direction, and that scales all the way up to 128GB/second/direction for a full x16 link. For devices that are already suturing a link of a given width, the extra bandwidth represents a significant increase in bus limits; meanwhile for devices that aren’t yet saturating a link, PCIe 6.0 offers an opportunity to reduce the width of a link, maintaining the same bandwidth while bringing down hardware costs.
| PCI Express Bandwidth (Full Duplex: GB/second/direction) |
||||||||
| Slot Width | PCIe 1.0 (2003) |
PCIe 2.0 (2007) |
PCIe 3.0 (2010) |
PCIe 4.0 (2017) |
PCIe 5.0 (2019) |
PCIe 6.0 (2022) |
||
| x1 | 0.25GB/sec | 0.5GB/sec | ~1GB/sec | ~2GB/sec | ~4GB/sec | 8GB/sec | ||
| x2 | 0.5GB/sec | 1GB/sec | ~2GB/sec | ~4GB/sec | ~8GB/sec | 16GB/sec | ||
| x4 | 1GB/sec | 2GB/sec | ~4GB/sec | ~8GB/sec | ~16GB/sec | 32GB/sec | ||
| x8 | 2GB/sec | 4GB/sec | ~8GB/sec | ~16GB/sec | ~32GB/sec | 64GB/sec | ||
| x16 | 4GB/sec | 8GB/sec | ~16GB/sec | ~32GB/sec | ~64GB/sec | 128GB/sec | ||
PCI Express was first launched in 2003, and today’s 6.0 release essentially marks the third major revision of the technology. Whereas PCIe 4.0 and 5.0 were “merely” extensions to earlier signaling methods – specifically, continuing to use PCIe 3.0’s 128b/130b signaling with NRZ – PCIe 6.0 undertakes a more significant overhaul, arguably the largest in the history of the standard.
In order to pull of another bandwidth doubling, the PCI-SIG has upended the signaling technology entirely, moving from the Non-Return-to-Zero (NRZ) tech used since the beginning, and to Pulse-Amplitude Modulation 4 (PAM4).
As we wrote at the time that development on PCIe 6.0 was first announced:
PAM4 itself is not a new technology, but up until now it’s been the domain of ultra-high-end networking standards like 200G Ethernet, where the amount of space available for more physical channels is even more limited. As a result, the industry already has a few years of experience working with the signaling standard, and with their own bandwidth needs continuing to grow, the PCI-SIG has decided to bring it inside the chassis by basing the next generation of PCIe upon it.
The tradeoff for using PAM4 is of course cost. Even with its greater bandwidth per Hz, PAM4 currently costs more to implement at pretty much every level, from the PHY to the physical layer. Which is why it hasn’t taken the world by storm, and why NRZ continues to be used elsewhere. The sheer mass deployment scale of PCIe will of course help a lot here – economies of scale still count for a lot – but it will be interesting to see where things stand in a few years once PCIe 6.0 is in the middle of ramping up.
Meanwhile, not unlike the MLC NAND in my earlier analogy, because of the additional signal states a PAM4 signal itself is more fragile than a NRZ signal. And this means that along with PAM4, for the first time in PCIe’s history the standard is also getting Forward Error Correction (FEC). Living up to its name, Forward Error Correction is a means of correcting signal errors in a link by supplying a constant stream of error correction data, and it’s already commonly used in situations where data integrity is critical and there’s no time for a retransmission (such as DisplayPort 1.4 w/DSC). While FEC hasn’t been necessary for PCIe until now, PAM4’s fragility is going to change that. The inclusion of FEC shouldn’t make a noticeable difference to end-users, but for the PCI-SIG it’s another design requirement to contend with. In particular, the group needs to make sure that their FEC implementation is low-latency while still being appropriately robust, as PCIe users won’t want a significant increase in PCIe’s latency.
It’s worth noting that FEC is also being paired with Cyclic Redundancy Checking (CRC) as a final layer of defense against bit errors. Packets that, even after FEC still fail a CRC – and thus are still corrupt – will trigger a full retransmission of the packet.
The upshot of the switch to PAM4 then is that by increasing the amount of data transmitted without increasing the frequency, the signal loss requirements won’t go up. PCIe 6.0 will have the same 36dB loss as PCIe 5.0, meaning that while trace lengths aren’t officially defined by the standard, a PCIe 6.0 link should be able to reach just as far as a PCIe 5.0 link. Which, coming from PCIe 5.0, is no doubt a relief to vendors and engineers alike.
Alongside PAM4 and FEC, the final major technological addition to PCIe 6.0 is its FLow control unIT (FLIT) encoding method. Not to be confused with PAM4, which is at the physical layer, FLIT encoding is employed at the logical level to break up data into fixed-size packets. It’s by moving the logical layer to fixed size packets that PCIe 6.0 is able to implement FEC and other error correction methods, as these methods require said fixed-size packets. FLIT encoding itself is not a new technology, but like PAM4, is essentially being borrowed from the realm of high-speed networking, where it’s already used. And, according to the PCI-SIG, it’s one of the most important pieces of the specification, as it’s the key piece to enabling (continued) low-latency operation of PCIe with FEC, as well as allowing for very minimal overhead. All told, PCI-SIG considers PCIe 6.0 encoding to be a 1b/1b encoding method, as there’s no overhead in the data encoding itself (there is however overhead in the form of additional FEC/CRC packets).
As it’s more of an enabling piece than a feature of the specification, FLIT encoding should be fairly invisible to users. However, it’s important to note that the PCI-SIG considered it important/useful enough that FLIT encoding is also being backported in a sense to lower link rates; once FLIT is enabled on a link, a link will remain in FLIT mode at all times, even if the link rate is negotiated down. So, for example, if a PCIe 6.0 graphics card were to drop from a 64 GT/s (PCIe 6.0) rate to a 2.5GT/s (PCIe 1.x) rate to save power at idle, the link itself will still be operating in FLIT mode, rather than going back to a full PCIe 1.x style link. This both simplifies the design of the spec (not having to renegotiate connections beyond the link rate) and allows all link rates to benefit from the low latency and low overhead of FLIT.
As always, PCIe 6.0 is backwards compatible with earlier specifications; so older devices will work in newer hosts, and newer devices will work in older hosts. As well, the current forms of connectors remain supported, including the ubiquitous PCIe card edge connector. So while support for the specification will need to be built into newer generations of devices, it should be a relatively straightforward transition, just like previous generations of the technology.
Unfortunately, the PCI-SIG hasn’t been able to give us much in the way of guidance on what this means for implementations, particularly in consumer systems – the group just makes the standard, it’s up to hardware vendors to implement it. Because the switch to PAM4 means that the amount of signal loss for a given trace length hasn’t gone up, conceptually, placing PCIe 6.0 slots should be about as flexible as placing PCIe 5.0 slots. That said, we’re going to have to wait and see what AMD and Intel devise over the next few years. Being able to do something, and being able to do it on a consumer hardware budget are not always the same thing.
Wrapping things up, with the PCIe 6.0 specification finally completed, the PCI-SIG tells us that, based on previous adoption timelines, we should start seeing PCIe 6.0 compliant hardware hit the market in 12-18 months. In practice this means that we should see the first server gear next year, and then perhaps another year or two for consumer gear.
Source: PCI-SIG
Facebook
Twitter
RSS
77 Comments
View All Comments
Duncan Macdonald - Wednesday, January 12, 2022 - link
Consumer products are unlikely to see PCIe 6.0 for some time due to the bandwidth limits of the main memory. Even a dual DIMM DDR5 memory can not provide enough bandwidth for a 128GB/sec PCIe link. (Corsair DDR5-6400 can manage 51GB/sec per DIMM,)To provide the required bandwidth will require a workstation or server processor with more memory channels (Threadripper, EPYC or Xeon).
The additional costs from a 4 channel memory system mean the there will be little demand for a consumer CPU to have such a capability.
Tomatotech - Wednesday, January 12, 2022 - link
As other posters have said, this is about:(1) expanding the gateway between CPU and everything else. A good set of PCIe 6.0 lanes will make getting data into and out of CPU much easier;
(2) reducing hardware costs by reducing the number of lanes needed for a hardware item. E.g. a SSD that needed 4x PCIe 4.0 lanes will need only 2x PCIe 5.0 lanes, and only 1x PCIe 6.0 lane, with a reduction in costs at each step.
(3) enabling higher SSD speeds. PCIe 5.0 has only been out for a few months, and already some 5.0 SSDs are maxing it out. It's embarrassing for the PCI-SIG standards commission that a standard that has only been out a few months is already inadequate. These standards are intended to be sufficient for at least the next few years. Hence there is a bit of a rush to get 6.0 out relatively soon to enable the industry to move forward.
Yes these high-speed SSDs are unaffordable to the typical person in the street, but there seems to be high demand from datacentres for large ultra-fast SSDs, and the PCI-SIG standard commission is in the unusual position of being the ones holding up this work.
I have to say I'm very impressed at the success of flash-based SSDs. I remember when they first came out - they were slower than HDDs and expensive and low capacity. Now the flash-based tech seems almost infinitely extensible to faster and faster speeds - it's almost like a doubling of speed every two years, combined with staggering capacities (for these with deep pockets).
mode_13h - Wednesday, January 12, 2022 - link
> expanding the gateway between CPU and everything else.Most communication the CPU does with devices is via memory. Doing otherwise would kill CPU performance, due to the latencies involved in PCIe transactions, and PCIe 6.0 won't fix that. CXL should help, but I'm not aware of a roadmap for bringing it to consumers.
> reducing hardware costs by reducing the number of lanes needed for a hardware item.
Could be, but it depends on what hardware costs you're talking about. If it's the motherboard, then what you're saying is that it can wire up fewer lanes in its various slots, however that will provide terrible performance on older, wider peripherals. So, that seems unlikely.
The next issue is that you're presuming it's a net-savings for device makers to upgrade to the next PCIe version and halve the width, which won't always be true. I'm sure it's not currently true of PCIe 5.0, and won't be true of PCIe 6.0 for a while, if ever.
> PCIe 5.0 has only been out for a few months, and already some 5.0 SSDs are maxing it out.
> It's embarrassing for the PCI-SIG standards commission
You're talking about the Samsung PM1743? That's a U.2 drive, which means it's only x4. Are you aware there are x8 and I think even x16 PCIe SSDs? All you'd have to do is put one in a 2.5" U.2 form factor with an upgraded PCIe controller.
Also, the PCI-SIG is an industry consortium. It's primarily run through the action of member companies who are building products incorporating PCIe. It's not an independent organization that's like competing with industry in some sort of race. I think they know where industry is at and what's needed, which is why they pushed onward to finalize PCIe 6.0, so soon after 5.0.
Also, I wouldn't say flash is the main use case for PCIe 6.0. It's probably compute accelerators, networking, and CXL-connected DRAM (since CXL piggybacks on part of PCIe and is backed by many of the same companies) and maybe Optane.
Of course, none of those are consumer use cases. I think Intel was over-ambitious in even rolling out PCIe 5.0. Whether it was a smart move or not depends partly on how many issues customers have with PCIe 5.0 peripherals they actually try to use in these motherboards. The other concern is board cost, although supply-chain issues make it hard to know much much PCIe 5.0 is to blame for that mess.
IntelUser2000 - Wednesday, January 12, 2022 - link
Uhh, who cares about sequential bandwidth on an SSD at this point? I think PCIe 4.0 SSDs pretty much cover all the scenarios. Maybe even 3.0.The sequential bandwidth is only achievable on select code under special circumstances and the real bandwidth falls far short of that. Far, far short.
mode_13h - Thursday, January 13, 2022 - link
I agree, in general. I happen to have a specific use case of editing large video files without transcoding, where saving the edited video clip actually runs at the sequential speed of my NVMe SSD. Granted, it's an older SSD, but it's something that takes long enough that I have to wait for it.DougMcC - Thursday, January 13, 2022 - link
It seems inevitable to me that SSDs will move to an x16 link standard, it's just a matter of time. Either Intel or AMD will get motherboards with true dual x16 in the next generation, and someone will build an SSD that plugs into that and outclasses the competition in performance, and then all the other makers will push for standardization.mode_13h - Thursday, January 13, 2022 - link
> It seems inevitable to me that SSDs will move to an x16 link standardThere are some datacenter x8 or x16 SSDs, for a long time already.
> Either Intel or AMD will get motherboards with true dual x16 in the next generation
Okay, so I guess you mean for consumers. Well, PCIe SSD cards for consumers are nothing new, nor are carrier-cards for M.2 drives. It's a niche product, with most users probably plugging them into workstation or HEDT motherboards.
DougMcC - Thursday, January 13, 2022 - link
Yes for consumers. I think the tipping point into x16 being standard is close.mode_13h - Thursday, January 13, 2022 - link
Anyone with a use case for so much I/O can *already* get a workstation or HEDT motherboard and something like this:https://www.anandtech.com/show/16247/highpoint-upd...
The reason why PCIe x16 SSDs won't go mainstream is that the market for such insane amounts of I/O throughput is too small. It's a niche product and a very expensive one, at that.
Dolda2000 - Wednesday, January 12, 2022 - link
If PCIe 6.0 is 1b/1b, what does that mean for clock recovery? As far as I'm aware, the whole reason for the bit scrambling was to provide enough transitions to recover the clock reliably, so surely there has to be some replacement for it?