400G Optics: Enabling Network Scale and Device Consolidation

Following servers, interconnect is the second highest hardware cost in a cloud/service provider's data center. Interconnect includes: data center to user, data center to data center and traffic within the data center, or server to server. Of these three categories, over 71 percent of interconnect traffic is within the data center.

To give you some perspective, a cloud provider (AWS, GoogleCloud, Azure, etc.) can easily have 300,000 servers in one region. Yes, cloud customers need to reach these servers, but what really drives the interconnect demand is that the servers need connectivity to one another for cloud applications and data replication for redundancy. 

You can imagine the challenge and hardware requirements needed to provide redundant interconnectivity between 300,000 servers. Now, consider that AWS (Amazon Web Services) alone currently has 76 of these regions across the globe.

400G Optics, the "why"

For cloud providers like AWS to continue to grow, scale and keep capex down, networking vendors and ASIC developers continue to create new routers and switches with ever increasing capacity. The increased capacity produces switches with a lower cost per bit, less power per bit and a smaller footprint. That's great — problem solved, right?

Not quite. Yes, one switch in one rack unit (1.75" high x 19" wide) can handle 14.4Tb+ of traffic, but how do you get the bits physically in and out of that switch cost effectively? Using 100G QSFP28 pluggables, for 14.4Tb, it would require 144 pluggables per switch, making the switch cost, space and power prohibitive and negating the cost savings of the new switch technology. 

The reality is that the optics, or pluggables, are significantly more expensive than the switch they are going into; denser optical capacity at lower cost is a must to realize any savings. To keep pace with ASIC and switch innovation and cost per bit reduction, optics vendors must innovate too, and they have. The latest innovation in pluggable optics is 400G QSFP-DD, providing either 1x 400G or 4x 100G in the same footprint of a 100G QSFP28. To learn more about 400G pluggable optics, please read on.

QSFP-DD, the form factor of choice

First let's look at form factor. There are two form factor specifications for 400G modules: QSFP-DD and OSFP. The one that would become the industry standard has been highly debated for the past 2 years. QSFP-DD is physically smaller than OSFP (see Figure 1 below), allowing for more ports per switch/router, but those in the OSFP camp contend that the smaller size QSFP-DD could not provide adequate heat dissipation for the 20+ watts of component power consumption. 

Those developing QSFP-DD, most notably Cisco Systems who initiated the QSFP-DD standard, assert and demonstrated at the 2019 OFC (Optical Fiber Communication Conference) that QSFP-DD can in fact effectively dissipate heat generated by modules requiring 20 watts or more of power. Read more about Cisco's 400G power demonstration. 

Currently, QSFP-DD has been widely adopted as the industry standard with virtually all OEMs building products based on it. It is important to note that even though they both have the same height and width, due to the additional QSFP-DD electrical interfaces, QSFP-DD pluggables are not physically compatible with QSFP28 based routers and switches.

400G Optics practical applications

Maybe not physically compatible, 400G QSFP-DD interface speed is certainly backwards compatible. OEMs do not expect customers to tear out all their 100G based devices and replace them with new 400G versions. Like anything, it will be a gradual migration. With that in mind, Cisco's 400G QSFP-DD pluggables allow you to break out a 400G port into 4x 100G interfaces via a breakout cable. 

Figure 2 illustrates how a 400G module in a new Cisco 8000 router can breakout to an existing ASR with 100G interfaces.  Read more about upgrading to Cisco 400G optics. This allows for flexibility in upgrades and limits downtime because not all devices need to be replaced at once. What about future compatibility you ask? QSFP-DD has that covered too. With the ability to support 20 watts of power, it can support the projected requirements of even faster speeds like 800G.

If "QSFP" sounds familiar, well that's because it is the same QSFP associated with pluggables like the 40G QSFP+ and 100G QSFP28. These form factors both have four electrical interfaces that connect to the device they are plugged into, and each interface supports 10Gbps or 25Gbps respectively. 

In our case, QSFP-DD, the "DD" stands for double density; we now have 8 electrical interfaces that can support PAM4 electrical modulation, providing 50Gbs per interface and giving us a total of 400Gbps of throughput. The physical size of QSFP-DD allows for greater switch and router port density over using the larger OSFP form factor. It also allows OEMs to keep switch and router port density consistent between existing 100G and new 400G switches and routers. Learn about Cisco's 8000 series 400G capable routers.

Figure 3 illustrates how increasing uplink speed in the same footprint has a cascading effect enabling every layer to scale, minimizing investment in rack space, power and additional fiber. 

400G-ZR: 400G DWDM in a pluggable

There is one specific 400G technology that I have not mentioned yet, and it will benefit any size network where two or more locations need connectivity. Up until now we have been looking at 400Gb Ethernet, which can be referred to as "grey optics." Allow me to introduce 400G-ZR.  

400G-ZR is the same form factor (QSFP-DD), sits in the same router or switch but is a DWDM (dense wave division multiplexing) optic. This allows you to interconnect 2 sites that are up to 100Km away, with multiple 400G connections with only 2 strands of fiber between each site. This capability currently requires a separate layer 1 device (transponder). With 400G-ZR, that device has been consolidated into the QSFP-DD form factor. 

As you can see in Figure 4, the "Tunable Transponder" has been removed from the image on the right and that functionality now resides in the pluggable seated in the router. This consolidation saves power, rack space and reduces management complexity. 

Conclusion

I hope this article helped you see why 400G optics are necessary for providers to scale and reduce costs, and that it provided relevant background on the technology and how it will make a significant impact on our industry. 

We have only scratched the surface on the details of 400G. Stay tuned for my next article, where we will take a closer look at the various 400G modules Cisco has to offer, the technology enabling 400G-ZR DWDM and explore more use cases. Feel free to contact us today with any questions or to discuss your specific use case.