Preparing the Path for More Efficient Data Centers

IEEE standards project to help data centers handle more traffic at a lower cost

7 August 2013

No two ways about it: Today’s data centers are badly strained. The Internet, mobile computing, cloud computing, virtual servers, and higher-performance computing applications, along with advances in processors, are all adding to a gigantic load. As the market for 100 gigabits per second (Gb/s) Ethernet grows, data centers need high-throughput LAN connections that have higher density, cost less to operate, and are more energy-efficient. Finding ways to allow all this is the focus of the task force working on the IEEE P802.3bm 40 Gb/s and 100 Gb/s Fiber Optic, a recently formed project within the IEEE 802.3 working group, which continues to be responsible for the development of standards for Ethernet.

“The demand for data throughput is going up extremely fast, so if we had not started on IEEE 802.3bm, data centers would not be able to keep up with it,” says the chair of the task force, IEEE Member Daniel Dove. He is a senior director of technology for connectivity solutions at AppliedMicro, in Sunnyvale, Calif.

The group is working on several fronts. These include adding 100 Gb/s Physical Layer (PHY) specifications and management parameters, and using a four-lane electrical interface for devices intended to operate over single-mode fiber (SMF) and multimode fiber (MMF) optic cables. SMF is used for longer-distance communication and MMF for distances of less than 100 meters. The new standard will also extend the reach of 40 Gb/s SMF solutions, and make the operation of 40 Gb/s and 100 Gb/s fiber networks more energy-efficient.

KEEPING PACE

In 2010, the IEEE Std. 802.3ba first defined transmitting Ethernet frames at rates of 40 and 100 Gb/s. At that time the maximum speed of commercial input/output devices was around 10 Gb/s. To get to 100 Gb/s, data centers had to use 10-by-10 Gb/s electrical interfaces, which meant that the components had to be physically wide.

“The interfaces to do 10-wide consumed significant power and tended to be relatively expensive,” Dove says, “so the initial work produced by IEEE 802.3ba was focused on a solution that was interoperable; cost was not the driving factor. The wider interfaces required more optical devices and/or gearbox ICs, hence more power and bigger optical modules, which means you got less density.

“This was fine for the initial 100 Gb/s deployment because the products for handling this, depending on the technology, didn’t need the higher density that is demanded today,” he continues. “Today’s newer data center architectures are driving demand for more 100 Gb/s links, and therefore power, cost, and density are becoming critical.”

The task force is proposing 100 Gb/s MMF PHY(s) that would use four 25 Gb/s electrical interconnect lanes in each direction. This allows for an eight-fiber link that provides for higher switch density, improves rack utilization, and reduces the number of optical components. For SMF PHY(s), the project does this by eliminating the need for a gearbox inside the optical module. The gearbox is a logic function or a device that maps data between a 10-by-10 Gb/s interface and a 4-by-25 Gb/s interface, where the input and output data-path lane widths and line rates are not evenly divisible. It does this by performing multiplexing, de-multiplexing, and shift and retiming operations on the received data stream.

But a gearbox is a power-consuming device that occupies a substantial portion of the optical module, according to Dove. By eliminating the gearbox, it’s possible to reduce the width of module interfaces and remove the power and cost of a gearbox. Today’s C Form-factor Pluggable (CFP) modules can be replaced by CFP2 and/or CFP4, and some claim it's possible to fit into the QSFP-28 form factor.

“The ability to lower cost, power, and size by going to 4-by-25 is what drove the development of IEEE P802.3bm,” Dove says. “If you can double the number of ports, which is done by eliminating the gearbox, you essentially cut in half the cost-per-port of the overall switch infrastructure such as fans, power supplies, and chassis. There is a cost savings of simply doubling density, even if you can’t lower the cost of the components, but optical module costs are expected to also be reduced substantially.”

MORE REACH AND IMPROVED EFFICIENCY

IEEE P802.3bm also defines extended-reach optics for 40 Gb/s single-mode fiber, which is currently up to 10 kilometers. It calls for a 40 Gb/s PHY for operation over at least 40 kilometers. As of press time, a 100 Gb/s PHY for operation of at least 500 meters was under consideration.

The project also establishes a higher-density short-reach optical interface for multimode cables. Thus, it defines a 100 Gb/s PHY for operation on OM4 cabling up to 100 meters using 8 optical fibers versus the 20 fibers required for 100GBASE-SR10.

The project is also defining an optional Energy-Efficient Ethernet setting for operation on fiber optic cables, which allows data centers to reduce power consumption by 50 percent or more during periods of low activity.

The standard is on track to be approved between March and June of 2015.

Learn More