firmware - multicastbits.com

I first heard about MRC while out on a run, listening to my weekly networking news podcast. I stopped dead in my tracks and listened to the rest of the discussion. Then I went home and started obsessively researching it.

MRC already puts some of the techniques developed by the Ultra Ethernet Consortium to work in production networks. I think Ultra Ethernet will eventually become the default stack for datacenter networks.

But first, what the hell is MRC?

MRC (Multipath Reliable Connection) extends RoCE v2. OpenAI, Microsoft, and NVIDIA developed the extension; the Open Compute Project published it as an open spec. OpenAI’s GB200 training clusters already run it in production.

It borrows a lot of ideas from Ultra Ethernet spec but it runs on NICs you can buy today.

To understand it, we need to cover a few networking concepts that have been around for years.

ECMP (Equal-Cost Multipath)

ECMP lets a router or Layer 3 switch forward traffic through multiple equal-cost next hops.

Suppose two servers have four equal-cost paths between them. ECMP can use all four paths, but how does it decide which one to use?

Most conventional ECMP implementations use per-flow hashing.

A flow is typically identified by five fields: source IP address, destination IP address, source port, destination port, and transport protocol. These are commonly called the five-tuple.

The switch runs these fields through a hash function to select one of the available next hops. Packets belonging to the same flow normally take the same path, helping preserve packet ordering.

ECMP distributes flows, not individual packets. So Imagine four equal-cost paths between two servers.

Several large, bursty transfers, like KV-cache transfers between disaggregated prefill and decode workers, can hash onto the same path while others have spare capacity. You end up with a few congested links even though there’s plenty of bandwidth available elsewhere in the network.

🗨️ If you’ve ever dealt with partner-operated OCI FastConnect, Azure ExpressRoute, or AWS Direct Connect links, you’ll know the frustration.

You call the provider because performance has gone to shit, only to hear, “Oh yeah, we had a fiber cut…. Failover worked, but ECMP ended up putting too much traffic on one of the remaining links.”

Your handoff never went down. BGP/BFD is happy, but latency is through the roof. A classic network brownout.

RDMA and RoCE

You may have come across RDMA in HCI solutions like Storage Spaces Direct (S2D), VMware vSAN or Proxmox VE.

It’s used to reduce CPU overhead for storage traffic between hosts.

How does it work?

RDMA (Remote Direct Memory Access) lets network adapters move data directly between application memory on different servers, without the receiving CPU processing each packet.
For an RDMA write, Server A’s application tells its RDMA capable network card or NIC, what to send and where.
The NIC reads the data from local memory and sends it to Server B.
Server B’s NIC writes it directly into a registered application buffer. The receiving CPU does not process each packet.
Server B’s CPU then reads the data from memory once the application knows it is ready. A plain RDMA write does not automatically notify the receiving application, so the software coordinates that separately.

Why MRC?

Sender

🗨️ Going back to our ECMP example above, with fixed hash inputs, packets from one connection keep taking the same path. Two large transfers can end up competing for the same link while another link sits unused.

MRC takes advantage of the hashing the switches already do.
With MRC the sending NIC varies the UDP source port from packet to packet.
The switches continue hashing normally because the source port is one of the fields switches already use in their ECMP hash. (Switches don’t need new firmware to understand MRC)
But now the changing input lets packets from the same RDMA connection take different paths. They call this packet spraying.
Now one transfer can spread its packets across all four paths.

Receiver

UDP source ports differ but the packets still belong to one RDMA connection since they all use the same RDMA transport header
The receiving NIC (with MRC support) uses the headers to ID the connection and track packet delivery.
Each data packet also carries the placement information needed to write its payload into the correct destination memory location.
If packet 3 arrives before packet 2 (out of order), the NIC can write packet 3’s data straight into its destination memory location and record that packet 2 has not arrived. It does not have to hold or buffer packet 3 while waiting for packet 2.
The receiver sends selective acknowledgements (SACKs) to report which packets have arrived and where there are gaps. In this example, its feedback effectively says: “I have packets 1 and 3; packet 2 has not arrived yet.”
Packet 2 might just be delayed. The sender uses that feedback, along with timing information, to decide whether to wait or retransmit it. No need to resend the data that already arrived.
The operation completes only once every packet has been placed and the sender has received acknowledgement for all of them.

What do we need to run it?

MRC is designed to run over best-effort Ethernet. Packets can get dropped, and the endpoints handle recovering them without relying on Ethernet flow control.

Ethernet flow control (PFC)?

Say your server is sending RDMA traffic into a switch faster than the switch can forward it. The buffer holding that traffic starts filling up.

Before it overflows, the switch sends a PFC pause frame back to the server’s NIC saying: “Stop sending this traffic class for a moment. I need to clear some buffer space.” The NIC pauses that traffic, then resumes when the pause expires or the switch tells the server nic to resume.

All connections sharing that paused priority have to wait, including those that did not cause the congestion

But this is not a turnkey solution. Just like RDMA, both endpoints (NICs) need the firmware support. The host’s drivers, RDMA libraries and communication software also need to support and use the MRC protocol.

Major vendors already have MRC implementations on High end 800GE NICs like NVIDIA ConnectX-8, AMD Pensando and Broadcom Thor Ultra.

Right now MRC is for frontier labs and hyperscales with training clusters at a scale only a handful of companies can pay for.

But we will all reap the benefits of this in more tokens, faster and cheaper, once it gets adopted widely across the world.

Preface-

After I started working with Open networking switches, wanted to know more about the Cisco catalyst range I work with every day.

Information on older ASICS is very hard to find, but recently they have started to talk a lot about the new chips like UADP 2.0 with the Catalyst 9k / Nexus launch, This is more likely due to the rise of Desegregated Network Operating Systems DNOS such as Cumulus and PICA8, etc forcing customers to be more aware of what’s under the hood rather than listening and believing shiny PDF files with a laundry list of features.

The information was there but scattered all over the web, I went though CiscoLive, TechFieldDay slides/videos, interviews, partner update PDFs, Press leases and whitepapers and even LinkedIn profiles to gather information

If you notice a mistake please let me know

Scope –

we are going to focus on the ASIC’s used in the well-known 2960S/X/XR and the 36xx,37xx,38XX and the new Cat 9K series

Timeline

Summary

Cisco Brought a bunch of companies to acquire the switching technology they needed that later bloomed into the switching platforms we know today

Crescendo Communications (1993) – Catalyst 5K and 6K chassis
Kalpana (1994) – Catalyst 3K (Fun Fact they invented VLANs that later got standardized as 802.1q)
Grand Junction (1995) – Catalyst 17xx, 19xx, 28xx, 29xx
Granite Systems (1996) – Catalyst 4K (K series ASIC)

After the original Cisco 3750/2950 switches, Cisco 3xxx/2xxx-G (G for Gigabit) was released

Next, the Cisco 3xxx-E series with enterprise management functions was released

later, Cisco developed the Cisco 3750-V series with the function of energy-saving version for –E series, later replaced by Cisco 3750 V2 series (Possibly a die shrink)

G series and E series were later phased out and integrated into Cisco X series. which is still being sold and supported

in 2017-2018 Cisco released the catalyst 9k family to replace the 2K and 3K families

Sasquatch ASIC

from what I could find there are two variants of this ASIC

The initial release in 2003

Fixed pipeline ASIC
180 Nano-meter process
60 Million Transistors

Shipped with the 10/100 3750 and 2960

Die Shrink to 130nm in 2005

Fixed pipeline ASIC
130 Nano-meter process
210 Million Transistors

Shipped in the 2960-G/3560-G/3750-G series

I couldn’t find much info about the chip design. will update this section when I find more.

Strider ASIC

Initially Release in 2010

Fixed pipeline ASIC
Built on the 65-nanometer process
1.3 Billion Transistors

Strider ASIC (circa 2010) was an improved design based on the 3750-E series was first shipped with the 2960-S family.

later in 2012-2013 with a die shrink to 45-nanometer, they managed to fit 2 ASICs in the same silicon, This shipped with the 2960-X/XR which replaced the 2960-S

higher stack speeds and features
limited layer 3 capabilities IP Lite feature (2960-XR only)
Better QoS and Netflow lite

Later down the line they silently rolled the ASIC design to a 32-nanometer process for better yield to achieve cheaper manufacturing costs

this switch is still being sold with no EOL announced as a cheaper Acess layer switch

On a side note – in 2017 Cisco released another version of the 2960 family the WS-2960-L This is a cheaper version built on a Marvel ASIC (Same as the SG55x) with a web UI and fanless design. I personally think this is the next evolution of their SMB market-oriented family the popular Cisco SG-5xx series. for the first the time the 2960 family had a fairly usable and pleasant web interface for the configuration and management. the new 9K series seems to be containing a more polished version of the web-UI

Unified Access Data Plane (UADP)

Due to the limitations in the fixed pipeline architecture and the costs involved with the re-rolling process to fix bugs they needed something new and had three options

As a compromise between all three Cisco Started dreaming up this programmable ASIC design in 2007-2008 the idea was to build a chip with programmable stages that can be updated with firmware updates instead of writing the logic into the silicon permanently.

they released the programmable ASIC technology initially for their QFP (Quantum flow processor) ASIC in the ISR router family to meet the customer needs (service providers and large enterprises)

This chip allowed them to support advanced routing technologies and new protocols without changing hardware simply via firmware updates improving the longevity of the investment allowing them to make more money out of the chips extended life cycle.

Eventually, this technology trickled downstream and the Doppler 1.0 was born

Improvements and features in UADP 1.0/1.1/2.0

Programmable stages in the pipeline
Cisco intent Driven networking support – DNA Center with ISE
Intergarted Stacking Support with Stack power – ASIC is built with pinouts for the stacking fabric allowing faster stacking performance
Rapid Recirculation (Encapsulation such as MPLS, VXLAN)
TrustSec
Advance on-chip QOS
Software-defined networking support – integrated NetFlow, SD access
Flex Parser – Programmable packet parser allowing them to introduce support for new protocols with firmware updates
On-chip Micro-engines – Highly specialized engines within in the chip to perform repetitive tasks such as
- Reassembly
- Encryption/Decryption
- Fragmentation
CAPWAP – Switch can function as a wireless Lan Controller
- Mobility agent – Offload Qo and other functions from the WLC (IMO Works really nicely with multi-site wireless deployments)
- Mobility Controller – Fullblown Wireless LAN controller (WLC)
Extended life cycle allowed integration of Cisco security technologies such as Cisco DNA + ISE later down the line even on the first generation switches
Multigig and 40GE speed support
Advanced malware detection capabilities via packet fingerprinting

Legacy Fixed pipeline architecture

Programmable pipeline architecture

Doppler/UADP 1.0 (2013)

While doppler1.0 programmable ASIC handling the Data plane coupled with Cavium CPU for the control plane the first generation of the switches to ship with these chip was the 3650-x and 3850-x gigabit versions

Built on 65 Nanometer Process
1.3 billion transistors

UADP 1.1 (2015)

Die Shrink to 45 Nanometer
3 billion transistors

UADP 2.0 (2017)

Built on 28nm/16nm Process
Equipped with an Intel Xeon D (Dual-core X86) CPU for the control plane
Open-IOS-XE

7.4 billion transistors

Flexible ASIC Templates –

Allows Cisco to write templates that can optimize the chip resources for different use cases

the new Catalyst 9000 series will replace the following campus switching families built on the older Strider and more recent UADP 1 and 1.1 ASICS

Catalyst 2K —–> Catalyst 9200
Catalyst 3K —–> Catalyst 9300
Catalyst 4K —–> Catalyst 9400
Catalyst 6K —–> Catalyst 9500

I’m will update/fix this post when I find more info about the UADP 2 and the next evolution, stay tuned for a few more articles based on the silicon used in open networking X86 chassis.

MRC: RDMA Across Multiple Paths