I first heard about MRC while out on a run, listening to my weekly networking news podcast. I stopped dead in my tracks and listened to the rest of the discussion. Then I went home and started obsessively researching it.
MRC already puts some of the techniques developed by the Ultra Ethernet Consortium to work in production networks. I think Ultra Ethernet will eventually become the default stack for datacenter networks.
But first, what the hell is MRC?
MRC (Multipath Reliable Connection) extends RoCE v2. OpenAI, Microsoft, and NVIDIA developed the extension; the Open Compute Project published it as an open spec. OpenAI’s GB200 training clusters already run it in production.
It borrows a lot of ideas from Ultra Ethernet spec but it runs on NICs you can buy today.
To understand it, we need to cover a few networking concepts that have been around for years.
ECMP (Equal-Cost Multipath)
ECMP lets a router or Layer 3 switch forward traffic through multiple equal-cost next hops.
Suppose two servers have four equal-cost paths between them. ECMP can use all four paths, but how does it decide which one to use?
Most conventional ECMP implementations use per-flow hashing.
A flow is typically identified by five fields: source IP address, destination IP address, source port, destination port, and transport protocol. These are commonly called the five-tuple.
The switch runs these fields through a hash function to select one of the available next hops. Packets belonging to the same flow normally take the same path, helping preserve packet ordering.
ECMP distributes flows, not individual packets. So Imagine four equal-cost paths between two servers.
Several large, bursty transfers, like KV-cache transfers between disaggregated prefill and decode workers, can hash onto the same path while others have spare capacity. You end up with a few congested links even though there’s plenty of bandwidth available elsewhere in the network.
🗨️ If you’ve ever dealt with partner-operated OCI FastConnect, Azure ExpressRoute, or AWS Direct Connect links, you’ll know the frustration.
You call the provider because performance has gone to shit, only to hear, “Oh yeah, we had a fiber cut…. Failover worked, but ECMP ended up putting too much traffic on one of the remaining links.”
Your handoff never went down. BGP/BFD is happy, but latency is through the roof. A classic network brownout.
RDMA and RoCE
You may have come across RDMA in HCI solutions like Storage Spaces Direct (S2D), VMware vSAN or Proxmox VE.
It’s used to reduce CPU overhead for storage traffic between hosts.
How does it work?
RDMA (Remote Direct Memory Access) lets network adapters move data directly between application memory on different servers, without the receiving CPU processing each packet.
For an RDMA write, Server A’s application tells its RDMA capable network card or NIC, what to send and where.
The NIC reads the data from local memory and sends it to Server B.
Server B’s NIC writes it directly into a registered application buffer. The receiving CPU does not process each packet.
Server B’s CPU then reads the data from memory once the application knows it is ready. A plain RDMA write does not automatically notify the receiving application, so the software coordinates that separately.
Why MRC?
Sender
🗨️ Going back to our ECMP example above, with fixed hash inputs, packets from one connection keep taking the same path. Two large transfers can end up competing for the same link while another link sits unused.
MRC takes advantage of the hashing the switches already do.
With MRC the sending NIC varies the UDP source port from packet to packet.
The switches continue hashing normally because the source port is one of the fields switches already use in their ECMP hash. (Switches don’t need new firmware to understand MRC)
But now the changing input lets packets from the same RDMA connection take different paths. They call this packet spraying.
Now one transfer can spread its packets across all four paths.
Receiver
UDP source ports differ but the packets still belong to one RDMA connection since they all use the same RDMA transport header
The receiving NIC (with MRC support) uses the headers to ID the connection and track packet delivery.
Each data packet also carries the placement information needed to write its payload into the correct destination memory location.
If packet 3 arrives before packet 2 (out of order), the NIC can write packet 3’s data straight into its destination memory location and record that packet 2 has not arrived. It does not have to hold or buffer packet 3 while waiting for packet 2.
The receiver sends selective acknowledgements (SACKs) to report which packets have arrived and where there are gaps. In this example, its feedback effectively says: “I have packets 1 and 3; packet 2 has not arrived yet.”
Packet 2 might just be delayed. The sender uses that feedback, along with timing information, to decide whether to wait or retransmit it. No need to resend the data that already arrived.
The operation completes only once every packet has been placed and the sender has received acknowledgement for all of them.
What do we need to run it?
MRC is designed to run over best-effort Ethernet. Packets can get dropped, and the endpoints handle recovering them without relying on Ethernet flow control.
Ethernet flow control (PFC)?
Say your server is sending RDMA traffic into a switch faster than the switch can forward it. The buffer holding that traffic starts filling up.
Before it overflows, the switch sends a PFC pause frame back to the server’s NIC saying: “Stop sending this traffic class for a moment. I need to clear some buffer space.” The NIC pauses that traffic, then resumes when the pause expires or the switch tells the server nic to resume.
All connections sharing that paused priority have to wait, including those that did not cause the congestion
But this is not a turnkey solution. Just like RDMA, both endpoints (NICs) need the firmware support. The host’s drivers, RDMA libraries and communication software also need to support and use the MRC protocol.
Major vendors already have MRC implementations on High end 800GE NICs like NVIDIA ConnectX-8, AMD Pensando and Broadcom Thor Ultra.
Right now MRC is for frontier labs and hyperscales with training clusters at a scale only a handful of companies can pay for.
But we will all reap the benefits of this in more tokens, faster and cheaper, once it gets adopted widely across the world.
Let me address the question of why I decided to put a DNS server (Pihole) exposed to the internet (not fully open but still).
I needed/wanted to set up an Umbrella/NextDNS/CF type DNS server that’s publicly accessible but secured to certain IP addresses.
Sure NextDNS is an option and its cheap with similar features, but i wanted roll my own solution so i can learn a few things along the way
I can easily set this up for my family members with minimal technical knowledge and unable to deal with another extra device (Raspberry pi) plugged into their home network.
This will also serve as a quick and dirty guide on how to use Docker compose and address some Issues with Running Pi-hole, Docker with UFW on Ubuntu 20.x
So lets get stahhhted…….
Scope
Setup Pi-hole as a docker container on a VM
Enable IPV6 support
Setup UFW rules to prune traffic and a cronjob to handle the rules to update with the dynamic WAN IPs
Deploy and test
What we need
Linux VM (Ubuntu, Hardened BSD, etc)
Docker and Docker Compose
Dynamic DNS service to track the changing IP (Dyndns,no-Ip, etc)
Deployment
Setup Dynamic DNS solution to track your Dynamic WAN IP
for this demo, we are going to use DynDNS since I already own a paid account and its supported on most platforms (Routers, UTMs, NAS devices, IP camera-DVRs, etc)
Use some google-fu there are multiple ways to do this without having to pay for the service, all we need is a DNS record that's up-to-date with your current Public IP address.
For Network A and Network B, I’m going to use the routers built-in DDNS update features
Network A gateway – UDM Pro
Network B Gateway – Netgear R6230
Confirmation
Setup the VM with Docker-compose
Pick your service provider, you can and should be able to use a free tier VM for this since its just DNS
Linode
AWS lightsail
IBM cloud
Oracle cloud
Google Compute
Digital Ocean droplet
Make sure you have a dedicated (static) IPv4 and IPv6 address attached to the resource
For this deployment, I’m going to use a Linode – Nanode, due to their native IPv6 support and cause I prefer their platform for personal projects
Lets Configure the docker networking side to fit our Needs
Create a Seperate Bridge network for the Pi-holecontainer
I guess you could use the default bridge network, but I like to create one to keep things organized and this way this service can be isolated from the other containers I have
With the new ubuntu version 20.x, Systemd will start a local DNS stub client that runs on 127.0.0.53:53
which will prevent the container from starting. because Pi-hole binds to the same port UDP 53
we could disable the service but that breaks DNS resolution on the VM causing more headaches and pain for automation and updates
After some google fu and trickering around this this is the workaround i found.
Disable the stub-listener
Change the symlink to the /etc/resolved.conf to /run/systemd/resolve/resolv.conf
push the external name servers so the VM won’t look at loopback to resolve DNS
Restart systemd-resolved
Resolving Conflicts with the systemd-resolved stub listener
We need to disable the stub listener thats bound to port 53, as i mentioned before this breaks the local dns resolution we will fix it in a bit.
sudo nano /etc/systemd/resolved.conf
Find and uncomment the line “DNSStubListener=yes” and change it to “no”
After this we need to push the external DNS servers to the box, this setting is stored on the following file
/etc/resolv.conf
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
nameserver 127.0.0.53
But we cant manually update this file with out own DNS servers, lets investigate
ls -l /etc/resolv.conf
its a symlink to the another system file
/run/systemd/resolve/stub-resolv.conf
When you take a look at the directory where that file resides, there are two files
When you look at the other file you will see that /run/systemd/resolve/resolv.conf is the one which really is carrying the external name servers
You still can’t manually edit This file, and it gets updated by whatever the IPs provided as DNS servers via DHCP. netplan will dictate the IPs based on the static DNS servers you configure on Netplan YAML file
i can see there two entries, and they are the default Linode DNS servers discovered via DHCP, I’m going to keep them as is, since they are good enough for my use case
container_name – This is the name of the container on the docker container registry
image – What image to pull from the Docker Hub
hostname – This is the host-name for the Docker container – this name will show up on your lookup when you are using this Pi-hole
ports – What ports should be NATed via the Docker Bridge to the host VM
TZ – Time Zone
DNS1 – DNS server used with in the image
DNS2 – DNS server used with in the image
WEBPASSWORD – Password for the Pi-Hole web console
ServerIP – Use the IPv4 address assigned to the VMs network interface(You need this for the Pi-Hole to respond on the IP for DNS queries)
IPv6 – Enable Disable IPv6 support
ServerIPv6 – Use the IPv4 address assigned to the VMs network interface (You need this for the Pi-Hole to respond on the IP for DNS queries)
volumes – These volumes will hold the configuration data so the container settings and historical data will persist reboots
cap_add:- NET_ADMIN – Add Linux capabilities to edit the network stack – link
restart: always – This will make sure the container gets restarted every time the VM boots up – Link
networks:default:external:name: Piholev6 – Set the container to use the network bridge we created before
Now lets bring up the Docker container
docker-compose up -d
-d switch will bring up the Docker container in the background
Run ‘Docker ps’ to confirm
Now you can access the web interface and use the Pihole
verifying its using the bridge network you created
Grab the network ID for the bridge network we create before and use the inspect switch to check the config
docker network ls
docker network inspect f7ba28db09ae
This will bring up the full configuration for the Linux bridge we created and the containers attached to the bridge will be visible under the “Containers”: tag
Testing
I manually configured my workstations primary DNS to the Pi-Hole IPs
Updating the docker Image
Pull the new image from the Registry
docker pull pihole/pihole
Take down the current container
docker-compose down
Run the new container
docker-compose up -d
Your settings will persist this update
Securing the install
now that we have a working Pi-Hole with IPv6 enabled, we can login and configure the Pihole server and resolve DNS as needed
but this is open to the public internet and will fall victim to DNS reflection attacks, etc
lets set up firewall rules and open up relevant ports (DNS, SSH, HTTPS) to the relevant IP addresses before we proceed
Disable IPtables from the docker daemon
Ubuntu uses UFW (uncomplicated firewall) as an obfuscation layer to make things easier for operators, but by default, Docker will open ports using IPtables with higher precedence, Rules added via UFW doesn’t take effect
So we need to tell docker not to do this when launching a container so we can manage the firewall rules via UFW
This file may not exist already if so nano will create it for you
sudo nano /etc/docker/daemon.json
Add the following lines to the file
{
"iptables": false
}
restart the docker services
sudo systemctl restart docker
now doing this might disrupt communication with the container until we allow them back in using UFW commands, so keep that in mind.
Automatically updating Firewall Rules based on the DYN DNS Host records
we are going to create a shell script and run it every hour using crontab
Shell Script Dry run
Get the IP from the DYNDNS Host records
remove/Cleanup existing rules
Add Default deny Rules
Add allow rules using the resolved IPs as the source
Dynamic IP addresses are updated on the following DNS records
trusted-Network01.selfip.net
trusted-Network02.selfip.net
Lets start by creating the script file under /bin/*
#!/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
now=$(date +"%m/%d/%T")
DYNDNSNetwork01="trusted-Network01.selfip.net"
DYNDNSNetwork02="trusted-Network02.selfip.com"
#Get the network IP using dig
Network01_CurrentIP=`dig +short $DYNDNSNetwork01`
Network02_CurrentIP=`dig +short $DYNDNSNetwork02`
echo "-----------------------------------------------------------------"
echo Network A WAN IP $Network01_CurrentIP
echo Network B WAN IP $Network02_CurrentIP
echo "Script Run time : $now"
echo "-----------------------------------------------------------------"
#update firewall Rules
#reset firewall rules
#
sudo ufw --force reset
#
#Re-enable Firewall
#
sudo ufw --force enable
#
#Enable inbound default Deny firewall Rules
#
sudo ufw default deny incoming
#
#add allow Rules to the relevant networks
#
sudo ufw allow from $Network01_CurrentIP to any port 22 proto tcp
sudo ufw allow from $Network01_CurrentIP to any port 8080 proto tcp
sudo ufw allow from $Network01_CurrentIP to any port 53 proto udp
sudo ufw allow from $Network02_CurrentIP to any port 53 proto udp
#add the ipV6 DNS allow all Rule - Working on finding an effective way to lock this down, with IPv6 rick is minimal
sudo ufw allow 53/udp
#find and delete the allow any to any IPv4 Rule for port 53
sudo ufw --force delete $(ufw status numbered | grep '53*.*Anywhere.' | grep -v v6 | awk -F"[][]" '{print $2}')
echo "--------------------end Script------------------------------"
Lets run the script to make sure its working
I used a online port scanner to confirm
Setup Scheduled job with logging
lets use crontab and setup a scheduled job to run this script every hour
Make sure the script is copied to the /bin folder with the executable permissions
using crontab -e (If you are launching this for the first time it will ask you to pick the editor, I picked Nano)
Let’s talk a little bit about this code and unpack this
Vagrant API version
Vagrant uses API versions for its configuration file, this is how it can stay backward compatible. So in every Vagrantfile we need to specify which version to use. The current one is version 2 which works with Vagrant 1.1 and up.
Provisioning the Ansible VM
This will
Provision the controller Ubuntu VM
Create a bridged network adapter
Set the host-name – LAB-Controller
Set the static IP – 172.17.10.120/24
Run the Shell script that installs Ansible using apt-get install (We will get to this below)
Lets start digging in…
Specifying the Controller VM Name, base box and hostname
Vagrant uses a base image to clone a virtual machine quickly. These base images are known as “boxes” in Vagrant, and specifying the box to use for your Vagrant environment is always the first step after creating a new Vagrantfile.
Define the shell script to customize the VM config and install the Ansible Package
Now this is where we define the provisioning shell script
this script installs Ansible and set the host file entries to make your life easier
In case you are wondering VLS stands for V=virtual,L – linux S – server.
I use this naming scheme for my VMs. Feel free to use anything you want; make sure it matches what you defined on the Vagrantfile under node.vm.hostname
We covered most of the code used above, the only difference here is we are using each method to create 3 VMs with the same template (I’m lazy and it’s more convenient)
This will create three Ubuntu VMs with the following Host-names and IP addresses, you should update these values to match you LAN, or use a private Adapter
vls-node1 – 172.17.10.121
vls-node2 – 172.17.10.122
vls-node1 – 172.17.10.123
So now that we are done with explaining the code, let’s run this
Building the Lab environment using Vagrant
Issue the following command to check your syntax
Vagrant status
Issue the following command to bring up the environment
Vagrant up
If you get this message Reboot in to UEFI and make sure virtualization is enabled
Intel – VT-D
AMD Ryzen – SVM
If everything is kumbaya you will see vagrant firing up the deployment
It will provision 4 VMs as we specified
Notice since we have the “vagrant-vbguest” plugin installed, it will reinstall the relevant guest tools along with the dependencies for the OS
==> vls-node3: Machine booted and ready!
[vls-node3] No Virtualbox Guest Additions installation found.
rmmod: ERROR: Module vboxsf is not currently loaded
rmmod: ERROR: Module vboxguest is not currently loaded
Reading package lists...
Building dependency tree...
Reading state information...
Package 'virtualbox-guest-x11' is not installed, so not removed
The following packages will be REMOVED:
virtualbox-guest-utils*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 5799 kB disk space will be freed.
(Reading database ... 61617 files and directories currently installed.)
Removing virtualbox-guest-utils (6.0.14-dfsg-1) ...
Processing triggers for man-db (2.8.7-3) ...
(Reading database ... 61604 files and directories currently installed.)
Purging configuration files for virtualbox-guest-utils (6.0.14-dfsg-1) ...
Processing triggers for systemd (242-7ubuntu3.7) ...
Reading package lists...
Building dependency tree...
Reading state information...
linux-headers-5.3.0-51-generic is already the newest version (5.3.0-51.44).
linux-headers-5.3.0-51-generic set to manually installed.
Check the status
Vagrant status
Testing
Connecting via SSH to your VMs
vagrant ssh controller
“Controller” is the VMname we defined before not the hostname, You can find this by running Vagrant status on posh or your terminal
We are going to connect to our controller and check everything
Little bit more information on the networking side
Vagrant Adds two interfaces, for each VM
NIC 1 – Nat’d in to the host (control plane for Vagrant to manage the VMs)
NIC 2 – Bridged adapter we provisioned in the script with the IP Address
Default route is set via the Private(NAT’d) interface (you cant change it)
Netplan configs
Vagrant creates a custom netplan yaml for interface configs
I hope this helped someone. when I started with Vagrant a few years back it took me a few tries to figure out the system and the logic behind it, this will give you a basic understanding on how things are plugged together.
let me know in the comments if you see any issues or mistakes.