I’ve really enjoyed the Ubiquity equipment upgrade I made this past year to my home network. The level of insight and control I now have is well beyond my previous Xfinity system. In particular, I’ve become curious about the SFP+ ports on the UDM-Pro. Reaching speeds of 10 gigabytes over a network connection at home blew my mind.
After doing research about these ports, I decided to explore Link Aggregation (LAG) to learn if it is possible to gain speeds beyond the one-gigabit download speed offered by Comcast. This required purchasing a small amount of equipment and building more knowledge for myself on networking systems.
The short of it: Yes, I was able to achieve the 1.2 Gbps download speeds that I read about online. But this came at a cost of losing speed on the upload side from 40 Mbps to 17 Mbps or roughly half.
This post shares how I performed this experiment. I won’t go into detail about every change and why I made it. Instead, I’ll include links to write-ups and videos smarter people than me have created that cover different parts of this setup.
The Equipment
We need to start by listing the equipment. The links include where to purchase them yourself:
- UniFi Dream Machine Pro (UDM-Pro) running the UniFi controller version 6.1.61.
- UniFi Switch PoE 16 (150W) (UniFi Switch).
- NETGEAR Nighthawk Cable Modem CM1200 – Comcast approved and has LAG capabilities.
- Mikrotik 5 port Switch/Router CRS305-1G-4S+IN – Will live between UDM-Pro and the CM1200.
- 2 QSFPTEK 10GBASE-T SFP+ transceivers – used to connect CAT 8 ethernet cables to CM1200 and UDM-Pro.
- 1 SFP+ DAC Twinax Cable – used to connect the CRS305 to the Switch over the SFP port.
- 2 QSFPTEK UF-MM-10G SFP+ Multimode transceiver – used to connect the CRS305 to the UDM-Pro.
- 1 FLYPROFiber @M OM4 LC to LC Fiber patch cable – fiber cable used with the previous transceivers.
Additionally, Comcast provides internet service with their Gigabit Internet speed and Unlimited Data Option.
Preparing the Network
The goal was to place the CRS305 in bridge mode between the UDM-Pro and the cable modem. The Mikrotik router would connect to the SFP+ WAN2 port on the UDM-Pro over a fiber connection. And it would connect with the cable modem on ports 1 and 2 using CAT 8 ethernet cables into the QSFPTEK transceivers into the SFP+ ports.
I also wanted to “see” what was going on inside the CRS305. This meant that it needed to connect to the network so that the web config page would be accessible. Early tests showed this would not be possible across the WAN ports. So that meant a LAN connection. To figure that out, I used this Medium post by Igor Zevaka that described how he did this with his modem and a UDM-Pro.
Based on that, here is what this network will look like.
Defining VLANs
The first changes I made were to define some VLANs and the appropriate firewall rules to segment my network. I created an IOT network on 196.128.20.0 and a Bridge network on 196.128.88.0. All my IOT devices are isolated from my core systems.
The same is true on the Bridge network. This network uses 88 since the CRS305 defaults to 192.168.88.1 if you reset it. I felt that would allow me a chance to remember this in case something happened in my experiment.
How I set up those VLANs and firewall rules was based on this excellent instruction video from The Hook Up. He goes into great depth on how to segment your network, the appropriate firewall rules, and the whys behind all these settings.
There is a specific setting on the UDM-Pro that is important for the Bridge network. To make sure the Bridge network Gateway has a different IP than the CRS305, I used the IP address 192.168.88.10. This will be important as the CRS305 will need to point back at this address in its setup.
Here is what the Bridge VLAN settings look like.
The final Bridge VLAN network setup is with my UniFi Switch. On the UniFi Switch, Port 18 (SFP 2) has the Port Profile as “Bridge.” This is where we will plug the CRS305 into the network so we can access the web configuration tool.
Configuring the CRS305
With the CRS305, there are plenty of videos on how to configure the device fresh out of the box. I recommend this video from Crosstalk Solutions to get an understanding of the device and how to initially configure it. To make this work for my network, I kept the default IP address, set the Gateway to what we defined for the Bridge network, and added specific DNS Servers.
The CRS305 now works on the network through Port 18 on the UniFi Switch. Lastly, set the IP address to the bridge and not a specific port.
Configuring the LAG Connection
The CRS305 is now ready to be configured for the LAG connection. To start that work, I found a great Reddit post that describes Link Aggregation on Cable Modems. All the settings you need to bond the ports together are there. There are three steps to this process.
1 – Release the two ports you want to bond from the bridge. In my case, this was SFP+ Ports 3 and 4. This can be done from the Bridge->Ports page in the WebFig too by double-clicking on the specific port and setting Interface to “None” for the port.
2 – Define a new bonding interface with those ports together. It should look like this when you are done.
3 – Apply this new bonding interface to the bridge.
The CRS305 now has four interfaces consisting of one ethernet port, SFP+ 1 (for the UDM-Pro WAN port), SFP+ 2 (to the UDM-Pro LAN port), and a bonded port with SFP+ 3 and SFP+ 4.
Making the UDM-PRO WAN2 Function
The last piece of magic (and the hardest part) was getting the UDM-Pro to use the WAN2 (Port 10) for all internet traffic. We need to use the WAN2 port to take advantage of the 10 Gbps speed offered there. My first few attempts at this ended in a mess, including a hard reset of my UDM-Pro.
However, WAN2 by default is set up for failover when the primary ethernet WAN (Port 9) goes down.
So here is how I got WAN2 to function:
- Unplug the ethernet cord connecting the UDM-Pro’s WAN port to the CM1200 Cable Modem.
- The UDM-Pro should show a message on its screen telling you to plug in a cable.
- Connect an ethernet cord from the CM1200’s Port 1 to the CRS305’s ethernet port.
- Reset the CM1200.
- When the CM1200 starts to reset, plug the CRS305’s SFP+ 1 into the UDM-Pro’s WAN2 port. I did this using the fiber cable.
- When the CM1200 finishes its reset process, it will supply an IP address to the UDM-Pro’s WAN2 port.
- Internet connectivity should now exist, so run a test.
I did this process a couple of times to make sure I could switch between the WAN port and WAN2 port on the UDM-Pro. I noticed an odd thing in the UDM-Pro’s Internet Settings page, though. It showed the IP address as belonging to the WAN port, but it had the details of the service provider assigned to the WAN2 port. And when I looked at the Ports on the UDM-Pro, it clearly showed the WAN port as disconnected but WAN2 as active at 10.0 Gbps.
Activating the LAG Connection
At this point, the CRS305 is ready for the LAG connection to the CM1200. This requires accessing the NETGEAR’s Modem CM1200 configuration page at 192.168.100.1. From here you can navigate to the page to enable ethernet port aggregation.
Hit “Apply” to reset the modem with port aggregation enabled.
Now you remove the ethernet cable from the CRS305’s ethernet port and connect two ethernet cables from ports 1 and 2 of the CM1200 to the CRS305’s SFP+ 3 and SFP+ 4 ports. Traffic will flow over the bonded interface once the CM1200 has finished its reset. That’s it!
The Final Results
What kind of performance did we get after doing all that configuration? Are there other configurations that are better? What happens if we disconnect one of the ethernet cables in the CM1200 with LAG turned on?
To answer these questions, I ran a series of simple unscientific experiments to test six configurations. I ran the Speed Test found in the UniFi controller software five consecutive times for each configuration. Speed values are in Mbps. I took the results and looked at the Max and Average values for the download and upload speeds. Here is what I found:
As you can see, the LAG configuration gave us the highest download speed of 1.2 Gbps! However, it came with a big drop in upload speed of 16.6 Mbps. The upload speed hit is so large, it doesn’t seem like there is value in keeping this configuration. Pulling one of the ethernet cables while in LAG mode saw an expected drop in the download speed to the 800 Mbps realm. But the upload speed dropped by half to, in some cases, being in the 8 Mbps realm. Wow.
Clearly, the next step is to understand the “why” behind the upload penalty for a LAG configuration. Is there a configuration change that would improve this number? Through my current internet research, I found this problem exists with just about everyone who has set up a similar LAG configuration. And there is no solid answer on how to improve this.
Until I have a solution, I’ve decided to keep the CRS305 connected to the UDM-Pro’s WAN2 port, just not in a LAG configuration. I’m looking forward to going deeper into this problem space to learn if a solution exists.
Thanks for the post! You pointed me in the right direction. I have almost the exact same setup. I was able to achieve 1400 down / 46 up without issues and simply leaving the primary wan port disconnected.