🎬 Intro #
I’ve recently been looking to move all my firewalls from pfSense VMs and Mikrotik hardware to OPNsense. Virtual pfSense has been great for me over many, many years and all the benefits a VM brings to a firewall have helped a lot, such as hypervisor level HA and snapshots. These days, however, I am leaning more towards physical installs, especially with the ridiculous amounts of cheap, powerful mini PCs available today.
The drive to move off pfSense has been a long time coming, I won’t go too deep into this here but OPNsense seems like the better FOSS solution. It is mature enough for me, has all the packages I require, and is more frequently updated than pfSense.
This post, however, will be about replacing a Mikrotik RB5009UG+S+ I have managing a remote site, my main homelab site. I actually really like the firewall, it does everything I need and was a great price for the hardware. One of the reasons for picking it was because I needed something cheap with SFP+ compatibility and fast routing speeds as the WAN link is 3Gb/3Gb.
Initially, I was using a UDM Pro as I wanted something that ‘just worked’. After getting a root shell on the thing and installing FRR on it for BGP, I was met with nothing but issues. After trying to speak with Ubiquiti support I got the standard “you’ve installed stuff on it, we don’t support that, wipe is pls” response which is fair enough, but I needed that functionality for my Wireguard tunnels with BGP, something which required a fair bit of unsupported config at the time.
I ended up replacing it with the Mikrotik as the stock UDM software is, in my opinion, a toy.
As previous, the Mikrotik has been working well, and I was planning to roll out Mikrotik firewalls to all my sites but the need to try the new hotness that is OPNsense is too great, so instead I’ve decided to put a physical OPNsense install here, and move the Mikrotik to my parents.
✅ Requirements #
The requirements are as follows:
- Low power - Around 20w would good.
- Smol - I need this to fit inside a small network cab on a wall, so it needs to be small.
- 10Gb/s capable - I don’t expect to route at 10Gb/s, but I do at least need to route 3Gb/s to satisfy my WAN link.
🤔 Options #
Now, there are a lot of options if you don’t need to route at over 2.5Gb. The market is currently flooded with cheap mini PCs with very capable hardware but all of these options have a cap of 2.5Gb ports. For most people, this is fine, but it would be beyond foolish to not use the entire 3Gb pipe that I pay for.
The main contender is the R86S machine which is exceptional value and fit for purpose. This thing is pretty insane and is a great example of the kinds of hardware that’s available now but it does come at a price. £300 is a good price for something like this don’t get me wrong, but I could do something similar for 1/3 of the price so I just did that instead…
Any modern processor can route decently fast and with hardware offload on decent NICs 10Gb, or close enough to it, shouldn’t be an issue. The issue is finding a machine that can fit a card in.
The M720q is one of the few SFF machines that can accept a PCIE card. There are other Lenovo machines that can take cards too but the M720q is older and half the price of the other options on that list, a Coffee Lake 8th Gen CPU is just fine anyway.
Another option at a similar price point was the HP T730 but this is much older and larger so not really worth considering.
🗿 Lenovo M720q #
The M720q is a well-designed SFF machine. One screw at the back removes both the top and bottom cover.
My machine shipped with a SATA SSD which is pointless to me as this space needs to be taken up by the PCIE card.
Here’s a shot of the motherboard with the SSD and caddy removed.
I got this machine for £70 which is especially cheap because one of the USB ports on the rear was faulty, I probably would never even know. The seller has labelled this. Above the USB ports is the expansion slot which is currently filled with a blanking plate. This is easily removed with two screws, one visable in the image and another around the corner of the machine.
The bottom of the machine has a panel that slides out which houses the dual channel memory slots and an NVME slot. This is where I slapped in one of the many cheapo NVME drives I usually buy on sale for these applications.
The memory configuration is 1 stick of DDR4 4GB, dual channel would, of course, be superior but I don’t see this being an issue for a router.
Now I’m just left with this thing which I will never throw out just in case I ever need it (I won’t). 🙃
⬆️ Risers #
The non-standard PCIE slot of the M720q is a bit of a pain. The slot on the motherboard is proprietary, but a riser makes it standard. There are x4, x8, and x16 risers, electrically, however, it’s only x8. There seem to be a few options:
| PN | BUS |
|---|---|
| 01AJ929 | x4 |
| 01AJ902 | x8 |
| 01AJ940 | x16 |
I have read that the x8 risers have some issues, and it’s better to go with the other options but I don’t know how true this is. I just got whatever was cheapest and most readily available, which was the 01AJ940.
A PCIE Gen3 x4 slot has a theoretical throughput of 31.5 Gbps so even if both 10Gb ports were pulling line rate which is impossible with my 3Gb WAN, so a x4 slot is more than enough.
🏃♂️ Fitting 10GbE #
Yeah so, this won’t work.
Turns out I didn’t even need to 3D design a bracket, there are models already out there. I printed this model from thingiverse that covers the slot of the M720q and also allows for one of the screws to be installed to keep it moving, neat.
The final product isn’t 100% perfect but it is 100% acceptable and much better than just having it chilling in there with no support.
💀 I Hate Computers #
I had this thing on my desk for weeks, tinkering every now and then with the OPNsense config and everything was working fine all throughout this. The goal was to have the setup exactly as I would need it so as to simplify my migration when I finally visited my offsite lab. I wanted to unplug the Mikrotik there, plug this thing in and have everything just work.
If you work in IT though, you know full well that is never the case. For weeks everything had been working just fine and it had survived multiple reboots, updates etc but when it came to boot up at the site, all of a sudden one of the 10Gb/s interfaces was just …gone. The card was being detected fine, but nothing I did would allow the OS to see a second 10Gb/s port.
After many reboots and head scratching, I decided to just plug my WAN link into my 10Gb/s switch and use the single interface that was appearing to trunk all VLANs, including the WAN. this is how it has been for several months now and it’s working great.
Of course, a few days later I had to do a remote reboot and all of a sudden the second NIC magically appeared. It’s been there ever since.
ix0: flags=28963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,NOMAP>
ether 90:e2:ba:f3:29:d4
inet6 fe80::92e2:baff:fef3:29d4%ix0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
ix1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,NOMAP>
ether 90:e2:ba:f3:29:d5
media: Ethernet autoselect
status: no carrier
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
I do plan on moving my WAN into the second port so that the only thing I am reliant on to get into the site remotely is the OPNsense box itself, and the ISP fibre equipment. I am slightly worried that one day the same might happen though.
Anyway, this is entirely an OPNsense thing and not an issue with the hardware, I just wanted to vent some frustration.
🔌 Power Usage #
I’m happy with the power usage of this box. The x520 isn’t the most power efficient these days and will depend on what transceivers are used. Something like a Connectx-3 will be more power efficient, but will most likely come at a higher cost initially.
The power usage when booting up is about 24w but quickly settles down once booted, which is about 45~ seconds.
I will say that this is higher than the Mikrotik, which would idle at something stupid like 3-5w but this thing is an entire desktop x86 PC.
⛰️ Doubling Down #
Not very long after installing this machine into my offsite lab I’d decided that it was the perfect machine for other sites, like my parents. I know I said previously that the plan was to put the Mikrotik there, but I came across a good deal on a higher spec’d machine so thought why the heck not.
The setup at my parents is a virtual pfSense setup and when something goes wrong having a physical firewall for them to ‘unplug and replug’ is vastly superior to what I currently have to do.
My parents live in a pretty remote (for London) house and there is only VDSL there, despite my best efforts to try and get ISPs to fibre the area. Their internet speed is about 50Mb/20Mb so really anything would do, in fact those N100 passive machines would be perfect for this application but I decided to just replicate this setup as it was a similar cost.
I found the following on eBay:
My thinking here was that I would pinch the processor and memory in this machine and replace it with the one in my lab. This is exactly what I ended up doing making my lab OPNsense box run with an i5-9400T and 16GB of memory.
I have quite a few Intel 4 port PCIE cards and this machine was a perfect candidate. The fit was a lot more snug than the 10Gb/s card installation. I had to remove more from the machine but it did just fit. No fancy 3D printed bracket here, I just put some foam underneath the card to stop it touching the mainboard.
🚀 Performance #
The machine at this site now looks like this in terms of spec:
Without any IPS rules enabled, the router is able to route at the full speed of my WAN, which is exactly why I’ve gone down this route.
❯ speedtest
Speedtest by Ookla
Server: Community Fibre Limited - London (id: 30690)
ISP: Community Fibre Limited
Idle Latency: 0.74 ms (jitter: 0.03ms, low: 0.69ms, high: 0.77ms)
Download: 3142.90 Mbps (data used: 1.5 GB)
5.38 ms (jitter: 0.17ms, low: 0.94ms, high: 7.72ms)
Upload: 3218.88 Mbps (data used: 1.6 GB)
2.69 ms (jitter: 0.23ms, low: 0.95ms, high: 3.13ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/98ad2300-6687-4e88-aef3-603d6c7cd7a4
Inter-VLAN routing is not as fast as I would like, however:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.31 GBytes 4.84 Gbits/sec 292 sender
[ 5] 0.00-10.00 sec 5.30 GBytes 4.84 Gbits/sec receiver
My network design doesn’t require really high throughput between VLANs, as everything that needs fast access to a resource should be on the same L2 segment, and as long as I can route the full speed of my 3Gb WAN then this is okay.
Having said that, there are several tunables that can be applied to OPNsense to help improve performance, so at some point I would like to mess with the config and try to get this routing as fast as I can.
Under normal operation (without any IPS), the CPU spikes around 30% when throughput via the WAN is being pushed.
During high throughput situations, the CPU does get rather toasty, but the environment this machine is not in is not ideal. It’s currently summer so ambient is about 27c, the room my lab is in is not ventilated, and the M720q sits ontop of my 10Gb/s switch, which itself does get rather toasty.
You can see the CPU temp gets close to 70c here when data rate is high, but this is rarely sustained, if it were the CPU would just downclock after a while of course and 70c isn’t too bad considering.
🛡️ IPS #
I stupidly enabled all the IPS rules to ’test’ and no surprises, the CPU was pegged at 100% and routing speed dropped to around 200Mb/s. Obviously, enabling all the rules possible isn’t realistic but this was part of my reasoning to swap the CPUs between the boxes.
With 10 IPS rulesets enabled, equalling 17552 individual rules my WAN speed drops to about 2.3Gbp/s download and 1.7Gbp/s upload. CPU usage during this is around 70% whilst pushing traffic.
Inter-VLAN traffic however falls off a cliff.
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 758 MBytes 684 Mbits/sec 53 sender
[ 5] 0.00-10.00 sec 755 MBytes 682 Mbits/sec receiver
For this reason I currently do not have any IPS rules enabled. I’m not sure how to improve this at current as the CPU usage is not high but I’m sure there is a lot of tuning that I’ve not looked into yet that will greatly help.
As well as tunables, enabling hardware acceleration and even installing a newer NIC may help this.
💭 Final Thoughts #
So, is it good? I would certainly say so.
It’s able to do everything I need of it as a router/firewall without breaking a sweat, and most importantly my lab environment is able to make full use of the 3Gb/s WAN.
In the age of the N100 router, however, I’m not entirely sure this setup has much weight behind it bar niche setups like mine where 2.5Gb/s is not enough. While this M720q solution can come out cheaper, especially if you’re repurposing existing hardware, having a vastly newer platform with the N100 does make things difficult to justify in many cases.
The N100-based routers offer several advantages:
- More recent architecture with potentially better power efficiency
- Often come in purpose-built, compact form factors
- May have better out-of-the-box compatibility with modern networking features
Whereas, the M720q has its own advatages:
- Flexibility and upgradeability - easily swap CPUs, add RAM, or change network cards
- Potentially higher build quality from a major manufacturer
- Ability to handle speeds beyond 2.5Gb/s with the right network card
There’s an argument to be made that this machine is likely built better than most of the Chinese N100 boxes, but I wouldn’t know for sure.
The decision ultimately comes down to your specific needs, budget, and comfort level with different hardware platforms. For my use case, with a speedy-boii WAN and the need for flexibility, the M720q hits the sweet spot.
For many home users with gigabit or slower connections, an N100-based router is probably the more straightforward and future-proof choice.
Thanks for reading. Fin.
