Sponsored Links

OpenMPTCProuter anyone?

dabigm

ULTIMATE Member
I had been thinking of doing this for a while. I have two connections at the moment, I can have 3 if I want. Currently im doing load balancing via an edgerouter x and when you're a family of 5 that works pretty well, connections get spread across the available links equally.

Of course there are two issues with this setup

1. Failover is noticeable. A failing link will kill the connections using that link. That of course means that if you're in a zoom conference, or SSH'd somewhere or doing a download and the link fails then it's going to cut you off.

2. Links are not aggregated. They are round-robin'd. So as mentioned this works when you have several people using it (or you use peer-to-peer downloading).

OpenMPTCPRouter claims to solve both of those issues by using multi-point VPNs to a location in a data centre somewhere with a fat pipe at the end.

So, is anyone doing this setup ? What have you found out about it by running it? Is it true that it will only operate at the speed of the lowest link? Will it aggregate uploads or not ?
 
It utilises MPTCP, which is a mechanism for TCP to take multiple paths. It was initially designed with datacentres in mind where a link to a switch may die and things just "continue as normal". It's been used for a while in iOS (for Facetime) and is in the Linux kernel nowadays.

Your traffic is then encapsulated in a VPN, with MPTCP packets which can span all your interfaces.

Failover *may* be noticeable, as some traffic will need to go through the TCP retransmit mechanism if it gets lost. But ultimately, all data will get from A to B.

It will aggregate the links, correct. I tried this specific project years ago and it was just a pain to get working right but it looks like it's matured quite a bit now.

The one thing I'd keep in mind is that things like Netflix are going to have a bad time when they realise you're using a data centre IP. You could get around this with some DNS trickery and a local SNI proxy that doesn't take the MPTCP route of course. Just something to keep in mind.

In terms of where to run the other end of the VPN, I'd look at Linode, as they are very flexible with network interfaces, and they allow you to increase bandwidth in increments of a gbit.
 
It utilises MPTCP, which is a mechanism for TCP to take multiple paths. It was initially designed with datacentres in mind where a link to a switch may die and things just "continue as normal". It's been used for a while in iOS (for Facetime) and is in the Linux kernel nowadays.

Your traffic is then encapsulated in a VPN, with MPTCP packets which can span all your interfaces.

Failover *may* be noticeable, as some traffic will need to go through the TCP retransmit mechanism if it gets lost. But ultimately, all data will get from A to B.

It will aggregate the links, correct. I tried this specific project years ago and it was just a pain to get working right but it looks like it's matured quite a bit now.

The one thing I'd keep in mind is that things like Netflix are going to have a bad time when they realise you're using a data centre IP. You could get around this with some DNS trickery and a local SNI proxy that doesn't take the MPTCP route of course. Just something to keep in mind.

In terms of where to run the other end of the VPN, I'd look at Linode, as they are very flexible with network interfaces, and they allow you to increase bandwidth in increments of a gbit.
Looking at linode yeah. Also looking at splitting a dedicated box from Hetzner with a friend who also wants a box in a data centre. Thinking to carve it up with two KVM VMs.

It's interesting you mention that it was a pain to get working. The demos of it look really simple, one YT video did the setup live in under 30 mins. But I also read several people asking for help that it doesn't work / doesn't work as expected.

Not too bothered about streaming (I barely use them) and they have a bypass method built in now (it will pick the requested link to do it rather than send it via the VPN)
 
Sponsored Links
Looking at linode yeah. Also looking at splitting a dedicated box from Hetzner with a friend who also wants a box in a data centre. Thinking to carve it up with two KVM VMs.
Keep in mind this is probably only a gbit uplink?
It's interesting you mention that it was a pain to get working. The demos of it look really simple, one YT video did the setup live in under 30 mins. But I also read several people asking for help that it doesn't work / doesn't work as expected.
This was very early days, I found it when trying to setup the same kind of tunnel (TCP proxy over MPTCP).

Not too bothered about streaming (I barely use them) and they have a bypass method built in now (it will pick the requested link to do it rather than send it via the VPN)
Oh neat!
 
Yeah Hetzner are 1Gig link unlimited traffic. Since i'll probably be hitting that occasionally it's probably better to look for 2gig. OVH has 10Gig ones but the prices reflect it.
 
@dabigm been toying with the idea for a while, but ultimately laziness won.

Going for hetzner/ovh might complicate your life as you'll start getting served german language content on various sites or services.

Try a UK based data centre.
 
Yeah Hetzner are 1Gig link unlimited traffic. Since i'll probably be hitting that occasionally it's probably better to look for 2gig. OVH has 10Gig ones but the prices reflect it.

Each tier adds another gbit in. I don't know how CPU intensive the VPN software is though? Also UK based.

Let us know how it goes! I am curious how much it really has matured.
 
Sponsored Links

Each tier adds another gbit in. I don't know how CPU intensive the VPN software is though? Also UK based.

Let us know how it goes! I am curious how much it really has matured.
interesting. but the data transfer caps seem quite small (I guess they're not for a VPS, but they are for a VPN).
 
Ok I was asked if I could do a basic write up on OMR, instead of mess up the thread about IONOS ill write it up here.

I'm assuming you are using a VM for it, like in proxmox or vmware etc. and that you use VLANs for your ISPs / ISP routers. You will need

1 x VLAN for OMR's "LAN"
1 VLAN per ISP

I'm also assuming your ISP routers use DHCP but you can adjust as necessary

Start by downloading the ISO / image from here

at the time of writing this is the one you probably want. download and unzip this


then upload it to your hypervisor, in my case, proxmox


In proxmox
==========
Create a VM with host CPU, 2GB RAM, No disk, required networks

download the x86 vm disk, and gunzip it. then import it into proxmox and add it to the new VM

qm importdisk <ID OF VM e.g. 100> openmptcprouter-v0.60-6.1-r0+24843-acf40c022e-x86-64-generic-ext4-combined.img local-lvm

on the GUI you should now see an "unused" disk attached to the vm. Edit it, and ensure the SCSI BUS is set properly. You can also use VIRTIO. You might also want to click discard if you have an SSD.
you should now be able to boot it. If you can't boot it, go into options and check the "boot order". launch another vm on the "lan" (or attach a machine to the same VLAN as you picked for OMR (first) interface you gave the openmptcprouter VM.

open a browser to 192.168.100.1 login as root with no pw.

You now need a VPS. You can use something like Ionos, or OVH. I suggest two cores and 2GB minimum. I am currently using the Ionos "medium" sized VPS (2 cores, 4GB, 160GB HDD)

I suggest you install latest debian (12 I think)

Then login to the VPS and run this command (it's going to take a long time)

Code:
apt-get update && apt-get upgrade
wget -O - https://www.openmptcprouter.com/server/debian-x86_64.sh | KERNEL="6.1" sh

When done, you will get an output with tons of codes and strings of stuff, don't worry you only need the "Openmptcprouter server key". It will be a string of random charachters like

Code:
OpenMPTCProuter Server key (you need OpenMPTCProuter >= 0.42):
3AF5A255535AFB38D928B3DA026696B0722F9E801F4C9DF3FC81A0543EB56314
1724181829647.webp

Your keys are saved in /root/openmptcprouter_config.txt btw. so don't worry if you rebooted and forgot to save it.

go to the browser on the OMR LAN network

go to System > OpenMPTCPRouter

paste in the key you got above.
check the key! at least check the start byte and end byte.

leave LAN interface alone

for me, i set "protocol" to "DHCP" for my wan interfaces

VPS must be rebooted after installation.!!!!!! (new kernel, reboot required!)

The ports on your server will be changed. For example, SSH will now be port 65222

Bypasses:

Since you will be routing all your data via a VPN now (the VPS) you might find some places block you. You can bypass the VPN by going to Services > OMR Bypass and enter a domain. You can pick an interface (ISP) or just leave it blank.

I also recommend you turn on the "TCP Low Latency" option.

Lastly, you should tell OMR which interface is your "Master" interface - I would set this to your most reliable ISP. Example if you have FTTP and 5G then set FTTP to "master" and others to "Enabled" .

1724181917349.webp
 
What my VM in proxmox looks like

1724182353579.webp


Vlan tag 16 = my "OMR LAN"
Vlan tag 14 = my FTTP
Vlan tag 15 = my 5G Router

if it doesn't boot check the "boot order"

1724182406835.webp
 
Sponsored Links
I should also mention if you want to use the bypass method, then OMR needs to be your DNS server. You can either have your DHCP server give OMR as the DNS IP, or you can have that as the upstream to your existing one. For example, I use adguard home, and in adguard i have the upstream DNS IP set to 192.168.100.1

You should be able to configure your existing router to join the OMR VLAN and route clients from your regular LAN to use OMR gateway as your gateway too.
 
Thanks, that's really useful. I'm going to set up a new proxmox box with multiple NICs soon. How do you configure your main router to route Internet traffic out onto VLAN 16? Also what are your ISPs? I have one Openreach FTTC and one Huawei 5G router currently connected to a Ubiquiti Unifi USG 3P
 
Thanks, that's really useful. I'm going to set up a new proxmox box with multiple NICs soon. How do you configure your main router to route Internet traffic out onto VLAN 16? Also what are your ISPs? I have one Openreach FTTC and one Huawei 5G router currently connected to a Ubiquiti Unifi USG 3P
so i have vodafone FTTP and three 5G.
set your main router (your Unifi) to have an interface on the same VLAN as the OMR "LAN" VLAN (the first interface you give OMR will be the LAN) then you just set that as your default gateway. In OPNSense you would configure a firewall rule to send traffic from (wherever) your LAN to use that gateway. On my Mikrotik I have to set a routing mark to use the routing table that this interface uses. I'm not sure how this is configured on the Unifi as I don't have one, but presumably if you have more than one WAN (OMR being one of your WANs) you do it the same way, just route traffic to that as your gateway.
 
I need to do PPPoE authentication on the FTTC line. Can I do that in the Proxmox VM? And am I right in saying I don't actually need another two port NIC on the server as these connections will be running on VLANs and the Proxmox hypervisor is VLAN aware?

Answering my own questions a little - I've just spun up the VM and had a look at the web interface. If I am reading it all right I can set two ports on my switch to VLAN 14 and 15 (using your config so I don't confuse myself), plug the internet connections into those, then either set a third port to VLAN 16 and connect that to WAN1 on the Unifi USG, or leave it unplugged and set the connection to use VLAN 16 through the LAN connection (this is the bit I'm not sure about until I have a play around with it).

Either way, I can't have a proper look at this until the weekend (work and kids need the internet so can't go reconfiguring things right now).
 
Last edited:
You should be able to do the PPPoE authentication just fine inside OMR on a VM yes. You could also just put the OMR router on to your existing LAN that points to that WAN.

Example lets say you've got your FTTC and existing router setup like this

FTTC modem -> Ubiquiti Router -> LAN

You could make the OMR "WAN" be on the "LAN" of the Ubiquiti so like this

FTTC modem -> Ubiquiti Router -> LAN <- OMR "WAN".


This way you could do some fun routing like send some clients via OMR and others via the regular WAN. Up to you but a few ways to skin this cat. Or if you're happy to use OMR for everything (and use bypass for specific domains) then connect the FTTC direct to the OMR VM and do the PPPoE there. You just need to change it from DHCP to PPPoE and provide the details.

1724255225012.webp


and yeah you don't need multiple ports, just one will do because your hypervisor vswitch is vlan aware as you say. Perfectly fine to do this as a "router on a stick" model yep.
 
Sponsored Links
Thanks again. I've had a go at this and have got the VPS and the 5G connection going but I cannot get the VM to connect to the PPPoE FTTC connection. I think its because it doesn't like the Openreach FTTC modem connecting over a VLAN to the Proxmox VM. So I've got a cheap USB NIC on order from Amazon so I can test that tomorrow. If this all works (and specifically works well - i.e. makes proper use of the upload speed on both my connections) then I'll buy some hardware like a router or Pi to install this on permanently.
 
yeah odd, if it's a VM in proxmox, the VM shouldn't even know it's on a VLAN, to the VM it's just another ethernet interface. I wonder if it might be MTU related. I terminate the PPPoE via my main router, and then use the "lan" out from my router as a VLAN for OMR, I haven't tried doing the PPP on OMR itself. But let us know how it goes.
 
I think I'm getting closer with the USB NIC. Now I'm having problems connecting to the vps. Do you use the VPN tunnel on your set up? There's an option 'MPTCP over VPN' but I don't know if this should be on of off.
 
i don't use that option no. i don't think it even works to be quite honest. You shouldn't need it no. What sort of connection problems? were you able to run the script on the VPS and reboot ? can you ssh back in after? (bearing in mind the port changed)
 
Top
Cheap BIG ISPs for 100Mbps+
Community Fibre UK ISP Logo
150Mbps
Gift: None
Virgin Media UK ISP Logo
Virgin Media £22.99
132Mbps
Gift: None
Vodafone UK ISP Logo
Vodafone £24.00 - 26.00
150Mbps
Gift: None
NOW UK ISP Logo
NOW £24.00
100Mbps
Gift: None
Plusnet UK ISP Logo
Plusnet £25.99
145Mbps
Gift: £50 Reward Card
Large Availability | View All
Cheapest ISPs for 100Mbps+
Gigaclear UK ISP Logo
Gigaclear £17.00
200Mbps
Gift: None
Community Fibre UK ISP Logo
150Mbps
Gift: None
Virgin Media UK ISP Logo
Virgin Media £22.99
132Mbps
Gift: None
Hey! Broadband UK ISP Logo
150Mbps
Gift: None
Youfibre UK ISP Logo
Youfibre £23.99
150Mbps
Gift: None
Large Availability | View All
Sponsored Links
The Top 15 Category Tags
  1. FTTP (6024)
  2. BT (3638)
  3. Politics (2720)
  4. Business (2439)
  5. Openreach (2405)
  6. Building Digital UK (2330)
  7. Mobile Broadband (2143)
  8. FTTC (2083)
  9. Statistics (1899)
  10. 4G (1813)
  11. Virgin Media (1762)
  12. Ofcom Regulation (1582)
  13. Fibre Optic (1467)
  14. Wireless Internet (1462)
  15. 5G (1404)
Sponsored

Copyright © 1999 to Present - ISPreview.co.uk - All Rights Reserved - Terms  ,  Privacy and Cookie Policy  ,  Links  ,  Website Rules