Linux has very powerful routing capabilities, and it would be a waste not to leverage them. If you also happen to have two ISPs, you can combine those routing capabilities to obtain an highly resilient and maybe even load-balanced home network.

Let's assume you have two upstream connections (for example, one could be a cell phone link, only for emergencies) it would be great to be able to:

• Switch between the two ISPs when one goes down
• Route access to specific servers trough ISP1 or ISP2
• Route specific services trough ISP1 or ISP2
• Load-balance your traffic

Having two ISPs is important for redundancy. When you start to rely on your home services for your everyday life you want them to be always accessible, so if ISP1 goes down switch to ISP2.

If your ISP1 is, for example, much faster but with a data-cap, while ISP2 is slower, but with unlimited data? It would be great to route all traffic trough ISP1, but some apps (like usenet or torrent) trough ISP2…

More over, you will want to set-up two SSH tunnels one trough ISP1 and one trough ISP2 so in any case you have remote access.

To achieve this you need to operate on two levels:

• At NAT level to set specific rules for packet filtering & modification inside the kernel
• At route level, because packets need to be properly routed outside

I will assume ISP1 is your default gateway, and you can have only one default route. The basic idea is that if i want to reach external-server2 via ISP2, i need to add one route rule and one nft rule.

The nft rule will instruct the NAT to send any request from the internal network trough ISP2 interface and not the default gateway. Without this, 77.77.77.77 will not be reachable from the internal network.

The route rule will make sure that 77.77.77.77 is accessed trough ISP2 and not ISP1. This will work only for the home server, unless the nft rule is also applied this is because our NAT goes trough the ISP1, but home server route for 77.77.77.77 goes trough ISP2, making that IP address unreachable for devices on the home network.

nft add rule nat postrouting oifname "enp59s0u2u4c2" ip daddr 77.77.77.77 snat to 192.168.1.254
ip route add 77.77.77.77 via 192.168.1.254 dev enp59s0u2u4c2

While it is possible to route individual apps via specific interfaces, this is a bit convoluted (implies using network namespaces or netns) and each single executable needs to be started by root, which is a bit complex to setup.

Luckly, there is a much better way. Since we will setup each service running under it's own user, it's simpler to apply custom routing tables to specific users.

The idea is to create a custom routing table which has the desired interface/gateway different from the default one, then create a rule to force packets created by a specific range of UIDs (user IDs) to go trough that one custom table.

The tricky part is the return packet. Since when packets gets into your interface they will not have any user-id associated with them, they will be destroyed by the kernel that will assume a DDoS attach via IP spoofing. Luckly for you, this can be easily fixed by enabling loose mode.

This is probably not the best solution as it opens your home server to possible IP spoofing attacks, but i believe the risk is limited as those packets would need to pass trough your ISP gateway and internal networks first. Anyway when i will come up with a better solution (probably using conntrack and by marking outgoing packets) i will update this page.

Start with enabling loose mode on the target interface for ISP2:

echo 2 > /proc/sys/net/ipv4/conf/enp59s0u2u4c2/rp_filter

Then create the new routing table with the default gateway:

ip route add default via 192.168.1.254 dev enp59s0u2u4c2 table 100

this will create a new table, called 100 (names are not allowed, only numbers here), which has ISP2 as default gateway and interface. The table 100 will be automatically created.

To check what's in the table:

ip route show table 100

Then add the rule to route packets from user with UID 1000 to the table 100:

ip rule add uidrange 1000-1000 lookup 100

From this moment, user 1000 packets will all be routed trough ISP2 even if ISP1 is the default gateway for your home network!

routes setup script

Now you can access internet safely from the home network.

To learn how to reach the internal server from the internet, head to the SSH tunnel description