OpenBSD and Failover NAT

I have a project that absolutely requires internet access, and Comcast isn't particularly good at doing things like SLAs or being particularly regular in the first place.

I've been lucky in that I haven't had any outages during the show, but that's luck more than anything. So I got a WiMAX modem from FreedomPOP to be a failover internet connection.

This is what Equal-Cost Multipath is for!

No, no it is not.

It is not for a simple reason– the links are not equal cost. The Comcast link is a business account that is essentially unmetered bandwidth, and FreedomPOP has a strict limit of 1GB per month before things get really expensive.

I could probably do a bit of PF tomfoolery to get it working the way I need, but nothing I tried seemed to do it, and frankly, I'd get worried.

What I need is to always go over Comcast, but when Comcast goes away, redirect the NAT traffic over FreedomPOP, and then recover when things get better.

I've got static IPs, but when this happens, they can suck it. The most critical thing is my phones, and I can lose those.

Assumptions

You know what OpenBSD is, you know how PF works, and you know how to get NAT setup without having a stroke. You are not going to email, call, or otherwise fidget at me about this if you can't get it to work for you.

Shut up already! How do you do it?

First things first, lose /etc/mygate.

Make dhclient not overwrite DNS or gateway info

The WiMAX modem won't route addresses that aren't handed out by it's DHCP server, so we need to talk DHCP without having it completely bung up routing. It hands out 192.168.15/24 addresses1).

Basically, do this to /etc/dhclient.conf as the only line:

request subnet-mask, broadcast-address, time-offset, host-name, lpr-servers, ntp-servers;

I could probably stand to lose more of those, but I'm making minimal changes here.

Setting up Interfaces

Fuck Yeah, Routing!

Something I didn't need before and only vaguely remembered, you can specify multiple routes to the same address space with different priorities. Priorities are integers, with the lower the int, the higher the priority. If the lower int interface is deactivated or loses link, traffic is automatically moved to the next lower int router.

To specify that, you'll need to call route in your hostname.if file. Lines prepended with a ! are executed as commands.

This doesn't manage problems farther afield– there's an daemon called ifstated that handles that. The other thing with routing is that we need a way to check to see if interfaces are back up. The way I'm doing that is by statically routing IPs to known good public DNS servers over different links. That's what the other route commands are for.

So, here are the hostname.if files:

hostname.comcast0
inet 10.2.2.236 255.255.255.0 10.2.2.239 description "Comcast Interface"
!route add default -priority 10 10.2.2.238
!route add -host 4.2.2.1 10.2.2.238
!route add -host 8.8.8.8 10.2.2.238
hostname.fpop0
dhcp
!ifconfig fpop0 description "FreedomPOP Backup Interface"
!route add default -priority 12 192.168.15.1
!route add -host 4.2.2.2 192.168.15.1
!route add -host 8.8.4.4 192.168.15.1
hostname.int0
inet 10.32.0.1 255.255.255.0 10.32.0.255 description "NAT Interface"

Yeah, yeah, but for completeness.

Firewall Rules

This is pretty basic, but you can include multiple nat-to lines so traffic going out those interfaces are automatically translated. Here's a really basic, far-from-complete pf.conf:

pf.conf
ext_if=comcast0
int_if=int0

bkp_if=fpop0

int_net="10.32.0.0/24"

nat_addr="10.2.2.236"


set block-policy drop

pass out on $ext_if from $int_net nat-to $nat_addr 
pass out on $bkp_if from $int_net nat-to ( $bkp_if )

Seriously, you're going to want something that looks better than this, but this is enough for a proof-of-concept.

Watching Changes Further Afield: ifstated

ifstated watches for network changes, and then does things in response to those changes. We can watch the state of comcast0 by pinging one of the publicly available DNS servers we static routed over it. If it goes down, it'll drop the comcast0 route and touch an approprate state file. When it comes back, it'll put the default route back and touch other appropriate state files.

ifstated.conf
init-state start

comcast_check = '( "ping -q -c 3 -i 1 -w 1 4.2.2.1 > /dev/null" every 10 )'

state start {
        if $comcast_check {
              set-state comcast_online
        }
        if !$comcast_check {
              set-state comcast_offline
        }
}

state comcast_online {
	init {
		run "route add default -priority 10 10.2.2.238"
		run "rm /tmp/USING_BACKUP_ROUTE"
		run "touch /tmp/USING_MAIN_ROUTE"
	}
	if !$comcast_check {
		set-state comcast_offline
	}
}

state comcast_offline {
	init {
		run "route delete default -priority 10 10.2.2.238"
		run "rm /tmp/USING_MAIN_ROUTE"
		run "touch /tmp/USING_BACKUP_ROUTE"
	}
	if $comcast_check {
		set-state comcast_online
	}
}

every 10 means every ten seconds. You probably want to do less than that, but in my case, ten seconds of downtime during a show is about the longest I can stand before stations go to offline content.

Still Working on This

Among other things, I wouldn't mind a way to simply prevent the route from repairing. If we kick to the secondary connection during a show, I pretty much want to stay there until the end.

This Documentation Sucks

Yes, yes it does.

1)
Screw you! I only route over DOUBLE NAT!