Stochastic Ideations

Attempting to move to Sonic.net proved to be more challenging than expected. In particular, all of my hosts that had static IP addresses behind my firewall wouldn't work. This should be a pretty conventional setup for most business users. It turned out to be an issue with ARP that Sonic doesn't seem to want to admit is a problem. I finally got it to work in an amazingly non-obvious way. This post is to help out you folks who are trying to do the same.

Follow up:

My Environment

I've got what is probably an unusually complicated network for a home, but not for a business user. Three networks live behind the firewall, which I'll call LAN, DMZ, and WLS (wireless). These are mutually firewalled from one another. My router is a Soekris Net4801 box running m0n0wall. The static addresses route to the internal network using 1:1 NAT and Proxy ARP. For example, an external address of 64.109.217.5 might NAT to the internal address 192.168.4.5, and the 1:1 nature means that all outbound WAN traffic will go using the external address. This is important: there are multiple static IP addresses all of which are using one interface, as far as Sonic is concerned.

The way this is supposed to work is relatively simple: when a machine with a static IP wants to send outgoing traffic it routes to the firewall, which changes the address to the external address and sends it on the WAN interface. When the network wants to send some traffic back it uses ARP to determine the ethernet address that corresponds to the IP address and then sends to that interface with the original IP address; the firewall converts that back to the internal address and forwards on the proper interface.

ARP is a simple protocol. Each host has a table mapping 32-bit IPv4 addresses to 48-bit ethernet addresses. In principle, every ethernet interface in the known universe has a unique address. To populate this table, a sending host sends out an ethernet broadcast packet (this is below the IP level) saying "who-has <IP>", i.e., which interface understands that IP address. That interface responds with "is-at <ethernet>" and communication commences.

It turns out that interfaces can also spontaneously send out ARP packets; in particular, a "who-has" request can be used when an interface is starting up to make sure that the IP address isn't already in use, and an "is-at" response can be broadcast to announce to the world that a particular interface is interested in a particular IP address. This is called "Gratuitous ARP".

The Problem

In a nutshell, the problem is that Sonic isn't sending ARP "who-has" requests. It isn't clear to me if it never sends them, or only sends them under some circumstances. Hosts with static IP addresses that are connected directly to the DSL modem work, possibly because they send Gratuitous ARP "is-at" packets when they are connected. But hosts behind a firewall don't do this, so the only address that is visible as far as Sonic is concerned is the address of the WAN interface on the firewall.

Lest you be saying "no, you must be wrong", I have the ethernet packet traces to prove it. Sonic didn't even want to look at them.

My Hack, a.k.a. The Solution

I won't bore you with everything I tried, which was considerable. I worked on this nearly full time for a week. So, here's the trick:

• Install an ethernet hub between the DSL modem and the firewall. Note that this has to be a true hub, not a switch. Nearly everything you find these days is a switch. Hubs broadcast all traffic to all of their interfaces. Switches learn which ethernet addresses are on which of their interfaces and only send to the correct interface. Hubs fell out of fashion because they have higher contention than switches.
• Install the arping utility if you do not already have it. It's in the FreeBSD ports collection (net/arping).
• Connect an interface on another machine to the hub. In my case, I had a spare port on another Soekris box that is serving as my web server. Note: you do not want this to be an active interface. It shouldn't even have an IP address.
• Set both interfaces attached to the hub the same ethernet address. It doesn't matter whether you change the address on the firewall or on your helper box, as long as they are the same. At least on BSD you can do this using the ifconfig command.
• Set up a cron job to run the attached script periodically. I'm currently running it once every ten minutes, but that may need tuning. Briefly, that script configures the interface up, runs arping once for each static IP address, and then configures the interface down again.

The basic idea is to force Gratuitous ARP packets which appear to come from the interface on the firewall. This tricks Sonic into properly updating its ARP tables so that it knows about the static addresses.

I hope this saves someone a really torturous week.