netfilter NAT vs. pump

[Werner Almesberger <almesber@xxxxxxxxxxx>]

I've come across a rather nasty problem: if I configure my 2.4.0test1*
kernel without "Full NAT", DHCP works as expected. If I add NAT support,
DHCP fails miserably. Here are the gory details:

Setup:
 - I have a box running 2.4.0test1-ac17 (also happens with older kernels,
   at least including test1-ac8) with RedHat 6.2
 - this machine has two Ethernet interfaces, eth0 and eth1.
 - eth0 points to an internal network and has static IP address 10.0.0.1
 - eth1 points to my cable modem and gets its IP address via DHCP
 - I have netfilter's "Full NAT" (CONFIG_IP_NF_NAT) enabled

The problem:
 - pump (RedHat's DHCP client) fails to configure eth1
 - I see DHCP requests on eth0, i.e. on the wrong interface

I've chased this through the kernel, and the reasons for this are as
follows:
 - whenever NAT is enabled, even if it's doing absolutely nothing, it
   sets NFC_ALTERED in skb->nfcache
   (net/ipv4/netfilter/ip_nat_standalone.c:ip_nat_fn)
 - this eventually leads to the packet being re-routed
   (linux/net/ipv4/ip_output.c:route_me_harder)
 - BEFORE the re-routing, the interface is eth1, as expected, destination
   IP is 255.255.255.255 (okay), and source IP is 10.0.0.1 (not nice, but
   probably doesn't really matter to DHCP)
 - AFTER re-routing, the interface becomes eth0, because the source IP
   was used to determine the interface. Very bad.
 - pump doesn't explicitly set the interface (it binds to { 0.0.0.0,68}),
   but eth1 gets picked for some lucky reason anyway
 - in the call to ip_route_output in net/ipv4/udp.c:udp_sendmsg, the
   source is 0.0.0.0, but afterwards, the 10.0.0.1 from rt->rt_src
   becomes the new source

Fixing this is a little tricky, because it seems that several parties
are at fault:
 1) I'm not sure why ip_route_output returns such a strange
    interface/address combination (didn't look up the details since it
    happens to err on the convenient side)
 2) as shown, route_me_harder doesn't necessarily perform the same
    operation as the original routing. This is a problem, because the
    system behaves differently with and without NAT support, even if
    nothing else changes. I see three possible approaches to correct
    this:
     - make it happen less often by setting NFC_ALTERED only when
       something has changed (probably a good idea in any case)
     - copy all the route selection subtleties (including multicast,
       etc.) into route_me_harder
     - declare any application that fails because of this as broken ;-)
       (probably a reasonable approach, although it breaks backward-
       compatibility)
 3) pump really ought to be a little more explicit about that interface
    (BTW, dhcpcd apparently has the same problem, but I didn't examine
    that one any further)

I've attached a patch to pump that seems to avoid the problem. At least
it has obtained my IP address and kept it for the last half hour.

Does anybody know who's taking care of pump ? I've only found the SRPM,
and it is remarkably devoid of any hint to its origin :-(

- Werner

---------------------------------- cut here -----------------------------------

--- pump-0.7.8-orig/dhcp.c      Tue Feb 15 21:59:11 2000
+++ pump-0.7.8/dhcp.c   Tue Jun 13 22:16:03 2000
@@ -940,6 +940,13 @@
     return s;
 }
 
+
+static int bind_to_device(int s,const char *itf)
+{
+    return setsockopt(s,SOL_SOCKET,SO_BINDTODEVICE,itf,strlen(itf));
+}
+
+
 int pumpDhcpRelease(struct pumpNetIntf * intf) {
     struct bootpRequest breq, bresp;
     unsigned char messageType;
@@ -957,6 +964,8 @@
 
     if ((s = createSocket()) < 0) return 1;
 
+    if (bind_to_device(s,intf->device) < 0) return 1;
+
     if ((chptr = prepareRequest(&breq, s, intf->device, pumpUptime()))) {
        close(s);
        while (1) {
@@ -1020,6 +1029,8 @@
 
     s = createSocket();
 
+    if (bind_to_device(s,intf->device) < 0) return 1;
+
     if ((chptr = prepareRequest(&breq, s, intf->device, pumpUptime()))) {
        close(s);
        while (1);      /* problem */
@@ -1147,6 +1158,12 @@
        pumpDisableInterface(intf->device);
        close(s);
        return perrorstr("bind");
+    }
+
+    if (bind_to_device(s,intf->device) < 0) {
+       pumpDisableInterface(intf->device);
+       close(s);
+       return perrorstr("SO_BINDTODEVICE");
     }
 
     serverAddr.sin_family = AF_INET;

-- 
  _________________________________________________________________________
 / Werner Almesberger, ICA, EPFL, CH       werner.almesberger@xxxxxxxxxxx /
/_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/