In alpha2lamar 3 hop configuration : (As of 4/25/03) (i.e. alpha-beta-lamar. There is no direct LSP between alpha and lamar) The 'forwardcontrol=yes' paramter in the ipsec.conf files seemed to done the trick as far as being able to ping between the subnets. Test it again. Also, what might have helped is that the iptables were simpleified very mcu. The details of what have been ommitted on alpha and lamar are given below in the appropriate sections to begin with, cleanup the routing table and set up the routes ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ alpha and lamar have external IP addresses. alpha = 128.198.61.15 on eth1 (Def GW = 128.198.61.1) lamar = 128.198.60.168 on eth1 (Def GW = 128.198.60.129) Beta,Delta,Gamma, Titan do not have direct internet access NOTE : On these machines, for each card, make sure the gateway is set to the card on hte machine at the other end NOTE : for runnin the following commands, do not use remote login. 'cause, after the 'iptables' step, you can lose contact with the machin if you do a remote login Machine cardIP Talkingto alpha 192.168.0.1(eth0) 192.168.0.3(eth0) on beta alpha 192.168.1.1(eth0) 192.168.1.4(eth0) on delta beta 192.168.2.3(eth1) 192.168.2.4(eth1) on gamma gamma 192.168.4.4(eth1) 192.168.4.2(eth2) on lamar lamar 192.168.3.2(eth0) 192.168.3.4(eth0) on titan Note that all private IP addresses ending with '1' are on alpha, ending with 2 are on lamar and '3' are on beta. ===================================================== (ma = alias for /root/mpls-linux-1.1/utils/mplsadm2) (iptables = /root/iptables-1.2.7a/iptables) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ on alpha : +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ /sbin/iptables -F ipsec setup stop echo "1" > /proc/sys/net/ipv4/ip_forward echo "0" > /proc/sys/net/ipv4/conf/eth0/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth2/rp_filter /sbin/iptables -P INPUT DROP /sbin/iptables -P FORWARD DROP /sbin/iptables -P OUTPUT DROP // The following 2 lines to be able to talk between the subnets /sbin/iptables -A FORWARD -s 192.168.3.0/24 -d 192.168.1.0/24 -j ACCEPT /sbin/iptables -A FORWARD -d 192.168.3.0/24 -s 192.168.1.0/24 -j ACCEPT // The following 2 lines to be able to talk between the gateways // PROBABLY NO NEED FOR$ THE FOLLOWING. 'CAUSE the OUTPUT entry below with // MPLS wiil do /sbin/iptables -A INPUT -s 192.168.4.0/24 -d 192.168.0.0/24 -j ACCEPT /sbin/iptables -A OUTPUT -d 192.168.4.0/24 -s 192.168.0.0/24 -j ACCEPT // The above 2 lines about INPUT and 0.0 and 4.0 added in an attempt to make // ipsec establish contact AFTER MPLS is setup ============== towards beta : // note that the key is assumed to be 0x00000002 ============== ma -A -O 0 ma -O 0x00000002 -o push:gen:23:set:eth0:ipv4:192.168.0.3 // When IPSec is also used, an outgoing packet will already have // the source and destination as hte two IPSec gateways. So, just // add the entry that involves the remote IPSec gateway. // Note that, this means, you HAVE TO run IPSEC with MPLS. If not // the packet will be dropped iptables -A OUTPUT -t mangle -d 192.168.4.2/24 -j MPLS --set-key 0x00000002 ma -A -I gen:22:0 -i pop:peek ma -L eth0:0 // associate eth0 with label space 0 // For traffic from the subnet into the other side // the one below to 4.2 can be done away with // Theforwarding should be taken care of by IPSec. MPLS should never // see a destination address of the subnet. So, you should not need // the followibg 'FORWARD' entry. ==================IMPORTANT=========== // But add this entry if you want // to turn off IPSEC and just see how things work with MPLS alone. But for some reason, // this rule breaks IPSEC. i.e. if you do not somehow flush out this rule and bring uo // ipsec, the traffic from the private subnets bypasses IPSec. Traffic from within // thge gateway is OK. This is happening because FORWARD seems to have precedence over // OUTPUT, and IPSEC does NOT read from FORWARD // iptables -A FORWARD -t mangle -d 192.168.3.4/24 -j MPLS --set-key 0x00000002 // The above line added so that MPLS can be done bewtween the IPSec // gateways - 0.1. and 4.2 /sbin/iptables -P INPUT ACCEPT /sbin/iptables -P FORWARD ACCEPT /sbin/iptables -P OUTPUT ACCEPT ------- BEGIN : Firewall rules for ipsec -------------- /sbin/iptables -A INPUT -p udp -d 192.168.0.1/24 -s 192.168.4.2 --dport 500 -j ACCEPT /sbin/iptables -A INPUT -p 50 -d 192.168.0.1/24 -s 192.168.4.2 -j ACCEPT /sbin/iptables -A INPUT -p 51 -d 192.168.0.1/24 -s 192.168.4.2 -j ACCEPT ------- END : Firewall rules for ipsec -------------- +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ on beta : +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ (ma = alias for /root/mpls-linux-1.1/utils/mplsadm2) (iptables = /root/iptables-1.2.7a/iptables) /sbin/iptables -F /sbin/iptables -P INPUT ACCEPT /sbin/iptables -P FORWARD ACCEPT echo "1" > /proc/sys/net/ipv4/ip_forward echo "0" > /proc/sys/net/ipv4/conf/eth0/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth2/rp_filter ============== towards alpha : // the assumption is that the key is 0x00000002 ============== ma -A -O 0 ma -O 0x00000002 -o push:gen:22:set:eth0:ipv4:192.168.0.1 ma -A -I gen:23:0 -i pop:peek ma -L eth0:0 iptables -A OUTPUT -t mangle -d 192.168.0.1/24 -j MPLS --set-key 0x00000002 ============== towards gamma : // the assumption is that the key is 0x00000003 ============== ma -A -O 0 ma -O 0x00000003 -o push:gen:21:set:eth1:ipv4:192.168.2.4 ma -A -I gen:20:0 -i pop:peek ma -L eth1:0 iptables -A OUTPUT -t mangle -d 192.168.2.4/24 -j MPLS --set-key 0x00000003 ========================================== For label switching at beta for traffic between alpha and gamma =========================================== // The following steps only if beta is an LSR. if it is LER, do not // include his step (This line associates the incoming label 20 with outgoing 22) (towards alpha) ma -B -I gen:20:0 -O 0x00000002 ma -B -I gen:23:0 -O 0x00000003 (The above line associates the incoming label 23, with outgoing 21) (towards gamma) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ on gamma : +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ (ma = alias for /root/mpls-linux-1.1/utils/mplsadm2) (iptables = /root/iptables-1.2.7a/iptables) /sbin/iptables -F /sbin/iptables -P INPUT ACCEPT /sbin/iptables -P FORWARD ACCEPT echo "1" > /proc/sys/net/ipv4/ip_forward echo "0" > /proc/sys/net/ipv4/conf/eth0/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth2/rp_filter ============== towards beta : // The assumption is tha tthe key is 0x00000002 ============== ma -A -O 0 ma -O 0x00000002 -o push:gen:20:set:eth1:ipv4:192.168.2.3 ma -A -I gen:21:0 -i pop:peek iptables -A OUTPUT -t mangle -d 192.168.2.3/24 -j MPLS --set-key 0x00000002 ma -L eth1:0 ============== towards lamar : // the assumption is the key is 0x00000003 ============== ma -A -O 0 ma -O 0x00000003 -o push:gen:19:set:eth0:ipv4:192.168.4.2 ma -A -I gen:18:0 -i pop:peek iptables -A OUTPUT -t mangle -d 192.168.4.2/24 -j MPLS --set-key 0x00000003 ma -L eth0:0 ========================================== For label switching at gamma for traffic between lamar and beta =========================================== // The following steps only if beta is an LSR. if it is LER, do not // include his step (This line associates the incoming label 18, with outgoing 20) (towards beta) ma -B -I gen:18:0 -O 0x00000002 (bind incoming 18 to outgoing 19) ma -B -I gen:21:0 -O 0x00000003 (The above line associates the incoming label 21 with outgoing 19) (towards lamar) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ on lamar : +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ /sbin/iptables -F ipsec setup stop echo "1" > /proc/sys/net/ipv4/ip_forward echo "0" > /proc/sys/net/ipv4/conf/eth0/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth2/rp_filter /sbin/iptables -P INPUT DROP /sbin/iptables -P FORWARD DROP /sbin/iptables -P OUTPUT DROP /sbin/iptables -A FORWARD -s 192.168.3.0/24 -d 192.168.1.0/24 -j ACCEPT /sbin/iptables -A FORWARD -d 192.168.3.0/24 -s 192.168.1.0/24 -j ACCEPT // The following 2 lines to be able to talk between the gateways // PROBABLY NO NEED FOR$ THE FOLLOWING. 'CAUSE the OUTPUT entry below with // MPLS wiil do /sbin/iptables -A INPUT -d 192.168.4.0/24 -s 192.168.0.0/24 -j ACCEPT /sbin/iptables -A OUTPUT -s 192.168.4.0/24 -d 192.168.0.0/24 -j ACCEPT // The above 2 lines about INPUT and 0.0 and 4.0 added in an attempt to make // ipsec establish contact AFTER MPLS is setup ============== towards gamma : // the assumption is that the key is 0x00000002 ============== ma -A -O 0 ma -O 0x00000002 -o push:gen:18:set:eth2:ipv4:192.168.4.4 // to get ipsec to run, u need to be able to ping between the gateways. // for that, u need the following OUTPUT entry iptables -A OUTPUT -t mangle -d 192.168.0.1/24 -j MPLS --set-key 0x00000002 ma -A -I gen:19:0 -i pop:peek ma -L eth2:0 // associate eth2 with label space 0 // By the time ain outgoing packet gets to MPLS, it has been through // the IPSEc module (hopefully) and hence will have the two IPSEC // gateway names as the source and destination. So, the folloqing //'FORWARD' entry shoud not be needed. But, if you want to turn off IPSEc // and see how things work with MPLS alone, use this statement ==================IMPORTANT=========== // iptables -A FORWARD -t mangle -d 192.168.1.4/24 -j MPLS --set-key 0x00000002 /sbin/iptables -P INPUT ACCEPT /sbin/iptables -P FORWARD ACCEPT /sbin/iptables -P OUTPUT ACCEPT ------- BEGIN : Firewall rules for ipsec -------------- /sbin/iptables -A INPUT -p udp -s 192.168.0.1/24 -d 192.168.4.2/24 --dport 500 -j ACCEPT /sbin/iptables -A INPUT -p 50 -s 192.168.0.1/24 -d 192.168.4.2/24 -j ACCEPT /sbin/iptables -A INPUT -p 51 -s 192.168.0.1/24 -d 192.168.4.2/24 -j ACCEPT ------- END : Firewall rules for ipsec -------------- +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Additional routing table stuff that needs to be done +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ============== BEGIN : To ping between private subnets ================================= Also, the machines on the 'private subnet' (delta on 1.0 and titan on 3.0) should have entries pointing to each other. Otherwise, the packets willnot even get out of the machines i.e. on 192.168.1.4, have the following entry route add -net 192.168.0.0 netmask 255.255.255.0 dev eth0 (The above entry about 0.0 is needed for the following entry about gw=0.1 to be accepted) route add -net 192.168.3.0 netmask 255.255.255.0 gw 192.168.0.1 dev eth0 Make a similar entry on titan.i.e. route add -net 192.168.4.0 netmask 255.255.255.0 dev eth1 route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.4.2 dev eth1 To assist in this, The one additional entry is on alpha and lamar. pointing what interface to take (on alpha and lamar respectively) to get to the appropriate private subnet. Even htough the 'iptable-mpls' entry while setting up MPLS defines the interface,looks like this entry in the routing table is needed to reroute the packet within alpha or lamar to the right interface, on which MPLS is being used. Still, when u send a packet from 1.4 to 3.4 or vice versa, the initial ARP fails because 1.4 does not know what 3.4 is and vice versa - A PROBLEM. See if this can be fixed by crearting a tunnel with ipsec (cause, at this point, we can ping between 1.1 and 3.2. ============== END : To ping between private subnets ================================= If there are any troubles setting up ipsec tunneli with a message like '(not authenticated), chances are that there is a firewall issue. so, start with an iptables -F, build the tunenl and then add conditions