setfib (ez)jails and wierd routing

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

setfib (ez)jails and wierd routing

Marko Cupać
Hi,

I notice wierd routing in my setfib (ez)jails setup.

I have a server with multiple NICs. setfib should ensure that LAN jails
(setfib 1) can not talk to DMZ jails (setfib 2) over loopbacks, but
need to go through firewalls as though they were physical boxes.

pacija@warden3:~ % sudo setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.30.19.190       UGS        bce0
10.30.19.160/27    00:1c:c4:de:0a:86  US         bce0
127.0.0.1          lo0                UHS         lo0
127.0.1.0/24       lo1                US          lo1

pacija@warden3:~ % sudo setfib 2 netstat -rn
Routing tables (fib: 2)

Internet:
Destination        Gateway            Flags     Netif Expire
default            193.53.106.254     UGS        bce1
127.0.0.1          lo0                UHS         lo0
127.0.2.0/24       lo2                US          lo2
193.53.106.0/24    00:1c:c4:de:0a:84  US         bce1

Host has the same default route as fib 1:

pacija@warden3:~ % sudo netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.30.19.190       UGS        bce0
...

If I ssh from the Internet into DMZ jail, everything works as expected.
But if I ping DMZ jail from the Internet, I see reply packets leaving
not the interface they came from (bce1, public address space, DMZ), but
another one (bce0, private address space, LAN). This is kinda
understandable, because jail on fib2 does not have ICMP enabled, so
it is not DMZ jail, but the host (which is in fib 0) who replies to
packets via its default gateway (router on a private LAN).

Is there an easy and elegant way to solve this? Like binding IP address
to fib? I wouldn't like to have to fire up pf on host and meddle with
reply-to rules in order to achieve this, I'd rather revert to old setup
of separate physical servers for each network.

Thank you in advance,
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: setfib (ez)jails and wierd routing

Andrew Hotlab
Hi Marko. I'm running an almost identical setup, but I do not have this issue: ICMP echo reply packets are sent from the right interface.
The only difference is that I didn't defined additional lo1 and lo2 interfaces, but I guess it shouldn't be the cause.

I'm running releng/10.3. Which release are you working on?

Andrew
________________________________________
From: [hidden email] [[hidden email]] on behalf of Marko Cupać [[hidden email]]
Sent: Friday, September 29, 2017 10:32 AM
To: [hidden email]
Subject: setfib (ez)jails and wierd routing

Hi,

I notice wierd routing in my setfib (ez)jails setup.

I have a server with multiple NICs. setfib should ensure that LAN jails
(setfib 1) can not talk to DMZ jails (setfib 2) over loopbacks, but
need to go through firewalls as though they were physical boxes.

pacija@warden3:~ % sudo setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.30.19.190       UGS        bce0
10.30.19.160/27    00:1c:c4:de:0a:86  US         bce0
127.0.0.1          lo0                UHS         lo0
127.0.1.0/24       lo1                US          lo1

pacija@warden3:~ % sudo setfib 2 netstat -rn
Routing tables (fib: 2)

Internet:
Destination        Gateway            Flags     Netif Expire
default            193.53.106.254     UGS        bce1
127.0.0.1          lo0                UHS         lo0
127.0.2.0/24       lo2                US          lo2
193.53.106.0/24    00:1c:c4:de:0a:84  US         bce1

Host has the same default route as fib 1:

pacija@warden3:~ % sudo netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.30.19.190       UGS        bce0
...

If I ssh from the Internet into DMZ jail, everything works as expected.
But if I ping DMZ jail from the Internet, I see reply packets leaving
not the interface they came from (bce1, public address space, DMZ), but
another one (bce0, private address space, LAN). This is kinda
understandable, because jail on fib2 does not have ICMP enabled, so
it is not DMZ jail, but the host (which is in fib 0) who replies to
packets via its default gateway (router on a private LAN).

Is there an easy and elegant way to solve this? Like binding IP address
to fib? I wouldn't like to have to fire up pf on host and meddle with
reply-to rules in order to achieve this, I'd rather revert to old setup
of separate physical servers for each network.

Thank you in advance,
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: setfib (ez)jails and wierd routing

Marko Cupać
On Sat, 30 Sep 2017 10:38:58 +0000
Andrew Hotlab <[hidden email]> wrote:

> Hi Marko. I'm running an almost identical setup, but I do not have
> this issue: ICMP echo reply packets are sent from the right
> interface. The only difference is that I didn't defined additional
> lo1 and lo2 interfaces, but I guess it shouldn't be the cause.
>
> I'm running releng/10.3. Which release are you working on?

Hi Andrew,

sorry for late reply. I'm running 11.1-RELEASE-p1. I am definitely
seeing packets with source addresses of my DMZ jails (fib2) exiting
through interface on local LAN. Those are mostly icmp echo replies that
should be coming from jails but are not due to the fact that jails
don't have raw sockets enables. So, echo replies are returned from
host (and not jails), whose default gateway is on internal network.

Would freebsd-net be more appropriate list for this problem?

Thank you in advance,
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: setfib (ez)jails and wierd routing

Andrew Hotlab
________________________________________
From: Marko Cupać <[hidden email]>
Sent: Monday, October 16, 2017 4:18 PM
To: Andrew Hotlab
Cc: [hidden email]
Subject: Re: setfib (ez)jails and wierd routing

> On Sat, 30 Sep 2017 10:38:58 +0000
> Andrew Hotlab <[hidden email]> wrote:
>
> > I'm running releng/10.3. Which release are you working on?
>
> sorry for late reply. I'm running 11.1-RELEASE-p1. I am definitely
> seeing packets with source addresses of my DMZ jails (fib2) exiting
> through interface on local LAN. Those are mostly icmp echo replies that
> should be coming from jails but are not due to the fact that jails
> don't have raw sockets enables. So, echo replies are returned from
> host (and not jails), whose default gateway is on internal network.
>

I just setup a similar scenario on a FreeBSD 11.1 host. It seems that
all is working fine (172.21.10.0/24 is the DMZ, while 192.168.1.0/24
is the LAN). Please see the following transcript:

root@BSD11:~ # uname -msr
FreeBSD 11.1-RELEASE amd64

root@BSD11:~ # ifconfig | egrep '^[a-z]|inet '
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        inet 172.21.10.100 netmask 0xffffff00 broadcast 172.21.10.255
        inet 172.21.10.101 netmask 0xffffffff broadcast 172.21.10.101
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000

root@BSD11:~ # netstat -rnfinet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.254      UGS         em1
127.0.0.1          link#3             UH          lo0
172.21.10.0/24     link#1             U           em0
172.21.10.100      link#1             UHS         lo0
172.21.10.101      link#1             UHS         lo0
172.21.10.101/32   link#1             U           em0
192.168.1.0/24     link#2             U           em1
192.168.1.100      link#2             UHS         lo0

root@BSD11:~ # setfib 1 netstat -rfinet
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.21.10.254      UGS         em0
localhost          link#3             UH          lo0
172.21.10.0/24     link#1             U           em0
172.21.10.101/32   link#1             U           em0
192.168.1.0/24     link#2             U           em1

root@BSD11:~ # cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
jtest01 {
  host.hostname = "jtest01.test.lab";
  path = /usr/jails/jtest01;
  ip4.addr = "em0|172.21.10.101/32";
  persist;
  allow.raw_sockets;
  exec.fib = "1";
}

root@BSD11:~ # jls
   JID  IP Address      Hostname                      Path
     8  172.21.10.101   jtest01.test.lab          /usr/jails/jtest01

root@BSD11:~ # ssh 172.21.10.101 'sysctl net.my_fibnum'
Password for [hidden email]:
net.my_fibnum: 1

root@BSD11:~ # tcpdump -i em0 -n -p icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:07:19.524839 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315, seq 0, length 64
17:07:20.539686 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315, seq 1, length 64
17:07:21.551653 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315, seq 2, length 64
17:07:22.562764 IP 172.21.1.81 > 172.21.10.101: ICMP echo request, id 65315, seq 3, length 64
^C
4 packets captured
12 packets received by filter
0 packets dropped by kernel


> Would freebsd-net be more appropriate list for this problem?

Maybe, but I would double check your jail configuration before ask to that list.
My guess is that your jail might not be associated to the right fib.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: setfib (ez)jails and wierd routing

Marko Cupać
On Tue, 17 Oct 2017 15:17:16 +0000
Andrew Hotlab <[hidden email]> wrote:

> root@BSD11:~ # cat /etc/jail.conf
> exec.start = "/bin/sh /etc/rc";
> exec.stop = "/bin/sh /etc/rc.shutdown";
> exec.clean;
> mount.devfs;
> jtest01 {
>   host.hostname = "jtest01.test.lab";
>   path = /usr/jails/jtest01;
>   ip4.addr = "em0|172.21.10.101/32";
>   persist;
>   allow.raw_sockets;
>   exec.fib = "1";
> }

Andrew,

do you have the ability to remove allow.raw_sockets line from jtest01
jail and try to ping it while tcpdumping icmp on em1? You should see
reply packets leaving em1.

Thank you in advance.
--
Before enlightenment - chop wood, draw water.
After  enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: setfib (ez)jails and wierd routing

Andrew Hotlab
________________________________________
From: Marko Cupać <[hidden email]>
Sent: Monday, October 23, 2017 1:58 PM
To: Andrew Hotlab
Cc: [hidden email]
Subject: Re: setfib (ez)jails and wierd routing

> On Tue, 17 Oct 2017 15:17:16 +0000
> Andrew Hotlab <[hidden email]> wrote:
>
> > root@BSD11:~ # cat /etc/jail.conf
> > exec.start = "/bin/sh /etc/rc";
> > exec.stop = "/bin/sh /etc/rc.shutdown";
> > exec.clean;
> > mount.devfs;
> > jtest01 {
> >   host.hostname = "jtest01.test.lab";
> >   path = /usr/jails/jtest01;
> >   ip4.addr = "em0|172.21.10.101/32";
> >   persist;
> >   allow.raw_sockets;
> >   exec.fib = "1";
> > }
>
> Andrew,
>
> do you have the ability to remove allow.raw_sockets line from jtest01
> jail and try to ping it while tcpdumping icmp on em1? You should see
> reply packets leaving em1.
>

So sorry: I didn't notice that my own transcript shown exactly the
behaviour you are describing... in fact you can see "echo request"
packets, but no "echo reply" on em0 interface!!

And I can confirm you that the problem does not happen in the same
topology with a FreeBSD 10.3 host.

At this point I guess that all responses to ICMP requests received on
IP addresses assigned to jails linked to specific FIB on FreeBSD 11.x
are not influenced by the FIB, while in FreeBSD 10.x they are.

(No problem from ICMP traffic generated from the jail itself: I saw packets
leaving and coming back through the right interface).

Unfortunately I haven't the competence to point you to the right direction
to solve, but I think it is a jail-related issue, thus this should be the right
mailing list to discuss about this.

I'll come back if I'll be able to understand something more.


Andrew
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"