exec.fib and a jail in two subnets

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

exec.fib and a jail in two subnets

Grzegorz Junka-2
Hi,

I am not sure if this question fits better to net or jail list so please
delete one crosspost when replying.

I have two routers in separate subnets (say 10.0.0.0/16 and
172.16.0.0/16). I have enabled multiple fibs on the host and I am trying
to setup a jail so that packets from one router are returned to the same
router. The second subnet is configured like this:

setfib 1 route add -net 172.16.0.0/16 -iface lagg0
setfib 1 route add default 172.16.0.1

When the jail configuration is (differences in red):

mta {
   exec.fib=1;
   ip4.addr = 172.16.0.2;
   interface = lagg0;
}

router 172.16.0.1 is able to send to and receive packets from the jail
as expected.

When the jail configuration is:

mta {
   ip4.addr = 10.0.0.2,172.16.0.2;
   interface = lagg0;
}

then router 10.0.0.1 is also able to send and receive packets from the
jail as expected.

However, when the configuration is:

mta {
exec.fib=1;
   ip4.addr = 10.0.0.2,172.16.0.2;
   interface = lagg0;
}

then router 172.16.0.1 is no longer able to receive a response from the
jail. The router's event log shows entry similar to the following two
about 2 minutes apart:

IN: ACCEPT [54] Connection opened (Port Forwarding: TCP [172.16.0.2]:80
<-​-​> [212.159.95.213]:80 -​ -​ -​ [111.202.101.2]:34172
CLOSED/SYN_SENT ppp3 NAPT)
IN: ACCEPT [57] Connection closed (Port Forwarding: TCP [172.16.0.2]:80
<-​-​> [212.159.95.213]:80 -​ -​ -​ [111.202.101.2]:34172
CLOSED/SYN_SENT ppp3 NAPT)

My question is why the 10.0.0.1 router is able to communicate with the
jail in the second configuration but 172.16.0.1 is not able to
communicate with the jail in the third configuration. Is it because of
order of IPs in ip4.addr?

When the jail is started jls shows only the first IP from either of the
configuration list above (i.e. 10.0.0.2 even if exec.fib is set to 1).
So my guess is that the first IP is somehow a default IP?

Then my additional question is if it's possible for a jail to be in two
subnets at the same time, i.e. so that when the jail responds to a
packet received from router 10.0.0.1 it sends it to the default route
from fib0 and when it responds to a packet received from 172.16.0.1 it
sends it to the default route from fib1. What exec.fib should be in such
a case?

Any help would be greatly appreciated. Thanks!

GrzegorzJ

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: exec.fib and a jail in two subnets

James Gritton
On 2019-03-10 13:40, Grzegorz Junka wrote:

> Hi,
>
> I am not sure if this question fits better to net or jail list so
> please delete one crosspost when replying.
>
> I have two routers in separate subnets (say 10.0.0.0/16 and
> 172.16.0.0/16). I have enabled multiple fibs on the host and I am
> trying to setup a jail so that packets from one router are returned to
> the same router. The second subnet is configured like this:
>
> setfib 1 route add -net 172.16.0.0/16 -iface lagg0
> setfib 1 route add default 172.16.0.1
>
> When the jail configuration is (differences in red):
>
> mta {
>   exec.fib=1;
>   ip4.addr = 172.16.0.2;
>   interface = lagg0;
> }
>
> router 172.16.0.1 is able to send to and receive packets from the jail
> as expected.
>
> When the jail configuration is:
>
> mta {
>   ip4.addr = 10.0.0.2,172.16.0.2;
>   interface = lagg0;
> }
>
> then router 10.0.0.1 is also able to send and receive packets from the
> jail as expected.
>
> However, when the configuration is:
>
> mta {
> exec.fib=1;
>   ip4.addr = 10.0.0.2,172.16.0.2;
>   interface = lagg0;
> }
>
> then router 172.16.0.1 is no longer able to receive a response from
> the jail. The router's event log shows entry similar to the following
> two about 2 minutes apart:
>
> IN: ACCEPT [54] Connection opened (Port Forwarding: TCP
> [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​
> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT)
> IN: ACCEPT [57] Connection closed (Port Forwarding: TCP
> [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​
> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT)
>
> My question is why the 10.0.0.1 router is able to communicate with the
> jail in the second configuration but 172.16.0.1 is not able to
> communicate with the jail in the third configuration. Is it because of
> order of IPs in ip4.addr?
>
> When the jail is started jls shows only the first IP from either of
> the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to
> 1). So my guess is that the first IP is somehow a default IP?
>
> Then my additional question is if it's possible for a jail to be in
> two subnets at the same time, i.e. so that when the jail responds to a
> packet received from router 10.0.0.1 it sends it to the default route
> from fib0 and when it responds to a packet received from 172.16.0.1 it
> sends it to the default route from fib1. What exec.fib should be in
> such a case?
>
> Any help would be greatly appreciated. Thanks!

You're correct in your assumption that a jail's first IP address is its
default: in the absence of binding a particular address for an outgoing
connection, the first-listed address will be used.  So then the problem
with the third jail is you have a packing being sent from 10.0.0.2 with
only the routing table that doesn't include 10.0/16.  I can't say
exactly why your second example *does* work, but at least from the jail
side it has a default address that's reachable in its routing table.  
I'm thinking you're saying that the second jail works not only with 10.0
but also with 172.16 (it's the 172.16 part I'm unsure about).

To answer your last question: sure, a jail can be in two subnets - but
it will still use its first address by default for any outbound packets.
  Note that the FIB associated with the jail isn't *really* associated
with the jail, but with the processes jail(8) starts for it - the reason
for the "exec" in "exec.fib".  You're still free to call setfib from
inside the jail to access a different table.

I haven't tried using two different routing tables in one jail at the
same time; the closest I've come is one jail that routed on the
non-default network.  Outside of the jail world, I believe multiple
routing tables implies multiple instances of servers, and that would be
the same for inside a jail.  Your router log shows port 80, so that
would imply two different apache (or whatever) processes running the
jail, each pointing to its own address, and rung under its own routing
table.

- Jamie
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: exec.fib and a jail in two subnets

Grzegorz Junka-2
On 12/03/2019 19:19, James Gritton wrote:

> On 2019-03-10 13:40, Grzegorz Junka wrote:
>> Hi,
>>
>> I am not sure if this question fits better to net or jail list so
>> please delete one crosspost when replying.
>>
>> I have two routers in separate subnets (say 10.0.0.0/16 and
>> 172.16.0.0/16). I have enabled multiple fibs on the host and I am
>> trying to setup a jail so that packets from one router are returned to
>> the same router. The second subnet is configured like this:
>>
>> setfib 1 route add -net 172.16.0.0/16 -iface lagg0
>> setfib 1 route add default 172.16.0.1
>>
>> When the jail configuration is (differences in red):
>>
>> mta {
>>   exec.fib=1;
>>   ip4.addr = 172.16.0.2;
>>   interface = lagg0;
>> }
>>
>> router 172.16.0.1 is able to send to and receive packets from the jail
>> as expected.
>>
>> When the jail configuration is:
>>
>> mta {
>>   ip4.addr = 10.0.0.2,172.16.0.2;
>>   interface = lagg0;
>> }
>>
>> then router 10.0.0.1 is also able to send and receive packets from the
>> jail as expected.
>>
>> However, when the configuration is:
>>
>> mta {
>> exec.fib=1;
>>   ip4.addr = 10.0.0.2,172.16.0.2;
>>   interface = lagg0;
>> }
>>
>> then router 172.16.0.1 is no longer able to receive a response from
>> the jail. The router's event log shows entry similar to the following
>> two about 2 minutes apart:
>>
>> IN: ACCEPT [54] Connection opened (Port Forwarding: TCP
>> [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​
>> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT)
>> IN: ACCEPT [57] Connection closed (Port Forwarding: TCP
>> [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​
>> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT)
>>
>> My question is why the 10.0.0.1 router is able to communicate with the
>> jail in the second configuration but 172.16.0.1 is not able to
>> communicate with the jail in the third configuration. Is it because of
>> order of IPs in ip4.addr?
>>
>> When the jail is started jls shows only the first IP from either of
>> the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to
>> 1). So my guess is that the first IP is somehow a default IP?
>>
>> Then my additional question is if it's possible for a jail to be in
>> two subnets at the same time, i.e. so that when the jail responds to a
>> packet received from router 10.0.0.1 it sends it to the default route
>> from fib0 and when it responds to a packet received from 172.16.0.1 it
>> sends it to the default route from fib1. What exec.fib should be in
>> such a case?
>>
>> Any help would be greatly appreciated. Thanks!
>
> You're correct in your assumption that a jail's first IP address is
> its default: in the absence of binding a particular address for an
> outgoing connection, the first-listed address will be used.  So then
> the problem with the third jail is you have a packing being sent from
> 10.0.0.2 with only the routing table that doesn't include 10.0/16.  I
> can't say exactly why your second example *does* work, but at least
> from the jail side it has a default address that's reachable in its
> routing table.  I'm thinking you're saying that the second jail works
> not only with 10.0 but also with 172.16 (it's the 172.16 part I'm
> unsure about).
>
> To answer your last question: sure, a jail can be in two subnets - but
> it will still use its first address by default for any outbound
> packets.  Note that the FIB associated with the jail isn't *really*
> associated with the jail, but with the processes jail(8) starts for it
> - the reason for the "exec" in "exec.fib". You're still free to call
> setfib from inside the jail to access a different table.
>
> I haven't tried using two different routing tables in one jail at the
> same time; the closest I've come is one jail that routed on the
> non-default network.  Outside of the jail world, I believe multiple
> routing tables implies multiple instances of servers, and that would
> be the same for inside a jail.  Your router log shows port 80, so that
> would imply two different apache (or whatever) processes running the
> jail, each pointing to its own address, and rung under its own routing
> table.
>

Many thanks for your response. The second example works with 10.0.0.1
but not with 172.16.0.1, otherwise there would be no post. Following on
your response, lets assume that a process (e.g. nginx) listens on both
IPs, 10.0.0.2,172.16.0.2. Is it possible to configure fibs or default
routes or whatever so that when a packet arrives from 10.0.0.1 it is
send back to 10.0.0.1 and if it arrives from 172.16.0.1 it is send back
to 172.16.0.1 (thus using default routes from either fib0 or fib1
depending if the packet came from a router in one of those network)? If
not, would it be possible to do this with some iptables/pf rules (which
I understand in FreeBSD 12 should work in a jail with VNET)?


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: exec.fib and a jail in two subnets

James Gritton
On Tue, Mar 12, 2019 at 2:05 PM Grzegorz Junka <[hidden email]> wrote:

>
> On 12/03/2019 19:19, James Gritton wrote:
> > On 2019-03-10 13:40, Grzegorz Junka wrote:
> >> Hi,
> >>
> >> I am not sure if this question fits better to net or jail list so
> >> please delete one crosspost when replying.
> >>
> >> I have two routers in separate subnets (say 10.0.0.0/16 and
> >> 172.16.0.0/16). I have enabled multiple fibs on the host and I am
> >> trying to setup a jail so that packets from one router are returned to
> >> the same router. The second subnet is configured like this:
> >>
> >> setfib 1 route add -net 172.16.0.0/16 -iface lagg0
> >> setfib 1 route add default 172.16.0.1
> >>
> >> When the jail configuration is (differences in red):
> >>
> >> mta {
> >>   exec.fib=1;
> >>   ip4.addr = 172.16.0.2;
> >>   interface = lagg0;
> >> }
> >>
> >> router 172.16.0.1 is able to send to and receive packets from the jail
> >> as expected.
> >>
> >> When the jail configuration is:
> >>
> >> mta {
> >>   ip4.addr = 10.0.0.2,172.16.0.2;
> >>   interface = lagg0;
> >> }
> >>
> >> then router 10.0.0.1 is also able to send and receive packets from the
> >> jail as expected.
> >>
> >> However, when the configuration is:
> >>
> >> mta {
> >> exec.fib=1;
> >>   ip4.addr = 10.0.0.2,172.16.0.2;
> >>   interface = lagg0;
> >> }
> >>
> >> then router 172.16.0.1 is no longer able to receive a response from
> >> the jail. The router's event log shows entry similar to the following
> >> two about 2 minutes apart:
> >>
> >> IN: ACCEPT [54] Connection opened (Port Forwarding: TCP
> >> [172.16.0.2]:80 <--> [212.159.95.213]:80 - - -
> >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT)
> >> IN: ACCEPT [57] Connection closed (Port Forwarding: TCP
> >> [172.16.0.2]:80 <--> [212.159.95.213]:80 - - -
> >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT)
> >>
> >> My question is why the 10.0.0.1 router is able to communicate with the
> >> jail in the second configuration but 172.16.0.1 is not able to
> >> communicate with the jail in the third configuration. Is it because of
> >> order of IPs in ip4.addr?
> >>
> >> When the jail is started jls shows only the first IP from either of
> >> the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to
> >> 1). So my guess is that the first IP is somehow a default IP?
> >>
> >> Then my additional question is if it's possible for a jail to be in
> >> two subnets at the same time, i.e. so that when the jail responds to a
> >> packet received from router 10.0.0.1 it sends it to the default route
> >> from fib0 and when it responds to a packet received from 172.16.0.1 it
> >> sends it to the default route from fib1. What exec.fib should be in
> >> such a case?
> >>
> >> Any help would be greatly appreciated. Thanks!
> >
> > You're correct in your assumption that a jail's first IP address is
> > its default: in the absence of binding a particular address for an
> > outgoing connection, the first-listed address will be used.  So then
> > the problem with the third jail is you have a packing being sent from
> > 10.0.0.2 with only the routing table that doesn't include 10.0/16.  I
> > can't say exactly why your second example *does* work, but at least
> > from the jail side it has a default address that's reachable in its
> > routing table.  I'm thinking you're saying that the second jail works
> > not only with 10.0 but also with 172.16 (it's the 172.16 part I'm
> > unsure about).
> >
> > To answer your last question: sure, a jail can be in two subnets - but
> > it will still use its first address by default for any outbound
> > packets.  Note that the FIB associated with the jail isn't *really*
> > associated with the jail, but with the processes jail(8) starts for it
> > - the reason for the "exec" in "exec.fib". You're still free to call
> > setfib from inside the jail to access a different table.
> >
> > I haven't tried using two different routing tables in one jail at the
> > same time; the closest I've come is one jail that routed on the
> > non-default network.  Outside of the jail world, I believe multiple
> > routing tables implies multiple instances of servers, and that would
> > be the same for inside a jail.  Your router log shows port 80, so that
> > would imply two different apache (or whatever) processes running the
> > jail, each pointing to its own address, and rung under its own routing
> > table.
> >
>
> Many thanks for your response. The second example works with 10.0.0.1
> but not with 172.16.0.1, otherwise there would be no post. Following on
> your response, lets assume that a process (e.g. nginx) listens on both
> IPs, 10.0.0.2,172.16.0.2. Is it possible to configure fibs or default
> routes or whatever so that when a packet arrives from 10.0.0.1 it is
> send back to 10.0.0.1 and if it arrives from 172.16.0.1 it is send back
> to 172.16.0.1 (thus using default routes from either fib0 or fib1
> depending if the packet came from a router in one of those network)? If
> not, would it be possible to do this with some iptables/pf rules (which
> I understand in FreeBSD 12 should work in a jail with VNET)?

> My understanding (which I admit is imperfect) is that it's not
> possible with default routes alone.  At the application level, it
> would be possible if nginx was either fib-aware, or if it explicitly
> bound the source address of its replies - but neither of those are
> things typically done at the application level.


> It is possible however at the firewall level; At least I know it's
> possible for ipfw (the small corner of the firewall world that I
> use).  A quick check of ipf and ipfilter man pages didn't show "fib"
> anywhere, but don't take my word on those.  It also may require a
> VNET jail; I've never run a system with your exact setup so I'm
> unsure whether the binding to the first (non-vnet) jail address
> happens before or after the ipfilter rules.

- Jamie
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: exec.fib and a jail in two subnets

Grzegorz Junka-2

>> Many thanks for your response. The second example works with 10.0.0.1
>> but not with 172.16.0.1, otherwise there would be no post. Following on
>> your response, lets assume that a process (e.g. nginx) listens on both
>> IPs, 10.0.0.2,172.16.0.2. Is it possible to configure fibs or default
>> routes or whatever so that when a packet arrives from 10.0.0.1 it is
>> send back to 10.0.0.1 and if it arrives from 172.16.0.1 it is send back
>> to 172.16.0.1 (thus using default routes from either fib0 or fib1
>> depending if the packet came from a router in one of those network)? If
>> not, would it be possible to do this with some iptables/pf rules (which
>> I understand in FreeBSD 12 should work in a jail with VNET)?

> My understanding (which I admit is imperfect) is that it's not
> possible with default routes alone.  At the application level, it
> would be possible if nginx was either fib-aware, or if it explicitly
> bound the source address of its replies - but neither of those are
> things typically done at the application level.
>
> It is possible however at the firewall level; At least I know it's
> possible for ipfw (the small corner of the firewall world that I
> use).  A quick check of ipf and ipfilter man pages didn't show "fib"
> anywhere, but don't take my word on those.  It also may require a
> VNET jail; I've never run a system with your exact setup so I'm
> unsure whether the binding to the first (non-vnet) jail address
> happens before or after the ipfilter rules.
>
> - Jamie


I am just playing with this now and what I see is that a jail can't be
in two fibs at the same time. It looks like the host is able to select
the default route depending on the subnet in which is the IP I want to
reach, but in the jail, telneting or otherwise trying to reach any IP
that isn't in the same subnet as the fib specified in exec.fib, is not
working.

For example, in jail this works:

telnet 172.16.0.1 80

but this doesn't

telnet 10.0.0.1 80

On the host both works. And both, the host and the jail have an IP and
an alias in both subnets.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"