Jails routing and localhost

classic Classic list List threaded Threaded
8 messages Options
Ole
Reply | Threaded
Open this post in threaded view
|

Jails routing and localhost

Ole
Hi,

I have some questions about how routing works for jails.

I have a FreeBSD 11.1 host in a datacenter. Which has only a routed IP
and different /29 routed networks. The IP is setup as /32 and there is a
default route to the router of the datacenter:


  #ifconfig em1
    (...)
    inet a.a.a.57 netmask 0xffffffff broadcast a.a.a.57
    (...)


  # netstat -rn
    (...)
    Destination        Gateway            Flags     Netif Expire
    default            a.a.a.1            UGS         em1
    (...)


If I create jails like

  # ezjail-admin create somejail 'lo1|b.b.b.238,lo1|127.b.b.238'

everything is fine until some service in the jail tries to bind to
127.0.0.1. Because it will bind to the public IP b.b.b.238.
The Handbook [1] tells

  "Inside a jail, access to the loopback address 127.0.0.1 is
  redirected to the first IP address assigned to the jail."

If I change the order of the IP-Adresses the service will bind to
127.b.b.238. But inside the Jail Networking fails in a way that I can't
debug. I can conntect from the outside via ssh but I can't connect from
the Jail to an external Server. I can't find any differences in
routing table or ifconfig between both setups.


I also tried to use tap interfaces instead of lo, but it results in the
same.

I wonder how others solve this problem. I searched a lot, but couldn't
find a solution. Maybe you don't have a solution, but can give me a
hint to debug the Problem. Thank you!


regards
Ole

[1] https://www.freebsd.org/doc/handbook/jails-ezjail.html

attachment0 (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Jails routing and localhost

Luke Crooks
Hi Ole,

I am by no means an expert, but to me I see your problem is here..


  # ezjail-admin create somejail 'lo1|b.b.b.238,lo1|127.b.b.238'

You are binding the jail to the same network controller lo1.

Usually you would bind the jail like..

  # ezjail-admin create somejail 'lo1|127.0.0.238, emX|10.1.1.238'


Where 10.1.1.0/24 is your subnet of your host. And you have free range on
the network and want to create the jail as a fully fledged host.

Seeing as you have only been assigned a /32 for your host. I would imagine
you would either need to possibly do something like...

  # ezjail-admin create somejail 'lo1|127.0.0.238, lo0|127. 0.0.237'

E.g bind the jail loopback of lo1 to the host loopback lo0. But I have
never seen a configuration like yours using the same device twice, but I
could be totally wrong.



On 18 Jan 2018 12:58, "Ole" <[hidden email]> wrote:

> Hi,
>
> I have some questions about how routing works for jails.
>
> I have a FreeBSD 11.1 host in a datacenter. Which has only a routed IP
> and different /29 routed networks. The IP is setup as /32 and there is a
> default route to the router of the datacenter:
>
>
>   #ifconfig em1
>     (...)
>     inet a.a.a.57 netmask 0xffffffff broadcast a.a.a.57
>     (...)
>
>
>   # netstat -rn
>     (...)
>     Destination        Gateway            Flags     Netif Expire
>     default            a.a.a.1            UGS         em1
>     (...)
>
>
> If I create jails like
>
>   # ezjail-admin create somejail 'lo1|b.b.b.238,lo1|127.b.b.238'
>
> everything is fine until some service in the jail tries to bind to
> 127.0.0.1. Because it will bind to the public IP b.b.b.238.
> The Handbook [1] tells
>
>   "Inside a jail, access to the loopback address 127.0.0.1 is
>   redirected to the first IP address assigned to the jail."
>
> If I change the order of the IP-Adresses the service will bind to
> 127.b.b.238. But inside the Jail Networking fails in a way that I can't
> debug. I can conntect from the outside via ssh but I can't connect from
> the Jail to an external Server. I can't find any differences in
> routing table or ifconfig between both setups.
>
>
> I also tried to use tap interfaces instead of lo, but it results in the
> same.
>
> I wonder how others solve this problem. I searched a lot, but couldn't
> find a solution. Maybe you don't have a solution, but can give me a
> hint to debug the Problem. Thank you!
>
>
> regards
> Ole
>
> [1] https://www.freebsd.org/doc/handbook/jails-ezjail.html
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Jails routing and localhost

Isaac (.ike) Levy-2
In reply to this post by Ole
Hi Ole,

I cannot comment on ezjail specifics, but can add notes on how jails fundamentally handle localhost, which may help you,

On Thu, Jan 18, 2018, at 7:23 AM, Ole wrote:

> Hi,
>
> I have some questions about how routing works for jails.
>
> I have a FreeBSD 11.1 host in a datacenter. Which has only a routed IP
> and different /29 routed networks. The IP is setup as /32 and there is a
> default route to the router of the datacenter:
>
>
>   #ifconfig em1
>     (...)
>     inet a.a.a.57 netmask 0xffffffff broadcast a.a.a.57
>     (...)
>
>
>   # netstat -rn
>     (...)
>     Destination        Gateway            Flags     Netif Expire
>     default            a.a.a.1            UGS         em1
>     (...)
>
>
> If I create jails like
>
>   # ezjail-admin create somejail 'lo1|b.b.b.238,lo1|127.b.b.238'
>
> everything is fine until some service in the jail tries to bind to
> 127.0.0.1. Because it will bind to the public IP b.b.b.238.
> The Handbook [1] tells
>
>   "Inside a jail, access to the loopback address 127.0.0.1 is
>   redirected to the first IP address assigned to the jail."

Right- so if you don't assign a loopback address at all, loopback will use to the first IP assigned the host (another response in this thread).

Because your hardware host has the 127.0.0.1 address, if you were to assign it to jails, that would mean all jails would all be communicating using it, which would be bad - (I'm not sure if it even works or if jail(2) prevents it from working).

RFC 3330 tells us, http://www.ietf.org/rfc/rfc3330.txt

   127.0.0.0/8 - This block is assigned for use as the Internet host
   loopback address.  A datagram sent by a higher level protocol to an
   address anywhere within this block should loop back inside the host.
   This is ordinarily implemented using only 127.0.0.1/32 for loopback,
   but no addresses within this block should ever appear on any network
   anywhere [RFC1700, page 5].

So, here's what I've been doing for many years:

- Add another 127.x.x.x address to lo0 on your hardware host  (I typically match the last octects to the public address I'm using, just my way of keeping track of things), for example, this IPv4 address could look like: "127.4.4.4/32".  You can add these single /32 addresses right to the lo0 interface on the host machine.

- Start your jail, first binding "127.4.4.4/32" to it as one of the IP interfaces- follow with your other IP's.

- In your jail, edit /etc/resolv.conf so that the 'localhost' entry, matches your IP above, "127.4.4.4".

Viola- you now have localhost!

--
This process is quite counter-intuitive,  since who ever really thinks about 127.0.0.0/8 as an actual netblock?  (Since these addresses "no addresses within this block should ever appear on any network anywhere [RFC1700, page 5]."

The same principle applies to IPv6 localhost in jails.

>
> If I change the order of the IP-Adresses the service will bind to
> 127.b.b.238. But inside the Jail Networking fails in a way that I can't
> debug. I can conntect from the outside via ssh but I can't connect from
> the Jail to an external Server. I can't find any differences in
> routing table or ifconfig between both setups.

From what you wrote above, I agree with the other person who responded- it may be the order of when you specify interfaces, (or how ezjail does).  Or, it may be that you're not making the localhost address a /32 to isolate it.

--
One more caveat: bad software :)
I've seen plenty of fine software which follows very bad form and hardcodes 127.0.0.1, instead of calling 'localhost' for various operations.  Simple answer here: file a bug and point to internet RFC's if 3rd party software, or, go have a chat with your colleagues if the software is in-house.

>
>
> I also tried to use tap interfaces instead of lo, but it results in the
> same.

  (From a practical security perspective, I've wondered for years if making abstracted interfaces for each localhost in each jail had any advantages, but that's a tangent here.)

>
> I wonder how others solve this problem. I searched a lot, but couldn't
> find a solution. Maybe you don't have a solution, but can give me a
> hint to debug the Problem. Thank you!

Hope this helps, tell us how it goes!

Best,
.ike



>
>
> regards
> Ole
>
> [1] https://www.freebsd.org/doc/handbook/jails-ezjail.html
> Email had 1 attachment:
> + Attachment2
>   1k (application/pgp-signature)
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Jails routing and localhost

Dewayne Geraghty-4

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Jails routing and localhost

Dan Langille
In reply to this post by Isaac (.ike) Levy-2
> On Jan 18, 2018, at 2:32 PM, Isaac (.ike) Levy <[hidden email]> wrote:
>
> Hi Ole,
>
> I cannot comment on ezjail specifics, but can add notes on how jails fundamentally handle localhost, which may help you,
>
> On Thu, Jan 18, 2018, at 7:23 AM, Ole wrote:
>> Hi,
>>
>> I have some questions about how routing works for jails.
>>
>> I have a FreeBSD 11.1 host in a datacenter. Which has only a routed IP
>> and different /29 routed networks. The IP is setup as /32 and there is a
>> default route to the router of the datacenter:
>>
>>
>>  #ifconfig em1
>>    (...)
>>    inet a.a.a.57 netmask 0xffffffff broadcast a.a.a.57
>>    (...)
>>
>>
>>  # netstat -rn
>>    (...)
>>    Destination        Gateway            Flags     Netif Expire
>>    default            a.a.a.1            UGS         em1
>>    (...)
>>
>>
>> If I create jails like
>>
>>  # ezjail-admin create somejail 'lo1|b.b.b.238,lo1|127.b.b.238'
>>
>> everything is fine until some service in the jail tries to bind to
>> 127.0.0.1. Because it will bind to the public IP b.b.b.238.
>> The Handbook [1] tells
>>
>>  "Inside a jail, access to the loopback address 127.0.0.1 is
>>  redirected to the first IP address assigned to the jail."
>
> Right- so if you don't assign a loopback address at all, loopback will use to the first IP assigned the host (another response in this thread).
>
> Because your hardware host has the 127.0.0.1 address, if you were to assign it to jails, that would mean all jails would all be communicating using it, which would be bad - (I'm not sure if it even works or if jail(2) prevents it from working).
>
> RFC 3330 tells us, http://www.ietf.org/rfc/rfc3330.txt
>
>   127.0.0.0/8 - This block is assigned for use as the Internet host
>   loopback address.  A datagram sent by a higher level protocol to an
>   address anywhere within this block should loop back inside the host.
>   This is ordinarily implemented using only 127.0.0.1/32 for loopback,
>   but no addresses within this block should ever appear on any network
>   anywhere [RFC1700, page 5].
>
> So, here's what I've been doing for many years:
>
> - Add another 127.x.x.x address to lo0 on your hardware host  (I typically match the last octects to the public address I'm using, just my way of keeping track of things), for example, this IPv4 address could look like: "127.4.4.4/32".  You can add these single /32 addresses right to the lo0 interface on the host machine.

I do similar, except *sometimes* I create lo1 and assign those addresses there.

I use this approach on jails with no public presence (e.g. database server).

--
Dan Langille - BSDCan / PGCon
[hidden email]



_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Ole
Reply | Threaded
Open this post in threaded view
|

Re: Jails routing and localhost

Ole
In reply to this post by Luke Crooks
Hi Luke,


Thu, 18 Jan 2018 19:03:32 +0000 - Luke Crooks
<[hidden email]>:

> Hi Ole,
>
> I am by no means an expert, but to me I see your problem is here..
>
>
>   # ezjail-admin create somejail 'lo1|b.b.b.238,lo1|127.b.b.238'
>
> You are binding the jail to the same network controller lo1.
>
> Usually you would bind the jail like..
>
>   # ezjail-admin create somejail 'lo1|127.0.0.238, emX|10.1.1.238'

If I do this (and ad first I tried exactly this) the networking on the
host system will fail a few minutes after the jail start. And I have no
remote connection to the Server. So I only can do a hard reset.

I don't know why this happens. At the moment I only have production
Servers in this datacenter, so I can't play with them to reproduce. But
I will organize another and report here.

Usually I have Servers with a public IP in a /24 Layer2 network. Then

  # ezjail-admin create somejail 'lo1|127.b.b.238, emX|b.b.b.238'

woks fine.

 

> Where 10.1.1.0/24 is your subnet of your host. And you have free
> range on the network and want to create the jail as a fully fledged
> host.
>
> Seeing as you have only been assigned a /32 for your host. I would
> imagine you would either need to possibly do something like...
>
>   # ezjail-admin create somejail 'lo1|127.0.0.238, lo0|127. 0.0.237'
>
> E.g bind the jail loopback of lo1 to the host loopback lo0. But I have
> never seen a configuration like yours using the same device twice,
> but I could be totally wrong.
But then I also have to set a static route like

  # route add b.b.b.238 127. 0.0.237

to reach the server with the public IP?


Thank you all for your replies
Ole

attachment0 (836 bytes) Download Attachment
Ole
Reply | Threaded
Open this post in threaded view
|

Re: Jails routing and localhost

Ole
In reply to this post by Dewayne Geraghty-4
Hi Dewayne,


Fri, 19 Jan 2018 10:36:43 +1100 - Dewayne Geraghty
<[hidden email]>:

> If you're paranoid, I also add a firewall rule to restrict traffic
> from/to specific ports and IP's over lo0.  If you have anything
> sensitive you might also consider this restriction.   Though I would
> recommend using "tcpdump -ni $INTERFACE" to learn how jails and
> routing works in your environment.  I was surprised to observe: when
> two jails are assigned IP's on their external interface the traffic
> between, expecting to use their external interfaces, traverses lo0. 

Until now I thought that Jails with two different /32 loopback
addresses can not communicate over loopback. Because it is /32. But you
are right. I need a firewall rule to block traffic between the jails.

> PS Sadly there are many examples of ports using 127.0.0.1 instead of
> localhost, there are 104 different files in the Samba 4.7 suite that
> use 127.0.0.1 :/

Yes. I think there are two standards. On is like Isaac told RFC 3330.
And the other one was "vote with the feet" and is localhost = 127.0.0.1
There is too many software with this address hardcoded. So it is a
security feature that software will not bind to public IP by accident.



I wonder why it is such a difference if the IP address of the host
is /32 or not. And I cant' just change it to /24, because than I
couldn't reach the other Server in this /24 Network. And some of them
are also mine :-(


Ole

attachment0 (836 bytes) Download Attachment
Ole
Reply | Threaded
Open this post in threaded view
|

Re: Jails routing and localhost

Ole
In reply to this post by Isaac (.ike) Levy-2
Thu, 18 Jan 2018 14:32:06 -0500 - "Isaac (.ike) Levy"
<[hidden email]>:

> Hope this helps, tell us how it goes!

Sorry for the delay. I couldn't figure out how to solve this problem
and I decided to to take the bull by the horns and migrated everything
from ezjail to iocage with vnet interfaces.

This solves the problem.

Ole

attachment0 (836 bytes) Download Attachment