Restricting IP ranges for guests over tap devices

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Restricting IP ranges for guests over tap devices

freebsd-virtualization mailing list
Hello!

Let's say I have a machine running a few dozen bhyve guests. Each bhyve
guest gets its own tap device, and all of the tap devices are connected
to a bridge.

Everything works fine. I can write pf rules that control access between
each guest, and between each guest and the world. I can't directly
observe the IP addresses that the guests have assigned to the tap
devices I gave them, but if I know the addresses beforehand, I can for
example write pf rules that say things like:

  block log all
  pass in on tap23 proto tcp \
    from any to $guest_23_ip port ssh modulate state

That then means that even if the guest is compromised and tries to bind
a server to another address, the pf rules won't allow anyone else to
actually connect to it.

The good thing about this is also the bad thing about this; I have to
write specific rules that say "only allow access to this specific IP
via this specific tap device". Over dozens of guests, that can multiply
to hundreds of laboriously maintained rules.

Is there some more general way I can supply a mapping between tap
devices and allowed addresses? Remember that pf can't see the guest
addresses on the host sides of the tap devices, so I can't use the
(device) syntax to expand to "the address of a NIC called 'device'".

I can generate rule sets, but perhaps there's something "better"[0]? The
documentation isn't suggesting much.

[0] Better in the sense that, for example, a table is usually better
    than a massive list of macros. :)

--
Mark Raynsford | https://www.io7m.com


attachment0 (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Restricting IP ranges for guests over tap devices

Jason Tubnor
On Sun, 2 Aug 2020 at 00:51, Mark Raynsford via freebsd-virtualization <
[hidden email]> wrote:

> Hello!
>
> Let's say I have a machine running a few dozen bhyve guests. Each bhyve
> guest gets its own tap device, and all of the tap devices are connected
> to a bridge.
>
> Everything works fine. I can write pf rules that control access between
> each guest, and between each guest and the world. I can't directly
> observe the IP addresses that the guests have assigned to the tap
> devices I gave them, but if I know the addresses beforehand, I can for
> example write pf rules that say things like:
>
>   block log all
>   pass in on tap23 proto tcp \
>     from any to $guest_23_ip port ssh modulate state
>
> That then means that even if the guest is compromised and tries to bind
> a server to another address, the pf rules won't allow anyone else to
> actually connect to it.
>
> The good thing about this is also the bad thing about this; I have to
> write specific rules that say "only allow access to this specific IP
> via this specific tap device". Over dozens of guests, that can multiply
> to hundreds of laboriously maintained rules.
>
> Is there some more general way I can supply a mapping between tap
> devices and allowed addresses? Remember that pf can't see the guest
> addresses on the host sides of the tap devices, so I can't use the
> (device) syntax to expand to "the address of a NIC called 'device'".
>
>
>
Treat the tap interface as a bridge and only define the destination port.
That way you are able to protect the guest from the host without knowing
the guest IP address.

I'd do it a bit differently though.  I'd treat the bridge that everything
is tapped into as being a hostile environment.  As such, each guest would
protect itself as if you had a VPN on the public internet, using the guests
built-in firewall.

Another way is isolating each guest or bunch of guests on private VLANs and
then protect these subnets on the host.

Cheers,

Jason.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Restricting IP ranges for guests over tap devices

Miroslav Lachman
On 02/08/2020 03:36, Jason Tubnor wrote:

> On Sun, 2 Aug 2020 at 00:51, Mark Raynsford via freebsd-virtualization <
> [hidden email]> wrote:
>
>> Hello!
>>
>> Let's say I have a machine running a few dozen bhyve guests. Each bhyve
>> guest gets its own tap device, and all of the tap devices are connected
>> to a bridge.
>>
>> Everything works fine. I can write pf rules that control access between
>> each guest, and between each guest and the world. I can't directly
>> observe the IP addresses that the guests have assigned to the tap
>> devices I gave them, but if I know the addresses beforehand, I can for
>> example write pf rules that say things like:
>>
>>    block log all
>>    pass in on tap23 proto tcp \
>>      from any to $guest_23_ip port ssh modulate state
>>
>> That then means that even if the guest is compromised and tries to bind
>> a server to another address, the pf rules won't allow anyone else to
>> actually connect to it.
>>
>> The good thing about this is also the bad thing about this; I have to
>> write specific rules that say "only allow access to this specific IP
>> via this specific tap device". Over dozens of guests, that can multiply
>> to hundreds of laboriously maintained rules.
>>
>> Is there some more general way I can supply a mapping between tap
>> devices and allowed addresses? Remember that pf can't see the guest
>> addresses on the host sides of the tap devices, so I can't use the
>> (device) syntax to expand to "the address of a NIC called 'device'".
>>
>>
>>
> Treat the tap interface as a bridge and only define the destination port.
> That way you are able to protect the guest from the host without knowing
> the guest IP address.
>
> I'd do it a bit differently though.  I'd treat the bridge that everything
> is tapped into as being a hostile environment.  As such, each guest would
> protect itself as if you had a VPN on the public internet, using the guests
> built-in firewall.
>
> Another way is isolating each guest or bunch of guests on private VLANs and
> then protect these subnets on the host.

For me the more serious issue is that malicious guest can assign IP of
another guest or the main host and cause some collisions or
malfunctions. I am looking for the right solution for a long time.

Miroslav Lachman
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Restricting IP ranges for guests over tap devices

John-Mark Gurney-2
In reply to this post by freebsd-virtualization mailing list
Mark Raynsford via freebsd-virtualization wrote this message on Sat, Aug 01, 2020 at 14:51 +0000:

> Let's say I have a machine running a few dozen bhyve guests. Each bhyve
> guest gets its own tap device, and all of the tap devices are connected
> to a bridge.
>
> Everything works fine. I can write pf rules that control access between
> each guest, and between each guest and the world. I can't directly
> observe the IP addresses that the guests have assigned to the tap
> devices I gave them, but if I know the addresses beforehand, I can for
> example write pf rules that say things like:
>
>   block log all
>   pass in on tap23 proto tcp \
>     from any to $guest_23_ip port ssh modulate state
>
> That then means that even if the guest is compromised and tries to bind
> a server to another address, the pf rules won't allow anyone else to
> actually connect to it.
>
> The good thing about this is also the bad thing about this; I have to
> write specific rules that say "only allow access to this specific IP
> via this specific tap device". Over dozens of guests, that can multiply
> to hundreds of laboriously maintained rules.
>
> Is there some more general way I can supply a mapping between tap
> devices and allowed addresses? Remember that pf can't see the guest
> addresses on the host sides of the tap devices, so I can't use the
> (device) syntax to expand to "the address of a NIC called 'device'".
>
> I can generate rule sets, but perhaps there's something "better"[0]? The
> documentation isn't suggesting much.
>
> [0] Better in the sense that, for example, a table is usually better
>     than a massive list of macros. :)
Don't think there is anything better...

bridge does have sticky that binds the mac address to an interface, but
that doesn't deal w/ IP ARP.

One issue w/ this is how do you know the difference between one machine
that's been down for a long time, and an attacking machine that takes
over the down'd machine's IP address?

I assume that these addresses are assigned via DHCP server, otherwise
if you are launching the VM's w/ known static IP's, you could use
pf's anchor directive, and each start/stop of a VM, update the rule for
that tap's anchor.

--
  John-Mark Gurney Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."

signature.asc (968 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Restricting IP ranges for guests over tap devices

Joachim Durchholz
In reply to this post by Miroslav Lachman
Am 02.08.20 um 14:45 schrieb Miroslav Lachman:
> For me the more serious issue is that malicious guest can assign IP of
> another guest or the main host and cause some collisions or
> malfunctions. I am looking for the right solution for a long time.

As of FreeBSD 12, you can put Bhyve into a jail.
Jails can use VNETs, which can be configured for restricted IPs.

https://forums.freebsd.org/threads/bhyve-inside-jails-why.69109/ talks
about this.

Disclaimer: I don't use bhyve so I don't know how accurate the postings are.

Regards,
Jo
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "[hidden email]"