vnet NAT'd jails extremely slow, connection dies

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

vnet NAT'd jails extremely slow, connection dies

Farhan Khan
Hi all,

I have a jail NAT'd to a base system, but the connection is extremely
slow and frequently disconnects drops, whereas the base is fine has
perfectly fine connectivity.

My configuration is as follows:
vtnet0: Has routeable IPv4 address and 172.16.0.1/16
Jail uses epair4b, base has epair4a. Jail's IP is 172.16.0.5/16.
The base and jail can ping each other.
bridge0: contains vtnet0 and epair4a.

I have gateway_enable="YES"
My pf.conf is as follows:
nat pass from 172.16.0.0/16 to any -> (vtnet0)

When I try to run clamav, the connectivity stalls after a few minutes
and eventually disconnects. I ran tcpdump on the bridge and saw a lot
of HTTP seq and ack packets but no actual data. I am not using IPv6
yet.

Assistance please.
Thanks
--
Farhan Khan
PGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vnet NAT'd jails extremely slow, connection dies

Farhan Khan
On Mon, Feb 4, 2019 at 2:29 PM Farhan Khan <[hidden email]> wrote:

>
> Hi all,
>
> I have a jail NAT'd to a base system, but the connection is extremely
> slow and frequently disconnects drops, whereas the base is fine has
> perfectly fine connectivity.
>
> My configuration is as follows:
> vtnet0: Has routeable IPv4 address and 172.16.0.1/16
> Jail uses epair4b, base has epair4a. Jail's IP is 172.16.0.5/16.
> The base and jail can ping each other.
> bridge0: contains vtnet0 and epair4a.
>
> I have gateway_enable="YES"
> My pf.conf is as follows:
> nat pass from 172.16.0.0/16 to any -> (vtnet0)
>
> When I try to run clamav, the connectivity stalls after a few minutes
> and eventually disconnects. I ran tcpdump on the bridge and saw a lot
> of HTTP seq and ack packets but no actual data. I am not using IPv6
> yet.
>
> Assistance please.
> Thanks
> --
> Farhan Khan
> PGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE

Just to provide more context to my previous email, outside of the jail
I can download the FreeBSD ISO installer image at 3 MBps. Within the
jail it drops to 12KBps.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vnet NAT'd jails extremely slow, connection dies

Michael Grimm-4
Farhan Khan <[hidden email]> wrote:
> On Mon, Feb 4, 2019 at 2:29 PM Farhan Khan <[hidden email]> wrote:

>> I have a jail NAT'd to a base system, but the connection is extremely
>> slow and frequently disconnects drops, whereas the base is fine has
>> perfectly fine connectivity.
>>
>> My configuration is as follows:
>> vtnet0: Has routeable IPv4 address and 172.16.0.1/16
>> Jail uses epair4b, base has epair4a. Jail's IP is 172.16.0.5/16.
>> The base and jail can ping each other.
>> bridge0: contains vtnet0 and epair4a.
>>
>> I have gateway_enable="YES"
>> My pf.conf is as follows:
>> nat pass from 172.16.0.0/16 to any -> (vtnet0)
>>
>> When I try to run clamav, the connectivity stalls after a few minutes
>> and eventually disconnects. I ran tcpdump on the bridge and saw a lot
>> of HTTP seq and ack packets but no actual data. I am not using IPv6
>> yet.
>
> Just to provide more context to my previous email, outside of the jail
> I can download the FreeBSD ISO installer image at 3 MBps. Within the
> jail it drops to 12KBps.

This sounds familiar to me ;-)

Please have a look at https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html
Solution in https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049484.html

I ended up with the following additions to /boot/loader.conf (and a subsequent reboot):

        # needs to become turned off (LRO) in order to restore tcp performance within VNET jails:
        hw.vtnet.lro_disable="1"  
        hw.vtnet.tso_disable="1"

HTH,
Michael

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vnet NAT'd jails extremely slow, connection dies

Kristof Provost
On 2019-02-05 18:47:23 (+0100), Michael Grimm <[hidden email]> wrote:

> Farhan Khan <[hidden email]> wrote:
> > On Mon, Feb 4, 2019 at 2:29 PM Farhan Khan <[hidden email]> wrote:
>
> >> I have a jail NAT'd to a base system, but the connection is extremely
> >> slow and frequently disconnects drops, whereas the base is fine has
> >> perfectly fine connectivity.
> >>
> >> My configuration is as follows:
> >> vtnet0: Has routeable IPv4 address and 172.16.0.1/16
> >> Jail uses epair4b, base has epair4a. Jail's IP is 172.16.0.5/16.
> >> The base and jail can ping each other.
> >> bridge0: contains vtnet0 and epair4a.
> >>
> >> I have gateway_enable="YES"
> >> My pf.conf is as follows:
> >> nat pass from 172.16.0.0/16 to any -> (vtnet0)
> >>
> >> When I try to run clamav, the connectivity stalls after a few minutes
> >> and eventually disconnects. I ran tcpdump on the bridge and saw a lot
> >> of HTTP seq and ack packets but no actual data. I am not using IPv6
> >> yet.
> >
> > Just to provide more context to my previous email, outside of the jail
> > I can download the FreeBSD ISO installer image at 3 MBps. Within the
> > jail it drops to 12KBps.
>
> This sounds familiar to me ;-)
>
> Please have a look at https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html
> Solution in https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049484.html
>
> I ended up with the following additions to /boot/loader.conf (and a subsequent reboot):
>
> # needs to become turned off (LRO) in order to restore tcp performance within VNET jails:
> hw.vtnet.lro_disable="1"  
> hw.vtnet.tso_disable="1"
>
Farhan has also solved his issue by turning off lro/tso. (We talked on
IRC).

I've not seen this issue myself, but I'm interested in a couple of
points to hopefully pinpoint and maybe even fix the problem.

These are questions for anyone who's running pf on top of a hypervisor
and has vnet or other jails, and has seen slowdowns.

 * What hypervisor are you running?
 * Does the problem affect only the jails, or also the host system?
 * Does it only happen with NAT, or with routed packets as well?

If anyone is affected and not using pf that'd be interesting information
as well.

Regards,
Kristof
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vnet NAT'd jails extremely slow, connection dies

Michael Grimm-4
Kristof Provost <[hidden email]> wrote:

> These are questions for anyone who's running pf on top of a hypervisor
> and has vnet or other jails, and has seen slowdowns.
>
> * What hypervisor are you running?

I do not know. It is a cloud hosted on OVH infrastructure in France https://www.ovh.co.uk/public-cloud/instances/technologies/
But I do not know which hypervisor they use, sorry.

> * Does the problem affect only the jails, or also the host system?

The host didn't show any slowdown. Only within a jail the performance drop dramatically.

> * Does it only happen with NAT, or with routed packets as well?

In my case it happens with NAT, never tried routed packets.

HTH,
Michael

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: vnet NAT'd jails extremely slow, connection dies

Farhan Khan
In reply to this post by Kristof Provost
On Tue, Feb 5, 2019 at 12:58 PM Kristof Provost <[hidden email]> wrote:

>
> On 2019-02-05 18:47:23 (+0100), Michael Grimm <[hidden email]> wrote:
> > Farhan Khan <[hidden email]> wrote:
> > > On Mon, Feb 4, 2019 at 2:29 PM Farhan Khan <[hidden email]> wrote:
> >
> > >> I have a jail NAT'd to a base system, but the connection is extremely
> > >> slow and frequently disconnects drops, whereas the base is fine has
> > >> perfectly fine connectivity.
> > >>
> > >> My configuration is as follows:
> > >> vtnet0: Has routeable IPv4 address and 172.16.0.1/16
> > >> Jail uses epair4b, base has epair4a. Jail's IP is 172.16.0.5/16.
> > >> The base and jail can ping each other.
> > >> bridge0: contains vtnet0 and epair4a.
> > >>
> > >> I have gateway_enable="YES"
> > >> My pf.conf is as follows:
> > >> nat pass from 172.16.0.0/16 to any -> (vtnet0)
> > >>
> > >> When I try to run clamav, the connectivity stalls after a few minutes
> > >> and eventually disconnects. I ran tcpdump on the bridge and saw a lot
> > >> of HTTP seq and ack packets but no actual data. I am not using IPv6
> > >> yet.
> > >
> > > Just to provide more context to my previous email, outside of the jail
> > > I can download the FreeBSD ISO installer image at 3 MBps. Within the
> > > jail it drops to 12KBps.
> >
> > This sounds familiar to me ;-)
> >
> > Please have a look at https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html
> > Solution in https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049484.html
> >
> > I ended up with the following additions to /boot/loader.conf (and a subsequent reboot):
> >
> >       # needs to become turned off (LRO) in order to restore tcp performance within VNET jails:
> >       hw.vtnet.lro_disable="1"
> >       hw.vtnet.tso_disable="1"
> >
> Farhan has also solved his issue by turning off lro/tso. (We talked on
> IRC).
>
> I've not seen this issue myself, but I'm interested in a couple of
> points to hopefully pinpoint and maybe even fix the problem.
>
> These are questions for anyone who's running pf on top of a hypervisor
> and has vnet or other jails, and has seen slowdowns.
>
>  * What hypervisor are you running?
>  * Does the problem affect only the jails, or also the host system?
>  * Does it only happen with NAT, or with routed packets as well?
>
> If anyone is affected and not using pf that'd be interesting information
> as well.
>
> Regards,
> Kristof

Michael, thank you very much. This appears to do the trick, as Kristof
also directed me.

A. This was on a Vultr instance. Per they are using KVM, per a support ticket.
B. Just the Jail, not the post
C. I will have to get back to you on that, as I do not have a publicly
routeable IP to test on at the moment.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "[hidden email]"