CFT: netdump

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

CFT: netdump

Mark Johnston-2
Hi,

In user/markj/netdump I've been working on a feature which allows the
kernel to transmit kernel dumps over a network rather than writing to a
local storage device. This is convenient for VMs, diskless or embedded
devices, and devices with limited storage space. The server component of
netdump is a UDP daemon which is in ports as ftp/netdumpd.

The original code has been around for a long time and wasn't authored by
me. However, I've fixed a number of bugs, made it more reliable, and
integrated it with dumpon(8). I also largely rewrote the userland
daemon. At this point I'd like to propose committing it to HEAD.

The netdump(4) man page included in the patch lists the drivers
currently supported by netdump[*]. To be supported, a driver must be
augmented with a set of routines for use by the generic code. These
allow the generic code to transmit packets and poll for completions. I
added a mechanism to overwrite the mbuf/cluster UMA zone pointers at
panic time so that they instead fetch mbufs from a pre-allocated pool.
This way, existing driver code can be used without modification, and we
don't impose any runtime overhead in the regular mbuf allocation path.

Of course, the netdump data path is not fully reliable since the panic
may have interrupted a thread as it was processing tx/rx ring entries
and thus left them in an inconsistent state. In my experience this
hasn't proved to be an issue, and when testing I will typically have
several iperf streams running over the interface at panic time. That
said, the same issue exists in principle with local storage devices
as well.

A kernel dump will generally contain sensitive data. Thus, netdump
should only be used on trusted networks. It is possible to use the
kernel dump encryption feature (dumpon -k) with netdump, however I
am not certain that the encryption scheme used is well-suited for
this situation. Any feedback on this would be appreciated. Please note
that you need netdumpd-20180411 or later when using this feature, as the
daemon needs to accept and save the symmetric key in addition to the
dump itself and support for this isn't present in earlier versions.

A single large patch for the src tree is available here:
https://people.freebsd.org/~markj/patches/netdump/netdump-20180411.diff
Of course, the user/markj/netdump branch in SVN or its clone in the
GitHub mirror may be used as well. Documentation for configuration is in
the dumpon(8) man page. The gist of it is that one specifies some
additional network parameters to dumpon(8), along with an interface
name. For instance,

# dumpon -c 192.168.2.11 -s 192.168.2.1 vtnet0

tells the kernel to transmit to netdumpd at 192.168.2.1 using a source
address of 192.168.2.11. If there is a router between the two, the
gateway IP for the client must be specified as well, with -g.

I'm interested in any testing reports that folks may have, especially
with ixgbe or bnxt, or with NICs under load. If you have a favourite
NIC driver which does not yet support netdump, I'm happy to help add
support or review patches. Depending on how testing goes, I plan to
create reviews for various components of the patch over the next several
days.

[*] Currently, alc(4), bge(4), bxe(4), cxgb(4), mlx4en(4), re(4) and
vtnet(4) are supported. Drivers implemented using iflib are also
supported in principle, though I've only tested em(4) and igb(4).
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: CFT: netdump

Mark Johnston-2
On Wed, Apr 11, 2018 at 05:04:45PM -0400, Mark Johnston wrote:

> Hi,
>
> In user/markj/netdump I've been working on a feature which allows the
> kernel to transmit kernel dumps over a network rather than writing to a
> local storage device. This is convenient for VMs, diskless or embedded
> devices, and devices with limited storage space. The server component of
> netdump is a UDP daemon which is in ports as ftp/netdumpd.
>
> The original code has been around for a long time and wasn't authored by
> me. However, I've fixed a number of bugs, made it more reliable, and
> integrated it with dumpon(8). I also largely rewrote the userland
> daemon. At this point I'd like to propose committing it to HEAD.
>
> The netdump(4) man page included in the patch lists the drivers
> currently supported by netdump[*]. To be supported, a driver must be
> augmented with a set of routines for use by the generic code. These
> allow the generic code to transmit packets and poll for completions. I
> added a mechanism to overwrite the mbuf/cluster UMA zone pointers at
> panic time so that they instead fetch mbufs from a pre-allocated pool.
> This way, existing driver code can be used without modification, and we
> don't impose any runtime overhead in the regular mbuf allocation path.
>
> Of course, the netdump data path is not fully reliable since the panic
> may have interrupted a thread as it was processing tx/rx ring entries
> and thus left them in an inconsistent state. In my experience this
> hasn't proved to be an issue, and when testing I will typically have
> several iperf streams running over the interface at panic time. That
> said, the same issue exists in principle with local storage devices
> as well.
>
> A kernel dump will generally contain sensitive data. Thus, netdump
> should only be used on trusted networks. It is possible to use the
> kernel dump encryption feature (dumpon -k) with netdump, however I
> am not certain that the encryption scheme used is well-suited for
> this situation. Any feedback on this would be appreciated. Please note
> that you need netdumpd-20180411 or later when using this feature, as the
> daemon needs to accept and save the symmetric key in addition to the
> dump itself and support for this isn't present in earlier versions.
>
> A single large patch for the src tree is available here:
> https://people.freebsd.org/~markj/patches/netdump/netdump-20180411.diff
> Of course, the user/markj/netdump branch in SVN or its clone in the
> GitHub mirror may be used as well. Documentation for configuration is in
> the dumpon(8) man page. The gist of it is that one specifies some
> additional network parameters to dumpon(8), along with an interface
> name. For instance,
>
> # dumpon -c 192.168.2.11 -s 192.168.2.1 vtnet0
>
> tells the kernel to transmit to netdumpd at 192.168.2.1 using a source
> address of 192.168.2.11. If there is a router between the two, the
> gateway IP for the client must be specified as well, with -g.
>
> I'm interested in any testing reports that folks may have, especially
> with ixgbe or bnxt, or with NICs under load. If you have a favourite
> NIC driver which does not yet support netdump, I'm happy to help add
> support or review patches. Depending on how testing goes, I plan to
> create reviews for various components of the patch over the next several
> days.

I've created reviews for various components of the change:

https://reviews.freebsd.org/D15251 (post-panic mbuf allocator)
https://reviews.freebsd.org/D15252 (kernel dump refactoring for netdump)
https://reviews.freebsd.org/D15253 (core netdump client)
https://reviews.freebsd.org/D15254 (dumpon(8) extension for netdump)
https://reviews.freebsd.org/D15255 (alc(4) hooks)
https://reviews.freebsd.org/D15256 (bge(4) hooks)
https://reviews.freebsd.org/D15257 (bxe(4) hooks)
https://reviews.freebsd.org/D15258 (cxgb(4) hooks)
https://reviews.freebsd.org/D15259 (mlxen(4) hooks)
https://reviews.freebsd.org/D15260 (re(4) hooks)
https://reviews.freebsd.org/D15261 (vtnet(4) hooks)
https://reviews.freebsd.org/D15262 (iflib hooks)

If anyone is interested in reviewing some or all of these diffs, please
feel free to add yourself to the corresponding review. I plan to add
driver maintainers as reviewers to the corresponding driver changes.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"