Re: How to use the DMA Engine in FreeBSD?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Re: How to use the DMA Engine in FreeBSD?

Rajesh Kumar

Thanks everyone for your inputs. I assume, "PCI busmaster"ing refers to a
peripheral device being made as the owner of the bus by the DMA engine and
do data transfer.  Please correct me if I am wrong. But, will this bus_dma
API uses the DMA engine in hardware (does it support PCI busmastering)?

As I understand, DMA engine will expose DMA channels, which can be used by
the client drivers to queue their DMA requests (without CPU intervention)
and completions are given through the callback functions. If this is
considered the real DMA operation, I don't see options to request DMA
channels using bus_dma API's, and the callbacks in bus_dma API is meant to
receive just the DMA map details (doesn't indicate a DMA completion).
ioat(4) driver seems to be doing what I described above, but that seems to
be Intel specific. Can it be used with any DMA controllers?

So, I am wondering what needs to be done in FreeBSD to do the actual DMA
involving DMA engine in a more generic way?


On Sat, Dec 15, 2018 at 6:30 AM Conrad Meyer <[hidden email]> wrote:

> On Fri, Dec 14, 2018 at 8:17 AM Alan Somers <[hidden email]> wrote:
> > > For some Intel based server hardware, there is the "ioat" driver, which
> > > allows for user code to schedule DMA operations.  See ioat(4) for
> > > details, including a pointer to the test program.
> This isn't true (or at least, only for testing).  It's only usable by
> the kernel at the moment.
> > * In what context are callbacks called?  Are they called from a signal
> > handler, or in a separate thread, or something else?
> Directly from the ithread, mostly.  Callers can't take sleep locks or
> do very much besides set a flag and wakeup, or kick off a callout or
> taskqueue.
> > * Why isn't ioat.h installed?
> Why would it be?  It's a kernel interface.
> > * Are "interrupts" synonymous with callbacks?
> I don't understand the question.  Interrupts are synonymous with
> interrupts.  Like, MSI-X.
> > * Do you have a rough idea for about the minimum buffer size that
> > makes sense to use with ioat?
> What makes sense will depend on your use case.  It's not necessarily
> faster than CPU memcpy, but it may be worth some slowdown to offload
> latency-insensitive bulk copies.  (It may make a lot of sense for
> asynchronous page zero, if someone wants to bring that back.)  It's
> worth benchmarking your specific model.
> We use it for >= 8kB copies on Broadwell, although that's largely an
> artifact of our use case (write sizes are exactly 8 bytes, 512 bytes,
> or multiples of 8kB).  IIRC, it shows up as a gen3 PCIe device with
> some small number of lanes on Broadwell, although I may be mistaken;
> and it may not communicate with RAM over the PCIe bus anyway, given
> the tight integration with the CPU.
> Best,
> Conrad
> _______________________________________________
> [hidden email] mailing list
> To unsubscribe, send any mail to "[hidden email]"
[hidden email] mailing list
To unsubscribe, send any mail to "[hidden email]"