Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

Rajesh Kumar
Hi,

I am trying to run FIO benchmark test with NVMe devices and see how FreeBSD
performs. There are lot of variables and combination. So, can anyone
suggest a Ideal way to do FIO benchmarking in FreeBSD? My intent is to
check what is the maximum throughput and IOPS the device delivers in
FreeBSD.

Few questions regarding the same,

   1. Should we use "posixaio" as the ioengine (or) something else?
   2. Should we use single thread (or) multiple threads for test? If
   multiple threads, how can we decide on the optimal thread count?
   3. Should we use "raw device files" (Eg: nvme namespace file -
   /dev/nvme0ns1) without filesystem (or) use a mounted filesystem with a
   regular file (Eg: /mnt/nvme/test1). Looks like raw device files give better
   numbers.
   4. Should we use a shared file (or) one file per thread?
   5. I believe 1Job should be fine for benchmarking. (or) should we try
   multiple jobs?

Please let me know your suggestions. Also, please suggest performance
tuning methods for NVMe and storage devices in general.

Thanks,
Rajesh.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

Enji Cooper

> On Feb 22, 2019, at 12:51 AM, Rajesh Kumar <[hidden email]> wrote:
>
> Hi,
>
> I am trying to run FIO benchmark test with NVMe devices and see how FreeBSD
> performs. There are lot of variables and combination. So, can anyone
> suggest a Ideal way to do FIO benchmarking in FreeBSD? My intent is to
> check what is the maximum throughput and IOPS the device delivers in
> FreeBSD.
>
> Few questions regarding the same,
>
>   1. Should we use "posixaio" as the ioengine (or) something else?
>   2. Should we use single thread (or) multiple threads for test? If
>   multiple threads, how can we decide on the optimal thread count?
>   3. Should we use "raw device files" (Eg: nvme namespace file -
>   /dev/nvme0ns1) without filesystem (or) use a mounted filesystem with a
>   regular file (Eg: /mnt/nvme/test1). Looks like raw device files give better
>   numbers.
>   4. Should we use a shared file (or) one file per thread?
>   5. I believe 1Job should be fine for benchmarking. (or) should we try
>   multiple jobs?
>
> Please let me know your suggestions. Also, please suggest performance
> tuning methods for NVMe and storage devices in general.

Hi Rajesh,
        Is there a data sheet for the NVMe device?
Cheers,
-Enji
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

Rajesh Kumar
Hi Enji Cooper,

I am using Samsung 960 PRO

https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/960pro/


On Fri, Feb 22, 2019 at 2:36 PM Enji Cooper <[hidden email]> wrote:

>
> > On Feb 22, 2019, at 12:51 AM, Rajesh Kumar <[hidden email]> wrote:
> >
> > Hi,
> >
> > I am trying to run FIO benchmark test with NVMe devices and see how
> FreeBSD
> > performs. There are lot of variables and combination. So, can anyone
> > suggest a Ideal way to do FIO benchmarking in FreeBSD? My intent is to
> > check what is the maximum throughput and IOPS the device delivers in
> > FreeBSD.
> >
> > Few questions regarding the same,
> >
> >   1. Should we use "posixaio" as the ioengine (or) something else?
> >   2. Should we use single thread (or) multiple threads for test? If
> >   multiple threads, how can we decide on the optimal thread count?
> >   3. Should we use "raw device files" (Eg: nvme namespace file -
> >   /dev/nvme0ns1) without filesystem (or) use a mounted filesystem with a
> >   regular file (Eg: /mnt/nvme/test1). Looks like raw device files give
> better
> >   numbers.
> >   4. Should we use a shared file (or) one file per thread?
> >   5. I believe 1Job should be fine for benchmarking. (or) should we try
> >   multiple jobs?
> >
> > Please let me know your suggestions. Also, please suggest performance
> > tuning methods for NVMe and storage devices in general.
>
> Hi Rajesh,
>         Is there a data sheet for the NVMe device?
> Cheers,
> -Enji
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

freebsd-hackers mailing list
In reply to this post by Rajesh Kumar
On 2/22/19 1:51 AM, Rajesh Kumar wrote:

>     1. Should we use "posixaio" as the ioengine (or) something else?
>     2. Should we use single thread (or) multiple threads for test? If
>     multiple threads, how can we decide on the optimal thread count?
>     3. Should we use "raw device files" (Eg: nvme namespace file -
>     /dev/nvme0ns1) without filesystem (or) use a mounted filesystem with a
>     regular file (Eg: /mnt/nvme/test1). Looks like raw device files give better
>     numbers.
>     4. Should we use a shared file (or) one file per thread?
>     5. I believe 1Job should be fine for benchmarking. (or) should we try
>     multiple jobs?


I just ran a quick test on a filesystem on my machine which has an M.2
NVMe drive, and it seems posixaio performs pretty poorly compared to the
sync ioengine: around 700 MB/s vs. 1100 MB/s!

I _was_ going to suggest using posixaio and setting iodepth to something
like 32, but since it performs badly I'd suggest playing around with the
numjobs parameter and seeing where the best performance is achieved -
whether that's latency or throughput.


On my system, single-threaded achieves ~530 MB/s, 8 jobs/threads 1150
MB/s and 32 1840 MB/s with a 4 KB block size.

Bumping the block size from 4 KB to 16 KB makes the throughput more
jumpy, but appears to average 2300 MB/s when used with 32 jobs.


--
Rebecca Cran

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

Alan Somers-2
On Fri, Feb 22, 2019 at 5:29 AM Rebecca Cran via freebsd-hackers
<[hidden email]> wrote:

>
> On 2/22/19 1:51 AM, Rajesh Kumar wrote:
> >     1. Should we use "posixaio" as the ioengine (or) something else?
> >     2. Should we use single thread (or) multiple threads for test? If
> >     multiple threads, how can we decide on the optimal thread count?
> >     3. Should we use "raw device files" (Eg: nvme namespace file -
> >     /dev/nvme0ns1) without filesystem (or) use a mounted filesystem with a
> >     regular file (Eg: /mnt/nvme/test1). Looks like raw device files give better
> >     numbers.
> >     4. Should we use a shared file (or) one file per thread?
> >     5. I believe 1Job should be fine for benchmarking. (or) should we try
> >     multiple jobs?
>
>
> I just ran a quick test on a filesystem on my machine which has an M.2
> NVMe drive, and it seems posixaio performs pretty poorly compared to the
> sync ioengine: around 700 MB/s vs. 1100 MB/s!

When AIO is run on a filesystem, it uses an internal thread pool to
process requests.  But if you run it on a bare drive, then the I/O is
direct and should be faster than the sync ioengine.

-Alan

>
> I _was_ going to suggest using posixaio and setting iodepth to something
> like 32, but since it performs badly I'd suggest playing around with the
> numjobs parameter and seeing where the best performance is achieved -
> whether that's latency or throughput.
>
>
> On my system, single-threaded achieves ~530 MB/s, 8 jobs/threads 1150
> MB/s and 32 1840 MB/s with a 4 KB block size.
>
> Bumping the block size from 4 KB to 16 KB makes the throughput more
> jumpy, but appears to average 2300 MB/s when used with 32 jobs.
>
>
> --
> Rebecca Cran
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

Warner Losh
On Fri, Feb 22, 2019 at 8:04 AM Alan Somers <[hidden email]> wrote:

> On Fri, Feb 22, 2019 at 5:29 AM Rebecca Cran via freebsd-hackers
> <[hidden email]> wrote:
> >
> > On 2/22/19 1:51 AM, Rajesh Kumar wrote:
> > >     1. Should we use "posixaio" as the ioengine (or) something else?
> > >     2. Should we use single thread (or) multiple threads for test? If
> > >     multiple threads, how can we decide on the optimal thread count?
> > >     3. Should we use "raw device files" (Eg: nvme namespace file -
> > >     /dev/nvme0ns1) without filesystem (or) use a mounted filesystem
> with a
> > >     regular file (Eg: /mnt/nvme/test1). Looks like raw device files
> give better
> > >     numbers.
> > >     4. Should we use a shared file (or) one file per thread?
> > >     5. I believe 1Job should be fine for benchmarking. (or) should we
> try
> > >     multiple jobs?
> >
> >
> > I just ran a quick test on a filesystem on my machine which has an M.2
> > NVMe drive, and it seems posixaio performs pretty poorly compared to the
> > sync ioengine: around 700 MB/s vs. 1100 MB/s!
>
> When AIO is run on a filesystem, it uses an internal thread pool to
> process requests.  But if you run it on a bare drive, then the I/O is
> direct and should be faster than the sync ioengine.
>


Here's the script I typically use to get first pass raw numbers. The key to
getting good benchmark numbers is getting as many I/O requests on the
device as possible. At present, posixaio is the only thing outside of the
kernel that can do that. I usually get close to datasheet numbers using
this test:

; SSD testing: 128k I/O 64 jobs 32 deep queue

[global]
direct=1
rw=randread
refill_buffers
norandommap
randrepeat=0
bs=128k
ioengine=posixaio
iodepth=128
numjobs=64
runtime=30
group_reporting
thread

[nvme128k]

And I agree with Alan: to get the best numbers, you need to go to the raw
device. I've not cared about from userland I/O performance, so I've not
looked at making the UFS or ZFS cases work better in fio.

Warner

>
> > I _was_ going to suggest using posixaio and setting iodepth to something
> > like 32, but since it performs badly I'd suggest playing around with the
> > numjobs parameter and seeing where the best performance is achieved -
> > whether that's latency or throughput.
> >
> >
> > On my system, single-threaded achieves ~530 MB/s, 8 jobs/threads 1150
> > MB/s and 32 1840 MB/s with a 4 KB block size.
> >
> > Bumping the block size from 4 KB to 16 KB makes the throughput more
> > jumpy, but appears to average 2300 MB/s when used with 32 jobs.
> >
> >
> > --
> > Rebecca Cran
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> > To unsubscribe, send any mail to "
> [hidden email]"
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

Enji Cooper
In reply to this post by Rajesh Kumar

> On Feb 22, 2019, at 01:29, Rajesh Kumar <[hidden email]> wrote:
>
> Hi Enji Cooper,
>
> I am using Samsung 960 PRO
>
> https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/960pro/

Hi Rajesh,
    I asked about the datasheet, because there might be some hints in terms of the number of parallel jobs you might want to apply as well as the I/O queue depth, which will affect the performance of the drive. Otherwise you’ll be throwing values against a wall, seeing what will stick, which is sort of ok if you bound your testing and adjust based on performance, but if your initial hunch is off, it can mislead you.
    Similarly, check to see if there are any tunables or sysctls that will bound the device in terms of the queue depth and I/O requests.
    As others have noted, test the device directly if you want to know its raw performance. Only test with a filesystem if your intent is to see how the device will perform with a filesystem on it (and disable filesystem sync if you want to test max throughput with the overhead of a filesystem). Testing with a filesystem can reveal some potentially interesting characteristics, in terms of limitations in VFS / the filesystem implementation, which might be helpful if you’re trying to determine why there’s a difference between raw device speed and the speed with a filesystem on it. Testing with the same file in different directories is ok, as long as you blow the drive cache—which will have a noticeable performance impact on conventional (PMR, SMR, etc) drives, as you want to test the worst case speed of the device, instead of the cache. It should matter a bit less with faster devices like SSDs/NVMe drives. Testing with files in the same lateral filesystem hierarchy (same directory), might reveal issues with filesystem locking (directory inode performance), but that shouldn’t be the primary goal of your testing. It’s just something to keep in mind.
    Happy testing!
HTH,
-Enji
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Any ideal way to run FIO benchmarking for NVMEe devices in FreeBSD

Rajesh Kumar
Thanks Enji Cooper, Warner Losh, Alan Somers and Rebecca Cran for your
Inputs.

I did some test and here's my observation.
1) iogine "posixaio" (or) "sync" - no much difference. Is this expected?
2) As said, raw device gives better numbers compared to filesystem.
3) With "thread" option, I see the numbers going considerably down. Is this
expected?
4) I see good numbers for IOPS, but the MB/s is less comparatively. Any
reasons?

Enji Cooper, the mentioned link has all the documents (data sheet, user
guide etc.,). Not sure if you have seen them. Anyway, I will consider about
the points you have mentioned in my testing and see how they impact the
performance. Thanks.

Thanks,
Rajesh.

On Fri, Feb 22, 2019 at 9:24 PM Enji Cooper <[hidden email]> wrote:

>
> On Feb 22, 2019, at 01:29, Rajesh Kumar <[hidden email]> wrote:
>
> Hi Enji Cooper,
>
> I am using Samsung 960 PRO
>
> https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/960pro/
>
>
> Hi Rajesh,
>     I asked about the datasheet, because there might be some hints in
> terms of the number of parallel jobs you might want to apply as well as the
> I/O queue depth, which will affect the performance of the drive. Otherwise
> you’ll be throwing values against a wall, seeing what will stick, which is
> sort of ok if you bound your testing and adjust based on performance, but
> if your initial hunch is off, it can mislead you.
>     Similarly, check to see if there are any tunables or sysctls that will
> bound the device in terms of the queue depth and I/O requests.
>     As others have noted, test the device directly if you want to know its
> raw performance. Only test with a filesystem if your intent is to see how
> the device will perform with a filesystem on it (and disable filesystem
> sync if you want to test max throughput with the overhead of a filesystem).
> Testing with a filesystem can reveal some potentially interesting
> characteristics, in terms of limitations in VFS / the filesystem
> implementation, which might be helpful if you’re trying to determine why
> there’s a difference between raw device speed and the speed with a
> filesystem on it. Testing with the same file in different directories is
> ok, as long as you blow the drive cache—which will have a noticeable
> performance impact on conventional (PMR, SMR, etc) drives, as you want to
> test the worst case speed of the device, instead of the cache. It should
> matter a bit less with faster devices like SSDs/NVMe drives. Testing with
> files in the same lateral filesystem hierarchy (same directory), might
> reveal issues with filesystem locking (directory inode performance), but
> that shouldn’t be the primary goal of your testing. It’s just something to
> keep in mind.
>     Happy testing!
> HTH,
> -Enji
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"