ZFS read performance

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

ZFS read performance

markham breitbach
   Hi,
   I'm trying to figure out a ZFS read performance issue that I am seeing
   on FreeBSD9.2 (amd64).
   CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (2000.11-MHz
   K8-class CPU)
   real memory  = 4294967296 (4096 MB)
   avail memory = 4059762688 (3871 MB)
   I have an LSI 9240-8i controller.  (8ports @ 6GB/s ea, x8 PCIE 2.0)
   with the drives installed JBOD.  These are intended for use as a
   storage pool.  The main system drives are using the onboard SATA
   controller for the system.

   With a single drive formatted UFS, I can achieve ~ 175MBps Read and
   Write speeds using dd as follows:
   WRITE:
   dd if=/dev/zero of=/vol0/test bs=1m count=10000
   READ
   dd if=/vol0/test of=/dev/null
   When I create a zfs pool with a single drive:
   zpool create vol0 /dev/mfisyspd0
   using dd again, I can write ~175MBps, but my read speed is only about
   60MBps.
   I have set
   vfs.zfs.prefetch_disable = 0
   and this improves reads to about 70MBps, but no much beyond that.
   Now, this gets even more interesting.
   If I add a drive to the pool:
   zpool add vol0 /dev/mfisyspd1
   My write speeds increase to ~280MBps, but my read speed is still about
   60MBps.
   Using gstat, I can see that the load is shared pretty equally amongst
   the drives.
   This trend continues with the addition of a 3rd and 4th drive with
   write speeds ramping up to about 500MBps (I didn't test past that), but
   read speeds stuck around 60MBps.
   I have done similar testing under 9.1 making sure to align 4K sectors
   using the gnop trick (I am using WD RED 3TB AF drives) with no change.
   I have looked through the wiki and handbook for tuning and done some
   googling around, but I'm pretty much out of ideas now.  Everything I
   have read seems to indicate that ZFS should run fine on 4GB without any
   tuning, and at this point I haven't even begun any serious stress
   testing.
   Any ideas of where to go from here would be greatly appreciated.
   Thanks,
   -Markham

   ---
   [1]Markham Breitbach
   Network Operations
   SSi   People, Ideas, Technology
   - - - - - - - - - - - - - - - - - - - - -
   +1 867 669 7500 work
   +1 867 669 7510 fax
   [2][hidden email]
   [3]www.ssimicro.com
   356B Old Airport Road
   Yellowknife , NT X1A 3T4
   Canada
   - - - - - - - - - - - - - - - - - - - - -
   Visit some of our other networks
   [4]www.qiniq.com   &   [5]www.airware.ca

References

   1. http://www.ssimicro.com/
   2. mailto:[hidden email]
   3. http://www.ssimicro.com/
   4. http://www.qiniq.com/
   5. http://www.airware.ca/
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ZFS read performance

Steven Hartland
First off avoid using mfi its a RAID card which adds a extra layer
you dont want even in jbob mode.

Whats the mfi max_cmd set to? If you havent already set it to -1
/boot/loader.conf (hw.mfi.max_cmds=-1) which means controller max.

Next 4GB on a machine avoid it if possible, thats whats disabling
prefetch by default, I'd recommend a min of 8Gb more if posslbe.

When you "add" a disk your creating a stripe which is bad news if
any single disk fails, so avoid that at all costs.

Finally your issue with read performance is likely your block size,
you set it for write to 1mb (which is reasonable) but you didn't
on read so will be 512b which will likely cpu limit in the dd
process instead of bottlenecking on the disk / FS performance.

    Regards
    Steve
----- Original Message -----
From: "markham breitbach" <[hidden email]>
To: <[hidden email]>
Sent: Wednesday, January 29, 2014 12:02 AM
Subject: ZFS read performance


>   Hi,
>   I'm trying to figure out a ZFS read performance issue that I am seeing
>   on FreeBSD9.2 (amd64).
>   CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (2000.11-MHz
>   K8-class CPU)
>   real memory  = 4294967296 (4096 MB)
>   avail memory = 4059762688 (3871 MB)
>   I have an LSI 9240-8i controller.  (8ports @ 6GB/s ea, x8 PCIE 2.0)
>   with the drives installed JBOD.  These are intended for use as a
>   storage pool.  The main system drives are using the onboard SATA
>   controller for the system.
>
>   With a single drive formatted UFS, I can achieve ~ 175MBps Read and
>   Write speeds using dd as follows:
>   WRITE:
>   dd if=/dev/zero of=/vol0/test bs=1m count=10000
>   READ
>   dd if=/vol0/test of=/dev/null
>   When I create a zfs pool with a single drive:
>   zpool create vol0 /dev/mfisyspd0
>   using dd again, I can write ~175MBps, but my read speed is only about
>   60MBps.
>   I have set
>   vfs.zfs.prefetch_disable = 0
>   and this improves reads to about 70MBps, but no much beyond that.
>   Now, this gets even more interesting.
>   If I add a drive to the pool:
>   zpool add vol0 /dev/mfisyspd1
>   My write speeds increase to ~280MBps, but my read speed is still about
>   60MBps.
>   Using gstat, I can see that the load is shared pretty equally amongst
>   the drives.
>   This trend continues with the addition of a 3rd and 4th drive with
>   write speeds ramping up to about 500MBps (I didn't test past that), but
>   read speeds stuck around 60MBps.
>   I have done similar testing under 9.1 making sure to align 4K sectors
>   using the gnop trick (I am using WD RED 3TB AF drives) with no change.
>   I have looked through the wiki and handbook for tuning and done some
>   googling around, but I'm pretty much out of ideas now.  Everything I
>   have read seems to indicate that ZFS should run fine on 4GB without any
>   tuning, and at this point I haven't even begun any serious stress
>   testing.
>   Any ideas of where to go from here would be greatly appreciated.
>   Thanks,
>   -Markham
>
>   ---
>   [1]Markham Breitbach
>   Network Operations
>   SSi   People, Ideas, Technology
>   - - - - - - - - - - - - - - - - - - - - -
>   +1 867 669 7500 work
>   +1 867 669 7510 fax
>   [2][hidden email]
>   [3]www.ssimicro.com
>   356B Old Airport Road
>   Yellowknife , NT X1A 3T4
>   Canada
>   - - - - - - - - - - - - - - - - - - - - -
>   Visit some of our other networks
>   [4]www.qiniq.com   &   [5]www.airware.ca
>
> References
>
>   1. http://www.ssimicro.com/
>   2. mailto:[hidden email]
>   3. http://www.ssimicro.com/
>   4. http://www.qiniq.com/
>   5. http://www.airware.ca/
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "[hidden email]"
>

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to [hidden email].

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: ZFS read performance

markham breitbach
Hi Steve,

At this point, I am just doing some evaluation with what's on-hand.  Ultimately I would
add RAM and redundancy, etc.

bs was definitely a big part of the issue.  (*forehead slap*)

hw-mfi.max_cmds=-1 also make a significant impact.

Onwards to some more strenuous testing...

Thanks for the help!

-Markham


On 14-01-28 6:34 PM, Steven Hartland wrote:

> First off avoid using mfi its a RAID card which adds a extra layer
> you dont want even in jbob mode.
>
> Whats the mfi max_cmd set to? If you havent already set it to -1
> /boot/loader.conf (hw.mfi.max_cmds=-1) which means controller max.
>
> Next 4GB on a machine avoid it if possible, thats whats disabling
> prefetch by default, I'd recommend a min of 8Gb more if posslbe.
>
> When you "add" a disk your creating a stripe which is bad news if
> any single disk fails, so avoid that at all costs.
>
> Finally your issue with read performance is likely your block size,
> you set it for write to 1mb (which is reasonable) but you didn't
> on read so will be 512b which will likely cpu limit in the dd
> process instead of bottlenecking on the disk / FS performance.
>
>    Regards
>    Steve
> ----- Original Message ----- From: "markham breitbach" <[hidden email]>
> To: <[hidden email]>
> Sent: Wednesday, January 29, 2014 12:02 AM
> Subject: ZFS read performance
>
>
>>   Hi,
>>   I'm trying to figure out a ZFS read performance issue that I am seeing
>>   on FreeBSD9.2 (amd64).
>>   CPU: Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz (2000.11-MHz
>>   K8-class CPU)
>>   real memory  = 4294967296 (4096 MB)
>>   avail memory = 4059762688 (3871 MB)
>>   I have an LSI 9240-8i controller.  (8ports @ 6GB/s ea, x8 PCIE 2.0)
>>   with the drives installed JBOD.  These are intended for use as a
>>   storage pool.  The main system drives are using the onboard SATA
>>   controller for the system.
>>
>>   With a single drive formatted UFS, I can achieve ~ 175MBps Read and
>>   Write speeds using dd as follows:
>>   WRITE:
>>   dd if=/dev/zero of=/vol0/test bs=1m count=10000
>>   READ
>>   dd if=/vol0/test of=/dev/null
>>   When I create a zfs pool with a single drive:
>>   zpool create vol0 /dev/mfisyspd0
>>   using dd again, I can write ~175MBps, but my read speed is only about
>>   60MBps.
>>   I have set
>>   vfs.zfs.prefetch_disable = 0
>>   and this improves reads to about 70MBps, but no much beyond that.
>>   Now, this gets even more interesting.
>>   If I add a drive to the pool:
>>   zpool add vol0 /dev/mfisyspd1
>>   My write speeds increase to ~280MBps, but my read speed is still about
>>   60MBps.
>>   Using gstat, I can see that the load is shared pretty equally amongst
>>   the drives.
>>   This trend continues with the addition of a 3rd and 4th drive with
>>   write speeds ramping up to about 500MBps (I didn't test past that), but
>>   read speeds stuck around 60MBps.
>>   I have done similar testing under 9.1 making sure to align 4K sectors
>>   using the gnop trick (I am using WD RED 3TB AF drives) with no change.
>>   I have looked through the wiki and handbook for tuning and done some
>>   googling around, but I'm pretty much out of ideas now.  Everything I
>>   have read seems to indicate that ZFS should run fine on 4GB without any
>>   tuning, and at this point I haven't even begun any serious stress
>>   testing.
>>   Any ideas of where to go from here would be greatly appreciated.
>>   Thanks,
>>   -Markham
>>
>>   ---
>>   [1]Markham Breitbach
>>   Network Operations
>>   SSi   People, Ideas, Technology
>>   - - - - - - - - - - - - - - - - - - - - -
>>   +1 867 669 7500 work
>>   +1 867 669 7510 fax
>>   [2][hidden email]
>>   [3]www.ssimicro.com
>>   356B Old Airport Road
>>   Yellowknife , NT X1A 3T4
>>   Canada
>>   - - - - - - - - - - - - - - - - - - - - -
>>   Visit some of our other networks
>>   [4]www.qiniq.com   &   [5]www.airware.ca
>>
>> References
>>
>>   1. http://www.ssimicro.com/
>>   2. mailto:[hidden email]
>>   3. http://www.ssimicro.com/
>>   4. http://www.qiniq.com/
>>   5. http://www.airware.ca/
>> _______________________________________________
>> [hidden email] mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>> To unsubscribe, send any mail to "[hidden email]"
>>
>
> ================================================
> This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or
> entity to whom it is addressed. In the event of misdirection, the recipient is
> prohibited from using, copying, printing or otherwise disseminating it or any
> information contained in it.
> In the event of misdirection, illegible or incomplete transmission please telephone +44
> 845 868 1337
> or return the E.mail to [hidden email].
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"