Disk Throughput test

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Disk Throughput test

ssanders
FreeBSD 6.3
Dual Quad Core Xeon 5450@3Ghz
16GB RAM
3Ware 9650SE-ML / 256MB cache
14 Seagate 750GB/7200RPM/ST375033ONS SATA drives

We have an application that is streaming data to disk at the maximum
rate the controller can sustain.  The controller should be able to
develop something on the order of 600MB/s but we're only getting 450MB/s.

The application writes about 50GB into a file, closes the file, and then
starts another file.

I've set all of the recommended settings from 3ware.  I have tried
various settings for vfs.hirunningspace - various settings include
everything from 5MB to the cache size of the card (256MB).

Does anyone know if there is some file system performance limit?

Thanks

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk Throughput test

Chuck Swiger-2
On Oct 23, 2008, at 8:27 AM, Stephen Sanders wrote:
> We have an application that is streaming data to disk at the maximum
> rate the controller can sustain.  The controller should be able to
> develop something on the order of 600MB/s but we're only getting  
> 450MB/s.

Are you using RAID-5 or RAID-10 config?  RAID-5 is best suited for  
read-only or read-mostly volumes; if you are doing an even mix  
RAID-1/10 will generally perform better.

> The application writes about 50GB into a file, closes the file, and  
> then
> starts another file.
>
> I've set all of the recommended settings from 3ware.  I have tried
> various settings for vfs.hirunningspace - various settings include
> everything from 5MB to the cache size of the card (256MB).
>
> Does anyone know if there is some file system performance limit?

You can also try running diskinfo -t to look for raw performance  
numbers; if they are much different from the bandwidth your app is  
getting, perhaps you need to tune your app...

Regards,
--
-Chuck

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk Throughput test

Stephen Sanders-2
Good point about the RAID.  It is set for RAID 5 as the data is supposed
to be protected.

Interestingly enough, diskinfo is telling me that the drive throughput
is 175MB/s.  I'm guessing that this means diskinfo is a mixture of reads
and writes?

Thanks

Chuck Swiger wrote:

> On Oct 23, 2008, at 8:27 AM, Stephen Sanders wrote:
>> We have an application that is streaming data to disk at the maximum
>> rate the controller can sustain.  The controller should be able to
>> develop something on the order of 600MB/s but we're only getting
>> 450MB/s.
>
> Are you using RAID-5 or RAID-10 config?  RAID-5 is best suited for
> read-only or read-mostly volumes; if you are doing an even mix
> RAID-1/10 will generally perform better.
>
>> The application writes about 50GB into a file, closes the file, and then
>> starts another file.
>>
>> I've set all of the recommended settings from 3ware.  I have tried
>> various settings for vfs.hirunningspace - various settings include
>> everything from 5MB to the cache size of the card (256MB).
>>
>> Does anyone know if there is some file system performance limit?
>
> You can also try running diskinfo -t to look for raw performance
> numbers; if they are much different from the bandwidth your app is
> getting, perhaps you need to tune your app...
>
> Regards,

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk Throughput test

Josh Paetzel-2
In reply to this post by ssanders
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stephen Sanders wrote:

> FreeBSD 6.3
> Dual Quad Core Xeon 5450@3Ghz
> 16GB RAM
> 3Ware 9650SE-ML / 256MB cache
> 14 Seagate 750GB/7200RPM/ST375033ONS SATA drives
>
> We have an application that is streaming data to disk at the maximum
> rate the controller can sustain.  The controller should be able to
> develop something on the order of 600MB/s but we're only getting 450MB/s.
>
> The application writes about 50GB into a file, closes the file, and then
> starts another file.
>
> I've set all of the recommended settings from 3ware.  I have tried
> various settings for vfs.hirunningspace - various settings include
> everything from 5MB to the cache size of the card (256MB).
>
> Does anyone know if there is some file system performance limit?
>
> Thanks
>

3ware has recently released new firmware for the 9650 and 9690 cards
that makes a huge difference in performance.  I've noticed anywhere from
10-25% increases in real world disk I/O performance.

There is no "file system performance limit" that magically caps
sequential read/write speed to some arbitrary value.  A lot of factors
come together to give you a max speed.

Obviously you are hitting a bottleneck somewhere that is lower than you
think it should be.

Does 3ware advertise the card is capable of 600 Megs/sec?  It seems an
8x pci-e card would be capable of higher performance than that.

- --
Thanks,

Josh Paetzel

PGP: 8A48 EF36 5E9F 4EDA 5ABC 11B4 26F9 01F1 27AF AECB
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)

iD8DBQFJALr3JvkB8SevrssRAqleAJ9oJCRkv9jOVwxCqYDlfaSj+blweACdFsBP
TUEcv2yhQ3CBk/tLC5JmyJs=
=h/mH
-----END PGP SIGNATURE-----
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk Throughput test

Chuck Swiger-2
In reply to this post by Stephen Sanders-2
On Oct 23, 2008, at 10:50 AM, Stephen Sanders wrote:
> Good point about the RAID.  It is set for RAID 5 as the data is  
> supposed
> to be protected.

RAID-10 provides somewhat better data protection, but less available  
space and better write performance especially for small writes.  (For  
big writes, RAID-5 can do OK also, but small writes tend to really  
degrade RAID-5 performance, down to an order of magnitude *less*  
throughput compared with a single raw drive.)

> Interestingly enough, diskinfo is telling me that the drive throughput
> is 175MB/s.  I'm guessing that this means diskinfo is a mixture of  
> reads
> and writes?

No, it should only be doing reads.  For best-case read performance,  
it's likely to be at or near:

   dd if=_your_disks_ of=/dev/null bs=_raid_stripe_size_

...or adjust bs up or down by factors of 2 and benchmark.

Regards,
--
-Chuck

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Disk Throughput test

Ivan Voras
In reply to this post by ssanders
Stephen Sanders wrote:
> FreeBSD 6.3
> Dual Quad Core Xeon 5450@3Ghz

FreeBSD 6.3 isn't very suited for your CPU. If your workload isn't
completely CPU bound (i.e. if isn't Seti@Home), you will not only not
make use of all 8 CPU cores but will probably get worse performance with
8 CPUs than with 2 CPUs. Upgrade to at least 7.0 and you'll probably
need 7.1-BETA2 to maximally use your CPU.

But if you don't have much IO contention it's probably not related to
your problem.

> 16GB RAM

You're using the AMD64 version, right?

> 3Ware 9650SE-ML / 256MB cache
> 14 Seagate 750GB/7200RPM/ST375033ONS SATA drives

Which RAID mode? What is the total size of your array?

> We have an application that is streaming data to disk at the maximum
> rate the controller can sustain.  The controller should be able to
> develop something on the order of 600MB/s but we're only getting 450MB/s.

Can you use something like bonnie++ to test your configuration?


signature.asc (266 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Disk Throughput test

Stephen Sanders-2
Good point.  I plan to switch to 7.0, though the system is running the
ULE scheduler.

The cpu's are amd64.  I tested early on with bonnie++ and was getting
similar numbers to the application test.  It's a good idea to try it
again so I will.

The RAID is a RAID 6 as the application calls for data integrity though
I do plan on doing a RAID 0 test as well.  The array is something like
7TB total.

Thanks.

On Thu, 2008-10-23 at 22:16 +0200, Ivan Voras wrote:

> Stephen Sanders wrote:
> > FreeBSD 6.3
> > Dual Quad Core Xeon 5450@3Ghz
>
> FreeBSD 6.3 isn't very suited for your CPU. If your workload isn't
> completely CPU bound (i.e. if isn't Seti@Home), you will not only not
> make use of all 8 CPU cores but will probably get worse performance with
> 8 CPUs than with 2 CPUs. Upgrade to at least 7.0 and you'll probably
> need 7.1-BETA2 to maximally use your CPU.
>
> But if you don't have much IO contention it's probably not related to
> your problem.
>
> > 16GB RAM
>
> You're using the AMD64 version, right?
>
> > 3Ware 9650SE-ML / 256MB cache
> > 14 Seagate 750GB/7200RPM/ST375033ONS SATA drives
>
> Which RAID mode? What is the total size of your array?
>
> > We have an application that is streaming data to disk at the maximum
> > rate the controller can sustain.  The controller should be able to
> > develop something on the order of 600MB/s but we're only getting 450MB/s.
>
> Can you use something like bonnie++ to test your configuration?
>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"