7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

alan bryan-4
Hi,

I've got a new server with a 3ware 9550SXU with the
Battery.  I am using FreeBSD 7.0-Release (tried both
4BSD and ULE) using AMD64 and the 3ware performance
for writes is just plain horrible.  Something is
obviously wrong but I'm not sure what.

I've got a 4 disk RAID 10 array.

According to 3dm2 the cache is on.  I even tried
setting The StorSave preference to "Performance" with
no real benefit.  There seems to be something really
wrong with disk performance.  Here's the results from
bonnie:

File './Bonnie.2551', size: 104857600
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start
'em...done...done...done...
             -------Sequential Output--------
---Sequential Input-- --Random--
             -Per Char- --Block--- -Rewrite-- -Per
Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec
%CPU K/sec %CPU  /sec %CPU
         100  9989  4.8  6739  1.0 18900  7.8 225973
98.5 1914662
99.9 177210.7 259.7

Any ideas?  Anybody have one of these that's working
with FreeBSD 7?


      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page.
http://www.yahoo.com/r/hs
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

Lamont Lucas
alan bryan wrote:
> Hi,
>
> I've got a new server with a 3ware 9550SXU with the
> Battery.  I am using FreeBSD 7.0-Release (tried both
> 4BSD and ULE) using AMD64 and the 3ware performance
> for writes is just plain horrible.  Something is
> obviously wrong but I'm not sure what.

Hello Alan.

Have you ever used this card with any other OS?  I have about 6 of them
running under linux and I too get horrible write performance from them.  
After several days of tuning and research, I've concluded that the cards
are just stinkers reguardless of what OS they are running.   That seems
to be the consensus if you google 3ware write performance.  I've put
them in 64 and 32 bit slots, various speeds and with various drives.

I had an older 3ware card running under FreeBSD 6.2 that would give
decent read performance in a raid0 configuration but terrible write
performance.  I performed no tuning on that setup, as my application was
mostly read and it was "good enough" for the time.

Not terribly helpful, I admit, but I wanted you to at least think about
it being the card rather than the OS.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

mdtancsa
In reply to this post by alan bryan-4
At 12:50 PM 3/4/2008, alan bryan wrote:
>Hi,
>
>I've got a new server with a 3ware 9550SXU with the
>Battery.  I am using FreeBSD 7.0-Release (tried both
>4BSD and ULE) using AMD64 and the 3ware performance
>for writes is just plain horrible.  Something is
>obviously wrong but I'm not sure what.

Not sure about 7.0, but I have this card on a 6.3 box.  Doing
something simple like

% cat /dev/zero > big

% iostat -c 100
       tty             da0            pass0             cpu
  tin tout  KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
    0    0 53.92   6  0.31   0.00   0  0.00   3  0  0  0 96
    3  197  4.00   1  0.00   0.00   0  0.00   0  0  0  0 100
    1   64 127.94 1100 137.42   0.00   0  0.00   0  0 49  5 46
    0   63 128.00 1012 126.50   0.00   0  0.00   2  0 34  5 59
    0   63 128.00 969 121.13   0.00   0  0.00   1  0 30  2 66
    0   62 127.85 758 94.67   0.00   0  0.00   1  0 26  4 69
    0   61 127.82 1252 156.25   0.00   0  0.00   1  0 50  8 42
    0   63 127.59 542 67.59   0.00   0  0.00   1  0 26  1 72
    0   61 127.66 1026 127.90   0.00   0  0.00   0  0 47  5 49
    1   87 127.56 513 63.97   0.00   0  0.00   0  0 19  4 77
    0   61  0.00   0  0.00   0.00   0  0.00   0  0  0  0 100


Shows pretty OK write performance.

% dd if=/dev/zero of=/var/tmp/big bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 8.646134 secs (121276863 bytes/sec)


This is on RAID10 on 4 Segate ST380811AS drives.

3ware device driver for 9000 series storage controllers, version: 3.60.04.003
twa0: <3ware 9000 series Storage Controller> port 0xac00-0xac3f mem
0xf4000000-0xf5ffffff,0xff2ff000-0xff2fffff irq 16 at device 3.0 on pci2
twa0: [GIANT-LOCKED]
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-4LP, 4
ports, Firmware FE9X 3.01.01.028, BIOS BE9X 3.01.00.024
da0: <AMCC 9550SX-4LP DISK 3.01> Fixed Direct Access SCSI-3 device
da0: 100.000MB/s transfers
da0: 152566MB (312455168 512 byte sectors: 255H 63S/T 19449C)

I have write cache enabled and performance set for the StorSav. Its
in a PCI-x slot as well.

         ---Mike

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

Boris Samorodov
In reply to this post by alan bryan-4
On Tue, 4 Mar 2008 09:50:23 -0800 (PST) alan bryan wrote:

> I've got a new server with a 3ware 9550SXU with the
> Battery.  I am using FreeBSD 7.0-Release (tried both
> 4BSD and ULE) using AMD64 and the 3ware performance
> for writes is just plain horrible.  Something is
> obviously wrong but I'm not sure what.

> I've got a 4 disk RAID 10 array.

> According to 3dm2 the cache is on.  I even tried
> setting The StorSave preference to "Performance" with
> no real benefit.  There seems to be something really
> wrong with disk performance.  Here's the results from
> bonnie:

> File './Bonnie.2551', size: 104857600
> Writing with putc()...done
> Rewriting...done
> Writing intelligently...done
> Reading with getc()...done
> Reading intelligently...done
> Seeker 1...Seeker 2...Seeker 3...start
> 'em...done...done...done...
>              -------Sequential Output--------
> ---Sequential Input-- --Random--
>              -Per Char- --Block--- -Rewrite-- -Per
> Char- --Block--- --Seeks---
> Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec
> %CPU K/sec %CPU  /sec %CPU
>          100  9989  4.8  6739  1.0 18900  7.8 225973
> 98.5 1914662
> 99.9 177210.7 259.7

> Any ideas?  Anybody have one of these that's working
> with FreeBSD 7?

I had almost the same problem. The transfer rate was less that 10Mb/s
for raid-10. When a BBU arrived I thought my problems gone. Not that
at once. Firstly none changed. I tested many combinatins with no avail
(it lasted some hours). Then it so happed that I set StorSave
"Ballanced", cache "on" and marked (by an accident) the raid for
checking (or verifying... can't recall how it's named at the 3WARE
BIOS). The test has began, I stopped it. Rebooted the OS.

To my surprise after booting the transfer rate was quite good. BTW,
the test proceeded after booting... Maybe the card did not use the
cache while the battery was uncharged? Don't know (but noticed
"twa0: INFO: (0x04: 0x0056): Battery charging completed:" at dmesg).

Here it is now with WD RE2 1T disks:
-----
3ware device driver for 9000 series storage controllers, version: 3.70.05.001
twa0: <3ware 9000 series Storage Controller> port 0xc800-0xc8ff mem 0xf8000000-0xf9ffffff,0xfeaff000-0xfeafffff irq 16 at device 0.0 on pci4
twa0: [ITHREAD]
twa0: INFO: (0x04: 0x0029): Verify started: unit=0
twa0: INFO: (0x04: 0x003D): Verify paused: unit=0
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-8LPML, 8 ports, Firmware FE9X 3.08.02.005, BIOS BE9X 3.08.00.002
-----
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9650SE-8LP DISK 3.08> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 953632MB (1953038336 512 byte sectors: 255H 63S/T 121571C)
-----
/dev/da0s1d                        902G     36G    794G     4%    /space
-----
btest% uname -a        
FreeBSD btest.ipt.ru 7.0-RELEASE FreeBSD 7.0-RELEASE #14: Mon Mar  3 18:27:26 MSK 2008     [hidden email]:/usr/obj/usr/src/sys/BTEST  i386
btest% sudo dd if=/dev/da0 of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 5.073393 secs (206681412 bytes/sec)
btest% sudo dd if=/dev/zero of=/space/dd.file bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 5.670188 secs (184927909 bytes/sec)
-----


WBR
--
Boris Samorodov (bsam)
Research Engineer, http://www.ipt.ru Telephone & Internet SP
FreeBSD committer, http://www.FreeBSD.org The Power To Serve
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

Boris Samorodov
In reply to this post by alan bryan-4
On Tue, 4 Mar 2008 09:50:23 -0800 (PST) alan bryan wrote:

> I've got a new server with a 3ware 9550SXU with the
> Battery.  I am using FreeBSD 7.0-Release (tried both
> 4BSD and ULE) using AMD64 and the 3ware performance
> for writes is just plain horrible.  Something is
> obviously wrong but I'm not sure what.

> I've got a 4 disk RAID 10 array.

> According to 3dm2 the cache is on.  I even tried
> setting The StorSave preference to "Performance" with
> no real benefit.  There seems to be something really
> wrong with disk performance.  Here's the results from
> bonnie:

> File './Bonnie.2551', size: 104857600
> Writing with putc()...done
> Rewriting...done
> Writing intelligently...done
> Reading with getc()...done
> Reading intelligently...done
> Seeker 1...Seeker 2...Seeker 3...start
> 'em...done...done...done...
>              -------Sequential Output--------
> ---Sequential Input-- --Random--
>              -Per Char- --Block--- -Rewrite-- -Per
> Char- --Block--- --Seeks---
> Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec
> %CPU K/sec %CPU  /sec %CPU
>          100  9989  4.8  6739  1.0 18900  7.8 225973
> 98.5 1914662
> 99.9 177210.7 259.7

> Any ideas?  Anybody have one of these that's working
> with FreeBSD 7?

I had almost the same problem. The transfer rate was less that 10Mb/s
for raid-10. When a BBU arrived I thought my problems gone. Not that
at once. Firstly none changed. I tested many combinatins with no avail
(it lasted some hours). Then it so happed that I set StorSave
"Ballanced", cache "on" and marked (by an accident) the raid for
checking (or verifying... can't recall how it's named at the 3WARE
BIOS). The test has began, I stopped it. Rebooted the OS.

To my surprise after booting the transfer rate was quite good. BTW,
the test proceeded after booting... Maybe the card did not use the
cache while the battery was uncharged? Don't know (but noticed
"twa0: INFO: (0x04: 0x0056): Battery charging completed:" at dmesg).

Here it is now with WD RE2 1T disks:
-----
3ware device driver for 9000 series storage controllers, version: 3.70.05.001
twa0: <3ware 9000 series Storage Controller> port 0xc800-0xc8ff mem 0xf8000000-0xf9ffffff,0xfeaff000-0xfeafffff irq 16 at device 0.0 on pci4
twa0: [ITHREAD]
twa0: INFO: (0x04: 0x0029): Verify started: unit=0
twa0: INFO: (0x04: 0x003D): Verify paused: unit=0
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-8LPML, 8 ports, Firmware FE9X 3.08.02.005, BIOS BE9X 3.08.00.002
-----
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9650SE-8LP DISK 3.08> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 953632MB (1953038336 512 byte sectors: 255H 63S/T 121571C)
-----
/dev/da0s1d                        902G     36G    794G     4%    /space
-----
btest% uname -a        
FreeBSD btest.ipt.ru 7.0-RELEASE FreeBSD 7.0-RELEASE #14: Mon Mar  3 18:27:26 MSK 2008     [hidden email]:/usr/obj/usr/src/sys/BTEST  i386
btest% sudo dd if=/dev/da0 of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 5.073393 secs (206681412 bytes/sec)
btest% sudo dd if=/dev/zero of=/space/dd.file bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 5.670188 secs (184927909 bytes/sec)
-----


WBR
--
Boris Samorodov (bsam)
Research Engineer, http://www.ipt.ru Telephone & Internet SP
FreeBSD committer, http://www.FreeBSD.org The Power To Serve
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

Boris Samorodov
On Tue, 04 Mar 2008 23:25:00 +0300 Boris Samorodov wrote:

> Here it is now with WD RE2 1T disks:

Sorry, these are just plain WD-JS 0.5T disks here.


WBR
--
Boris Samorodov (bsam)
Research Engineer, http://www.ipt.ru Telephone & Internet SP
FreeBSD committer, http://www.FreeBSD.org The Power To Serve
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

alan bryan-4
In reply to this post by alan bryan-4

--- alan bryan <[hidden email]> wrote:

> Hi,
>
> I've got a new server with a 3ware 9550SXU with the
> Battery.  I am using FreeBSD 7.0-Release (tried both
> 4BSD and ULE) using AMD64 and the 3ware performance
> for writes is just plain horrible.  Something is
> obviously wrong but I'm not sure what.
>
> I've got a 4 disk RAID 10 array.
>
> According to 3dm2 the cache is on.  I even tried
> setting The StorSave preference to "Performance"
> with
> no real benefit.  There seems to be something really
> wrong with disk performance.  Here's the results
> from
> bonnie:


OK - so, I ran the server for about 24 hrs while it
did its battery test.  After that it automatically
turned on it's write cache.  So, even though initially
3dm2 was reporting the cache was on I guess it must
not have been.  I ran some more tests:

Version 1.93d       ------Sequential Output------
--Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per
Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP
K/sec %CP K/sec %CP  /sec %CP
server 16G   530  99 120160  23 28207   8   903  99
109688  16 518.7  18
Latency             18651us     541ms     447ms  
11461us     271ms   66793us
Version 1.93d       ------Sequential Create------
--------Random Create--------
server -Create-- --Read--- -Delete-- -Create--
--Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP
/sec %CP  /sec %CP  /sec %CP
                 16 20737  34 +++++ +++ +++++ +++
20429  37 +++++ +++ +++++ +++
Latency               146ms      22us     101ms    
150ms      24us      31us
1.93c,1.93d,server,1,1204745494,16G,,530,99,120160,23,28207,8,903,99,109688,16,518.7,18,16,,,,,20737,34,+++++,+++,+++++,+++,20429,37,+++++,+++,+++++,+++,18651us,541ms,447ms,11461us,271ms,66793us,146ms,22us,101ms,150ms,24us,31us

And my original test again:

Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start
'em...done...done...done...
              -------Sequential Output--------
---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per
Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec
%CPU K/sec %CPU  /sec %CPU
          100 154436 71.4 86016 10.8 104301 13.7
226587 100.0 1934776 101.6 173626.2 261.3


>dd if=/dev/zero of=/tmp/dd.file bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 7.814102 secs
(134190213 bytes/sec)

So, it's better but am I still getting what I should
be seeing?  


      ____________________________________________________________________________________
Looking for last minute shopping deals?  
Find them fast with Yahoo! Search.  http://tools.search.yahoo.com/newsearch/category.php?category=shopping
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

alan bryan-4

--- alan bryan <[hidden email]> wrote:

>
> --- alan bryan <[hidden email]> wrote:
>
> > Hi,
> >
> > I've got a new server with a 3ware 9550SXU with
> the
> > Battery.  I am using FreeBSD 7.0-Release (tried
> both
> > 4BSD and ULE) using AMD64 and the 3ware
> performance
> > for writes is just plain horrible.  Something is
> > obviously wrong but I'm not sure what.
> >
> > I've got a 4 disk RAID 10 array.
> >
> > According to 3dm2 the cache is on.  I even tried
> > setting The StorSave preference to "Performance"
> > with
> > no real benefit.  There seems to be something
> really
> > wrong with disk performance.


So, all of this seems to be due to the Battery unit.
What seems to be happening is that if the battery
isn't fully charged the write cache get disabled.

So, when I was testing and removing the card, changing
cabling, etc... with the server powered off the
battery would get ever so slightly discharged and then
when booting back up the server has the cache off
until it gets topped back up again.  Thus, I was
seeing weird inconsistent results all over the place.

I had to move the BBU and disconnected it for a few
minutes and then turned the server back on.  When it
came back up 3dm2 reporting that the BBU was charging
and that write cache was off.  Once recharged (15 min
or so) the cache was turned back on.

This could be really bad in a loaded server if you
have to down it for a few min service window and then
boot back up and the write cache is then off.  I found
that I could log into 3dm2 and change the StorSave
profile to "Performance" which would then enable the
cache again (it ignores whether you have the BBU).
Then once the battery is charged you can go back to
"Balanced" profile.

Guess the lesson learned is make sure the server has
been powered up for 24hrs and battery fully charged
before doing any tests against this 3ware card.


      ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ 

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance

Ivan Voras
In reply to this post by alan bryan-4
alan bryan wrote:
> --- alan bryan <[hidden email]> wrote:

>> I've got a 4 disk RAID 10 array.

> Version 1.93d       ------Sequential Output------
> --Sequential Input- --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per
> Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP
> K/sec %CP K/sec %CP  /sec %CP
> server 16G   530  99 120160  23 28207   8   903  99
> 109688  16 518.7  18

> So, it's better but am I still getting what I should
> be seeing?  

It seems about right - twice the performance of a single drive is OK for
a 4-drive RAID10.



signature.asc (196 bytes) Download Attachment