PCIe SATA HBA for ZFS on -STABLE

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

PCIe SATA HBA for ZFS on -STABLE

Matt Thyer-2
I'm not on the -STABLE list so please reply to me.

I'm using an Intel Core i3-530 on a Gigabyte H55M-D2H motherboard with 8 x
2TB drives & 2 x 1TB drives.
The plan is to have the 1 TB drives in a zmirror and the 8 in a raidz2.

Now the Intel chipset has only 6 on board SATA II ports so ideally I'm
looking for a non RAID SATA II HBA to give me 6 extra ports (4 min).
Why 6 extra ?
Well the case I'm using has 2 x eSATA ports so 6 would be ideal, 5 OK, and 4
the minimum I need to do the job.

So...

What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
using ZFS ?

Not wanting to break the bank.
Not interested in SATA III 6GB at this time... though it could be useful if
I add an SSD for... (is it ZIL ?).
Can this be added at any time ?

The main issue is I need at least 10 ports total for all existing drives...
ZIL would require 11 so ideally we are talking a 6 port HBA.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Steven Hartland
Areca's work well. The ARC-1220 (8 ports) should do you, not the cheapest but
good support and performance.

    Regards
    Steve

----- Original Message -----
From: "Matt Thyer" <[hidden email]>
To: <[hidden email]>
Sent: Tuesday, May 31, 2011 1:48 PM
Subject: PCIe SATA HBA for ZFS on -STABLE


> I'm not on the -STABLE list so please reply to me.
>
> I'm using an Intel Core i3-530 on a Gigabyte H55M-D2H motherboard with 8 x
> 2TB drives & 2 x 1TB drives.
> The plan is to have the 1 TB drives in a zmirror and the 8 in a raidz2.
>
> Now the Intel chipset has only 6 on board SATA II ports so ideally I'm
> looking for a non RAID SATA II HBA to give me 6 extra ports (4 min).
> Why 6 extra ?
> Well the case I'm using has 2 x eSATA ports so 6 would be ideal, 5 OK, and 4
> the minimum I need to do the job.
>
> So...
>
> What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
> using ZFS ?
>
> Not wanting to break the bank.
> Not interested in SATA III 6GB at this time... though it could be useful if
> I add an SSD for... (is it ZIL ?).
> Can this be added at any time ?
>
> The main issue is I need at least 10 ports total for all existing drives...
> ZIL would require 11 so ideally we are talking a 6 port HBA.
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "[hidden email]"
>

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to [hidden email].

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Freddie Cash-8
In reply to this post by Matt Thyer-2
On Tue, May 31, 2011 at 5:48 AM, Matt Thyer <[hidden email]> wrote:

> What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
> using ZFS ?
>
> Not wanting to break the bank.
> Not interested in SATA III 6GB at this time... though it could be useful if
> I add an SSD for... (is it ZIL ?).
> Can this be added at any time ?
>
> The main issue is I need at least 10 ports total for all existing drives...
> ZIL would require 11 so ideally we are talking a 6 port HBA.
>

SuperMicro AOC-USAS2-L8i works exceptionally well.  These are 8-port HBAs
using the LSI1068 chipset, supported by the mpt(4) driver.  Support 3 Gpbs
SATA/SAS, using multi-lane cables (2 connectors on the card, each connector
supports 4 SATA ports), hot-plug, hot-swap.

These are UIO cards, so the bracket that comes with it doesn't work with
normal cases (the bracket is on the wrong side of the card; they're made for
SuperMicro's UIO-based motherboards).  However, these are normal PCIe cards
and work in any PCIe slot.  You either have to remove the bracket, or you
can purchase separate brackets online.

These cards are recommended on the zfs-discuss mailing list.  They are only
~$120 CDN at places like cdw.ca and newegg.ca.

--
Freddie Cash
[hidden email]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Artem Belevich-3
On Tue, May 31, 2011 at 7:31 AM, Freddie Cash <[hidden email]> wrote:

> On Tue, May 31, 2011 at 5:48 AM, Matt Thyer <[hidden email]> wrote:
>
>> What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
>> using ZFS ?
>>
>> Not wanting to break the bank.
>> Not interested in SATA III 6GB at this time... though it could be useful if
>> I add an SSD for... (is it ZIL ?).
>> Can this be added at any time ?
>>
>> The main issue is I need at least 10 ports total for all existing drives...
>> ZIL would require 11 so ideally we are talking a 6 port HBA.
>>
>
> SuperMicro AOC-USAS2-L8i works exceptionally well.  These are 8-port HBAs
> using the LSI1068 chipset, supported by the mpt(4) driver.  Support 3 Gpbs
> SATA/SAS, using multi-lane cables (2 connectors on the card, each connector
> supports 4 SATA ports), hot-plug, hot-swap.
>
> These are UIO cards, so the bracket that comes with it doesn't work with
> normal cases (the bracket is on the wrong side of the card; they're made for
> SuperMicro's UIO-based motherboards).  However, these are normal PCIe cards
> and work in any PCIe slot.  You either have to remove the bracket, or you
> can purchase separate brackets online.
>
> These cards are recommended on the zfs-discuss mailing list.  They are only
> ~$120 CDN at places like cdw.ca and newegg.ca.

+1 for LSI1068(e) controller + mpt driver. It's cheap and it works.
Those LSI controllers are often hiding behind other brands. SuperMicro
mentioned above is one. Intel would be another -- search for Intel
SASUC8I. Tyan also sells one as TYAN P3208SR. LSI-branded controllers
tend to be a bit more expensive than rebranded ones, though
functionality is the same and you can often cross-flash firmware.

Keep in mind that HBAs based on LSI1068(e) can't handle hard drives
larger than 2TB and will truncate larger drive capacity to 2TB.

As for the SSD, you may want to hook them up to on-board SATA ports.
In my not-very scientific benchmark Intel's X25-M SSD connected to
on-board SATA port on ICH10 was able to deliver ~20% more reads/sec
than the same SSD connected to LSI1068 based controller.

--Artem
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

TJ Varghese-2
In reply to this post by Freddie Cash-8
On Tue, May 31, 2011 at 10:31 PM, Freddie Cash <[hidden email]> wrote:

> On Tue, May 31, 2011 at 5:48 AM, Matt Thyer <[hidden email]> wrote:
>
> > What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
> > using ZFS ?
> >
> > Not wanting to break the bank.
> > Not interested in SATA III 6GB at this time... though it could be useful
> if
> > I add an SSD for... (is it ZIL ?).
> > Can this be added at any time ?
> >
> > The main issue is I need at least 10 ports total for all existing
> drives...
> > ZIL would require 11 so ideally we are talking a 6 port HBA.
> >
>
> SuperMicro AOC-USAS2-L8i works exceptionally well.  These are 8-port HBAs
> using the LSI1068 chipset, supported by the mpt(4) driver.  Support 3 Gpbs
> SATA/SAS, using multi-lane cables (2 connectors on the card, each connector
> supports 4 SATA ports), hot-plug, hot-swap.
>
>
The USAS2 (6Gbps) is supported by the mps driver (on -CURRENT, not sure if
it's in 8-STABLE yet). Perhaps you're referring to the earlier USAS which
does 3Gbps and is supported by the mpt driver.


--
TJ
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Jeremy Chadwick
On Wed, Jun 01, 2011 at 02:34:55PM +0800, TJ Varghese wrote:

> On Tue, May 31, 2011 at 10:31 PM, Freddie Cash <[hidden email]> wrote:
>
> > On Tue, May 31, 2011 at 5:48 AM, Matt Thyer <[hidden email]> wrote:
> >
> > > What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
> > > using ZFS ?
> > >
> > > Not wanting to break the bank.
> > > Not interested in SATA III 6GB at this time... though it could be useful
> > if
> > > I add an SSD for... (is it ZIL ?).
> > > Can this be added at any time ?
> > >
> > > The main issue is I need at least 10 ports total for all existing
> > drives...
> > > ZIL would require 11 so ideally we are talking a 6 port HBA.
> > >
> >
> > SuperMicro AOC-USAS2-L8i works exceptionally well.  These are 8-port HBAs
> > using the LSI1068 chipset, supported by the mpt(4) driver.  Support 3 Gpbs
> > SATA/SAS, using multi-lane cables (2 connectors on the card, each connector
> > supports 4 SATA ports), hot-plug, hot-swap.
> >
> >
> The USAS2 (6Gbps) is supported by the mps driver (on -CURRENT, not sure if
> it's in 8-STABLE yet). Perhaps you're referring to the earlier USAS which
> does 3Gbps and is supported by the mpt driver.

Folks considering use of mps(4), which was committed to RELENG_8 roughly
around 2011/02/18 (thus is not in 8.2-RELEASE), should read the below
threads just in case.  Always good to be educated.  Of course, the
mailing lists are usually filled with complaints rather than success
stories, so the tone of my mail here will therefore sound negative; I
don't mean it that way, I just ask that people "be aware".

* 2011/04/29 -- mps driver instability under stable/8
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62507
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-May/thread.html#62518

* 2011/04/27 -- MPS driver: force bus rescan after remove SAS cable
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62438
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62443

* 2011/03/10 -- LSI SAS2008 performance with mps(4) driver
  http://lists.freebsd.org/pipermail/freebsd-stable/2011-March/thread.html#61862

To the OP (Matt Thyer):

Sadly I don't have a recommendation for you, since you effectively want
a 6-port SATA300 controller that's reliable, you're almost certainly
going to be paying Big Bucks(tm) given the number of ports and your
requirement that it be PCIe-based.  You state quite boldly "not wanting
to break the bank", but what you're asking for almost certainly WILL
break the bank.

For example, an "affordable" controller might be one driven by Silicon
Image's SiI3124 chip -- four (4) SATA300 ports, but it's only hooked to
PCI or PCI-X, not PCIe, which means you're susceptible to a much more
severe bus bottleneck than with PCIe:

http://www.siliconimage.com/products/family.aspx?id=3

FreeBSD does have support for many Silicon Image chips via the siis(4)
driver, and support is quite good since SI provides mav@ with technical
documentation and support.  I commend SI for that; it's good to see
companies supporting developers, regardless of OS.

I tend to avoid consumer-grade Marvell and JMicron SATA chipsets like
the plague, however.  That's based on my experiences with them under
Windows, where I would expect (truly) the drivers to be rock solid given
the marketing demographic of the chips in question.

Be aware that SATA port multipliers (if someone recommends them to you
as a way of providing expansion) will also limit your I/O bottleneck,
especially when multiple drives are used over a single multiplier port.
E.g. 4 drives operating at 100MByte/sec (common read speed with
consumer-grade Caviar Black drives!) will saturate a SATA300 connection
easily.  One port per drive solves this dilemma of course, putting the
focus back on the PCI/PCI-X/PCIe bus as a bottleneck.

Anyway, you need to ask yourself what your requirements really are, or
what sort of monetary limitations you have, then make a decision based
on that.  Remember: a good, solid controller will probably be a one-time
purchase.  But also think about the future and if in 2-3 years you want
to go about buying another controller (likely costing more than whatever
it is you buy now).

Good luck, and please let us know what controller you *do* end up going
with and your experience with it!  Positives are as important as
negatives.

--
| Jeremy Chadwick                                   [hidden email] |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Daniel Kalchev
In reply to this post by TJ Varghese-2


On 01.06.11 09:34, TJ Varghese wrote:
>
>> SuperMicro AOC-USAS2-L8i works exceptionally well.  These are 8-port HBAs
>> using the LSI1068 chipset, supported by the mpt(4) driver.  Support 3 Gpbs
>> SATA/SAS, using multi-lane cables (2 connectors on the card, each connector
>> supports 4 SATA ports), hot-plug, hot-swap.
> The USAS2 (6Gbps) is supported by the mps driver (on -CURRENT, not sure if
> it's in 8-STABLE yet). Perhaps you're referring to the earlier USAS which
> does 3Gbps and is supported by the mpt driver.
>

One should also bear in mind, that the 1068e based controllers
(AOC-USAS-L8i) apparently have 2TB drive size limitation, therefore
cannot be used with current 3TB (and who knows what capacities by the
end of the year) drives. Otherwise, this is an well supported, high
performance, stable and relatively cheap HBA to consider. The SAS2 (LSI
2008 based) HBAs are also good, despite some firmware issues (mostly are
related to use with SAS expanders) and do not have obvious limitations
yet. These might be just a bit more expensive, but my Supermicro
supplier advised delivery times for the older SAS (3Gbps) versions would
be much longer.

Daniel
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Stefan Esser-3
In reply to this post by Jeremy Chadwick
Am 01.06.2011 10:07, schrieb Jeremy Chadwick:

> Sadly I don't have a recommendation for you, since you effectively want
> a 6-port SATA300 controller that's reliable, you're almost certainly
> going to be paying Big Bucks(tm) given the number of ports and your
> requirement that it be PCIe-based.  You state quite boldly "not wanting
> to break the bank", but what you're asking for almost certainly WILL
> break the bank.
>
> For example, an "affordable" controller might be one driven by Silicon
> Image's SiI3124 chip -- four (4) SATA300 ports, but it's only hooked to
> PCI or PCI-X, not PCIe, which means you're susceptible to a much more
> severe bus bottleneck than with PCIe:
>
> http://www.siliconimage.com/products/family.aspx?id=3

FYI: There is at least one PCIe card with Sil3124 (with PCIe to PCI-X
bridge on the controller card):

        http://www.leaf-computer.com/#24e

Price is 69 Euro plus 5 Euro shipping to Europe or North America (i.e.
some US$110 total).

Regards, STefan
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Freddie Cash-8
In reply to this post by TJ Varghese-2
On Tue, May 31, 2011 at 11:34 PM, TJ Varghese <[hidden email]> wrote:

> On Tue, May 31, 2011 at 10:31 PM, Freddie Cash <[hidden email]> wrote:
>
>> On Tue, May 31, 2011 at 5:48 AM, Matt Thyer <[hidden email]> wrote:
>>
>> > What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
>> > using ZFS ?
>> >
>> > Not wanting to break the bank.
>> > Not interested in SATA III 6GB at this time... though it could be useful
>> if
>> > I add an SSD for... (is it ZIL ?).
>> > Can this be added at any time ?
>> >
>> > The main issue is I need at least 10 ports total for all existing
>> drives...
>> > ZIL would require 11 so ideally we are talking a 6 port HBA.
>> >
>>
>> SuperMicro AOC-USAS2-L8i works exceptionally well.  These are 8-port HBAs
>> using the LSI1068 chipset, supported by the mpt(4) driver.  Support 3 Gpbs
>> SATA/SAS, using multi-lane cables (2 connectors on the card, each
>> connector
>> supports 4 SATA ports), hot-plug, hot-swap.
>>
>>
> The USAS2 (6Gbps) is supported by the mps driver (on -CURRENT, not sure if
> it's in 8-STABLE yet). Perhaps you're referring to the earlier USAS which
> does 3Gbps and is supported by the mpt driver.
>
>
Oops, you're right.  We have the USAS, not the USAS2, using the mpt(4)
driver.

--
Freddie Cash
[hidden email]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Matt Thyer-2
In reply to this post by Jeremy Chadwick
On 1 June 2011 17:37, Jeremy Chadwick <[hidden email]> wrote:

> On Wed, Jun 01, 2011 at 02:34:55PM +0800, TJ Varghese wrote:
> > On Tue, May 31, 2011 at 10:31 PM, Freddie Cash <[hidden email]>
> wrote:
>

[snip]


> > > SuperMicro AOC-USAS2-L8i works exceptionally well.  These are 8-port
> HBAs
> > > using the LSI1068 chipset, supported by the mpt(4) driver.  Support 3
> Gpbs
> > > SATA/SAS, using multi-lane cables (2 connectors on the card, each
> connector
> > > supports 4 SATA ports), hot-plug, hot-swap.
> > >
> > >
> > The USAS2 (6Gbps) is supported by the mps driver (on -CURRENT, not sure
> if
> > it's in 8-STABLE yet). Perhaps you're referring to the earlier USAS which
> > does 3Gbps and is supported by the mpt driver.
>
> Folks considering use of mps(4), which was committed to RELENG_8 roughly
> around 2011/02/18 (thus is not in 8.2-RELEASE), should read the below
> threads just in case.  Always good to be educated.  Of course, the
> mailing lists are usually filled with complaints rather than success
> stories, so the tone of my mail here will therefore sound negative; I
> don't mean it that way, I just ask that people "be aware".
>
> * 2011/04/29 -- mps driver instability under stable/8
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62507
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2011-May/thread.html#62518
>
> * 2011/04/27 -- MPS driver: force bus rescan after remove SAS cable
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62438
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2011-April/thread.html#62443
>
> * 2011/03/10 -- LSI SAS2008 performance with mps(4) driver
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2011-March/thread.html#61862
>

Those threads assure me that the SuperMicro AOC-USAS2-L8i with version 9
firmware and the mps(4) driver work very well as long as I'm running FreeBSD
9-CURRENT or 8-STABLE (not 8.2-RELEASE).  As I'm running -STABLE
I'm quite happy to give it a go.

To the OP (Matt Thyer):
>
> Sadly I don't have a recommendation for you, since you effectively want
> a 6-port SATA300 controller that's reliable, you're almost certainly
> going to be paying Big Bucks(tm) given the number of ports and your
> requirement that it be PCIe-based.  You state quite boldly "not wanting
> to break the bank", but what you're asking for almost certainly WILL
> break the bank.
>

Jeremy, I think you need to have another look at current prices.
I have now bought a SuperMicro AOC-USAS2-L8i on EBay from bakamuzko
with the cables I need for only $US 210.99 (I do know about the UIO
bracket).

For example, an "affordable" controller might be one driven by Silicon
> Image's SiI3124 chip -- four (4) SATA300 ports, but it's only hooked to
> PCI or PCI-X, not PCIe, which means you're susceptible to a much more
> severe bus bottleneck than with PCIe:
>

I defintely would not consider PCI for part of a ZFS array with 4 drives on
that one controller.

<http://www.siliconimage.com/products/family.aspx?id=3>

> I tend to avoid consumer-grade Marvell and JMicron SATA chipsets like
> the plague, however.  That's based on my experiences with them under
> Windows, where I would expect (truly) the drivers to be rock solid given
> the marketing demographic of the chips in question.
>

I've had the same bad experiences.

Good luck, and please let us know what controller you *do* end up going
> with and your experience with it!  Positives are as important as
> negatives.
>

I'll let you know how it works out.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Matthew Dillon
In reply to this post by Matt Thyer-2
:I'm not on the -STABLE list so please reply to me.
:
:I'm using an Intel Core i3-530 on a Gigabyte H55M-D2H motherboard with 8 x
:2TB drives & 2 x 1TB drives.
:The plan is to have the 1 TB drives in a zmirror and the 8 in a raidz2.
:
:Now the Intel chipset has only 6 on board SATA II ports so ideally I'm
:looking for a non RAID SATA II HBA to give me 6 extra ports (4 min).
:Why 6 extra ?
:Well the case I'm using has 2 x eSATA ports so 6 would be ideal, 5 OK, and 4
:the minimum I need to do the job.
:
:So...
:
:What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
:using ZFS ?
:
:Not wanting to break the bank.
:Not interested in SATA III 6GB at this time... though it could be useful if
:I add an SSD for... (is it ZIL ?).
:Can this be added at any time ?
:
:The main issue is I need at least 10 ports total for all existing drives...
:ZIL would require 11 so ideally we are talking a 6 port HBA.

    The absolute cheapest solution is to buy a Sil-3132 PCIe card
    (providing 2 E-SATA ports), and then connect an external port multiplier
    to each port.  External port multiplier enclosures typically support
    5 drives each so that would give you your 10 drives.

    Even the 3132 is a piss-ant little card it does support FIS-Based
    switching so performance will be very good... it will just be limited
    to SATA-II speeds is all.

    Motherboard AHCI-based SATA ports typically do NOT have FIS-Based
    switching support (this would be the FBSS capability flag when the AHCI
    driver probes the chipset).  This means that while you can attach an
    external port multiplier enclosure to mobo SATA ports (see later
    on E-SATA vs SATA), read performance from multiple drives concurrently
    will be horrible.  Write performance will still be decent due to drive
    write caches despite being serialized.

    On E-SATA vs SATA.  Essentially there are only two differences between
    E-SATA and SATA.  One is the cable and connector format.  The other is
    hot swap detection.  Most mobo SATA ports can be strung out to E-SATA
    with an appropriate adapter.  High-end Intel ASUS mobos often come with
    such adapters (this is why they usually don't sport an actual E-Sata
    port on the backplane) and the BIOS has setup features to specify E-SATA
    on a port-by-port basis.

    --

    For SSDs you want to directly connect the SSD to a mobo SATA port and
    then either mount the SSD in the case or mount it in a hot-swap gadget
    that you can screw into a PCI slot (it doesn't actually use the PCI
    connector, just the slot).  A SATA-III port with a SATA-III SSD really
    shines here and 400-500 MBytes/sec random read performance from a single
    SSD is possible, but it isn't an absolute requirement.  A SATA-II port
    will still work fine as long as you don't mind maxing out the bandwidth
    at 250 MBytes/sec.

    --

    I can't help with any of the other questions.  Someone also suggested
    the MPS driver for FreeBSD, with caveats.

    I'll add a caveat on the port multiplier enclosures.  Nearly all such
    enclosures use another SIL chipset internally and it works pretty well
    EXCEPT that it isn't 100% dependable if you try to hot-swap drives in the
    enclosure while other drives in the enclosure are active.  So with that
    caveat, I recommend the port multiplier enclosure as the cheapest solution.

    To get robust hot-swap enclosures you either need to go with SAS or you
    need to go with discrete SATA ports (no port multiplication), and the
    ports have to support hot-swap.  The best hot-swap support for an AHCI
    port is if the AHCI chipset supports cold-presence-detect (CPD), and
    again Mobo AHCI chipsets usually don't.  Hot-swap is a bit hit or miss
    without CPD because power savings modes can effectively prevent hot-swap
    detect from working properly.  Drive disconnects will always be detected
    but drive connects might not be.

    And even with discrete SATA ports the AHCI firmware on mobos does not
    necessarily handle hot-swap properly.  For example my Intel-I7 ASUS mobo
    will generate spurious interrupts and status on a DIFFERENT discrete
    SATA port when I hot swap on some other discrete SATA port, in addition
    to generating the status interrupt on the correct port.  So then it comes
    down to the driver in the operating system properly handling the
    spurious status and properly stopping and restarting pending commands
    when necessary.   So, again, it is best for the machine to be idle before
    attempting a hot-swap.

    Lots of caveats.  Sorry... you can blame Intel for all the blasted issues
    with AHCI and SATA.  Intel didn't produce a very good chipset spec and
    vendors took all sorts of liberties.

                                        -Matt
                                        Matthew Dillon
                                        <[hidden email]>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Alexander Motin-3
In reply to this post by Matt Thyer-2
On 07.06.2011 05:33, Matthew Dillon wrote:
>      The absolute cheapest solution is to buy a Sil-3132 PCIe card
>      (providing 2 E-SATA ports), and then connect an external port multiplier
>      to each port.  External port multiplier enclosures typically support
>      5 drives each so that would give you your 10 drives.
>
>      Even the 3132 is a piss-ant little card it does support FIS-Based
>      switching so performance will be very good... it will just be limited
>      to SATA-II speeds is all.

SiI3132 is indeed good for it's price and it is quite good for random
I/O. But at burst speeds it is limited lower then SATA-II. Even lower
then PCIe 1.0 x1 it uses. IIRC I've seen about 150MB/s from one port and
about 170MB/s from two.

If burst rate is important, SiI3124 chip is much better -- up to about
900MB/s measured from 4 ports. The only issue is PCI-X interface: either
motherboard with PCI-X needed, or card with PCIe x8 bridge (like these
http://www.addonics.com/products/host_controller/adsa3gpx8-4e.asp), but
last case is too expensive.

There are also much cheaper (~$50) PCIe x1 bridge SiI3124 cards
(http://www.sybausa.com/productInfo.php?iid=537). They are not so fast
-- about 200MB/s, but still more then SiI3132. And they still have 4
SATA ports.

>      For SSDs you want to directly connect the SSD to a mobo SATA port and
>      then either mount the SSD in the case or mount it in a hot-swap gadget
>      that you can screw into a PCI slot (it doesn't actually use the PCI
>      connector, just the slot).  A SATA-III port with a SATA-III SSD really
>      shines here and 400-500 MBytes/sec random read performance from a single
>      SSD is possible, but it isn't an absolute requirement.  A SATA-II port
>      will still work fine as long as you don't mind maxing out the bandwidth
>      at 250 MBytes/sec.

Agree. Intel on-board ports rock! Recently I've built new system with
two OCZ Vertex 3 SSDs connected to 6Gbps SATA ports on Intel Sandy
Bridge class motherboard. UFS on top of graid RAID0 volume gives me
about 950MB/s on both read and write!

>      To get robust hot-swap enclosures you either need to go with SAS or you
>      need to go with discrete SATA ports (no port multiplication), and the
>      ports have to support hot-swap.  The best hot-swap support for an AHCI
>      port is if the AHCI chipset supports cold-presence-detect (CPD), and
>      again Mobo AHCI chipsets usually don't.  Hot-swap is a bit hit or miss
>      without CPD because power savings modes can effectively prevent hot-swap
>      detect from working properly.  Drive disconnects will always be detected
>      but drive connects might not be.

I would say it depends. In some cases it is easier to detect hot-plug
then hot-unplug, as device sends COMINIT that should wake up port even
from power-save state. With ICH10, for example, I've managed to make
both hot plug and unplug work even with power-management enabled:
hot-plug via tracking COMINIT, unplug via it's CPD capability. Without
PM it "just works". :)

--
Alexander Motin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Matt Thyer-2
In reply to this post by Matthew Dillon
On 7 June 2011 12:03, Matthew Dillon <[hidden email]> wrote:

>    The absolute cheapest solution is to buy a Sil-3132 PCIe card
>    (providing 2 E-SATA ports), and then connect an external port multiplier
>    to each port.  External port multiplier enclosures typically support
>    5 drives each so that would give you your 10 drives.
>

I've decided to avoid issues with port multiplication by going for a
Supermicro AOC-USAS2-L8i and then to flash it to IT (as opposed to IR) mode
to make it run as a standard non-RAID HBA.

As I've got 8 x 2 TB drives for the ZFS raidz2 I'll put them all on the
AOC-USAS2-L8i and save my onboard SATA-II ports for my 2 x 1TB drives for
the FreeBSD O.S. and any eSATA use.

Now the only remaining issue is whether to go with the Supermicro firmware
or the generic Lsi Logic firmware as some have reported better performance
with the version 9 Lsi Logic firmware.

I'll report on my experiences (as I keep a record of the revision of my
-STABLE build this should actually be useful!).
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PCIe SATA HBA for ZFS on -STABLE

Matt Thyer-2
On 8 June 2011 20:55, Matt Thyer <[hidden email]> wrote:

> On 7 June 2011 12:03, Matthew Dillon <[hidden email]> wrote:
>
>>    The absolute cheapest solution is to buy a Sil-3132 PCIe card
>>    (providing 2 E-SATA ports), and then connect an external port
>> multiplier
>>    to each port.  External port multiplier enclosures typically support
>>    5 drives each so that would give you your 10 drives.
>>
>
> I've decided to avoid issues with port multiplication by going for a
> Supermicro AOC-USAS2-L8i and then to flash it to IT (as opposed to IR) mode
> to make it run as a standard non-RAID HBA.
>
> As I've got 8 x 2 TB drives for the ZFS raidz2 I'll put them all on the
> AOC-USAS2-L8i and save my onboard SATA-II ports for my 2 x 1TB drives for
> the FreeBSD O.S. and any eSATA use.
>
> Now the only remaining issue is whether to go with the Supermicro firmware
> or the generic Lsi Logic firmware as some have reported better performance
> with the version 9 Lsi Logic firmware.
>
> I'll report on my experiences (as I keep a record of the revision of my
> -STABLE build this should actually be useful!).
>
>
I could not be happier with the result.

As planned, I've put all 8 2TB Western Digital WD20EARS drives on the 8 lane
PCIe 2.0 AOC-USAS2-L8i (flashed with the latest SuperMicro firmware to
behave as an AOC-USAS2-L8e).

Many of you are now screaming "Why is he using the dreaded WD20EARS ?".  The
answer is that I bought the first 4 drives before I knew their issues and
later decided to continue with them once I knew how to mitigate their issues
by:


   - Using the DOS based WDIDLE3.EXE utility to change the default "park the
   heads after 8 seconds of idle time" to the maximum of 5 minutes
   - Avoiding alignment problems by putting ZFS on the whole disks (with no
   GPT or MBR partition table)
   - Convincing ZFS to use 4KiB transfers by creating the pool on top of
   4KiB block sized devices created with "gnop -S 4096"


And the resulting performance...

Previously I could sustain about 45 MiB/s writing from high powered Windows
7 machines (via Samba using asynchronous I/O) on to my old arrangement of 4
drives in raidz1 (using only the Intel on-board SATA ports on the H55
chipset motherboard).

I can now sustain 90 MiB/s over the network with the 8 drive raidz2 but
that's only because the Windows 7 machines can't feed the data fast enough.

I can see from "zpool iostat pool 1" that ZFS is idle for about 5-7 seconds
and then it will write at around 390 - 410 MB/s for about 3 seconds.

dd tests are:
> sudo dd if=/dev/zero of=/export/1G bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 3.218604 secs (333604841 bytes/sec)
> sudo dd if=/dev/random of=/export/1G bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 15.531615 secs (69132658 bytes/sec)

So I'm very happy that I can keep my home network users happy with the
limits now being due to my gigabit ethernet network (and I'm not going 10
GbE any time soon!).

This is on 8-STABLE at r220359 (~ 5th of April 2011).
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"