Samba server performance -- only 33MiB/s with one client on 1G network!

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Samba server performance -- only 33MiB/s with one client on 1G network!

Lev Serebryakov
Hello, Freebsd-performance.

  I have 9-STABLE/amd64 server on E4500 CPU (Core2Duo 2.4Ghz), with 6GiB of
 DDR2-800 RAM and intel 1G adapter (em0, "82566DM-2 Gigabit Network
 Connection").

  It has SATA software RAID5 with 5 drives.

  Filesystem (UFS2, 32K blocks) is exported with samba36 (latest), with
 enabled AIO, sendfile and TCP options for 256K window (SMB2 is turned off,
 as Win7 clients has confirmed, but not fixed bug in it).

  Local read of directory like this:

  tar cf - "dir" | dd bs=128k of=/dev/null

  shows 280MiB/s read according to "dd" diagnostics. It is about 24GiB of
 data, so it could not be cached. Good result, I like it.

  This server is connected to small home netwrok via HP ProCurve 1400-8G
 switch.

  Also, I have client, which is connected to same switch and runs Win 7. It
 mounts this FS via Samba.

  iperf from server to Win7 client (Server sends) shows ~820 Mbits/sec. It
 is not best performanse, but client has loosy network adapter (Atheros 1G
 adapter).

  But when I run

  tar cf - "dir" | dd bs=128k of=/dev/null

  on client it shows only 33MiB/s. It doesn't look like something near
 820Mbit/s!

  And smbd utilize only 50% of one core.


  So, smbd is NOT limited by:

    (1) FS (33MiB/s vs 280MiB/s).
    (2) Network (33MiB/s vs 820Mbit/s / 97MiB/s).
    (3) CPU (50% vs 100%, CPU is 150% idle!).

  Why samba performacne is so loosy? What could

--
// Black Lion AKA Lev Serebryakov <[hidden email]>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Samba server performance -- only 33MiB/s with one client on 1G network!

Lev Serebryakov
Hello, Lev.
You wrote 7 июля 2013 г., 23:37:04:

LS>   Why samba performacne is so loosy? What could
 ...be improved?

 Sorry for sending incomplete message.

--
// Black Lion AKA Lev Serebryakov <[hidden email]>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Samba server performance -- only 33MiB/s with one client on 1G network!

Lev Serebryakov
In reply to this post by Lev Serebryakov
Hello, Lev.
You wrote 7 июля 2013 г., 23:37:04:

LS>   So, smbd is NOT limited by:
LS>     (1) FS (33MiB/s vs 280MiB/s).
LS>     (2) Network (33MiB/s vs 820Mbit/s / 97MiB/s).
LS>     (3) CPU (50% vs 100%, CPU is 150% idle!).
LS>   Why samba performacne is so loosy? What could
 I've "ktrace" smbd process in question and see a lot of

 67106 100935 smbd     0.003483 CALL  geteuid
 67106 100935 smbd     0.003485 RET   geteuid 0
 67106 100935 smbd     0.003487 CALL  getegid
 67106 100935 smbd     0.003490 RET   getegid 0
 67106 100935 smbd     0.003493 CALL  __sysctl(0x7fffffffd130,0x2,0x7fffffffd14c,0x7fffffffd140,0,0)
 67106 100935 smbd     0.003496 SCTL  "kern.ngroups"
 67106 100935 smbd     0.003500 RET   __sysctl 0
 67106 100935 smbd     0.003503 CALL  setgroups(0x2,0x804c45280)
 67106 100935 smbd     0.003506 RET   setgroups 0
 67106 100935 smbd     0.003509 CALL  setregid(0xffffffff,0)
 67106 100935 smbd     0.003513 RET   setregid 0
 67106 100935 smbd     0.003516 CALL  getegid
 67106 100935 smbd     0.003518 RET   getegid 0
 67106 100935 smbd     0.003521 CALL  setreuid(0xffffffff,0)
 67106 100935 smbd     0.003525 RET   setreuid 0
 67106 100935 smbd     0.003529 CALL  geteuid
 67106 100935 smbd     0.003543 RET   geteuid 0

  sequences, really, there is (for 60 seconds of dump):

2131018 geteuid calls  (35516 calls/s)
2130595 getegid calls  (35509 calls/s)
 959106 setreuid calls (15985 calls/s)
 959082 setregid calls (15984 calls/s)
 957925 __sysctl calls (15965 calls/s)

 It looks strange to me! And very ineffective!
--
// Black Lion AKA Lev Serebryakov <[hidden email]>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Samba server performance -- only 33MiB/s with one client on 1G network!

Lev Serebryakov
Hello, Lev.
You wrote 8 июля 2013 г., 1:02:24:

LS> Hello, Lev.
LS> You wrote 7 июля 2013 г., 23:37:04:

LS>>   So, smbd is NOT limited by:
LS>>     (1) FS (33MiB/s vs 280MiB/s).
LS>>     (2) Network (33MiB/s vs 820Mbit/s / 97MiB/s).
LS>>     (3) CPU (50% vs 100%, CPU is 150% idle!).
LS>>   Why samba performacne is so loosy? What could
LS>  I've "ktrace" smbd process in question and see a lot of
 Ok, poll() takes 98.5% of time spent by samba process! Is it Ok?!
 And getegid()/geteuid() is takes 30% of calls (but only 0.1% of time)!
 But this poll() prevalence is not clear to me. So, it wait something most
of time? What?

--
// Black Lion AKA Lev Serebryakov <[hidden email]>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Samba server performance -- only 33MiB/s with one client on 1G network!

Mark Saad-5
On Tue, Jul 9, 2013 at 12:10 PM, Lev Serebryakov <[hidden email]> wrote:

> Hello, Lev.
> You wrote 8 июля 2013 г., 1:02:24:
>
> LS> Hello, Lev.
> LS> You wrote 7 июля 2013 г., 23:37:04:
>
> LS>>   So, smbd is NOT limited by:
> LS>>     (1) FS (33MiB/s vs 280MiB/s).
> LS>>     (2) Network (33MiB/s vs 820Mbit/s / 97MiB/s).
> LS>>     (3) CPU (50% vs 100%, CPU is 150% idle!).
> LS>>   Why samba performacne is so loosy? What could
> LS>  I've "ktrace" smbd process in question and see a lot of
>  Ok, poll() takes 98.5% of time spent by samba process! Is it Ok?!
>  And getegid()/geteuid() is takes 30% of calls (but only 0.1% of time)!
>  But this poll() prevalence is not clear to me. So, it wait something most
> of time? What?
>
> --
> // Black Lion AKA Lev Serebryakov <[hidden email]>
>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "
> [hidden email]"
>

Lev
  What can you tell us about the system tunings. Can you post a copy of your

1. sysctl.conf
2. loader.conf
3. smb.conf
4. /etc/fstab

Also the output from

1. pciconf -lv
2. dmidecode
3. ifconfig
4. mount

sanitize them as needed.
--
mark saad | [hidden email]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Samba server performance -- only 33MiB/s with one client on 1G network!

Lev Serebryakov
Hello, Mark.
You wrote 9 июля 2013 г., 20:41:31:

MS>   What can you tell us about the system tunings. Can you post a copy of your
 I need to note, that system as tuned for (a) maximum local read/write speed
 and (b) maximum TCP throughput in local network, as seen by "iperf".

    Results of these separate tunings could be seen in very first message in
 this thread (280MiB/s sustained local read from FS and 820Mbit/s of TCP
 throughput to client in question).

MS> 1. sysctl.conf
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=131072
net.inet.tcp.recvbuf_max=1048576
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=131072
net.inet.tcp.sendbuf_max=1048576
net.inet.tcp.maxtcptw=102400
net.inet.tcp.mssdflt=8800
net.inet.tcp.recvspace=262144
net.inet.tcp.sendspace=262144
kern.ipc.nmbclusters=204800
kern.ipc.nmbjumbop=192000
kern.ipc.maxsockbuf=2097152
net.inet.ip.intr_queue_maxlen=8192
vfs.read_max=32

MS> 2. loader.conf
hw.em.rxd=4096
hw.em.txd=4096
net.link.ifqmaxlen=16384

MS> 3. smb.conf
 Attached

MS> 4. /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/label/swap         none            swap    sw              0       0
/dev/ufs/root           /               ufs     rw              1       1
/dev/ufs/tmp            /tmp            ufs     rw              2       2
/dev/ufs/usr            /usr            ufs     rw              2       2
/dev/ufs/var            /var            ufs     rw              2       2
/dev/raid5/storage      /usr/home       ufs     rw              2       2
linproc                 /compat/linux/proc      linprocfs       rw      0       0
/usr/home               /backup/home    nullfs  rw              0       0

 FS in question is "/usr/home" (exported as "storage" SMB share).

MS> Also the output from
MS> 1. pciconf -lv
  attached.

MS> 2. dmidecode
dmidecode
MS> 3. ifconfig
ifconfig -v em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:1e:8c:75:03:0d
        inet 192.168.134.2 netmask 0xffffff00 broadcast 192.168.134.255
        inet6 fe80::21e:8cff:fe75:30d%em0 prefixlen 64 scopeid 0x1
        inet6 2001:470:923f:1:21e:8cff:fe75:30d prefixlen 64 autoconf
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet 1000baseT <full-duplex>
        status: active

MS> 4. mount
/dev/ufs/root on / (ufs, local, soft-updates)
devfs on /dev (devfs, local)
/dev/ufs/tmp on /tmp (ufs, local, journaled soft-updates)
/dev/ufs/usr on /usr (ufs, NFS exported, local, journaled soft-updates)
/dev/ufs/var on /var (ufs, local, journaled soft-updates)
/dev/raid5/storage on /usr/home (ufs, NFS exported, local, journaled soft-updates)
linprocfs on /usr/local/linux/proc (linprocfs, local)
/usr/home on /backup/home (nullfs, local)



--
// Black Lion AKA Lev Serebryakov <[hidden email]>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"

dmidecode.txt.bz2 (5K) Download Attachment
pciconf.txt.bz2 (1K) Download Attachment
smb.conf.bz2 (952 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Samba server performance -- only 33MiB/s with one client on 1G network!

Adrian Chadd-2
Wait a sec - your system is spending most of its time in poll() ?

Have you run hwpmc to see if it's actually spinning in poll(), or if
its just doing a lot of poll calls that are very short-lived?



-adrian
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Samba server performance -- only 33MiB/s with one client on 1G network!

Lev Serebryakov
Hello, Adrian.
You wrote 9 июля 2013 г., 23:56:14:

AC> Wait a sec - your system is spending most of its time in poll() ?
AC> Have you run hwpmc to see if it's actually spinning in poll(), or if
AC> its just doing a lot of poll calls that are very short-lived?
  I need to re-test all four cases (palin, aio, sendfile, aio + sendfile)
 with hwpmc. What should I read about hwpmc?
  Anyway, all 4 cases shows same performance.

--
// Black Lion AKA Lev Serebryakov <[hidden email]>

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"