Debug slow lan (ssh, nfs) file transfer

I've got two linux boxes attached to a gigabit switch. They both have gigabit NICs, cables are cat7.

Testing the network with iperf shows a fast connection but transferring files with rsync, scp, or nfs share is slow.

I'm testing with one 1GB file.

iperf result:

Client connecting to odroid, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.26 port 58788 connected with 192.168.1.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   979 MBytes   821 Mbits/sec

The transfer speed with rsync, scp or nfs is all about ~13Mb/s

scp:

 scp bigfile odroid:/mnt/usb1/               [10:19:12]
bigfile                                        57%  590MB  12.2MB/s   00:35 ETA^CKilled by signal 2.

rsync:

 rsync --progress bigfile /mnt/usb1/
bigfile
     44,695,552   4%   12.15MB/s    0:01:11  ^C

nfs:

binaryplease➜~(master✗)» time cp bigfile /mnt/nfs/usb1/      
cp -i bigfile /mnt/nfs/usb1/  0.01s user 0.94s system 1% cpu 1:11.06 total

1024MB/71sec = 14,42 MB/s

Since the iperf test shows a fast network connection, I assumed a problem with the storage devices being slow, but that doesn't seem to be the case either:

Client, SSD, internal:

binaryplease➜~(master✗)» sudo hdparm -tT /dev/sda                    
/dev/sda:
 Timing cached reads:   20344 MB in  2.00 seconds = 10181.50 MB/sec
 Timing buffered disk reads: 1498 MB in  3.00 seconds = 498.98 MB/sec

binaryplease➜~(master✗)» dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 2.03861 s, 263 MB/s

binaryplease➜~(master✗)» dd if=test of=/dev/null iflag=direct bs=8M  [12:29:01]
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 1.11392 s, 482 MB/s

Server, USB 3.0 Drive, external:

➜  ~ git:(master) ✗ sudo hdparm -tT /dev/sda   
/dev/sda:
 Timing cached reads:   1980 MB in  2.00 seconds = 991.66 MB/sec
 Timing buffered disk reads: 266 MB in  3.01 seconds =  88.27 MB/sec
➜  usb1   dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 6.53386 s, 82.2 MB/s
➜  usb1  dd if=test of=/dev/null iflag=direct bs=8M
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 7.13567 s, 75.2 MB/s

OS on client (Linux arch):

Linux binaryplease-laptop 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015 x86_64 GNU/Linux

OS on server (Ubuntu server for odroid):

Linux odroid 3.10.92 #1 SMP PREEMPT Tue Nov 17 00:15:24 BRST 2015 armv7l armv7l armv7l GNU/Linux

On both systems neither the cpu or the ram is maxed out.

If I interpret the results correctly the write speed of the servers drive (82.2 MB/s) should be easily matched by the network. How is the file transfer so slow?

I hope the information provided is sufficient and someone can help me find the bottleneck.

Thanks.

5
задан 22 January 2016 в 02:01
1 ответ

Судя по полученным числам, в вашей сети что-то не гигабитное.

Используйте ethtool или что-то подобное, чтобы проверить скорость на каждой сетевой карте компьютера.

Проверьте переключатель и убедитесь, что порты действительно активны на гигабитной скорости (проверьте светодиоды или состояние порта конфигурации, если коммутатор поддерживает управление).

Запустите iperf или альтернативный вариант между клиентами.

Наконец, проверьте использование ЦП - возможно, что-то не так там.

0
ответ дан 3 December 2019 в 02:09

Теги

Похожие вопросы