Posts Tagged BBCP

BBCP Test Report

Environment:
Tested 10 times between west-server1 and east-server1.

Speed:
Within the same network, sometimes busy sometimes not, via the internet.
bbcp speed was more stable, and always faster than scp, mostly only spent 30% time of scp, sometimes the time was even shorter.

System Load:
During the whole transfer process:
CPU Load changed very few, only around “0.5-1”.
The value of “-/+ buffers/cache” changed very few, only around “100M”, sometimes even lower.

Bigfile and Smallfiles:
Tested the models directory(1.9G, 204 directories, 82 files) with a single bigfile(file.2g)
Single bigfile was faster, modles directory(9m25s), file.2g(6m12s), still faster than “scp -r”(24m8s)

Stop in the middle of transaction and then resume:
By default, files only appear on remote side when they are completed.
So if there was a file on remote side with the same name, it skipped with messages:

"bbcp: File /home/heydevops/file.2g already exists."

But “-a” option gives bbcp the ability to pick up where it left off, it creates a file on remote side for checkpoint informations:
[heydevops.guo@east-server1 heydevops]$ bbcp -k -a /home/heydevops/ -r -P 2 -V -f -w 9m -s 16 -i /home/heydevops/.ssh/id_rsa file.256m west-server1:/home/heydevops/
[heydevops@west-server1 ~]$ cat bbcp.172.16.4.11.905004000a0.file.256m

0 f 278176444449211 644 268435456 531fd087 531fcffd serverymp file.256m

[heydevops@east-server1 ~]$ bbcp -k -a /home/heydevops/ -r -P 2 -V -f -w 9m -s 16 -i /home/heydevops/.ssh/id_rsa file.256m west-server1:/home/heydevops/

...
bbcp: Will try to complete copying /home/heydevops/file.256m
bbcp: Appending to /home/heydevops/file.256m at offset 199233536
...

Command explain:
# bbcp -k -a /home/heydevops/ -r -P 2 -V -f -w 9m -s 16 -i /home/heydevops/.ssh/id_rsa file.256m west-server1:/home/heydevops/

-k keeps any partially created target files and allows full recovery after a copy failure.
-a /home/heydevops/ appends data to the end of the target file if the target is found to be incomplete due to a previously failed copy.
-r performs a recursive copy by copying all files starting at the source directory
-P 2 produces progress messages every 2 seconds.
-V produces verbose output, including detailed transfer-speed statistics.
-f forces the copy by erasing the target prior to copying the source file.
-w 9m sets the size of the disk input/output (I/O) buffers.
(window = netspeed/8*RTT = 1000Mb/8*74ms = 1000/1000/8*74 = 9.25 M)
[http://www.slac.stanford.edu/~abh/bbcp/#_Toc332986061]
-i specifies the name of the ssh identity file.
-s 16 sets the number of parallel network streams to 16 as suggested:
[http://www.slac.stanford.edu/~abh/bbcp/#_Streams_(-s)]

System optimization:
Increase the os default max windows size.
Otherwise the bbcp shows the autotuning may be misconfigured with given window size:

"bbcp: Target autotuning may be misconfigured; max set to 245760 bytes(240K)."

# vim /etc/sysctl.conf

net.core.rmem_max = 536870912
net.core.wmem_max = 536870912
net.core.rmem_default = 536870912
net.core.wmem_default = 536870912
net.core.optmem_max = 536870912
net.core.netdev_max_backlog = 1000000

1 Comment

Try BBCP to speed up the data transfers on internet

[heydevops@east-server1 ~]$ sudo wget http://www.slac.stanford.edu/~abh/bbcp/bin/amd64_rhel60/bbcp -O /usr/bin/bbcp
[heydevops@east-server1 ~]$ sudo chmod +x /usr/bin/bbcp

[heydevops@west-server1 ~]$ sudo wget http://www.slac.stanford.edu/~abh/bbcp/bin/amd64_rhel60/bbcp -O /usr/bin/bbcp
[heydevops@west-server1 ~]$ sudo chmod +x /usr/bin/bbcp

[heydevops@east-server1 ~]$ which bbcp

/usr/bin/bbcp

[heydevops@east-server1 ~]$ ssh west-server1 which bbcp

/usr/bin/bbcp

[heydevops@east-server1 ~]$ cd heydevops
[heydevops@east-server1 heydevops]$ sudo dd if=/dev/zero of=/home/heydevops/heydevops/file.2g bs=1024M count=2

2+0 records in
2+0 records out
2147483648 bytes (2.1 GB) copied, 45.9129 s, 46.8 MB/s

[heydevops@east-server1 heydevops]$ ls -lh

total 2.0G
-rw-r--r-- 1 root   root   2.0G Mar  4 06:40 file.2g

[heydevops@east-server1 heydevops]$ time bbcp -r -P 2 -V -w 8m -s 16 file.2g west-server1:/home/heydevops/heydevops/

bbcp: Window size reduced to 245760 bytes.
bbcp: Indexing files to be copied...
bbcp: Copying 0 files in 0 directories.
Source east-server1.heylinux.com using initial send window of 18700
Target west-server1.heylinux.com using initial recv window of 87380
bbcp: Creating /home/heydevops/heydevops/file.2g
bbcp: 140304 06:46:12  0% done; 8.5 MB/s, avg 8.5 MB/s
bbcp: 140304 06:46:14  1% done; 6.9 MB/s, avg 7.5 MB/s
...
bbcp: 140304 06:51:46  99% done; 7.7 MB/s, avg 6.1 MB/s
bbcp: 140304 06:51:48  99% done; 3.3 MB/s, avg 6.1 MB/s
Source cpu=3.643 (sys=3.552 usr=0.091).
File /home/heydevops/heydevops/file.2g created; 2147483648 bytes at 6.0 MB/s
48 buffers used with 0 reorders; peaking at 0.
Source east-server1.heylinux.com using a final send window of 433840
Target cpu=15.149 (sys=14.505 usr=0.644).
Target west-server1.heylinux.com using a final recv window of 2298624
1 file copied at effectively 6.0 MB/s

real    5m42.236s
user    0m0.104s
sys     0m3.567s

[heydevops@east-server1 heydevops]$ time scp file.2g west-server1:/home/heydevops/heydevops/

file.2g   100%   2048MB   2.1MB/s   16:06    

real    16m8.448s
user    0m43.497s
sys     0m7.548s

, ,

No Comments

Fork me on GitHub