Try BBCP to speed up the data transfers on internet


[heydevops@east-server1 ~]$ sudo wget http://www.slac.stanford.edu/~abh/bbcp/bin/amd64_rhel60/bbcp -O /usr/bin/bbcp
[heydevops@east-server1 ~]$ sudo chmod +x /usr/bin/bbcp

[heydevops@west-server1 ~]$ sudo wget http://www.slac.stanford.edu/~abh/bbcp/bin/amd64_rhel60/bbcp -O /usr/bin/bbcp
[heydevops@west-server1 ~]$ sudo chmod +x /usr/bin/bbcp

[heydevops@east-server1 ~]$ which bbcp

/usr/bin/bbcp

[heydevops@east-server1 ~]$ ssh west-server1 which bbcp

/usr/bin/bbcp

[heydevops@east-server1 ~]$ cd heydevops
[heydevops@east-server1 heydevops]$ sudo dd if=/dev/zero of=/home/heydevops/heydevops/file.2g bs=1024M count=2

2+0 records in
2+0 records out
2147483648 bytes (2.1 GB) copied, 45.9129 s, 46.8 MB/s

[heydevops@east-server1 heydevops]$ ls -lh

total 2.0G
-rw-r--r-- 1 root   root   2.0G Mar  4 06:40 file.2g

[heydevops@east-server1 heydevops]$ time bbcp -r -P 2 -V -w 8m -s 16 file.2g west-server1:/home/heydevops/heydevops/

bbcp: Window size reduced to 245760 bytes.
bbcp: Indexing files to be copied...
bbcp: Copying 0 files in 0 directories.
Source east-server1.heylinux.com using initial send window of 18700
Target west-server1.heylinux.com using initial recv window of 87380
bbcp: Creating /home/heydevops/heydevops/file.2g
bbcp: 140304 06:46:12  0% done; 8.5 MB/s, avg 8.5 MB/s
bbcp: 140304 06:46:14  1% done; 6.9 MB/s, avg 7.5 MB/s
...
bbcp: 140304 06:51:46  99% done; 7.7 MB/s, avg 6.1 MB/s
bbcp: 140304 06:51:48  99% done; 3.3 MB/s, avg 6.1 MB/s
Source cpu=3.643 (sys=3.552 usr=0.091).
File /home/heydevops/heydevops/file.2g created; 2147483648 bytes at 6.0 MB/s
48 buffers used with 0 reorders; peaking at 0.
Source east-server1.heylinux.com using a final send window of 433840
Target cpu=15.149 (sys=14.505 usr=0.644).
Target west-server1.heylinux.com using a final recv window of 2298624
1 file copied at effectively 6.0 MB/s

real    5m42.236s
user    0m0.104s
sys     0m3.567s

[heydevops@east-server1 heydevops]$ time scp file.2g west-server1:/home/heydevops/heydevops/

file.2g   100%   2048MB   2.1MB/s   16:06    

real    16m8.448s
user    0m43.497s
sys     0m7.548s

, ,

  1. #1 by dylanlee on December 7, 2017 - 5:00 pm

    pls tell me how to lock the target directory of the bbcp user?

(will not be published)

*


Fork me on GitHub