在CentOS 6.4上安装配置GlusterFS


参考资料:
http://www.sohailriaz.com/glusterfs-howto-on-centos-6-x/
http://navyaijm.blog.51cto.com/4647068/1258250

背景介绍:
项目目前在文件同步方面采用的是rsync,在尝试用分布式文件系统替换的时候,使用过MooseFS,效果差强人意,在了解到了GlusterFS之后,决定尝试一下,因为它跟MooseFS相比,感觉部署上更加简单一些,同时没有元数据服务器的特点使其没有单点故障的存在,感觉非常不错。

环境介绍:
OS: CentOS 6.4 x86_64 Minimal
Servers: idc1-server1,idc1-server2,idc1-server3,idc1-server4
Client: idc1-server5

具体步骤:
1. 在idc1-server{1-4}上安装GlusterFS软件包:
# wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
# yum install -y glusterfs glusterfs-server glusterfs-fuse

# /etc/init.d/glusterd start
# chkconfig glusterd on

2. 在idc1-server1上配置整个GlusterFS集群:
[root@idc1-server1 ~]# gluster peer probe idc1-server1

peer probe: success: on localhost not needed

[root@idc1-server1 ~]# gluster peer probe idc1-server2

peer probe: success

[root@idc1-server1 ~]# gluster peer probe idc1-server3

peer probe: success

[root@idc1-server1 ~]# gluster peer probe idc1-server4

peer probe: success

注意事项:
在某些情况下,idc1-server1在peer列表中会被识别为IP地址,这会造成一些通讯的问题。
假设idc1-server1的IP地址为10.100.1.11,则需要通过以下步骤来手动修复。
[root@idc1-server2 ~]# gluster peer detach 10.100.1.11

peer detach: success

[root@idc1-server2 ~]# gluster peer probe idc1-server1

peer probe: success

[root@idc1-server2 ~]# gluster peer status

Number of Peers: 3
  
Hostname: idc1-server3
Uuid: 01f25251-9ee6-40c7-a322-af53a034aa5a
State: Peer in Cluster (Connected)
  
Hostname: idc1-server4
Uuid: 212295a6-1f38-4a1e-968c-577241318ff1
State: Peer in Cluster (Connected)
  
Hostname: idc1-server1
Port: 24007
Uuid: ed016c4e-7159-433f-88a5-5c3ebd8e36c9
State: Peer in Cluster (Connected)

4. 在idc1-server1上创建GlusterFS磁盘:
[root@idc1-server1 ~]# gluster volume create datavolume1 replica 2 transport tcp idc1-server1:/usr/local/share/datavolume1 idc1-server2:/usr/local/share/datavolume1 idc1-server3:/usr/local/share/datavolume1 idc1-server4:/usr/local/share/datavolume1 force

volume create: datavolume1: success: please start the volume to access data

[root@idc1-server1 ~]# gluster volume create datavolume2 replica 2 transport tcp idc1-server1:/usr/local/share/datavolume2 idc1-server2:/usr/local/share/datavolume2 idc1-server3:/usr/local/share/datavolume2 idc1-server4:/usr/local/share/datavolume2 force

volume create: datavolume2: success: please start the volume to access data

[root@idc1-server1 ~]# gluster volume create datavolume3 replica 2 transport tcp idc1-server1:/usr/local/share/datavolume3 idc1-server2:/usr/local/share/datavolume3 idc1-server3:/usr/local/share/datavolume3 idc1-server4:/usr/local/share/datavolume3 force

volume create: datavolume3: success: please start the volume to access data

[root@idc1-server1 ~]# gluster volume start datavolume1

volume start: datavolume1: success

[root@idc1-server1 ~]# gluster volume start datavolume2

volume start: datavolume2: success

[root@idc1-server1 ~]# gluster volume start datavolume3

volume start: datavolume3: success

[root@idc1-server1 ~]# gluster volume info

Volume Name: datavolume1
Type: Distributed-Replicate
Volume ID: aea76c2a-b754-4037-9634-c2062e2955c3
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: idc1-server1:/usr/local/share/datavolume1
Brick2: idc1-server2:/usr/local/share/datavolume1
Brick3: idc1-server3:/usr/local/share/datavolume1
Brick4: idc1-server4:/usr/local/share/datavolume1
 
Volume Name: datavolume2
Type: Distributed-Replicate
Volume ID: 1ed65c6e-ee23-475a-82c7-2967e2fc2569
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: idc1-server1:/usr/local/share/datavolume2
Brick2: idc1-server2:/usr/local/share/datavolume2
Brick3: idc1-server3:/usr/local/share/datavolume2
Brick4: idc1-server4:/usr/local/share/datavolume2
 
Volume Name: datavolume3
Type: Distributed-Replicate
Volume ID: b63bb4ea-bd37-4dd6-9a4c-230e6d236afa
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: idc1-server1:/usr/local/share/datavolume3
Brick2: idc1-server2:/usr/local/share/datavolume3
Brick3: idc1-server3:/usr/local/share/datavolume3
Brick4: idc1-server4:/usr/local/share/datavolume3

5. 在idc1-server5上部署客户端并mount GlusterFS文件系统:
[root@idc1-server5 ~]# wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
[root@idc1-server5 ~]# yum install -y glusterfs glusterfs-server glusterfs-fuse
[root@idc1-server5 ~]# mkdir -p /mnt/{datavolume1,datavolume2,datavolume3}
[root@idc1-server5 ~]# mount -t glusterfs -o ro idc1-server1:datavolume1 /mnt/datavolume1/
[root@idc1-server5 ~]# mount -t glusterfs -o ro idc1-server1:datavolume2 /mnt/datavolume2/
[root@idc1-server5 ~]# mount -t glusterfs -o ro idc1-server1:datavolume3 /mnt/datavolume3/
[root@idc1-server5 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_t-lv_root
                       59G  8.0G   48G  15% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/xvda1            485M   33M  428M   8% /boot
idc1-server1:datavolume1       393G   63G  311G  17% /mnt/datavolume1
idc1-server1:datavolume2        393G   63G  311G  17% /mnt/datavolume2
idc1-server1:datavolume3        393G   63G  311G  17% /mnt/datavolume3

6. 相关数据读写可用性测试:
在idc1-server5挂载点上写入数据:
[root@idc1-server5 ~]# umount /mnt/datavolume1
[root@idc1-server5 ~]# mount -t glusterfs idc1-server1:datavolume1 /mnt/datavolume1/
[root@idc1-server5 ~]# echo "This is idc1-server5" > /mnt/datavolume1/hello.txt
[root@idc1-server5 ~]# mkdir /mnt/datavolume1/testdir
在idc1-server1数据目录中进行查看:
[root@idc1-server1 ~]# ls /usr/local/share/datavolume1/

hello.txt testdir

结果: 数据写入成功

在idc1-server1数据目录中直接写入数据:
[root@idc1-server1 ~]# echo "This is idc1-server1" > /usr/local/share/datavolume1/hello.2.txt
[root@idc1-server1 ~]# mkdir /usr/local/share/datavolume1/test2
在idc1-server5挂载点上进行查看:
[root@idc1-server5 ~]# ls /mnt/datavolume1
[root@idc1-server5 ~]# ls -l /mnt/datavolume1

hello.txt testdir

结果: 数据未同步

在idc1-server1挂载点上写入数据:
[root@idc1-server1 ~]# mount -t glusterfs localhost:datavolume1 /mnt/datavolume1/
[root@idc1-server1 ~]# echo "This is idc1-server1" > /mnt/datavolume1/hello.3.txt
[root@idc1-server1 ~]# mkdir /mnt/datavolume1/test3
在idc1-server5挂载点上进行查看:
[idc1-server5][root@idc1-server5 datavolume1]# ls /mnt/datavolume1

hello.2.txt  hello.3.txt hello.txt  test2  test3  testdir

结果: 数据同步成功,同时之前未同步的数据也成功加载了。

最终结论:
在数据目录中直接写入数据,会导致其它节点因为得不到通知而使数据同步失败。
正确的做法是所有的读写操作都通过挂载点来进行。

7. 其它操作笔记:
删除GlusterFS磁盘:
# gluster volume stop datavolume1
# gluster volume delete datavolume1

卸载GlusterFS磁盘:
# gluster peer detach idc1-server4

ACL访问控制:
# gluster volume set datavolume1 auth.allow 10.100.1.*,10.200.1.*

添加GlusterFS节点:
# gluster peer probe idc1-server5
# gluster peer probe idc1-server6
# gluster volume add-brick datavolume1 idc1-server5:/usr/local/share/datavolume1 idc1-server6:/usr/local/share/datavolume1

迁移GlusterFS磁盘数据:
# gluster volume remove-brick datavolume1 idc1-server1:/usr/local/share/datavolume1 idc1-server5:/usr/local/share/datavolume1 start
# gluster volume remove-brick datavolume1 idc1-server1:/usr/local/share/datavolume1 idc1-server5:/usr/local/share/datavolume1 status
# gluster volume remove-brick datavolume1 idc1-server1:/usr/local/share/datavolume1 idc1-server5:/usr/local/share/datavolume1 commit

数据重新分配:
# gluster volume rebalance datavolume1 start
# gluster volume rebalance datavolume1 status
# gluster volume rebalance datavolume1 stop

修复GlusterFS磁盘数据(例如在idc1-server1宕机的情况下):
# gluster volume replace-brick datavolume1 idc1-server1:/usr/local/share/datavolume1 idc1-server5:/usr/local/share/datavolume1 commit -force
# gluster volume heal datavolume1 full

,

  1. #1 by Magine on 2014/04/06 - 22:30

    第6点,创建testdir目录时,路径好像有点问题。
    另外在添加和移除贡献磁盘时,最好做一次数据重新分配:
    gluster volume rebalance datavolume1 start|status|stop

    • #2 by mcsrainbow on 2014/04/08 - 10:39

      谢谢,已更正。

  2. #3 by chris on 2014/07/28 - 16:12

    当idc1-server1 宕机的时候,client 能快速切换到其他的挂载服务器上吗?

    • #4 by mcsrainbow on 2014/07/28 - 19:10

      使用Redhat版本的RPM包,可以在Client上实现高可用自动切换。
      具体内容可以看我的这篇英文文章:
      Setup GlusterFS with Distributed Replicated Volumes and Native client

      • #5 by ffmm on 2014/08/27 - 18:49

        英文与中文是同样的吧?没看出如何在Client实现高可用自动切换。。。

        • #6 by mcsrainbow on 2014/08/27 - 21:44

          mount -t glusterfs -o backup-volfile-servers=idc1-server2:idc1-server3:idc1-server4,ro idc1-server1:datavolume2 /mnt/datavolume2/

(will not be published)
*