关于 十二月, 2014 的文章

AWS自动化运维脚本分享

背景介绍:
目前项目中使用了大量的AWS EC2 Instances作为服务器,在自动化运维方面,我们之前一直使用的是AWS CLI命令行工具,然后在Shell脚本中调用。
最近我想通过脚本实现一个“Clone”的功能,模拟Web Console上的“Launch More Like This”来创建Instance。但在Shell脚本中实现起来感觉不太舒服,于是就直接利用Python的boto库写了一个,在此分享给大家。

具体内容:
脚本地址:https://github.com/mcsrainbow/python-demos/blob/master/demos/awscli.py

相关示例:
温馨提示:我的Blog页面默认没有采用宽屏模式,如果觉得下面的代码不太美观,可以点击右上角的“<>”切换到宽屏模式。
$ ./awscli.py -h

usage: awscli.py [-h] (--create | --clone | --terminate) --region REGION
                 [--instance_name INSTANCE_NAME] [--image_id IMAGE_ID]
                 [--instance_type INSTANCE_TYPE] [--key_name KEY_NAME]
                 [--security_group_ids SECURITY_GROUP_IDS]
                 [--subnet_id SUBNET_ID]
                 [--src_instance_name SRC_INSTANCE_NAME]
                 [--dest_instance_name DEST_INSTANCE_NAME]
                 [--private_ip_address PRIVATE_IP_ADDRESS]
                 [--instance_id INSTANCE_ID] [--volume_size VOLUME_SIZE]
                 [--volume_type {standard,io1,gp2}]
                 [--volume_zone VOLUME_ZONE] [--volume_iops VOLUME_IOPS]
                 [--volume_delete_on_termination]
                 [--load_balancer_name LOAD_BALANCER_NAME]
                 [--ignore_load_balancer]
                 [--quick]

examples:
  ./awscli.py --create --region us-west-1 --instance_name idc1-server2 \
              --image_id ami-30f01234 --instance_type t1.micro \
              --key_name idc1-keypair1 --security_group_ids sg-eaf01234f \
              --subnet_id subnet-6d901234
  ./awscli.py --create --region us-west-1 --instance_name idc1-server3 \
              --image_id ami-30f01234 --instance_type t1.micro \
              --key_name idc1-keypair1 --security_group_ids sg-eaf01234f \
              --subnet_id subnet-6d901234 --volume_size 10 --volume_type gp2 \
              --volume_zone us-west-1a --volume_delete_on_termination \
              --load_balancer_name idc1-elb1 --private_ip_address 172.16.2.23
  ./awscli.py --clone --region us-west-1 --src_instance_name idc1-server1 \
              --dest_instance_name idc1-server2
  ./awscli.py --clone --region us-west-1 --src_instance_name idc1-server1 \
              --dest_instance_name idc1-server3 --private_ip_address 172.16.2.23
  ./awscli.py --clone --region us-west-1 --src_instance_name idc1-server1 \
              --dest_instance_name idc1-server3 --private_ip_address 172.16.2.23 \
              --ignore_load_balancer
  ./awscli.py --terminate --region us-west-1 --instance_name idc1-server3
  ./awscli.py --terminate --region us-west-1 --instance_id i-01234abc
  ./awscli.py --terminate --region us-west-1 --instance_id i-01234abc --quick
  ...

optional arguments:
  -h, --help            show this help message and exit
  --create              create instance
  --clone               clone instance
  --terminate           terminate instance
  --region REGION
  --instance_name INSTANCE_NAME
  --image_id IMAGE_ID
  --instance_type INSTANCE_TYPE
  --key_name KEY_NAME
  --security_group_ids SECURITY_GROUP_IDS
  --subnet_id SUBNET_ID
  --src_instance_name SRC_INSTANCE_NAME
  --dest_instance_name DEST_INSTANCE_NAME
  --private_ip_address PRIVATE_IP_ADDRESS
  --instance_id INSTANCE_ID
  --volume_size VOLUME_SIZE
                        in GiB
  --volume_type {standard,io1,gp2}
  --volume_zone VOLUME_ZONE
  --volume_iops VOLUME_IOPS
  --volume_delete_on_termination
                        delete volumes on termination
  --load_balancer_name LOAD_BALANCER_NAME
  --ignore_load_balancer
                        ignore load balancer setting
  --quick               no wait on termination

$ ./awscli.py --create --region us-west-1 --instance_name idc1-server1 --image_id ami-30f01234 \
--instance_type t1.micro --key_name idc1-keypair1 --security_group_ids sg-eaf01234f \
--subnet_id subnet-6d901234 --volume_size 10 --volume_type gp2 --volume_zone us-west-1a \
--volume_delete_on_termination --load_balancer_name idc1-elb1 --private_ip_address 172.16.2.21

1. Launching instance: idc1-server1
2. Creating tag as instance name: {"Name": idc1-server1}
Instance state: pending
Instance state: running
3. Creating secondary volume for instance: idc1-server1 as gp2 10G
Volume status: available
4. Attaching volume: vol-4ba6a54c to instance: idc1-server1 as device: /dev/sdf
5. Adding instance: idc1-server1 to ELB: idc1-elb1

$ ./awscli.py --clone --region us-west-1 --src_instance_name idc1-server1 --dest_instance_name idc1-server2

1. Launching instance: idc1-server2
2. Creating tag as instance name: {"Name": idc1-server2}
Instance state: pending
Instance state: running
3. Creating secondary volume for instance: idc1-server2 as gp2 10G
Volume status: available
4. Attaching volume: vol-5b61635c to instance: idc1-server2 as device: /dev/sdf
5. Adding instance: idc1-server2 to ELB: idc1-elb1

$ ./awscli.py --terminate --region us-west-1 --instance_name idc1-server2

Terminating instance: idc1-server2 id: i-86976d62
Instance state: shutting-down
Instance state: shutting-down
Instance state: terminated

, ,

4 Comments

在AWS EC2中创建不含Marketplace code的CentOS6 AMI

参考资料:
https://www.caseylabs.com/remove-the-aws-marketplace-code-from-a-centos-ami/
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage_expand_partition.html

背景介绍:
在AWS EC2中,从Marketplace里面可以很方便的选择最新的CentOS6的官方Minimal版本的AMI,来创建Instance。
但是这里面却埋了一个大坑,那就是,所有基于Marketplace里面的AMI所创建的Instance,都会带有一个Marketplace code。
它会导致你无法通过为现有根分区所在的EBS Volume创建Snapshot和新的Volume的方式来对其扩容。
在Detach了现有的根分区所在的Volume后,将无法再次将其Attach到Instance当中,在Attach新的Volume时也会遇到相同的报错:

Client.OperationNotPermitted:
'vol-xxxxxxx' with Marketplace codes may not be attached as a secondary device.

这个Marketplace code,顾名思义,应该就是为了保护一些付费的AMI不被随意的克隆,但不知道为什么没有对费用为$0的CentOS6 AMI做单独的处理。
上面的限制,主要影响到的是,默认创建好的CentOS6 Instance的EBS Volume只有8G,即使在创建时指定了50G的EBS Volume,创建后的根分区空间也只有8G。这样的大小是无法满足线上需求的,只能对其进行扩容,而因为有上面的Marketplace code的限制,又使扩容变得很艰难。
还好最终我通过参考上面的两篇文章,从官方的CentOS6 AMI中移除了Marketplace code,并成功的对根分区进行了扩容并创建了相应的AMI。

具体步骤:
1. 从现有的CentOS6 AMI中移除Marketplace code
1.1 从AWS的Marketplace搜索CentOS6 AMI,并创建一个根分区所在的EBS Volume为8G(默认大小)的Instance;
1.2 在AWS EC2 web console中,再创建一个新的大小为8G的EBS Volume;
1.3 将新创建的EBS Volume Attach到Instance上,通常会默认识别为/dev/xvdj(HVM版本的AMI会识别为/dev/xvdf);
1.4 通过SSH登陆到Instance,并通过dd克隆根分区所在的EBS Volume(HVM版本的AMI会将根目录所在的EBS Volume识别为/dev/xvda):

dd bs=65536 if=/dev/xvde of=/dev/xvdj

1.5 当克隆完成以后,关闭Instance;
1.6 Detach现有根分区所在的EBS Volume;
1.7 Detach新创建的EBS Volume,并重新Attach到Instance,作为/dev/sda(HVM版本的AMI需要指定为/dev/sda1);
1.8 启动Instance;
1.9 在确认Instance正常启动后,在EC2 web console中右键点击Instance,并选择Create Image,即可创建一个新的不含Marketplace code的CentOS6 AMI了,我一般将其命名为official_centos6_x86_64_minimal_ebs8g。

2. 将现有的AMI根分区所在的EBS Volume扩容为50G,并创建新的AMI official_centos6_x86_64_minimal_ebs50g
2.1 基于AMI official_centos6_x86_64_minimal_ebs8g创建一个Instance;
2.2 为Instance所在的EBS Volume创建一个Snapshot;
2.3 创建一个新的大小为50G的Volume,并包含刚刚创建的Snapshot;
2.4 将新创建的Volume Attach到Instance,作为第二块EBS Volume,默认会识别为/dev/xvdj(HVM版本的AMI会识别为/dev/xvdf);
2.5 在Instance上对第二块EBS Volume进行扩容,详细步骤如下(HVM版本的AMI会将根目录所在的EBS Volume识别为/dev/xvda):

[root@ip-172-17-4-12 ~]# parted /dev/xvdj
GNU Parted 2.1
Using /dev/xvdj
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit s
(parted) print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdj: 104857600s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End        Size       Type     File system  Flags
 1      2048s  16777215s  16775168s  primary  ext4         boot

(parted) rm 1
(parted) mkpart primary 2048s 100%
(parted) print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdj: 104857600s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End         Size        Type     File system  Flags
 1      2048s  104857599s  104855552s  primary  ext4

(parted) set 1 boot on
(parted) print
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdj: 104857600s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End         Size        Type     File system  Flags
1      2048s  104857599s  104855552s  primary  ext4         boot

(parted) quit
Information: You may need to update /etc/fstab.

[root@ip-172-17-4-12 ~]# e2fsck -f /dev/xvdj1
e2fsck 1.41.12 (17-May-2010)
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway<y>? yes

/dev/xvdj1: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/xvdj1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/xvdj1: 18425/524288 files (0.2% non-contiguous), 243772/2096896 blocks

[root@ip-172-17-4-12 ~]# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvde    202:0    0   8G  0 disk
└─xvde1 202:1    0   8G  0 part /
xvdj    202:80   0  50G  0 disk
└─xvdj1 202:     0  50G  0 part

[root@ip-172-17-4-12 ~]# resize2fs /dev/xvdj1
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/xvdj1 to 13106944 (4k) blocks.
The filesystem on /dev/xvdj1 is now 13106944 blocks long.

2.6 关闭Instance;
2.7 Detach现有根分区所在的EBS Volume;
2.8 Detach扩容后的第二块EBS Volume,并重新Attach到Instance,作为/dev/sda(HVM版本的AMI需要指定为/dev/sda1);
2.9 启动Instance;
2.10 在确认Instance正常启动后,在EC2 web console中右键点击Instance,并选择Create Image,即可创建一个新的根分区大小为50G的CentOS6 AMI了,我一般将其命名为official_centos6_x86_64_minimal_ebs50g。

PS: 自己制作的镜像,如果要支持创建时自定义指定SSH Public Key,需要确保/etc/rc.local中包含以下代码,同时将镜像中的/root/.ssh/authorized_keys文件删除。

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

# set a random pass on first boot
if [ -f /root/firstrun ]; then
  dd if=/dev/urandom count=50|md5sum|passwd --stdin root
  passwd -l root
  rm /root/firstrun
fi

if [ ! -d /root/.ssh ]; then
  mkdir -m 0700 -p /root/.ssh
  restorecon /root/.ssh
fi

# Get the root ssh key setup
ReTry=0
while [ ! -f /root/.ssh/authorized_keys ] && [ $ReTry -lt 10 ]; do
  sleep 2
  curl -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /root/.ssh/pubkey
  if [ 0 -eq 0 ]; then
    mv /root/.ssh/pubkey /root/.ssh/authorized_keys
  fi
  ReTry=$[Retry+1]
done
chmod 600 /root/.ssh/authorized_keys && restorecon /root/.ssh/authorized_keys

, , , , ,

No Comments