Archive for September, 2012

How to migrate the VMs’ IP addresses of CloudStack

Backgroud: 
We had two CloudStack Servers in Wuhan, which were running over 30 VMs. Their IP addresses were all like 172.16.x.x, after we moved them to Chengdu office, the network became a big trouble, because we are using the 10.6.x.x, so we have to migrate all VMs’ IP addresses to make sure they can be accessed. 

We searched a lot on Google and official forum, asked the official technical support team. Then we knew that the CloudStack doesn’t provide this function yet, but they suggested that we can try to change all IP addresses of VMs in Database, then restart the CloudStack. 

Solution: 
Finally, we made it by the following steps: 

1. First, we should backup all configuration files and the Database. 
Log into the management server, stop all VMs in page “Instances”. 
Waiting until all VMs stopped, the time depends on the number of the running VMs. 
Stop the management server and agent:

$ sudo /etc/init.d/cloud-management stop
$ sudo /etc/init.d/cloud-agent stop

Backup the configuration files:

$ sudo cp -rpa /etc/cloud /etc/cloud.bak
$ sudo cp -rpa /var/lib/cloud /var/lib/cloud.bak

Backup the Database:

$ mysqldump -uroot -p123456 cloud > cloud.sql

2. Change the IP addresses of OS 
Backup the network settings:

$ sudo cp -p /etc/network/interfaces /etc/network/interfaces.bak

Update the network settings:

$ sudo vi /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 10.6.8.200
netmask 255.255.0.0
gateway 10.6.255.1

auto cloudbr0
iface cloudbr0 inet static
bridge_ports eth0
address 10.6.8.201
netmask 255.255.0.0

Restart the OS:

$ sudo sync
$ sudo reboot

After the OS restart, we may need to do the following steps to make the network works:

$ sudo ifdown cloudbr0
$ sudo ifup cloudbr0

$ sudo ifdown eth0 
$ sudo ifup eth0 

3. Update the configuration files. 
Check all IP settings in configuration files:

$ sudo grep 172.16 -r /etc/cloud/

Update all settings to 10.6.x.x:

$ sudo vi /etc/cloud/management/db.properties
cluster.node.IP=10.6.8.201

$ sudo vi /etc/cloud/agent/agent.properties
host.ip=10.6.8.201

$ sudo vi /etc/cloud/agent/agent.properties
host=10.6.8.201

4. Update all IP settings in Database: 
There are so many data about the IP settings in Database, so we need some scripts. 

Update the IP addresses of table “user_ip_address”:

#!/bin/sh
mysql -uroot -p123456 cloud -e "select * from user_ip_address" > cloud.user_ip_address.list
sed -i '1d' cloud.user_ip_address.list

for public_ip_address in `cat cloud.user_ip_address.list|awk '{print $5}'`
 do
        value3=`echo $public_ip_address | awk -F "." '{print $3}'`
        value4=`echo $public_ip_address | awk -F "." '{print $4}'`
        id=`grep -w $public_ip_address cloud.user_ip_address.list | awk '{print $1}'`
        echo mysql -uroot -p123456 -e "update cloud.user_ip_address set public_ip_address="10.6.$value3.$value4" where id="$id";"
        mysql -uroot -p123456 -e "update cloud.user_ip_address set public_ip_address='10.6.$value3.$value4' where id='$id';"
done

Update the IP addresses of table “nics”:

#!/bin/sh
mysql -uroot -p123456 cloud -e "select * from nics" | grep 172.16 > cloud.nics.list

for ip4_address in `cat cloud.nics.list|awk '{print $5}'`
do
        value3=`echo $ip4_address | awk -F "." '{print $3}'`
        value4=`echo $ip4_address | awk -F "." '{print $4}'`
        for id in `grep -w $ip4_address cloud.nics.list.2 | awk '{print $1}'`
        do
                echo mysql -uroot -p123456 -e "update cloud.nics set ip4_address='10.6.$value3.$value4' where id="$id";"
        echo mysql -uroot -p123456 -e "update cloud.nics set gateway='10.6.255.1' where id="$id";"
                mysql -uroot -p123456 -e "update cloud.nics set ip4_address='10.6.$value3.$value4' where id="$id";"
                mysql -uroot -p123456 -e "update cloud.nics set gateway='10.6.255.1' where id="$id";"
        done
done

Update the IP addresses of table “op_dc_ip_address_alloc”:

#!/bin/sh
mysql -uroot -p123456 cloud -e "select * from op_dc_ip_address_alloc" > cloud.op_dc_ip_address_alloc.list
sed -i '1d' cloud.op_dc_ip_address_alloc.list

for ip_address in `cat cloud.op_dc_ip_address_alloc.list|awk '{print $2}'`
do
    value3=`echo $ip_address | awk -F "." '{print $3}'`
    value4=`echo $ip_address | awk -F "." '{print $4}'`
    id=`grep -w $ip_address cloud.op_dc_ip_address_alloc.list | awk '{print $1}'`
    echo mysql -uroot -p123456 -e "update cloud.op_dc_ip_address_alloc set ip_address="10.6.$value3.$value4" where id="$id";"
    mysql -uroot -p123456 -e "update cloud.op_dc_ip_address_alloc set ip_address='10.6.$value3.$value4' where id='$id';"
done

Update the IP addresses of table “vm_instance”:

#!/bin/sh
mysql -uroot -p123456 cloud -e "select id,private_ip_address from vm_instance;" > cloud.vm_instance.list
sed -i '1d' cloud.vm_instance.list

for private_ip_address in `cat cloud.vm_instance.list|awk '{print $2}'`
do
    value3=`echo $private_ip_address | awk -F "." '{print $3}'`
    value4=`echo $private_ip_address | awk -F "." '{print $4}'`
    echo mysql -uroot -p123456 -e "update cloud.vm_instance set private_ip_address="10.6.$value3.$value4" where private_ip_address="172.16.$value3.$value4";"
    mysql -uroot -p123456 -e "update cloud.vm_instance set private_ip_address='10.6.$value3.$value4' where private_ip_address='172.16.$value3.$value4';"
done

Then we should update the IP addresses of other tables manually. 

We can use this script to find out the tables which has the IP addresses data:

#!/bin/sh
mysql -uroot -p123456 cloud -e "show tables;" > tables.list

for table in `cat tables.list`
do
    tabledata=`mysql -uroot -p123456 cloud -e "select * from $table;"`
    echo $tabledata | grep --color=auto "$1" && echo ============= $table
done

Save the script as check.sh then use the following command to check:

$ ./check.sh 172.16

All data in DB related the 172.16 will be displayed, then we should update them all except these tables: 

“alert;usage_event” 

It takes a lot of time but we really need. 

5. Check the consistency of settings (pod,zone,cluster). 
If we started the agent or did something else before we’ve done all DB IP addresses update, the settings in /etc/cloud/agent/agent.properties could be changed by the agent itself. 

So we should check the consistency of settings between the file and Database:

$ mysql -uroot -p123456 cloud -e "select * from pod_vlan_map;"
+----+--------+------------+
| id | pod_id | vlan_db_id |
+----+--------+------------+
|  1 |      1 |          1 |
+----+--------+------------+

$ mysql -uroot -p123456 cloud -e "select * from cluster;"
+----+--------+--------------------------------------+------+--------+----------------+-----------------+--------------+------------------+---------------+---------+
| id | name   | uuid                                 | guid | pod_id | data_center_id | hypervisor_type | cluster_type | allocation_state | managed_state | removed |
+----+--------+--------------------------------------+------+--------+----------------+-----------------+--------------+------------------+---------------+---------+
|  1 | beluga | 51b5c695-8c94-4dfd-9f22-017dc9cf9af6 | NULL |      1 |              1 | KVM             | CloudManaged | Enabled          | Managed       | NULL    |
+----+--------+--------------------------------------+------+--------+----------------+-----------------+--------------+------------------+---------------+---------+

$ sudo cat /etc/cloud/agent/agent.properties
#Storage
#Fri Sep 14 10:04:27 CST 2012
guest.network.device=cloudbr0
workers=5
private.network.device=cloudbr0
port=8250
resource=com.cloud.agent.resource.computing.LibvirtComputingResource
pod=1
host.mac.address=c8\:60\:00\:6d\:55\:ad
vm.migrate.speed=0
zone=1
guid=c075ffc5-d6da-382b-826a-e2011c7badd1
host.ip=10.6.8.201
public.network.device=cloudbr0
cluster=1
local.storage.uuid=5ddd48d8-717b-4ba7-96b3-8690a6182c5f
domr.scripts.dir=scripts/network/domr/kvm
host=10.6.8.201
LibvirtComputingResource.id=1

6. Start the agent and management:

$ sudo /etc/init.d/cloud-agent start
$ sudo /etc/init.d/cloud-management start

7. Start the VMs 

Go to the page “Infrastructure” - “Zones” - “wuhan” - “System VMs”. 
Start the VMs “s-1-VM” and “v-2-VM” 

Go to the page “Infrastructure” - “Zones” - “wuhan” - “Physical Network” - “PhysicalNetworkInBasicZone” - “Network Service Providers” - “Virtual Router” - “Instances”. Start the VM “r-4-VM” 

Go to the page “Infrastructure” - “Zones” - “wuhan” - “Compute and Storage” - “Hosts”. 
Check the State of host, if it shows “Up”, means it’s OK. Otherwise we should check the log (/var/log/cloud/management/management-server.log) and fix it. 

After the host up, try to restart the Guest Network. 
“Network” - “guestNetworkForBasicZone” - “Restart Network” 

Then, we can try to start the VMs in Page “Instances” one by one. 

When we meet any Errors we should keep investigating from the logs: 
/var/log/cloud/management/management-server.log 
/var/log/cloud/agent/agent.log 

8. Something could help us 
We can log into the VMs as Local on the management web - “View console”. 
The Root Password of System VMs is “6m1ll10n”.

,

1 Comment

How to upgrade Zabbix from 1.8.x to 2.0.x

http://www.zabbix.com/documentation/2.0/manual/installation/upgrade_notes
http://www.zabbix.com/documentation/2.0/manual/installation/upgrade?s[]=upgrade

Upgraded from 1.8.2 to 2.0.2.

1. Stop Zabbix server
$ su - zabbix
$ sudo /etc/init.d/zabbix_server stop

2. Back up the existing Zabbix database
$ mkdir -p /home/zabbix/zabbix_upgrade/backup
$ cd /home/zabbix/zabbix_upgrade/backup
$ mysqldump -uroot -pzabbixadmin zabbix > zabbix.backup.sql

3. Back up configuration files, PHP files and zabbix binaries
$ cp -rp /etc/zabbix .
$ cp -rp /home/zabbix/public_html .
$ cp -rp /etc/init.d/zabbix-* .
$ cp -rp /usr/sbin/zabbix* .
$ cp -rp /usr/bin/zabbix* .

4. Install new server binaries
Download zabbix-2.0.2.tar.gz from website.
$ cd /home/zabbix/zabbix_upgrade/
$ tar xzvf zabbix-2.0.2.tar.gz
$ cd zabbix-2.0.2
$ ./configure --prefix=/usr --with-mysql --with-net-snmp --with-libcurl --enable-server --enable-agent
$ make
$ sudo make install
$ sudo zabbix_server --help
Zabbix server v2.0.2 (revision 29214) ...

5. Review server configuration parameters
$ cd /usr/etc/
$ sudo cp -rp /home/zabbix/zabbix_upgrade/backup/zabbix/zabbix_* .

6. Upgrade the database
Clean the history data:
$date +%s -d "2012/9/1 00:00:00"
1346428800

mysql> DELETE FROM `history_uint` WHERE `clock` < 1346428800; mysql> optimize table history_uint;

mysql> DELETE FROM `history` WHERE `clock` < 1346428800; mysql> optimize table history;

$ cd /home/zabbix/zabbix_upgrade/zabbix-2.0.2/upgrades/dbpatches/2.0/mysql
$ mysql -uroot -p123456 zabbix < patch.sql 7. Install new Zabbix web interface $ cd /home/zabbix/zabbix_upgrade/ $ sudo mv /home/zabbix/public_html /home/zabbix/public_html_bak $ sudo mkdir /home/zabbix/public_html $ sudo chown -R zabbix:zabbix /home/zabbix/public_html $ cp -r zabbix-2.0.2/frontends/php/* /home/zabbix/public_html/ 8. Start new Zabbix binaries $ /etc/init.d/zabbix-server start $ ps aux | grep zabbix_server Open the webpage http://IP/zabbix/ Then follow the steps to setup the Zabbix WebUI. Download the zabbix.conf.php and put it into /home/zabbix/public_html/conf/ Restart Zabbix server $ /etc/init.d/zabbix-server stop $ /etc/init.d/zabbix-server start 9. Upgrade Zabbix agent clients We don't have to upgrade all Zabbix agent clients, but we'd better to do it if we can. Download zabbix-2.0.2.tar.gz from website. $ sudo /etc/init.d/zabbix_agentd stop $ mkdir -p /home/zabbix/zabbix_upgrade/backup $ cd /home/zabbix/zabbix_upgrade/backup $ cp -rp /etc/zabbix . $ cp -rp /etc/init.d/zabbix-* . $ cp -rp /usr/sbin/zabbix* . $ cp -rp /usr/bin/zabbix* . $ cd /home/zabbix/zabbix_upgrade/ $ tar xzvf zabbix-2.0.2.tar.gz $ cd zabbix-2.0.2 $ ./configure --prefix=/usr --with-net-snmp --with-libcurl --enable-agent $ make $ sudo make install $ cd /usr/etc/ $ sudo cp -rp /home/zabbix/zabbix_upgrade/backup/zabbix/zabbix_* . $ sudo zabbix_agent --help Zabbix agent v2.0.2 (revision 29214) ... $ sudo /etc/init.d/zabbix_agentd start 10. Done 11. Something we should pay attention Configuration files location changed: /etc/zabbix → /usr/etc But except the /etc/zabbix/alert.d/. {IPADDRESS} macro deprecated, use {HOST.IP}

5 Comments

Fork me on GitHub