Troubleshooting huge MySQL temp files

Tonight I got an alert about the disk usage on db1.idc1, its root partition usage was higher than 96% and was increasing.

[root@db1.idc1 ~]# df -hP
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/mylvm-root 102G 92G 4.5G 96% /
tmpfs 32G 0 32G 0% /dev/shm
/dev/sda1 240M 30M 198M 13% /boot
/dev/mapper/mylvm2-var 1.1T 152G 893G 15% /var

After a quick troubleshoot, I found some MySQL temp files took most spaces.

[root@db1.idc1 ~]# cd /tmp
[root@db1.idc1 tmp]# ls -lh | grep G
total 78G
-rw-rw---- 1 mysql mysql 32G Aug 5 07:28 #sql_4862_0.MYD
-rw-rw---- 1 mysql mysql 7.5G Aug 5 07:28 #sql_4862_0.MYI
-rw-rw---- 1 mysql mysql 32G Aug 5 07:28 #sql_4862_2.MYD
-rw-rw---- 1 mysql mysql 7.5G Aug 5 07:28 #sql_4862_2.MYI

[root@db1.idc1 tmp]# lsof | grep /tmp/#sql_4862_0.MYD
mysqld 18530 mysql 282u REG 253,0 33339280800 2883602 /tmp/#sql_4862_0.MYD

[root@db1.idc1 tmp]# ps aux | grep 18530
root 16365 0.0 0.0 103304 916 pts/0 S+ 07:30 0:00 grep --colour=auto 18530
mysql 18530 66.5 4.2 15718552 2795960 ? Sl May20 73373:06 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/db1.idc1.err --pid-file=/var/lib/mysql/db1.idc1.pid --socket=/var/lib/mysql/mysql.sock --port=3306

Then found that two slow queries were running on db1.idc1.
And those two queries were selecting data from 52320599850 == 52 billion rows, from the host 10.8.8.88.

[root@db1.idc1 tmp]# mysql -uusername -ppassword -e 'show full processlist\G'
...
*************************** 10. row ***************************
Id: 34770751
User: username
Host: 10.8.8.88:50657
db: dbname
Command: Query
Time: 303040
State: Sending data
Info: Select ...
Rows_sent: 0
Rows_examined: 52320599850
Rows_read: 52320599850

*************************** 11. row ***************************
Id: 34824847
User: username
Host: 10.8.8.88:50829
db: dbname
Command: Query
Time: 302476
State: Sending data
Info: Select ...
Rows_sent: 0
Rows_examined: 52240222853
Rows_read: 52240222853
...

I killed those two queries then saved the db1.idc1, asked the developers to improve the queries.

[root@db1.idc1 tmp]# mysql -uusername -ppassword
mysql> kill 34770751;
Query OK, 0 rows affected (0.00 sec)

mysql> kill 34824847;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye


[root@db1.idc1 tmp]# df -hP
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/mylvm-root 102G 12G 85G 12% /
tmpfs 32G 0 32G 0% /dev/shm
/dev/sda1 240M 30M 198M 13% /boot
/dev/mapper/mylvm2-var 1.1T 152G 892G 15% /var

No Comments

Zabbix practices

Some useful informations about Zabbix server in production.

1. Zabbix server summary:
Hosts: 2,000+
Items: 170,000+ (After disabled 50% non-important items from 300,000+ items)
Zabbix Server: 1
Zabbix Proxy: 3 (For other 3 data centers, total backend hosts < 500)
MySQL Data size: 200G(with TokuDB engine and keep history data in 30 days)
MySQL QPS: 10,000~15,000

2. Zabbix server hardware:
CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz * 24 Cores
Memory: 64G RAM
Hard drives: 300G SAS*2(for OS)
Data drives: Shannon Direct-IO SSD G2(for MySQL)

3. Zabbix Server optimization:
MySQL: Percona TokuDB engine + Split big tables about history data
Items: Over 95% items are active/trapper

No Comments

pfSense sucks

Never use pfSense as the Gateway in production environment, sometimes you have to reboot the whole OS to fix some issues, but the reboot means serious downtime especially when there are some VPN connections between different colocations.

Just buy some professional firewall network devices...

2 Comments

how to create RPM and YUM repository

Reference:
http://www.ibm.com/developerworks/cn/linux/management/package/rpm/part1/
http://www.ibm.com/developerworks/cn/linux/management/package/rpm/part2/

Goal:
Make a RPM package for compiled nginx 1.1.9 with pcre 8.30 and zlib 1.2.6
Crate a private repository to manage the RPM packages

1. Create a softlink for my home
$ cd /home
$ sudo ln -s dong dong.guo

2. Install some compilation tools
$ sudo yum install gcc gcc-c++ autoconf automake zlib libxml ncurses-devel libgcrypt libtool

3. Install the RPM build tools
$ sudo yum install rpm-build rpmdevtools

4. Configure the RPM build environment
$ cd /home/dong.guo
$ mkdir rpmbuild
$ cd rpmbuild
$ mkdir -pv {BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS,TMP}

$ cd /home/dong.guo
$ sudo vim .rpmmacros
$ sudo cp .rpmmacros /root/

%_topdir /home/dong.guo/rpmbuild
%_tmppath /home/dong.guo/rpmbuild/TMP

5. Download the source packages for nginx-1.1.9
$ cd /home/dong.guo/rpmbuild/SOURCES
$ wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.30.tar.gz
$ wget http://sourceforge.net/projects/libpng/files/zlib/1.2.6/zlib-1.2.6.tar.gz/download
$ wget http://nginx.org/download/nginx-1.1.9.tar.gz

$ sudo vim nginx

#!/bin/sh
#
# nginx - this script starts and stops the nginx daemin
#
# chkconfig:   - 85 15
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /usr/local/nginx/conf/nginx.conf
# pidfile:     /usr/local/nginx/logs/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/opt/nginx/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/opt/nginx/conf/nginx.conf"

lockfile=/opt/nginx/lock/nginx.lock

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    start
}

reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit 2
esac

$ sudo chmod +x nginx

$ tree

├── BUILD
├── BUILDROOT
├── RPMS
├── SOURCES
│   ├── nginx
│   ├── nginx-1.1.9.tar.gz
│   ├── pcre-8.30.tar.gz
│   └── zlib-1.2.6.tar.gz
├── SPECS
├── SRPMS
└── TMP

6. Configure the SPEC file
$ cd /home/dong.guo/rpmbuild/SPECS
$ vim heylinux-nginx-1.1.9.spec

Name:           heylinux-nginx
Version:        1.1.9
Release:        el6
Summary:        Compiled nginx 1.1.9 with pcre 8.30 and zlib 1.2.6

Group:          System Environment/Daemons
License:        GPL
URL:            http://www.drawbrid.ge
Source0:        nginx-1.1.9.tar.gz
Source1:        pcre-8.30.tar.gz
Source2:        zlib-1.2.6.tar.gz

Vendor:         guosuiyu@gmail.com

BuildRequires:  gcc gcc-c++ autoconf automake openssl-devel

%description
This RPM package is made for compiled nginx 1.1.9, pcre 8.30 and zlib 1.2.6 are not installed,
they just provide the packages for nginx build process.

%prep
tar xzvf $RPM_SOURCE_DIR/pcre-8.30.tar.gz
tar xzvf $RPM_SOURCE_DIR/zlib-1.2.6.tar.gz
tar xzvf $RPM_SOURCE_DIR/nginx-1.1.9.tar.gz

%build
cd nginx-1.1.9
./configure --prefix=/opt/nginx \
--user=nginx \
--group=nginx \
--pid-path=/opt/nginx/run/nginx.pid \
--lock-path=/opt/nginx/lock/nginx.lock \
--with-http_ssl_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_realip_module \
--with-http_gzip_static_module \
--with-http_stub_status_module \
--with-mail \
--with-mail_ssl_module \
--with-pcre=../pcre-8.30 \
--with-zlib=../zlib-1.2.6 \
--with-debug \
--http-client-body-temp-path=/opt/nginx/tmp/client \
--http-proxy-temp-path=/opt/nginx/tmp/proxy \
--http-fastcgi-temp-path=/opt/nginx/tmp/fastcgi \
--http-uwsgi-temp-path=/opt/nginx/tmp/uwsgi \
--http-scgi-temp-path=/opt/nginx/tmp/scgi

%install
rm -rf $RPM_BUILD_ROOT
cd nginx-1.1.9/
mkdir -p $RPM_BUILD_ROOT/opt/nginx/tmp
mkdir -p $RPM_BUILD_ROOT/opt/nginx/run
mkdir -p $RPM_BUILD_ROOT/opt/nginx/lock
make install DESTDIR=%{buildroot}

mkdir -p $RPM_BUILD_ROOT/etc/init.d
cp -p $RPM_SOURCE_DIR/nginx $RPM_BUILD_ROOT/etc/init.d/

%clean
rm -rf $RPM_BUILD_DIR/pcre-8.30
rm -rf $RPM_BUILD_DIR/zlib-1.2.6
rm -rf $RPM_BUILD_DIR/nginx-1.1.9

%preun

%postun
userdel nginx

%files
/opt/nginx
/etc/init.d/nginx

%post
useradd -M nginx -s /sbin/nologin

%changelog

7. Build the RPM package
$ sudo rpmbuild -bb heylinux-nginx-1.1.9.spec

8. Found the RPM package
$ cd /home/dong.guo/rpmbuild/RPMS/x86_64/

$ ls

heylinux-nginx-1.1.9-el6.x86_64.rpm

9. Copy the RPM package to another server
$ scp -i /home/dong.guo/.ssh/id_rsa heylinux-nginx-1.1.9-el6.x86_64.rpm dong.guo@172.16.1.21:/home/dong/

10. Login the server 172.16.1.21 and install the RPM package
$ sudo rpm -ivh heylinux-nginx-1.1.9-el6.x86_64.rpm

Preparing...                ########################################### [100%]
   1:heylinux-nginx       ########################################### [100%]

$ sudo /etc/init.d/nginx start

Starting nginx:                                            [  OK  ]

$ sudo netstat -lntp | grep 80

tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      13581/nginx

11. Crate a private YUM repository
$ sudo yum install httpd
$ sudo yum install createrepo
$ sudo mkdir -p /var/www/html/heylinux/RPMS/x86_64
$ cd /var/www/html/heylinux/RPMS/x86_64
$ sudo cp /home/dong.guo/rpmbuild/RPMS/x86_64/heylinux-nginx-1.1.9-el6.x86_64.rpm .
$ sudo createrepo /var/www/html/heylinux/RPMS/x86_64/
$ sudo /etc/init.d/httpd start

12. Configure the YUM source on another server
$ sudo vim heylinux.repo

[heylinux]
name=heylinux
baseurl=http://172.16.1.21/heylinux/RPMS/x86_64/
gpgcheck=0
enabled=1

$ sudo yum install heylinux-nginx

Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
heylinux            |  951 B     00:00
heylinux/primary    |  978 B     00:00
heylinux                                                                                      1/1
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package heylinux-nginx.x86_64 0:1.1.9-el6 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package                     Arch              Version                  Repository             Size
====================================================================================================
Installing:
 heylinux-nginx            x86_64            1.1.9-el6               heylinux            386 k

Transaction Summary
====================================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)

Total download size: 386 k
Installed size: 940 k
Is this ok [y/N]: y
Downloading Packages:
heylinux-nginx-1.1.9-el6.x86_64.rpm      | 386 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing: 
  heylinux-nginx-1.1.9-el6.x86_64    1/1

Installed:
  heylinux-nginx.x86_64 0:1.1.9-el6

Complete!

13. Done

,

No Comments

BEC Notes - Ways of working

freelance - You sell your work or services to a number of different companies.
teleworking - You work for a company from home via email, phone or the Internet.
job-sharing - You do your job for part of the week and another person does it for the other part.
shift work - You work during different parts of the day (eg nights).
part-time - You only work for some of the week (not full-time).
temping - You work for different companies for a short time without a permanent contract.
consultancy - You aren't employed by a company, but are paid to give specialist advice.
flexitime - You work a number of hours per week or month but you decide when you start or finish.
hot-desking - You don't have a permanent place or office to work at, but you find a place to work when you arrive.

It's great because I'm my own boss, but I still work with lots of different people. - freelance
I wish I had my own space. I have to carry everything around in my bag and sometimes there's nowhere to sit. - hot-desking
I like it because it's only for a couple of months and I'm saving up to go round the world. - consultancy (me: A well paid part-time job for specialist.)
The only problem is sleeping. Your body never knows if it's night or day! - shift work (me: It sucks!)
It can get a bit lonely at times. And I miss my colleagues and all the office gossip. - part-time
My Children are at school so it lets me spend more time with them. - teleworking/temping
When one of us wants a week off, the other person does a few extra days so it's fairly flexible. - flexitime/job-sharing

How to job-share
1. Find the perfect partner.
2. Open your mind. Embrace the benefits.
3. Plan for disaster. Always discuss the worst-case scenarios, you may lost your job-share partner.
4. Get organised. Plan the system for handing work over carefully and play to each other's different strengths. Delegate the workload according to each other's particular skills and qualities. (me: It's not easy, but very helpful in teamwork.)
5. Set your limits. Managers should clarify what they expect in terms of hours, availability and results, and employees should manage their employer's expectations.
6. Put pen to pager. Agree in writing arrangements for holidays, parental leave, retirement, etc. Everyone should know where they stand from the beginning.
7. Don't fell guilty. Do not work until 1am at home to make up for not being in the office every day.
8. Two become one. Clients may not liking having to deal with two people working closely together. A job-share should be like a marriage.
(me: I really think that job-share is a difficult way of working, only few companies accept it and it's very difficult to find a good partner. )

To be continued...

No Comments

AWS automation script which supports "--clone" as "Launch More Like This" on web console

I just completed an automation script for AWS, it supports "--clone" function, likes the "Launch More Like This" on AWS web console, based on Python boto library.

Address:https://github.com/mcsrainbow/python-demos/blob/master/demos/awscli.py

$ ./awscli.py -h

usage: awscli.py [-h] (--create | --clone | --terminate) --region REGION
                 [--instance_name INSTANCE_NAME] [--image_id IMAGE_ID]
                 [--instance_type INSTANCE_TYPE] [--key_name KEY_NAME]
                 [--security_group_ids SECURITY_GROUP_IDS]
                 [--subnet_id SUBNET_ID]
                 [--src_instance_name SRC_INSTANCE_NAME]
                 [--dest_instance_name DEST_INSTANCE_NAME]
                 [--private_ip_address PRIVATE_IP_ADDRESS]
                 [--instance_id INSTANCE_ID] [--volume_size VOLUME_SIZE]
                 [--volume_type {standard,io1,gp2}]
                 [--volume_zone VOLUME_ZONE] [--volume_iops VOLUME_IOPS]
                 [--volume_delete_on_termination]
                 [--load_balancer_name LOAD_BALANCER_NAME]
                 [--ignore_load_balancer]
                 [--quick]

examples:
  ./awscli.py --create --region us-west-1 --instance_name idc1-server2 \
              --image_id ami-30f01234 --instance_type t1.micro \
              --key_name idc1-keypair1 --security_group_ids sg-eaf01234f \
              --subnet_id subnet-6d901234
  ./awscli.py --create --region us-west-1 --instance_name idc1-server3 \
              --image_id ami-30f01234 --instance_type t1.micro \
              --key_name idc1-keypair1 --security_group_ids sg-eaf01234f \
              --subnet_id subnet-6d901234 --volume_size 10 --volume_type gp2 \
              --volume_zone us-west-1a --volume_delete_on_termination \
              --load_balancer_name idc1-elb1 --private_ip_address 172.16.2.23
  ./awscli.py --clone --region us-west-1 --src_instance_name idc1-server1 \
              --dest_instance_name idc1-server2
  ./awscli.py --clone --region us-west-1 --src_instance_name idc1-server1 \
              --dest_instance_name idc1-server3 --private_ip_address 172.16.2.23
  ./awscli.py --clone --region us-west-1 --src_instance_name idc1-server1 \
              --dest_instance_name idc1-server3 --private_ip_address 172.16.2.23 \
              --ignore_load_balancer
  ./awscli.py --terminate --region us-west-1 --instance_name idc1-server3
  ./awscli.py --terminate --region us-west-1 --instance_id i-01234abc
  ./awscli.py --terminate --region us-west-1 --instance_id i-01234abc --quick
  ...

optional arguments:
  -h, --help            show this help message and exit
  --create              create instance
  --clone               clone instance
  --terminate           terminate instance
  --region REGION
  --instance_name INSTANCE_NAME
  --image_id IMAGE_ID
  --instance_type INSTANCE_TYPE
  --key_name KEY_NAME
  --security_group_ids SECURITY_GROUP_IDS
  --subnet_id SUBNET_ID
  --src_instance_name SRC_INSTANCE_NAME
  --dest_instance_name DEST_INSTANCE_NAME
  --private_ip_address PRIVATE_IP_ADDRESS
  --instance_id INSTANCE_ID
  --volume_size VOLUME_SIZE
                        in GiB
  --volume_type {standard,io1,gp2}
  --volume_zone VOLUME_ZONE
  --volume_iops VOLUME_IOPS
  --volume_delete_on_termination
                        delete volumes on termination
  --load_balancer_name LOAD_BALANCER_NAME
  --ignore_load_balancer
                        ignore load balancer setting
  --quick               no wait on termination
  ...

optional arguments:
  -h, --help            show this help message and exit
  --create              create instance
  --clone               clone instance
  --terminate           terminate instance
  --region REGION
  --instance_name INSTANCE_NAME
  --image_id IMAGE_ID
  --instance_type INSTANCE_TYPE
  --key_name KEY_NAME
  --security_group_ids SECURITY_GROUP_IDS
  --subnet_id SUBNET_ID
  --src_instance_name SRC_INSTANCE_NAME
  --dest_instance_name DEST_INSTANCE_NAME
  --private_ip_address PRIVATE_IP_ADDRESS
  --instance_id INSTANCE_ID
  --volume_size VOLUME_SIZE
                        in GiB
  --volume_type {standard,io1,gp2}
  --volume_zone VOLUME_ZONE
  --volume_iops VOLUME_IOPS
  --volume_delete_on_termination
                        delete volumes on termination
  --load_balancer_name LOAD_BALANCER_NAME
  --quick               no wait on termination

$ ./awscli.py --create --region us-west-1 --instance_name idc1-server1 --image_id ami-30f01234 \
--instance_type t1.micro --key_name idc1-keypair1 --security_group_ids sg-eaf01234f \
--subnet_id subnet-6d901234 --volume_size 10 --volume_type gp2 --volume_zone us-west-1a \
--volume_delete_on_termination --load_balancer_name idc1-elb1 --private_ip_address 172.16.2.21

1. Launching instance: idc1-server1
2. Creating tag as instance name: {"Name": idc1-server1}
Instance state: pending
Instance state: running
3. Creating secondary volume for instance: idc1-server1 as gp2 10G
Volume status: available
4. Attaching volume: vol-4ba6a54c to instance: idc1-server1 as device: /dev/sdf
5. Adding instance: idc1-server1 to ELB: idc1-elb1

$ ./awscli.py --clone --region us-west-1 --src_instance_name idc1-server1 --dest_instance_name idc1-server2

1. Launching instance: idc1-server2
2. Creating tag as instance name: {"Name": idc1-server2}
Instance state: pending
Instance state: running
3. Creating secondary volume for instance: idc1-server2 as gp2 10G
Volume status: available
4. Attaching volume: vol-5b61635c to instance: idc1-server2 as device: /dev/sdf
5. Adding instance: idc1-server2 to ELB: idc1-elb1

$ ./awscli.py --terminate --region us-west-1 --instance_name idc1-server2

Terminating instance: idc1-server2 id: i-86976d62
Instance state: shutting-down
Instance state: shutting-down
Instance state: terminated

, , ,

No Comments

Create an Ansible Module and then use Module Provided 'Facts'

Sometimes the default Ansible Facts are not enough.
For example, Ansible doesn't provide "ansible_private_ipv4_address" which with private IP address.
With it, we don't need to worry about which IP on which NIC should be used.

But we can create an Ansible module and then use module provided 'Facts'

The steps:
[root@idc-server2 ~]# ifconfig

eth0      Link encap:Ethernet  HWaddr 1B:2B:3B:4B:5B:6B
          inet addr:172.16.1.2  Bcast:172.16.1.255  Mask:255.255.252.0
...

eth1      Link encap:Ethernet  HWaddr 1A:2A:3A:4A:5A:6A
          inet addr:100.100.100.100  Bcast:100.100.100.255  Mask:255.255.255.240
...

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
...

[root@idc-server1 ansible]# vim myfacts.yml

---
- hosts: idc-server2
  roles:
  - myfacts

[root@idc-server1 ansible]# mkdir -p roles/myfacts/{tasks,templates}

[root@idc-server1 ansible]# vim roles/myfacts/tasks/main.yml

---
- name: run myfacts module to get customized facts
  myfacts: get_facts=yes

- name: update file with the customized facts
  template: src=myfacts.txt.j2 dest=/tmp/myfacts.txt

[root@idc-server1 ansible]# vim roles/myfacts/templates/myfacts.txt.j2

ansible_private_ipv4_address : {{ ansible_private_ipv4_address }}

[root@idc-server1 ansible]# mkdir -p library/heylinux
[root@idc-server1 ansible]# vim library/heylinux/myfacts

#!/usr/bin/python

import json
import commands
import re

def get_ansible_private_ipv4_address():
    iprex = "(^192\.168)|(^10\.)|(^172\.1[6-9])|(^172\.2[0-9])|(^172\.3[0-1])"
    output = commands.getoutput("""/sbin/ifconfig |grep "Link encap" |awk '{print $1}' |grep -wv 'lo'""")
    nics = output.split('\n')
    for i in nics:
        ipaddr = commands.getoutput("""/sbin/ifconfig %s |grep -w "inet addr" |cut -d: -f2 | awk '{print $1}'""" % (i))
        if re.match(iprex,ipaddr):
            ansible_private_ipv4_address = ipaddr
            return ansible_private_ipv4_address

def main():
    global module
    module = AnsibleModule(
        argument_spec = dict(
            get_facts=dict(default="yes", required=False),
        ),
        supports_check_mode = True,
    )

    ansible_facts_dict = {
        "changed" : False,
        "ansible_facts": {
            }
    }

    if module.params['get_facts'] == 'yes':
        ansible_private_ipv4_address = get_ansible_private_ipv4_address()
        ansible_facts_dict['ansible_facts']['ansible_private_ipv4_address'] = ansible_private_ipv4_address

    print json.dumps(ansible_facts_dict)

from ansible.module_utils.basic import *
from ansible.module_utils.facts import *
main()

[root@idc-server1 ansible]# ansible-playbook -u root myfacts.yml -i hosts

PLAY [idc1-server2] ***************************************************************

GATHERING FACTS ***************************************************************
ok: [idc1-server2]

TASK: [myfacts | run myfacts module to get customized facts] **************
ok: [idc1-server2]

TASK: [myfacts | update file with the customized facts] *********************
changed: [idc1-server2]

PLAY RECAP ********************************************************************
idc1-server2                   : ok=3    changed=1    unreachable=0    failed=0  

[root@idc-server1 ansible]# ssh idc1-server2 'cat /tmp/myfacts.txt'

ansible_private_ipv4_address : 172.16.1.2

,

No Comments

Setup GlusterFS with Distributed Replicated Volumes and Native client

OS: CentOS 6.4 x86_64 Minimal

1. Install packages, on idc1-server{1-4}:
# wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
# yum install -y glusterfs glusterfs-server glusterfs-fuse

# /etc/init.d/glusterd start
# chkconfig glusterd on

2. Configure peers, just on idc1-server1:
[root@idc1-server1 ~]# gluster peer probe idc1-server2

peer probe: success

[root@idc1-server1 ~]# gluster peer probe idc1-server3

peer probe: success

[root@idc1-server1 ~]# gluster peer probe idc1-server4

peer probe: success

NOTE:
The following step is very important with the version 3.4.3!
If configured all peers with hostname on one host, must remember to recreate the peer of this host on another host.
Otherwise all other hosts cannot recognize this host by hostname.
10.1.1.35 is the ip address of idc1-server1, detach it then probe with hostname.

Just on idc1-server2:
[root@idc1-server2 ~]# gluster peer detach 10.1.1.35

peer detach: success

[root@idc1-server2 ~]# gluster peer probe idc1-server1

peer probe: success

Just on idc1-server2:
[root@idc1-server2 ~]# gluster peer status

Number of Peers: 3
  
Hostname: idc1-server3
Uuid: 01f25251-9ee6-40c7-a322-af53a034aa5a
State: Peer in Cluster (Connected)
  
Hostname: idc1-server4
Uuid: 212295a6-1f38-4a1e-968c-577241318ff1
State: Peer in Cluster (Connected)
  
Hostname: idc1-server1
Port: 24007
Uuid: ed016c4e-7159-433f-88a5-5c3ebd8e36c9
State: Peer in Cluster (Connected)

3. Create directories, on idc1-server{1-4}:
# mkdir -p /usr/local/share/{datavolume1,datavolume2,datavolume3}
# chown -R root:root /mnt/{datavolume1,datavolume2,datavolume3}
# ls -l /usr/local/share/

total 24
drwxr-xr-x.  2 root root 4096 Sep 23  2011 applications
drwxr-xr-x   2 root root 4096 Apr  1 12:19 datavolume2
drwxr-xr-x.  2 root root 4096 Sep 23  2011 info
drwxr-xr-x. 21 root root 4096 Feb 20  2013 man
drwxr-xr-x   2 root root 4096 Apr  1 12:19 datavolume1
drwxr-xr-x   2 root root 4096 Apr  1 12:19 datavolume3

4. Create Distributed Replicated Volumes, just on idc1-server1:
[root@idc1-server1 ~]# gluster volume create datavolume1 replica 2 transport tcp idc1-server1:/usr/local/share/datavolume1 idc1-server2:/usr/local/share/datavolume1 idc1-server3:/usr/local/share/datavolume1 idc1-server4:/usr/local/share/datavolume1 force

volume create: datavolume1: success: please start the volume to access data

[root@idc1-server1 ~]# gluster volume create datavolume2 replica 2 transport tcp idc1-server1:/usr/local/share/datavolume2 idc1-server2:/usr/local/share/datavolume2 idc1-server3:/usr/local/share/datavolume2 idc1-server4:/usr/local/share/datavolume2 force

volume create: datavolume2: success: please start the volume to access data

[root@idc1-server1 ~]# gluster volume create datavolume3 replica 2 transport tcp idc1-server1:/usr/local/share/datavolume3 idc1-server2:/usr/local/share/datavolume3 idc1-server3:/usr/local/share/datavolume3 idc1-server4:/usr/local/share/datavolume3 force

volume create: datavolume3: success: please start the volume to access data

[root@idc1-server1 ~]# gluster volume start datavolume1

volume start: datavolume1: success

[root@idc1-server1 ~]# gluster volume start datavolume2

volume start: datavolume2: success

[root@idc1-server1 ~]# gluster volume start datavolume3

volume start: datavolume3: success

[root@idc1-server1 ~]# gluster volume info

Volume Name: datavolume1
Type: Distributed-Replicate
Volume ID: aea76c2a-b754-4037-9634-c2062e2955c3
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: idc1-server1:/usr/local/share/datavolume1
Brick2: idc1-server2:/usr/local/share/datavolume1
Brick3: idc1-server3:/usr/local/share/datavolume1
Brick4: idc1-server4:/usr/local/share/datavolume1
 
Volume Name: datavolume2
Type: Distributed-Replicate
Volume ID: 1ed65c6e-ee23-475a-82c7-2967e2fc2569
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: idc1-server1:/usr/local/share/datavolume2
Brick2: idc1-server2:/usr/local/share/datavolume2
Brick3: idc1-server3:/usr/local/share/datavolume2
Brick4: idc1-server4:/usr/local/share/datavolume2
 
Volume Name: datavolume3
Type: Distributed-Replicate
Volume ID: b63bb4ea-bd37-4dd6-9a4c-230e6d236afa
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: idc1-server1:/usr/local/share/datavolume3
Brick2: idc1-server2:/usr/local/share/datavolume3
Brick3: idc1-server3:/usr/local/share/datavolume3
Brick4: idc1-server4:/usr/local/share/datavolume3

5. Configure the Network ACL, just on idc1-server1:
[root@idc1-server1 ~]# gluster volume set datavolume1 auth.allow 10.1.1.*,10.1.2.*

6. Tests on idc1-client15, install the version '-3.4.0.57rhs-1.el6_5' which supports option: 'backup-volfile-servers'
[root@idc1-client15 ~]# yum install -y glusterfs-3.4.0.57rhs-1.el6_5 glusterfs-libs-3.4.0.57rhs-1.el6_5 glusterfs-fuse-3.4.0.57rhs-1.el6_5
[root@idc1-client15 ~]# mkdir -p /mnt/{datavolume1,datavolume2,datavolume3}
[root@idc1-client15 ~]# chown -R root:root /mnt/{datavolume1,datavolume2,datavolume3}
[root@idc1-client15 ~]# mount -t glusterfs -o backup-volfile-servers=idc1-server2:idc1-server3:idc1-server4,ro idc1-server1:datavolume1 /mnt/datavolume1/
[root@idc1-client15 ~]# mount -t glusterfs -o backup-volfile-servers=idc1-server2:idc1-server3:idc1-server4,ro idc1-server1:datavolume2 /mnt/datavolume2/
[root@idc1-client15 ~]# mount -t glusterfs -o backup-volfile-servers=idc1-server2:idc1-server3:idc1-server4,ro idc1-server1:datavolume3 /mnt/datavolume3/

[root@idc1-client15 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_t-lv_root
                       59G  7.7G   48G  14% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/xvda1            485M   33M  428M   8% /boot
idc1-server3:datavolume1     98G  8.6G   85G  10% /mnt/datavolume1
idc1-server3:datavolume2     98G  8.6G   85G  10% /mnt/datavolume2
idc1-server3:datavolume3     98G  8.6G   85G  10% /mnt/datavolume3

Tests:
Write on idc1-client15 mount point /mnt/datavolume1:
[root@idc1-client15 ~]# umount /mnt/datavolume1
[root@idc1-client15 ~]# mount -t glusterfs idc1-server3:datavolume1 /mnt/datavolume1/
[root@idc1-client15 ~]# echo "This is idc1-client15" > /mnt/datavolume1/hello.txt
[root@idc1-client15 ~]# mkdir /mnt/testdir
[root@idc1-server1 ~]# ls /usr/local/share/datavolume1/

hello.txt testdir

Saw hello.txt and testdir

Write on idc1-server1 volumes dir /usr/local/share/datavolume1:
[root@idc1-server1 ~]# echo "This is idc1-server1" > /usr/local/share/datavolume1/hello.2.txt
[root@idc1-server1 ~]# mkdir /usr/local/share/datavolume1/test2
[idc1-client15][root@idc1-client15 ~]# ls /mnt/datavolume1
[idc1-client15][root@idc1-client15 datavolume1]# ls -l /mnt/datavolume1

hello.txt testdir

Didn't see hello.2.txt and test2

Write on idc1-server1 mount point /mnt/datavolume1:
[root@idc1-server1 ~]# mount -t glusterfs idc1-server1:datavolume1 /mnt/datavolume1/
[root@idc1-server1 ~]# echo "This is idc1-server1" > /mnt/datavolume1/hello.3.txt
[root@idc1-server1 ~]# mkdir /mnt/datavolume1/test3
[idc1-client15][root@idc1-client15 datavolume1]# ls /mnt/datavolume1

hello.2.txt  hello.3.txt hello.txt  test2  test3  testdir

Saw hello.3.txt and test3, and the hello.2.txt and test2 also.

So I guess if we just write or delete file outside the mount point, it doesn't notify other nodes, so the changes didn't take effect.
We should mount it on all servers which we want to write or delete files.

Other Notes:
Delete:
# gluster volume stop datavolume1
# gluster volume delete datavolume1

Move:
# gluster peer detach idc1-server4

ACL:
# gluster volume set datavolume1 auth.allow 10.1.1.*,10.1.2.*

Add:
# gluster peer probe idc1-server5
# gluster peer probe idc1-server6
# gluster volume add-brick datavolume1 idc1-server5:/usr/local/share/datavolume1 idc1-server6:/usr/local/share/datavolume1

Shrink and migration:
# gluster volume remove-brick datavolume1 idc1-server1:/usr/local/share/datavolume1 idc1-server5:/usr/local/share/datavolume1 start
# gluster volume remove-brick datavolume1 idc1-server1:/usr/local/share/datavolume1 idc1-server5:/usr/local/share/datavolume1 status
# gluster volume remove-brick datavolume1 idc1-server1:/usr/local/share/datavolume1 idc1-server5:/usr/local/share/datavolume1 commit

Rebalance:
# gluster volume rebalance datavolume1 start
# gluster volume rebalance datavolume1 status
# gluster volume rebalance datavolume1 stop

If idc1-server1 dead:
# gluster volume replace-brick datavolume1 idc1-server1:/usr/local/share/datavolume1 idc1-server5:/usr/local/share/datavolume1 commit -force
# gluster volume heal datavolume1 full

No Comments

Install Zimbra MTA 8 on CentOS 6.5

Just a copy from http://scrble.net/install-zimbra-mta-8-on-centos-6-5/

Make sure no mail server and http server is running. Otherwise stop them and disable or uninstall them.

Make sure SELINUX is disabled. By default it is disabled on CentOS 6.5. To confirm it, check using getforce

# getenforce

Disabled

# cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing - SELinux security policy is enforced.
#       permissive - SELinux prints warnings instead of enforcing.
#       disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
#       targeted - Only targeted network daemons are protected.
#       strict - Full SELinux protection.
SELINUXTYPE=targeted

Stop and Disable Postfix and Iptables
# service postfix stop
# service ip6tables stop
# service iptables stop

# chkconfig postfix off
# chkconfig ip6tables off
# chkconfig iptables off

Make sure that domain name is set in correct format at /etc/hosts file. The format is:

ip_address     server.domain.com      server

Also set the hostname at /etc/sysconfig/network and also:
# service network restart
# hostname server.domain.com

Make sure you have proper MX record configured for your domain
# host -t mx server.domain.com

server.domain.com mail is handled by 10 server.domain.com.

Install the required packages:
# yum install perl nc sysstat -y

Download the latest Zimbra MTA and extract it:
# cd /opt/
# wget http://files2.zimbra.com/downloads/8.0.6_GA/zcs-8.0.6_GA_5922.RHEL6_64.20131203103705.tgz
# tar -zxvf zcs-8.0.6_GA_5922.RHEL6_64.20131203103705.tgz
# cd zcs-8.0.6_GA_5922.RHEL6_64.20131203103705

Start the installation:
# ./install.sh --platform-override

Just accept all default:
Do you agree with the terms of the software license agreement? [N] Y
Do you agree with the terms of the software license agreement? [N] Y
Select the packages to install
Install zimbra-ldap [Y]
Install zimbra-logger [Y]
Install zimbra-mta [Y]
Install zimbra-snmp [Y]
Install zimbra-store [Y]
Install zimbra-apache [Y]
Install zimbra-spell [Y]
Install zimbra-memcached [N]
Install zimbra-proxy [N]

Give “Y” when it asked the following:

The system will be modified.  Continue? [N] Y

Set the Admin password:

Password for admin@server.domain.com (min 6 characters): [_yTalZUL9OZ] password

Finally press ‘a’ to complete the installation.

*** CONFIGURATION COMPLETE - press 'a' to apply
 Select from menu, or press 'a' to apply config (? - help) a
 Save configuration data to a file? [Yes]
 Save config in file: [/opt/zimbra/config.26430]
 Saving config in /opt/zimbra/config.26430...done.
 The system will be modified - continue? [No] YES
Configuration complete - press return to exit

Switch user as zimbra
# su - zimbra

Start Zimbra services:
$ zmcontrol start

Host server.domain.com
        Starting zmconfigd...Done.
        Starting logger...Done.
        Starting mailbox...Done.
        Starting antispam...Done.
        Starting antivirus...Done.
        Starting opendkim...Done.
        Starting snmp...Done.
        Starting spell...Done.
        Starting mta...Done.
        Starting stats...Done.

Check Zimbra services status:
$ zmcontrol status

Host server.domain.com
        antispam           Running
        antivirus          Running
        ldap               Running
        logger             Running
        mailbox            Running
        mta                Running
        opendkim           Running
        snmp               Running
        spell              Running
        stats              Running
        zmconfigd          Running

You can check and manage through Zimbra Admin Control Panel at https://server.domain.com:7071 and client web-mail at https://server.domain.com

zimbra

No Comments

Fix the OpenSSL Heartbleed vulnerability without upgrade

In some environments, upgrade all OpenSSL packages may not be easy, with iptables there is a solution.

#Log
iptables -t filter -A INPUT -p tcp --dport 443 -m u32 --u32 "52=0x18030000:0x1803FFFF" -j LOG --log-prefix "BLOCKED: HEARTBLEED"
#Block
iptables -t filter -A INPUT -p tcp --dport 443 -m u32 --u32 "52=0x18030000:0x1803FFFF" -j DROP

,

No Comments

Fork me on GitHub