关于 九月, 2014 的文章

Shell脚本之并发编程

在Python中,有很多模块都可以实现并发编程,比如 threading, processing, eventlet 与 Stackless Python 等。

那么对于Shell而言,又如何实现呢?其实原理很简单,我采用的方法是:
1. 将需要执行的任务分批放入后台执行;
2. 将后台执行的命令结果汇总到指定的文件中;
3. 使用wait命令来等待所有任务执行结束。

下面的脚本就用到了这样的并发编程方法,实现的功能是:
快速(3-4秒内)对相同C网内的所有IP(255个)通过命令ping进行测试并返回结果。

vim fastping.sh

#!/bin/bash 

# default settings
subnet=$1 # C type subnet
retry=2 # retry times
timeout=3 # timeout seconds
output=/tmp/ping.output # output file

# function print_help
function print_help(){
  echo "Examples:"
  echo ${0} 172.17.32
  echo ${0} 192.168.1 unable
  exit 1
}

# check the parameter
if [ $# -lt 1 ]; then
  print_help
fi

# check the network parameter's format
count=0
for i in $(echo $1 |sed 's/\./ /g')
do
  count=$((${count}+1))
done
if [ ${count} -ne 3 ]; then
  print_help
fi

# clean the output file
> ${output}

function pingable(){
  ping -c ${retry} -w ${timeout} -q ${subnet}.${i} &> /dev/null && echo ${i} >> ${output}
}

function unpingable(){
  ping -c ${retry} -w ${timeout} -q ${subnet}.${i} &> /dev/null || echo ${i} >> ${output}
}

# get the check type
if [ "$2" == "unable" ]; then
  status="unpingable"
else
  status="pingable"
fi

# ping as paraller mode and write output into file
for i in {1..255}
do 
  ${status} &
done

# wait for all ping processes done
wait

# print output with better order
sum=$(wc -l ${output} |awk '{print $1}')
echo "There are '${sum}' '${status}' IPs begin with '${subnet}.' :"
cat ${output} |sort |xargs -n 20 echo " "

chmod +x fastping.sh
./fastping.sh

Examples:
./fastping 172.17.32
./fastping 192.168.1 unable

time ./fastping.sh 192.168.1

There are '142' 'pingable' IPs begin with '192.168.1' :
  1 10 12 13 14 15 16 18 19 20 21 22 23 24 25 26 27 28 29 30
  31 32 33 34 35 36 37 38 40 41 42 43 44 45 46 47 48 49 50 51
  52 53 54 55 56 57 59 60 61 62 63 64 65 66 67 68 69 70 71 72
  73 74 75 76 77 78 80 81 83 84 85 86 87 88 89 90 91 92 93 94
  95 96 97 98 99 100 101 102 103 104 105 106 107 108 112 113 114 115 116 117
  118 119 120 121 122 123 124 125 126 127 128 133 134 135 136 137 138 139 140 141
  142 143 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166
  170 254

real    0m3.201s
user    0m0.135s
sys     0m0.495s

No Comments

使用Pelican + Markdown + GitHub Pages来撰写Blog

参考资料:
http://www.linuxzen.com/shi-yong-pelicanda-zao-jing-tai-bo-ke.html
http://www.dongxf.com/3_Build_Personal_Blog_With_Pelican_And_GitHub_Pages.html
https://github.com/razius/razius.com/blob/master/pelicanconf.py

背景介绍:
在我还自以为使用Wordpress作为Blog程序很高大上时,身边有不少运维哥们儿却已经采用Pelican作为博客生成器,使用Markdown语法编写文章,并托管在GitHub Pages上,逼格立刻就体现出来了。
为了让自己不至于太落后,我经过一阵子摸索,终于也完成了我的GitHub Page: http://mcsrainbow.github.io

具体步骤:
1. 安装所需软件
sudo yum install python python-devel python-pip
sudo pip install pelican markdown

2. 创建工作目录
cd ~
mkdir pelican

cd pelican
pelican-quickstart

Welcome to pelican-quickstart v3.4.0.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.


> Where do you want to create your new web site? [.]
> What will be the title of this web site? HeyDevOps
> Who will be the author of this web site? mcsrainbow
> What will be the default language of this web site? [en] zh
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n)
> What is your URL prefix? (see above example; no trailing slash) http://mcsrainbow.github.io
> Do you want to enable article pagination? (Y/n)
> How many articles per page do you want? [10]
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n)
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n)
> Do you want to upload your website using FTP? (y/N)
> Do you want to upload your website using SSH? (y/N)
> Do you want to upload your website using Dropbox? (y/N)
> Do you want to upload your website using S3? (y/N)
> Do you want to upload your website using Rackspace Cloud Files? (y/N)
> Do you want to upload your website using GitHub Pages? (y/N) Y
> Is this your personal page (username.github.io)? (y/N) Y
Done. Your new project is available at /home/dong/pelican

3. 下载风格包pelican-themes与插件包pelican-plugins
git clone git://github.com/getpelican/pelican-themes.git
git clone git://github.com/getpelican/pelican-plugins.git

4. 配置pelicanconf.py

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

# Site settings
AUTHOR = u'mcsrainbow'
AUTHOR_EMAIL = u'guosuiyu@gmail.com'
SITENAME = u'HeyDevOps'
TAGLINE = 'Whatever is worth doing is worth doing well.'
SITEURL = 'http://mcsrainbow.github.io'
DEFAULT_DATE_FORMAT = ('%Y-%m-%d')

TIMEZONE = 'Asia/Shanghai'

DEFAULT_LANG = u'zh'
DEFAULT_METADATA = (
)

DELETE_OUTPUT_DIRECTORY = False

# Blogroll
LINKS = (
    ('HeyLinux', 'http://heylinux.com'),
)

# Social widget
SOCIAL = (
    ('Github', 'http://github.com/mcsrainbow/heydevops'),
    ('Twitter', 'http://twitter.com/heydevops'),
)

MENUITEMS = (
)

# Disqus
DISQUS_SITENAME = u"mcsrainbow"

# Content path
PATH = 'content'
PAGE_PATHS = ['pages']
ARTICLE_PATHS = ['articles']
STATIC_PATHS = ['images', 'files']
EXTRA_PATH_METADATA = {
    'files/robots.txt': {'path': 'robots.txt'},
    'images/favicon.ico': {'path': 'favicon.ico'},
}

ARTICLE_URL = ('articles/{slug}.html')
ARTICLE_SAVE_AS = ('articles/{slug}.html')
PAGE_LANG_SAVE_AS = False

# Feed
FEED_DOMAIN = SITEURL
FEED_ALL_ATOM = 'feeds/all.atom.xml'
CATEGORY_FEED_ATOM = 'feeds/%s.atom.xml'
TRANSLATION_FEED_ATOM = None

# Theme
THEME = 'pelican-themes/zurb-F5-basic'
DEFAULT_PAGINATION = 10

MD_EXTENSIONS = (['codehilite(css_class=highlight)', 'extra',
                  'fenced_code', 'tables', 'sane_lists'])

# Plugin
PLUGIN_PATHS = ['pelican-plugins']
PLUGINS = [ 'sitemap', 'gravatar' ]

# Sitemap
SITEMAP = {
    'format': 'xml',
    'priorities': {
        'articles': 1,
        'pages': 0.9,
        'indexes': 0.8,
    },
    'changefreqs': {
        'indexes': 'daily',
        'articles': 'daily',
        'pages': 'weekly'
    }
}

# Can be useful in development, but set to False when you're ready to publish
RELATIVE_URLS = False

5. 配置 publishconf.py

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

# This file is only used if you use `make publish` or
# explicitly specify it as your config file.

import os
import sys
sys.path.append(os.curdir)
from pelicanconf import *

SITEURL = 'http://mcsrainbow.github.io'
RELATIVE_URLS = False

FEED_ALL_ATOM = 'feeds/all.atom.xml'
CATEGORY_FEED_ATOM = 'feeds/%s.atom.xml'

DELETE_OUTPUT_DIRECTORY = True

# Following items are often useful when publishing

DISQUS_SITENAME = u"mcsrainbow"
#GOOGLE_ANALYTICS = ""

6. 撰写第一篇文章
cd content
mkdir articles files images pages
vim articles/hello.md

Title: Hello
Date: 2014-09-24 02:06
Category: Uncategorized
Tags: Pelican, Markdown
Slug: hello
Author: mcsrainbow
Summary: Hello Pelican, Markdown and GitHub Pages.

Hello Pelican, Markdown and GitHub Pages.

7. 生成robots.txt与favicon.ico
vim files/robots.txt

User-agent: *
Sitemap: http://mcsrainbow.github.io/sitemap.xml

scp heylinux.com:/webserver/blog/rainbow/favicon.ico .

8. 配置Disqus
Disqus上注册一个用户并生成一个站点mcsrainbow.disqus.com;
设置mcsrainbow.disqus.com站点使其允许域名mcsrainbow.github.io;
设置以上配置文件为DISQUS_SITENAME = u"mcsrainbow",mcsrainbow 为站点ID

9. 创建GitHub Pages
直接创建一个新的repo,但是其名称必须与ID相同,并加上github.io或github.com后缀。
就我而言,就必须创建一个repo名为mcsrainbow.github.io

10. 创建好GitHub Pages之后,生成Blog静态HTML文件
cd ~
cd pelican
make html

pelican /home/dong/pelican/content -o /home/dong/pelican/output -s /home/dong/pelican/pelicanconf.py
WARNING: AUTHOR_SAVE_AS is set to False
Done: Processed 1 article(s), 0 draft(s) and 0 page(s) in 0.23 seconds.

11. 进入output目录,将生成好的静态HTML文件上传到GitHub Pages站点mcsrainbow.github.io中
cd output
git init
git remote add origin https://github.com/mcsrainbow/mcsrainbow.github.io.git
git add -A
git commit -m "Update Blog"
git push -u origin master

12. 等待15分钟左右,访问mcsrainbow.github.io即可看到生成的网站效果

13. 注意
由于我设置了在重新生成HTML时默认不删除output目录,因此每次更新Blog时都需要手动执行'rm -rf output/*'。
这样做的目的,是为了避免删除output/.git目录,方面在生成之后立刻提交到GitHub Pages。
当然在实际操作当中我是编写了alias和scripts来完成这一系列动作的。
具体可以参考我的Pelican工作环境:https://github.com/mcsrainbow/pelican

, , ,

1 Comment

使用Bind9搭建DNS主从服务器

背景介绍:
服务器数量较多时,有效的规划主机名并搭建一个内网DNS来进行解析,会大幅度提高工作效率。
在我们的线上环境中,就是通过Bind9搭建的一套DNS主从服务器来完成的,并且自动化伸缩的脚本(新建/删除EC2虚拟机)也与DNS相结合,自动调用一个简单的Shell脚本来实现DNS记录的新增和删除。

具体步骤:
环境介绍:
OS: CentOS 6.4 x86_64 Minimal
Domain: heylinux.com
Master: 172.16.8.246
Slave: 172.16.8.247

1. 安装配置DNS主服务器
1.1 安装Bind9所需软件包
yum install bind bind-devel bind-libs bind-utils bind-chroot

1.2 创建/var/named/heylinux.com.key用于主从服务器之间的校验
cd /var/named/
dnssec-keygen -a HMAC-MD5 -b 128 -n USER heylinux.com.

Kheylinux.com.+157+59510

ls Kheylinux.com*

Kheylinux.com.+157+59510.key  Kheylinux.com.+157+59510.private

grep -w Key Kheylinux.com.+157+59510.private

Key: 4Cn0wwI+STjsW+3S5dEEdQ==

vim heylinux.com.key

key "heylinux.com." {
        algorithm hmac-md5;
        secret "4Cn0wwI+STjsW+3S5dEEdQ==";
};

1.3 修改文件属性
chown named:named *heylinux.com*
chown named:named /var/named

1.4 启动named服务,生成/etc/rndc.key文件
service named start

1.5 配置named服务开机自启动
chkconfig named on
阅读全文 »

No Comments