标签为 CDH4 的文章

Hadoop集群(CHD4)实践之 (4) Oozie搭建

目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

本文内容
Hadoop集群(CHD4)实践之 (4) Oozie搭建

参考资料
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

环境准备
OS: CentOS 6.4 x86_64
Servers:
hadoop-master: 172.17.20.230 内存10G
- namenode
- hbase-master

hadoop-secondary: 172.17.20.234 内存10G
- secondarybackupnamenode,jobtracker
- hive-server,hive-metastore
- oozie

hadoop-node-1: 172.17.20.231 内存10G sudo yum install hbase-regionserver
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-2: 172.17.20.232 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-3: 172.17.20.233 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

对以上角色做一些简单的介绍:
namenode - 整个HDFS的命名空间管理服务
secondarynamenode - 可以看做是namenode的冗余服务
jobtracker - 并行计算的job管理服务
datanode - HDFS的节点服务
tasktracker - 并行计算的job执行服务
hbase-master - Hbase的管理服务
hbase-regionServer - 对Client端插入,删除,查询数据等提供服务
zookeeper-server - Zookeeper协作与配置管理服务
hive-server - Hive的管理服务
hive-metastore - Hive的元存储,用于对元数据进行类型检查与语法分析
oozie - Oozie是一种Java Web应用程序,用于工作流的定义和管理

本文定义的规范,避免在配置多台服务器上产生理解上的混乱:
以下操作都只需要在 Oozie 所在主机,即 hadoop-secondary 上执行。

1. 安装前的准备
Hadoop集群(CDH4)实践之 (3) Hive搭建

2. 安装Oozie
$ sudo yum install oozie oozie-client

3. 创建Oozie数据库
$ mysql -uroot -phiveserver

 
mysql> create database oozie;
mysql> grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';
mysql> grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';
mysql> exit;

4.配置oozie-site.xml
$ sudo vim /etc/oozie/conf/oozie-site.xml

 
<?xml version="1.0"?>
<configuration>
    <property>
        <name>oozie.service.ActionService.executor.ext.classes</name>
        <value>
            org.apache.oozie.action.email.EmailActionExecutor,
            org.apache.oozie.action.hadoop.HiveActionExecutor,
            org.apache.oozie.action.hadoop.ShellActionExecutor,
            org.apache.oozie.action.hadoop.SqoopActionExecutor,
            org.apache.oozie.action.hadoop.DistcpActionExecutor
        </value>
    </property>
    <property>
        <name>oozie.service.SchemaService.wf.ext.schemas</name>
        <value>shell-action-0.1.xsd,shell-action-0.2.xsd,email-action-0.1.xsd,hive-action-0.2.xsd,hive-action-0.3.xsd,hive-action-0.4.xsd,hive-action-0.5.xsd,sqoop-action-0.2.xsd,sqoop-action-0.3.xsd,ssh-action-0.1.xsd,ssh-action-0.2.xsd,distcp-action-0.1.xsd</value>
    </property>
    <property>
        <name>oozie.system.id</name>
        <value>oozie-${user.name}</value>
    </property>
    <property>
        <name>oozie.systemmode</name>
        <value>NORMAL</value>
    </property>
    <property>
        <name>oozie.service.AuthorizationService.security.enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>oozie.service.PurgeService.older.than</name>
        <value>30</value>
    </property>
    <property>
        <name>oozie.service.PurgeService.purge.interval</name>
        <value>3600</value>
    </property>
    <property>
        <name>oozie.service.CallableQueueService.queue.size</name>
        <value>10000</value>
    </property>
    <property>
        <name>oozie.service.CallableQueueService.threads</name>
        <value>10</value>
    </property>
    <property>
        <name>oozie.service.CallableQueueService.callable.concurrency</name>
        <value>3</value>
    </property>
    <property>
        <name>oozie.service.coord.normal.default.timeout
	</name>
	<value>120</value>
    </property>

    <property>
        <name>oozie.db.schema.name</name>
        <value>oozie</value>
    </property>
    <property>
        <name>oozie.service.JPAService.create.db.schema</name>
        <value>true</value>
    </property>

    <property>
        <name>oozie.service.JPAService.jdbc.driver</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>oozie.service.JPAService.jdbc.url</name>
        <value>jdbc:mysql://localhost:3306/oozie</value>
    </property>
    <property>
        <name>oozie.service.JPAService.jdbc.username</name>
        <value>oozie</value>
    </property>
    <property>
        <name>oozie.service.JPAService.jdbc.password</name>
        <value>oozie</value>
    </property>

    <property>
        <name>oozie.service.JPAService.pool.max.active.conn</name>
        <value>10</value>
    </property>

    <property>
        <name>oozie.service.HadoopAccessorService.kerberos.enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>local.realm</name>
        <value>LOCALHOST</value>
    </property>
    <property>
        <name>oozie.service.HadoopAccessorService.keytab.file</name>
        <value>${user.home}/oozie.keytab</value>
    </property>
    <property>
        <name>oozie.service.HadoopAccessorService.kerberos.principal</name>
        <value>${user.name}/localhost@${local.realm}</value>
    </property>
    <property>
        <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
        <value> </value>
    </property>
    <property>
        <name>oozie.service.HadoopAccessorService.nameNode.whitelist</name>
        <value> </value>
    </property>

    <property>
        <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
        <value>*=/etc/hadoop/conf</value>
    </property>
    <property>
        <name>oozie.service.WorkflowAppService.system.libpath</name>
        <value>/user/${user.name}/share/lib</value>
    </property>

    <property>
        <name>use.system.libpath.for.mapreduce.and.pig.jobs</name>
        <value>false</value>
    </property>

    <property>
        <name>oozie.authentication.type</name>
        <value>simple</value>
    </property>
    <property>
        <name>oozie.authentication.token.validity</name>
        <value>36000</value>
    </property>
    <property>
        <name>oozie.authentication.signature.secret</name>
        <value>oozie</value>
    </property>

    <property>
      <name>oozie.authentication.cookie.domain</name>
      <value></value>
    </property>

    <property>
        <name>oozie.authentication.simple.anonymous.allowed</name>
        <value>true</value>
    </property>

    <property>
        <name>oozie.authentication.kerberos.principal</name>
        <value>HTTP/localhost@${local.realm}</value>
    </property>

    <property>
        <name>oozie.authentication.kerberos.keytab</name>
        <value>${oozie.service.HadoopAccessorService.keytab.file}</value>
    </property>

    <property>
        <name>oozie.authentication.kerberos.name.rules</name>
        <value>DEFAULT</value>
    </property>

    <property>
        <name>oozie.service.ProxyUserService.proxyuser.oozie.hosts</name>
        <value>*</value>
    </property>

    <property>
        <name>oozie.service.ProxyUserService.proxyuser.oozie.groups</name>
        <value>*</value>
    </property>

    <property>
        <name>oozie.service.ProxyUserService.proxyuser.hue.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>oozie.service.ProxyUserService.proxyuser.hue.groups</name>
        <value>*</value>
    </property>

    <property>
        <name>oozie.action.mapreduce.uber.jar.enable</name>
        <value>true</value>
    </property>
    <property>
        <name>oozie.service.HadoopAccessorService.supported.filesystems</name>
        <value>hdfs,viewfs</value>
    </property>
</configuration>

5. 配置Oozie Web Console
$ cd /tmp/
$ wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip
$ cd /var/lib/oozie/
$ sudo unzip /tmp/ext-2.2.zip
$ cd ext-2.2/
$ sudo -u hdfs hadoop fs -mkdir /user/oozie
$ sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie

6. 配置Oozie ShareLib
$ mkdir /tmp/ooziesharelib
$ cd /tmp/ooziesharelib
$ tar xzf /usr/lib/oozie/oozie-sharelib.tar.gz
$ sudo -u oozie hadoop fs -put share /user/oozie/share
$ sudo -u oozie hadoop fs -ls /user/oozie/share
$ sudo -u oozie hadoop fs -ls /user/oozie/share/lib
$ sudo -u oozie hadoop fs -put /usr/lib/hive/lib/hbase.jar /user/oozie/share/lib/hive/
$ sudo -u oozie hadoop fs -put /usr/lib/hive/lib/zookeeper.jar /user/oozie/share/lib/hive/
$ sudo -u oozie hadoop fs -put /usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.5.0.jar /user/oozie/share/lib/hive/
$ sudo -u oozie hadoop fs -put /usr/lib/hive/lib/guava-11.0.2.jar /user/oozie/share/lib/hive/
$ sudo ln -s /usr/share/java/mysql-connector-java.jar /var/lib/oozie/mysql-connector-java.jar

7. 启动Oozie
$ sudo service oozie start

8. 访问Oozie Web Console
http://hadoop-secondary:11000/oozie

9. 至此,Oozie的搭建就已经完成。

, ,

5 Comments

Hadoop集群(CDH4)实践之 (3) Hive搭建

目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

本文内容
Hadoop集群(CDH4)实践之 (3) Hive搭建

参考资料
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

环境准备
OS: CentOS 6.4 x86_64
Servers:
hadoop-master: 172.17.20.230 内存10G
- namenode
- hbase-master

hadoop-secondary: 172.17.20.234 内存10G
- secondarybackupnamenode,jobtracker
- hive-server,hive-metastore

hadoop-node-1: 172.17.20.231 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-2: 172.17.20.232 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-3: 172.17.20.233 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

对以上角色做一些简单的介绍:
namenode - 整个HDFS的命名空间管理服务
secondarynamenode - 可以看做是namenode的冗余服务
jobtracker - 并行计算的job管理服务
datanode - HDFS的节点服务
tasktracker - 并行计算的job执行服务
hbase-master - Hbase的管理服务
hbase-regionServer - 对Client端插入,删除,查询数据等提供服务
zookeeper-server - Zookeeper协作与配置管理服务
hive-server - Hive的管理服务
hive-metastore - Hive的元存储,用于对元数据进行类型检查与语法分析

本文定义的规范,避免在配置多台服务器上产生理解上的混乱:
以下操作都只需要在 Hive 所在主机,即 hadoop-secondary 上执行。

1. 安装前的准备
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建

2. 安装Hive
$ sudo yum install hive hive-metastore hive-server
$ sudo yum install hive-jdbc hive-hbase

3. 安装MySQL JDBC Connector
$ sudo yum install mysql-connector-java
$ sudo ln -s /usr/share/java/mysql-connector-java.jar /usr/lib/hive/lib/mysql-connector-java.jar

4. 安装MySQL
$ sudo yum install mysql-server
$ sudo /etc/init.d/mysqld start

$ sudo /usr/bin/mysql_secure_installation

 
[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] y
New password: hiveserver
Re-enter new password: hiverserver
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!

5. 创建数据库并授权
$ mysql -u root -phiveserver

 
mysql> CREATE DATABASE metastore;
mysql> USE metastore;
mysql> SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.10.0.mysql.sql;

mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hiveserver';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'%';
mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'%';

mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hiveserver';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'localhost';
mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'localhost';

mysql> CREATE USER 'hive'@'127.0.0.1' IDENTIFIED BY 'hiveserver';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON metastore.* TO 'hive'@'127.0.0.1';
mysql> REVOKE ALTER,CREATE ON metastore.* FROM 'hive'@'127.0.0.1';

6. 配置hive-site.xml
$ sudo vim /etc/hive/conf/hive-site.xml

 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://hadoop-secondary/metastore</value>
  <description>the URL of the MySQL database</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>hiveserver</value>
</property>
<property>
  <name>datanucleus.autoCreateSchema</name>
  <value>false</value>
</property>
<property>
  <name>datanucleus.fixedDatastore</name>
  <value>true</value>
</property>
<property>
  <name>datanucleus.autoStartMechanism</name> 
  <value>SchemaTable</value>
</property> 
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://hadoop-secondary:9083</value>
  <description>IP address (or fully-qualified domain name) and port of the metastore host</description>
</property>
<property>
  <name>hive.aux.jars.path</name>
  <value>file:////usr/lib/hive/lib/hbase.jar,file:///usr/lib/hive/lib/zookeeper.jar,file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.5.0.jar,file:///usr/lib/hive/lib/guava-11.0.2.jar</value>
</property>
<property>
  <name>hbase.zookeeper.quorum</name>
  <value>hadoop-node-1,hadoop-node-2,hadoop-node-3</value>
</property>
</configuration>

7. 启动Hive
$ /etc/init.d/hive-metastore start
$ /etc/init.d/hive-server start

8. 创建Hive所需的HDFS目录
$ sudo -u hdfs hadoop fs -mkdir /user/hive
$ sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse
$ sudo -u hdfs hadoop fs -ls -R /user
$ sudo -u hdfs hadoop fs -chown -R hive /user/hive
$ sudo -u hdfs hadoop fs -chmod -R 1777 /user/hive/warehouse

$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hadoop-mapred
$ sudo -u hdfs hadoop fs -chmod -R 777 /tmp/hive-hive
$ sudo chown -R hive:hive /var/lib/hive/.hivehistory

9. 至此,Hive的搭建就已经完成。

, ,

No Comments

Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建

目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

本文内容
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建

参考资料
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

环境准备
OS: CentOS 6.4 x86_64
Servers:
hadoop-master: 172.17.20.230 内存10G
- namenode
- hbase-master

hadoop-secondarynamenode: 172.17.20.234 内存10G
- secondarybackupnamenode,jobtracker

hadoop-node-1: 172.17.20.231 内存10G sudo yum install hbase-regionserver
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-2: 172.17.20.232 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

hadoop-node-3: 172.17.20.233 内存10G
- datanode,tasktracker
- hbase-regionserver,zookeeper-server

对以上角色做一些简单的介绍:
namenode - 整个HDFS的命名空间管理服务
secondarynamenode - 可以看做是namenode的冗余服务
jobtracker - 并行计算的job管理服务
datanode - HDFS的节点服务
tasktracker - 并行计算的job执行服务
hbase-master - Hbase的管理服务
hbase-regionServer - 对Client端插入,删除,查询数据等提供服务
zookeeper-server - Zookeeper协作与配置管理服务

本文定义的规范,避免在配置多台服务器上产生理解上的混乱:
所有直接以 $ 开头,没有跟随主机名的命令,都代表需要在所有的服务器上执行,除非后面有单独的//开头或在标题说明。

1. 安装前的准备
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建

配置NTP时钟同步
$ sudo yum install ntp
$ sudo /etc/init.d/ntpd start

配置ulimit与nproc参数
$ sudo vim /etc/security/limits.conf

 
hdfs  -       nofile  32768
hbase -       nofile  32768

退出并重新登录SSH使设置生效

2. 在hadoop-master上安装hbase-master
$ sudo yum install hbase-master
$ sudo yum install hbase-rest
$ sudo yum install hbase-thrift

3. 在hadoop-node上安装hbase-regionserver
$ sudo yum install hbase-regionserver

4. 在HDFS中创建HBase的目录
以下HDFS操作仅需在任意一台主机上执行一次
$ sudo -u hdfs hadoop fs -mkdir /hbase
$ sudo -u hdfs hadoop fs -chown hbase /hbase

5. 配置hbase-site.xml
$ sudo vim /etc/hbase/conf/hbase-site.xml
$ cat /etc/hbase/conf/hbase-site.xml

 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
  <name>hbase.rest.port</name>
  <value>60050</value>
</property>
<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
</property>
<property>
  <name>hbase.rootdir</name>
  <value>hdfs://hadoop-master:8020/hbase</value>
</property>
<property>
  <name>hbase.zookeeper.quorum</name>
  <value>hadoop-node-1,hadoop-node-2,hadoop-node-3</value>
</property>
</configuration>

6. 配置regionservers
$ sudo vim /etc/hbase/conf/regionservers

 
hadoop-node-1
hadoop-node-2
hadoop-node-3

7. 安装Zookeeper
$ sudo yum install zookeeper
$ sudo vim /etc/zookeeper/conf/zoo.cfg
$ cat /etc/zookeeper/conf/zoo.cfg

 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=0
server.1=hadoop-node-1:2888:3888
server.2=hadoop-node-2:2888:3888
server.3=hadoop-node-3:2888:3888

8. 在hadoop-node上安装zookeeper-server并创建myid文件
$ sudo yum install zookeeper-server
$ sudo touch /var/lib/zookeeper/myid
$ sudo chown -R zookeeper:zookeeper /var/lib/zookeeper
$ echo 1 > /var/lib/zookeeper/myid //仅在hadoop-node-1上执行
$ echo 2 > /var/lib/zookeeper/myid //仅在hadoop-node-2上执行
$ echo 3 > /var/lib/zookeeper/myid //仅在hadoop-node-3上执行

$ sudo /etc/init.d/zookeeper-server init //仅在任一hadoop-node上执行一次
$ sudo /etc/init.d/zookeeper-server start

9. 启动Hbase服务
仅在hadoop-master上
$ sudo /etc/init.d/hbase-master start
$ sudo /etc/init.d/hbase-thrift start
$ sudo /etc/init.d/hbase-rest start

仅在hadoop-node上
$ sudo /etc/init.d/hbase-regionserver start

10. 查看服务的状态
通过网页查看 http://hadoop-master:60010

11. 至此,HBase&Zookeeper的搭建就已经完成。

, , ,

No Comments

Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建

目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

本文内容
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建

参考资料
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/CDH4-Installation-Guide.html

环境准备
OS: CentOS 6.4 x86_64
Servers:
hadoop-master: 172.17.20.230 内存10G
- namenode

hadoop- secondarynamenode: 172.17.20.234 内存10G
- secondarybackupnamenode,jobtracker

hadoop-node-1: 172.17.20.231 内存10G
- datanode,tasktracker

hadoop-node-2: 172.17.20.232 内存10G
- datanode,tasktracker

hadoop-node-3: 172.17.20.233 内存10G
- datanode,tasktracker

对以上角色做一些简单的介绍:
namenode - 整个HDFS的命名空间管理服务
secondarynamenode - 可以看做是namenode的冗余服务
jobtracker - 并行计算的job管理服务
datanode - HDFS的节点服务
tasktracker - 并行计算的job执行服务

本文定义的规范,避免在配置多台服务器上产生理解上的混乱:
所有直接以 $ 开头,没有跟随主机名的命令,都代表需要在所有的服务器上执行,除非后面有单独的//开头或在标题说明。

1. 选择最好的安装包
为了更方便和更规范的部署Hadoop集群,我们采用Cloudera的集成包。
因为Cloudera对Hadoop相关的系统做了很多优化,避免了很多因各个系统间版本不符产生的很多Bug。
这也是很多资深Hadoop管理员所推荐的。
https://ccp.cloudera.com/display/DOC/Documentation/

2. 安装Java环境
由于整个Hadoop项目主要是通过Java开发完成的,因此需要JVM的支持。
登陆www.oracle.com(需要创建一个ID),从以下地址下载一个64位的JDK,如jdk-7u45-linux-x64.rpm
http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

$ sudo rpm -ivh jdk-7u45-linux-x64.rpm
$ sudo vim /etc/profile.d/java.sh

 
export JAVA_HOME=/usr/java/jdk1.7.0_45
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

$ sudo chmod +x /etc/profile.d/java.sh
$ source /etc/profile

3. 配置Hadoop安装源
$ sudo rpm --import http://archive.cloudera.com/cdh4/redhat/5/x86_64/cdh/RPM-GPG-KEY-cloudera
$ cd /etc/yum.repos.d/
$ sudo wget http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/cloudera-cdh4.repo

4. 安装Hadoop相关套件,选择MRv1的框架支持
$ sudo yum install hadoop-hdfs-namenode //仅在hadoop-master上安装

$ sudo yum install hadoop-hdfs-secondarynamenode //仅在hadoop-secondary上安装
$ sudo yum install hadoop-0.20-mapreduce-jobtracker //仅在hadoop-secondary上安装

$ sudo yum install hadoop-hdfs-datanode //仅在hadoop-node上安装
$ sudo yum install hadoop-0.20-mapreduce-tasktracker //仅在hadoop-node上安装

$ sudo yum install hadoop-client

5. 创建Hadoop配置文件
$ sudo cp -r /etc/hadoop/conf.dist /etc/hadoop/conf.my_cluster

6. 激活新的配置文件
$ sudo alternatives --verbose --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.my_cluster 50
$ sudo alternatives --set hadoop-conf /etc/hadoop/conf.my_cluster
$ cd /etc/hadoop/conf

7. 添加hosts记录并修改对应的主机名
$ sudo vim /etc/hosts

 
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.17.20.230 hadoop-master
172.17.20.234 hadoop-secondary
172.17.20.231 hadoop-node-1
172.17.20.232 hadoop-node-2
172.17.20.233 hadoop-node-3

8. 安装LZO支持
$ cd /etc/yum.repos.d
$ sudo wget http://archive.cloudera.com/gplextras/redhat/6/x86_64/gplextras/cloudera-gplextras4.repo
$ sudo yum install hadoop-lzo-cdh4

9. 配置hadoop/conf下的文件
$ sudo vim /etc/hadoop/conf/masters

 
hadoop-master

$ sudo vim /etc/hadoop/conf/slaves

 
hadoop-node-1
hadoop-node-2
hadoop-node-3

10. 创建hadoop的HDFS目录
$ sudo mkdir -p /data/{1,2,3,4}/mapred/local
$ sudo chown -R mapred:hadoop /data/{1,2,3,4}/mapred/local

$ sudo mkdir -p /data/1/dfs/nn /nfsmount/dfs/nn /data/1/dfs/ns /data/{1,2,3,4}/dfs/dn
$ sudo chown -R hdfs:hdfs /data/1/dfs/nn /nfsmount/dfs/nn /data/1/dfs/ns /data/{1,2,3,4}/dfs/dn
$ sudo chmod 700 /data/1/dfs/nn /nfsmount/dfs/nn /data/1/dfs/ns /data/{1,2,3,4}/dfs/dn

$ sudo mkdir /data/tmp
$ sudo chmod 1777 /data/tmp

11. 配置core-site.xml
$ sudo vim /etc/hadoop/conf/core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
 <name>fs.defaultFS</name>
 <value>hdfs://hadoop-master:8020</value>
</property>
<property>
 <name>hadoop.tmp.dir</name>
 <value>/data/tmp/hadoop-${user.name}</value>
</property>

<property>
  <name>hadoop.proxyuser.oozie.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.oozie.groups</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.hive.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.hive.groups</name>
  <value>*</value>
</property>
</configuration>

12. 配置hdfs-site.xml
$ sudo vim /etc/hadoop/conf/hdfs-site.xml

 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
 <name>dfs.namenode.name.dir</name>
 <value>/data/1/dfs/nn,/nfsmount/dfs/nn</value>
</property>
<property>
  <name>dfs.namenode.http-address</name>
  <value>hadoop-master:50070</value>
</property>

<property>
  <name>fs.namenode.checkpoint.period</name>
  <value>3600</value>
</property>
<property>
  <name>fs.namenode.checkpoint.dir</name>
  <value>/data/1/dfs/ns</value>
</property>
<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>hadoop-secondary:50090</value>
</property>

<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>
<property>
 <name>dfs.permissions.superusergroup</name>
 <value>supergroup</value>
</property>
<property>
 <name>dfs.datanode.data.dir</name>
 <value>/data/1/dfs/dn,/data/2/dfs/dn,/data/3/dfs/dn</value>
</property>
<property>
  <name>dfs.datanode.max.xcievers</name>
  <value>4096</value>
</property>
</configuration>

阅读全文 »

, ,

4 Comments

Hadoop集群(CDH4)实践之 (0) 前言

目录结构
Hadoop集群(CDH4)实践之 (0) 前言
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

本文内容
Hadoop集群(CDH4)实践之 (0) 前言

下面进入正文
在我初学Hadoop的期间,我写过一个系列的Hadoop入门文章,第一篇就是《Hadoop集群实践 之 (0) 完整架构设计
在之前的系列文章中,我对Hadoop的一些入门概念也进行了讲解,主要是针对我曾经所遇到过的一些疑惑。
同时,在之前的系列文章中,我还列出了一些小的操作Demo来加深对各个工具的理解。

那么为什么这次又要写这个系列的文章呢,看起来内容感觉都是重复的。
其实,主要是由于以下原因:
1. 之前的文章是基于Ubuntu 10.10 系统,也同样适用于新版的Ubuntu,但是采用CentOS作为生产环境的情况更多;
同时由于Ubuntu有一些改动与开源社区的步伐不太一致,因此目前有唱衰Ubuntu的趋势。
2. CentOS随着EPEL等扩展库的规范和快速发展,目前已经具备了和Ubuntu同等规模的丰富的软件库,通过YUM安装和部署软件也非常的方便;
3. 之前的文章是基于CDH3的,而目前Hadoop的发展,CDH4已经成为了主流,同时具备CDH3所不具备的一些功能,我觉得最有用的功能有以下:
a) NameNode HA,与secondary namenode不同,CDH4提供了一种HA的方式,可以确保双节点NameNode;
b) TaskTracker 提供了容错机制,可以确保并行计算过程中,不会因为某一个节点出错而导致整个并行计算的失败;

因此,基于以上原因,本文是在CentOS 6.4 x86_64的系统上,基于CDH4的环境下完成的。
不过,目前还没有完成Namenode HA 和 TaskTracker容错的测试,相关内容暂时还无法看到。
同时,本文采用了非YARN方式,而是与CDH3相同的MRv1计算框架,为了确保公司之前线上环境所开发的代码能够准确无误的运行。

下面,就让我们开始整个实战演练过程:
Hadoop集群(CDH4)实践之 (1) Hadoop(HDFS)搭建
Hadoop集群(CDH4)实践之 (2) HBase&Zookeeper搭建
Hadoop集群(CDH4)实践之 (3) Hive搭建
Hadoop集群(CHD4)实践之 (4) Oozie搭建
Hadoop集群(CHD4)实践之 (5) Sqoop安装

,

1 Comment

伪分布式安装部署CDH4.2.1与Impala[原创实践]

参考资料:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Quick-Start/cdh4qs_topic_3_3.html
http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/Installing-and-Using-Impala.html
http://blog.cloudera.com/blog/2013/02/from-zero-to-impala-in-minutes/

什么是Impala?
Cloudera发布了实时查询开源项目Impala,根据多款产品实测表明,它比原来基于MapReduce的Hive SQL查询速度提升3~90倍。Impala是Google Dremel的模仿,但在SQL功能上青出于蓝胜于蓝。

1. 安装JDK
$ sudo yum install jdk-6u41-linux-amd64.rpm

2. 伪分布式模式安装CDH4
$ cd /etc/yum.repos.d/
$ sudo wget http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/cloudera-cdh4.repo
$ sudo yum install hadoop-conf-pseudo

格式化NameNode.
$ sudo -u hdfs hdfs namenode -format

启动HDFS
$ for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done

创建/tmp目录
$ sudo -u hdfs hadoop fs -rm -r /tmp
$ sudo -u hdfs hadoop fs -mkdir /tmp
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp

创建YARN与日志目录
$ sudo -u hdfs hadoop fs -mkdir /tmp/hadoop-yarn/staging
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp/hadoop-yarn/staging

$ sudo -u hdfs hadoop fs -mkdir /tmp/hadoop-yarn/staging/history/done_intermediate
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp/hadoop-yarn/staging/history/done_intermediate

$ sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/hadoop-yarn/staging

$ sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
$ sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
阅读全文 »

, , ,

1 Comment