Sqoop1和Sqoop2的安装

2016-07-13 09:38:48   作者：MangoCool   来源：MangoCool

Sqoop稳定版本是sqoop-1.4.6，Sqoop最新版本是sqoop-1.99.6，1.99.6和1.4.6这两个版本完全不兼容，也就是我们所说的Sqoop1和Sqoop2，有不同的特征。官网说明Sqoop2（也就是sqoop-1.99.6）并不用于生产环境，这也是跟它本身稳定性相关的。

Sqoop1安装部署：

1、安装环境：

hadoop-2.7.2, hive-2.1.0, sqoop-1.4.6

2、下载及解压：

地址：http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

解压：

tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz sqoop-1.4.6

3、配置sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/hadoop/SW/hadoop

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/hadoop/SW/hadoop

#set the path to where bin/hbase is available export
export HBASE_HOME=/home/hadoop/SW/hbase

#Set the path to where bin/hive is available
export HIVE_HOME=/home/hadoop/SW/hive-2.0.0

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/home/hadoop/SW/zookeeper

4、添加所须包至sqoop/lib下：

我这里用到了MySQL和Oracle，所以添加包：mysql-connector-java-5.1.38.jar和ojdbc6.jar

5、配置环境变量：

vim /etc/profile

添加：

#Sqoop  
export SQOOP_HOME=/home/hadoop/SW/Sqoop-1.4.6
export PATH=$PATH:$SQOOP_HOME/bin

保存执行生效：

source /etc/profile

Sqoop2安装部署：

1、安装环境：

hadoop-2.7.2, hive-2.1.0, sqoop-1.4.6

2、下载及解压：

地址：http://mirrors.hust.edu.cn/apache/sqoop/1.99.6/sqoop-1.99.6-bin-hadoop200.tar.gz

解压：

tar -zxvf sqoop-1.99.6-bin-hadoop200.tar.gz
mv sqoop-1.99.6-bin-hadoop200.tar.gz sqoop-1.99.6

3、配置sqoop.properties

路径：sqoop-1.99.6/server/conf

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/hadoop/SW/hadoop/etc/hadoop

4、配置catalina.properties

路径：sqoop-1.99.6/server/conf

common.loader=/home/hadoop/SW/hadoop/share/hadoop/common/*.jar,/home/hadoop/SW/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/hdfs/*.jar,/home/hadoop/SW/hadoop/share/hadoop/hdfs/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/SW/hadoop/share/hadoop/mapreduce/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/tools/*.jar,/home/hadoop/SW/hadoop/share/hadoop/tools/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/yarn/*.jar,/home/hadoop/SW/hadoop/share/hadoop/yarn/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/httpfs/tomcat/lib/*.jar,/home/hadoop/SW/hive-2.1.0/lib/*.jar

5、添加所须包至lib下：

路径：sqoop-1.99.6/server/lib

我这里用到了MySQL和Oracle，所以添加包：mysql-connector-java-5.1.38.jar和ojdbc6.jar

6、配置环境变量：

vim /etc/profile

添加：

#Sqoop export SQOOP_HOME=/home/hadoop/SW/sqoop-1.99.6
export PATH=$SQOOP_HOME/bin:$PATH
export CATALINA_BASE=/home/hadoop/SW/sqoop-1.99.6/server
export LOGDIR=$SQOOP_HOME/logs/

保存执行生效：

source /etc/profile

7、启动：

sqoop2-server start

同时启动一下Hadoop的historyserver进程：

./mr-jobhistory-daemon.sh start historyserver --config ../etc/hadoop

以避免后面遇到以下问题：

Call From hadoop/192.168.21.181 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused。

注：最好其他节点也启动此进程。

客户端启动：

sqoop2-shell

sqoop执行过程中详细信息显示可以执行以下命令：

sqoop:000> set option --name verbose --value true
Verbose option was changed to true

标签： Sqoop1 Sqoop2

分享：

上一篇yum安装MySQL5.7

下一篇javax.jdo.JDODataStoreException: Required table missing : "`VERSION`" in Catalog "" Schema "".