Sqoop1和Sqoop2的安装

2016-07-13 09:38:48   作者:MangoCool   来源:MangoCool

Sqoop版本是sqoop-1.4.6,Sqoop最新版本是sqoop-1.99.6,1.99.6和1.4.6这两个版本完全不兼容,也就是我们所说的Sqoop1和Sqoop2,有不同的特征。官网说明Sqoop2(也就是sqoop-1.99.6)并不用于生产环境,这也是跟它本身稳定性相关的。


Sqoop1安装部署:

1、安装环境:

hadoop-2.7.2, hive-2.1.0, sqoop-1.4.6

2、下载及解压:

地址:http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

解压:

tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz sqoop-1.4.6

3、配置sqoop-env.sh

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/hadoop/SW/hadoop

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/hadoop/SW/hadoop

#set the path to where bin/hbase is available export
export HBASE_HOME=/home/hadoop/SW/hbase

#Set the path to where bin/hive is available
export HIVE_HOME=/home/hadoop/SW/hive-2.0.0

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/home/hadoop/SW/zookeeper

4、添加所须包至sqoop/lib下:

我这里用到了MySQL和Oracle,所以添加包:mysql-connector-java-5.1.38.jar和ojdbc6.jar

5、配置环境变量:

vim /etc/profile

添加:

#Sqoop  
export SQOOP_HOME=/home/hadoop/SW/Sqoop-1.4.6
export PATH=$PATH:$SQOOP_HOME/bin

保存执行生效:

source /etc/profile


Sqoop2安装部署:

1、安装环境:

hadoop-2.7.2, hive-2.1.0, sqoop-1.4.6

2、下载及解压:

地址:http://mirrors.hust.edu.cn/apache/sqoop/1.99.6/sqoop-1.99.6-bin-hadoop200.tar.gz

解压:

tar -zxvf sqoop-1.99.6-bin-hadoop200.tar.gz
mv sqoop-1.99.6-bin-hadoop200.tar.gz sqoop-1.99.6

3、配置sqoop.properties

路径:sqoop-1.99.6/server/conf

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/hadoop/SW/hadoop/etc/hadoop

4、配置catalina.properties

路径:sqoop-1.99.6/server/conf

common.loader=/home/hadoop/SW/hadoop/share/hadoop/common/*.jar,/home/hadoop/SW/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/hdfs/*.jar,/home/hadoop/SW/hadoop/share/hadoop/hdfs/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/SW/hadoop/share/hadoop/mapreduce/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/tools/*.jar,/home/hadoop/SW/hadoop/share/hadoop/tools/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/yarn/*.jar,/home/hadoop/SW/hadoop/share/hadoop/yarn/lib/*.jar,/home/hadoop/SW/hadoop/share/hadoop/httpfs/tomcat/lib/*.jar,/home/hadoop/SW/hive-2.1.0/lib/*.jar

5、添加所须包至lib下:

路径:sqoop-1.99.6/server/lib

我这里用到了MySQL和Oracle,所以添加包:mysql-connector-java-5.1.38.jar和ojdbc6.jar

6、配置环境变量:

vim /etc/profile

添加:

#Sqoop export SQOOP_HOME=/home/hadoop/SW/sqoop-1.99.6
export PATH=$SQOOP_HOME/bin:$PATH
export CATALINA_BASE=/home/hadoop/SW/sqoop-1.99.6/server
export LOGDIR=$SQOOP_HOME/logs/

保存执行生效:

source /etc/profile

7、启动:

sqoop2-server start

同时启动一下Hadoop的historyserver进程:

./mr-jobhistory-daemon.sh start historyserver --config ../etc/hadoop

以避免后面遇到以下问题:

Call From hadoop/192.168.21.181 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused。

注:最好其他节点也启动此进程。

客户端启动:

sqoop2-shell

sqoop执行过程中详细信息显示可以执行以下命令:

sqoop:000> set option --name verbose --value true
Verbose option was changed to true

标签: Sqoop1 Sqoop2

分享:

上一篇yum安装MySQL5.7

下一篇javax.jdo.JDODataStoreException: Required table missing : "`VERSION`" in Catalog "" Schema "".

关于我

一个喜欢唱歌,热衷旅行,爱好电子产品的码农。没事,跟三五好友吼上几嗓子,约上几个背着行囊去露营或者宅在家里抱着孩子敲代码。

座右铭:当你的才华还撑不起你的野心的时候,你就应该静下心来学习,永不止步!

            人生之旅历途甚长,所争决不在一年半月,万不可因此着急失望,招精神之萎葸。


Copyright 芒果酷(mangocool.com) All rights reserved. 湘ICP备14019394号

免责声明:本网站部分文章转载其他媒体,意在为公众提供免费服务。如有信息侵犯了您的权益,可与本网站联系,本网站将尽快予以撤除。