org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

2016-06-28 14:30:39   作者:MangoCool   来源:MangoCool

执行sqoop命令将Oracle数据导入hive,并用Parquet作为存储:

./sqoop import --connect jdbc:oracle:thin:@192.168.20.19:1521:orcl --username xxx --password xxx --table TB_DIAG_MACHINE_RUN_TIME --fields-terminated-by '\t' --verbose -m 1 --hive-import --hive-database xbdp --hive-table tb_machine --as-parquetfile


出现报错:

16/06/28 12:39:38 INFO mapreduce.Job: Task Id : attempt_1467084753393_0003_m_000000_0, Status : FAILED
Container launch failed for container_1467084753393_0003_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

16/06/28 12:39:45 INFO mapreduce.Job:  map 100% reduce 0%
16/06/28 12:39:45 INFO mapreduce.Job: Job job_1467084753393_0003 failed with state FAILED due to: Task failed task_1467084753393_0003_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0

16/06/28 12:39:45 INFO mapreduce.Job: Counters: 9
        Job Counters
                Failed map tasks=21
                Killed map tasks=3
                Launched map tasks=21
                Other local map tasks=21
                Total time spent by all maps in occupied slots (ms)=14
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=7
                Total vcore-milliseconds taken by all map tasks=7
                Total megabyte-milliseconds taken by all map tasks=10752
16/06/28 12:39:45 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
16/06/28 12:39:45 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 59.7048 seconds (0 bytes/sec)
16/06/28 12:39:45 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/06/28 12:39:45 INFO mapreduce.ImportJobBase: Retrieved 0 records.
16/06/28 12:39:45 ERROR tool.ImportTool: Error during import: Import job failed!


问题原因:

因为Hive底层执行job是hadoop的MP,如果auxService:mapreduce_shuffle不存在,我们就须要配置。


解决办法:

设置yarn-site.xml文件

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

重启yarn即可。


参考来源:

http://stackoverflow.com/questions/30921838/auxservicemapreduce-shuffle-does-not-exist-on-hive-1-2-0

标签: Sqoop Hive Oracle error

分享:

关于我

一个喜欢唱歌,热衷旅行,爱好电子产品的码农。没事,跟三五好友吼上几嗓子,约上几个背着行囊去露营或者宅在家里抱着孩子敲代码。

座右铭:当你的才华还撑不起你的野心的时候,你就应该静下心来学习,永不止步!

            人生之旅历途甚长,所争决不在一年半月,万不可因此着急失望,招精神之萎葸。


Copyright 芒果酷(mangocool.com) All rights reserved. 湘ICP备14019394号

免责声明:本网站部分文章转载其他媒体,意在为公众提供免费服务。如有信息侵犯了您的权益,可与本网站联系,本网站将尽快予以撤除。