org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

2016-06-28 14:30:39   作者：MangoCool   来源：MangoCool

执行sqoop命令将Oracle数据导入hive，并用Parquet作为存储：

./sqoop import --connect jdbc:oracle:thin:@192.168.20.19:1521:orcl --username xxx --password xxx --table TB_DIAG_MACHINE_RUN_TIME --fields-terminated-by '\t' --verbose -m 1 --hive-import --hive-database xbdp --hive-table tb_machine --as-parquetfile

出现报错：

16/06/28 12:39:38 INFO mapreduce.Job: Task Id : attempt_1467084753393_0003_m_000000_0, Status : FAILED
Container launch failed for container_1467084753393_0003_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
        at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

16/06/28 12:39:45 INFO mapreduce.Job:  map 100% reduce 0%
16/06/28 12:39:45 INFO mapreduce.Job: Job job_1467084753393_0003 failed with state FAILED due to: Task failed task_1467084753393_0003_m_000001
Job failed as tasks failed. failedMaps:1 failedReduces:0

16/06/28 12:39:45 INFO mapreduce.Job: Counters: 9
        Job Counters
                Failed map tasks=21
                Killed map tasks=3
                Launched map tasks=21
                Other local map tasks=21
                Total time spent by all maps in occupied slots (ms)=14
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=7
                Total vcore-milliseconds taken by all map tasks=7
                Total megabyte-milliseconds taken by all map tasks=10752
16/06/28 12:39:45 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
16/06/28 12:39:45 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 59.7048 seconds (0 bytes/sec)
16/06/28 12:39:45 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
16/06/28 12:39:45 INFO mapreduce.ImportJobBase: Retrieved 0 records.
16/06/28 12:39:45 ERROR tool.ImportTool: Error during import: Import job failed!

问题原因：

因为Hive底层执行job是hadoop的MP，如果auxService:mapreduce_shuffle不存在，我们就须要配置。

解决办法：

设置yarn-site.xml文件

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

重启yarn即可。

参考来源：

http://stackoverflow.com/questions/30921838/auxservicemapreduce-shuffle-does-not-exist-on-hive-1-2-0

标签： Sqoop Hive Oracle error

分享：

上一篇Hive on Spark

下一篇org.kitesdk.data.DatasetNotFoundException: Unknown dataset URI: hive:/xbdp/tb_machine. Check that JARs for hive datasets are on the classpath.