Hadoop中的若干不胜的消灭办法,Linux单机运转ha

作者: 计算机网络  发布:2019-11-04

异常1

Hadoop-0.19.2的代码能够到Apache上下载,使用的Linux机器是昂CoraHEL 5,Linux上安装的Java版本为1.6.0_16,并且JAVA_HOME=/usr/java/jdk1.6.0_16
推行进度

Hadoop@Ubuntu:~$ hadoop/bin/hadoop jar hadoop-0.20.2-examples.jar wordcount input01 output01
Exception in thread "main" java.io.IOException: Error opening job jar: hadoop-0.20.2-examples.jar
    at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:131)
    at java.util.jar.JarFile.<init>(JarFile.java:150)
    at java.util.jar.JarFile.<init>(JarFile.java:87)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:88)

1、ssh无密码验证登入localhost
有限支持Linux系统的ssh服务业已运转,并确定保证能够通过无密码验证登入本机Linux系统。若是不能够作保,能够遵照如下的手续去做:
(1卡塔 尔(英语:State of Qatar)运转命令行窗口,试行命令行:
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
(2卡塔 尔(英语:State of Qatar)ssh登录localhost,实践命令行:
$ ssh localhost
先是次登陆,会唤醒您不可能树立到127.0.0.1的接连,是或不是要创造,输入yes就可以,上边是能够由此无密码验证登入的音讯:
[root@localhost hadoop-0.19.2]# ssh localhost
Last login: Sun Aug  1 18:35:37 2010 from 192.168.0.104
[root@localhost ~]#

发出这一个特别后,找了多数帖子都还未有解答,也许有多数个人超出了看似的状态。其实那相近实际不是java包有标题,难题也大概的可笑,便是地点的通令行中*
**
hadoop-0.20.2-examples.jar
路子不完全形成的,须要当心一下命令行业前的职分,例如对于自身的动静,改为
hadoop/hadoop-0.20.2-examples.jar***就足以了

2、Hadoop-0.19.0配置
下载hadoop-0.19.0.tar.gz,大概是40.3M,解压缩到Linux系统钦赐目录,www.linuxidc.com此间作者的是/root/hadoop-0.19.2目录下。
上面遵照有序的步调来验证配置进度:
(1)修改hadoop-env.sh配置
将Java境遇的配置实行订正后,并撤回注释“#”,修正后的表现:
export JAVA_HOME=/usr/java/jdk1.6.0_16
(2)修改hadoop-site.xml配置
在<configuration>与</configuration>加上3个本性的配备,改良后的安插文件内容为:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

异常2

3运行wordcount实例
wordcount例子是hadoop发行李包裹中自带的实例,通过运维实例能够心得并尝试精通hadoop在实施MapReduce任务时的实践进度。遵照法定的“Hadoop Quick Start”教程为主得以轻松地得以达成,下边轻易说一下作者的幼学壮行进度。
导航到hadoop目录上边,作者的是/root/hadoop-0.19.0。
(1)格式化HDFS
执行格式化HDFS的命令行:
[root@localhost hadoop-0.19.2]# bin/hadoop namenode -format
格式化实行音信如下所示:
10/08/01 19:04:02 INFO namenode.NameNode: STARTUP_MSG:

hadoop@ubuntu:~$ hadoop/bin/hadoop jar hadoop/hadoop-0.20.2-examples.jar wordcount input01 output02

Re-format filesystem in /tmp/hadoop-root/dfs/name ? (Y or N) y
Format aborted in /tmp/hadoop-root/dfs/name
10/08/01 19:04:05 INFO namenode.NameNode: SHUTDOWN_MSG:

java.io.IOException: Task process exit with nonzero status of 1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)

(2卡塔尔运维Hadoop相关后台进度
实施命令行:
[root@localhost hadoop-0.19.2]# bin/start-all.sh
起步试行音讯如下所示:
starting namenode, logging to /root/hadoop-0.19.0/bin/../logs/hadoop-root-namenode-localhost.out
localhost: starting datanode, logging to /root/hadoop-0.19.0/bin/../logs/hadoop-root-datanode-localhost.out
localhost: starting secondarynamenode, logging to /root/hadoop-0.19.0/bin/../logs/hadoop-root-secondarynamenode-localhost.out
starting jobtracker, logging to /root/hadoop-0.19.0/bin/../logs/hadoop-root-jobtracker-localhost.out
localhost: starting tasktracker, logging to /root/hadoop-0.19.0/bin/../logs/hadoop-root-tasktracker-localhost.out
(3卡塔尔国盘算进行wordcount职务的数据
先是,这里在地点创造了多个多少目录input,并拷贝一些文本到该目录下边,如下所示:
[root@localhost hadoop-0.19.2]# mkdir input
[root@localhost hadoop-0.19.2]# cp CHANGES.txt LICENSE.txt NOTICE.txt README.txt input/
下一场,将当地目录input上流传HDFS文件系统上,推行如下命令:
[root@localhost hadoop-0.19.2]# bin/hadoop fs -put input/ input
(4)启动wordcount任务
推行如下命令行:
[root@localhost hadoop-0.19.2]# bin/hadoop jar hadoop-0.19.2-examples.jar wordcount input output
元数据目录为input,输出数据目录为output。
任务实践音信如下所示:
10/08/01 19:06:15 INFO mapred.FileInputFormat: Total input paths to process : 4
10/08/01 19:06:15 INFO mapred.JobClient: Running job: job_201008011904_0002
10/08/01 19:06:16 INFO mapred.JobClient:  map 0% reduce 0%
10/08/01 19:06:22 INFO mapred.JobClient:  map 20% reduce 0%
10/08/01 19:06:24 INFO mapred.JobClient:  map 40% reduce 0%
10/08/01 19:06:25 INFO mapred.JobClient:  map 60% reduce 0%
10/08/01 19:06:27 INFO mapred.JobClient:  map 80% reduce 0%
10/08/01 19:06:28 INFO mapred.JobClient:  map 100% reduce 0%
10/08/01 19:06:38 INFO mapred.JobClient:  map 100% reduce 26%
10/08/01 19:06:40 INFO mapred.JobClient:  map 100% reduce 100%
10/08/01 19:06:41 INFO mapred.JobClient: Job complete: job_201008011904_0002
10/08/01 19:06:41 INFO mapred.JobClient: Counters: 16
10/08/01 19:06:41 INFO mapred.JobClient:   File Systems
10/08/01 19:06:41 INFO mapred.JobClient:     HDFS bytes read=301489
10/08/01 19:06:41 INFO mapred.JobClient:     HDFS bytes written=113098
10/08/01 19:06:41 INFO mapred.JobClient:     Local bytes read=174004
10/08/01 19:06:41 INFO mapred.JobClient:     Local bytes written=348172
10/08/01 19:06:41 INFO mapred.JobClient:   Job Counters
10/08/01 19:06:41 INFO mapred.JobClient:     Launched reduce tasks=1
10/08/01 19:06:41 INFO mapred.JobClient:     Launched map tasks=5
10/08/01 19:06:41 INFO mapred.JobClient:     Data-local map tasks=5
10/08/01 19:06:41 INFO mapred.JobClient:   Map-Reduce Framework
10/08/01 19:06:41 INFO mapred.JobClient:     Reduce input groups=8997
10/08/01 19:06:41 INFO mapred.JobClient:     Combine output records=10860
10/08/01 19:06:41 INFO mapred.JobClient:     Map input records=7363
10/08/01 19:06:41 INFO mapred.JobClient:     Reduce output records=8997
10/08/01 19:06:41 INFO mapred.JobClient:     Map output bytes=434077
10/08/01 19:06:41 INFO mapred.JobClient:     Map input bytes=299871
10/08/01 19:06:41 INFO mapred.JobClient:     Combine input records=39193
10/08/01 19:06:41 INFO mapred.JobClient:     Map output records=39193
10/08/01 19:06:41 INFO mapred.JobClient:     Reduce input records=10860
(5卡塔尔国查看职责实行结果
能够透过如下命令行:
bin/hadoop fs -cat output/*
实践结果,截取部分显得如下所示:
vijayarenu      20
violations.     1
virtual 3
vis-a-vis       1
visible 1
visit   1
volume  1
volume, 1
volumes 2
volumes.        1
w.r.t   2
wait    9
waiting 6
waiting.        1
waits   3
want    1
warning 7
warning,        1
warnings        12
warnings.       3
warranties      1
warranty        1
warranty,       1
(6卡塔尔国终止Hadoop相关后台过程
试行如下命令行:
[root@localhost hadoop-0.19.2]# bin/stop-all.sh
举行音信如下所示:
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode
一度将下边列出的5个经过jobtracker、tasktracker、namenode、datanode、secondarynamenode
特别深入分析
在实行上述推行进程中,大概会遇见某种非常情状,大概解析如下:
1、Call to localhost/127.0.0.1:9000 failed on local exception异常
(1卡塔 尔(英语:State of Qatar)万分描述
兴许你会在推行如下命令行的时候现身:
[root@localhost hadoop-0.19.2]# bin/hadoop jar hadoop-0.19.0-examples.jar wordcount input output
出错万分音讯如下所示:
10/08/01 19:50:55 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).
10/08/01 19:50:56 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s).
10/08/01 19:50:57 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s).
10/08/01 19:50:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s).
10/08/01 19:50:59 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s).
10/08/01 19:51:00 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s).
10/08/01 19:51:01 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s).
10/08/01 19:51:02 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s).
10/08/01 19:51:03 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s).
10/08/01 19:51:04 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s).
java.lang.RuntimeException: java.io.IOException: Call to localhost/127.0.0.1:9000 failed on local exception: Connection refused
        at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:323)
        at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:295)
        at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:268)
        at org.apache.hadoop.examples.WordCount.run(WordCount.java:146)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: java.io.IOException: Call to localhost/127.0.0.1:9000 failed on local exception: Connection refused
        at org.apache.hadoop.ipc.Client.call(Client.java:699)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
        at $Proxy0.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319)
        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:104)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:177)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:74)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1367)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:56)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1379)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
        at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:319)
        ... 21 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:299)
        at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:772)
        at org.apache.hadoop.ipc.Client.call(Client.java:685)
        ... 33 more
(2卡塔 尔(英语:State of Qatar)非凡深入分析
从上述十分音讯深入分析,那句是首要:
Retrying connect to server: localhost/127.0.0.1:9000.
是说在品尝13次三回九转到“server”时都力所不如成功,那就表明到server的通讯链路是不通的。大家曾在hadoop-site.xml中配置了namenode结点的值,如下所示:
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
就此,很恐怕namenode进度根本就从未运维,更不必谈要施行职责了。
上述非常,进程是:
格式化了HDFS,可是尚未试行bin/start-all.sh,直接开发银行wordcount职责,就涌出上述非常。
为此,应该实施bin/start-all.sh现在再起步wordcount职责。
2、Input path does not exist异常
(1卡塔 尔(阿拉伯语:قطر‎格外描述
当你在眼下hadoop目录下边创制一个input目录,并cp有些文件到个中,初阶实行:
[root@localhost hadoop-0.19.2]# bin/hadoop namenode -format
[root@localhost hadoop-0.19.2]# bin/start-all.sh 
那会儿,你认为input已经存在,应该能够实行wordcount职责了:
[root@localhost hadoop-0.19.2]# bin/hadoop jar hadoop-0.19.0-examples.jar wordcount input output
结果抛出一批分外,音讯如下:
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/root/input
        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:179)
        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:190)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:782)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1127)
        at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
上述非常,作者模拟的历程是:
[root@localhost hadoop-0.19.2]# bin/hadoop fs -rmr input
Deleted hdfs://localhost:9000/user/root/input
[root@localhost hadoop-0.19.2]# bin/hadoop fs -rmr output
Deleted hdfs://localhost:9000/user/root/output
(2卡塔 尔(阿拉伯语:قطر‎卓殊解析
本地的input目录并不曾上传到HDFS上,所现身org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/root/input
只供给实行上传的授命就能够:
[root@localhost hadoop-0.19.2]# bin/hadoop fs -put input/ input

11/03/15 12:54:09 WARN mapred.JobClient: Error reading task output
......

图片 1

以此主题材料苦闷了自家总体意气风发晚间,汉语博客基本没搜到什么有参谋价值的篇章,老外的不菲博客提到了,不过不菲也没说领会。此中有一点点有提示效果,比如:

Just an FYI, found the solution to this problem.

Apparently, it's an OS limit on the number of sub-directories that can be reated in another directory.  In this case, we had 31998 sub-directories uder hadoop/userlogs/, so any new tasks would fail in Job Setup.

From the unix command line, mkdir fails as well:
  $ mkdir hadoop/userlogs/testdir
  mkdir: cannot create directory `hadoop/userlogs/testdir': Too many links

Difficult to track down because the Hadoop error message gives no hint whasoever.  And normally, you'd look in the userlog itself for more info, butin this case the userlog couldn't be created.

标题是,作者得以因此这么些小测验,在userlogs上边能够加上狂妄的文本夹和文件,当然也可能有希望有个别人真的就是其生机勃勃标题,不能够丰盛。
然后本身的解决办法是,间接把这几个userlogs给去掉可能换一个文本夹名

hadoop@ubuntu:~$ mv /home/hadoop/hadoop/logs/uerlogs/ /home/hadoop/hadoop/logs/uerlogsOLD/

即,把本来的文件夹改名成userlogsOLD(也就是风流罗曼蒂克种移除、保存方法了卡塔尔,重新运维

hadoop@ubuntu:~$ hadoop/bin/hadoop jar hadoop/hadoop-0.20.2-examples.jar wordcount input01 output03

11/03/15 14:21:23 INFO input.FileInputFormat: Total input paths to process : 3
11/03/15 14:21:23 INFO mapred.JobClient: Running job: job_201103151252_0004
11/03/15 14:21:24 INFO mapred.JobClient:  map 0% reduce 0%
11/03/15 14:21:32 INFO mapred.JobClient:  map 66% reduce 0%
11/03/15 14:21:35 INFO mapred.JobClient:  map 100% reduce 0%
11/03/15 14:21:44 INFO mapred.JobClient:  map 100% reduce 100% 11/03/15 14:21:46 INFO mapred.JobClient: Job complete: job_201103151252_0004
......

标题之后解除了!不过我要么不懂那是怎么着来头引致的,但能够一定的是有关日志的存款和储蓄量的主题材料。因为才起来学,eclpse下新建MapReduce工程也能跑起来了,稳步猜想会了然。留此权当笔记!

图片 2

本文由今晚开什么码发布于计算机网络,转载请注明出处:Hadoop中的若干不胜的消灭办法,Linux单机运转ha

关键词:

上一篇:访问数据库
下一篇:没有了