cdh deployment notes

  • Shutting down NameNode at java.net.UnknownHostException : hmaster

This problems caused by can’t analysis domain name : hmaster. We need to add localhost mapping in /etc/hosts, like the following :


127.0.0.1 hmaster

  • when executing `for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done`, showing ‘Error: JAVA_HOME is not set and could not be found.’

For example, we need to export $JAVA_HOME in /etc/init.d/hadoop-hdfs-datanode.


export JAVA_HOME=/home/hadoop/software/jdk1.7.0_60

  •  All directories in dfs.datanode.data.dir are invalid ‘/data/1/dfs/dn /data/2/dfs/dn /data/3/dfs/dn’

It is because the authority of the directories are invalid, we need to do like the following :


hadoop@hslave sudo chown -R hdfs:hdfs /data/1/dfs/dn /data/2/dfs/dn /data/3/dfs/dn /data/4/dfs/dn

hadoop@hslave sudo chmod 700 /data/1/dfs/dn /data/2/dfs/dn /data/3/dfs/dn /data/4/dfs/dn

  •  Invalid directory or I/O error occurred for dir: /data/1/dfs/nn

hadoop@hmaster sudo chmod 700 /data/1/dfs/nn /nfsmount/dfs/nn

  •  Call From hmaster/127.0.0.1 to hmaster.dw.imdada.cn:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

This error caused by confusing mappings, in my /etc/hosts, i set these two mappings :


10.10.2.2 hmaster.dw.imdada.cn

127.0.0.1 hmaster.dw.imdada.cn

  •   FAILED:Execution Error,return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

This is because the authorities of hdfs folders, make sure your hdfs input folder authority is 0777

  • Incompatible clusterIDs in /data/1/dfs/dn

This is because the folder ‘/data/1/dfs/dn ‘ reserved the last namenode -format informs, so i need to delete the folder ‘/data/1/dfs/dn ‘.

  • Hbase-master can not start.

The reason to the phenomenon is because that hmaster should not start HregionServer once Hmaster has been started. Another reason is that we should first stop HregionServer on hslave, then to start hmaster server. (Which I mean in distributed cluster with zookeeper controlling cluster.)

  •  Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask] with root cause

It is because that hdfs dir has already exist and it was operated by ‘truncate’ command, we need to clear the dir in hdfs.


$ hadoop dfs -rm -r /user/hive/warehouse/alogs/date=<a class="explorer-browse-links">date=0115</a>

 

 

 

发表评论

电子邮件地址不会被公开。 必填项已用*标注

您可以使用这些HTML标签和属性: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>