Hadoop 2.6.0 Multi Node Cluster Setup on Ubuntu 14.10


Namenode > hadoopmaster > 192.168.23.132

Datanodes >  hadoopslave1 > 192.168.23.133
                      hadoopslave2 > 192.168.23.134
                      hadoopslave3 > 192.168.23.135

Clone Hadoop Single node cluster as hadoopmaster

Hadoopmaster Node

          $ sudo gedit /etc/hosts

                      hadoopmaster   192.168.23.132
                      hadoopslave1   192.168.23.133
                      hadoopslave2   192.168.23.134
                      hadoopslave3   192.168.23.135

          $ sudo gedit /etc/hostname

                      hadoopmaster

          $ cd /usr/local/hadoop/etc/hadoop

          $ sudo gedit core-site.xml

                       replace localhost as hadoopmaster

          $ sudo gedit hdfs-site.xml

                       replace value 1 as 3 (represents no of datanode)

          $ sudo gedit yarn-site.xml

                       add the following configuration
       
                       <configuration>
                              <property>
                                  <name>yarn.resourcemanager.resource-tracker.address</name>
                                  <value>hadoopmaster:8025</value>
                       <property>
                       <property>
                                  <name>yarn.resourcemanager.scheduler.address</name>
                                  <value>hadoopmaster:8030</value>
                       <property>
                       <property>
                                  <name>yarn.resourcemanager.address</name>
                                  <value>hadoopmaster:8050</value>
                             </property>
                       </configuration>

          $ sudo gedit yarn-site.xml

                       replace mapreduce.framework.name as mapred.job.tracker

                       replace yarn as hadoopmaster:54311

          $ sudo rm -rf /usr/local/hadoop/hadoop_data

Shutdown hadoopmaster node

Clone Hadoopmaster Node as hadoopslave1, hadoopslave2, hadoopslave3

Hadoopslave Node (conf should be done on each slavenode)

          $ sudo gedit /etc/hostname

                      hadoopslave<nodenumberhere>

          $ sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/datanode

          $ sudo chown -R chaalpritam:chaalpritam /usr/local/hadoop

          $ sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

                       remove dfs.namenode.name.dir property section

reboot all nodes

Hadoopmaster Node

          $ sudo gedit /usr/local/hadoop/etc/hadoop/masters

                       hadoopmaster

          $ sudo gedit /usr/local/hadoop/etc/hadoop/slaves

                       remove localhost and add 

                       hadoopslave1
                       hadoopslave2
                       hadoopslave3

          $ sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

                       remove dfs.datanode.data.dir property section

          $ sudo mkdir -p /usr/local/hadoop/hadoop_data/hdfs/namenode

          $ sudo chown -R chaalpritam:chaalpritam /usr/local/hadoop

          $ sudo ssh-copy-id -i ~/.ssh/id_dsa.pub chaalpritam@hadoopmaster

          $ sudo ssh-copy-id -i ~/.ssh/id_dsa.pub chaalpritam@hadoopslave1

          $ sudo ssh-copy-id -i ~/.ssh/id_dsa.pub chaalpritam@hadoopslave2

          $ sudo ssh-copy-id -i ~/.ssh/id_dsa.pub chaalpritam@hadoopslave3

          $ sudo ssh hadoopmaster

          $ exit

          $ sudo ssh hadoopslave1

          $ exit

          $ sudo ssh hadoopslave2

          $ exit

          $ sudo ssh hadoopslave3

          $ exit

          $ hadoop namenode -format

          $ start-all.sh

          $ jps (check in all 3 datanodes)


http://hadoopmaster:8088/
http://hadoopmaster:50070/
http://hadoopmaster:50090/

http://hadoopmaster:50075/



15 comments:

  1. how can I clone the hadoop systems in ubutu
    I configured single node cluster in one ubutu system then how can I clone the configuration other vubutu systems AND I am following installation steps iam getting "sudo: unable to resolve the host hadoopslave1 in slave systems How can I solve this ?
    I AM NOT DOING THIS INSTALLATION STEPS IN VMWARE
    IAM DOING THIS STEPS DIRECTLY IN UNBUNTU

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Excellent blog. But DataNode in not showing on slave node. I have master and one slave. How to resolve ?

    ReplyDelete
  4. $ sudo gedit yarn-site.xml
    replace mapreduce.framework.name as mapred.job.tracker
    replace yarn as hadoopmaster:54311

    You should change this word "sudo gedit yarn-site.xml" because this word "mapreduce.framework.name" just have mapred-site.xml.

    ReplyDelete
  5. You need change when If you was changed hostname also hadoopmaster .

    http://localhost:8088/
    http://localhost:50070/
    http://localhost:50090/

    not

    http://master:8088/
    http://master:50070/
    http://master:50090/

    ReplyDelete
  6. then u can install vitual box or vmware and establish slave nodes there :)

    ReplyDelete
  7. check
    $ sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml

    dfs.namenode.name.dir property section (on master)

    dfs.datanode.data.dir property section (on slave)

    ReplyDelete
  8. thank you fr your suggestion :)

    ReplyDelete
  9. I had same problem and solved it.
    On slave, Drop the folder as follows
    /usr/local/hadoop/hadoop_store/~(remove)
    and try start-all.sh on master


    The folder that you remove will be created again. Don't worry

    ReplyDelete
  10. Thanks! I was struggling every day because of this.. i will give you something delicious :) ThanksThanksThanks

    ReplyDelete
  11. Have you a example for testing the cluster?
    I have done all (single and multi), but my nodes don't appear on browser.
    But when I execute "jps" on machines (master and slaves), some lines appear. What is the lines?

    ReplyDelete
  12. jps tells u what stuff running on your node :) just run a word count program to test http://chaalpritam.blogspot.com/2015/01/running-wordcount-on-hadoop-multi-node.html

    ReplyDelete
  13. Nice post thanks for sharing. If any one need Hadoop Interview Questions & Answers and Free Material Click Here

    ReplyDelete
  14. please help me.
    I run the program pi Examples of hadoop, but does not show me the applications running on localhost: 8088
    I see only the nodes that I have connected, but nothing appears applications.
    please help, I want to see the application running on localhost: 8088, as is running on: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.15/bk_using-apache-hadoop/content/running_mapreduce_examples_on_yarn.html

    ReplyDelete
  15. Enormously invigorating! I was drawing in dependably in light of this.. i wanna give you something delectable :)

    Informatica Training in Chennai | Hadoop Training in Chennai

    ReplyDelete

 

Flickr Photostream

Twitter Updates