HADOOP SINGLE NODE INSTALLATION ON UBUNTU 14.04(CS6712-GRID AND CLOUD LABORATORY-ANNA UNIVERSITY 2013 Regulation)

HADOOP SINGLE NODE INSTALLATION ON UBUNTU 14.04

PREREQUISITES

* Java (version 1.6.0 or above) should be installed

[ If java is not installed you can try any of these methods to install java

Method 1:To install the openJDK JDK and JRE 8 use (replace 8 with the version you want, such as 7 or 6):

                          sudo apt-get install openjdk-8-jdk

Method 2:If you instead want to install the official Oracle JDK and JRE and definitely want to install   through apt-get then do (you can replace the 8 with other versions such as 9, and 7):

                          sudo add-apt-repository ppa:webupd8team/java

                          sudo apt-get update

                          sudo apt-get install oracle-java8-installer

To automatically setup the Java 7 environment variables JAVA-HOME and PATH

            Sudo apt-get install oracle-java7-set-default ]

* SSH should be installed and sshd must be running.

            [ If ssh is not installed, you can run the following command to install it

                                    sudo apt-get install openssh-server

               check ssh using the following commands after installing

                                    which ssh

                                    output should be -/usr/bin/ssh

                                    which sshd

                                    output should be -/usr/sbin/sshd

HADOOP USER CREATION

user@node:~$ sudo addgroup hadoop

[sudo] password for user:

Adding group `hadoop' (GID 1001) ...

Done.

user@node:~$ sudo adduser --ingroup hadoop hdpuser

Adding user `hdpuser' ...

Adding new user `hdpuser' (1001) with group `hadoop' ...

Creating home directory `/home/hdpuser' ...

Copying files from `/etc/skel' ...

Enter new UNIX password:

Retype new UNIX password:

passwd: password updated successfully

Changing the user information for hdpuser

Enter the new value, or press ENTER for the default

            Full Name []:

            Room Number []:

            Work Phone []:

            Home Phone []:

            Other []:

Is the information correct? [Y/n]

SWITCH TO SUPER USER TO ADD  HADOOP USER TO SUDOERS GROUP

Switch to root user                 -       su root

Add the hadoop user to sudoers list by additing the below entry in the file /etc/sudoers

            hadpuser ALL=(ALL:ALL) ALL

                        (under      # User privilege specification

                                                  root    ALL=(ALL:ALL) ALL )

VERIFY JAVA INSTALLATION

Switch to hadoop user              -      su hadoop

hdpuser@node:~$ java -version

java version "1.7.0_80"

Java(TM) SE Runtime Environment (build 1.7.0_80-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

hdpuser@node:~$ update-alternatives --config java

There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status

------------------------------------------------------------

  0            /usr/lib/jvm/java-7-oracle/jre/bin/java          1072      auto mode

  1            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071      manual mode

* 2            /usr/lib/jvm/java-7-oracle/jre/bin/java          1072      manual mode

Press enter to keep the current choice[*], or type selection number: 

hdpuser@node:~$

UPDATE JAVA VARIABLES IN THE ~/.BASHRC FILE

Add the below entry in the ~/.bashrc file

export JAVA_HOME=/usr/lib/jvm/java-7-oracle

export PATH=$PATH:/usr/lib/jvm/java-7-oracle/bin

source the .bashrc file using the command

source .bashrc

VERIFY SSH INSTALLATION

hdpuser@node:~$ which ssh

/usr/bin/ssh

hdpuser@node:~$ which sshd

/usr/sbin/sshd

SSH KEY GENERATION

hdpuser@node:~$ ssh-keygen -t rsa -P ""

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hdpuser/.ssh/id_rsa):

Created directory '/home/hdpuser/.ssh'.

Your identification has been saved in /home/hdpuser/.ssh/id_rsa.

Your public key has been saved in /home/hdpuser/.ssh/id_rsa.pub.

The key fingerprint is:

da:4c:9a:89:bb:02:ac:7e:00:70:16:11:bc:fa:49:5e hdpuser@node

The key's randomart image is:

+--[ RSA 2048]----+

| .++             |

|. +              |

|.o .             |

|. .              |

|o.      S        |

|oo. E. O         |

|.=.o. = o        |

|. =. .           |

|....o.           |

+-----------------+

hdpuser@node:~$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

DOWNLOADING AND INSTALLING HADOOP

[ Hadoop can be downloaded using the below link if you don't have the pac kage in your system    wget http://mirrors.sonic.net/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz]

hdpuser@node:~$ cd /home/user/Documents/

hdpuser@node:/home/user/Documents$ sudo mv hadoop-2.6.0.tar.gz /usr/local/

[sudo] password for hdpuser:

hdpuser@node:/home/user/Documents$ cd /usr/local/

hdpuser@node:/usr/local$ sudo tar xvzf hadoop-2.6.0.tar.gz

hdpuser@node:/usr/local$ sudo chown -R hdpuser:hadoop hadoop-2.6.0

hdpuser@node:/usr/local$ sudo ln -s hadoop-2.6.0 hadoop

Add the below entry in the ~/.bashrc file and source the .bashrc file

export HADOOP_HOME=/usr/local/hadoop

hdpuser@node:/usr/local$ hadoop version

Hadoop 2.6.0

Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1

Compiled by jenkins on 2014-11-13T21:10Z

Compiled with protoc 2.5.0

From source with checksum 18e43357c8f927c0695f1e9522859d6a

This command was run using /usr/local/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar

SETTING UP HADOOP ENVIRONMENT VARIABLES

—   You can set Hadoop environment variables by appending the following commands to ~/.bashrc file.

—        export JAVA_HOME=/usr/lib/jvm/java-7-oracle

—        export HADOOP_HOME=/usr/local/hadoop

—        export HADOOP_MAPRED_HOME=$HADOOP_HOME

—        export HADOOP_COMMON_HOME=$HADOOP_HOME

—        export HADOOP_HDFS_HOME=$HADOOP_HOME

—        export YARN_HOME=$HADOOP_HOME

—        export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

—        export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

—        export HADOOP_INSTALL=$HADOOP_HOME

—  Now apply all the changes into the current running system.

            $ source ~/.bashrc

HADOOP CONFIGURATION

—        Next we need to configure some of the Hadoop files, namely:

                                    —        hadoop-env.sh

                                    —        core-site.xml

                                    —        hdfs-site.xml

                                    —        mapred-site.xml

These files are located in $HADOOP_HOME/etc/hadoop

—        hadoop-env.sh

                        —        In this file, add the following line to define the Java home

                                                export JAVA_HOME=/usr/lib/jvm/java-7-oracle

—        mapred-site.xml

                        —        This file may not be present with the same name. In that case we need to first copy this file from the template file

                        —        cp mapred-site.xml.template mapred-site.xml

                        —        Then add the following property within the configuration tabs

            <property>

             <name>mapred.job.tracker</name>

            <value>localhost:54311</value>

            <description>The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task.

             </description>

            </property>

—        core-site.xml

            —        Add the following property in the configuration tabs

                        <property>

                          <name>fs.default.name</name>

                        <value>hdfs://localhost:54310</value>

<description>The name of the default file system. A URI whose scheme and  authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class.The uri's authority is used to determine the host, port, etc. for a filesystem.</description>

                        </property>

—        hdfs-site.xml

                        —        We need to create a couple of directories that would be used by the namenode                              and the datanode in the Hadoop cluster.

                        —        $ sudo mkdir -p /usr/local/hadoop_store/hdfs/namenode

                        —        $ sudo mkdir -p /usr/local/hadoop_store/hdfs/datanode

                                    $ sudo chown -R hdpuser:hadoop /usr/local/hadoop_store

—        Next we add the following properties within the configuration tabs

                        <property>

                         <name>dfs.replication</name>

                         <value>1</value>

                         <description>Default block replication.

                         The actual number of replications can be specified when the file is created.                                    The default is used if replication is not specified in create time.

                         </description>

                        </property>

                        <property>

                          <name>dfs.namenode.name.dir</name>

                          <value>file:/usr/local/hadoop_store/hdfs/namenode</value>

                        </property>

                        <property>

                         <name>dfs.datanode.data.dir</name>

                         <value>file:/usr/local/hadoop_store/hdfs/datanode</value>

                        </property>

FORMATTING THE NAMENODE

—        Once the Hadoop configuration is over, we need to format the Namenode.

            —        The Hadoop system can be formatted by the following command:

                        —        hadoop namenode –format

            —        The Namenode should be successfully formatted before proceeding further.

START THE HADOOP DAEMONS

—        Next we need to start the Hadoop Daemons which run as individual Java services.

—        Hadoop provides a set of scripts to start and stop the Daemons.

—        To start the DFS Daemons, issue the following command in the terminal:

                        —        start-dfs.sh

—        To start the Yarn Daemons, issue the following command in the terminal:

                        —        start-yarn.sh

VERIFYING HADOOP INSTALLATION

—        Hadoop installation can be verified by checking if all the Daemons are running successfully.

—        Since all the Daemons are Java processes, issue the following command on the terminal:

                        —        $ jps

—        It should list the following processes:

                        —        Namenode

                        —        SecondaryNamenode

                        —        Datanode

                        —        NodeManager

                        —        ResourceManager

HADOOP WEB INTERFACES

—        Hadoop Namenode and ResourceManager can be monitored using the web interfaces.

—        Usually used by Hadoop Administrators.

                        —        For NameNode:

                                    —        http://localhost:50070

                        —        For ResourceManger:

                                    —        http://localhost:8088

                        —        For Secondary NameNode:

                                    —        http://localhost:50090

                        —        For DataNode:

                                    —        http://localhost:50075

Linux Commands(Continues)

Linux Commands

SSH

ssh user@host – connect to host as user

ssh -p port user@host – connect to host on port

port as user

ssh-copy-id user@host – add your key to host for

user to enable a keyed or passwordless login

Searching

grep pattern files – search for pattern in files

grep -r pattern dir – search recursively for

pattern in dir

command | grep pattern – search for pattern in the

output of command

locate file – find all instances of file

System Info

date – show the current date and time
cal – show this month's calendar
uptime – show current uptime
w – display who is online
whoami – who you are logged in as
finger user – display information about user
uname -a – show kernel information
cat /proc/cpuinfo – cpu information
cat /proc/meminfo – memory information
man command – show the manual for command
df – show disk usage
du – show directory space usage
free – show memory and swap usage
whereis app – show possible locations of app
which app – show which app will be run by default

Compression

tar cf file.tar files – create a tar named
file.tar containing files
tar xf file.tar – extract the files from file.tar
tar czf file.tar.gz files – create a tar with
Gzip compression
tar xzf file.tar.gz – extract a tar using Gzip
tar cjf file.tar.bz2 – create a tar with Bzip2
compression
tar xjf file.tar.bz2 – extract a tar using Bzip2
gzip file – compresses file and renames it to
file.gz
gzip -d file.gz – decompresses file.gz back to
file

Network

ping host – ping host and output results
whois domain – get whois information for domain
dig domain – get DNS information for domain
dig -x host – reverse lookup host
wget file – download file
wget -c file – continue a stopped download\

Installation

Install from source:
./configure
make
make install
dpkg -i pkg.deb – install a package (Debian)
rpm -Uvh pkg.rpm – install a package (RPM)

Shortcuts in Linux

Ctrl+C – halts the current command
Ctrl+Z – stops the current command, resume with
fg in the foreground or bg in the background
Ctrl+D – log out of current session, similar to exit
Ctrl+W – erases one word in the current line
Ctrl+U – erases the whole line
Ctrl+R – type to bring up a recent command
!! - repeats the last command
exit – log out of current session

* use with extreme caution.

Linux Commands

Linux Commands

File Commands

ls – directory listing

ls -al – formatted listing with hidden files

cd dir - change directory to dir

cd – change to home

pwd – show current directory

mkdir dir – create a directory dir

rm file – delete file

rm -r dir – delete directory dir

rm -f file – force remove file

rm -rf dir – force remove directory dir *

cp file1 file2 – copy file1 to file2

cp -r dir1 dir2 – copy dir1 to dir2; create dir2 if it

doesn't exist

mv file1 file2 – rename or move file1 to file2

if file2 is an existing directory, moves file1 into

directory file2

ln -s file link – create symbolic link link to file

touch file – create or update file

cat > file – places standard input into file

more file – output the contents of file

head file – output the first 10 lines of file

tail file – output the last 10 lines of file

tail -f file – output the contents of file as it

grows, starting with the last 10 lines 

Process Management

ps – display your currently active processes

top – display all running processes

kill pid – kill process id pid

killall proc – kill all processes named proc *

bg – lists stopped or background jobs; resume a

stopped job in the background

fg – brings the most recent job to foreground

fg n – brings job n to the foreground 

File Permissions

chmod octal file – change the permissions of file

to octal, which can be found separately for user,

group, and world by adding:

● 4 – read (r)

● 2 – write (w)

● 1 – execute (x)

Examples:

chmod 777 – read, write, execute for all

chmod 755 – rwx for owner, rx for group and world

For more options, see man chmod.