What is Hadoop

What is Hadoop?
Before entering into Hadoop, we have to understand the issues related to Big Data and traditional processing system. In the previous blog, we already discussed about Big Data in detail. Hadoop is an open-source batch processing framework developed in Java, used to store and analyze the large sets of data. It is being used by Google, Yahoo, Facebook, Twitter, LinkedIn and etc.

Components of Hadoop

  •  HDFS - Hadoop Distributed File System, Used to store huge amount of dataset across the cluster.
  •  YARN - Yet Another Resource Negotiator, Used for managing the cluster.
  •  MapReduce - It's a software framework to process huge data in parallel on a cluster.

Features of Hadoop 

  •   Cluster storage - Hadoop will split the single data set into multiple and stored across cluster (more than one storage system working together) with replication (default 3). It's used to increase performance and reliability.
  •   Distributed computing - A single problem divides into multiple sub-problem and each sub-problem solve by different computer. Each computer will communicate internally to avoid duplication. If all sub-problem done the solution will come as single. 
  •   Commodity hardware - Commodity hardware is nothing but cheap hardware, which is inexpensive and more availability. We can use our traditional systems on the cluster. No need to buy expensive system.
  •   Parallel processing - A single task will split into many and every task will run separate CPU to reduce program running time.
  •   Low latency - Process huge data sets with less time.
  •   Data availability - Data will store across cluster with replication. If any machine on the cluster will down the specific machine data will available on another machine in the cluster.
  •   Fault-tolerance - The system will work as usual without any data loss, even if some system has failed. This is one of the main advantages of Hadoop.
  •   Horizontal scalability - We can add or remove server (system) on the cluster without interrupting  existing servers.

Hadoop Daemons

Hadoop Daemons
Daemons

  • NameNode - It is the master node used to store the meta information of all the files, such as no of Blocks, location, replicas and etc.. It will manage the Slave node.
  • Secondary NameNode - It's called as Checkpoint Node. It is backup node for NameNode. It gee FsImage and EditLogs from the NameNode and merges EditLogs with the Fsimage regularly.
  • DataNode - It is the slave node used to store the actual data in HDFS. It's used to perform read and write operation as per request.
  • ResourceManager - It's runs on the master node. This is one for each cluster.
  • NodeManager - It's runs on the each slave node.



<< Previous                                                                                                                                 Next >>

Comments

Post a Comment

Popular posts from this blog

HDFS Commands Part - I

Image
Prequirement Before start Hadoop shell have to install Hadoop . File System Shell Most of the commands in FS shell is like corresponding Linux commands. The FileSystem (FS) shell is invoked by bin/hadoop fs <args> . All the FS shell commands take path URIs as arguments. For HDFS the scheme is hdfs , and for the local filesystem the scheme is file . The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://<namenodehost>/dir_1/dir_2 or simply as /dir_1/dir_2 ( given that your configuration is set to point to hdfs://<namenodehost> ). Error information is sent to stderr and the output is sent to stdout . Basic Commands 1. version   This HDFS command prints the Hadoop version.        Example: hdfs dfs version 2. cat  This HDFS command used to displays the contents of the filename on stdout/conso
Read more

HDFS Commands Part - II

Image
In part - I session we learned about HDFS basic commands, in this session will see the intermediate level commands. Before read this article I suggest you to learn basic hdfs commands.   Commands 1. copyFromLocal This HDFS command is similar to put command, but the source is restricted to a local file reference.      Usages: hdfs dfs -copyFromLocal <local_path> <hdfs_path>      Example: hdfs dfs -copyFromLocal /home/user/Desktop/file.orc /dir_1/ 2. copyToLocal This HDFS command will copy file/directory from HDFS to local file system.      Usages: hdfs dfs -copyToLocal <hdfs_path <local_path>      Example: hdfs dfs -copyToLocal /dir_1/file.orc /home/user/Desktop/ 3. text This HDFS command will take the source file and display the file content in text formad.      Usages: hdfs dfs -text <hdfs_file_path>      Example: hdfs dfs -text /dir_1/file 4. tail This HDFS command will displays last kilobyte of the file to
Read more

Install Hadoop On Ubuntu

Image
Prequirement Before installing Hadoop, you have to install Java . Hadoop Installation Steps Step 1: Create Separate Login          $ sudo addgroup hadoop          $ sudo adduser –ingroup hadoop hdfs user          $ sudo adduser hdfsuser sudo   Step 2: Install SSH          $ sudo apt-get update          $ sudo apt-get install ssh          $ sudo su hdfsuser          $ sudo ssh-keygen -t rsa -p ""                >> If it's asking for file name or location, leave it blank.                $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys          $ chmod 0600 ~/.ssh/authorized_keys          $ exit   Step 3: Install Hadoop on Ubuntu           $ wget http://www-us.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz          $ tar xvzf hadoop-3.0.0.tar.gz          $ sudo mkdir -p /usr/local/hadoop          $ cd hadoop-3.1.0/            $ sudo mv * /usr/local/hadoop          $ sudo chown -R hdf
Read more