Apache Hadoop MapReduce

MapReduce


Before drive into this tutorial, I suggest you to read what is Bigdata, Hadoop, HDFS and YARN. If you not aware of those topics.


What is MapReduce

MapReduce is an Apache framework used to process large amount of data in parallel which is presented on hadoop cluster. It's will do the job as divide and conquer manner.

There are two key components in Hadoop MapReduce.

Component

  1. Mapper and
  2. Reducer

Mapper

It takes input from input split and process the each input split. The result of processed input split will be collection of key, value pairs. The result will be persist on the local disk. The number of mapper will be desired based on input split. How many number of inputsplit is there, those many number of mapper will be run.

Reducer

It takes Mapper output as a input and process that intermediate result (collection of key, value pairs) and combine those key, value pairs and create a smaller set of collection (Key, value pairs). The final output will be produced by reducer only. We can configure the number of reducers. The number of output file will be create based on reducer configuration. How many number of reducer is there those many number of output file will be generated.

Apache Hadoop MapReduce Workflow

Inputsplit

The piece of data from input file. The piece is segregate by anything like splitting by space, comma, semicolon, new line, tab space and etc..

Mapping

We already learned in above component topic.

Shuffling

This is the phase where trying to combine all values associated to single identical key. How many number of key is there in mapping phase those many number of shuffling will happen.

Reducing

We already learned in above component topic.

Final output

The final output will be collection of key, value pairs. It will be persist in hdfs storage. 
 

What next?

In next post we will see about Apache Hive.

Post your queries in comment section :)

Comments

Post a Comment

Popular posts from this blog

HDFS Commands Part - I

Image
Prequirement Before start Hadoop shell have to install Hadoop . File System Shell Most of the commands in FS shell is like corresponding Linux commands. The FileSystem (FS) shell is invoked by bin/hadoop fs <args> . All the FS shell commands take path URIs as arguments. For HDFS the scheme is hdfs , and for the local filesystem the scheme is file . The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://<namenodehost>/dir_1/dir_2 or simply as /dir_1/dir_2 ( given that your configuration is set to point to hdfs://<namenodehost> ). Error information is sent to stderr and the output is sent to stdout . Basic Commands 1. version   This HDFS command prints the Hadoop version.        Example: hdfs dfs version 2. cat  This HDFS command used to displays the contents of the filename on stdout/conso
Read more

HDFS Commands Part - II

Image
In part - I session we learned about HDFS basic commands, in this session will see the intermediate level commands. Before read this article I suggest you to learn basic hdfs commands.   Commands 1. copyFromLocal This HDFS command is similar to put command, but the source is restricted to a local file reference.      Usages: hdfs dfs -copyFromLocal <local_path> <hdfs_path>      Example: hdfs dfs -copyFromLocal /home/user/Desktop/file.orc /dir_1/ 2. copyToLocal This HDFS command will copy file/directory from HDFS to local file system.      Usages: hdfs dfs -copyToLocal <hdfs_path <local_path>      Example: hdfs dfs -copyToLocal /dir_1/file.orc /home/user/Desktop/ 3. text This HDFS command will take the source file and display the file content in text formad.      Usages: hdfs dfs -text <hdfs_file_path>      Example: hdfs dfs -text /dir_1/file 4. tail This HDFS command will displays last kilobyte of the file to
Read more

Install Hadoop On Ubuntu

Image
Prequirement Before installing Hadoop, you have to install Java . Hadoop Installation Steps Step 1: Create Separate Login          $ sudo addgroup hadoop          $ sudo adduser –ingroup hadoop hdfs user          $ sudo adduser hdfsuser sudo   Step 2: Install SSH          $ sudo apt-get update          $ sudo apt-get install ssh          $ sudo su hdfsuser          $ sudo ssh-keygen -t rsa -p ""                >> If it's asking for file name or location, leave it blank.                $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys          $ chmod 0600 ~/.ssh/authorized_keys          $ exit   Step 3: Install Hadoop on Ubuntu           $ wget http://www-us.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz          $ tar xvzf hadoop-3.0.0.tar.gz          $ sudo mkdir -p /usr/local/hadoop          $ cd hadoop-3.1.0/            $ sudo mv * /usr/local/hadoop          $ sudo chown -R hdf
Read more