What is HDFS

Hadoop HDFS

What is HDFS

HDFS (Hadoop Distributed File System) is a file system like our normal desktop/laptop file system which is used to store the data. It's specially designed for storing huge datasets with cluster of commodity hardware and with streaming access pattern.   The data may be text file, image, audio, video, etc...

Streaming access pattern

Streaming access pattern means write once read many number of time but don't change content of the file is called as streaming access pattern.

Operations in HDFS

  1. Write Operation
  2. Read Operation 

Write Operation

Assume that you are writing file into HDFS. Your write request will go NameNode (NN) Distributed File System (DFS). The DFS will make RPC call to the namenode for create new file.

Before creating file the namenode will do couple of things. It will check whether the file is not exist and user has permission to create new file. Once all the check is done successfully the namenode will provide available DataNode (DN).

Now, the datanode will start write file and it will take care of splitting the file into blocks and replication of each block based on configuration. The datanode will send success acknowledge after completion of replication of blocks.

Read Operation

Assume that you are trying to read file from HDFS. Now the request will go to namenode DFS. The namenode will check whether the file is exist and use has permission to read this specific file. If all check done successfully it will give the file meta information like in which datanode has requested file and block and replica detail. based on that datanode will collect the block of data and return.

What next?

Interact with HDFS using Command Line Interface (CLI).


Post your queries in comment section :)

 

Location: Chennai, Tamil Nadu, India

Comments

Post a Comment

Popular posts from this blog

HDFS Commands Part - I

Image
Prequirement Before start Hadoop shell have to install Hadoop . File System Shell Most of the commands in FS shell is like corresponding Linux commands. The FileSystem (FS) shell is invoked by bin/hadoop fs <args> . All the FS shell commands take path URIs as arguments. For HDFS the scheme is hdfs , and for the local filesystem the scheme is file . The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://<namenodehost>/dir_1/dir_2 or simply as /dir_1/dir_2 ( given that your configuration is set to point to hdfs://<namenodehost> ). Error information is sent to stderr and the output is sent to stdout . Basic Commands 1. version   This HDFS command prints the Hadoop version.        Example: hdfs dfs version 2. cat  This HDFS command used to displays the contents of the filename on stdout/conso
Read more

HDFS Commands Part - II

Image
In part - I session we learned about HDFS basic commands, in this session will see the intermediate level commands. Before read this article I suggest you to learn basic hdfs commands.   Commands 1. copyFromLocal This HDFS command is similar to put command, but the source is restricted to a local file reference.      Usages: hdfs dfs -copyFromLocal <local_path> <hdfs_path>      Example: hdfs dfs -copyFromLocal /home/user/Desktop/file.orc /dir_1/ 2. copyToLocal This HDFS command will copy file/directory from HDFS to local file system.      Usages: hdfs dfs -copyToLocal <hdfs_path <local_path>      Example: hdfs dfs -copyToLocal /dir_1/file.orc /home/user/Desktop/ 3. text This HDFS command will take the source file and display the file content in text formad.      Usages: hdfs dfs -text <hdfs_file_path>      Example: hdfs dfs -text /dir_1/file 4. tail This HDFS command will displays last kilobyte of the file to
Read more

Install Hadoop On Ubuntu

Image
Prequirement Before installing Hadoop, you have to install Java . Hadoop Installation Steps Step 1: Create Separate Login          $ sudo addgroup hadoop          $ sudo adduser –ingroup hadoop hdfs user          $ sudo adduser hdfsuser sudo   Step 2: Install SSH          $ sudo apt-get update          $ sudo apt-get install ssh          $ sudo su hdfsuser          $ sudo ssh-keygen -t rsa -p ""                >> If it's asking for file name or location, leave it blank.                $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys          $ chmod 0600 ~/.ssh/authorized_keys          $ exit   Step 3: Install Hadoop on Ubuntu           $ wget http://www-us.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz          $ tar xvzf hadoop-3.0.0.tar.gz          $ sudo mkdir -p /usr/local/hadoop          $ cd hadoop-3.1.0/            $ sudo mv * /usr/local/hadoop          $ sudo chown -R hdf
Read more