HDFS Commands Part - II

HDFS Commands

In part - I session we learned about HDFS basic commands, in this session will see the intermediate level commands. Before read this article I suggest you to learn basic hdfs commands. 

Commands

1. copyFromLocal

This HDFS command is similar to put command, but the source is restricted to a local file reference.

     Usages: hdfs dfs -copyFromLocal <local_path> <hdfs_path>

     Example: hdfs dfs -copyFromLocal /home/user/Desktop/file.orc /dir_1/

2. copyToLocal

This HDFS command will copy file/directory from HDFS to local file system.

     Usages: hdfs dfs -copyToLocal <hdfs_path <local_path>

     Example: hdfs dfs -copyToLocal /dir_1/file.orc /home/user/Desktop/

3. text

This HDFS command will take the source file and display the file content in text formad.

     Usages: hdfs dfs -text <hdfs_file_path>

     Example: hdfs dfs -text /dir_1/file

4. tail

This HDFS command will displays last kilobyte of the file to stdout.

     Usages: hdfs dfs -tail <hdfs_file_path>

     Example: hdfs dfs -tail /dir_1/file

5. find

This HDFS command will finds all files that match the specified expression and applies selected actions to them. If no path is specified then defaults to the current working directory. If no expression is specified then defaults to -print.

     Usages: hdfs dfs -find <hdfs_path> <expression>

     Example: hdfs dfs -find -name test -print

6. checksum

This HDFS command will return checksum information.

     Usages: hdfs dfs -checksum <file_path>

     Example: hdfs dfs -checksum /dir_1/file.txt

7. df

This HDFS command used to check the free space.

     Usages: hdfs dfs -df -h <path>

     Example: hdfs dfs -df -h /dir_1/

8. du

This HDFS command used to displays sizes of files and directories contained in the given directory or the length of a file in case its just a file.

     Usages: hdfs dfs -du <hdfs_path>

     Example: hdfs dfs -df /dir_1/

9. expunge

This HDFS command will makes trash empty.

     Usages: hdfs dfs -expunge

     Example: hdfs dfs -expunge

10. fsck

This HDFS command will check the health of the hadoop file system.

     Usages: hdfs dfs -fsck /

     Example: hdfs dfs -fsck /

11. usage

This HDFS command will returns the help for individual command.

     Usages: hdfs dfs -usage <command>

     Example: hdfs dfs -usage <cat> 

12. help

This HDFS command will displays help for given command or all commands if not specified any.

     Usages: hdfs dfs -help

     Example: hdfs dfs -help


What next?

Apache YARN.

Post your queries in comment section :)

 

Comments

Post a Comment

Popular posts from this blog

HDFS Commands Part - I

Image
Prequirement Before start Hadoop shell have to install Hadoop . File System Shell Most of the commands in FS shell is like corresponding Linux commands. The FileSystem (FS) shell is invoked by bin/hadoop fs <args> . All the FS shell commands take path URIs as arguments. For HDFS the scheme is hdfs , and for the local filesystem the scheme is file . The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://<namenodehost>/dir_1/dir_2 or simply as /dir_1/dir_2 ( given that your configuration is set to point to hdfs://<namenodehost> ). Error information is sent to stderr and the output is sent to stdout . Basic Commands 1. version   This HDFS command prints the Hadoop version.        Example: hdfs dfs version 2. cat  This HDFS command used to displays the contents of the filename on stdout/conso
Read more

Install Hadoop On Ubuntu

Image
Prequirement Before installing Hadoop, you have to install Java . Hadoop Installation Steps Step 1: Create Separate Login          $ sudo addgroup hadoop          $ sudo adduser –ingroup hadoop hdfs user          $ sudo adduser hdfsuser sudo   Step 2: Install SSH          $ sudo apt-get update          $ sudo apt-get install ssh          $ sudo su hdfsuser          $ sudo ssh-keygen -t rsa -p ""                >> If it's asking for file name or location, leave it blank.                $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys          $ chmod 0600 ~/.ssh/authorized_keys          $ exit   Step 3: Install Hadoop on Ubuntu           $ wget http://www-us.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz          $ tar xvzf hadoop-3.0.0.tar.gz          $ sudo mkdir -p /usr/local/hadoop          $ cd hadoop-3.1.0/            $ sudo mv * /usr/local/hadoop          $ sudo chown -R hdf
Read more