Big Data

Big data

 

What is Data?

Data is an unprocessed or raw format information. It can be any character,  text, numbers, images, audio, or video.

What is Big Data?

Big data is a term that describes the large amount of data. It’s used to process huge and complex data. Data that is structured, unstructured, semi-structured and very large cannot be processed by relational database engines on given time. Moreover, The data will be growing exponentially based on time. This type of data called "big data".

Categories Of Big Data

  1. Structured

    Structured data refers to any data that fixed fields and records for example RDBMS and csv files.

    Structured data
    Example of Structured data

  2. Unstructured

    Unstructured data refers to any data that does not have predefined format, for example machine generated logs and web page. 

    UnStructured data
    Example of Unstructured data

  3. Semi-Structured

    Semi-Structured data refers to any data that would be a raw data or typed data in a conventional database system for example JSON, XML.

    Semi-Structured data
    Example of Semi-Structured data
     

  4V's of Big Data

  1. Volume 

    Earlier human created data, Today data generated by machines, networks and social media. Technologies have capable of store this, more data. 

  2. Velocity

    Data is coming at extremely high speed and it will be real-time. Velocity also describes how fast data are processed and analyzed.

  3. Variety

    Data comes in different formats. Like numeric, email, video, audio, stock ticker data and financial transactions. 

  4. Veracity

    Basically the quality of data, not all data is always good. That’s why the system should be able to extract good data instead of saving both the good and bad data.

 Why Big Data?

  1. Monitoring -  To watch daily activities and prevent the Vulnerabilities. 
  2. Alert - To send the alert. If any vulnerability or data theft occurred.
  3. Fraud Detection - To find crime on our organization and etc..

  Who Uses Big Data?

  1. Education - To find the students and staff's progress.
  2. Health Care -In analyzing hospital activity and protect patients Identity.
  3. Banking -To detect fraud and Security issues.
  4. Retail and Manufacturing - To analyze sales growth, etc.. 


                                                                                    
                                                                                                                                                                                 Next >>

Comments

Post a Comment

Popular posts from this blog

HDFS Commands Part - I

Image
Prequirement Before start Hadoop shell have to install Hadoop . File System Shell Most of the commands in FS shell is like corresponding Linux commands. The FileSystem (FS) shell is invoked by bin/hadoop fs <args> . All the FS shell commands take path URIs as arguments. For HDFS the scheme is hdfs , and for the local filesystem the scheme is file . The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://<namenodehost>/dir_1/dir_2 or simply as /dir_1/dir_2 ( given that your configuration is set to point to hdfs://<namenodehost> ). Error information is sent to stderr and the output is sent to stdout . Basic Commands 1. version   This HDFS command prints the Hadoop version.        Example: hdfs dfs version 2. cat  This HDFS command used to displays the contents of the filename on stdout/conso
Read more

HDFS Commands Part - II

Image
In part - I session we learned about HDFS basic commands, in this session will see the intermediate level commands. Before read this article I suggest you to learn basic hdfs commands.   Commands 1. copyFromLocal This HDFS command is similar to put command, but the source is restricted to a local file reference.      Usages: hdfs dfs -copyFromLocal <local_path> <hdfs_path>      Example: hdfs dfs -copyFromLocal /home/user/Desktop/file.orc /dir_1/ 2. copyToLocal This HDFS command will copy file/directory from HDFS to local file system.      Usages: hdfs dfs -copyToLocal <hdfs_path <local_path>      Example: hdfs dfs -copyToLocal /dir_1/file.orc /home/user/Desktop/ 3. text This HDFS command will take the source file and display the file content in text formad.      Usages: hdfs dfs -text <hdfs_file_path>      Example: hdfs dfs -text /dir_1/file 4. tail This HDFS command will displays last kilobyte of the file to
Read more

Install Hadoop On Ubuntu

Image
Prequirement Before installing Hadoop, you have to install Java . Hadoop Installation Steps Step 1: Create Separate Login          $ sudo addgroup hadoop          $ sudo adduser –ingroup hadoop hdfs user          $ sudo adduser hdfsuser sudo   Step 2: Install SSH          $ sudo apt-get update          $ sudo apt-get install ssh          $ sudo su hdfsuser          $ sudo ssh-keygen -t rsa -p ""                >> If it's asking for file name or location, leave it blank.                $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys          $ chmod 0600 ~/.ssh/authorized_keys          $ exit   Step 3: Install Hadoop on Ubuntu           $ wget http://www-us.apache.org/dist/hadoop/common/hadoop-3.1.0/hadoop-3.1.0.tar.gz          $ tar xvzf hadoop-3.0.0.tar.gz          $ sudo mkdir -p /usr/local/hadoop          $ cd hadoop-3.1.0/            $ sudo mv * /usr/local/hadoop          $ sudo chown -R hdf
Read more