Cloudera hadoop cluster demo VM

July 11, 2016 by S4

Filed under Hadoop

Last modified July 15, 2016

Cloudera Hadoop cluster demo VM

This document will hep java developers to kick start how to run hadoop cluster VM. To understand this document you need to possess basic theoretical knowledge on Hadoop, hdfs and map reduce jobs. It is also advisable to have some prior knowledge on basic linux commands.

It is possible to try sample map reduce jobs on your windows PC without any cumbersome hadoop setup if you have cloudera demo VM with you. This environment is ideal for functionality testing on very small sample data volumes, larger input data will not be supported due to memory constrains posed by the cloudera VM.

To test your sample map reduce job on local hadoop environment (cloudera Demo VM) follow the below mentioned steps in order.

Download the following software and cloudera image on your windows pc.

  • Cloudera VM

http://www.cloudera.com/downloads/quickstart_vms/5-7.html

  • VM Ware Player

User credential for cloudera VM

Username :- cloudera

Password:- cloudera

 

Login through Linux console and here is some basic command that you can run and test VM

Command Syntax:

hadoop fs -mkdir  <full path of directory in hdfs>

Example

hadoop fs -mkdir  /user1

  1. Copy the contents from the linux box input folder to hdfs input folder

Command Syntax:

hadoop fs –copyFromLocal  <source directory from local linux box>  <destination directory in hdfs>

Example

hadoop fs -copyFromLocal  /var/log/message  /user1

  1. Check the availability of the input files on HDFS(not a mandatory step)

Command Syntax:

hadoop fs –ls  <full path of hdfs directory>

Example

hadoop fs -ls  /user1

Leave a Comment