Protected: openshift_test

Posted on September 4, 2013 by bennojoy

Hadoop cluster deployment in eucalyptus cloud using puppet.

Posted on November 15, 2012 by bennojoy

I would recommend to read this great article to get a basic understanding on concepts of hadoop

http://bradhedlund.com/2011/09/10/understanding-hadoop-clusters-and-the-network/

Another great article on deploying hadoop clusters.

http://hadoop.apache.org/docs/r1.0.4/single_node_setup.html

An short description on how application deployment can be automated on eucalyptus cloud using puppet.

https://bennojoy.wordpress.com/2012/11/09/automating-application-deployment-using-puppet-in-eucalyptusaws-cloud/

This article list the manifests that were used to automate deployment of a two node Hadoop cluster. One node acts as the master( name node,datanode & Jobtracker,task tracker) while the second node runs as slave (datanode,task tracker).

As the first step we create an instance in eucalyptus passing user data variable ‘role’ with value hadoopmaster which would be later read by puppet master to determine which manifests to be sent down to client.

euca-run-instances -k benkey emi-0AA43BEE -t m1.large -d “role=hadoopmaster”

A script in rc.local of the template assigns hadoopmaster as a facter in the running instance and contacts the puppet master.

curl –retry 3 –retry-delay 10 -o $TMP_FILE http://10.101.1.118:8773/latest/user-data

if [ -s $TMP_FILE ]; then

echo “Downloaded user data in $TMP_FILE”

if [ “`head -c 2 $TMP_FILE`” = “#!” ]; then

chmod a+x $TMP_FILE

echo “User data is a script: executing it”

sh $TMP_FILE

ROLE=`cat $TMP_FILE | grep role | cut -d= -f2 | col -b`

cat > /usr/lib/ruby/site_ruby/1.8/facter/bentest.rb <<EOF

Facter.add(:role) do

setcode do

role = ‘$ROLE’

end

EOF

Once the puppet master is contacted by the puppet agent in the running instance, the puppet server check for the faster value in role and send back appropriate configurations.

[root@e-20 modules]# cat /etc/puppet/modules/vmnodes/manifests/classifiers.pp

class vmnodes::classifiers{

case $role {

“dbserver” : {

include “mysql”

}

case $role {

“appserver” : {

include “httpd”

}

case $role {

“hadoopmaster” : {

include “hadoopmaster”

}

case $role {

“hadoopslave” : {

include “hadoopslave”

}

Here’s the manifest that would be sent down to the hadoopmaster node.

[root@e-20 modules]# cat /etc/puppet/modules/hadoopmaster/manifests/init.pp

class hadoopmaster{

#install the JDK package

package { “java-1.6.0-openjdk-devel”:

ensure => “present”,

}

#install the hadoop rpm

package { ‘hadoop’:

ensure => installed,

provider => ‘rpm’,

source => ‘http://apache.techartifact.com/mirror/hadoop/common/stable/hadoop-1.0.4-1.x86_64.rpm’,

require => Package[“java-1.6.0-openjdk-devel”],

}

#The env file for hadoop where appropriate java_home is configured.

file { “/etc/hadoop/hadoop-env.sh”:

source => “puppet:///hadoopmaster/hadoop-env.sh”,

require => Package[“hadoop”],

mode => “0755”,

}

#the configured core-site file, (should be tweaked according to you site needs,) the below files are hadoop configuration files and the contents depends based on the deployment.

file { “/etc/hadoop/core-site.xml”:

source => “puppet:///hadoopmaster/core-site.xml”,

require => Package[“hadoop”],

mode => “0755”,

}

file { “/etc/hadoop/hdfs-site.xml”:

source => “puppet:///hadoopmaster/hdfs-site.xml”,

require => Package[“hadoop”],

mode => “0755”,

}

file { “/etc/hadoop/mapred-site.xml”:

source => “puppet:///hadoopmaster/mapred-site.xml”,

require => Package[“hadoop”],

mode => “0755”,

}

#Contains the hostnane of the node which would act as the master.

file { “/etc/hadoop/masters”:

source => “puppet:///hadoopmaster/masters”,

require => Package[“hadoop”],

mode => “0755”,

}

#contains the hostnames for the slaves.

file { “/etc/hadoop/slaves”:

source => “puppet:///hadoopmaster/slaves”,

require => Package[“hadoop”],

mode => “0755”,

}

#Public key for pas wordless access to slaves by the master node.

file { “/root/.ssh/id_dsa”:

source => “puppet:///hadoopmaster/id_dsa”,

mode => “0600”,

}

file { “/root/.ssh/id_dsa.pub”:

source => “puppet:///hadoopmaster/id_dsa.pub”,

mode => “0644”,

}

file { “/root/.ssh/authorized_keys”:

source => “puppet:///hadoopmaster/authorized_keys”,

mode => “0400”,

}

#make the hostnane of the master server as master and the ip address would be private ip, we can’t make the public ip here as the java process’s wouldnt be able to bind to that ip. on the slave nodes this would point to public ip of the master node.(use elastic IP’s here).

host { “master”:

ip => “$ipaddress”,

name => “master”,

}

#host entry for slave, use elastic ip here.

host { “slave”:

ip => “10.101.5.100”,

name => “slave”,

}

#formats the hadfs filesytem.

exec { “hadoop namenode -format”:

creates => “/tmp/hadoop-root/dfs”,

path => [“/bin”, “/usr/bin”, “/usr/sbin”],

require => Package[“hadoop”],

}

#the rpm doesn’t set the execute permission on the startup scripts so we do it using puppet.

exec { “chmod 755 /usr/sbin/*.sh”:

path => [“/bin”, “/usr/sbin”],

require => Package[“hadoop”],

}

The manifest for the hadoopslave is as follows, only difference in host entries for master and slave , master get the elastic ip instead of it’s private ip.

[root@e-20 modules]# cat /etc/puppet/modules/hadoopslave/manifests/init.pp

class hadoopslave{

package { “java-1.6.0-openjdk-devel”:

ensure => “present”,

}

package { ‘hadoop’:

ensure => installed,

provider => ‘rpm’,

source => ‘http://apache.techartifact.com/mirror/hadoop/common/stable/hadoop-1.0.4-1.x86_64.rpm’,

require => Package[“java-1.6.0-openjdk-devel”],

}

file { “/etc/hadoop/hadoop-env.sh”:

source => “puppet:///hadoopslave/hadoop-env.sh”,