Getting status out of JBoss

Recent question arose on how to monitor JBoss from the load balancer. Since we can't use JMX or the jmx-console from the load balancer (our usual method) we needed some HTTP endpoints that were easy to use out of the box (i.e. we didn't want to write our own servlet/jsp page). Two good candidates were:

http://hostname/jboss/status

and (if you have are using web services)

http://hostname/jbossws/services

Ok, thats it for now!

Book review - Pulling Strings with Puppet; configuration mgmt made easy

Puppet is an amazing tool for keeping everything on a cluster in sync. I/we use it for apache cassandra, hadoop and our own internal software distribution. 
Its a short book but given the light level of documentation around the puppet opensource project I found this book useful to get you started automating machine administration. However, I realize now that I am using puppet more and more what holds it back is that its such a thin book it lacks complete examples. This is really a critical flaw. Maybe if it came with source code on a website this hole could be plugged. In the end I got a lot out of online tutorials and then used this book for reference/reminders. Eventually I've mostly moved past this book and I use the reference guides on the reductivelabs/puppetlabs website: 
http://docs.puppetlabs.com/references/stable/configuration.html
http://docs.puppetlabs.com/references/latest/type.html#package

Another annoyance is the lack of an index so I recommend the ebook.
Final comments: 
- Puppet is based on Ruby, so a light understanding of Ruby does help (especially if you need to patch puppet).
- I did have to patch my redhat 4 box's version of puppet that i got from epel yum repo since it was failing on templates and causing a SEGV in ruby. See http://projects.reductivelabs.com/issues/2604

Book review - Hadoop: The Definitive Guide by Tom White

I really enjoyed the book "Hadoop: The Definitive Guide by Tom White". 
It has everything you need to: 
a) Get started running your own cluster and writing your own MR jobs 
b) Understand how to administer the cluster 
c) Troubleshoot your programs 
d) Learn about really important side projects like Pig, Hive, Zookeeper and HBase (of which I think Hive is the most amazing) 

One thing I wish I'd done is go through the cloudera online tutorials BEFORE reading this book. If I'd done that (instead of doing so afterwards) I think I'd have got through certain sections of the book much quicker; basically I would have 'got it' quicker. See http://www.cloudera.com/resources/?type=Training


After reading the book I organized a little geek meet where I covered a synopsis of Hadoop, Pig and Hive with the development team. I also introduced them to the Cloudera training virtual machine. That is just an amazing resource for learning hadoop et al. It also introduced me to some unique cool things like the sqoop program (http://www.cloudera.com/developers/downloads/sqoop/) for reading tables out of an RDBMS like MySQL or Oracle and auto populating Hadoop and/or Hive...very useful!

Enabling ssh-agent for password-less ssh login on KDE/Gnome

So one of the things that had been bothering me was ssh'ing into remote machines with keys that had passwords. I wanted to use ssh-agent so that I would not have to type in my password. Trouble was that I couldn't figure out how to do it on my KDE desktop so that every time I opened a new shell the ssh-agent would be active. Everything I'd previously read talked about executing the command:

  ssh-agent bash

...but this only starts the agent for the shell started and any child processes of that shell. Consequently, every shell opened has its own ssh-agent and you have to do a ssh-add on each shell, typing in your password each time.

Well here is how to do it

start-ssh-agent script

#!/bin/bash
if [ -f ~/.ssh/ssh-agent.env ]; then
  #echo "Agent already started"
  i=1
  #I just needed something above so the then was a valid statement
  #...is there a noop in bash?
else
  ssh-agent > ~/.ssh/ssh-agent.env
  #we need to delete the echo from the source script since some
  #commands like scp and ssh hate it when .cshrc echos stuff out
  sed -e '/echo/d' ~/.ssh/ssh-agent.env > ~/.ssh/ssh-agent2.env
  mv ~/.ssh/ssh-agent2.env ~/.ssh/ssh-agent.env
  . ~/.ssh/ssh-agent.env
  #echo "Agent started"
  ssh-add
fi

Basically this script executes ssh-agent, captures the output that specifies the environment variables and writes them to a file for future reference from future shells. It then executes ssh-add to prompt you to enter the passwords for the private keys.


stop-ssh-agent script

#!/bin/bash
if [ -f ~/.ssh/ssh-agent.env ]; then
  . ~/.ssh/ssh-agent.env > /dev/null
  kill $SSH_AGENT_PID
  rm ~/.ssh/ssh-agent.env
  echo "Agent stopped"
else
  echo "Agent is not running"
fi

Then in ~/.bashrc file you add the following:

if [ -f ~/.ssh/ssh-agent.env ]; then
        . ~/.ssh/ssh-agent.env
else
        ~/bin/start-ssh-agent
fi

...this basically means...
if the ssh-agent.env file exists
  source it so that the environment vars point to the ssh-agent process running.
else
  run the script to start the ssh-agent and prompt for the passwords for any keys

This is not perfect and you need to be careful if you are doing agent forwarding into the box but for most general cases this works.

Good and bad: NUFC promotion to Premiership

Over the past year my Newcastle United RSS feed has been very different to the year before. We've only lost 4 times away and the results have often been 3-0, 4-1, etc Its been a joy to read the news. However, that good feeling that I get on a Monday morning is going to change with promotion back into the Premiership. Honestly, I've got mixed feelings about promotion now that I am so used to hearing good news and the most we can seem to hope for is solid middle table performance in the Premier League.