rdotwoolley

Ops, Entrepreneurism, Tools, and Thoughts.

Graylog2 and Dreamy Ocelots

| Comments

Logging! A repeat topic here these days. I have found myself in a position where there are multiple instances of our product running and logging the snot out of everything. Point of interest: we cannot virtualize our product at this point due do an OpenGL dependency and the need for some serious GPU power. So, back to the main topic, how do you deal with logs all over the place? Logstash and Graylog2 (though @lusis is doing some crazy stuff with logstash and 0mq) are your best bet. For our testing we did a graylog2 implementatation on Ubuntu \11.10 and logstash running on RHEL 5.5 piping the logs to our graylog2 instance. I want to cover our setup (there are some learnings that are worth noting as most people are ripping Ubuntu 11.04) and I am leaning towards setting up a chef cookbook soon.

Graylog2 Server

  1. Setup MongoDB So the mongo in the Ubuntu Repo is not the new hawtness. You’re best off to follow these steps to and get the latest
1
2
3
4
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
sudo echo -e "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen\n" >> /etc/apt/sources.list
sudo apt-get update
sudo apt-get install mongodb-10gen
  1. Install the recommended JRE
1
sudo apt-get install openjdk-6-jre
  1. Grab and Install the Graylog2 Server (0.9.6)
1
2
3
4
cd /opt/graylog2
sudo wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz
sudo tar -xvf graylog2-server-0.9.6.tar.gz
sudo cp /opt/graylog2/graylog2-server-0.9.6/graylog2.conf.example /etc/graylog2.conf
  1. Add the Graylog2 user to Mongo
1
2
3
4
mongo
use graylog2
db.addUser("graylog2","password")
exit
  1. Edit the Graylog2 config file
1
2
3
4
5
6
7
sudo vim /etc/graylog2.conf

# Match These Settings
mongodb_useauth=truemongodb_user=graylog2
mongodb_password=passwordmongodb_host=127.0.0.1
mongodb_database=graylog2 
mongodb_port=27017
  1. Kickoff the Graylog2 server and see how you did
1
2
3
sudo ln -s /opt/graylog2/graylog2-server-0.9.6 /opt/graylog2/graylog2-server
sudo ./graylog2ctl start
Check logs

Elastic Search Installation

  1. One Shot Install
1
2
3
sudo wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.tar.gz
sudo tar -xvf elasticsearch-0.18.7.tar.gz
sudo elasticserach/bin/elasticearch #manual start

More Details

Graylog2 WebUI

A few points here: * For testing purposes we’re just running the webui out of rails. I’m not sure I’d do this in production * RVM wasn’t installed. We don’t do RVM in product environments. Ocelet’s Ruby is good enough to roll into production * I ran into issues with the date formats in the gemspec files. Just hack them A few points here: * For testing purposes we’re just running the webui out of rails. I’m not sure I’d do this in production * RVM wasn’t installed. We don’t do RVM in product environments. Ocelet’s Ruby is good enough to roll into production * I ran into issues with the date formats in the gemspec files. Just hack them

  1. Install Ruby Ocelot comes with 1.8.7 standard so there is nothing to do here. RVM is definitely an option if you want.
  2. Get the Graylog2 WebUI and ramp up rails
1
2
3
4
5
6
7
8
cd /opt/graylog2/
sudo wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
sudo tar -xvf graylog2-web-interface-0.9.6.tar.gz
sudo ln -sf graylog2-web-interface-0.9.6 graylog2-web-interface
sudo apt-get install rubygems
sudo gem install bundler
sudo bundle install
sudo apt-get install rails
  1. Config and Launch
  2. Edit the mongoid.yml to match your config. I suggest commenting out the test and development sections.
  3. Launch the WebUI from the cmdline:
1
/opt/graylog2/graylog2-web-interface/scripts/rails server -e production

Annnnd you’re done

At this point you should be read to rip logstash and start pumping all the logs into the server. In a later post I will announce a sweet cookbook for this AND how to setup your logstash on the app server. Also, I’m assessing how graylog2 can assist in compliance situations (HIPAA, SOX, ISO, etc) so stay tuned for my thoughts and findings.

Comments