Install and setup graylog2 on Ubuntu 12.04

References

• http://mrickert.com/2013/04/04/install-and-setup-graylog2-on-ubuntu-12-04/
• http://dean.io/getting-started-with-graylog2-for-logging-updated-for-096/

• Setting GrayLog2 Server
  • Up to date Ubuntu 12.04 server x64 
  • Installing mongodb
  • create the mongodb user: 
  • Installing Java
  • Installing elasticsearch
    • Check Elasticsearch service
  • Installing graylog2-server
    • Configuring service
  • Installing graylog2-web-interface
  • Installing RUBY 2.0
    • Preparing 
  • Installing Ruby 2.0
  • Check ruby is working
  • Installing bundler and others
  • Change the version of json in the Gemfile
    • Run bundle command for update json
• Installing web server daemon
• Get passenger and make sure you pull the pre version
  • Installing the passenger module (for apache)
  • Creating configuration files (for apache)
• Editing elasticsearch and other items

• Restarting services
• Access Graylog2 Web

• Gathering log from other servers(syslog clients)

Setting GrayLog2 Server

Up to date Ubuntu 12.04 server x64

apt-get update && apt-get upgrade

Installing mongodb

echo -e "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen\n" > /etc/apt/sources.list.d/mongodb-10gen.list
apt-get update
apt-get install mongodb-10gen

create the mongodb user:

mongo
use graylog2
db.addUser("grayloguser", "123")
exit

Installing Java

apt-get install openjdk-6-jdk 
ln -s /usr/lib/jvm/java-6-openjdk-amd64 java-6-openjdk
cat <<EOF > /etc/profile.d/java.sh
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-amd64
EOF

source /etc/profile.d/java.sh

Installing elasticsearch

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.20.4.deb 
dpkg -i elasticsearch- 0.20.4.deb
service elasticsearch start

Check Elasticsearch service

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

output

{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

Installing graylog2-server

mkdir -p /opt/graylog2 && cd /tmp
wget http://download.graylog2.org/graylog2-server/graylog2-server-0.11.0.tar.gz
tar -xzvf graylog2-server-0.11.0.tar.gz -C /opt/graylog2
cd /opt/graylog2
ln -sf graylog2-server-0.11.0 graylog2-server
cp graylog2-server/graylog2.conf.example /etc/graylog2.conf

You should change the value of the mongodb_password too If you changed the mongodb_user's password in mongodb console.

Configuring service

cat <<EOF > /etc/init/graylog2-server.conf
description "graylog2 server"
author  "Mick Pollard <aussielunix@gmail.com>"
modified  "DaeHyung <daehyung@gmail.com>"
start on runlevel [2345]
stop on runlevel [06]
# tell upstart we're creating a daemon
# upstart manages PID creation for you.
expect fork
script
  cd /opt/graylog2/graylog2-server
  exec sudo java -jar graylog2-server.jar > /opt/graylog2/graylog2-server/log/graylog2.log 2>&1 &
  emit graylog2-server_running
end script
EOF
touch /opt/graylog2/graylog2-server/log/graylog2.log
cd /var/log && ln -s /opt/graylog2/graylog2-server/log/graylog2.log
service graylog2-server start

Installing graylog2-web-interface

apt-get install curl
 
cd /tmp
wget http://download.graylog2.org/graylog2-web-interface/graylog2-web-interface-0.11.0.tar.gz
tar -xzvf graylog2-web-interface-0.11.0.tar.gz -C /opt/graylog2
cd /opt/graylog2
ln -sf graylog2-web-interface-0.11.0 graylog2-web-interface
chown www-data.www-data -R /opt/graylog2/graylog2-web-interface

Installing RUBY 2.0

Preparing

apt-get install build-essential libcurl4-openssl-dev libssl-dev zlib1g-dev

Installing Ruby 2.0

cd /tmp
wget http://ftp.ruby-lang.org/pub/ruby/2.0/ruby-2.0.0-p0.tar.gz
tar -xzvf ruby-2.0.0-p0.tar.gz -C /opt/graylog2
cp ruby-2.0.0-p0 
cd /opt/graylog2/ruby-2.0.0-p0 
./configure && make && make install
update-alternatives --install /usr/bin/ruby ruby /usr/local/bin/ruby 20000

Check ruby is working

ruby -v

Installing bundler and others

gem install bundler --no-rdoc --no-ri

Change the version of json in the Gemfile

/opt/graylog2/graylog2-web-interface/Gemfile

...skiped...
gem 'json', '~> 1.5.5'               ===> change 1.5.5 to 1.7.7
...skiped...

Run bundle command for update json

bundle update json
bundle install

Installing web server daemon

apt-get install apache2-mpm-prefork apache2-prefork-dev libapr1-dev libaprutil1-dev
or
apt-get install nginx

Get passenger and make sure you pull the pre version

The Passenger is A modern web server and application server for Ruby, Python and Node.js, optimized for performance, low memory usage and ease of use.

Passenger web site : http://rubygems.org/gems/passenger

Installing the passenger module (for apache)

gem install passenger --no-rdoc --no-ri --pre
(For Apache) passenger-install-apache2-module
(For Nginx) passenger-install-nginx-module

Creating configuration files (for apache)

cd /etc/apache2/mods-available
cat <<EOF > passenger.conf
PassengerRoot /usr/local/lib/ruby/gems/2.0.0/gems/passenger-4.0.23
PassengerDefaultRuby /usr/local/bin/ruby
EOF
 
cat <<EOF > passenger.load
LoadModule passenger_module /usr/local/lib/ruby/gems/2.0.0/gems/passenger-4.0.23/buildout/apache2/mod_passenger.so
EOF
 
cd ../mods-enabled
ln -sf ../mods-available/passenger.conf
ln -sf ../mods-available/passenger.load
 
cd ../sites-available
mv default default.backup
 
cat <<EOF > default
<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        DocumentRoot /opt/graylog2/graylog2-web-interface/public
        RailsEnv 'production'
        <Directory /opt/graylog2/graylog2-web-interface/public>
                 # This relaxes Apache security settings.
                 AllowOverride all
                 # MultiViews must be turned off.
                 Options -MultiViews
        </Directory>
        ErrorLog ${APACHE_LOG_DIR}/error.log
        LogLevel warn
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
EOF

Editing elasticsearch and other items

/etc/elasticsearch/elasticsearch.yml

...skipped...
cluster.name: graylog2
node.name: "graylog2-elasticserver"
node.master: true
node.data: true

/etc/graylog2-elasticsearch.yml

cluster.name: graylog2
node.name: "graylog2-server"
transport.tcp.port: 9390

Restarting services

service apache2 start

Access Graylog2 Web

Screenshots

Gathering log from other servers(syslog clients)

CentOS6 case

cat <<EOF > /etc/rsyslog.d/graylog2.conf
# Forward all logs to Graylog2
*.* @192.168.126.133 #udp forwarding
EOF
 
service restart rsyslog

SAR time format을 24시간 형식으로 변경하기

SAR은 디폴트로 AM/PM형태의 시간 형식을 사용한다. 이를 24시간 형식으로 변경하기위해서는 다음과 같이 한다.

• 설정으로 변경하기

~/.bashrc

export LC_TIME="POSIX"

or

alias sar='LC_TIME="POSIX" sar'

• 일회성 설정 사용

LC_TIME="POSIX" sar

Jasper Reporting Integration

Install JDK & Tomcat

Installing Apache Tomcat 6 on RHEL or CentOS 참조

Change default JVM Options

/usr/share/tomcat6/bin/setenv.sh

export CATALINA_OPTS="$JAVA_OPTS -Xms1024m -Xmx2048m -XX:PermSize=32m -XX:MaxPermSize=128m -Xss2m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled"

Firewall

/etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT

Install JasperServer

Download URL

이 항목은 JasperServer에 번들로 제공되는 Tomcat과 PostgreSQL을 사용하지 않고 JasperReports Server WAR File Installer 을 통해 설치를 진행한다.

Extract file

unzip jasperreports-server-cp-5.1.0-bin_2.zip -d /tmp/

Create database

CREATE DATABASE jasperserver character set utf8;
GRANT ALL on *.* to 'jasperdb'@'localhost' IDENTIFIED BY 'tRe56Tr4V';
FLUSH PRIVILEGES; 

jasperdb account password : tRe56Tr4V (tango - ROMEO - echo - Five - Six - TANGO - romeo - Four - VICTOR)

Setting install configuration

cd /tmp/jasperreports-server-cp-5.1.0-bin/buildomatic/
cp sample_conf/mysql_master.properties default_master.properties

default_master.properties

appServerType = tomcat6
appServerDir = /usr/share/tomcat6
CATALINA_HOME = /usr/share/tomcat6
CATALINA_BASE = /usr/share/tomcat6


dbType=mysql


dbHost=localhost
dbUsername=jasperdb
dbPassword=tRe56Tr4V


deployJDBCDriver=false

Download MySQL JDBC Connector

Download URL

tar zxvf mysql-connector-java-5.1.25.tar.gz -C /tmp
cd /tmp/mysql-connector-java-5.1.25/
cp mysql-connector-java-5.1.25-bin.jar /tmp/jasperreports-server-cp-5.1.0-bin/buildomatic/conf_source/db/mysql/jdbc/
cp mysql-connector-java-5.1.25-bin.jar /usr/share/tomcat6/lib/
chown tomcat: /usr/share/tomcat6/lib/mysql-connector-java-5.1.25-bin.jar

Deploying

./js-install-ce.sh

Change Permission JasperServer Web Context

chown -R tomcat: /usr/share/tomcat6/webapps/jasperserver

Connect

http://<IP>:8080/jasperserver

• default account : User: jasperadmin Password: jasperadmin
• sample end-user : User: joeuser Password: joeuser
• CAUTION: For security reasons, always change the default passwords immediately after installing JasperReports Server.

Install Icinga Reports

Extract package

tar zxvf icinga-reports-1.9.0.tar.gz -C /tmp
cd /tmp/icinga-reports-1.9.0

Modifying Icinga Database

mysql -uroot -p icinga < db/icinga/mysql/availability.sql

Check Grant options

SHOW GRANTS FOR 'icinga'@'localhost';
+---------------------------------------------------------------------------------------------------------------+
| Grants for icinga@localhost                                                                                   |
+---------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'icinga'@'localhost' IDENTIFIED BY PASSWORD '*88D4515B04D41BAB8B950A069A26A2D449460144' |
| GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW ON `icinga`.* TO 'icinga'@'localhost'                 |
+---------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

MySQL

GRANT EXECUTE ON icinga.* to 'icinga'@'localhost';
SHOW GRANTS FOR 'icinga'@'localhost';
+---------------------------------------------------------------------------------------------------------------+
| Grants for icinga@localhost                                                                                   |
+---------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'icinga'@'localhost' IDENTIFIED BY PASSWORD '*88D4515B04D41BAB8B950A069A26A2D449460144' |
| GRANT SELECT, INSERT, UPDATE, DELETE, DROP, EXECUTE, CREATE VIEW ON `icinga`.* TO 'icinga'@'localhost'        |
+---------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)


FLUSH PRIVILEGES;

Import Icinga jasper report & Copy icinga report JAR

./configure --with-jasper-server=/tmp/jasperreports-server-cp-5.1.0-bin
make js-import-icinga
cp ./jsp-server/classes/icinga/icinga-reporting.jar /usr/share/tomcat6/webapps/jasperserver/WEB-INF/lib

make install 명령은 Jasper Report Server 설치본으로 설치한 경우에만 제대로 동작하므로 여기서는 각각 설치하도록 하였다.

Change IDO Settings

• root -> Icinga -> Data Sources -> IDO Select -> Click Edit Button on the right panel

• Change icinga password and Click Test Connection button
• Click Save button when the connection test is passed