Friday 24 April 2015

CloudStack 4.4 + KVM + OVS Installation

Table of Contents

CloudStack 4.4 Installation

Physical Network Structure
System Specification
IP
CloudStack Type
OS
Kernel
10.40.205.2ManagementCentOS 6.62.6.32-504.1.3.el6.x86_64
10.40.205.3KVM HostCentOS 6.62.6.32-504.8.1.el6.x86_64
10.40.205.4KVM HostCentOS 6.62.6.32-504.8.1.el6.x86_64
10.40.205.5KVM HostCentOS 6.62.6.32-504.8.1.el6.x86_64




10.40.205.253Secondary StorageFreeNAS 9.3
10.40.205.254Primary StorageFreeNAS 9.3

Storage Settings

I'm using FreeNAS as a main storage server at this time.

Secondary Storage(NFS)

NFS export(share) setting

Primary Storage(iSCSI + CLVM)

Primary Storage 1(iSCSI)
Please refer the part Storage(FreeNAS) Settings of the wiki : ISCSI + CLVM as a primary storage

Management Server

  • Hostname: syseng-dev-mgmt.localdomain
  • Timezone: Asia/Seoul
  • root password: !@******(Default Password)
  • Partitioning Layout: Replace Existing Linux Systems - Changed lv_home -> lv_var (for MySQL database)
  • Installation Type: Basic Server
  • IP: 10.40.205.2
  • Netmask: 255.255.255.0
  • Gateway: 10.40.205.1

Basic settings

/etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.1.2  syseng-dev-mgmt.localdomain syseng-dev-mgmt
/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=syseng-dev-mgmt.localdomain
GATEWAY=10.40.205.1
/etc/sysconfig/network-scripts/ifcfg-em1
DEVICE=em1
TYPE=Ethernet
UUID=77568336-91d6-4277-a510-0ee8e50ee069
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
HWADDR=78:2B:CB:76:45:10
IPADDR=10.40.205.2
PREFIX=24
GATEWAY=10.40.205.1
DNS1=8.8.8.8
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System em1"
/etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=em2
HWADDR=78:2B:CB:76:45:11
TYPE=Ethernet
UUID=593d205f-ed2a-4b98-a108-3d112c3ab9c9
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=172.16.1.2
NETMASK=255.255.255.0

Repository settings

I copied all files from the CloudStack repository. also I created a new repository for installing CloudStack packages quickly in the Intra network.
Follow steps of the WIKI Create CloudStack mirror repo, if you want.
/etc/yum.repos.d/cloudstack.repo
[cloudstack]
name=cloudstack
baseurl=http://cloudstack.apt-get.eu/rhel/4.4/
enabled=0
gpgcheck=0
/etc/yum.repos.d/cloudstack-local.repo
[cloudstack-local]
name=cloudstack
baseurl=http://127.0.0.1/cloudstack
enabled=1
gpgcheck=0
yum update -y
yum install nfs-utils -y
Mount secondary storage's NFS export
I've already set the NFS export on the FreeNAS server(10.40.205.253, 172.16.1.253) for this.
mkdir /mnt/secondary
mount -t nfs 172.16.1.253:/mnt/tank/cs-sec-storage /mnt/secondary/

Database Settings

MySQL
# yum install -y mysql-server
# mv /etc/my.cnf{,.back}
# cat <<EOF > /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
EOF
# /usr/bin/mysqladmin -u root password '3EnzhLKtuH'
MySQL Password
Icon
You must change the password of the root user.
Install CloudStack management packages 
yum -y install cloudstack-management
Install optional utility file for XenServer
Actually, I don't use XenServer as hypervisor, therefore I don't need it.
Optional
wget http://download.cloud.com.s3.amazonaws.com/tools/vhd-util -O /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util

Initialize CloudStack database
cloudstack-setup-databases cloud:<Your Password>@localhost --deploy-as=root:<Your Password>
Please see the document
Icon
You should check out the link about-password-key-encryption for the better safety. 
MySQL Password
Icon
You must change the password of the cloud user.
The above command will create a file called db.properties which has multiple key-value pairs for the database connection, When you have a time please refer this file.

Setup management
cloudstack-setup-management
Setup systemVM template
CloudStack uses a number of system VMs to provide functionality for accessing the console of virtual machines, providing various networking services, and managing various aspects of storage.
This step will acquire those system images ready for deployment when we bootstrap your cloud.
Icon
For the convenient, I copied the system template file from the CloudStack repository into the directory /var/www/html for accessing this file from the rest of hypervisor hosts.
from mirror
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-m /mnt/secondary \
-u http://localhost/systemvm64template-4.4.1-7-kvm.qcow2.bz2 \
-h kvm -F
Get from the origin
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-m /mnt/secondary \
-u http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.1-7-kvm.qcow2.bz2 \
-h kvm -F

Change IP address of the management server (Important)

Icon
I'm going to use the private IP address of the management server instead of the public IP.
If I use a public IP 10.40.205.2, the SSVM(System Storage VM) can not find the management server.
Essentially the SSVM tries to connect the management server via eth1.because this private nic used as management interface between the management server and itself(SSVM).
Restart service
service cloudstack-management restart

KVM Hosts

  • Timezone: Asia/Seoul
  • network configuration: I only set the IP address on the em1(nic 1), please refer to the below table
  • root password: ******(Your password>
  • Partitioning Layout: Replace Existing Linux Systems (lv_home(/home) -> lv_var(/var))
  • Installation Type: Basic Server
HostnameNetwork
cloudbr0
(public traffic)		cloudbr1
(guest traffic)		cloudbr2
(Management traffic)		em4(SAN)
IPNMGWIPNMIPNMIPNM
syseng-dev-kvmhost0110.40.205.3255.255.255.010.40.205.1- -172.16.1.3255.255.255.0192.168.1.3255.255.255.0
syseng-dev-kvmhost0210.40.205.4255.255.255.010.40.205.1--172.16.1.4255.255.255.0192.168.1.4255.255.255.0
syseng-dev-kvmhost0310.40.205.5255.255.255.010.40.205.1--172.16.1.5255.255.255.0192.168.1.5255.255.255.0

Basic settings

/etc/hosts  (deploy this to all hosts)
Icon
This file is needed by the cluster management daemon(cman).
this daemon needs to identify the status of the member of the cluster by their names.
If you are planning to use local DNS, you don't need this process.
/etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.1.3 syseng-dev-kvmhost01.localdomain syseng-dev-kvmhost01
172.16.1.4 syseng-dev-kvmhost02.localdomain syseng-dev-kvmhost02
172.16.1.5 syseng-dev-kvmhost03.localdomain syseng-dev-kvmhost03
Register CloudStack YUM repository
/etc/yum.repos.d/cloudstack.repo
cat <<EOF > /etc/yum.repos.d/cloudstack.repo
[cloudstack]
name=cloudstack
baseurl=http://10.40.205.2/cloudstack
enabled=1
gpgcheck=0
EOF
Updating & Applying updates
yum update -y && reboot

Install CloudStack Agent 

Install virtualization & CloudStack agent packages
yum -y groupinstall virtualization-tools virtualization virtualization-client virtualization-platform
yum -y install cloudstack-agent

KVM Configuration

 QEMU Configuration
/etc/libvirt/qemu.conf
vnc_listen=0.0.0.0
Libvirt Configuration 
CloudStack uses libvirt for managing virtual machines. It is vital that libvirt is configured correctly. Libvirt is a dependency of cloud-agent and should already be installed.
/etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
tcp_port = "16059"
auth_tcp = "none"
mdns_adv = 0
Make sure below LIBVIRTD_ARGS is not commented
/etc/sysconfig/libvirtd
LIBVIRTD_ARGS="--listen"
service libvirtd restart

Configure CPU model for KVM Guest
Add this line in the file /etc/cloudstack/agent/agent.properties.
/etc/cloudstack/agent/agent.properties
echo "guest.cpu.mode=host-passthrough" >> /etc/cloudstack/agent/agent.properties
Icon
Please refer to Hypervisor setup section of the CloudStack Installation Documentation.

SELinux

You should set the SELinux as at least permissive mode.
sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config
setenforce 0

Open vSwitch (OVS)

 Install dependency packages
Prerequisites
yum install -y @development
yum install -y openssl-devel
 Build & Install OVS
cd ~/
mkdir -p rpmbuild/{SOURCES,SPECS}
wget http://openvswitch.org/releases/openvswitch-2.3.1.tar.gz
tar zxf openvswitch-2.3.1.tar.gz
cp openvswitch-2.3.1.tar.gz ~/rpmbuild/SOURCES/
cp openvswitch-2.3.1/rhel/openvswitch-kmod.files ~/rpmbuild/SOURCES/
cp openvswitch-2.3.1/rhel/openvswitch*.spec ~/rpmbuild/SPECS/
  
cd rpmbuild
rpmbuild -bb SPECS/openvswitch.spec
rpmbuild -bb SPECS/openvswitch-kmod-rhel6.spec
  
yum -y localinstall RPMS/x86_64/kmod-openvswitch-2.3.1-1.el6.x86_64.rpm
yum -y localinstall RPMS/x86_64/openvswitch-2.3.1-1.x86_64.rpm
 Bridge Network settings 
Common configurations
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-em1
DEVICE=em1
IPV6INIT=no
IPV6_AUTOCONF=no
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
NM_CONTROLLED=no
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=cloudbr0
EOF
  
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=em2
IPV6INIT=no
IPV6_AUTOCONF=no
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
NM_CONTROLLED=no
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=cloudbr1
EOF
  
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-em3
DEVICE=em3
IPV6INIT=no
IPV6_AUTOCONF=no
ONBOOT=yes
HOTPLUG=no
BOOTPROTO=none
NM_CONTROLLED=no
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=cloudbr2
EOF
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-cloudbr1
DEVICE=cloudbr1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
IPV6INIT=no
IPV6_AUTOCONF=no
BOOTPROTO=none
HOTPLUG=no
EOF
syseng-dev-kvmhost01
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
IPV6INIT=no
IPV6_AUTOCONF=no
BOOTPROTO=static
IPADDR=10.40.205.3
NETMASK=255.255.255.0
GATEWAY=10.40.205.1
EOF
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-cloudbr2
DEVICE=cloudbr2
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
IPV6INIT=no
IPV6_AUTOCONF=no
BOOTPROTO=static
IPADDR=172.16.1.3
NETMASK=255.255.255.0
EOF
syseng-dev-kvmhost02
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
IPV6INIT=no
IPV6_AUTOCONF=no
BOOTPROTO=static
IPADDR=10.40.205.4
NETMASK=255.255.255.0
GATEWAY=10.40.205.1
EOF
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-cloudbr2
DEVICE=cloudbr2
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
IPV6INIT=no
IPV6_AUTOCONF=no
BOOTPROTO=static
IPADDR=172.16.1.4
NETMASK=255.255.255.0
EOF
syseng-dev-kvmhost03
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
IPV6INIT=no
IPV6_AUTOCONF=no
BOOTPROTO=static
IPADDR=10.40.205.5
NETMASK=255.255.255.0
GATEWAY=10.40.205.1
EOF
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-cloudbr2
DEVICE=cloudbr2
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
IPV6INIT=no
IPV6_AUTOCONF=no
BOOTPROTO=static
IPADDR=172.16.1.5
NETMASK=255.255.255.0
EOF
echo 'blacklist bridge' >> /etc/modprobe.d/blacklist.conf
echo "network.bridge.type=openvswitch" >> /etc/cloudstack/agent/agent.properties
echo "libvirt.vif.driver=com.cloud.hypervisor.kvm.resource.OvsVifDriver" >> /etc/cloudstack/agent/agent.properties
For iSCSI traffic
Icon
I used the em4 interface to communication with a iSCSI storage(currently I'm using a FreeNAS server).
A iSCSI volume will be used as a volume as a primary storage, and this volume should be mounted on each KVM host servers.

syseng-dev-kvmhost01
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-em4
DEVICE=em4
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=192.168.1.3
NETMASK=255.255.255.0
MTU=9000
EOF
syseng-dev-kvmhost02
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-em4
DEVICE=em4
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=192.168.1.4
NETMASK=255.255.255.0
MTU=9000
EOF
syseng-dev-kvmhost03
cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-em4
DEVICE=em4
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=192.168.1.5
NETMASK=255.255.255.0
MTU=9000
EOF
Reboot for applying OVS 
reboot
 Check 
[root@syseng-dev-host01 rpmbuild]# ovs-vsctl -V
ovs-vsctl (Open vSwitch) 2.3.1
Compiled Dec 12 2014 08:54:49
DB Schema 7.6.2
[root@syseng-dev-host01 rpmbuild]#
[root@syseng-dev-host01 ~]# ovs-vsctl show
94404a34-774c-4e7e-b1ac-5c667bee0083
    Bridge "cloudbr0"
        Port "em1"
            Interface "em1"
        Port "cloudbr0"
            Interface "cloudbr0"
                type: internal
    Bridge "cloudbr1"
        Port "cloudbr1"
            Interface "cloudbr1"
                type: internal
        Port "em2"
            Interface "em2"
    ovs_version: "2.3.1"

Primary Storage Settings

  • ISCSI + CLVM as a primary storage

Access Management Web UI

  • http://10.40.205.2:8080/client (admin/********)

Zone Settings

  • Zone Setting for SYSENG-DEV in the office
Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)