Daniel Camara's home page

Virtualization with OpenVZ

Introduction and Disclaimer :)

Disclaimer: I resume here what I have learned about the virtualization with the OpenVZ tool. I DO NOT assume any responsibility about rights and wrongs of what you find here. These are my understandings of the tool, they can be right or wrong, so if you are using this information, evaluate it by yourself first.

Open Vz is a virtualization tool that is a co project with the Virtuoso one, the lest one commercial. A HUGE amount of information about open vz can be found in the Open VZ project's page, and in the forum.

What we will discuss here are related to debian based distributions, what does include ubuntu. I will present:

The installation process
Network configuration (Veth and Bridge)
Scripts to create and use the virtual machines

Conventions:

host #> : denotes commands executed in the real machine, the host machine
virtual 104 #>: denotes commands executed in the virtual machine, the exemplified one

Scripts: What is explained below is also summarized in two bash scripts, I hope them are useful for you.

Creating OpenVZ shell script

Running OpenVZ shell script

Comments:

IP tool: Open vz uses a tool to configure the network, if you haven't it installed try to install, if you are unable to install it forget and use the route add command. It works exactly as the same and it becomes clearly to you what is going on.

A good tutorial on veth configuration can be found here.

Installation

Add to the /etc/apt/sources.list file the following lines

host #> cat >> /etc/apt/sources.list
deb http://download.openvz.org/debian etch main
deb http://debian.systs.org/ stable openvz
deb-src http://debian.systs.org/ stable openvz

Install the open VZ kernel version 2.6.18-028stab035.1 or subversion (stab035) grater. There is a bug in the previous versions that prevent the use of the multicast communication. Use synaptic or with the apt-get in this way

host #> apt-get update
host #> apt-get install ovzkernel-2.6.18-enterprise

Install the management applications

host #> apt-get install vzctl vzquota vzdump vzctl-ostmpl-debian

Update grub with the new kernel

host #> update-grub

# If the grub is not installed it will be necessary to install it. FOR THE TESTED CASE!!!!!! in this way:
host #> grub-install /dev/hda4
host #> /sbin/update-grub

Edit the /boot/grub/menu.lst to enable the visualization of the menu, got to the hilenmenu line

(# hiddenmenu) remove the #

Reboot in your new Debian Stable OpenVZ System, Simple like that!!!

host #> reboot

The modprobe vzethdev modules should be loaded, but if they aren't, load them

host #> modprobe vzethdev

Virtual machine creation

The virtual machines are based into templates. There are a series of templates do download through the Internet. But basically the template is a tar.gz of a Linux system. The templates are sotered in the directory

To create an image based on a template

host #> vzctl create 104 --ostemplate debian-4.0-i386-minimal

To verify if it was created

host #> vzlist -a
VEID NPROC STATUS IP_ADDR HOSTNAME
104 - stopped - -

Network configuration

There are two or three different ways to configure the network and, off course, hundreds of different ways doing this configuration. We will show only one, a simple and, I expect, understandable one. The IP address is guessed, and in this case, part of an internal network. We will say that the 192.168.12.100 is the gateway and name server for the network we are configuring. You need to see which are this numbers for your case.

host #> vzctl set 104 --ipadd 192.168.12.104 --save
Saved parameters for VE 104
host #> rvzctl set 104 --nameserver 192.168.12.100 --save
Saved parameters for VE 104

Attributes the mac addresses to the two interfaces, the internal to the virtual machine and the virtual interface created at the real machine. The MAC addresses can be any valid ones, but inexistent in the same sub net. We will use as standard a random number followed by the machine id, in this case 104, for the virtual interface on the real machine and the same random number followed by the inverse of the ID 401 for the interface inside the virtual machine.

host #> vzctl set 104 --netif_add eth0,00:12:34:56:74:01,veth104.0,00:12:34:56:71:04 --save
Saved parameters for VE 104

To configure the interface the virtual machine must to be started.

host #> vzctl start 104
   Starting VE ...
   VE is mounted
   Adding IP address(es): 192.168.12.104
   Setting CPU units: 1000
   File resolv.conf was modified
   Configure veth devices: veth104.0
   VE start in progress...

Configures the virtual interface and default interface in the real machine

host #> ifconfig veth104.0 0
host #> echo 1 > /proc/sys/net/ipv4/conf/veth104.0/forwarding
host #> echo 1 > /proc/sys/net/ipv4/conf/veth104.0/proxy_arp
host #> echo 1 > /proc/sys/net/ipv4/conf/eth0/forwarding
host #> echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
host #> ip route add 192.168.12.104 dev veth104.0

host #> enable () { for file in $@; do echo $file; echo 1 > $file; done }
host #> enable /proc/sys/net/ipv4/conf/*/send_redirects
host #> cat 0 > /proc/sys/net/ipv4/icmp_echo_ignore_all
host #> cat 0 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

Enters in the virtual machine to configure the interface inside it

host #> vzctl enter 104

Check the interfaces

virtual 104 #> ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

venet0    Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:127.0.0.1 P-t-P:127.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

Configure the eth0

virtual 104 #> /sbin/ifconfig eth0 0
virtual 104 #> ifconfig eth0 192.168.12.104

Check the interfaces again and verify if every thing is as expected

virtual 104 #> ifconfig
eth0      Link encap:Ethernet HWaddr 00:12:34:56:74:01
          inet addr:192.168.12.104 Bcast:0.0.0.0 Mask:255.255.255.255
          inet6 addr: fe80::212:34ff:fe56:7401/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:384 (384.0 b)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

venet0    Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:127.0.0.1 P-t-P:127.0.0.1 Bcast:0.0.0.0 Mask:255.255.255.255
          UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

Configure the routing, if there are some thing like this, you need to fix. These are the configurations of the image, that MAYBE are not suitable to your network

virtual 104 #> route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.0.2.1       *               255.255.255.255 UH    0      0        0 venet0
0.0.0.0         192.0.2.1       0.0.0.0         UG    0      0        0 venet0

Add your own default gateway

virtual 104 #> route add default gw 192.168.12.100

And remove anything not needed from the routing table, in this case:

virtual 104 #> route del default gw 192.0.2.1
virtual 104 #> route del -host 192.0.2.1

Verify if everything is fine

virtual 104 #> route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.12.0    *               255.255.255.0   U     0      0        0 eth0
0.0.0.0         192.168.12.100 0.0.0.0         UG    0      0        0 eth0

To REALY see if every thing is fine

virtual 104 #> ping 192.168.12.100
PING 192.168.12.100 (192.168.12.100) 56(84) bytes of data.
64 bytes from 192.168.12.100: icmp_seq=1 ttl=63 time=979 ms
64 bytes from 192.168.12.100: icmp_seq=2 ttl=63 time=0.348 ms
64 bytes from 192.168.12.100: icmp_seq=3 ttl=63 time=0.349 ms

--- 192.168.12.100 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.348/326.606/979.122/461.398 ms

# If the access the names are not working, add the name server of your network to the /etc/resolv.conf

virtual 104 #> cat > /etc/resolv.conf
nameserver 192.168.12.100

Bridge setup

To add more than one virtual machine, one may use a bridge. Each individual machine should be configured as it is described previously here, but the added routing to the new machine through the virtual interface should be suppressed. If not, there will be two different routes for the same machine and the OS will not understand what to do with packets to this address. So,configure all the machines as the 104 above and after that create and configure the bridge as follows:

host #> brctl addbr vzbr0

# Add veth devices to bridge
host #> brctl addif vzbr0 veth104.0
host #> brctl addif vzbr0 veth105.0
...
host #> brctl addif vzbr0 vethXXX.N

Configure bridge device

host #> ifconfig vzbr0 0
host #> echo 1 > /proc/sys/net/ipv4/conf/vzbr0/forwarding
host #> echo 1 > /proc/sys/net/ipv4/conf/vzbr0/proxy_arp

Add routes in VE0

host #> ip route add 192.168.12.104 dev vzbr0
host #> ip route add 192.168.12.105 dev vzbr0
host #> ip route add 192.168.12.106 dev vzbr0
....
host #> ip route add 192.168.xxx.xxx dev vzbr0

Just to be sure configure the interfaces at the end

host #> enable () { for file in $@; do echo $file; echo 1 > $file; done }
host #> enable /proc/sys/net/ipv4/conf/*/send_redirects
host #> cat 0 > /proc/sys/net/ipv4/icmp_echo_ignore_all
host #> cat 0 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts