VHACS-VM
The Linux SCSI Target Wiki
 |
This article needs a review and may need a cleanup or additional content. Please help improve this article to meet our quality standards. (12/7/2010) |
 | |
|---|---|
| Storage Cloud Controller | |
| Original author(s) | Jerome Martin |
| Developer(s) | Datera, Inc. |
| Initial release | June 26, 2008 |
| Preview release | 0.8.15 / August 25, 2008 |
| Development status | Deprecated |
| Written in | Python |
| Operating system | Linux |
| Type | Cloud controller |
| License | GNU GPLv2 |
| Website | datera.io |
VHACS-VM is an Alpha release of the VHACS high availability storage cloud stack running in a virtualized x86_64 environment. VHACS is a highly available storage cloud based on at least eight open source technologies running on Linux v2.6.25.
The VHACS-VM images are available, and oriented towards kernel developers, storage and network systems researchers, C programmers, Python and Shell programmers, and anyone who wants to run and see a real life open storage cloud on their virtualized x86 PC. Using VHACS-VM is the fastest and easiest method for users to get involved with the project. This is a developer-oriented release, and a complete development environment is included in the images.
Currently, VHACS can manage 4-8 storage clouds as long as the network bridge setup is working as expected (see below).
Contents |
Current setup
There are two (vhacs64-west and vhacs64-east) x86_64 VMware v6 compatible images, and there is one build .vmdk containing the source for the VHACS Alpha. The build disk image is distributed seperately in order to reduce the size for obtaining a two node VHACS-VM cloud of your own.
# ls -al -rw-r--r-- 1 root root 830805854 Jul 16 19:57 VHACS64-BUILD-0.8.15.zip -rw-r--r-- 1 root root 314996216 Jul 16 19:56 VHACS64-East-0.8.15-2.zip -rw-r--r-- 1 root root 299372440 Jul 16 19:56 VHACS64-West-0.8.15-2.zip # cat md5sum.txt e1923fbea84b2e91739ca1199e4695f4 VHACS64-BUILD-0.8.15.zip 32bc1e16c1961c3fb3e66ea52976060a VHACS64-East-0.8.15-2.zip 160e3cdee8344faedf148c5eb845ed2c VHACS64-West-0.8.15-2.zip
Each image has a Debian Etch v4.0 r3 x86_64 installation. The Linux kernel is v2.6.25.9 (see kernel.org) with KDB patches. The VHACS-VM kernel is also configured with ParaVirt-Ops for the least amount of virtualized execution overhead on modern PC hardware.
The .vmx images are configured with:
- Bridged networking mode, configured to grab a DHCP address
- 256 MB of system memory
- three virtual disks:
-
/dev/sda: 4 GB for rootfs -
/dev/sdb: 20 GB for /mnt/build -
/dev/sdc: 40 GB for the VHACS volume group (VG)
-
Requirements
A x86_64 CPU with hardware x86 virtualization is required, i.e. the CPU has a virtualized memory management unit (MMU) capable of handling multiple page tables from multiple different unmodified operating systems.
Hardware x86 virtualization is the easiest way to get a high-performance open-source storage cloud running on recent x86_64 CPUs quickly (for the purpose of the Alpha preview) - VHACS itself doesn't require x86 hardware virtualization, of course.
Setup
/etc/hosts
Based on your DHCP configured IP address (obtained with ifconfig eth0), set the vhacs64-west.linux-iscsi.org and vhacs64-east.linux-iscsi.org hostnames to your locally allocated IP addresses on both vhacs64-west and vhacs64-east VM images.
/etc/vhacs.conf
Configure the storage network and mask:
STORAGE_NETWORK = "192.168.0.0" STORAGE_NETMASK = "24"
Configure the range of IP addresses that are assigned to VHACS cloud from the resource pool.
WARNING: Make sure this range is either explictly not in use by your DHCP server, or won't have an affect on your active network.
FIRST_STORAGE_IP = "192.168.0.50" LAST_STORAGE_IP = "192.168.0.99"
NIC configuration
VHACS-VM can be configured to use different STORAGE_IFNAME and HEARTBEAT_IFNAME interfaces. For instance, this consists of:
- Having two physical hosts running VMware and VHACS-VM. This implies any PC with hardware x86_64 virtualization that can run VMware with the current 64-bit VHACS-VM images.
- Having two bridged network interfaces on the host machines running vhacs64-[west,east] demo images. They should be running on a different local subnet or network range from each other.
The current setup using two network bridges looks something like:
- 192.168.0.0 (eth0) for
STORAGE_IFNAME:
HOST0: 192.168.0.*/ethX via DHCP bridged into vhacs64-east VM HOST1: 192.168.0.*/ethX via DHCP bridged into vhacs64-west VM vhacs64-east: 192.168.0.*/eth0 via DHCP vhacs64-west: 192.168.0.*/eth0 via DHCP
- 10.10.0.0 (eth1) for
HEARTBEAT_IFNAME:
HOST0: 10.10.0.5/ethY bridged into vhacs64-east VM HOST1: 10.10.0.10/ethY bridged into vhacs64-west VM vhacs64-east: 10.10.0.15/eth1 via static IP vhacs64-west: 10.10.0.20/eth1 via static IP
Also, you need to change your /etc/ais/openais.conf to the 10.10.0.0 network for broadcast packets (example).
Results
A correctly running VHACS setup with working cluster communication should produce the following crm_mon -1 output after booting:
vhacs64-west:~# crm_mon -1 ============ Last updated: Thu Jul 3 00:38:29 2008 Current DC: vhacs64-east (vhacs64-east) 2 Nodes configured. 0 Resources configured. ============ Node: vhacs64-west (vhacs64-west): online Node: vhacs64-east (vhacs64-east): online
Problems
Failure to communicate
If cluster communication somehow cannot be established between nodes, try calling the VHACS cluster initialization logic to restart services on both vhacs64-west and vhacs64-east images:
vhacs cluster -I vhacs64-west,vhacd64-east
Again, double check that the hostnames vhacs64-west and vhacs64-east in both VHACS-VM images /etc/hosts files match each VHACS-VM images' individual ifconfig output.
Failure to join
If crm_mon -1 only shows the local node on each VHACS-VM image, it is an indication that the UDP broadcast packets for cluster communciation didn't come up cleanly or got wedged.
For reference purposes, the VHACS-VM prototype was a 2.2 Ghz Core 2 Duo with 1 GB of DDR2-800 running Debian Etch i386. The local bridged Ethernet interface is an on-board Realtek RTL8111/8168B PCI-Express 1 Gb/sec MAC.
Note that not all ethernet interfaces support bridging sufficently to run VHACS-VM. We still need to determine how to easily find the networking hardware that can be used with the various virtualization hypervisors with VHACS-VM images.
Launching the storage cloud
Now that active cluster communication between the two VHACS nodes (vhacs64-west and vhacs64-east) is established, the storage cloud can be launched as follows.
Assigning roles
Roles can be assigned to the nodes in the VHACS cluster as follows:
vhacs64-west:~# vhacs node -s ALL vhacs64-west,vhacs64-east role vhost now set on node vhacs64-west role storage now set on node vhacs64-west role vhost now set on node vhacs64-east role storage now set on node vhacs64-east
Allocating the storage cloud
The VHACS storage cloud can then be allocated as follows:
vhacs64-west:~# vhacs storage -c vmcloud0 -s 200M storage vmcloud0 has been assigned IP 192.168.0.50 LVM volume vmcloud0 created on vhacs64-west LVM volume vmcloud0 created on vhacs64-east Promoting node vhacs64-west Waiting until sync is finished (be patient) ... drbd devices configured and formated as ext3 on vhacs64-west and vhacs64-east master/slave drbd resource created and constrained for storage vmcloud0 iSCSI target iqn.2003-01.org.linux-iscsi:192.168.0.50 created and constrained for storage vmcloud0 iSCSI filesystem for iqn.2003-01.org.linux-iscsi:192.168.0.50 created and constrained for storage vmcloud0
Monitoring
The storage cloud can be monitord by opening up another ssh connection to one of the VHACS-VM nodes, and running the VHACS monitor. The following reflects the simple configuration descibed here:
vhacs64-east:~# vhacs cluster -m [00:56:00] [vhacs64-east] [VHACS cluster storage resources status] ____________________________________________________________________________________________________________________________________________________________________ | | | | | | | NODE | HA STATUS | FREE STORAGE | STORAGE ROLE | VHOST ROLE | |________________________________|________________________________|________________________________|________________________________|________________________________| | | | | | | | (A)vhacs64-west | online | 39.80G/39.99G | 1 exported | 0 mounted | | (A)vhacs64-east | online | 39.80G/39.99G | 0 exported | 1 mounted | |________________________________|________________________________|________________________________|________________________________|________________________________| ____________________________________________________________________________________________________________________________________________________________________ | | | | | | | STORAGE | PRI/SEC DRBD | DRBD TARGET | ISCSI MOUNT | FREE SPACE | |________________________________|________________________________|________________________________|________________________________|________________________________| | | | | | | | (A)vmcloud0 | (P)vhacs64-west/(S)vhacs64-east| (S)vhacs64-west | (S)vhacs64-east | 179M/194M (96%) | |________________________________|________________________________|________________________________|________________________________|________________________________|
A more complex VHACS storage cloud with four volumes could look as follows:
vhacs64-east:~# vhacs cluster -M _________________________________________________________________________________________________________________________________________________________________________ | | | | | | | NODE | HA STATUS | FREE STORAGE | STORAGE ROLE | VHOST ROLE | |_________________________________|_________________________________|_________________________________|_________________________________|_________________________________| | | | | | | | | | | | | | (A)vhacs64-west | online | 39.21G/39.99G | 3 exported | 2 mounted | | (A)vhacs64-east | online | 39.21G/39.99G | 1 exported | 2 mounted | |_________________________________|_________________________________|_________________________________|_________________________________|_________________________________| _________________________________________________________________________________________________________________________________________________________________________ | | | | | | | NODE | WHEN | OPERATION TYPE | RESOURCE | RETURN CODE | |_________________________________|_________________________________|_________________________________|_________________________________|_________________________________| | | | | | | | vhacs64-west | 2h30m28s ago | initial start | ms_drbd_vmcloud1_CLONE:0 | ERROR | |_________________________________|_________________________________|_________________________________|_________________________________|_________________________________| _________________________________________________________________________________________________________________________________________________________________________ | | | | | | | STORAGE | PRI/SEC DRBD | DRBD TARGET | ISCSI MOUNT | FREE SPACE | |_________________________________|_________________________________|_________________________________|_________________________________|_________________________________| | | | | | | | (A)vmcloud0 | (P)vhacs64-west/(S)vhacs64-east | (S)vhacs64-west | (S)vhacs64-east | 179M/194M (96%) | | (A)vmcloud1 | stopped/(P)vhacs64-east | (S)vhacs64-east | (S)vhacs64-west | 179M/194M (96%) | | (A)vmcloud2 | (P)vhacs64-west/(S)vhacs64-east | (S)vhacs64-west | (S)vhacs64-west | 179M/194M (96%) | | (A)vmcloud3 | (S)vhacs64-east/(P)vhacs64-west | (S)vhacs64-west | (S)vhacs64-east | 179M/194M (96%) | |_________________________________|_________________________________|_________________________________|_________________________________|_________________________________|
Trouble shooting
As this is a very developer oriented Alpha preview, so you can help us find bugs! Here is the 10,000 ft view:
VHACS kernel/systems developer
The included v2.6.25.9 x86_64 kernel is patched with SGI Kernel Debugger. At any time, you can break into the kernel debugger with the good old' PC key 'Pause/Break'. Note that not all laptops have this key, so you may have to use a USB keyboard to access KDB.
Cluster infrastructure developer
The cluster stack (see VHACS for the stack components) is enabled with verbose debug. You can view the complete logs with tail -f /var/log/syslog.
VHACS systems administator
vhacs cluster -m is the primary status monitor (see above). Exceptions related to cluster RAs will appear as they occur.
See also
- Cloud storage and management: RTS Director and VHACS
- High availability: DRBD
- Virtualization: LIO-VM
- VMs: VMware ESX, VMware vSphere and KVM
External links
- Cloud storage Wikipedia entry
- x86 virtualization Wikipedia entry
