Tag Archives: cloudstack

Usage metering and charging with Cloudstack

One of the prominent features of an IaaS cloud is that one can meter its resource usage by its consumers. Metrics are everywhere: From the hypervisor, virtual disk size, network I/O, occupied IP addresses, virtual CPUs and RAM, they are all over the place, waiting to be collected. As soon as you can grab a handful of metrics, you can implement chargeback policies and report back on your users on their resource consumption or, if you run a public IaaS shop, somehow transform these metrics to invoices.

Cloud.com’s cloudstack comes with an excellect usage server, recording metrics directly from its accounts. During installation, simply select the “Install Usage Server” option and perform some basic configuration, and you are all set to go. The usage server collect no less than thirteen (as of cloudstack release 2.2) metrics, which can be found here. In short, some of the most important ones are:

  • RUNNING_VM: Total hours a virtual machine is started and running on the hypervisor
  • ALLOCATED_VM: Total hours a VM exists (no matter if it’s up or down). Useful parameter for charging OS license usage, for example Microsoft SPLA licenses.
  • IP_ADDRESS: Self evident; applies to public (Internet) IP addresses consumed by a cloudstack account. These addresses are (according to cloudstack architecture) attached to the virtual router of the user
  • NETWORK_BYTES_SENT and NETWORK_BYTES_RECEIVED: Traffic passing through the virtual router of a user
  • VOLUME: Size in bytes of user volumes
  • TEMPLATE and ISO: Size in bytes of user-uploaded VM templates and ISO images
(For those who are not familiar with cloudstack’s architecture, cloudstack users are part of accounts. Virtual machines belonging to a single account live in their own private VLAN, totally isolated from other accounts. Access to the Internet, DHCP addressing, DNS and VPN termination, all take place in a special cloudstack virtual machine, a virtual router. Every account has its own virtual router, not directly controlled by the end user, but via the cloudstack API).

The service (“cloud-usage”) starts along with the rest cloud services on your cloudstack controller and its configuration variables are at the global parameters of cloudstack. The most important are usage.stats.job.aggregation.range and usage.stats.job.exec.time. The first controls the aggregation interval (in minutes) of collected metrics and the second the time the aggregation algorithm kicks in. Remember to restart the usage server service (“cloud-usage”) everytime you play with these variables.

All metrics are stored in a second database, called “cloud_usage”. To see if your usage server really works, connect to that database and see if its tables start to fill (all metrics tables start with “usage_*”). Data can be retrieved from the database, however, a more elegant way is to use the cloudstack API. The most useful API calls are:

  • listUsageRecords: Takes as arguments account, start & end date and returns usage records for the specified time interval.
  • generateUsageRecords: Starts the aggregation process asynchronously

Accessing the API is a breeze: Generate the secret and API keys from the console and pass them as arguments to a python script or a simple wget and target the API port (which is 8080 for plain http, or a designated SSL port).

So, what do you do with all these collected metrics? Well, there are two ways to deal with them. The first is to write a few complex scripts that collect the metrics from the API, sanitize them, implement your billing scheme and export to your reporting tool or ERP to generate invoices.

The second is to use an off the shelf charging/billing solution. As of January 2012, Amysta have a product in beta and Ubersmith offer complete cloudstack billing in their product, Ubersmith DE.

A quick tour of Cloudstack

Cloud.com, now a part of Citrix, has developed a neat, compact, yet powerful platform for cloud management: Enter cloudstack, a provisioning, management and automation platform for KVM, Xen and VMware, already trusted for private and public cloud management frmo companies like Zynga (got Farmville?), Tata communications (public IaaS) and KT (major Korean Service provider).

Recently I had the chance to give cloudstack a spin in a small lab installation with one NFS repository and two Xenservers. Interested in how it breathes and hums? Read on, then.

Cloudstack was installed in a little VM in our production vSphere environment. Although it does support vSphere 4.1, we decided to try it with Xen and keep it off the production ESX servers. Installation was completed in 5 minutes (including the provisioning of the Ubuntu 10.04 server from a ready VMware tremplate) and cloudstack came to life, waiting for us to login:

The entire interface is AJAX – no local client. In fact, cloudstack can be deployed in a really small scale (a standalone server) or in a full-blown fashion, with redundant application and database servers to fulfill scalability and availability policies.

Configuring cloudstack is a somewhat more lengthy process and requires reading the admin guide. We decided to follow the simple networking paradigm, without VLANs and use NFS storage for simplicity. Then, it was time to define zones, pods and clusters, primary and secondary storage. In a nutshell:

  • A zone is a datacenter. A zone has a distinct secondary storage, used to store boot ISO images and preconfigured virtual machine templates.
  • A pod is servers and storage inside a zone, sharing the same network segments
  • A cluster is a group of servers with identical CPUs (to allow VM migration) inside a pod. Clusters share the same primary storage.
We created a single zone (test zone) with one pod and two clusters, each cluster consisting of a single PC (one CPU, 8 GB RAM) running Xenserver 5.6. Configuring two clusters was mandatory, since the two Xenservers were of different architectures (Core 2 and Xeon). After the configuration was finished, logging in to Cloudstack as administrator brings us to the dashboard.

In a neat window, the datacenter status is shown in clear, with events and status in the same frame. From here an administrator has full power over the entire deployment. This is a host (processing node in Openstack terms) view:

You can see the zone hierarchy in the left pane and the virtual machines (instances) running on the host shown in the pane on the right.

Pretty much, what an administrator can do is more or less what Xencenter and vCenter do: Create networks, virtual machine templates, configure hosts and so on. Let’s see how the cloudstack templates look like:

Cloudstack comes with some sample templates and internal system virtual machine templates. These are used internally, but more on them later. The administrator is free to upload templates for all three hypervisor clans (KVM, Xen and Vcenter). For KVM, qemu images, for VMware, .ova and for Xenserver VHD. We created one Windows 2008 server template quite easily, by creating a new VM in Xencenter, installing Xentools and then uploading the VHD file in Cloudstack:

As soon as the VHD upload is finished, it is stored internally in the Zone secondary storage area and is ready to be used by users (or customers).

How does cloudstack look like from the user/customer side? We created a customer account (Innova) and delegated access to our test zone:

Customers (depending on their wallet…) have access to one or more pods and can create virtual machines freely, either from templates of from ISO boot images they have access to, without bringing into the loop cloudstack administrators. Creating a new virtual machine (instance) is done through a wizard. First, select your favorite template:

Then, select a service offering from preconfigured sizes (looks similar to EC2?)

Then, select a virtual disk. A template comes with its own disk (in our case the VHD we uploaded earlier), but you can add more disks to your instances. This can also be done after the instance is deployed.

…and after configuring the network (step 4), you are good to go:

The template will be cloned to your new instance, boot up, and form this point on, you can log in through the web browser – no RDP or VNC client needed!

It’s kind of magic — doing this via an app server seems impossible, right? Correct. Cloudstack deploys silently and automagically its own system VMs that take care of template deployment to computing nodes and storage. Three special kinds of VMs are used:

  • Console proxies that relay to a web browser VNC, KVM console or RDP sessions of instances. One console proxy runs in every zone.
  • Secondary storage VM, that takes care of template provisioning
  • Virtual router, one for every domain (that is, customers), which supplies instances with DNS services, DHCP addressing and firewalling.
Through the virtual router users can add custom firewall rules, like this:
All these system virtual machines are managed directly from cloudstack. Login is not permitted and they are restarted upon failure. This was demonstrated during an unexpected Xenserver crash, which brought down the zone secondary storage VM. After the Xenserver was booted up, the secondary storage VM was restarted automatically by cloudstack and relevant messages showed up in the dashboard. Cool, huh?

Customers have full power over their instances, for example, they can directly interact with virtual disks (volumes), including creating snapshots:

In all, from our little cloudstack deployment we were really impressed. The platform is very solid, all advertised features do work (VM provisioning, management, user creation and delegation, templates, ISO booting, VM consoles, networking) and the required resources are literally peanuts: It is open source and all you need are L2 switches (if you go with basic networking), servers and some NFS storage. Service providers investigating options for their production IaaS platform definitely should look into cloud.com offerings, which has been a part of Citrix since July 2011.