Monthly Archives: January 2012

Usage metering and charging with Cloudstack

One of the prominent features of an IaaS cloud is that one can meter its resource usage by its consumers. Metrics are everywhere: From the hypervisor, virtual disk size, network I/O, occupied IP addresses, virtual CPUs and RAM, they are all over the place, waiting to be collected. As soon as you can grab a handful of metrics, you can implement chargeback policies and report back on your users on their resource consumption or, if you run a public IaaS shop, somehow transform these metrics to invoices.’s cloudstack comes with an excellect usage server, recording metrics directly from its accounts. During installation, simply select the “Install Usage Server” option and perform some basic configuration, and you are all set to go. The usage server collect no less than thirteen (as of cloudstack release 2.2) metrics, which can be found here. In short, some of the most important ones are:

  • RUNNING_VM: Total hours a virtual machine is started and running on the hypervisor
  • ALLOCATED_VM: Total hours a VM exists (no matter if it’s up or down). Useful parameter for charging OS license usage, for example Microsoft SPLA licenses.
  • IP_ADDRESS: Self evident; applies to public (Internet) IP addresses consumed by a cloudstack account. These addresses are (according to cloudstack architecture) attached to the virtual router of the user
  • NETWORK_BYTES_SENT and NETWORK_BYTES_RECEIVED: Traffic passing through the virtual router of a user
  • VOLUME: Size in bytes of user volumes
  • TEMPLATE and ISO: Size in bytes of user-uploaded VM templates and ISO images
(For those who are not familiar with cloudstack’s architecture, cloudstack users are part of accounts. Virtual machines belonging to a single account live in their own private VLAN, totally isolated from other accounts. Access to the Internet, DHCP addressing, DNS and VPN termination, all take place in a special cloudstack virtual machine, a virtual router. Every account has its own virtual router, not directly controlled by the end user, but via the cloudstack API).

The service (“cloud-usage”) starts along with the rest cloud services on your cloudstack controller and its configuration variables are at the global parameters of cloudstack. The most important are usage.stats.job.aggregation.range and usage.stats.job.exec.time. The first controls the aggregation interval (in minutes) of collected metrics and the second the time the aggregation algorithm kicks in. Remember to restart the usage server service (“cloud-usage”) everytime you play with these variables.

All metrics are stored in a second database, called “cloud_usage”. To see if your usage server really works, connect to that database and see if its tables start to fill (all metrics tables start with “usage_*”). Data can be retrieved from the database, however, a more elegant way is to use the cloudstack API. The most useful API calls are:

  • listUsageRecords: Takes as arguments account, start & end date and returns usage records for the specified time interval.
  • generateUsageRecords: Starts the aggregation process asynchronously

Accessing the API is a breeze: Generate the secret and API keys from the console and pass them as arguments to a python script or a simple wget and target the API port (which is 8080 for plain http, or a designated SSL port).

So, what do you do with all these collected metrics? Well, there are two ways to deal with them. The first is to write a few complex scripts that collect the metrics from the API, sanitize them, implement your billing scheme and export to your reporting tool or ERP to generate invoices.

The second is to use an off the shelf charging/billing solution. As of January 2012, Amysta have a product in beta and Ubersmith offer complete cloudstack billing in their product, Ubersmith DE.


Bring out your (virtual) dead

Virtualization is cool. Especially for someone that was used to racking, cabling, imaging, installing apps, adding memory, swapping disks powering on/off and the rest. Jumping to vCenter (or XenCenter or virtmanager) is like scuttling your sailboat and getting aboard a hovercraft. We all love it!

Yet, as soon as you pass the thirty-forty virtual machine mark, something happens. Quite naturally, your virtual infrastructure goes into yellow, then orange, then red. Memory utilization is the first to suffer, then your thin provisioning disk array starts to fill up… Days before your virtual farm shuts down or refuses to deploy new virtual machines, you realize that lots of virtual servers are useless – and orphans: You don’t know why they are there.

This is called virtual sprawl. You have lots of “virtual assets” taking up CPU, memory, IP addresses and disk blocks for nothing; they just sit there. Worst, you don’t touch them because they are not yours. How do you clean up?

Oh, he is dead - No, I am not - Yes, you are

How do you control who has the right to create a VM, how long it is used and when it is time to let it go? Virtual infrastructure management tools usually leave the VM lifecycle management aside: What they do best is managing the hypervisors, not your IT policies and business procedures. Options?

The answer here is “private cloud management”. What is required is an automation layer on top of your hypervisor and virtual centers, which implements business procedures, capacity planning, virtual lifecycle management and end user self service portals. A nifty tool is Embotics vCommander, sitting atop your VMware infrastructure and letting your internal customers order a VM, have it approved by their supervisor, charged for its use and specify a decomissioning date. In addition, it can do capacity planning and let you know when you will be out of infrastructure.

Another option is enterprise private/public cloud platform. Abiquo is one of them, able to span across all hypervisors (HyperV, VMware, Xen, Oracle VirtualBox and KVM) with fine grained user control and resource tiering.

Whatever you choose, it is a good practice to have your virtual users accountable for the virtual assets they own and use. Adding a note on their virtual machine summary in vCenter or utilizing a private cloud controller, it is a practice that will pay off in the mid term.