OpenStack: Testing Upgrades with Tobiko

What is Tobiko?

Tobiko is an OpenStack upgrade testing framework. It aims to provide tooling and methods for easily developing upgrade type tests.

Note: At this moment it also provides you with several built-in networking tests.

If you are familiar with OpenStack you might wonder why the current OpenStack testing framework (Tempest) is not used for that purpose.

First of all, that’s an excellent question. Secondly, imagine the following scenario: you would like to set up several different OpenStack resources (instances, routers, networks, …) before upgrading the cloud, run the upgrade process and once the upgrade process is finished, run the same or similar tests to check if the resources you created are still there and working properly. Another scenario is to test your cloud during the upgrade and analyze what happens to the different resources while upgrading your cloud.

Tempest designed to run, test and cleanup a resource once the test is over. If you would try to run the same test before and after an update it would just create and remove the resources in each phase (pre and post) instead of cleaning them only post-upgrade.

Now, let’s move to the practical part of this post and learn how to use it.

How to install Tobiko?

Run the following code in order to install Tobiko in its own (Python) virtual environment

git clone
cd tobiko && pipenv install .

Verify it works by running a simple command like tobiko-list templates. It should list the Heat templates available in Tobiko

[abregman]$ tobiko-list --templates


Running tests

Tobiko at this point is coupled to Tempest. It acts as tempest plugin so you execute it the following way

tempest run --config-file etc/tempest.conf --regex tobiko\.\*test_pre

This will run all the pre-upgrade tests. In order to run the post tests, you simply change the regex

tempest run --config-file etc/tempest.conf --regex tobiko\.\*test_post

To run successfully, it will require you to provide a valid tempest configuration file which provides it with input it uses like authentication method and image to use.

For some commands, Tobiko merely requires the authentication info which can be provided with environment variables and doesn’t require you to pass a whole tempest configuration file.

How it works?

Each test class in Tobiko is associated with a stack/template. When a test runs, it uses the template (which is located with other templates in tobko/tests/scenario/templates)  to create the stack.

Both stack and template are named the same as the test file. This means that if your tests file called “” then the stack name is “test_floatingip” and template name is “test_floatingip.yaml”

If the stack already exists, Tobiko will simply skip the stack creation and proceed to run the tests.

Remember, once the test is over it will not remove the resources/stack. There is a util allows you to remove resources created by Tobiko called ‘tobiko-delete. We’ll cover it later on.

Adding a test

First, add a setUp method that will call the base class (that used by each test class) for creating the stack required by the tests. In order to create the stack, the base class needs the file name (of the test class)

def setUp(self):
        super(<YOUR_TEST_CLASS>, self).setUp(__file__)

Next, make sure you have a test for each phase – pre and post. It should be called test_pre_… and test_post_… to match other existing tests included in Tobiko.

Tobiko CLI

Tobiko provides you with tooling in order to ease the process of developing new tests and testing related resources.


tobiko-list lists the templates defined in Tobiko

tobiko-list --templates


It can also be used to list the stacks in your OpenStack project but only those that related to Tobiko (either created by it or match a template name defined in the project)

tobiko-list --stacks



If you would like only to create the stacks defined in Tobiko and not run the tests, you can use tobiko-create for that purpose.

You can either specify a specific stack the following way

tobiko-create -s test_floatingip

Or create all stacks

tobiko-create --all


Once you finished testing/developing, you can use tobiko-delete to clean up resources created during your development/testing. Similarly to tobiko-create you can delete a specific task

tobiko-delete -s test_floatingip

Or all stacks

tobiko-delete --all

Don’t worry about deleting all the stacks in your project. It will only delete stacks created by Tobiko or those that match templates names.


Tobiko code review is managed via OpenStack Gerrit system. Follow OpenStack Developer guide to submit patches. The short version is: clone the project, commit your changes and run git-review to submit your patch for a review. The long version is well documented by OpenStack.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s