What is Tobiko?
Tobiko is an OpenStack upgrade testing framework. It aims to provide tooling and methods for easily developing upgrade type tests.
Note: At this moment it also provides you with several built-in networking tests.
If you are familiar with OpenStack you might wonder why the current OpenStack testing framework (Tempest) is not used for that purpose.
First of all, that’s an excellent question. Secondly, imagine the following scenario: you would like to set up several different OpenStack resources (instances, routers, networks, …) before upgrading the cloud, run the upgrade process and once the upgrade process is finished, run the same or similar tests to check if the resources you created are still there and working properly. Another scenario is to test your cloud during the upgrade and analyze what happens to the different resources while upgrading your cloud.
Tempest designed to run, test and cleanup a resource once the test is over. If you would try to run the same test before and after an update it would just create and remove the resources in each phase (pre and post) instead of cleaning them only post-upgrade.
Now, let’s move to the practical part of this post and learn how to use it.
How to install Tobiko?
Run the following code in order to install Tobiko in its own (Python) virtual environment
git clone https://review.openstack.org/openstack/tobiko.git cd tobiko && pipenv install .
Verify it works by running a simple command like tobiko-list templates. It should list the Heat templates available in Tobiko
[abregman]$ tobiko-list --templates test_floatingip.yaml test_mtu.yaml
Running tests
Tobiko at this point is coupled to Tempest. It acts as tempest plugin so you execute it the following way
tempest run --config-file etc/tempest.conf --regex tobiko\.\*test_pre
This will run all the pre-upgrade tests. In order to run the post tests, you simply change the regex
tempest run --config-file etc/tempest.conf --regex tobiko\.\*test_post
To run successfully, it will require you to provide a valid tempest configuration file which provides it with input it uses like authentication method and image to use.
For some commands, Tobiko merely requires the authentication info which can be provided with environment variables and doesn’t require you to pass a whole tempest configuration file.
How it works?
Each test class in Tobiko is associated with a stack/template. When a test runs, it uses the template (which is located with other templates in tobko/tests/scenario/templates) to create the stack.
Both stack and template are named the same as the test file. This means that if your tests file called “test_floatingip.py” then the stack name is “test_floatingip” and template name is “test_floatingip.yaml”
If the stack already exists, Tobiko will simply skip the stack creation and proceed to run the tests.
Remember, once the test is over it will not remove the resources/stack. There is a util allows you to remove resources created by Tobiko called ‘tobiko-delete. We’ll cover it later on.
Adding a test
First, add a setUp method that will call the base class (that used by each test class) for creating the stack required by the tests. In order to create the stack, the base class needs the file name (of the test class)
def setUp(self):
super(<YOUR_TEST_CLASS>, self).setUp(__file__)
Next, make sure you have a test for each phase – pre and post. It should be called test_pre_… and test_post_… to match other existing tests included in Tobiko.
Tobiko CLI
Tobiko provides you with tooling in order to ease the process of developing new tests and testing related resources.
tobiko-list
tobiko-list lists the templates defined in Tobiko
tobiko-list --templates test_floatingip.yaml test_security_groups.yaml
It can also be used to list the stacks in your OpenStack project but only those that related to Tobiko (either created by it or match a template name defined in the project)
tobiko-list --stacks test_floatingip
tobiko-create
If you would like only to create the stacks defined in Tobiko and not run the tests, you can use tobiko-create for that purpose.
You can either specify a specific stack the following way
tobiko-create -s test_floatingip
Or create all stacks
tobiko-create --all
tobiko-delete
Once you finished testing/developing, you can use tobiko-delete to clean up resources created during your development/testing. Similarly to tobiko-create you can delete a specific task
tobiko-delete -s test_floatingip
Or all stacks
tobiko-delete --all
Don’t worry about deleting all the stacks in your project. It will only delete stacks created by Tobiko or those that match templates names.
Contributions
Tobiko code review is managed via OpenStack Gerrit system. Follow OpenStack Developer guide to submit patches. The short version is: clone the project, commit your changes and run git-review to submit your patch for a review. The long version is well documented by OpenStack.
