Tempest: An OpenStack Integration test suite

Integration Testing:

• Integration Testing is the phase in software testing in which individual software modules are combined and tested as a group.
• It occurs after unit testing and before system testing.
• Tests the actual functionality of the application/software you have written.

Why wasting time in integration tests when I have an option to check the functionality manually?

• Because every time you add a new feature you will have to check and re-check the functionalities manually which is tedious, time-consuming and error-prone.
• What if you have thousands of features, certainly you can’t afford to check if them one by one.
• If you claim that your software is platform-independent. How will you make sure that all the functionalities of your software are intact on multiple platforms? That’s where a ‘test-suite’ comes into picture. You can create a dedicated test environment to run your test-suite across different platforms adhering to your scale and stress needs.

I have written unit tests properly. Why should I write integration tests?

• Because unit-tests test the source code you have written, the semantic and the syntax, not the actual functionality of your software.
• Integration tests make sure that your source code, when integrated with all your components, serves the real purpose.
• With addition of a new feature you can just re-trigger the suite and hence make sure that your new feature doesn’t break any existing functionality.

What is tempest?

• One of Tempest’s prime function is to ensure that your OpenStack cloud works with the OpenStack API as documented. The current largest portion of Tempest code is devoted to test cases that do exactly this.
• It’s also important to test not only the expected positive path on APIs, but also to provide them with invalid data to ensure they fail in expected and documented ways. Over the course of the OpenStack project Tempest has discovered many fundamental bugs by doing just this.
• In order for some APIs to return meaningful results, there must be enough data in the system. This means these tests might start by spinning up a server, image, etc., then operating on it.
• Currently tempest has support to test both APIs and CLIs.

How to run the tempest tests?

• There are multiple ways to run the tests. You have a choice to run the entire test suite, all the tests in a directory, a test module or a single tests.
• The documents describes how to run the tempest tests using nosetests.
• Install the basic dependencies:
  • apt-get install pip (if you have Ubuntu)
  • pip install nosetests
  • pip install testresources
  • git clone http://github.com/openstack/tempest
    
• Go to the directory tempest and execute the tests with the following command;
  • nosetests -sv tempest.tests.identity.admin.test_services.py [ This would run all the tests inside the module test_services.py]
  • nosetests -sv tempest.tests.identity.admin.test_services:ServicesTestJSON.test_create_get_delete_service [This runs a specific tests inside the class ServiceTestJSON of the module test_services.py]

Basic Architecture

• Tempest tests can be executed from either outside the openstack setup or from the openstack controller itself.
• The API tests are based on rest. (http://ilearnstack.com/2013/04/25/understanding-rest-implementation/)
  • The tempest has a rest client implemented in the suite which takes care of the REST operations.
  • All the user inputs which are performing the rest operationon nova apis are read from tempest.conf file.
  • The suite first make a rest call to the keystone api to get the auth token and tenant details for authorization and authentication.
  • Using the auth token obtained from the response, the tests performs the required operations.

More to follow…

• Debugging using eclipse..
• Extending tempest framework for your project..

Unit-testing

What is unit-testing?

• Unit testing is a method by which individual units of source code are tested to determine if they are fit for use.
• Also a means of automatic code walkthrough.

Why unit-testing?

• It makes sure that you code is syntactically error-free
• It helps in resolving complex logical issues.
• It leads to modular, extensible code that is easy to test, understand, and maintain.
• If you stack together enough guesses, you may eventually build something that appears to function, but that no human could ever say with certainty ever worked properly.

“With a dynamic language like Python, Perl, or Ruby, it is easy to develop software by simply banging away at the problem, often interactively, until you get what seems to be the correct result and calling it a day.
Unfortunately, this approach, while tempting, often leads to a false sense of accomplishment that is fraught with danger.Much of the danger lies in not designing a solution to be testable, and part lies in not properly controlling the complexity of the software written. If you don’t have tests, you don’t know if your software works!! PERIOD”

Unit tests framework for python – pyunit

• Pyunit : A unit testing framework for python like JUnit for java
• unittest supports test automation, sharing of setup and shutdown code for tests, and independence of the tests from the reporting framework. The unittest module provides classes that make it easy to support these qualities for a set of tests.
• All your pyunit test class should extend the unittest.TestCase :
• It supports some important concepts like:
  • setup()
  • teardown()
  • setupUpClass()
  • tearDownClass()

setup():

• This is called immediately before calling the test method.
• It will be called each time the test_method is called.
• Exists to make sure that each method has its own environment to work on with and hence the each test_methods can be executed independently.
• Any exception raised by this method will be considered an error rather than a test failure

teardown():

• This is called immediately after the test method has been called.
• It will be called each time the test_method is called.
• Exists to make sure that after a test method has been executed the environment is destroyed so the next method to be executed can have a clean environment to work on with.
• Any exception raised by this method will be considered an error rather than a test failure.
• This method will only be called if the setUp() succeeds, regardless of the outcome of the test method.

setUpClass():

• This is called before tests in an individual class run.
• setUpClass is called with the class as the only argument and must be decorated as a classmethod():

  @classmethod
  

  def setUpClass(cls):
  

tearDownClass():

• This is called after tests in an individual class run.
• setUpClass is called with the class as the only argument and must be decorated as a classmethod():

  @classmethod
  

  def tearDownClass(cls):
  

A Simple Illustration

	# A class performing basic maths
	class Do_Mathematics:
	def __init__(self, a, b):
	self.a = a
	self.b = b
	def sum(self):
	return self.a+self.b
	def sub(self):
	return self.a-self.b

view raw

do_maths_simple.py

hosted with ❤ by GitHub

	#A simple test class to test the DoMathematics class
	class Test_do_maths_simple(unittest.TestCase):
	def setUp(self):
	self.do_maths = Do_Mathematics(10, 20)
	def tearDown(self):
	del self.do_maths
	def test_sum(self):
	result = self.do_maths.sum()
	self.assertEqual(result, 30, "Sum method is incorrect")
	def test_sub(self):
	result = self.do_maths.sub()
	self.assertEqual(result, -10, "Sub method is incorrect")

view raw

test_my_do_mathematics_simple.py

hosted with ❤ by GitHub

What if the method which I am testing calls other methods inside?

• Mocking the method is a solution to that.
• Mocking a method means you are faking the call and returning a desired value for the method being mocked without executing the method.
• The method to be mocked should exists in real, you can’t mock an imaginary method or class.
• You can check if the mocked method is actually put into use or not by putting a VerifyAll call. If the mocked method is not being used VerifyAll will let the test to fail.

By mocking am I comprising with the quality?

• Absolutely not.
• You mock a method to make sure a piece of source code you are testing is sane. If the piece of code is dependent on some other piece of code you should write different unit tests for checking their sanity.

	'''
	Created on Apr 25, 2013
	@author: abhsriva
	'''
	class Do_Mathematics:
	def __init__(self, a, b):
	self.a = a
	self.b = b
	def sum(self):
	return self.a+self.b
	def sub(self):
	return self.a-self.b
	def check_greater(self):
	if self.a > self.b:
	print "a is greater"
	else:
	print "b is greater"
	def _check_if_zero(self, num):
	if num == 0:
	return True
	return False
	def div(self):
	if self._check_if_zero(self.b):
	print 'Can\'t divide'
	else:
	return float(self.a)/self.b

view raw

do_maths.py

hosted with ❤ by GitHub

	'''
	Created on Apr 25, 2013
	@author: abhsriva
	'''
	import unittest
	import mox
	from class_to_be_tested import Do_Mathematics
	class Test_Do_Mathematics_class(unittest.TestCase):
	def setUp(self):
	print 'In setup', self.x
	self.mock = mox.Mox()
	self.do_maths = Do_Mathematics(10, 20)
	def tearDown(self):
	print 'In tear down', self.x
	del self.do_maths
	def test_sum(self):
	result = self.do_maths.sum()
	self.assertEqual(result, 30, "Sum method is incorrect")
	def test_sub(self):
	result = self.do_maths.sub()
	self.assertEqual(result, -10, "Sub method is incorrect")
	def test_div(self):
	self.mock.StubOutWithMock(Do_Mathematics, '_check_if_zero')
	self.do_maths._check_if_zero(mox.IgnoreArg()).AndReturn(False)
	self.mock.ReplayAll()
	result = self.do_maths.div()
	self.mock.VerifyAll()
	self.assertEqual(result, 0.5, "Div method is incorrect")
	def test_check_greater_with_zero(self):
	self.assertTrue(self.do_maths._check_if_zero(0))
	def test_check_greater_non_zero(self):
	self.assertFalse(self.do_maths._check_if_zero(10))
	if __name__ == "__main__":
	unittest.main()

view raw

test_do_maths_with_mox.py

hosted with ❤ by GitHub

Basics of Python

This is a basic python tutorial (basic enough to let you start with OpenStack)

Topics covered

1. Introduction
2. Basic Syntax
3. Data types and variables
4. Loops and conditionals
5. Functions and Methods
6. Classes and objects
7. Python Specifics

Introduction

1. Python is an easy to learn, powerful programming language
2. Dynamic type
3. Interpreted
4. Can be extended using C or C++
5. Comes with a large standard library that covers areas such as string processing (regular expressions, Unicode, calculating differences between files), Internet protocols (HTTP, FTP, SMTP, XML-RPC, POP, IMAP, CGI programming), software engineering (unit testing, logging, profiling, parsing Python code), and operating system interfaces (system calls, filesystems, TCP/IP sockets).
6. Standard modules and packages from : http://pypi.python.org/pypi
7. Download and install from here: http://www.python.org/download/releases/2.7.3/
8. FAQ on python: http://docs.python.org/2/faq/general.html
9. Python Success Stories: http://www.python.org/about/success/

Basic Syntax

1. Files should have the extension “.py”. They are called modules
   e.g.
   my_first_program.py
2. The file can have any name, any number of classes and methods.
3. There are no braces {} or semicolons ; in python. It is a very high level language. Instead of braces, blocks are identified by having the same indentation.
4. The program execution starts from main.
5. if the python interpreter is running that module (the source file) as the main program, it sets the special __name__ variable to have a value ”__main__”. If this file is being imported from another module, __name__ will be set to the module’s name.
6. Comments start with ‘#’ symbol, and there no block comments in python

e.g.




	if __name__=="__main__":
	print "Hello World"
	print "This is out of the if block"
	#This is a comment
	#This is also a comment
	print "Hello World"

view raw

BlockDesc1.py

hosted with ❤ by GitHub

Data types and variables

1. Variables do not need to be declared and their data-types are inferred from the assignment statement.

e.g.




	A = 'string var'
	A = 24
	A = 24.0

view raw

SimpleVars.py

hosted with ❤ by GitHub

2. Python supports the following data types:
   • Boolean:
     • True
     • False
   • Integer
   • long
   • float
   • String
   • List:
     • A list is a mutable sequence of mixed data type.
     • mix = [1, 2, "solaris", (1, 2, 3)]
   • Tuple:
     • A tuple is an immutable sequence of mixed data type.
     • mix = (1, 2, "solaris", (1, 2, 3))
     • return (3 + 7)# This is an expression
     • return (3 + 7, ) # This is a tuple
   • Set:
     • A set is an unordered collection of data with no duplicate elements.
     • A set supports operations like union, intersection or difference.
     • set1 = set([‘a’, ‘b’, ‘c’, ‘c’, ‘d’])
     • set2 = set([‘a’, ‘b’, ‘x’, ‘y’, ‘z’])
     • set supports operations like union, intersection or difference. Similar as in Mathematics.
   • Dictionary:
     • A Python dictionary is a group of key-value pairs. The elements in a dictionary are indexed by keys. Keys in a dictionary are required to be unique.
     • Diction = {1: 'cat', 2:'apple'}
   • Object
   • None

Loops and conditionals

1. if:
   if expr: statement
2. if-else:
   
if expr: statement1
else: statement2
if-elseif: if expr: statement1
elif expr: statement2
else: statement3

3. Multiple elifs can be included in the same if statement. There is no switch or case statement so multiple elifs must be used instead. While parenthesis are not required around the expression, they can be used.




	if a > b: print "a is greater than b";
	if (a > b):
	print "a is greater than b"
	print "blocks are defined by indentation"
	elif (a < b):
	print "a is less than b"
	else:
	print "a is equal to b"

view raw

SimpleCon.py

hosted with ❤ by GitHub

4. Loops
   • For:
     for var in range(start [,stop [,inc]]): statements
     var can be any variable. The range statement can take start and stop values, and an increment.
   • while:
     
while expr: statements

     Executes statements while the expression is true.
   • continue: continue
     Skips the rest of the body of the loop for the current iteration and continue execution at the beginning of the next iteration.
   • break: break
     Ends the execution of the current loop.
   • else: else
     for and while loops can both have else clauses, which are executed after the loop terminates normally by falsifying the conditional, but else clauses are not executed when a loop terminates via a break statement.
   • foreach: for x in array: statements
     Loops over the array given by array. On each iteration, the value of the current element is assigned to x and the internal array pointer is advanced by one.

   
   

   	value = ["abhi", "cena", "dhoni", "bruce wayne"]
   	for j in range(4): print "Value number " + str(j) +" is "+value[j]
   	x=0
   	for j in range(10,0,-2):
   	x = x + j
   	print x
   	b=1
   	a=5
   	while (b<a):
   	print "b is less than a."
   	b=b+1
   	else:
   	print "b is equal to a now"
   	k=0
   	for j in range(0,10):
   	while(k < j):
   	print "j = " + str(j) + " k = "+str(k)
   	if (j == 1): break
   	k=k+1
   	print "j equals k or j equals 1"
   	a = ["abc","def","ghi"]
   	for x in a:
   	print x

   view raw
   
   SimpleLooping.py
   
   hosted with ❤ by GitHub

Functions

1. Definition: Functions in Python are defined with the following syntax:
   
def myfunct(arg_1, arg_2, ..., arg_n):
return value

2. Any Python code, including other function and class definitions, may appear inside a function. Functions may also be defined within a conditional, but in that case the function’s definition must be processed prior to its being called.
3. Python does not support function overloading but does support variable number of arguments, default arguments, and keyword arguments.
4. Return types are not specified by functions.
5. Arguments: Function arguments are passed by value so that if you change the value of the argument within the function, it does not get changed outside of the function. If you want the function to be able to modify non-local variables, you must declare them as global in the first line of the function. Note that if you declare any variables as global, that name cannot be reused in the argument list, i.e. this would cause an error:




	def double(x):
	global x
	x = x*2
	return
	double(x)
	#Instead this could be done
	def double(n):
	n = n * 2
	return n
	x = double(x)
	# Or
	function doubleX():
	global x
	x = x * 2
	return
	doubleX()

view raw

SimpleMethodCall.py

hosted with ❤ by GitHub

6. Default Arguments: A function may define default values for arguments. The default must be a constant expression or array and any defaults should be on the right side of any non-default arguments.
   
def person_name(n = 'Abhi'):
return n + " Rocks!!"

   If this function is called with person_name(), it will return Abhi Rocks!!, Otherwise, if it is called with person_name(“John”), it will return John Rocks!!
7. Variable length argument lists: Variable length arguments are supported by being wrapped up in a tuple. Before the variable number of arguments, zero or more normal arguments may occur:The special syntax, *args and **kwargs in function definitions is used to pass a variable number of arguments to a function. The single asterisk form (*args) is used to pass a non-keyworded, variable-length argument list, and the double asterisk form is used to pass a keyworded, variable-length argument list.

e.g.




	def var_args(farg1, farg2, *args, **kargs):
	print "formal arg:", farg1
	print "formal arg:", farg2
	print 'Length of', len(args)
	for arg in args:
	print 'vararg:' , arg
	for key in kargs:
	print key
	sample_keyarg = {1:’abhi’, 2:’smart’}
	var_args(1,2, sample_keyarg, “abhi”, “kk”, “jj”, (“r”, “p”), ram = 85, shyam=45 )
	#Output:
	#formal arg: 1
	#formal arg: 2
	#Length of args 5
	#vararg: {1: 'abhi', 2: 'smart'}
	#vararg: abhi
	#vararg: kk
	#vararg: jj
	#vararg: ('r', 'p')
	#ram
	#shyam

view raw

MethodArgs.py

hosted with ❤ by GitHub

8. Return: Values are returned from the function with the return command: return var. You can’t return multiple values, but that can be achieved by returning an array or object.

Return immediately ends execution of the function and passes control back to the line from which it was called.

1. Variable Functions: Python supports the concept of variable functions. That means that if a variable can point to a function instead of a value. Objects within a method can be called similarly.




	def test():
	print 'This is a test.'
	var = test
	var() #this calls test()
	var = circle.setRadius
	var(3) #this calls circle.setRadius(3)

view raw

VarFun.py

hosted with ❤ by GitHub

Classes and Objects

1. The simplest form of class definition looks like this:
   class ClassName:
   <statement-1>
   .
   .
   .
   <statement-N>
2. The class instantiation calls the __init__ method of the class which can be over written:




	class MyClass:
	"""A simple example class"""
	def __init__(self):
	self.i = 12345
	def f(self):
	return 'hello world'

view raw

SimpleClass.py

hosted with ❤ by GitHub

3. The argument self tells that method is a class method




	# Function defined outside the class
	def f1(self, x, y):
	return min(x, x+y)
	class C:
	f = f1
	def g(self):
	return ‘hello world’
	h = gprint type(C)
	print type(C())
	my_new_obj = C()
	print my_new_obj.f(23,45)
	print my_new_obj.g()
	#Output:
	#<type 'classobj'>
	#<type 'instance'>
	#23
	#hello world

view raw

InlineMethod.py

hosted with ❤ by GitHub

4. Inheritance:class DerivedClassName(BaseClassName):
   <statement-1>
   .
   .
   .
   <statement-N>

Python Specifics

1. Packages
   • Packages are a way of structuring Python’s module namespace by using “dotted module names”.
   • Every package has __init__.py module which is empty bydefault.
   • The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable.
2. Importing * From a Package: Will import only those modules which have been listed in __all__ variable in the __init__.py module of the package.
   e.g. __all__ = ['mod1', 'mod2']
3. Decorators:
   • Decorators allow you to inject or modify code in functions or classes.
   • The actual method is compiled with the decoration and is substituted
   • A simple e.g. to illustrate decorator

   
   

   	def entryExit(f):
   	def new_f():
   	print "Entering", f.__name__
   	f()
   	print "Exited", f.__name__
   	new_f.__name__ = f.__name__
   	return new_f
   	@entryExit
   	def func1():
   	print “inside func1()
   	@entryExit
   	def func2():
   	print “inside func2()”func1()
   	func2()
   	print (“*”)*40
   	#without decorator
   	def func1_wd():
   	print “inside func1()”
   	def func2_wd():
   	print “inside func2()”
   	fwd1 = func1_wd
   	fwd2 = func2_wd
   	entryExit(fwd1)()
   	entryExit(fwd2)()

   view raw
   
   SimpleDecorator.py
   
   hosted with ❤ by GitHub