OpenStack-Cinder: create volume data/code flow

Hello Folks,

I guess, its been quite some time since my last post. Actually, didn’t get time some free time cover up this post.

Thanks to dear friend Rahul Updadhyaya, he has pull me out  today to write this up 🙂

Apologize for not putting diagrams right now, will work on it sooner.

This post is about cinder create volume data/code flow.

Example: cinder create 1 –display-name firstvol –volume-type FCVolumeType

1. Start with calling the cinder api:

File: cinder/api/v1/volumes.py

Method: def create(self, req, body): #validates cinder api arguments.

2. Above method calls “cinder.volume.api.API”, taken from  “volume_api_class” a flag from cinder.conf.

self.volume_api.create() # here self.volume_api created from “cinder.volume.api.API”

File: cinder.volume.api.API

Method: def create(self….)

This function store necessary data into database with volume status as ‘creating’.

3. Above method further calls cinder schedule.

self.scheduler_rpcapi.create_volume() # Making asynchronous cast on cinder-scheduler

4. Next asynchronous cast makes it to the cinder/scheduler/manager.py def create_volume()

this in turn calls: self.driver.schedule_create_volume() #here, the self.driver points to scheduler_driver flag in cinder.conf

Now, this could be SimpleScheduler or FilterScheduler (in case of multi-backend)

5. Incase of SimpleScheduler, above  method calls

File: cinder/scheduler/simple.py

Method: def schedule_create_volume()

above method next calls: self.volume_rpcapi.create_volume() # which makes asynchronous cast on selected host

Here, you can view host info with #cinder-manger host list

6. Message is reached to volume/manager.py

File: cinder/volume/manager.py

Method: def create_volume() # calls _create_volume() and make call to volume driver’s create_volume()

self.driver.create_volume() # driver is chosen with volume_driver flag from cinder.conf, this return metadata about volume

And volume status is changed from ‘creating’ to ‘available’.

Debug Openstack code Local / Remote with Eclipse and PyDev Plug-In

This article is a result of my exhaustive search for finding a concrete way to debug Openstack. After referring several places, I have come up with a manual of my own on how to setup eclipse environment to debug and understand openstack code flow. It should be a good read if you have similar questions as posted below in your mind.

• Is it possible to debug openstack code end-to-end?
• Should I debug locally (everything configure inside eclipse)?
• How to debug remotely running openstack services?Or, combination of above two?
• What developer tools/IDEs to use for debugging? (eclipse +pydev, pdb, winpdb, pycharm)?
• What’s the best/easiest/more sophisticated method, to get set everything quickly?

And there’s bunch of other questions, followed by multiple alternatives to chose from.

Here in this post, I have tried debugging using Eclipse with pydev plug-in.

Development Environment:

Linux Distro: centos/ ubuntu, (I used VM workstation)

Install eclipse as per os type 32/64 bit on one of the VM: http://www.eclipse.org/

Install pydev plugin: http://pydev.org/manual_101_install.html

Configure python Interpreter in eclipse.

Install git plugin (only for local debug): http://download.eclipse.org/egit/updates, add this in Help-> install new software

How to  Debug Openstack Services Locally?

To begin with, you can try with keystone in eclipse.

Check out: http://wiki.openstack.org/Setup%20keystone%20in%20Eclipse

Also, setup environment variables under debug configuration for keystone service to pick up.

OS_USERNAME,

OS_PASSWORD,

OS_TENANT_NAME,

OS_REGION_NAME,

OS_AUTH_URL

Optionally, Setup keystone.conf file as argument under debug configuration dialog.

For example, to test setup, put a break-point at:

File: keystone/identity/core.py

Method: def get_all_tenants(self, context, **kw):

Now, execute keystone-all (debug as-> python run) from eclipse

As you have already install keystoneclient by following above link, from terminal execute:

$keystone tenant-list

(check db is running, iptables service not blocking port – just in case if get 500 error with tenant-list)

This should hit break-point in keystone service running in eclipse and ask to move to debug perspective.

Voila, You have just got setup for local debugging.

Remote Debugging: 

Development Environment: 

In this case, I have used two VMs, one is centos and other is ubuntu 12.04.

Ubuntu VM- running  eclipse IDE with pydev plug-in.

Centos VM –  openstack services running.

Configure python Interpreter in eclipse.

Configure pydev debug server in eclipse.

To Remote debug, following link has most of the answers:

http://pydev.org/manual_adv_remote_debugger.html

Now, copy /pysrc directory from ubuntu vm to centos vm.

/pysrc – will be found in eclipse installation plugins/org.python.pydev_<version>/pysrc

On centos (Remote machine), preferred place to copy under python site-package.

Ex: /usr/lib/python2.6/site-packages/pysrc/

Example-1: Remote debug keystone

Run the debug server in eclipse, note the debug server port.

File: keystone/keystone/identity/core.py

Function: def get_all_tenants(self, context, **kw):  # gives tenant-list

Under this function add line:

import pydevd;pydevd.settrace(<IP addr of eclipse vm>, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

Next,File: /keystone/bin/keystone-all

To add pysrc to PYTHONPATH:  add following line after “import sys” line

sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=monkeypatch_thread)

Comment out above this line, and add following line:

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=False)

This most important for debugging, otherwise you will received “ThreadSuspended” error in eclipse.

As, the debug server listen to single thread, above line will take away green threading of thread module.

Restart keystone service

$service keystone restart

$keystone tenant-list

On eclipse, switch to debug perspective.

You should be able to hit break-point in core.py file, and step through further debug execution.

Example-2: Debugging keystone(get auth-token) + nova-api

$nova flavor-list                #will debug this cli

File: /keystone/keystone/service.py
Class: class TokenController
Method:def authenticate(self, context, auth=None):

Add following line:
import pydevd;pydevd.settrace(‘<IP addr of eclipse vm>’, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

Next,File: /keystone/bin/keystone-all

To add pysrc to PYTHONPATH:  add following line after “import sys” line

sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=monkeypatch_thread)

Comment this line, and add following line:

eventlet.patcher.monkey_patch(all=False, socket=True, time=True, thread=False)

This most important for debugging, otherwise you will received “ThreadSuspended” error in eclipse.

As, the debug server listen to single thread, above line will take away green threading of thread module.

Restart keystone service

Next, File: nova/nova/api/openstack/compute/flavors.py

Class: class Controller(wsgi.Controller):

Method:

@wsgi.serializers(xml=FlavorsTemplate)

def detail(self, req):

Add following line under this function:

import pydevd;pydevd.settrace((<IP addr of eclipse vm>, port=8765, stdoutToServer=True, stderrToServer=True,suspend=True)

File: nova/bin/nova-api

Add line after “import sys”

sys.path.append(‘/usr/lib/python2.6/site-packages/pysrc/’)

eventlet.monkey_patch(os=False)

comment above line, and change to:

eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)

$service keystone restart

$service nova-api restart

$ nova flavor-list

In eclipse, this should hit break-point in service.py for keystone

After keystone token generated, control move to flavors.py

Things observed:

Path resolution:

If python paths and/or openstack code paths different on both VMs, eclipse will not be able to locate correct file to open, and respond with open file dialog, just cancel the dialog and file from remote machine get displayed. This file will get store into prsrc/temporary_file directory.
To avoid this, on server running openstack service, go to pysrc directory, and modify the file, pydevd_file_utils.py.

More info on this: http://pydev.org/manual_adv_remote_debugger.html

The whole idea for this blog post is to try out alternatives to debug openstack code.

I have taken simplest possible examples in very short time, to demonstrate, it works!!

Setting up a Single Node Openstack Environment

Here we are going to deploy openstack on a single node, i.e all openstack components Nova, Glance, Keystone, Quantum and Horizon will be running in the same box. Nova-compute is also running on the same machine which will provision virtual machines on the KVM.

Requirements:

Ubuntu 12.04 running on Oracle VirtualBox with two NIC ‘s.

• Setup a ubuntu virtual machine in virtualBox.
• Add a second NIC to the VM.
  • Power Off the Virtual Machine
  • Right Click on the VM and select settings
  • In the Network Section select Adapter 2 tab
  • Select “Attached to:” to internal

• Power On the VM
• When the VM boots up configure to network for second NIC
• Edit /etc/network/interfaces to something like

• Restart Network service

service networking restart

• You can verify your network setting using “ ifconfig ”

Deploying Openstack

• Login to the ubuntu VM.
• Change the user to root.

sudo -i

• If your network is behind proxy then please export proxy variables.

export http_proxy=<proxy-server>
export https_proxy=<proxy-server>
export no_proxy=”localhost,127.0.0.1″

•  Install git on the machine

apt-get update -y

apt-get install git -y

• Clone the CloudGear git to install Openstack

git clone https://github.com/ilearnstack/cloudgear.git

• Execute the script to set up openstack

cd cloudgear/
python cloudgear.py

• Open Openstack Dashboard in browser from URL  http://<controller-ip>/horizon
• Login with credentials  admin/secret

• Add a image into Glance

                     Download image from the given link :http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img

• Create Network for Virtual Machines.

• Create subnet for the network created above.

• Launch a instance from the Project Tab , Instances page.

• Once VM is successfully spawned it will show as ACTIVE in Instances page. 

• Select Console tab to view VM console.

Setting up Ubuntu virtual machine in Oracle Virtualbox

VirtualBox is a free virtualization software offered by oracle for running virtual machines inside it. We are going to walk through creating a virtual machine inside virtualbox and installing Ubuntu server 12.04 LTS (Precise Pangolin) which comes with Openstack Folsom release .

Step 1: Creating a new virtual machine in virtulbox.

1. Download and install Oracle Virtualbox from https://www.virtualbox.org/wiki/Downloads .
2. Open Virtualbox .
3. Click “New” button to create new virtual Machine .
4. Give a name to the virtual machine. Select Type as “Linux” and version “Ubuntu (64 bit)” .
5. Select memory(RAM) as 1GB .
6. Press Next with default option on “Create a new virtual hard drive now” .
7. Select disk type as “VMDK” which is also compatible with Vmware Workstation.
8. Select “Dynamically allocated” option which will allocate the space as and when data is created.If you select the option “Fixed size”, it will allocate the total disk space at once. 
9. Allocate the disk space . I have selected 32GB to be the vm disk space. 
10. VM is created now, but it does not have any operating system installed inside it. We will now install ubuntu 12.04 inside this newly created virtual machine.

Step 2 : Installing ubuntu 12.04 inside Virtualbox virtual machine.

1. Download Ubuntu server 12.04 LTS iso from http://www.ubuntu.com/download/server .
2. Power on the vm and select “Choose a virtual CD/DVD disk file” from “Devices->CD/DVD Devices”.
3. Attach the download iso to the VM and reset the VM from “Machine->Reset”.
4. VM will boot with the attached ISO. Select “English” as language.
5. select “Install Ubuntu Server” .
6. Select installation language as “English” .
7. Select your country.
8. Configure your keyboard Layout. 
9. Enter the hostname. This will be name of the system.
10. Create user for accessing the machine as a non root user .   
11. Configure system clock.
12. Partition the disk. Use default options.
13. Configure the proxy if your network is behind a proxy server.
14. Select update policy.
15. Install “OpenSSH server” with the OS so that we can directly login via ssh. Press space to select the option.
16. Install GRUB boot loader. 
17. Ubuntu installation is now complete.
18. Ubuntu vm is ready to use now. Login with the user created above.

Note  :  If you cannot ping/ssh to the VM from the Host Machine then change the Network Settings to “Bridged Adapter”

• Power Off the Virtual Machine
  
• Right Click on the VM and select settings
• Select “Attached to:” to Bridged Adapter
• Power On the VM