cliff – Command Line Interface Formulation Framework – version 1.2.1

cliff is a framework for building command line programs. It uses
setuptools entry points to provide subcommands, output formatters, and
other extensions.

What’s New In This Release?

  • Fix problem with documentation packaging.
  • Fix problem with missing izip import in lister.py.

Documentation

Documentation for cliff is hosted on readthedocs.org

Installation

Use pip:

$ pip install cliff

See the installation guide for more details.

cliff – Command Line Interface Formulation Framework – version 1.2

cliff is a framework for building command line programs. It uses
setuptools entry points to provide subcommands, output formatters, and
other extensions.

What’s New In This Release?

  • Fix problem with interactive mode help command.
  • Disable logging by default but add a –log-file option to
    re-enable it at runtime.
  • Add support for python 2.6. (contributed by Mark McClain for
    OpenStack Quantum)

Documentation

Documentation for cliff is hosted on readthedocs.org

Installation

Use pip:

$ pip install cliff

See the installation guide for more details.

cliff – Command Line Interface Formulation Framework – version 1.1.2

cliff is a framework for building command line programs. It uses
setuptools entry points to provide subcommands, output formatters, and
other extensions.

What’s New In This Release?

  • This point release fixes a packaging problem introduced
    by removing several formatters in version 1.1.

Documentation

Documentation for cliff is hosted on readthedocs.org

Installation

Use pip:

$ pip install cliff

See the installation guide for more details.

cliff – Command Line Interface Formulation Framework – version 0.5

cliff is a framework for building command line programs. It uses
setuptools entry points to provide subcommands, output formatters, and
other extensions.

What’s New In This Release?

  • Asking for help about a command by prefix lists all matching
    commands.
  • Add formatters for HTML, JSON, and YAML.

Documentation

Documentation for cliff is hosted on readthedocs.org

Installation

Use pip:

$ pip install cliff

See the installation guide for more details.

Determining the Name of a Process from Python

Finding the name of the program from which a Python module is
running can be trickier than it would seem at first, and
investigating the reasons led to some interesting experiments.

A couple of weeks ago at the OpenStack Folsom Summit, Mark McClain
pointed out an interesting code snippet he had discovered in the Nova
sources
:

nova/utils.py: 339

script_dir = os.path.dirname(inspect.stack()[-1][1])

The code is part of the logic to find a configuration file that lives
in a directory relative to where the application startup script is
located. It looks at the call stack to find the main program, and
picks the filename out of the stack details.

The code seems to be taken from a response to a StackOverflow
question
, and when I saw it I thought it looked like a case of
someone going to more trouble than was needed to get the
information. Mark had a similar reaction, and asked if I knew of a
simpler way to determine the program name.

I thought it looked like a case of someone going to more trouble
than was needed…

Similar examples with inspect.stack() appear in four places in
the Nova source code (at last as-of today). All of them are either
building filenames relative to the location of the original “main”
program, or are returning the name of that program to be used to build
a path to another file (such as a log file or other program). Those
are all good reasons to be careful about the location and name of the
main program, but none explain why the obvious solution isn’t good
enough. I assumed that if the OpenStack developers were looking at
stack frames there must have been a reason. I decided to examine the
original code and spend a little time deciphering what it is doing,
and especially to see if there were cases where it did not work as
desired (so I could justify a patch).

The Stack

The call to inspect.stack() retrieves the Python interpreter
stack frames for the current thread. The return value is a list with
information about the calling function in position 0 and the “top”
of the stack at the end of the list. Each item in the list is a tuple
containing:

  • the stack frame data structure
  • the filename for the code being run in that frame
  • the line number within the file
  • the co_name member of the code object from the frame,
    giving the function or method name being executed
  • the source lines for that bit of code, when available
  • an index into the list of source lines showing the actual source
    line for the frame

show_stack.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import inspect


def show_stack():
    stack = inspect.stack()
    for s in stack:
        print 'filename:', s[1]
        print 'line    :', s[2]
        print 'co_name :', s[3]
        print


def inner():
    show_stack()


def outer():
    inner()


if __name__ == '__main__':
    outer()

The information is intended to be used for generating tracebacks or by
tools like pdb when debugging an application (although
pdb has its own implementation). To answer the question “Which
program am I running in?” the filename is most the interesting piece
of data.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
$ python show_stack.py
filename: show_stack.py
line    : 6
co_name : show_stack

filename: show_stack.py
line    : 15
co_name : inner

filename: show_stack.py
line    : 19
co_name : outer

filename: show_stack.py
line    : 23
co_name : <module>

One obvious issue with these results is that the filename in the stack
frame is relative to the start up directory of the application. It
could lead to an incorrect path if the process has changed its working
directory between startup and checking the stack. But there is
another mode where looking at the top of the stack produces completely
invalid results.

The simple one-liner is not always going to produce the right
results.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ python -m show_stack
filename: /Users/dhellmann/.../show_stack.py
line    : 6
co_name : show_stack

filename: /Users/dhellmann/.../show_stack.py
line    : 15
co_name : inner

filename: /Users/dhellmann/.../show_stack.py
line    : 19
co_name : outer

filename: /Users/dhellmann/.../show_stack.py
line    : 23
co_name : <module>

filename: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py
line    : 72
co_name : _run_code

filename: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py
line    : 162
co_name : _run_module_as_main

The -m option to the interpreter triggers the runpy module,
which takes the module name specified and executes it like a main
program. As the stack printout above illustrates, runpy is then at
the top of the stack, so the “main” part of our local module is
several levels down from the top. That means the simple one-liner is
not always going to produce the right results.

Why the Obvious Solution Fails

Now that I knew there were ways to get the wrong results by looking at
the stack, the next question was whether there was another way to find
the program name that was simpler, more efficient, and especially more
correct. The simplest solution is to look at the command line
arguments passed through sys.argv.

argv.py

1
2
3
import sys

print sys.argv[0]

Normally, the first element in sys.argv is the script that was run
as the main program. The value always points to the same file,
although the method of invoking it may cause the value to fluctuate
between a relative and full path.

1
2
3
4
5
6
7
8
$ python argv.py
argv.py

$ ./argv.py
./argv.py

$ python -m argv
/Users/dhellmann/.../argv.py

As this example demonstrates, when a script is run directly or passed
as an argument to the interpreter, sys.argv contains a relative
path
to the script file. Using -m we see the full path, so
looking at the command line arguments is more robust for that
case. However, we cannot depend on -m being used so we aren’t
guaranteed to get the extra details.

Using import

The next alternative I considered was probing the main program module
myself. Every module has a special property, __file__, which holds
the path to the file from which the module was loaded. To access the
main program module from within Python, you import a specially named
module __main__. To test this method, I created a main program
that loads another module:

import_main_app.py

1
2
3
import import_main_module

import_main_module.main()

And the second module imports __main__ and print the file it was
loaded from.

import_main.py

1
2
3
import __main__

print __main__.__file__

Looking at the __main__ module always pointed to the actual main
program module, but it did not always produce a full path. This makes
sense, because the filename for a module that goes into the stack
frame comes from the module itself.

1
2
3
4
5
$ python import_main.py
import_main.py

$ python -m import_main
/Users/dhellmann/.../import_main.py

Wandering Down the Garden Path

After I found such a simple way to reliably retrieve the program name,
I spent a while thinking about the motivation of the person who
decided that looking at stack frames was the best solution. I came up
with two hypotheses. First, it is entirely possible that they did not
know about importing __main__. It isn’t the sort of thing one
needs to do very often, and I don’t even remember where I learned
about doing it (or why, because I’m pretty sure I’ve never used the
feature in production code for any reason). That seems like the most
plausible reason, but the other idea I had was that for some reason it
was very important to have a relatively tamper-proof value –
something that could not be overwritten accidentally. This new idea
merited further investigation, so I worked back through the methods of
accessing the program name to determine which, if any, met the new
criteria.

I did not need to experiment with sys.argv to know it was
mutable. The arguments are saved in a normal list object, and
can be modified quite easily, as demonstrated here.

argv_modify.py

1
2
3
4
5
6
7
8
import sys

print 'Type  :', type(sys.argv)
print 'Before:', sys.argv

sys.argv[0] = 'wrong'

print 'After :', sys.argv

All normal list operations are supported, so replacing the program
name is a simple assignment statement. Because sys.argv is a list,
it is also susceptible to having values removed by pop(),
remove(), or a slice assignment gone awry.

$ python argv_modify.py
Type  : <type 'list'>
Before: ['argv_modify.py']
After : ['wrong']

The __file__ attribute of a module is a string, which is not
itself mutable, but the contents can be replaced by assigning a new
value to the attribute.

import_modify.py

1
2
3
4
5
6
7
import __main__

print 'Before:', __main__.__file__

__main__.__file__ = 'wrong'

print 'After :', __main__.__file__

This is less likely to happen by accident, so it seems somewhat
safer. Nonetheless, changing it is easy.

$ python import_modify.py
Before: import_modify.py
After : wrong

That leaves the stack frame.

Down the Rabbit Hole

As described above, the return value of inspect.stack() is a
list of tuples. The list is computed each time the function is called,
so it was unlikely that one part of a program would accidentally
modify it. The key word there is accidentally, but even a malicious
program would have to go to a bit of effort to return fake stack data.

stack_modify1.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import inspect


def faux_stack():
    full_stack = orig_stack()
    top = full_stack[-1]
    top = (top[0], 'wrong') + top[2:]
    full_stack[-1] = top
    return full_stack
orig_stack = inspect.stack
inspect.stack = faux_stack


stack_data = inspect.stack()
script_name = stack_data[-1][1]
print 'From stack:', script_name
print 'From frame:', stack_data[-1][0].f_code.co_filename

The filename actually appears in two places in the data returned by
inspect.stack(). The first location is in the tuple that is part
of the list returned as the stack itself. The second is in the code
object of the stack frame within that same tuple
(frame.f_code.co_filename).

$ python stack_modify1.py
From stack: wrong
From frame: stack_modify1.py

It turned out to be more challenging to change the code object.

Replacing the filename in the tuple was relatively easy, and would be
sufficient for code that trusted the stack contents returned by
inspect.stack(). It turned out to be more challenging to change
the code object. For C Python, the code class is implemented
in C as part of the set of objects used internally by the interpreter.

Objects/codeobject.c

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
static PyMemberDef code_memberlist[] = {
    {"co_argcount",     T_INT,          OFF(co_argcount),       READONLY},
    {"co_nlocals",      T_INT,          OFF(co_nlocals),        READONLY},
    {"co_stacksize",T_INT,              OFF(co_stacksize),      READONLY},
    {"co_flags",        T_INT,          OFF(co_flags),          READONLY},
    {"co_code",         T_OBJECT,       OFF(co_code),           READONLY},
    {"co_consts",       T_OBJECT,       OFF(co_consts),         READONLY},
    {"co_names",        T_OBJECT,       OFF(co_names),          READONLY},
    {"co_varnames",     T_OBJECT,       OFF(co_varnames),       READONLY},
    {"co_freevars",     T_OBJECT,       OFF(co_freevars),       READONLY},
    {"co_cellvars",     T_OBJECT,       OFF(co_cellvars),       READONLY},
    {"co_filename",     T_OBJECT,       OFF(co_filename),       READONLY},
    {"co_name",         T_OBJECT,       OFF(co_name),           READONLY},
    {"co_firstlineno", T_INT,           OFF(co_firstlineno),    READONLY},
    {"co_lnotab",       T_OBJECT,       OFF(co_lnotab),         READONLY},
    {NULL}      /* Sentinel */
};

The data members of a code object are all defined as READONLY,
which means you cannot modify them from within Python code directly.

code_modify_fail.py

1
2
3
4
5
6
7
import inspect

stack_data = inspect.stack()
frame = stack_data[0][0]
code = frame.f_code

code.co_filename = 'wrong'

Attempting to change a read-only property causes a TypeError.

1
2
3
4
5
$ python code_modify_fail.py
Traceback (most recent call last):
  File "code_modify_fail.py", line 8, in <module>
    code.co_filename = 'wrong'
TypeError: readonly attribute

Instead of changing the code object itself, I would have to replace it
with another object. The reference to the code object is accessed
through the frame object, so in order to insert my code object into
the stack frame I would need to modify the frame. Frame objects are
also immutable, however, so that meant creating a fake frame to
replace the original value. Unfortunately, it is not possible to
instantiate code or frame objects from within
Python, so I ended up having to create classes to mimic the originals.

stack_modify2.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
import collections
import inspect

# Define a namedtuple with the attributes of a stack frame
frame_attrs = ['f_back',
               'f_code',
               'f_builtins',
               'f_globals',
               'f_lasti',
               'f_lineno',
               ]
frame = collections.namedtuple('frame', ' '.join(frame_attrs))

# Define a namedtuple with the attributes of a code object
code_attrs = ['co_argcount',
              'co_nlocals',
              'co_stacksize',
              'co_flags',
              'co_code',
              'co_consts',
              'co_names',
              'co_varnames',
              'co_freevars',
              'co_cellvars',
              'co_filename',
              'co_name',
              'co_firstlineno',
              'co_lnotab',
              ]
code = collections.namedtuple('code', ' '.join(code_attrs))


def _make_fake_frame(original, filename):
    """Return a new fake frame object with the wrong filename."""
    new_c_attrs = dict((a, getattr(original.f_code, a))
                          for a in code_attrs)
    new_c_attrs['co_filename'] = filename
    new_c = code(**new_c_attrs)

    new_f_attrs = dict((a, getattr(original, a))
                           for a in frame_attrs)
    new_f_attrs['f_code'] = new_c
    new_f = frame(**new_f_attrs)
    return new_f


def faux_stack():
    full_stack = orig_stack()
    top = full_stack[-1]

    new_frame = _make_fake_frame(top[0], 'wrong')

    # Replace the top of the stack
    top = (new_frame, 'wrong') + top[2:]
    full_stack[-1] = top

    return full_stack
orig_stack = inspect.stack
inspect.stack = faux_stack


def show_app_name():
    stack_data = inspect.stack()
    script_name = stack_data[-1][1]
    print 'From stack:', script_name
    print 'From frame:', stack_data[-1][0].f_code.co_filename


if __name__ == '__main__':
    show_app_name()

I stole the idea of using namedtuple as a convenient way to
have a class with named attributes but no real methods from
inspect, which uses it to define a Traceback class.

$ python stack_modify2.py
From stack: wrong
From frame: wrong

Replacing the frame and code objects worked well for accessing the
“code” object directly, but failed when I tried to use
inspect.getframeinfo() because there is an explicit type check
with a TypeError near the beginning of getframeinfo()
(see line 16 below).

http://hg.python.org/cpython/file/35ef949e85d7/Lib/inspect.py#l987

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
def getframeinfo(frame, context=1):
    """Get information about a frame or traceback object.

    A tuple of five things is returned: the filename, the line number of
    the current line, the function name, a list of lines of context from
    the source code, and the index of the current line within that list.
    The optional second argument specifies the number of lines of context
    to return, which are centered around the current line."""
    if istraceback(frame):
        lineno = frame.tb_lineno
        frame = frame.tb_frame
    else:
        lineno = frame.f_lineno
    if not isframe(frame):
        raise TypeError('{!r} is not a frame or traceback object'.format(frame))

    filename = getsourcefile(frame) or getfile(frame)
    if context > 0:
        start = lineno - 1 - context//2
        try:
            lines, lnum = findsource(frame)
        except IOError:
            lines = index = None
        else:
            start = max(start, 1)
            start = max(0, min(start, len(lines) - context))
            lines = lines[start:start+context]
            index = lineno - 1 - start
    else:
        lines = index = None

    return Traceback(filename, lineno, frame.f_code.co_name, lines, index)

The solution was to replace getframeinfo() with a version that
skips the check. Unfortunately, getframeinfo() uses
getfile(), which performs a similar check, so that function
needed to be replaced, too.

stack_modify3.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
import inspect

from stack_modify2 import show_app_name


def getframeinfo(frame, context=1):
    """Get information about a frame or traceback object.

    A tuple of five things is returned: the filename, the line number of
    the current line, the function name, a list of lines of context from
    the source code, and the index of the current line within that list.
    The optional second argument specifies the number of lines of context
    to return, which are centered around the current line."""
    if inspect.istraceback(frame):
        lineno = frame.tb_lineno
        frame = frame.tb_frame
    else:
        lineno = frame.f_lineno
    # if not isframe(frame):
    #     raise TypeError('{!r} is not a frame or traceback object'.format(frame))

    filename = inspect.getsourcefile(frame) or inspect.getfile(frame)
    if context > 0:
        start = lineno - 1 - context//2
        try:
            lines, lnum = inspect.findsource(frame)
        except IOError:
            lines = index = None
        else:
            start = max(start, 1)
            start = max(0, min(start, len(lines) - context))
            lines = lines[start:start+context]
            index = lineno - 1 - start
    else:
        lines = index = None

    return inspect.Traceback(filename, lineno, frame.f_code.co_name, lines, index)
inspect.getframeinfo = getframeinfo


def getfile(object):
    """Work out which source or compiled file an object was defined in."""
    if hasattr(object, 'f_code'):
        return object.f_code.co_filename
    return orig_getfile(object)
orig_getfile = inspect.getfile
inspect.getfile = getfile


if __name__ == '__main__':
    show_app_name()
    s = inspect.stack()
    print inspect.getframeinfo(s[-1][0])

Now the caller can use inspect.getframeinfo() (really my
replacement function) and see the modified filename in the return
value.

1
2
3
4
5
$ python stack_modify3.py
From stack: wrong
From frame: wrong
Traceback(filename='wrong', lineno=51, function='<module>',
code_context=None, index=None)

After reviewing inspect.py one more time to see if I needed to
replace any other functions, I realized that a better solution was
possible. The implementation of inspect.stack() is very small,
since it calls inspect.getouterframes() to actually build the
list of frames. The seed frame passed to getouterframes() comes
from sys._getframe().

http://hg.python.org/cpython/file/35ef949e85d7/Lib/inspect.py#l1052

def stack(context=1):
    """Return a list of records for the stack above the caller's frame."""
    return getouterframes(sys._getframe(1), context)

The rest of the stack is derived from the first frame returned by
_getframe() using the f_back attribute to link from one
frame to the next.

http://hg.python.org/cpython/file/35ef949e85d7/Lib/inspect.py#l1025

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
def getouterframes(frame, context=1):
    """Get a list of records for a frame and all higher (calling) frames.

    Each record contains a frame object, filename, line number, function
    name, a list of lines of context, and index within the context."""
    framelist = []
    while frame:
        framelist.append((frame,) + getframeinfo(frame, context))
        frame = frame.f_back
    return framelist

If I modified getouterframes() instead of inspect.stack(),
then I could ensure that my fake frame information was inserted at the
beginning of the stack, and all of the rest of the inspect
functions would honor it.

stack_modify4.py

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
import collections
import inspect

# Define a namedtuple with the attributes of a stack frame
frame_attrs = ['f_back',
               'f_code',
               'f_builtins',
               'f_globals',
               'f_lasti',
               'f_lineno',
               ]
frame = collections.namedtuple('frame', ' '.join(frame_attrs))

# Define a namedtuple with the attributes of a code object
code_attrs = ['co_argcount',
              'co_nlocals',
              'co_stacksize',
              'co_flags',
              'co_code',
              'co_consts',
              'co_names',
              'co_varnames',
              'co_freevars',
              'co_cellvars',
              'co_filename',
              'co_name',
              'co_firstlineno',
              'co_lnotab',
              ]
code = collections.namedtuple('code', ' '.join(code_attrs))


def _make_fake_frame(original, filename):
    """Return a new fake frame object with the wrong filename."""
    new_c_attrs = dict((a, getattr(original.f_code, a))
                          for a in code_attrs)
    new_c_attrs['co_filename'] = filename
    new_c = code(**new_c_attrs)

    new_f_attrs = dict((a, getattr(original, a))
                           for a in frame_attrs)
    new_f_attrs['f_code'] = new_c
    new_f = frame(**new_f_attrs)
    return new_f


def getouterframes(frame, context=1):
    """Get a list of records for a frame and all higher (calling) frames.

    Each record contains a frame object, filename, line number, function
    name, a list of lines of context, and index within the context."""
    framelist = []
    while frame:
        if not frame.f_back:
            # Replace the top of the stack with a fake entry
            frame = _make_fake_frame(frame, 'wrong')
        framelist.append((frame,) + getframeinfo(frame, context))
        frame = frame.f_back
    return framelist
inspect.getouterframes = getouterframes


def getframeinfo(frame, context=1):
    """Get information about a frame or traceback object.

    A tuple of five things is returned: the filename, the line number of
    the current line, the function name, a list of lines of context from
    the source code, and the index of the current line within that list.
    The optional second argument specifies the number of lines of context
    to return, which are centered around the current line."""
    if inspect.istraceback(frame):
        lineno = frame.tb_lineno
        frame = frame.tb_frame
    else:
        lineno = frame.f_lineno
    # if not isframe(frame):
    #     raise TypeError('{!r} is not a frame or traceback object'.format(frame))

    filename = inspect.getsourcefile(frame) or inspect.getfile(frame)
    if context > 0:
        start = lineno - 1 - context // 2
        try:
            lines, lnum = inspect.findsource(frame)
        except IOError:
            lines = index = None
        else:
            start = max(start, 1)
            start = max(0, min(start, len(lines) - context))
            lines = lines[start:start + context]
            index = lineno - 1 - start
    else:
        lines = index = None

    return inspect.Traceback(filename, lineno, frame.f_code.co_name, lines, index)
inspect.getframeinfo = getframeinfo


def getfile(object):
    """Work out which source or compiled file an object was defined in."""
    if isinstance(object, frame):
        return object.f_code.co_filename
    return orig_getfile(object)
orig_getfile = inspect.getfile
inspect.getfile = getfile


def show_app_name():
    stack_data = inspect.stack()
    #print [(s[1], s[0].__class__, s[0].f_code.co_name) for s in stack_data]
    print 'From stack        :', stack_data[-1][1]
    print 'From code in frame:', stack_data[-1][0].f_code.co_filename
    print 'From frame info   :', inspect.getframeinfo(stack_data[-1][0]).filename


if __name__ == '__main__':
    show_app_name()

The customized versions of getframeinfo() and getfile()
are still required to avoid exceptions caused by the type checking.

$ python stack_modify4.py
From stack        : wrong
From code in frame: wrong
From frame info   : wrong

Enough of That

At this point I have proven to myself that while it is unlikely that
anyone would bother to do it in a real program (and they would
certainly not do it by accident) it is possible to intercept the
introspection calls and insert bogus information to mislead a program
trying to discover information about itself. This implementation does
not work to subvert pdb, because it does not use
inspect. Probably because it predates inspect,
pdb has its own implementation of a stack building function,
which could be replaced using the same technique as what was done
above.

This investigation led me to several conclusions. First, I still don’t
know why the original code is looking at the stack to discover the
program name. I should ask on the OpenStack mailing list, but in the
mean time I had fun experimenting while researching the question.
Second, given that looking at __main__.__file__ produces a value
at least as correct as looking at the stack in all cases, and more
correct when a program is launched using the -m flag, it seems
like the solution with best combination of reliability and
simplicity. A patch may be in order. And finally, monkey-patching can
drive you to excesses, madness, or both.

Updates

8 May – Updated styles around embedded source files and added a label
for each.

Notes from OpenStack Folsom Design Summit Spring 2012

These are my notes from the Design Summit and Conference kicking off
the Folsom release of OpenStack, held April 16-20, 2012 in San
Francisco, CA. Much more happened than I could hope to capture in a
single post, but I can publish most of my notes from the design
sessions.

tl;dr

This was a good week for the DreamHost development team. We began
establishing ourselves as contributors in the OpenStack community, met
a lot of people we will need to be working with over the next few
months, and learned a lot and OpenStack in general. We also identified
several projects that need our help, and found some tools we should be
able to use for our implementation.

Goals for the Week

I had several goals for this trip.

  1. Because this was my first summit, the most important thing for me
    to do was introduce myself to, and become acquainted with, members
    of the community. This will be ongoing work, but I made a good
    start.
  2. DreamHost is making it a priority to contribute to general
    technical discussions, not just the aspects of OpenStack that will
    affect us directly (they all do, in the long run). I participated
    in several of the summit sessions, sharing my own thoughts,
    expressing requirements for DreamHost, and repeating information
    discussed in other sessions. (see notes below)
  3. I also wanted to identify projects to which I or DreamHost can and
    should contribute early.

Session Notes

Dough: Billing Service

Zhongyue Luo of Sina Corporation presented “Dough,” the billing
service they have created. There was a lot of discussion, some of it
quite critical. The main issue raised was that Dough requires changing
Horizon to add explicit hooks to call into the new service, which is
brittle in the face of failed operations (such as when an instance
creation is requested but cannot be completed). It also combined
metering and billing too tightly and did not actually track compute or
network resource usage.

DreamHost needs a metering service, and may want some features of an
existing billing service to coalesce metered data before feeding it
into our charge-back system.

See also

Keystone Secret Storage

Justin Santa Barbara discussed a system for using chained secrets to
allow encryption of data within Swift, without requiring re-encrypting
that data when the user changes their password (the first step of the
chain). The idea is to create a second secret value associated with
the user and encrypt it using the password, then store it in Keystone
(or some system to which Keystone delegates). The chain of trust would
be the user password, a generated user key, a generated tenant key,
and a container key. The weak link is still the password, but using a
chain allows some of the values to be changed without re-encrypting
everything in the database and offers some compartmentalization for
protection in the case of a breach.

OpenStack Common

Mark McLoughlin led this session reviewing the progress on the common
library project to unify the implementations of features within
OpenStack where it makes sense to share code.

The plan for now is to provide a two-stage system for common
code. Incubated code would live in a special part of the common
repository and would be copied into projects that use it. When the API
for an incubated module stabilizes, the module can be moved into the
common library as an official module and it can be imported from that
namespace instead of the project namespace. That deprecation will
happen over the course of several releases (deprecate in N, keep
support in N+1, remove in N+2) to give downstream users of the project
code trees time to adjust. This process will begin with the config
module during the Folsom release cycle.

Existing APIs to be considered for common include:

  • config (cfg, pastedeploy, logging)
  • setup
  • importutils
  • authutils
  • timeutils
  • local – greenthread local storage
  • context
  • exception
  • wsgi, extensions (Nova)

The two existing developers on the project asked for help reviewing
patches and contributing. Since code cleanup is a priority for
DreamHost, I have added this to my todo list.

See also

Standardizing Database Management

This session as a grab-bag of topics related to database management
issues.

Database management works slightly differently between the
projects. Specifically, handling dropped connections for MySQL needs
work. Someone suggested contributing code back to sqlalchemy to help
with that.

Database migrations with sqlalchemy-migrate can be painful. Jonathan
LaCour suggested using Alembic instead.

Someone else proposed a rule that database migrations and code changes
should be submitted in separate changesets, sequenced so that nothing
breaks. There was some discussion, but I don’t think we agreed to do
this.

We did agree to disable automatic database creation and migration for
all projects.

See also

Stable Branch Management

Mark McLoughlin led this session on stable branches and support for
historical versions. There was quite a bit of discussion, and the
outcome looks good for users such as DreamHost.

First, representatives for several distros were in the room and they
all agreed that the original proposal of dropping support for a
release after 1 year was not going to work. They committed resources
to the stable branch management team to improve that situation, and
the core developers agreed to allow support for a given release to be
advertised so long as there are actual active developers maintaining
it.

The maintenance model will have two states: The project will receive
“active” maintenance by the stable team during development and the
next release cycle. It will then receive “passive” maintenance (only
security and high-impact fixes) as long as there are developers signed
up to provide it. The maintenance status of any given release will be
published (probably in the wiki) so deployers can make informed
decisions.

In all cases, patches should be applied to trunk before being
back-ported, and no back-port change will be approved without a
reference to the trunk changeset.

Mark also discussed a tool called “palaver” for reviewing git
changesets in master and tagging them for cherry-picking to be merged
into stable branches. There was some discussion of integrating the
features with gerrit to make triaging easier.

See also

Unified Command Line Interface

Dean Troyer of RCB has started working on a design for a unified
command line interface to replace all of the existing clients. The
project goal is to provide a consistent user interface with naming
conventions, argument names, etc. For Folsom we will develop a
prototype and work out the arguments supported by each command. Dean
has done a fair bit of analysis of the existing commands to map them
to a new structure he is proposing, and he and I are both creating
prototypes on which to base the design discussions.

We plan to start a new project to hold the base classes and core
components of the CLI, then provide a plugin system to allow other
projects to define their own sub-commands. The commands available on a
given system will depend on the packages installed.

There was also some discussion of sharing more code between the Python
client libraries for the various projects, which would make
implementing the new CLI easier. Those changes do not block work on
the CLI, so the projects can continue in parallel.

Dean and I had several conversations after the session about how to
implement the plugin loading. We both reviewed cement2, a library
identified by some of the Quantum developers, and found it unsuitable
(I categorized it as the unholy love child of a Zope developer and a
Java developer, but that may be a bit harsh). Dean posted his
prototype, and I have started a new framework called cliff that I
am proposing.

See also

Splitting up Nova

Thierry Carrez led a session on looking for other parts of Nova that
can be split out into their own projects to make maintenance
easier. Suggestions included volumes (already approved), hypervisor
drivers, utils (into openstack-common), the job/resource scheduler,
RPC and notifiers (also into openstack-common). Blocking issues
identified included database migrations and the lax API with
virtualization drivers.

The objective during the Folsom release cycle is to create a plugin
system to allow pieces of code coming from these other projects to be
brought into Nova at runtime when needed.

Important aspects of the plugin system are that plugin code needs to
be able to execute synchronously or asynchronously, plugins need the
option of two way communication (i.e., having Nova ask the scheduler a
question and wait for the answer), and error handling needs to support
aborting operations.

Andrew Bogott and I had a lengthy discussion about the plugin API. Our
approaches are a little different (Andrew has proposed a monolithic
plugin API which will have entry points for specific purposes and I
have been arguing in favor of more targeted plugin APIs where the code
that uses the plugin is responsible for loading it). We have some work
to do before we agree, but we have agreed that we need to agree and
will eventually, so the current disagreement is at least agreeable.

Eric Windisch suggested that all communication with the plugins should
be via RPC. Neither Andrew nor I thought that should be a hard
requirement.

See also

Making Nova Tenant Data “Pluggable”

Phil Day from HP led this session on changing Keystone to provide a
way to store generic user data such as SSH keys, quotas, and security
group definitions. The initial proposal discussed those items
explicitly, but I suggested making a more generic key/value store API
and that was received well by several participants.

It was unclear at the end of the session how Phil would be proceeding,
so I will need to keep an eye on the mailing list for discussions
about this.

See also

DevStackPy

Joshua Harlow of Yahoo! presented the work we have done on the
replacement for devstack. Although there was some interest, there were
some requirements met by devstack that are not met by devstackpy. The
primary objection seems to be the fact that the documentation team
does actually read and pick apart the shell script in order to create
the installation guides, and that would be more difficult with the
Python-based implementation. This was a little disappointing, since
DreamHost had contributed persona support and some significant
rearchitecting to DevStackPy to make it easier to add new platform
support.

See also

Nova Orchestration

This session discussed “orchestration” in somewhat generic
terms. There is a project with a code-name, but that wasn’t the
subject.

There was a good bit of back-and-forth about whether to use a state
machine or workflow model for orchestration. The general consensus was
that a state machine combined with idempotent task implementations
would allow for more reliable retry and cleanup, so it was preferred.

Transactional consistency across services is not a goal because it
leads to brittle solutions.

The discussions were focused around the sub-tasks for things like
creating instances in Nova, but implementation will need to extend to
Glance and Quantum in the future, since they also have complicated
multi-step operations. Managing parallel jobs such as creating
multiple instances was tabled for future work.

The first step for now is to add a way to store the “expected” state
of the system so the current state can be compared against it.

See also

Common Image Metadata

Glance provides a way to store custom properties with images. Justin
Santa Barbara would like to develop a standard naming scheme for these
so we can add useful metadata in a common way without requiring code
changes to add those fields to all images. The specific fields Justin
wants to add are mostly for identifying images programmatically, so
they include things like the image distro, vendor, and version. He
also has a date field for recording how up to date the image is in
case it is not a base image.

There was a bit of bike-shedding on syntax and naming, but I think in
the end we reached consensus on using a prefix of “org.openstack__” for common properties. Other providers who
define properties would use their own prefix. Property names should
have version numbers (so, “prefix__1”) in case the semantics
of the field change.

Distributions will be identified by a registered name such as
“com.ubuntu,” optionally followed by a qualifier or distribution
name. For example, “com.ubuntu.release” and “com.ubuntu.daily” might
both have the same version information but because the daily builds
are not tested as thoroughly Canonical wants to make sure they are
clearly identified.

I proposed using the standard in **PEP 396** for version string
format instead of inventing a new convention.

I also proposed investigating other standards such as DMTF and OVF for
properties we might be leaving out. No one had looked at those, yet.

See also

Efficient Metering

Nick Barcet proposed some requirements for a metering system. He
identified three parts to a full billing system: metering, analysis,
and billing. This session was limited to the metering part, since it
needs to be built first.

The proposal he has put forward includes agents collecting data for
“counters” on a regular basis (hourly), and the data being aggregated
at larger time periods until it is eventually discarded.

Someone brought up monitoring, which is related but would need more
timely and fine-grain data.

James Peinck from Yahoo! has started implementing data collection
agents which are not yet open but will be. He is interested in
adapting it to work with the standard.

See also

Nova Ops Pain Points

Michael Still of Canonical led this session during which many members
of the DevOps (or regular Ops) community described various issues they
have encountered running OpenStack. The big issues are around upgrades
and debugging, and there were some good suggestions for short and long
term solutions. There was a definite consensus that operators should
not need to access the database to debug or clean up an operation, so
we should keep that in mind as we design the metering solution and
other new features.

See also

Federated Zones

Christopher MacGown of Piston Cloud led this session talking about how
they handle the lack of zone awareness in clients (Keystone can return
a list of endpoints but all of the current clients only take the first
one). They use a load balancer so they only need one “end point” and
the work can be split out by the LB. Ken Pepple of Internap said they
are doing something very similar in their implementation.

The proposal was to add zone awareness to the client so it can make
decisions about where to run tasks based on geography, permission,
capability, etc. (depending on how zones are defined in a given
implementation). This would work by having the end-user specify an
explicit reference to some object such as a Swift container. The
client would determine the zone where the object lives, and use that
zone for the current task.

There was some discussion of having the zone awareness be more “hint
based” and allow the system to make alternative choices, but that sort
of guessing is really hard to get right so we decided that if the
operation cannot be performed with that zone for some reason, the job
should fail.

In order to achieve this, the Keystone API needs to be changed to
include the zone information associated with each endpoint that it
returns in the service catalog.

See also

Making Configuration Easier

Dan of Puppet Labs talked about some of the issues he has had modeling
the OpenStack configuration features to implement deployment
orchestration in Puppet. Two key issues are the paste.ini file in
Keystone (because it combines code and local configuration) and the
monolithic Nova configuration file.

I pointed out that in the Python 3 session Paste was brought up as a
blocker for moving to Python 3 support, so the configuration question
was another reason to consider dropping it.

Someone else mentioned that users/installers do actually need to
change the pipeline in the paste.ini, so we can’t just put that “in
code” somewhere.

Quantum CLI Rewrite

Jason proposed using cement2 to rebuild the Quantum command line
interface. Some requirements for the new tool are that it should be
able to figure out which extensions are installed on a given server so
it can pre-validate requests (or possibly turn off features in the
client). The output also needs to be more parsable so the tools can be
used in scripts.

This work will probably wait for the unified CLI project to establish
the proper framework before continuing.

See also

DevOps “Love-In”

Duncan McGreggor of DreamHost led this session trying to get the
DevOps community to combine efforts to raise awareness of their
issues. The general theme that I took away from the session was that
the operators need to be the squeaky wheel and commit resources to
opening and tracking bugs. They can also help by triaging and
commenting on existing bugs, discussing issues on the main mailing
list, and providing environments where devs can recreate problems that
only happen “at scale.” Duncan is going to lead an effort to summarize
ops issues for the weekly project status meetings on IRC.

Preparing My First Patch for OpenStack

I joined DreamHost about four weeks ago on Feb 6. and am on the
team building a cloud hosting service based on the open source project
OpenStack. I spent the first couple of weeks at the new job doing
the usual sorts of new-hire activities: reading a lot of documentation
and learning my way around the development environment and tool
chain. I’ve done a little bit of work on some internal code already,
and I recently had a good opportunity to start working on OpenStack
itself.

The OpenStack project team was running a coordinated “Global
Hack-In
” on March 1st to work on bugs for the upcoming Essex
release. Several members of the OpenStack Atlanta meetup, sponsored
by DreamHost, met to participate in the hack-in as a group. Since we
don’t have official offices in Atlanta, yet, we gathered over at
Duncan McGreggor’s house
. Brent Scotten has already posted a quick
recap of the day on the OpenStack blog, but I wanted to go into a
little more detail about my experiences preparing my first patch for
the project because for me this was a new code base on a new project
using new version control and code review tools.

Becoming an OpenStack Contributor

The process for signing up to be a contributor to OpenStack isn’t
terribly complicated, but there are a lot of steps and the
instructions are spread out over a couple of different documents. I
took care of subscribing to the required mailing lists and signing the
contributor agreement a couple of weeks ago, but I am including those
steps here along with the technical stuff I did yesterday, to provide
an end-to-end view of the process.

Community

The first set of instructions, in the wiki under HowToContribute,
explains how to join the OpenStack community. I registered an account
on Launchpad, the code hosting service managed by Canonical and
used by Ubuntu and other open source projects. Then I joined all of
the OpenStack projects (Nova, Swift, Glance, and Keystone). Joining
the individual teams under the OpenStack umbrella gave me access to
the project mailing lists, so I can eventually participate in
conversations about designs and implementation details.

Contributor Agreement

Please refer to this wiki page for the current process for signing
the CLA.

Development Environment

I am familiar with source control and virtualization, but not the
specific flavors used for this project. My next step was to set up a
development environment where I could work on the Nova project
documentation, which meant setting up an virtual machine where I could
build the project.

Virtualbox

My desktop system is running Mac OS X 10.7.3 (Lion), and OpenStack is
intended to run on Linux (primarily Ubuntu). To set up an Ubuntu
system for the build, I used virtualbox and vagrant. Duncan and
Mark had already found a Vagrantfile that would set up an Oneiric
image and then use DevStack to deploy all of the parts of OpenStack
into the new VM using these chef cookbooks.

I saved the Vagrantfile to an empty directory and started a VM using
vagrant up. After a short wait while devstack downloaded and
installed all of the dependencies, the VM was ready and running a copy
of OpenStack based on the git repositories created by the devstack
script. The local repositories are created in /opt/stack within
the VM, but I wanted to work in a copy that was mounted via NFS from
the host OS so I could edit under OS X and build in the VM.

I was going to work on Nova, so I cloned its git repository from
github and modified the Vagrantfile to have more RAM and so it
would mount the directory where my source code was checked out using
NFS to /home/vagrant/Devel.

# -*- mode: ruby -*-

Vagrant::Config.run do |config|
  sshdir = "#{ENV['HOME']}/.ssh/"
  cachedir = (ENV['CACHEDIR'] or "/Users/dhellmann/Documents/Devel/openstack-vm/cache")
  checkout = (ENV['COOKBOOKS'] or "/Users/dhellmann/Documents/Devel/openstack-vm/openstack-cookbooks")

  # Make /vagrant an NFS mount instead of using the default setting
  #config.vm.share_folder("v-root", "/vagrant", ".", :nfs => true)

  ip_prefix = (ENV['IP_PREFIX'] or "10.0.5.")
  mac_prefix = (ENV['MAC_PREFIX'] or "080027027")
  suffix = "100"
  ip = "#{ip_prefix}#{suffix}"
  config.vm.box = "oneiric"
  config.vm.box_url = "http://images.ansolabs.com/vagrant/oneiric64.box"
  config.vm.customize ['modifyvm', :id, '--memory', '2048']
  config.vm.network :hostonly, ip, :mac => "#{mac_prefix}#{suffix}"

  config.vm.share_folder("v-cache", "/home/vagrant/cache", cachedir, :nfs => true)
  config.vm.share_folder("v-ssh", "/home/vagrant/.host-ssh", sshdir)
  config.vm.share_folder("v-dev", "/home/vagrant/Devel", "/Users/dhellmann/Documents/Devel/nova-docs",
                         :nfs => true)

  config.vm.forward_port 80, 8080
  config.vm.provision :chef_solo do |chef|
    chef.cookbooks_path = "#{checkout}/cookbooks"
    chef.roles_path = "#{checkout}/roles"
    chef.log_level = :debug
    chef.run_list = [
      "recipe[anso::cache]",
      "recipe[nova::hostname]",
      "recipe[nova::source]",
      #"recipe[anso::settings]", # vim / screen / git settings for testing
    ]
    chef.json.merge!({
      :nova => {
        :source => {
          :mysql_password => "secrete",
          :rabbit_password => "secrete",
          :admin_password => "secrete",
          :service_token => "secrete",
          :flat_interface => "eth1",
          :public_interface => "eth1",
          :floating_range => (ENV['FLOATING'] or "#{ip_prefix}128/28"),
          :host_ip => ip,
        }
      },
    })
  end
end

After restarting the VM (vagrant halt && vagrant up) I was able to
login and switch Nova to use the git repository I had just checked
out:

$ vagrant ssh
$ cd Devel/nova
$ sudo python setup.py develop

I encountered some I/O issues using vagrant ssh to connect to the
VM, so in the end I switched to using ssh directly via the port that
was forwarded using the instructions in the Vagrantfile above and the
key vagrant used when setting up the VM.

$ ssh -i ~/.vagrant.d/insecure_private_key -p 2222 vagrant@127.0.0.1

Building the Documentation

The documentation (and test) build for Nova use a local virtualenv
created within the source tree. The instructions for creating the
virtualenv
are part of the Nova Developer Guide. We all had
issues with timeouts and installation failures of one sort or another
setting up the virtualenv within a virtualbox VM, and I eventually
resorted to installing the dependencies into the virtualenv by hand
using:

$ cd Devel/nova
$ source .venv/bin/activate
(.venv)$ pip install -r tools/pip-requires
(.venv)$ pip install -r tools/test-requires

Sphinx and the other related tools needed to build the documentation
are installed into the virtualenv so that, with the environment
active, it is easy to build the HTML documentation using make:

$ cd doc
$ make

Choosing Something to Fix

It takes Sphinx a while to process all of the documentation, and it
produced a lot of warnings and errors along the way. Duncan and Mark
worked on some bugs in the OpenStack code, but I decided that since I
was new to the tools as well as the code-base I would focus on
shepherding a small documentation change all the way through the
process this time around, to make sure I had everything set up
correctly and that I understood the process. Fixing some of the errors
in the reStructuredText formatting within the documentation met my
criteria nicely.

Tracking Changes

OpenStack uses Launchpad for bug reports and task management.
Launchpad’s operating model uses “blueprints” to group related related
bugs and feature tickets. After studying the errors produced by
Sphinx, I decided that there were basically three causes:

  1. Problems in the automatically generated markup used to produce the

    API guide.

  2. Problems in the manually written markup of the manual.

  3. Problems in the manually written reStructuredText within the Nova

    source code, used when the API guide is generated.

I created one blueprint called sphinx-doc-cleanup to track all of
the changes, and then opened three separate bug tickets to address the
different types of problems. After opening the bugs, I went back to
the blueprint page to associate the new tickets with the blueprint.

I decided to tackle the changes in the order they are listed above,
because the fix for the first one was just a few lines and it would
eliminate several error messages.

Crafting the Fix

When Sphinx runs to build the Nova documentation, the first thing it
does is generate skeleton files for some of the API documentation
using a script called nova/doc/generate_autodoc_index.sh. For each
module that needs to be documented and for which no documentation
exists, a reStructuredText file is emitted containing a header such
as:

The :mod:`nova.wsgi` Module
===========================
.. automodule:: nova.wsgi
   :members:
   :undoc-members:
   :show-inheritance:

The result is that even if there is no special local documentation for
the module, the doc-strings from the code will be processed and
included in the final output.

The bug I found was with the way generate_autodoc_index.sh
calculated the length of the underline to create the reStructuredText
heading. The original version of the script used a single long line of
=, but some of the module names were longer than the line. Rather
than just extend it, I changed the script to calculate the proper
length.

Because this was my first patch, I also added myself to the
Authors file at the top of the Nova source tree so I would be
listed among the project contributors. I committed the change to my
local repository, including the blueprint and bug id in the git commit
message. At that point I was ready to submit the patch for review.

Review Tools and Process

OpenStack uses github to host its code, but the project does not
accept
pull requests in the usual way. Instead, it uses a copy of
Gerrit tied to Launchpad and Jenkins so that changes can be reviewed
by other developers and then run through the test suite before being
merged into the main repository automatically.

The instructions for setting up git to be able to submit change
requests to Gerrit are in the GerritWorkflow page of the OpenStack
wiki. After installing the git-review plug-in described there, I
ran into some trouble authenticating with Gerrit. I finally figured
out that I needed to upload the public half of my my ssh key to the
Gerrit server. I don’t know if it was missing because I did not have
it installed on Launchpad when I authenticated the first time, or if I
would have had to take the extra step anyway. Installing the key
allowed me to complete the setup, though, and then submit my change
for review
with git review.

While I was looking for instructions about how to make sure someone
saw that the change was ready for review, Vish Ishaya reviewed the
change (email notification of reviews are sent to a review mailing
list populated, I assume, from people who have signed up for the
OpenStack Gerrit instance). I ended up needing to modify the git
metadata for my change to use my DreamHost email address instead of my
personal account, but after a bit of back-and-forth on IRC to ask for
another review, Vish approved the change. A little while later the
Jenkins build completed and the change was merged into the main
repository. My first commit was merged!

Final Thoughts

So far I have found the OpenStack community helpful and welcoming. I
especially want to thank Anne Gentle, who assisted me with finding
some of the development setup documentation I needed. I should also
thank Vish Ishaya and Brian Waldon for jumping right on the code
review as soon as I submitted the patch, which made it possible to
achieve my personal goal for the day. As I mentioned above, the
OpenStack developers are using several tools I hadn’t used extensively
before. The tools integrate well, especially those like gerrit that
can use Launchpad for authentication. I am confident that I have
everything I need set up now, which means I can start making more
significant contributions.