Evaluating Tools for Developing with SOAP in Python
In order to better meet the needs of partners, Racemi needed to build a private web service to facilitate tighter integration between our applications and theirs. After researching the state of SOAP development in Python, we were able to find a set of tools that met our needs quite well. In this article, we will describe the criteria we used to evaluate the available tools and the process we followed to decide which library was right for us.
Racemi’s product, DynaCenter, is a server provisioning and data center management software suite focusing on large private installations where automation is key for our end-users. Because we are a small company, our business model is organized around partnering with larger companies in the same industry and acting as an OEM. Those partners typically provide their own user interface, and drive DynaCenter’s capture and provision services through our API.
Many of our partners’ automation and workflow management systems are designed to call scripts or external programs, so the first version of our API was implemented as a series of command line programs. However, we are increasingly seeing a desire for more seamless integration through web service APIs. Since most of our partners are Java shops, in their minds the term web service is synonymous with SOAP (Simple Object Access Protocol), an HTTP and XML-based protocol for communicating between applications. Since Python’s standard library does not include support for SOAP, we knew we would need to research third-party libraries to find one suitable for creating a web service interface to DynaCenter. Our first step was to develop a set of minimum requirements.
Basic Requirements
DynaCenter is designed with several discrete layers that communicate with each other as needed. The command line programs that comprise the existing OEM API communicate with internal services running in daemons on a central control server or on the managed systems. This layered approach separates the exposed interface from the implementation details, allowing us to change the implementation but maintain a consistent API for use by partners. All of the real work for capturing and provisioning server images is implemented inside the DynaCenter core engine, which is invoked by the existing command line programs. The first requirement we established was that the new web service layer had to be thin so we could reuse as much existing code as possible, and avoid re-implementing any of the core engine specifically for the web service.
This project was unique in that many of the features of full-stack web frameworks would not be useful to meeting our short-term requirements. We have our own ORM for accessing the DynaCenter’s database, so any potential solution needed to be able to operate without a fully- configured ORM component. In addition, we were not building a human interface, so full-featured templating languages and integration with Javascript toolkits were largely irrelevant to the project. On the other hand, while we recognized that SOAP was a short-term requirement from some of our partners, we did anticipate wanting to support other protocols like JSON in the future without having to write a new service completely from scratch.
We also knew that creating a polished product would require comprehensive documentation.
We also knew that creating a polished product would require comprehensive documentation. The WSDL (Web Service Definition Language) file for the SOAP API, which is a formal machine-readable declaration of what calls and data types an API supports, would be helpful, but only as a reference. We planned to document the entire API in a reference manual as well as with sample Java and Python code bundled in a software development kit (SDK). We could write that documentation manually, but integration with the documentation tools was considered a bonus feature.
Finally, we needed support for complex data structures. Our data model uses a fairly sophisticated representation of image meta-data, including networking and storage requirements. DynaCenter also maintains data about the peripherals in a server so that we can reconfigure the contents of images as they are deployed to run under new hardware configurations. This information is used as parameters and return values throughout the API, so we needed to ensure that the tool we chose would support data types beyond the simple built-ins like strings and integers.
Meet the Candidates
Through our research, we were able to identify three viable candidate solutions for building SOAP-based web services in Python.
The Zolera Soap Infrastucture (ZSI), is a part of the pywebsvcs project. It provides complete server and client libraries for working with SOAP. To use it, a developer writes the WSDL file (by hand or using a WSDL editor), and then generates Python source for the client and stubs for the server. The data structures defined in the WSDL file are converted into Python classes that can be used in both client and server code.
soaplib is a lightweight library from Optio Software. It also supports only SOAP, but in contrast to ZSI it works by generating the WSDL for your service based on your Python source code. soaplib is not a full-stack solution, so it needs to be coupled with another framework such as TurboGears or Pylons to create a service.
TGWebServices (TGWS) is a TurboGears-specific library written by Kevin Dangoor and maintained by Christophe de Vienne. It provides a special controller base class to act as the root of the service. It is similar to soaplib in that it generates the WSDL for a service from the source at runtime. In fact, we found a reference to the idea of merging soaplib and TGWebServices, but that work seems to have stalled out. One difference between the libraries is that TGWS also supports JSON and “raw” XML messages for the same back-end code.
Now that we had the basic requirements identified and a few candidates to test, we were able to create a list of evaluation criteria to help us make our decision.
Installing
A primary concern was whether or not a tool could be installed and made to work at all using any tutorial or guide from the documentation. We used a clean virtualenv for each application and used Python 2.6.2 for all tests. Initial evaluations were made under Mac OS X 10.5 and eventually prototype servers were set up under CentOS 4 so the rest of Racemi’s libraries could be used and the service could work with real data.
The latest official release of ZSI (2.0-rc3) installed using
easy_install, including all dependencies and C extensions. A newer
alpha release (2.1-a1) also installed correctly from a source archive
we downloaded manually. The sample code provided with the source
archive had us up and running a test server in a short time.
We were less successful using easy_install with TGWS because we did
not start out with TurboGears installed and the dependencies were not
configured to bring it in automatically. After modifying the
dependencies in the package by hand, we were able to install it and
configure a test server following the documentation. Once we overcame
that problem, we found that the official distribution of TGWS is only
compatible with TurboGears 1.0. By asking on the support mailing list,
we found patches to make it compatible with TurboGears 1.1 and were
then able to bring up a test server. Since TurboGears 2.x has moved
away from CherryPy, and TGWS uses features of CherryPy, we did not try
to use TurboGears 2.x.
We never did get soaplib to install. It depends on lxml, and installation on both of our our test platforms failed with compilation and link errors. At this point, soaplib was moved off of the list of primary candidates. We kept it open as an option in case the other tools did not pan out, but not being able to install it hurt our ability to evaluate it completely.
Feature Completeness
Since we anticipated other web-related work, we also considered the completeness of the stack. Although ZSI provides a full SOAP server, it does not easily support other protocols. Since our only hard requirement for protocols in the first version of the service was SOAP, this limitation did not rule ZSI out immediately.
Because TGWS sits on top of TurboGears, we knew that if we eventually wanted to create a UI for the service we could use the same stack. It also supports JSON out of the box, so third-party JavaScript developers could create their own UI as well.
Interoperability
Another concern was whether the tool would be inter-operable with a wide variety of clients. We were especially interested in the Java applications we expected our partners to be writing. Since we are primarily a Python shop, we also wanted to be able to test the SOAP API using Python libraries. In order to verify that both sets of clients would work without issue, we constructed prototype servers using each tool and tested them using SOAP clients in Python and Java (using the Axis libraries).
Both ZSI and TGWS passed the compatibility tests we ran using both client libraries. The only interoperability issue we came across was with the SOAP faults generated by TGWS, which did not pass through the strict XML parser used by the Java Axis libraries. We were able to overcome this with a few modifications to TGWS (which we have published for possible inclusion in a future version of TGWS).
Freshness
Our investigations showed that there had not been much recent development of SOAP libraries in Python, even from the top contenders we were evaluating. It wasn’t clear whether this was because the existing tools were stable and declared complete, or if the Python community has largely moved on to other protocols like JSON. To get a sense of the “freshness” of each project, we looked for the last commit to the source repository and also examined mailing list archives for recent activity. We were especially interested in responses from developers to requests for support.
The recent activity on the ZSI forums on Sourceforge seemed mostly to be requests for help. The alpha release we used for one of the tests was posted to the project site in November of 2007. There had been more recent activity in the source tree, but we did not want to use an unreleased package if we could avoid it.
The situation with TGWS was confusing at first because we found several old sites. By following the chain of links from the oldest to the newest, we found the most recent code in a BitBucket repository being maintained by Christophe de Vienne. As mentioned earlier, the project mailing list was responsive to questions about making TGWS work with TurboGears 1.1, and pointed us towards a separate set of patches that were not yet incorporated in the official release.
Documentation
As new users, we wanted to find good documentation for any tool we selected. Having the source is useful for understanding how you’re doing something wrong, but learning what to do in the first place calls for separate instructions. All of the candidates provided enough documentation for us to create a simple prototype server without too much trouble.
Just as we expect to have documentation for third-party tools we use, we need to provide API references and tutorials for the users of our web service. We use Sphinx for all customer-facing documentation at Racemi, since it allows us to manage the documentation source along with our application code, and to build HTML and PDF versions of all of our manuals. TGWS includes a Sphinx extension that adds directives for generating documentation for web service controllers, so we could integrate it with our existing build process easily. ZSI has no native documentation features. We did consider building something to parse the WSDL file and generate API docs from that, but the existing Sphinx integration TGWS provided was a big bonus in our eyes.
Deployment Complexity
We evaluated the options for deploying all of the tools, including how much the deployment could be automated and how flexible they were. We decided to run our service behind an Apache proxy so we could encrypt the traffic with SSL. All of the tools support the standard options for doing this (mod_proxy, mod_python, and in some cases mod_wsdl) so there was no clear winner for this criteria.
In addition to simple production deployment, we also needed an option for running a server in “development” mode without requiring root access or modifications to a bunch of system services. We found that both ZSI and TGWS have good development server configurations, and could be run directly out of a project source tree (in fact, that is how the prototype servers were tested).
Packaging Complexity
As a packaged OEM product, DynaCenter is a small piece of a larger software suite being deployed on servers outside of our control. It needs to play well with others and be easy to install in the field. Most installations are performed by trained integrators, but they are not Python programmers and we don’t necessarily want to make them deal with a lot of our implementation details. We definitely do not want them downloading dependencies from the Internet, so we package our own copy of Python and the libraries we use so that installation is simpler and avoids version conflicts.
ZSI’s only external dependency are PyXML and zope.interface. We were
already packaging PyXML for other reasons, and zope.interface was
easy to add. TGWS depends on TurboGears, which is a collection of many
separate packages. This made re-distribution less convenient, since we
had to grab the sources for each component separately. Fortunately,
the complete list is documented clearly in the installation script for
TurboGears and we were able to distill it down to the few essential
pieces we would actually be using. Those packages were then integrated
with our existing processes so they could be included in the Python
package we build.
Licensing
Although Racemi does contribute to open source tools when possible,
DynaCenter is not itself open source. We therefore had to eliminate
from consideration any tool that required the use of a GNU Public
License. ZSI uses a BSD-like license, which matched our
requirements. The zope.interface package is licensed under the Zope
Public License, which is also BSD-like. TGWS and most of the
TurboGears components are licensed under a BSD or MIT license. The
only component that even mentioned GNU was SQLObject, which uses the
LGPL. That would have been acceptable, but since we have our own ORM
and do not need SQLObject, we decided to skip including it in our
package entirely to avoid any question.
Elegance
SOAP toolkits tend to fall in one of two camps: Those that generate source from a WSDL file and those that generate a WSDL document from source. We didn’t particularly care which solution we ended up with, as long as we didn’t have to write both the WSDL and the source code. We also wanted to avoid writing vast amounts of boilerplate code, if possible. As you will see from the examples below, the tools that generated the WSDL from Python source turned out to be a much more elegant in the long run.
We also considered the helpfulness of the error messages as part of evaluating the elegance and usability of the tools. With TGWS, most of what we were writing was Python. Many of the initial errors we saw were from the interpreter, and so the error types and descriptions were familiar. Once those were eliminated, the errors we saw generated by TGWS code were usually direct and clear, although they did not always point at the parts of our source code where the problem could be fixed.
In contrast, we found ZSI’s errors to be very obscure. It seemed many
were caused by a failure of the library to trap problems in the
underlying code, such as indexing into a None value as a tuple. Even
the errors that were generated explicitly by the ZSI code left us
scratching our heads on occasion. We continued evaluating both tools,
but by this time we were leaning towards TGWS and growing more
frustrated with ZSI.
Testing
Automated testing is especially important for a complex product like DynaCenter, so being able to write tests for the new web service and integrate them with our existing test suite was an important feature. ZSI does not preclude writing automated tests, but does not come with any obvious framework or features for supporting them, so we would need to roll our own. TGWS takes advantage of TurboGears’ integration with WebTest to let the developer write unit and integration tests in Python without even needing to start a test daemon.
Performance
Once we established the ease of creating and testing services with TGWS, we had basically made our choice for that library. However, there was one last criteria to check: performance. Using the prototype servers we had set up for experimenting with the tools, we took some basic timing measurements by writing a SOAP client in Python to invoke a service that returned a large data set (500 copies of a complex type with several properties of different types). We measured the time it took for the client to ask for the data and then parse it into usable objects.
The data structure definition was the same for both services, and we found no significant difference in the performance of the two SOAP implementations. Interestingly, as the amount of data increased, the JSON performance reached a 10x improvement over SOAP. Our hypothesis for the performance difference is that there was less data to parse, the parser was more efficient, and the objects being created in the client were simpler because JSON does not try to instantiate user- defined classes.
Prototyping with ZSI
We were somewhat familiar with ZSI because we had used it in the past for building a client for interacting with the VMware Virtual Center web service, so we started with ZSI as our first prototype. For both prototypes, we implemented a simple echo service that returns as output whatever it gets as input from the client. Listing 1 contains the hand-crafted WSDL inputs for the ZSI version of this service.
Listing 1
<?xml version="1.0" encoding="UTF-8"?>
<definitions
xmlns="http://schemas.xmlsoap.org/wsdl/"
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:http="http://schemas.xmlsoap.org/wsdl/http/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:tns="urn:ZSI"
targetNamespace="urn:ZSI" >
<types>
<xsd:schema elementFormDefault="qualified"
targetNamespace="urn:ZSI">
<xsd:element name="Echo">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:anyType"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
</types>
<message name="EchoRequest">
<part name="parameters" element="tns:Echo" />
</message>
<message name="EchoResponse">
<part name="parameters" element="tns:Echo"/>
</message>
<portType name="EchoServer">
<operation name="Echo">
<input message="tns:EchoRequest"/>
<output message="tns:EchoResponse"/>
</operation>
</portType>
<binding name="EchoServer" type="tns:EchoServer">
<soap:binding style="document"
transport="http://schemas.xmlsoap.org/soap/http"/>
<operation name="Echo">
<soap:operation soapAction="Echo"/>
<input>
<soap:body use="literal"/>
</input>
<output>
<soap:body use="literal"/>
</output>
</operation>
</binding>
<service name="EchoServer">
<port name="EchoServer" binding="tns:EchoServer">
<soap:address location="http://localhost:7000"/>
</port>
</service>
</definitions>
To generate the client and server code from the WSDL, feed it into the
wsdl2py program (included with ZSI). To add support for complex
types, add the -b option, but it isn’t required for this simple
example. wsdl2py will, in response, produce three files:
Listing 2
EchoServer_client.py is the code needed to build a client for the
SimpleEcho web service.
##################################################
# file: EchoServer_client.py
#
# client stubs generated by
# "ZSI.generate.wsdl2python.WriteServiceModule"
#
##################################################
from EchoServer_types import *
import urlparse, types
from ZSI.TCcompound import ComplexType, Struct
from ZSI import client
from ZSI.schema import GED, GTD
import ZSI
from ZSI.generate.pyclass import pyclass_type
# Locator
class EchoServerLocator:
EchoServer_address = "http://localhost:7000"
def getEchoServerAddress(self):
return EchoServerLocator.EchoServer_address
def getEchoServer(self, url=None, **kw):
return EchoServerSOAP(
url or EchoServerLocator.EchoServer_address,
**kw)
# Methods
class EchoServerSOAP:
def __init__(self, url, **kw):
kw.setdefault("readerclass", None)
kw.setdefault("writerclass", None)
# no resource properties
self.binding = client.Binding(url=url, **kw)
# no ws-addressing
# op: Echo
def Echo(self, request, **kw):
if isinstance(request, EchoRequest) is False:
raise TypeError, "%s incorrect request type" %
(request.__class__)
# no input wsaction
self.binding.Send(None, None, request, soapaction="Echo", **kw)
# no output wsaction
response = self.binding.Receive(EchoResponse.typecode)
return response
EchoRequest = GED("urn:ZSI", "Echo").pyclass
EchoResponse = GED("urn:ZSI", "Echo").pyclass
Listing 3
EchoServer_server.py contains code needed to build the SimpleEcho
web service server.
##################################################
# file: EchoServer_server.py
#
# skeleton generated by
# "ZSI.generate.wsdl2dispatch.ServiceModuleWriter"
#
##################################################
from ZSI.schema import GED, GTD
from ZSI.TCcompound import ComplexType, Struct
from EchoServer_types import *
from ZSI.ServiceContainer import ServiceSOAPBinding
# Messages
EchoRequest = GED("urn:ZSI", "Echo").pyclass
EchoResponse = GED("urn:ZSI", "Echo").pyclass
# Service Skeletons
class EchoServer(ServiceSOAPBinding):
soapAction = {}
root = {}
def __init__(self, post='', **kw):
ServiceSOAPBinding.__init__(self, post)
def soap_Echo(self, ps, **kw):
request = ps.Parse(EchoRequest.typecode)
return request,EchoResponse()
soapAction['Echo'] = 'soap_Echo'
root[(EchoRequest.typecode.nspname,EchoRequest.typecode.pname)] =
'soap_Echo'
Listing 4
EchoServer_types.py has type definitions used by both the client and
server code.
##################################################
# file: EchoServer_types.py
#
# schema types generated by
# "ZSI.generate.wsdl2python.WriteServiceModule"
#
##################################################
import ZSI
import ZSI.TCcompound
from ZSI.schema import (LocalElementDeclaration, ElementDeclaration,
TypeDefinition, GTD, GED)
from ZSI.generate.pyclass import pyclass_type
##############################
# targetNamespace
# urn:ZSI
##############################
class ns0:
targetNamespace = "urn:ZSI"
class Echo_Dec(ZSI.TCcompound.ComplexType, ElementDeclaration):
literal = "Echo"
schema = "urn:ZSI"
def __init__(self, **kw):
ns = ns0.Echo_Dec.schema
TClist = [ZSI.TC.AnyType(pname=(ns,"value"),
aname="_value", minOccurs=1, maxOccurs=1,
nillable=False, typed=False,
encoded=kw.get("encoded"))]
kw["pname"] = ("urn:ZSI","Echo")
kw["aname"] = "_Echo"
self.attribute_typecode_dict = {}
ZSI.TCcompound.ComplexType.__init__(self,None,TClist,
inorder=,**kw)
class Holder:
__metaclass__ = pyclass_type
typecode = self
def __init__(self):
# pyclass
self._value = None
return
Holder.__name__ = "Echo_Holder"
self.pyclass = Holder
# end class ns0 (tns: urn:ZSI)
Once generated, these files are not meant to be edited, because they will be regenerated as part of a build process whenever the WSDL input changes. The code in the files grows as more types and calls are added to the service definition.
The implementation of the server goes in a separate file that imports
the generated code. In the example, the actual service is on lines
18–25 of Listing 5. The @soapmethod decorator defines the input (an
EchoRequest) and the output (an EchoResponse) for the call. In the
example, the implementation of soap_Echo() just fills in the
response value with the request value, and returns both the request
and the response. From there, ZSI takes care of building the SOAP
response and sending it back to the client.
Listing 5
import os
import sys
from EchoServer_client import *
from ZSI.twisted.wsgi import (SOAPApplication,
soapmethod,
SOAPHandlerChainFactory)
class EchoService(SOAPApplication):
factory = SOAPHandlerChainFactory
wsdl_content = dict(name='Echo',
targetNamespace='urn:echo',
imports=(),
portType='',
)
def __call__(self, env, start_response):
self.env = env
return SOAPApplication.__call__(self, env, start_response)
@soapmethod(EchoRequest.typecode,
EchoResponse.typecode,
operation='Echo',
soapaction='Echo')
def soap_Echo(self, request, response, **kw):
# Just return what was sent
response.Value = request.Value
return request, response
def main():
from wsgiref.simple_server import make_server
from ZSI.twisted.wsgi import WSGIApplication
application = WSGIApplication()
httpd = make_server('', 7000, application)
application['echo'] = EchoService()
print "listening..."
httpd.serve_forever()
if __name__ == '__main__':
main()
Listing 6 includes a sample of how to use the ZSI client libraries to
access the servers from the client end. All that needs to be done is
to create a handle to the EchoServer web service, build an
EchoRequest, send it off to the web service, and read the response.
Listing 6
from EchoServer_client import *
import sys, time
loc = EchoServerLocator()
port = loc.getEchoServer(url='http://localhost:7000/echo')
print "Echo: ",
msg = EchoRequest()
msg.Value = "Is there an echo in here?"
rsp = port.Echo(msg)
print rsp.Value
Prototyping with TGWebServices
To get started with TGWebServices, first create a TurboGears project
by running tg-admin quickstart which will
prompt you to name the new project and Python package, and then
produce a directory structure full of skeleton code. The directory
names are based on the project and package names chosen when running
tg-admin. The top-level directory contains sample configuration
files and a script for starting the server, and a subdirectory
containing all the Python code for the web service.
tg-admin will generate several Python files, but the important file
for defining the web service is controllers.py. Listing 7 shows the
controllers.py file for our prototype echo server. The @wsexpose
decorator on line 7 exposes the web service call and defines the
return type as a string. On line 8, @wsvalidate defines the data
types for each parameter. As with the ZSI example, the actual
implementation of the echo call just returns what is passed in.
Listing 7
from turbogears import controllers, expose, flash
from tgwebservices.controllers import WebServicesRoot, wsexpose, wsvalidate
class EchoService(WebServicesRoot):
"""EchoService web service definition"""
@wsexpose(str)
@wsvalidate(value=str)
def echo(self, value):
"Echo the input back to the caller."
return value
class Root(controllers.RootController):
"""The root controller of the application."""
echo = EchoService('http://localhost:7000/echo/')
The auto-generated WSDL for the web service is accessible via
http://<server>/echo/soap/api.wsdl. Listing 8 shows an example of
the WSDL generated by TGWS for the prototype EchoService. It includes
definitions of all types used in the API (lines 3–20), the request and
response message wrappers for each call (lines 21–26), as well as the
ports (lines 27–45) and a service definition (lines 46–51) pointing to
the server generating the WSDL document. Each port includes the
docstring from the method implementing it (line 29).
Listing 8
<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="EchoService" xmlns:types="http://localhost:7000/echo/soap/types" xmlns:soapenc="http://www.w3.org/2001/09/soap-encoding" targetNamespace="http://localhost:7000/echo/soap/" xmlns:tns="http://localhost:7000/echo/soap/">
<wsdl:types>
<xsd:schema elementFormDefault="qualified" targetNamespace="http://localhost:7000/echo/soap/types">
<xsd:element name="echo">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
<xsd:element name="echoResponse">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="result" type="xsd:string"/>
</xsd:sequence>
</xsd:complexType>
</xsd:element>
</xsd:schema>
</wsdl:types>
<wsdl:message name="echoRequest" xmlns="http://localhost:7000/echo/soap/types">
<wsdl:part name="parameters" element="types:echo"/>
</wsdl:message>
<wsdl:message name="echoResponse" xmlns="http://localhost:7000/echo/soap/types">
<wsdl:part name="parameters" element="types:echoResponse"/>
</wsdl:message>
<wsdl:portType name="EchoService_PortType">
<wsdl:operation name="echo">
<wsdl:documentation>Echo the input back to the caller.</wsdl:documentation>
<wsdl:input message="tns:echoRequest"/>
<wsdl:output message="tns:echoResponse"/>
</wsdl:operation>
</wsdl:portType>
<wsdl:binding name="EchoService_Binding" type="tns:EchoService_PortType">
<soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
<wsdl:operation name="echo">
<soap:operation soapAction="echo"/>
<wsdl:input>
<soap:body use="literal"/>
</wsdl:input>
<wsdl:output>
<soap:body use="literal"/>
</wsdl:output>
</wsdl:operation>
</wsdl:binding>
<wsdl:service name="EchoService">
<wsdl:documentation>WSDL File for EchoService</wsdl:documentation>
<wsdl:port binding="tns:EchoService_Binding" name="EchoService_PortType">
<soap:address location="http://localhost:7000/echo/soap/"/>
</wsdl:port>
</wsdl:service>
</wsdl:definitions>
The tgwsdoc extension to Sphinx, distributed with TGWS, adds several
auto-documentation directives to make it easy to keep your
documentation in sync with your code. By using autotgwstype,
autotgwscontroller, and autotgwsfunction, you can insert
definitions of the complex types, controllers, or individual API calls
in with the rest of your hand-written documentation. This was
especially useful for us because we already had a lot of text
explaining our existing command line interface. We were able to reuse
a lot of the material and document all three interfaces (command line,
SOAP, and JSON) with a single tool.
Implementation Considerations
Once we had chosen TGWS as our framework, we set about working on the first implementation of our real service. This helped us uncover a few small problems with our original “pure” design, and some details we had not considered while prototyping.
For example, we wanted to make sure that our web service was not only interoperable with Java clients, but also that the API made sense to a Java developer. One tool that they might be using, the Java Axis client, is built by feeding the WSDL file into a code generator to produce source code for client classes. After we tried working with the generated Java code, we adjusted our web service API to make it more usable. For instance, Java doesn’t allow you to specify defaults for method arguments, which caused problems with a couple of web service calls that had a handful of required arguments along with many optional keyword arguments. On the Java side, the caller would have to pass in all 23 parameters to the call, most of them null placeholders for the optional parameters. To address that, we moved all the optional parameters to a separate “options” object that could be populated and passed in for advanced operations.
There were other minor annoyances, such as the way a camelCase
naming convention resulted in nicer-looking Java code than the
under_scored naming convention typically used by Python
programmers. We ended up going with camelCase names for attributes
and methods of classes used in the public side of the web
service. After making these tweaks, it is not difficult to design an
API with TGWS that makes sense to both Java and Python client
developers.
Testing in Java was another challenge for us to work out. We have a
large suite of Python tests driven by nose, and we ultimately were
able to automate the client-side Java testing using junit. We
then integrated the two suites by writing a single Python test to run
all of the junit tests in a separate process and parse the results
from the output.
In addition to developer tests, Racemi has a dedicated group of test engineers who perform QA and acceptance tests before each new version of DynaCenter is released. The QA team needed a client library to use for testing the new web service. None of them are Java programmers, so the Dev team took on the task of basic Java integration testing. But for full-on regression testing and automation, QA needed something lightweight and easy to get up and running with quickly. Suds fit this bill quite nicely. It is a client-only SOAP interface for Python that reads the WSDL file at runtime and provides client bindings for the web service API. Armed with our WSDL and the Suds documentation, our QA team was able to start building a client test harness almost immediately.
Conclusions
At the beginning of our evaluation process, we knew there were a lot of ways to compare the available tools. At first, we weren’t sure if the code-from-WSDL model used by ZSI or the WSDL-from-code model used by TGWebServices and soaplib would be easier to use. After creating the simple echo service prototype with both tools, we found that writing Python and generating the WSDL worked much better for us. Because WSDL is an XML format primarily concerned with types, we found it excessively verbose compared to the Python needed to back it up. It felt much more natural to express our API with Python code and then generate the description of it. Starting with the code also lead to fewer situations where translating to WSDL produced errors, unlike when we tried to manage the WSDL by hand.
Because WSDL is an XML format primarily concerned with types, we found it excessively verbose compared to the Python needed to back it up.
As mentioned earlier, we ended up needing to patch TGWebServices to make it work correctly with TurboGears 1.1. Those patches were available on the Internet as separate downloads, but we decided to “fork” the original Mercurial repository and create a new version that included them directly. We have also added a few other enhancements, such as the option of specifying which formats (JSON and/or XML) to use when documenting sample types, and better SOAP error message handling. We are working with Christophe de Vienne to move those changes upstream.
TGWebServices stood out as the clear winner for our needs.
Aside from the ease of use benefits and technical merits of TGWebServices, there were several bonus features that made it appealing. The integration with Sphinx for generating documentation meant that not only would we not have to write the reference guide as a completely separate task, but it would never grow stale as code (especially data structures) changed during the evolution of the API. Getting the JSON for “free” was another big win for us because it made testing easier and did not lock us in to a SOAP solution for all of our partners. Couple that with the benefit of having the TurboGears framework already in place for a possible web UI down the road, and TGWebServices stood out as the clear winner for our needs.