I am extending the contest until Feb the 1st – Mozillians, win a free pass for Pycon US

Cornice is growing steadily, and we are thinking about the different ways to use it for our needs. One use case that comes often when we build web services is the need to publish a SQL Database via HTTP.

For instance, in a project I am working on, we might expose a list of servers and some information about them, that are stored in a SQL DB . The goal is to allow some management scripts to interact with the DB, to set and retrieve information about the servers, like: “can I use server 12 as a node for application X ?”

Interacting with CURL or a similar tool is simpler and more portable than coding yet another SQL client for this, so the idea is to see how this kind of web service can be done is the minimum pain with Cornice.

What I am thinking about building is a small CRUD interface that glues Cornice and SQLAlchemy. The latter has a way to define a database schema explicitly via mappings meaning that it’s easy to write a generic layer that exposes the database to the web via Cornice definitions. The work consists of transforming POST & PUT requests that contains data to write to the DB into SQLAlchemy objects, and transforming select results asked via GET requests into the proper responses.

Nothing very new, there are tons of existing systems that implement CRUD on the top of ORMs or plain SQL libraries. The only reason to build yet another one is to use it in the context of our current toolset which is composed of Cornice, Pyramid & SQLAlchemy for most projects. The whole code will probably be less than 300 lines at the end anyways.

Oh my.

Turns out this idea is really freaking out some people around me. There’s a strong aversion of some coders against anything that looks a bit like Active Records — in the Rails Context. In other words anything that would completely automate the serialization & deserialization layer and make it hard to tweak some code.

Another criticism is that a CRUD system would not be able to scale in the context of a big database, like Firefox Sync, that uses numerous databases to shard data.

Turns out building a CRUD on tools like SQLAlchemy or Pyramid is not really going to ruin your scalability as long as:

• you can tweak the serialization / deserialization
• you can override any operation in the CRUD operations when needed
• you don’t shoot yourself in the foot by using CRUD with some code or DB that is not meant to be used that way
• you can use the power of the underlying tools without being blocked by the lib

For the latter, Ben Bangert was pointing me at SQLAlchemy horizontal feature, which is basically what I wrote from scratch last year to make the Sync server shard across databases… At this point I sense that Firefox Sync could have been built with a CRUD lib, and be as efficient as it is today, because when I look at the queries produced by the code and the one a CRUD lib would produce, we are one or two tweaks away.

Anyways, here’s a first attempt at such a library.

Defining the model

In SQLAlchemy, you can define the DB model using mappings, which are simple classes containing a description of the tables.

For example, if I have a class “users” with a field “id” and a field “name”, the mapping will look like this:

class Users(_Base):                                           
    __tablename__ = 'users'                                      
    id = Column(Integer, primary_key=True)                    
    name = Column(String(256), nullable=False)

What I started to do is write a meta class one can use in a class to publish the mapping via HTTP.

Here’s an example:

from cornicesqla import MetaDBView
from myapp import Users, DBSession

class UsersView(object):
    __metaclass__ = MetaDBView
    mapping = Users
    path = '/users/{id}'
    collection_path = '/users'
    session = DBSession

What we have here is the definition of a view for the Users mapping. The class defines an URI for the collection (collection_path) and for each user (path). The session attribute is an SQLAlchemy session object you usually define when you work with that tool.

That’s it.

The model gets published, and you can GET, PUT, POST and DELETE on /users and /users/someid.

The code of the prototype is here and you can find a working example in the tests here. It’s called cornice-sqla

Tweaking serialization & data validation

By default, cornice-sqla will serialize and deserialize using JSON but you can tweak these steps by providing a custom serializer, or deserializer (or both.)

Let’s say you want to use the Colander libary to validate and serialize the data. To do this, you just have to write your serializer method into the view class

class UsersValidation(colander.MappingSchema):
    name = colander.SchemaNode(colander.String())

class UsersView(object):
    __metaclass__ = MetaDBView

    mapping = Users
    path = '/users/{id}'
    collection_path = '/users'
    session = DBSession

    def serialize(self):
        """Unserialize the data from the request, to serialize it for the DB"""
        try:
            user = json.loads(self.request.body)
        except ValueError:
            request.errors.add('body', 'item', 'Bad Json data!')
            # let's quit
            return

        schema = UsersValidation()
        try:
            deserialized = schema.deserialize(user)
        except Invalid, e:
            # the struct is invalid
            request.errors.add('body', 'item', e.message)

        return deserialized

Colander is used here to validate the incoming request and create a flat mapping we can push into the DB. Cornice’s error system is in usage here, as explained here.

You can tweak the data that gets back from the DB with unserialize(), and for the collection URI, use collection_serialize() and collection_unserialize().

Tweaking C, R, U or D

cornice-sqla is based on a fresh feature Gael added into Cornice lately: resources. A resource is a class where you can define get(), post(), delete() and put() methods for a given URI.

cornice-sqla views are based on resources, meaning that you can override anyone of those methods and do whatever you want if you don’t want the CRUD part.

What’s next

I need to make sure everything you can do in Cornice (acls various options etc) can still be done in cornice-sqla, and start to work with more complex DB schema that include relations etc.  I also need to add basic missing features like batching and some docs.

My hope at the end is that this small library will reduce considerably the code needed in some of our projects that interact with SQL.

If you have been following closely the latest work done by Chris on WebOb, you know that WebOb and eventually Pyramid became Python 3 compatible.

That makes Python 3 a very tempting target for a new web project.

Paste & PasteScript still need to be ported to Python 3 and the Pyramid team has chosen not to. They have created their own paster replacer instead, which can be used to initiate a Pyramid project or run the app using the .ini file.

I am wondering if it would not be simpler at this point to drop Paste and use this replacer for all Python 3 frameworks that are using the Paste script and templates features.

Besides all the features Pyramid and its libs turns out most of the libs you usually need to build a classical web app already support Python 3, like SQLALchemy and PyMysql for MySQL access, Pylibmc for Memcached;

Things I am still missing in Python 3:

• gevent
• gunicorn
• python-ldap
• Cornice — I will port it soon

If you want to give it a shot, get the latest Python 3.2 and grab more details at : https://github.com/Pylons/pyramid/wiki/Python-3-Porting

And if you miss one lib, add it here

Merry Christmas !

At this stage, I think we’ve added enough helpers in Cornice to get anyone started in building web services in Python.

As a reminder, Cornice provides helpers to build & document REST-ish Web Services with Pyramid, a Python web framework. The main benefits of Cornice are:

• automatic handling of some HTTP errors – Ask yourself: is your app handling properly 405 or 406 errors?
• automatic web service documentation via a Sphinx extension.
• a simple way to validate and convert requests data, and return structured 400 responses.

This is a small tutorial, extracted from our documentation.

Let’s create a full working application with Cornice. We want to create a light messaging service.

You can find its whole source code at https://github.com/mozilla-services/cornice/blob/master/examples/messaging

Features:

• users can register to the service
• users can list all registered users
• users can send messages
• users can retrieve the latest messages
• messages have three fields: sender, content, color (red or black)
• adding a message is done through authentication

Limitations:

• there’s a single channel for all messages.
• if a user with the same name is already registered, he cannot register.
• all messages and users are kept in memory.

$ mkdir messaging
$ cd messaging
$ virtualenv --no-site packages .

$ bin/pip install Cornice

$ bin/paster create -t cornice messaging
Selected and implied templates:
cornice#cornice  A Cornice application

Variables:
egg:      messaging
package:  messaging
project:  messaging
Enter appname (Application name) ['']: Messaging
Enter description (One-line description of the project) ['']: A simple messaging service.
Enter author (Author name) ['']: Tarek
Creating template cornice
...
Generating Application...
Running python2.7 setup.py egg_info

$ cd messaging
$ ../bin/python setup.py develop
...

$ ../bin/paster serve messaging.ini
Starting server in PID 7618.
serving on 0.0.0.0:5000 view at http://127.0.0.1:5000

{'Hello': 'World'}

from cornice import Service

hello = Service(name='hello', path='/', description="Simplest app")

@hello.get()
def get_info(request):
    """Returns Hello in JSON."""
    return {'Hello': 'World'}

_USERS = {}

users = Service(name='users', path='/users', description="Users"0

@users.get(validator=valid_token)
def get_users(request):
    """Returns a list of all users."""
    return {'users': _USERS.keys()}

@users.put(validator=unique)
def create_user(request):
    """Adds a new user."""
    user = request.validated['user']
    _USERS[user['name']] = user['token']
    return {'token': '%s-%s' % (user['name'], user['token'])}

@users.delete(validator=valid_token)
def del_user(request):
    """Removes the user."""
    user = request.validated['user']
    del _USERS[user['name']]
    return {'goodbye': user['name']}

import os
import binascii
from webob import exc

def _create_token():
    return binascii.b2a_hex(os.urandom(20))

def valid_token(request):
    header = 'X-Messaging-Token'

    token = request.headers.get(header)
    if token is None:
        raise exc.HTTPUnauthorized()

    token = token.split('-')
    if len(token) != 2:
        raise exc.HTTPUnauthorized()

    user, token = token

    valid = user in _USERS and _USERS[user] == token
    if not valid:
        raise exc.HTTPUnauthorized()

    request.validated['user'] = user

def unique(request):
    name = request.body
    if name in _USERS:
        request.errors.add('url', 'name', 'This user exists!')
    else:
        user = {'name': name, 'token': _create_token()}
        request.validated['user'] = user

$ curl http://localhost:5000/users
{"status": "error", "errors": [{"location": "header",
                                "name": "X-Messaging-Token",
                                "description": "No token"}]}

$ curl -X PUT http://localhost:5000/users -d 'tarek'
{"token": "tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"}

$ curl http://localhost:5000/users -H "X-Messaging-Token:tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"
{'users': ['tarek']}

$ curl -X DELETE http://localhost:5000/users -H "X-Messaging-Token:tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"
{'Goodbye': 'tarek}

messages = Service(name='messages', path='/', description="Messages")

_MESSAGES = []

@messages.get()
def get_messages(request):
    """Returns the 5 latest messages"""
    return _MESSAGES[:5]

@messages.post(validator=(valid_token, valid_message))
def post_message(request):
    """Adds a message"""
    _MESSAGES.insert(0, request.validated['message'])
    return {'status': 'added'}

def valid_message(request):
    try:
        message = json.loads(request.body)
    except ValueError:
        request.errors.add('body', 'message', 'Not valid JSON')
        return

    # make sure we have the fields we want
    if 'text' not in message:
        request.errors.add('body', 'text', 'Missing text')
        return

    if 'color' in message and message['color'] not in ('red', 'black'):
        request.errors.add('body', 'color', 'only red and black supported')
    elif 'color' not in message:
        message['color'] = 'black'

    message['user'] = request.validated['user']
    request.validated['message'] = message

$ bin/pip install Sphinx

$ mkdir docs
$ sphinx-quickstart
Welcome to the Sphinx 1.0.7 quickstart utility.

..

Enter the root path for documentation.
> Root path for the documentation [.]: docs
...
> Separate source and build directories (y/N) [n]: y
...
> Project name: Messaging
> Author name(s): Tarek
...
> Project version: 1.0
...
> Create Makefile? (Y/n) [y]:
> Create Windows command file? (Y/n) [y]:

import cornice
extensions = ['cornice.sphinxext']

Welcome to Messaging's documentation!
=====================================

.. services::
   :package: messaging

Hey I did this in 2009, let’s try again — I am adding one extra question this year

1. What’s the coolest Python application, framework or library you have discovered in 2011 ?

GEvent & Pyramid. Not discoveries, but a daily usage. GEvent was for me a fantastic way to make the Firefox Sync Python server scale without being forced to write callback-style code. Pyramid is a very elegant framework, that takes the simplicity from Pylons and the power and experience from Repoze & the Zope world. A good sign for me is that we don’t have to deal with the ZCA

2. What new programming technique did you learn in 2011 ?

Better behaviour in high loaded server apps. During the last year, when we wrote all the pieces that makes Firefox Sync today, I’ve learned how to be more careful on how my apps would react when a back-end breaks or cease to reply, when a database gets slow, or when some service that’s used restarts — or when my own app restarts, still hammered by many requests. I did a fair amount of work on this, like smart pools of connectors and better testing techniques, and make decisions on what features survive when some third-party server is down, and what features just go 503.

3. What’s the name of the open source project you contributed the most in 2011 ? What did you do ?

Mozilla. I have not contributed as much as last year in Python because my work at Mozilla takes most of my time, but the good news is that all our stuff is open source so.. The most useful stuff for the community we’ve started is probably Cornice. But we’ve written and writing a myriad of apps and libs. See https://github.com/mozilla-services and http://hg.mozilla.org/services

In Python I still interact a bit with what’s going on in Packaging and hope I’ll be able to spend more time on it in 2012. But some packaging work I needed at work was also useful for the community, like pypi2rpm.

4. What was the Python blog or website you read the most in 2011 ?

Like in the past few years, Python Reddit. And I think I am not alone in that case. 90% of my blog hits come from Reddit

5. What are the three top things you want to learn in 2012 ?

I’d like to learn how to program in a few new languages, just to give them a shot. Maybe Haskell. I’d also like to finish a spare project I have started with Benoit, and try to launch it, promote & market it. Last, I’d like to learn more about Firefox arcanes — just for my culture.

6. What are the top software, app or lib you wish someone would write in 2012 ?

• I want to take a picture of a wine bottle and have it recognized in an online app, where I can share my thoughts about its taste.
• I want an Android virtual ping-pong application, where you can use your phone as paddle and see the e-ball through the camera & play with a friend.

Want to do your own list ? here’s how:

• copy-paste the questions and answer to them in your blog
• tweet it with the #2012pythonmeme hashtag

Yesterday, we’ve launched the Developer Preview for Apps, you can play with at https://apps-preview.mozilla.org

The server side is composed of many pieces, and while they are subject to change since this is just a preview, I think it’s quite interesting to describe some of them already — and maybe get more contributors in the process, since everything is open source and contributors are welcome.

Here’s an overview of the system — we used:

• Django for the App MarketPlace
• Cornice for the Sync APIs
• Node.js for the Sauropod APIs
• HBase for the Sauropod DB

The App MarketPlace – Django

The App MarketPlace located at https://apps-preview.mozilla.org is where you can upload your own Apps, or install some. There’s a payment process for non-free Apps, and you can see which one you have bought in your profile.

The development is driven by the WebDev team, and is based on Django — see https://github.com/mozilla/zamboni/tree/master/apps/webapps

I can’t really describe this part, as I did not follow it closely. But basically, the Market Place keeps track of your apps and payment receipts, for other parts of the system to interact with.

The Dashboard – HTML + JS

Once you start to install Open Web Applications, you are redirected to a Dashboard at https://myapps.mozillalabs.com. This Dashboard is an HTML application that lists your installed Apps associated to your Browser ID.

What’s pretty cool is that no matter where you’ve installed a given App — Firefox on your Desktop, your Phone, it will appear on this dashboard, and synced across devices.This is done via a Sync service called AppSync.

Code pointers for the Dashboard:

• https://github.com/mozilla/openwebapps/tree/develop/site
• https://github.com/mozilla/openwebapps/blob/develop/addons/jetpack/lib/sync.js

AppSync

AppSync is the part I worked on. Its design was mainly done by Ian Bicking, who then worked on the client side implementation while I was working on the server side one.

It’s quite similar to what we did for Firefox Sync, except that:

• AppSync supports BrowserID
• The data is stored in Sauropod

Securing this data is part of a much larger ongoing project called Sauropod. The idea is that any database access has to be done with credentials, and that Sauropod is in charge of controlling them and dealing with the storage.

In the long term, depending how Sauropod evolves with respect to encryption, and how Firefox Sync itself evolves with respect to Browser ID and Sauropod access, we might merge both projects.

Or maybe Sauropod will provide APIs one day that are good enough for direct clients interactions, turning AppSync into a simple proxy ?

Time will tell !

Anyway, here’s an overview below of the AppSync system we’ve set up for this preview.

We have the AppSync server itself, that’s built using Cornice. It provides the synchronisation APIs described in this document: https://wiki.mozilla.org/Apps/Sync/Spec

Every time a client wants to write new data, we call the Sauropod server which is a very simple GET/SET api built with Node.js.

The flow is:

• AppSync server asks for a new Sauropod session, using Browser ID credentials
• Sauropod verifies the Browser ID credentials then create a DB token into a session
• AppSync uses this token until its not valid anymore
• Sauropod calls in turn an HBase cluster to access the data
• Every write is mirrored in a MySQL database in AppSync. This mirroring was set so we can turn off Sauropod if we need to, and still have a working system. This will eventually go away later.

Find the code here:

• Server https://github.com/mozilla/appsync
• Client https://github.com/mozilla/openwebapps/tree/develop/sync

What’s Next

I am really excited by this project. There are a lot of people involved in the Mozilla community, and seeing all the moving pieces assembled to build an Open App environement is pretty cool.

We’re going to work in the upcoming months on consolidating the whole system, making sure it scales well and correct the design as the feedback comes back.

If you want to get involved, you can install the preview, play with the available Apps and even maybe write your own Apps for the Market Place, or help us in the coding.

We’re hanging in #openwebapps on Mozilla’s IRC

EDIT : Anant wrote a very nice blog post on the topic

The other task Alexis and I are going to work on this week, besides Cornice, is a QA script for web services.

The goal is simple : check that a set of web services are HTTP compliant. For example, does your application send the proper 406 error when an unsupported Accept is asked by the client ?

If you document properly your web services, asking for an unsupported Accept should not occur of course, but in most projects those protocol details are often a bit vague. And someone that writes a client software will inevitably make some assumptions based on his HTTP knowledge on how the application is supposed to behave.

Richard Newman came up with a fair list of tests we could run against a web app already, and we’ve started to summarise and add more of them here: https://wiki.mozilla.org/Services/WALint

The idea of the script is to print out a report of errors and warnings it found on a web app, exactly like a lint tool would do on some code. That’s what I called it WALint (Web App Lint). Alexis doesn’t like the name but he did not find a better name yet

The way it works is that you describe in a configuration file the URIs of your web services, then WALint runs tests against them, using what we called controllers.

Each controller is in charge of trying out something on the web app, using a small HTTP test client (using WebTest), given a path and a method.  WALint will provide built-in controllers and will be extensible. We will have Mozilla-specific tests, like the maximum size of a query string, or the maximum size of the request, since those limits are specific to the used stack.

We got bitten by this is the past in Sync – one web service failed to work properly because the client was building a super long query string, that was truncated along the way in our stack.

Our final goal with this tool is to be able to add in Jenkins these controls for all our web apps, and catch more problems before they occur in production.

Since it’s also useful while you build your code, WALint will have a UnitTest integration, so you can run it as a functional test from within your test suite — In that case, it will run directly against the code.

As usual, feedback & contributions are welcome. The code is being built here: https://github.com/mozilla-services/walint

Alexis started to work with me this week from my house and we’re having good times. More geeks from Afpy are joining us this week-end for wine-drinking and working on a Python technical writing project.

While working myself on AppSync with the fine folks from Mozilla Labs,  I am mentoring Alexis to get him up to speed on our projects & standards at Mozilla Services.

Cornice, is a perfect match for him to start working — it’s isolated enough so he can have fun and produce features that are immediately useful, yet learn our standards & processes.

So we worked on its design with the help of its principal user, Ben Bangert, and I asked Alexis to blog about the work & doc we produced since Monday.

Read up => Introducing Cornice

If you’re a VIM user and a Python coder, that could interest you.

Syntastic, a syntax checker for VIM now includes Flake8 as a checker for Python code.

Syntastic can be used to check for syntax errors in files opened in VIM. It has a pretty extensive list of supported languages.

For Python it had PyFlakes, and since the latest version, it has my small glue script “Flake8″, that reunites in a single stream: pep8, PyFlakes and a McCabe checker.

For Flake8 the next step is to support Python 3 — as soon as I find some time, or a volunteer

Thanks to Sylvain Soliman (and Clayton Parker) !

Since we’ve initially started Cornice at Services, we had more discussion about how we could make it easier for developers to validate an incoming request.

Our goal is :

• to be able to validate a request and if needed, to convert it to specific data structures
• to complete the documentation of our web services in Sphinx with those validation steps

Here’s a concrete example: A PUT request used to create a user in a database should come as a JSON object in the request body, which content would be validated and turned into a Person object, then sent to a SQL backend.

The web service needs in that case to reject any malformed request with a 400, and we’d also want to document this constraint in the web service documentation.

There are some existing tools in the validation arena, like FormEncode or Colander.

FormEncode and Colander both provide data validation via schemas, and FormEncode offers HTMLform generation.

When you need to validate an incoming request in your web service, those tools can fit your needs or simply be overkill. So, we wanted to integrate a validation step in Cornice without forcing the usage of a specific validation tool.

A simple callable

What we did is added a validator option that needs to point to a callable. The callable receives the request object and has to return an HTTP error code followed by a reason in case the request does not comply.

Here’s an example, a GET /foo that will return a 402 if the X-Paid header is missing :

    from cornice import Service

    foo = Service(name='foo', path='/foo')

    def has_paid(request):
        """The request must have an X-Paid header containing a token.

        This header proves the user has paid
        """
        if not 'X-Paid' in request.headers:
            return 402, 'You need to pay !'

    @foo.get(validator=has_paid)
    def get_value(request):
        """Returns the value.
        """
        return 'Hello'

From there, Cornice will call has_paid prior to get_value. Cornice is also able to build the documentation of the web service, by merging the docstrings of get_value and has_paid.

Sphinx integration

We provide a Sphinx extension for documenting a Cornice based project. You can inject in your Sphinx document the web service description, via a service directive.

.. services::
   :package: demoapp

In this example, the directive looks for all Cornice definitions in the demoapp package by scanning it, and injects their documentation in the Sphinx document.

The service directive provides a few options. For instance the name option will let you inject the documentation of one specific web service.

Colander integration

I said earlier simple callables where good enough for simple validation cases. Let’s take a more complex example.

Let’s say, you have a Person schema in Colander, that defines a person’s attributes — See Colander docs for details.

And you want to provide a PUT Web Service to create a person, where the request body is the person data serialized in JSON.

Here’s the full Cornice definition

   from cornice import Service
   from cornice.schemas import save_converted, get_converted

   def check_person(request):
      """The request body must be a JSON object describing the Person"""
      try:
          person = json.loads(request)
      except ValueError:
          return 400, 'Bad Json data!'

      schema = Person()
      try:
          deserialized = schema.deserialized(person)
      except InvalidError, e:
           # the struct is invalid
           return 400, e.message

      save_converted(request, 'person', deserialized)

    service = Service(name='person', path='/person/{id}')

    @service.put(validator=check_person)
    def data_posted(request):
        person = get_converted(request, 'person')
        ... do the work on person ...

In this example, the validator checks that the request body is valid Json, then pass the unserialized mapping to Colander to check that it’s a valid Person record. Last, it uses the save_converted function we provide, to save the Person object into the request.

The web service then can pick it up with the get_converted function.

Next Steps

Our next steps is to build a library of useful built-in validators. Things like:

• is the request body is JSON ?
• do we have param X, if yes is it an integer ?
• any reusable validator we’ll think about

The final goals here are :

1. have our web services code written in two phases: the validation phase, and the code itself. Because they are a lot of reusable bits in that first phase.
2. reuse as much as possible docstrings to document our web services, to avoid the doc-is-not-up-to-date-anymore plague

You can follow the development of Cornice here: https://github.com/mozilla-services/cornice

And even participate by joining our Mailing List: https://mail.mozilla.org/listinfo/services-dev