Fetchez le Python

Circus 0.4 coming soon

Tarek Ziadé — Sat, 12 May 2012 15:00:00 +0200

Note

Circus is a program that will let you run and watch multiple processes. Like Supervisord, BluePill and Daemontools.

The next release of Circus, that should be happening soon, has some new features I want to hilight in this post:

circusd-stats, a process that streams statistics on Circus.
circus-top, a top-like console
circushttpd, a Web Managment interface

circusd-stats

In Circus 0.3, if you want to know how much CPU and Memory each process takes, you can use circusctl:

$ bin/circusctl stats dummy
dummy:
1: 91238  python root 0 3M 2G 62.6 0.1 0:04.80
3: 91240  python root 0 3M 2G 63.2 0.1 0:04.80
2: 91239  python root 0 3M 2G 63.1 0.1 0:04.80
5: 91242  python root 0 3M 2G 62.8 0.1 0:04.82
4: 91241  python root 0 3M 2G 62.6 0.1 0:04.76

I was a bit amazed by the inaccuracy of our CPU stats when I compared them to a simple call to ps or top.

The problem was that psutil, the library we use to build the stats, has two ways of calculating the CPU usage:

a blocking call, that calculates the usage of the CPU for a bit of time, like 500ms.
a non-blocking call, that just returns the value since the last call, that will be accurate as long as you wait at least 100ms between two calls.

The problem with any of these techniques is that, when you have hundreds of processes running, you can't really do this calculation on the fly without making the client wait for a long time. And Circus naively used the non-blocking calls, meaning that the CPU usage values was relying on the frequency of client calls.

circusd-stats solves this problem by constantly calculating in a separate process all the stats for all the processes. It has a dedicated thread per watcher, and simply listens to Circus pub/sub socket to know when a process is added or removed.

circusd-stats itself publishes at a constant pace in a new, dedicated ZMQ pub/sub channel the stats for each process, and also aggregated stats for each watcher.

From there, any feature that needs accurate stats can register to that pub/sub channel to receive the stats. Stats can be filtered by watcher name, by PID, or the client can get all of them.

circusd-stats CPU usage can be quite important, so we've added configurable intervals between two stats calls.

circus-top

The first stats client we've added into Circus is circus-top, a console script that uses curses to display a console dashboard where every process CPU and memory usage is constantly refreshed.

It's a read-only console, so you can't do anything like killing a process, but it is a good way to check if there's any issue with your system

circushttpd

I have already talked about the Web Admin tool we were building here : http://blog.ziade.org/2012/04/23/a-web-admin-for-circus/

And well, thanks to Adnane, it now look really great. Check out the new screencast: https://plus.google.com/106436370949746015255/posts/U6yKgbLn4a4

circushttpd is a heavy user of stats, and instead of using the stats command circusd provides, it now uses the circusd-stats stream.

What's next

We're currently changing circusdhttp so it uses socket.io instead of the current polling. socket.io will use web sockets when the browser is compatible, and fallback to XHR-polling.

Once this integration is ready, we'll cut a 0.4

An exciting feature is planned for 0.5 : circus-cluster, a tool to drive several instances of circus we'll call Circus Nodes.

This new feature will allow you to run and manage a whole cluster. circus-top and circushttpd will be adapted to work with circus-cluster as well.

A Web admin for Circus

Tarek Ziadé — Mon, 23 Apr 2012 18:51:00 +0200

We're working on adding a Web Administration Panel to Circus. Like many tools out there, the idea is to let you control Circus from the web.

This new feature is currently built with:

The Bottle Framework -- A one-module wsgi framework.
The Sparkline jQuery Extension -- A JS lib to draw cute little charts.
The Circus Client class, that let you connect to a Circus system and interact with it via ZMQ messages.

The prototype currently does AJAX Polls to update the data on the page every second, but a lot of things are being worked out to improve this.

For instance, the web app keeps in memory the last 100 stats calls in order to build the charts series, and serves them to the clients on every poll.

We're going to change this by collecting and building the series on the client only.

We'll also add a bit of Memoization so we serve back the same stats values in a given timespan to avoid overloading Circus with web stats calls.

This is especially important when the Web UI displays charts for hundreds of processes.

And err... we need to make the UI pretty and usable. I am doing my best here, but let's face it -- I suck at CSS :)

If you are a designer and want to help there...

You can see a screencast of the prototype here: http://blog.ziade.org/circus-web-panel.m4v

This screencast was built with the crypto demo of the Powerhose project at: https://github.com/mozilla-services/powerhose/blob/master/examples/crypto.ini

This demo runs crypto workers, a broker, and a web app.

In this screencast I do the following:

connect the Circus Web Admin app to a Circus system
check how the processes of the workers watcher do
add & remove some processes
start & restart the web worker

Links:

the doc: http://circus.readthedocs.org/en/latest/index.html
the repo: https://github.com/mozilla-services/circus

Please let us know what you think !

Circus 0.3 released

Tarek Ziade — Tue, 17 Apr 2012 21:29:00 +0200

Note

Circus is a program that will let you run and watch multiple processes. Like Supervisord, BluePill and Daemontools.

The third release of Circus is here. It wanted to hightlight three new features we've added since 0.1:

rlimit support
stderr and stdout streaming
flapping detection

rlimit support

Circus comes now with rlimit support. For example, if you want a specific process to have a limited number of open files to 100, you can use in your Circus configuration file the rlimit_nofile option:

[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555

[watcher:myprogram]
cmd = myprogram
rlimit_nofile = 500

This feature is built-in in the Python standard library, in the resource module so that was easy to add. The next step here would be to see how Circus could interact with tools like cgroups.

stderr and stdout streaming

That feature is a must have, and we worked quite a bit on it to make sure it's fast even with hundreds of processes being watched : the ability to stream the standard output and standard error streams of all processes back into Circus.

For example, if you want all processes to write their stdout continously into a file, you can write:

[watcher:myprogram]
cmd = myprogram
numprocesses = 100

# will push in test.log the stream every 300 ms
stdout_stream.class = FileStream
stdout_stream.filename = myprogram.log
stdout_stream.refresh_time = 0.3

The core of this feature is a call to the select() function from the standard library on the PIPEs opened on each process.

The gist of this code is:

import fcntl

for pipe in pipes:
    fcntl.fcntl(pipe, fcntl.F_SETFL, os.O_NONBLOCK)

while True:
    rlist, __, __ = select(pipes, [], [])
    for pipe in rlist:
        try:
            data = pipe.read(self.buffer)
            redirect_to_circus(data)
        except IOError, ex:
            if ex[0] != errno.EAGAIN:
                raise

redirect_to_circus basically redirects the stream to whatever class you've configured in stdout_stream.class. You can provide your own class if you want to implement a specific stream handler.

We've implemented this stream redirector with one thread per watcher that operates the select() calls, but also have a Gevent implementation that uses greenlets instead of threads and Gevent's own select() implementation.

The threaded version is the default one, but you can pick the gevent backend with the stream_backend option.

Flapping

Yet another important feature to have: the ability to detect that a process that's launched constantly dies. That happens when the process command line is wrong or when a resource the program uses is not reachable for example.

Tools like daemontools will simply try again and again to run the service, eating in the process the whole CPU of your server.

With this feature built-in and enabled by default, Circus detects the flapping of processes, and try again later then eventually quit trying.

Of course, the flapping dance is published in the PUB/SUB channel so it's easy to build a script that will send you an alert when it happens.

What's next

We've been doing extensive load testing for a Mozilla project that's going in production next week, and Circus seems to be quite stable so far. It's handling around 150 processes per server right now, and everything's working like a charm.

I think Circus has reached a level now where it could replace tools like Daemontools in some of our set ups.

The next major step is to refactor a bit the tool to make it more pluggable.

For example, the flapping detection is built-in, but could be factored out as a plug-in since it subscribes to the PUB/SUB channel to get notified when a process is restarted, and could control everything as a client like circusctl.

In other words, it would be nice to provide a base class that gets notifications and acts upon. The auto-grow feature I want to add --a feature where Circus adds automatically workers depending on the load-- could be a plugin based on that class.

Links:

the doc: http://circus.readthedocs.org/en/latest/index.html
the release: http://pypi.python.org/pypi/circus/0.3.1
the repo: https://github.com/mozilla-services/circus

Please let us know what you think !

Signpic - Signing Pictures with PIL

Tarek Ziade — Sat, 14 Apr 2012 23:05:00 +0200

I am starting to really like taking pictures, thanks to the new Pentax K5 I bought.

My photo processing looks roughly like this:

I shoot many pics and delete 90% of them
I edit them in Adobe Lightroom 4
I push them on my openphoto.me account
they are stored in my Dropbox account

What I wanted to add is a way to sign my pictures by adding a small text on the bottom right corner. Not because I want to look like some kind of pro photographer -- I am not ;)

But signing is useful when you send pics around to your family & friends: I want them to know that they are more pics at http://tarek.openphoto.me.

I found a cool little script at http://www.turboradness.com/watermark-your-images-with-python that provides this feature with PIL and will let you sign pictures with a signature image.

signpic is a small script inspired from that blog post that adds:

a console script you can use to sign your pics or directories of pics using a provided signature image, or the signature image located at ~/.signature.jpg
a --powerhose mode that will run a Powerhose cluster to perform the operations. This is very useful to speed up the process when you need to sign a bunch of pictures - a single signing takes a few seconds.... Powerhose is a Request-Reply Broker pattern in ZMQ, and speed up the signing task.

If you want to give it a shot:

$ pip install signpic
$ bin/signpic ~/Desktop/pics --phose
Using signature file '/Users/tarek/.signature.jpg'
Looking for files in '/Users/tarek/Desktop/pics'
+++++++++++++.............Done.

GitHub based services

Tarek Ziade — Fri, 06 Apr 2012 10:20:00 +0200

Ok, maybe I am late at the party, and some people will think what I am writing here is obvious, but nevermind. ;)

There's a trend these days on Github-based online services. That is -- point me your Github repo and I'll do something with it everytime you push a change.

Notification hooks have existed for years now, and a lot of teams have their own Jenkins CI (we do at Mozilla), or their checkins mailing list. But the thing is : what's the point to maintain your own stuff anymore when there's a service for it you can hook in a few minutes.

For a Python project there are two absolutely awesome services:

ReadTheDocs -- creates a website out of your project Sphinx docs, exactly like what we've set with http://packages.python.org but it's all automatically done for you on every commit. win
Travis-CI -- run your tests on every change. Sends mail on failure. Again, nothing's new here, but as a developer, the only thing you have to do is to add a YAML file in your repo. win

Travis does not replace functional tests or tests you want to run on a specific infrastructure. But for day-to-day usage it's perfect.

In fact, there are more and more people that have standardized their README page that appears at PyPI with those two tools.

For example, look at Circus -- http://pypi.python.org/pypi/circus

It displays the Travis build status directly on PyPI, and provides a link to Read the docs.

Out of my head here are some services I'd love to enable from my Github admin panel:

A Coverage Dashboard like this one -- http://nedbatchelder.com/code/coverage/sample_html/
A similar Dashboard but with profiling info, using http://www.hexacosa.net/project/pyprof2html/
A Funkload Trending Dashboard. Read about this here: http://ziade.org/2011/06/10/continuous-load-testing-wint-funkload

The first and second ones are simple to produce - all it takes is a call to the script that generates the HTML report. It needs to run in some VM though.

Ah. mm wait. Maybe the ReadTheDocs project could add support to these, or maybe I can try to sneak in those dashboards as Sphinx extensions...

The last one is harder because it involves some thoughts on the load testing architecture and well, it won't tell you if your application is slow on your hardware -- it will just detect that your web service got suddenly slower after that commit.

There are hundreds of services on Github already -- curious to know which ones Python folk use.

Circus 0.1 released

Tarek Ziade — Tue, 20 Mar 2012 21:55:00 +0100

Note

Circus is a program that will let you run and watch multiple processes. Like Supervisord, BluePill and Daemontools.

We wanted to release a first version of Circus before Pycon originally. Then Pycon was here, so we decided we would release it before Pycon would end. And Pycon ended. And we had a few bugs to fix.

But today it's happening !

Circus 0.1 is an alpha release, don't use it in production yet !

But this first release contains a lot of stuff already:

a pub/sub channel so you can monitor what's going on.
circusctl : an amazing console script that let you interact with the system.
a full documentation for the circusctl CLI.
an API that's simple enough in most cases.
... many more things really. I had to stop Benoit from adding features ;) -- did I mention that Benoit is amazing ?

I also added a section on how Circus is different from other tools here: http://circus.readthedocs.org/en/latest/index.html#why-should-i-use-circus-instead-of-x , since a lot of people asked about it.

One very important feature for me is to be able to use Circus with a few lines of Python in a program that needs to spawn workers. The use case for me is to run Powerhose, a library that let you dispatch some tasks across several workers, no matter what language they're built with.

This is all it takes to run and maintain 4 workers:

from circus import get_arbiter

arbiter = get_arbiter("worker", 4)
try:
    arbiter.start()
finally:
    arbiter.stop()

Links:

the doc: http://circus.readthedocs.org/en/latest/index.html
the release: http://pypi.python.org/pypi/circus/0.1
the repo: https://github.com/mozilla-services/circus

Please let us know what you think !

Pycon - Packaging Sprint/Bof Report

Tarek Ziade — Thu, 15 Mar 2012 14:54:00 +0100

We did a packaging open space and a sprint during Pycon and the talks and work we did was quite constructive. The good thing is that most people that are involved in the maintenance of the packaging tools in our eco-system were present -- that helps a lot.

I don't want to go into great details here, mostly because I am currently in a train going back to Dijon, completely jet-lagged from the Pycon trip. But I do want to blog about it now to make sure I write down while it's fresh.

Note

Just a quick note on the terms I use to avoid confusion:

distutils is now frozen in Python
packaging is the new distutils, with backward incompatible changes
distutils2 is the backport of packaging for Python 2.5 to 2.7

On to the top of my head, here what we've decided to do and worked on. I am adding a [status] marker for the ones I know.

release a Python 2 Distutils2 alpha version for people to play with [done]
add oauth support in PyPI
make SSL the default way to register and upload projects on PyPI
intensive testing and debugging of Distutils2 [lots done here, but ongoing]
poke at using Distutils2 in zc.buildout [ongoing]
start a discussion on Distutils-SIG about the next binary format [done]
publish setup.cfg at PyPI so tools or humans can read it without downloading and/or executing some code.

That does not sound like a lot of achievements, but this Pycon was in my opinion the best Pycon ever to put all the people involved in packaging on the same page and agree on some plans for the future.

That did not include the people from the scientific community, like last year or the year before. I was told that's because they have an aversion for the Distutils codebase. Guess what - mee to :)

But I think we can find a mutual ground for several reasons:

what we're building in Python is based on standards we define in PEPs. So that's orthogonal to an implementation. For example Scientific build tools could definitely consume our metadata, or our setup.cfg (http://docs.python.org/dev/packaging/setupcfg.html).
the packaging package in the standard library has been revamped, uses clean, modern python. It has useful tools in it -- for example, if you need to browse PyPI or the installed packages, you have now reference implementations there, that will be used by pip and zc.buildout in the long term. Notice that they are backported in the standalone distutils2 package so they can already be used
the compilers part is still horrible, but we made it pluggable. If you build a build tool, distutils2 can be configured to use it. So you can hate that code, but you should try to be compatible with at least the definition of extensions in setup.cfg.

So, like last year and the year before, I'd like to say again that anyone from the scientific community interested in helping in the standards or anything else is more than welcome.

If you're curious, you can see the etherpad we used here during Pycon:

On a side-note, using a pad like this was extremely useful during the open space, since everyone could interact live in it.

Oh by the way, kudos to Eric Araujo for the work he's been doing in the last months in packaging.

Circus Sprint Report

Tarek Ziade — Wed, 14 Mar 2012 15:28:00 +0100

Benoit and I organized a sprint at Pycon for Circus, the process watcher we're building (see http://ziade.org/2012/02/24/circus-a-process-controller).

We had quite some interest on the topic, and a few contributors. I was myself naviguating between the Circus sprint and the Packaging one. On Circus we were most of the time discussing features and adding test coverage and documentation.

Unfortunately, a few bugs prevented us from releasing the first public version, but that should be out there in a few days.

You can read the doc while we're building it here: http://circus.readthedocs.org/en/latest/index.html

And have a look at the architecture here: http://circus.readthedocs.org/en/latest/architecture

Stuff we did during the sprint:

more test coverage
hooked the project in Travis - http://travis-ci.org/#!/mozilla-services/circus
more doc
flapping detection
added pub/sub channel to allow any client to watch over what's going on -- any event, like a process that restarts, is published in the pub/sub channel
started an pub/sub client example: a web monitoring page based on websockets, using Flask -- that app will be a subscriber of Circus. (not in the repo yet)

Stuff we need to finish / Features we're thinking about:

factor out the flapping detector as a pub/sub plugin.
stream the process stdout/stderr so clients can subscribe to them
refactor how the commands are plugged in Circus -- mainly to be able to produce a doc like this one : http://www.redis.io/commands
add an encryption layer for the zmq channels
discuss with Chris McDonough wrt Supervisord common bits.

And welcome to our new contributors, Neil Chintomby and Ori Livneh !

If you want to have a look, for us at https://github.com/mozilla-services/circus

PCard - Password Card

Tarek Ziade — Wed, 14 Mar 2012 12:00:00 +0100

If you go to http://passwordcard.org, there's a password card generator you can use to create a 8 x 30 card filled with random characters, like the one you see here.

The first line is composed of icons and the idea is that you memorize one icon per service you use, then get a pass phrase by reading the characters on that column, or going in diagonal.

Of course that does not work if you have hundreds of passwords. But if you have a dozen very important passwords, the great thing about this card is that it can be printed and fit in your wallet, so you don't depend on any computer or untrusted Internet access when you want a password.

I am not sure to understand why passwordcard.org has the key used to generate the card printed on it, since it can be easy for someone that gives a glimpse to memorize that short sequence (8 chars) and regenerate the card with it. He won't get your passwords but will be closer to them I guess. Of course when you use that card you are not supposed to put you finger on it and follow a line in public ;)

Anyways, I wanted to print out a card for myself, but wanted to run it on my own computer for more safety, and remove the key on the printed version -- I also wanted a command line tool to print out the card in a shell.

And you know what happens when a Python programmer tries to change a Java program that's quite small. He ends up coding it back in Python after staring at it a few minutes :o

Anyways, http://pcard.ziade.org is my own version, the code is at https://github.com/tarekziade/pcard . There's a command line tool and a bottle-based web generator.

Pycon WebDev Summit Report - Python Application Packages

Tarek Ziade — Sat, 10 Mar 2012 08:00:00 +0100

We had a web summit at Pycon last Thursday and a few discussions next on that Python application package idea. I had conversations myself with a bunch of different people, trying to understand where they were at and what were their goals in this.

If you want to know what it is all about, read Ian blog's here: http://blog.ianbicking.org/2012/02/29/python-application-package

He describes one possible solution to create Python application packages.

The final goal is to be able to make it easier for cloud providers to run our apps. I should mention that they don't really have an issue here, they already run Python apps (Mozilla too ;)). But they use their own standards -- so the idea is to see if a common standard is possible.

During the summit, and right after, I suggested we would work together on defining what we wanted to achieve and to start a draft of a standard in a PEP during the Pycon sprint.

It's interesting to see how saying the word PEP can scare some people out by the way. When related to packaging, that word induce to some people things like long process, consensus hard to achieve, fights on mailing list and so on. Some people want to solve their problems right now, which is awesome, but...

But if their ambition is to see their solution adopted by the community at large, they might fall into the trap we went in with setuptools a few years ago, which is to create and push a new de-facto standard on the top of an existing, official standard in a way that creates chaos in the packaging eco-system -- because of two competing standards. So while it's great to build tools, I think that kind of standard should be discussed together from the beginning to see where are the common grounds.

So I would like to step back a little bit and clarify a few things:

A Standard is not an implementation -- and this is in particular important when we're thinking about building something to describe a Python application standard -- so let's drop any bit of code for now imo.
Solving this problem is not specific to a Web application -- defining a Python application that can be launched is not specific to a web app. In fact, as Kenneth put it, we're talking about defining a service here.
A deployment format is not a standard -- I think some people were confusing formats and standards. We were talking about how Java WAR files were cool because you could just deploy that file and have the server run it, like the Python eggs a few years ago I guess.
This standard is not here to compete with other packaging system -- whathever we build should not be a way to "avoid having to deal with system libraries". It should be possible to deploy an application using virtualenv and a local directory, but again, this is orthogonal to having a standard. As a RPM packager I want to be able to deploy your service in my own fashion, and that standard should help me too.
A Python web app is also a Python package -- this is important to note because a package already has metadata information see PEP 345 - http://www.python.org/dev/peps/pep-0345. And in fact, many things I see in Ian proposal are already solved by that PEP.

Based on these remarks, I would like during the sprint to work in two directions:

thinking about extending PEP 345 to introduce fields specific to the notion of running a service and describe how it would look like in packaging/distutils2's setup.cfg
thinking about how to introduce this standard in the existing eco-system for people to use immediatly on existing projects.

So far my thinking is that we should provide a field called Application that points a script to run an application -- like what you would point with the script option in distutils or in distribute/setuptools console entry point.

I am not sure yet about the environment and the config files.

Next episode during the sprints !

Moving to Pelican

Tarek Ziade — Mon, 05 Mar 2012 20:44:00 +0100

So, in my current quest of reducing the ads around me, I realized my blog hosted at wordpress.com was full of them.

So I decided to move to Alexis' Pelican blog system : http://pelican.notmyidea.org

Pelican creates a static html blog using reStructuredText or Markdown files and that makes it easy enough to push blog entries at github and have them published automatically on a server.

Pelican can work by default with Disqus and has a default CSS that's simple and readable.

What I am losing from Wordpress is the whole Dashboard that I really liked, and in particular the stats. Knowing that one of my blog post made it to Reddit or hacker news was a good thing -- I could go there and answer questions or read the comments.

I almost got myself a pro account at wordpress because I did not want to maintain yet another stats tool on my server. But getting away from wordpress and getting back under my own domain was more tempting.

Pelican has an import feature for Wordpress, that failed to produce the proper reStructuredText for me, but worked quite well with Markdown.

I was also able to tweak the URLs so they would look exactly like the Wordpress ones, that is, URL with the year, month and day. I was also lucky enough about the slug part of the URL: the import generated the same slugs than Wordpress ! In other words a simple redirection of the domain does the trick to avoid losing any indexing.

I am paying 12 bucks to wordpress to get this redirect but that's worth it because people hitting tarekziade.wordpress.com should find back the result transparently on blog.ziade.org.

I also missed a way to create static pages, like what Wordpress has. So I added this feature to Pelican and will propose it for inclusion to Alexis.

Overall I really recommend Pelican. It's well crafted, easy to use and it's Python !

Thanks to Alexis and other contributors.

More privacy please

Tarek Ziadé — Wed, 29 Feb 2012 10:44:00 +0100

It became extremely annoying for me to get targeted ads, since I do most of my shopping online, even groceries. I am a great target for advertising operators :)

Google seems to be very transparent about letting you opt-out third party cookies as a logged-in user, there's a page for this here. You can also make sure your web history is disabled on this page.

[caption id="" align="alignnone" width="390" caption="(c) bhikku cc-by-nc 2.0"][][][/caption]

But I don't want to chase options on every online service I use, just to opt-out and protect my privacy. All these options should be off by default in my opinion, or at least you should be warned, no ? I don't know if it's still accurate but there's a law in Europe for this.

Anyways, this is not a new problem. But I realized that while I was shocked by how cookies could breach your privacy +10 years ago, I was not taking any action to protect myself. Let's fix this.

I am trying to fine-tune my browser and change my search habits to make sure I am leaking as less data as possible.

Here's a list of actions I have done, if you have better ideas please comment, I'll try to keep that list up-to-date.
- Use Firefox and disable third-party cookies. - Activate Do Not Track - there's now [a good list of ads companies][] that should honor it. - Install Cookie Monster to get more control on cookies. - Install the Ghostery pluginto detect and block ads scripts. It's pretty enlightening. - Install HTTPS Everywhere. (nothing really a counter-measure for data leaking but it's safer - use it!)

I have also decided to give a shot to duckduckgo.com instead of google.com for search.They don't track at all. You caninstall the ssl search bar add-on and also tweak your Awesome bar so it uses it too (I set it to https://duckduckgo.com/?q= and it seems to work).

It's been a week of Duck Duck Going, and I like it so far. It's refreshing to use a new search engine. I have not suffered from bad results so far -- nothing compared to what made me move from Altavista to Google a very long time ago :D

And oh... nooo.. what now.. Browser Fingerprint ??!

"bhikku cc-by-nc 2.0"

[]: http://www.flickr.com/photos/bhikku/1187679/sizes/z/in/photostream/

circus - a process controller

Tarek Ziadé — Fri, 24 Feb 2012 16:58:00 +0100

Benoit -- from Gunicorn fame -- and myself have started to work on Circus, a process controller.

I really like this small project for several reasons:
- it's the first time we're building a Mozilla Services library from the ground with a contributor that's not for the Mozilla community but rather from the Python community - this library is the last bit we're missing in our Python server stack to have a full control over it

Why a new lib ?

A process controller is mostly a script that manages processes, sends signals to them, re-spawns them whenever they die. There are numerous libraries out there that are already doing this so creating this new library can sound like NIH.

I have looked at several projects before we've started this one, like supervisord, BluePill, and so on. But none of them met exactly all of my requirements - here's my audacious list:
- can be used as a program and as a library -- so in Python - a way to query the system to perform live operations via a (secured) remote console such as
- add or remove workers - restart the workers, broadcast signals - get some stats on what's going on

advanced process management features, like
controlled flapping - trying to restart n times a worker that dies on startup -- then abandon
- auto-grow - add workers if all workers are 100% busy n seconds (for CPU-bound workers)
- auto-shrink - remove workers if they are bored
- run in containment tools like cgroups
meta-controller that can drive several controller remotely, to manage a cluster (later)

BluePill looked very promising because it has most features we wanted, but unfortunately, since we'd like to use it as a library, it's a blocker. Also, while the DSL is quite sexy, that's not something we'd want to use as-is because we're in a ini-file land were every tool is configured via a new [section] in a config file.

Supervisord is excellent -- and widely used.

I have tried to extend it and I must admit I had a little hard time to wrap my head into it. This is purely technical, but some choices made in Supervisord make it hard for me to extend it the way we want -- like the fact that the main class is driven by a configuration where I wanted to completely separate these two concepts. I want to be able to create a class and tell it to run n workers without having to create a configuration object in the middle. There's also now the subprocess module in Python, and while Supervisord is probably compatible with older versions of Python, we want 2.6+ so we can make the code way less verbose.

Here's an example on how to run 3 workers on a given command with Circus - KISS:
from circus import get_trainer

trainer = get_trainer(cmd, 3)

try:

    trainer.start()

finally:

    trainer.stop()

Anyways -- we'll still be suffering from our choice for a bit -- we'll encounter issues that other projects have encountered before. But I think that's for the best, and Benoit has a lot of experience in this area with Gunicorn - I expect both project to exchange a lot of pieces.

Current status

We're busy polishing the tool, but it's already in a usable state. For Mozilla, the main use case is to run Crypto Workers for Powerhose (read about Powerhose here) and we can already do this.

Turns out all Powerhose does is wrapping the get_trainer() call into a class called Workers. (see https://github.com/mozilla-services/powerhose/blob/master/powerhose/client/workers.py) and when the web application is launched, it runs a Powerhose Master and some PowerHose Workers that way -- delegating all the process management tasks to Circus.

For our Ops, Circus provide a small console that will let them watch the workers, add some, restart them, etc.

We've reached a point where we have almost all the features we wanted for our needs, but I suspect the project will gain many more features with the contributions of Benoit and maybe other folks in the future.

Nothing's released yet -- I'll wait for it to pass our benches, QA tests before I cut a release. But the code is growing here if you're curious : https://github.com/mozilla-services/circus.

Yeah, it's under-tested because I did not come up with a nice testing environment yet - it's hard to do this properly when you deal with processes and signals -- and mocking this is a bit of a non-sense. I suspect the best way will be to run functional tests with workers that produce some content the test can check out.

"The Flying Circus official food (cc)"
https://github.com/mozilla-services/powerhose/blob/master/powerhose/client/workers.py)

Defining a wsgi app deployment standard

Tarek Ziadé — Fri, 10 Feb 2012 00:14:00 +0100

Next month at Pycon, we'll have a web summit and I'm invited there to talk about how I deploy web applications. This is not a new topic, as it was already discussed a bit last year -- see Ian Bicking's thought on the topic.

My presentation at the summit will be in two parts. I want to 1/ explain how I organized our Python deployments at Mozilla (using RPMs) 2/ make an initial proposal for a deployment standard that would work for the community at large - I intend to work on this during Pycon and later on the dedicated SIG.

Here's an overview of the deployment standard idea...

How we deploy usually

If I want to roughly summarize how people deploy their web applications these days, from my knowledge I'd say that there are two main categories.
1. Deployments that need to be done in the context of an existing packaging system -- like RPM or DPKG 2. Deployments that are done in no particular context, where we want it to just work. -- like a directory containing a virtualenv and all the dependencies needed.

In both cases, preparing a deployment usually consists of fetching Python packages at PyPI and maybe compile some of them. These steps are usually done using tools like zc.buildout or virtualenv + pip, and in the case of Mozilla Services, a custom tool that transforms all dependencies into RPMs.

In one case we end up with a directory filled with everything needed to run the application, except the system dependencies, and in the other case with a collection of RPMs that can be deployed on the target system.

But in both cases, we end up using the same thing: a complete list of Python dependencies.

The trick with using tools like zc.buildout or pip is that from an initial list of dependencies, you end up pulling indirect dependencies. For instance, the Pyramid package will pull the Mako package and so on. A good practice is to have them listed in a single place and to pin each package to a specific version before releasing the app. Both pip and zc.buildout have tools to do this.

Deployments practices I have seen so far:
- a collection of rpms/debian packages/etc are built using tools like bdist_rpms etc. - a virtualenv-based directory is created in-place in production or as a pre-build binary release that's archived and copied in production - a zc-buildout-based directory is created in-place in production or as a pre-build binary release that's archived and copied in production

The part that's still fuzzy for everyone that is not using RPMs or Debian packages is how to list system-level dependencies. We introduced in PEP 345 the notion of hint where you can define system level dependencies which name may not be the actual name on the target system. So if you say you need libxml-dev, which is valid under Debian, people that deploy your system will know they'll need libxml-devel under Fedora. Yeah no magic here, it's a tough issue. see Requires-External.

The Standard

EDIT : Ian has a much more rich standard proposal here. (see the comments)

The standard I have in mind is a very lightweight standard that could be useful in all our deployment practices - it's a thin layer on the top of the WSGIstandard.

A wsgi application is a directory containing:
- a text file located in the directory at dependencies.txt, listing all dependencies - possibly reusing Pip's requirements format - a text file located in the directory at external-dependencies.txt, listing all system dependencies - possibly reusing PEP 345 format - a Python script located it the directory at bin/wsgiapp with an "application" variable. The shebang line of the Python script might also point to a local Python interpreter (a virtualenv version)

From there we have all kind of possible scenarios where the application can be built and/or run with the usual set of tools

Here's one example of a deployment from scratch :
- The repository of the project is cloned - A virtualenv is created in the repository clone - pip, which gets installed with virtualenv, is used to install all dependencies describes in dependencies.txt - gunicorn is used to run the app locally using "cd bin; gunicorn wsgiapp:application" - the directory is zipped and sent in production - the directory is unzipped - virtualenv is run again in the directory - the app is hooked to Apache+mod_wsgi

Another scenario I'd use in our RPM environment:
- The repository of the project is cloned - a RPM is built for each package in dependencies.txt - if possible, external-dependencies.txt is used to feed a spec file. - the app is deployed using the RPM collection

That's the idea, roughly -- a light standard to point a wsgi app and a list of dependencies.

Scaling Crypto work in Python

Tarek Ziadé — Mon, 06 Feb 2012 09:03:00 +0100

We're building a new service at Services called the Token Server - The idea is simple : give us a Browser ID assertion and a service name, and the Token Server will send you back a token that's good for 30 minutes to use for the specific service.

That indirection makes our live easier to manage user authentication and resource allocation for our services . A few examples:
- when a new user wants to use Firefox Sync, we can check which server has the smallest number of allocated users, and tell the user to go there - we can manage a user from a central place - we can manage a user we've never heard about before without asking her to register specifically to each service -- that's the whole point of Browser ID

I won't get into more details because that's not the intent of this blog post. But if you are curious the full draft spec is here -https://wiki.mozilla.org/Services/Sagrada/TokenServer

What's this post is really about is how to build this token server.

The server is a single web service that gets a Browser ID assertion and does the following:
1. verify the assertion 2. create a token, which is a simple JSON mapping 3. encrypt and sign the token

The GIL, Gevent, greenlet and the likes

Implementing this using Cornice and a crypto lib is quite simple, but has one major issue : the crypto work is CPU intensive, and even if the libraries we can use have C code under the hood, it seems that the GIL is not released enough to let your threads really use several cores. For example, we benched M2Crypto and it was obvious that a multi-threaded app was locked by the GIL.

But we don't use threads in our Python servers -- we use Gevent workers, which are based on greenlets. But while greenlets help on I/O bound calls, it won't help on CPU bound work : you're tied into a single thread in this case and each greenlet that does some CPU work blocks the other ones.

It's easy to demonstrate -- see http://tarek.pastebin.mozilla.org/1476644 If I run it on my Mac Book Air, the pure Python synchronous version is always faster (huh, the gevent version is *much* slower, not sure why..)

So the sanest option is to use separate processes and set up a messaging queue between the web service that needs some crypto work to be done and specialized crypto workers.

We're back in that case to our beloved 100% I/O bound model we know how to scale using NGinx + GUnicorn + GEvent

For the crypto workers, we want it to be as fast as possible, so we started to look at Crypto++ which seems promising because it uses CPU-specific calls in ASM. There's the pycryptopp binding that's available to work with Crypto++ but we happen to need to do some tasks that are not available in that lib yet -- like HKDF.

Yeah, at that point it became obvious we'd use pure C++ for that part, and drive it from Python.

Message passing

Back to our Token server -- we need to send crypto work to our workers and get back the result. The first option that comes in mind is to use multiprocessing to spawn our C++ workers and to feed them with work.

The model is quite simple, but now that we have one piece in C++, it's getting harder to use the built-in tools in multiprocessing to communicate with our workers -- we need to be lower level and start to work with signals or sockets. And well, I am not sure what would be left of multiprocessing then.

This is doable but a bit of a pain to do correctly (and in a portable way.) Moreover, if we want to have a robust system, we need to have things like a hearbeat, which requires more inter-process message passing. And now I need to code it in Python and C++

Hold on -- Let me summarize my requirements:
- inter-process communication - something less painful than signals or sockets - very very very fast

I got tempted by Memory Mapped Files, but the drawbacks I've read here and there scared me.

ZeroMQ

It turns out zeromq is perfect for this job - there are clients in Python and C++, and defining a protocol to exchange data from the Python web server to the crypto workers is quite simple.

In fact, this can be done as a reusable library that takes care of passing messages to workers and getting back results. It has been done hundreds of times, there are many examples in the zmq website, but I have failed to find any Python packaged library that would let me push some work to workers transparently, via a simple execute() call -- if you know one tell me!.

So I am building one since it's quite short and simple -- The project is called PowerHose and is located here :https://github.com/mozilla-services/powerhose.

Here is its descriptions/limitations:
- Powerhose is based on a single master and multiple workers protocol - The Master opens a socket and waits for workers to register themselves into it - The worker registers itself to the master, provides the path to its own socket, and wait for some work on it. - Workers are performing the work synchronously and send back the result immediatly. - The master load-balances on available workers, and if all are busy waits a bit before it times out. - The worker pings the master on a regular basis and exits if it's unable to reach it. It attempts several time to reconnect to give a chance to the master to come back. - Workers are language agnostic and a master could run heterogeneous workers (one in C, one in Python etc..) - Powerhose is not serializing/deserializing the data - it sends plain strings. This is the responsibility of the program that uses it. - Powerhose is not responsible to respawn a master or a worker that dies. I plan to use daemontools for this, and maybe provide a script that runs all workers at once. - Powerhose do not queue works and just rely on zeromq sockets.

The library implements this protocol and gives two tools to use it:
- A JobRunner class in Python, you can use to send some work to be done - A Worker class in Python and C++, you can use as a base class to implement workers

Here's an example of using Powerhose:
- The Server - [https://github.com/mozilla-services/powerhose/blob/master/examples/square_master.py][] - The Python worker - [https://github.com/mozilla-services/powerhose/blob/master/examples/square_worker.py][] - The C++ worker (don't look at the code :) - [https://github.com/mozilla-services/powerhose/blob/master/examples/square_worker.cpp][]

For the Token server, we'll have:
- A JobRunner in our Cornice application - A C++ worker that uses Crypto++

The first benches look fantastic -- probably faster that anything I'd have implemented myself using plain sockets :)

I'll try to package Powerhose so other projects at Mozilla can use it. I am wondering if this could be useful to more people, since I failed to find that kind of tool. How do you scale your CPU-bound web apps ?

https://github.com/mozilla-services/powerhose/blob/master/examples/square_master.py
https://github.com/mozilla-services/powerhose/blob/master/examples/square_worker.py
https://github.com/mozilla-services/powerhose/blob/master/examples/square_worker.cpp

Mozillians, win a free pass for Pycon US - take 2

Tarek Ziadé — Wed, 04 Jan 2012 23:41:00 +0100

I am extending the contest until Feb the 1st - Mozillians, win a free pass for Pycon US

The fear of CRUD

Tarek Ziadé — Tue, 03 Jan 2012 00:17:00 +0100

Cornice is growing steadily, and we are thinking about the different ways to use it for our needs. One use case that comes often when we build web services is the need to publish a SQL Database via HTTP.

For instance, in a project I am working on, we might expose a list of servers and some information about them, that are stored in a SQL DB . The goal is to allow some management scripts to interact with the DB, to set and retrieve information about the servers, like: "can I use server 12 as a node for application X ?"

Interacting with CURL or a similar tool is simpler and more portable than coding yet another SQL client for this, so the idea is to see how this kind of web service can be done is the minimum pain with Cornice.

What I am thinking about building is a small CRUD interface that glues Cornice and SQLAlchemy. The latter has a way to define a database schema explicitly via mappings meaning that it's easy to write a generic layer that exposes the database to the web via Cornice definitions. The work consists of transforming POST & PUT requests that contains data to write to the DB into SQLAlchemy objects, and transforming select results asked via GET requests into the proper responses.

Nothing very new, there are tons of existing systems that implement CRUD on the top of ORMs or plain SQL libraries. The only reason to build yet another one is to use it in the context of our current toolset which is composed of Cornice, Pyramid & SQLAlchemy for most projects. The whole code will probably be less than 300 lines at the end anyways.

*Oh my.*

Turns out this idea is really freaking out some people around me. There's a strong aversion of some coders against anything that looks a bit like Active Records -- in the Rails Context. In other words anything that would completely automate the serialization & deserialization layer and make it hard to tweak some code.

Another criticism is that a CRUD system would not be able to scale in the context of a big database, like Firefox Sync, that uses numerous databases to shard data.

Turns out building a CRUD on tools like SQLAlchemy or Pyramid is not really going to ruin your scalability as long as:
- you can tweak the serialization / deserialization - you can override any operation in the CRUD operations when needed - you don't shoot yourself in the foot by using CRUD with some code or DB that is not meant to be used that way - you can use the power of the underlying tools without being blocked by the lib

For the latter, Ben Bangert was pointing me at SQLAlchemy horizontal feature, which is basically what I wrote from scratch last year to make the Sync server shard across databases... At this point I sense that Firefox Sync could have been built with a CRUD lib, and be as efficient as it is today, because when I look at the queries produced by the code and the one a CRUD lib would produce, we are one or two tweaks away.

Anyways, here's a first attempt at such a library.

Defining the model

In SQLAlchemy, you can define the DB model using mappings, which are simple classes containing a description of the tables.

For example, if I have a class "users" with a field "id" and a field "name", the mapping will look like this:
class Users(_Base):

    __tablename__ = 'users'

    id = Column(Integer, primary_key=True)

    name = Column(String(256), nullable=False)

What I started to do is write a meta class one can use in a class to publish the mapping via HTTP.

Here's an example:
from cornicesqla import MetaDBView

from myapp import Users, DBSession

class UsersView(object):

    __metaclass__ = MetaDBView

    mapping = Users

    path = '/users/{id}'

    collection_path = '/users'

    session = DBSession

What we have here is the definition of a view for the Users mapping. The class defines an URI for the collection (collection_path) and for each user (path). The session attribute is an SQLAlchemy session object you usually define when you work with that tool.

That's it.

The model gets published, and you can GET, PUT, POST and DELETE on /users and /users/someid.

The code of the prototype is here and you can find a working example in the tests here. It's called cornice-sqla

Tweaking serialization & data validation

By default, cornice-sqla will serialize and deserialize using JSON but you can tweak these steps by providing a custom serializer, or deserializer (or both.)

Let's say you want to use the Colander libary to validate and serialize the data. To do this, you just have to write your serializer method into the view class
class UsersValidation(colander.MappingSchema):

    name = colander.SchemaNode(colander.String())

class UsersView(object):

    __metaclass__ = MetaDBView

    mapping = Users

    path = '/users/{id}'

    collection_path = '/users'

    session = DBSession

    def serialize(self):

        """Unserialize the data from the request, to serialize it for the DB"""

        try:

            user = json.loads(self.request.body)

        except ValueError:

            request.errors.add('body', 'item', 'Bad Json data!')

            # let's quit

            return

        schema = UsersValidation()

        try:

            deserialized = schema.deserialize(user)

        except Invalid, e:

            # the struct is invalid

            request.errors.add('body', 'item', e.message)

        return deserialized

Colander is used here to validate the incoming request and create a flat mapping we can push into the DB. Cornice's error system is in usage here, as explained here.

You can tweak the data that gets back from the DB with unserialize(), and for the collection URI, use collection_serialize() and collection_unserialize().

Tweaking C, R, U or D

cornice-sqla is based on a fresh feature Gael added into Cornice lately: resources. A resource is a class where you can define get(), post(), delete() and put() methods for a given URI.

cornice-sqla views are based on resources, meaning that you can override anyone of those methods and do whatever you want if you don't want the CRUD part.

What's next

I need to make sure everything you can do in Cornice (acls various options etc) can still be done in cornice-sqla, and start to work with more complex DB schema that include relations etc. I also need to add basic missing features like batching and some docs.

My hope at the end is that this small library will reduce considerably the code needed in some of our projects that interact with SQL.

Pyramid @ Python 3

Tarek Ziadé — Sun, 25 Dec 2011 11:15:00 +0100

If you have been following closely the latest work done by Chris on WebOb, you know that WebOb and eventually Pyramid became Python 3 compatible.

That makes Python 3 a very tempting target for a new web project.

Paste & PasteScript still need to be ported to Python 3 and the Pyramid team has chosen not to. They have created their own paster replacer instead, which can be used to initiate a Pyramid project or run the app using the .ini file.

I am wondering if it would not be simpler at this point to drop Paste and use this replacer for all Python 3 frameworks that are using the Paste script and templates features.

Besides all the features Pyramid and its libs turns out most of the libs you usually need to build a classical web app already support Python 3, like SQLALchemy and PyMysql for MySQL access, Pylibmc for Memcached;

Things I am still missing in Python 3:
- gevent - gunicorn - python-ldap - Cornice -- I will port it soon

If you want to give it a shot, get the latest Python 3.2 and grab more details at :https://github.com/Pylons/pyramid/wiki/Python-3-Porting

And if you miss one lib, add it here

Merry Christmas !

Tutorial - build your web services with Cornice

Tarek Ziadé — Wed, 21 Dec 2011 11:50:00 +0100

At this stage, I think we've added enough helpers in Cornice to get anyone started in building web services in Python.

As a reminder, Cornice provides helpers to build & document REST-ish Web Services with Pyramid, a Python web framework. The main benefits of Cornice are:
- automatic handling of some HTTP errors - Ask yourself: is your app handling properly 405 or 406 errors? - automatic web service documentation via a Sphinx extension. - a simple way to validate and convert requests data, and return structured 400 responses.

This is a small tutorial, extracted from our documentation.

Let’s create a full working application with Cornice. We want to create a light messaging service.

You can find its whole source code at https://github.com/mozilla-services/cornice/blob/master/examples/messaging

Features:
- users can register to the service - users can list all registered users - users can send messages - users can retrieve the latest messages - messages have three fields: sender, content, color (red or black) - adding a message is done through authentication

Limitations:
- there’s a single channel for all messages. - if a user with the same name is already registered, he cannot register. - all messages and users are kept in memory.

Design

The application provides two services:
- users, at /users: where you can list all users or register a new one - messages, at /: where you can read the messages or add new ones

On the server, the data is kept in memory.

We’ll provide a single CLI client in Python, using Curses.

Setting up the development environment

To create this application, we’ll use Python 2.7. Make sure you have it on your system, then install virtualenv (see http://pypi.python.org/pypi/virtualenv.)

Create a new directory and a virtualenv in it:

$ mkdir messaging

$ cd messaging

$ virtualenv --no-site packages .

Once you have it, install Cornice in it with Pip:

$ bin/pip install Cornice

Cornice provides a Paster Template you can use to create a new application:

$ bin/paster create -t cornice messaging

Selected and implied templates:

cornice#cornice  A Cornice application

Variables:

egg:      messaging

package:  messaging

project:  messaging

Enter appname (Application name) ['']: Messaging

Enter description (One-line description of the project) ['']: A simple messaging service.

Enter author (Author name) ['']: Tarek

Creating template cornice

...

Generating Application...

Running python2.7 setup.py egg_info

Once your application is generated, go there and call develop against it:

$ cd messaging

$ ../bin/python setup.py develop

...

The application can now be launched via Paster, it provides a default “Hello” service, you can check:

$ ../bin/paster serve messaging.ini

Starting server in PID 7618.

serving on 0.0.0.0:5000 view at http://127.0.0.1:5000

Once the application is running, visit http://127.0.0.1:5000 in your browser or Curl and make sure you get:

{'Hello': 'World'}

Defining the services

Let’s open the file in messaging/views.py, it contains all the Services:

from cornice import Service

hello = Service(name='hello', path='/', description="Simplest app")

@hello.get()

def get_info(request):

    """Returns Hello in JSON."""

    return {'Hello': 'World'}

Users managment

We’re going to get rid of the Hello service, and change this file in order to add our first service - the users managment

_USERS = {}

users = Service(name='users', path='/users', description="Users"0

@users.get(validator=valid_token)

def get_users(request):

    """Returns a list of all users."""

    return {'users': _USERS.keys()}

@users.put(validator=unique)

def create_user(request):

    """Adds a new user."""

    user = request.validated['user']

    _USERS[user['name']] = user['token']

    return {'token': '%s-%s' % (user['name'], user['token'])}

@users.delete(validator=valid_token)

def del_user(request):

    """Removes the user."""

    user = request.validated['user']

    del _USERS[user['name']]

    return {'goodbye': user['name']}

What we have here is 3 methods on /users:
- GET: simply return the list of users names – the keys of _USERS - PUT: adds a new user and returns a unique token - DELETE: removes the user.

Remarks:
- PUT uses the unique validator to make sure that the user name is not already taken. That validator is also in charge of generating a unique token associated with the user. - GET users the valid_token to verify that a X-Messaging-Token header is provided in the request, with a valid token. That also identifies the user. - DELETE also identifies the user then removes it.

Validators are filling the request.validated mapping, the service can then use.

Here’s their code:

import os

import binascii

from webob import exc

def _create_token():

    return binascii.b2a_hex(os.urandom(20))

def valid_token(request):

    header = 'X-Messaging-Token'

    token = request.headers.get(header)

    if token is None:

        raise exc.HTTPUnauthorized()

    token = token.split('-')

    if len(token) != 2:

        raise exc.HTTPUnauthorized()

    user, token = token

    valid = user in _USERS and _USERS[user] == token

    if not valid:

        raise exc.HTTPUnauthorized()

    request.validated['user'] = user

def unique(request):

    name = request.body

    if name in _USERS:

        request.errors.add('url', 'name', 'This user exists!')

    else:

        user = {'name': name, 'token': _create_token()}

        request.validated['user'] = user

When the validator finds errors, it adds them to the request.errors mapping, and that will return a 400 with the errors.

Let’s try our application so far with CURL:

$ curl http://localhost:5000/users

{"status": "error", "errors": [{"location": "header",

                                "name": "X-Messaging-Token",

                                "description": "No token"}]}

$ curl -X PUT http://localhost:5000/users -d 'tarek'

{"token": "tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"}

$ curl http://localhost:5000/users -H "X-Messaging-Token:tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"

{'users': ['tarek']}

$ curl -X DELETE http://localhost:5000/users -H "X-Messaging-Token:tarek-a15fa2ea620aac8aad3e1b97a64200ed77dc7524"

{'Goodbye': 'tarek}

Messages managment

Now that we have users, let’s post and get messages. This is done via two very simple functions we’re adding in the views.py file:

messages = Service(name='messages', path='/', description="Messages")

_MESSAGES = []

@messages.get()

def get_messages(request):

    """Returns the 5 latest messages"""

    return _MESSAGES[:5]

@messages.post(validator=(valid_token, valid_message))

def post_message(request):

    """Adds a message"""

    _MESSAGES.insert(0, request.validated['message'])

    return {'status': 'added'}

The first one simply returns the five first messages in a list, and the second one inserts a new message in the beginning of the list.

The POST uses two validators:
- valid_token(): the function we used previously that makes sure the user is registered - valid_message(): a function that looks at the message provided in the POST body, and puts it in the validated dict.

Here’s the valid_message() function:

def valid_message(request):

    try:

        message = json.loads(request.body)

    except ValueError:

        request.errors.add('body', 'message', 'Not valid JSON')

        return

    # make sure we have the fields we want

    if 'text' not in message:

        request.errors.add('body', 'text', 'Missing text')

        return

    if 'color' in message and message['color'] not in ('red', 'black'):

        request.errors.add('body', 'color', 'only red and black supported')

    elif 'color' not in message:

        message['color'] = 'black'

    message['user'] = request.validated['user']

    request.validated['message'] = message

This function extracts the json body, then checks that it contains a text key at least. It adds a color or use the one that was provided, and reuse the user name provided by the previous validator with the token control.

Generating the documentation

Now that we have a nifty web application, let’s add some doc.

Go back to the root of your project and install Sphinx:

$ bin/pip install Sphinx

Then create a Sphinx structure with sphinx-quickstart:

$ mkdir docs

$ sphinx-quickstart

Welcome to the Sphinx 1.0.7 quickstart utility.

..

Enter the root path for documentation.

> Root path for the documentation [.]: docs

...

> Separate source and build directories (y/N) [n]: y

...

> Project name: Messaging

> Author name(s): Tarek

...

> Project version: 1.0

...

> Create Makefile? (Y/n) [y]:

> Create Windows command file? (Y/n) [y]:

Once the initial structure is created, we need to declare the Cornice extension, by editing the source/conf.py file. We want to change extensions = [] into:

import cornice

extensions = ['cornice.sphinxext']

The last step is to document your services by editing the source/index.rst file like this:

Welcome to Messaging's documentation!

=====================================

.. services::

   :package: messaging

The services directive is told to look at the services in the messaging package. When the documentation is built, you will get a nice output of all the services we’ve described earlier.

The Client

A simple client to use against our service can do three things:
1. let the user register a name 2. poll for the latest messages 3. let the user send a message !

Without going into great details, there’s a Python CLI against messaging that uses Curses.

See https://github.com/mozilla-services/cornice/blob/master/examples/messaging/messaging/client.py

Going deeper

If you want to dig deeper, here are a few links:
- Documentation - http://packages.python.org/cornice - PyPI - http://pypi.python.org/pypi/cornice - Repository - https://github.com/mozilla-services/cornice

We'd love feedback & new contributors !

https://github.com/mozilla-services/cornice/blob/master/examples/messaging
https://github.com/mozilla-services/cornice/blob/master/examples/messaging/messaging/client.py

New Year's Python Meme 2011

Tarek Ziadé — Tue, 20 Dec 2011 11:28:00 +0100

Hey I did this in 2009, let's try again -- I am adding one extra question this year

1. What’s the coolest Python application, framework or library you have discovered in 2011 ?

GEvent & Pyramid. Not discoveries, but a daily usage. GEvent was for me a fantastic way to make the Firefox Sync Python server scale without being forced to write callback-style code. Pyramid is a very elegant framework, that takes the simplicity from Pylons and the power and experience from Repoze & the Zope world. A good sign for me is that we don't have to deal with the ZCA ;)

2. What new programming technique did you learn in 2011 ?

Better behaviour in high loaded server apps. During the last year, when we wrote all the pieces that makes Firefox Sync today, I've learned how to be more careful on how my apps would react when a back-end breaks or cease to reply, when a database gets slow, or when some service that's used restarts -- or when my own app restarts, still hammered by many requests. I did a fair amount of work on this, like smart pools of connectors and better testing techniques, and make decisions on what features survive when some third-party server is down, and what features just go 503.

3. What’s the name of the open source project you contributed the most in 2011 ? What did you do ?

Mozilla. I have not contributed as much as last year in Python because my work at Mozilla takes most of my time, but the good news is that all our stuff is open source so.. The most useful stuff for the community we've started is probably Cornice. But we've written and writing a myriad of apps and libs. See https://github.com/mozilla-services and http://hg.mozilla.org/services

In Python I still interact a bit with what's going on in Packaging and hope I'll be able to spend more time on it in 2012. But some packaging work I needed at work was also useful for the community, like pypi2rpm.

4. What was the Python blog or website you read the most in 2011 ?

Like in the past few years, Python Reddit. And I think I am not alone in that case. 90% of my blog hits come from Reddit

5. What are the three top things you want to learn in 2012 ?

I'd like to learn how to program in a few new languages, just to give them a shot. Maybe Haskell. I'd also like to finish a spare project I have started with Benoit, and try to launch it, promote & market it. Last, I'd like to learn more about Firefox arcanes -- just for my culture.

6. What are the top software, app or lib you wish someone would write in 2012 ?
- I want to take a picture of a wine bottle and have it recognized in an online app, where I can share my thoughts about its taste. - I want an Android virtual ping-pong application, where you can use your phone as paddle and see the e-ball through the camera & play with a friend.

Want to do your own list ? here's how:
- copy-paste the questions and answer to them in your blog - tweet it with the [#2012pythonmeme][] hashtag

Mozilla Apps -- server side

Tarek Ziadé — Wed, 14 Dec 2011 14:43:00 +0100

Yesterday, we've launched the Developer Preview for Apps, you can play with at https://apps-preview.mozilla.org

The server side is composed of many pieces, and while they are subject to change since this is just a preview, I think it's quite interesting to describe some of them already -- and maybe get more contributors in the process, since everything is open source and contributors are welcome.

Here's an overview of the system -- we used:
- Django for the App MarketPlace - Cornice for the Sync APIs - Node.js for the Sauropod APIs - HBase for the Sauropod DB

The App MarketPlace - Django

The App MarketPlace located at https://apps-preview.mozilla.org is where you can upload your own Apps, or install some. There's a payment process for non-free Apps, and you can see which one you have bought in your profile.

The development is driven by the WebDev team, and is based on Django -- see https://github.com/mozilla/zamboni/tree/master/apps/webapps

I can't really describe this part, as I did not follow it closely. But basically, the Market Place keeps track of your apps and payment receipts, for other parts of the system to interact with.

The Dashboard - HTML + JS

Once you start to install Open Web Applications, you are redirected to a Dashboard at https://myapps.mozillalabs.com. This Dashboard is an HTML application that lists your installed Apps associated to your Browser ID.

What's pretty cool is that no matter where you've installed a given App -- Firefox on your Desktop, your Phone, it will appear on this dashboard, and synced across devices.This is done via a Sync service called AppSync.

Code pointers for the Dashboard:
- https://github.com/mozilla/openwebapps/tree/develop/site - https://github.com/mozilla/openwebapps/blob/develop/addons/jetpack/lib/sync.js

AppSync

AppSync is the part I worked on. Its design was mainly done by Ian Bicking, who then worked on the client side implementation while I was working on the server side one.

It's quite similar to what we did for Firefox Sync, except that:
- AppSync supports BrowserID - The data is stored in Sauropod

Securing this data is part of a much larger ongoing project called Sauropod. The idea is that any database access has to be done with credentials, and that Sauropod is in charge of controlling them and dealing with the storage.

In the long term, depending how Sauropod evolves with respect to encryption, and how Firefox Sync itself evolves with respect to Browser ID and Sauropod access, we might merge both projects.

Or maybe Sauropod will provide APIs one day that are good enough for direct clients interactions, turning AppSync into a simple proxy ?

Time will tell !

Anyway, here's an overview below of the AppSync system we've set up for this preview.

We have the AppSync server itself, that's built using Cornice. It provides the synchronisation APIs described in this document: https://wiki.mozilla.org/Apps/Sync/Spec

Every time a client wants to write new data, we call the Sauropod server which is a very simple GET/SET api built with Node.js.

The flow is:
- AppSync server asks for a new Sauropod session, using Browser ID credentials - Sauropod verifies the Browser ID credentials then create a DB token into a session - AppSync uses this token until its not valid anymore - Sauropod calls in turn an HBase cluster to access the data - Every write is mirrored in a MySQL database in AppSync. This mirroring was set so we can turn off Sauropod if we need to, and still have a working system. This will eventually go away later.

Find the code here:
- Server https://github.com/mozilla/appsync - Client https://github.com/mozilla/openwebapps/tree/develop/sync

What's Next

I am really excited by this project. There are a lot of people involved in the Mozilla community, and seeing all the moving pieces assembled to build an Open App environement is pretty cool.

We're going to work in the upcoming months on consolidating the whole system, making sure it scales well and correct the design as the feedback comes back.

If you want to get involved, you can install the preview, play with the available Apps and even maybe write your own Apps for the Market Place, or help us in the coding.

We're hanging in #openwebapps on Mozilla's IRC

EDIT : Anant wrote a very nice blog post on the topic

https://github.com/mozilla/openwebapps/blob/develop/addons/jetpack/lib/sync.js

QA script on web services

Tarek Ziadé — Mon, 12 Dec 2011 11:00:00 +0100

The other task Alexis and I are going to work on this week, besides Cornice, is a QA script for web services.

The goal is simple : check that a set of web services are HTTP compliant. For example, does your application send the proper 406 error when an unsupported Accept is asked by the client ?

If you document properly your web services, asking for an unsupported Accept should not occur of course, but in most projects those protocol details are often a bit vague. And someone that writes a client software will inevitably make some assumptions based on his HTTP knowledge on how the application is supposed to behave.

Richard Newman came up with a fair list of tests we could run against a web app already, and we've started to summarise and add more of them here: https://wiki.mozilla.org/Services/WALint

The idea of the script is to print out a report of errors and warnings it found on a web app, exactly like a lint tool would do on some code. That's what I called it WALint (Web App Lint). Alexis doesn't like the name but he did not find a better name yet ;)

The way it works is that you describe in a configuration file the URIs of your web services, then WALint runs tests against them, using what we called controllers.

Each controller is in charge of trying out something on the web app, using a small HTTP test client (using WebTest), given a path and a method. WALint will provide built-in controllers and will be extensible. We will have Mozilla-specific tests, like the maximum size of a query string, or the maximum size of the request, since those limits are specific to the used stack.

We got bitten by this is the past in Sync - one web service failed to work properly because the client was building a super long query string, that was truncated along the way in our stack.

Our final goal with this tool is to be able to add in Jenkins these controls for all our web apps, and catch more problems before they occur in production.

Since it's also useful while you build your code, WALint will have a UnitTest integration, so you can run it as a functional test from within your test suite -- In that case, it will run directly against the code.

As usual, feedback & contributions are welcome. The code is being built here: https://github.com/mozilla-services/walint

Cornice - Validators and 406s brought to you by Alexis

Tarek Ziadé — Wed, 07 Dec 2011 19:09:00 +0100

Alexis started to work with me this week from my house and we're having good times. More geeks from Afpy are joining us this week-end for wine-drinking and working on a Python technical writing project.

While working myself on AppSync with the fine folks from Mozilla Labs, I am mentoring Alexis to get him up to speed on our projects & standards at Mozilla Services.[][]

Cornice, is a perfect match for him to start working -- it's isolated enough so he can have fun and produce features that are immediately useful, yet learn our standards & processes.

So we worked on its design with the help of its principal user, Ben Bangert, and I asked Alexis to blog about the work & doc we produced since Monday.

Read up => Introducing Cornice

"alexis"

[]: http://tarekziade.files.wordpress.com/2011/12/alexis1.jpg

Syntastic now with Flake8

Tarek Ziadé — Sun, 04 Dec 2011 22:36:00 +0100

If you're a VIM user and a Python coder, that could interest you.

Syntastic, a syntax checker for VIM now includes Flake8 as a checker for Python code.

Syntastic can be used to check for syntax errors in files opened in VIM. It has a pretty extensive list of supported languages.

For Python it had PyFlakes, and since the latest version, it has my small glue script "Flake8", that reunites in a single stream: pep8, PyFlakes and a McCabe checker.

For Flake8 the next step is to support Python 3 -- as soon as I find some time, or a volunteer ;)

Thanks to Sylvain Soliman (and Clayton Parker) !

Cornice -- web services with Pyramid

Tarek Ziadé — Tue, 29 Nov 2011 11:18:00 +0100

Since we've initially started Cornice at Services, we had more discussion about how we could make it easier for developers to validate an incoming request.

Our goal is :
- to be able to validate a request and if needed, to convert it to specific data structures - to complete the documentation of our web services in Sphinx with those validation steps

Here's a concrete example: A PUT request used to create a user in a database should come as a JSON object in the request body, which content would be validated and turned into a Person object, then sent to a SQL backend.

The web service needs in that case to reject any malformed request with a 400, and we'd also want to document this constraint in the web service documentation.

There are some existing tools in the validation arena, like FormEncode or Colander.

FormEncode and Colander both provide data validation via schemas, and FormEncode offers HTMLform generation.

When you need to validate an incoming request in your web service, those tools can fit your needs or simply be overkill. So, we wanted to integrate a validation step in Cornice without forcing the usage of a specific validation tool.

A simple callable

What we did is added a validator option that needs to point to a callable. The callable receives the request object and has to return an HTTP error code followed by a reason in case the request does not comply.

Here's an example, a GET /foo that will return a 402 if the X-Paid header is missing :
from cornice import Service

    foo = Service(name='foo', path='/foo')

    def has_paid(request):

        """The request must have an X-Paid header containing a token.

        This header proves the user has paid

        """

        if not 'X-Paid' in request.headers:

            return 402, 'You need to pay !'

    @foo.get(validator=has_paid)

    def get_value(request):

        """Returns the value.

        """

        return 'Hello'

From there, Cornice will call has_paid prior to get_value. Cornice is also able to build the documentation of the web service, by merging the docstrings of get_value and has_paid.

Sphinx integration

We provide a Sphinx extension for documenting a Cornice based project. You can inject in your Sphinx document the web service description, via a service directive.
.. services::

   :package: demoapp

In this example, the directive looks for all Cornice definitions in the demoapp package by scanning it, and injects their documentation in the Sphinx document.

The service directive provides a few options. For instance the name option will let you inject the documentation of one specific web service.

Colander integration

I said earlier simple callables where good enough for simple validation cases. Let's take a more complex example.

Let's say, you have a Person schema in Colander, that defines a person's attributes -- See Colander docs for details.

And you want to provide a PUT Web Service to create a person, where the request body is the person data serialized in JSON.

Here's the full Cornice definition
from cornice import Service

   from cornice.schemas import save_converted, get_converted

   def check_person(request):

      """The request body must be a JSON object describing the Person"""

      try:

          person = json.loads(request)

      except ValueError:

          return 400, 'Bad Json data!'

      schema = Person()

      try:

          deserialized = schema.deserialized(person)

      except InvalidError, e:

           # the struct is invalid

           return 400, e.message

      save_converted(request, 'person', deserialized)

    service = Service(name='person', path='/person/{id}')

    @service.put(validator=check_person)

    def data_posted(request):

        person = get_converted(request, 'person')

        ... do the work on person ...

In this example, the validator checks that the request body is valid Json, then pass the unserialized mapping to Colander to check that it's a valid Person record. Last, it uses the save_converted function we provide, to save the Person object into the request.

The web service then can pick it up with the get_converted function.

Next Steps

Our next steps is to build a library of useful built-in validators. Things like:
- is the request body is JSON ? - do we have param X, if yes is it an integer ? - any reusable validator we'll think about

The final goals here are :
1. have our web services code written in two phases: the validation phase, and the code itself. Because they are a lot of reusable bits in that first phase. 2. reuse as much as possible docstrings to document our web services, to avoid the doc-is-not-up-to-date-anymore plague

You can follow the development of Cornice here: https://github.com/mozilla-services/cornice

And even participate by joining our Mailing List: https://mail.mozilla.org/listinfo/services-dev

A French Python Subreddit

Tarek Ziadé — Thu, 03 Nov 2011 11:49:00 +0100

Are you a French speaking Python programmer ? Are you wishing sometimes you could debate on Python in la langue de Molière ? Are you tired of the current Economic crisis and want to find a peaceful place to hang in ?

Then, you should join us in our brand new French Speaking Python Subreddit and fetch some Vaches with us !

http://reddit.afpy.org

ldappool: a simple pool for python-ldap

Tarek Ziadé — Fri, 28 Oct 2011 19:16:00 +0200

Over the past few months, we've done quite some tweaking at Mozilla Service to make our Python platform work well for the Sync platform. One bottleneck we have is the number of open connections we can have on a given LDAP server.

On very high load, we needed to make sure we could control the flow of open connections, and when possible reduce the number of connectors as much as possible, by recycling/reusing them.

python-ldap does not come with a pool, so we've built our own with the Ops team.
The pool keeps LDAP connectors alive and let you reuse them, drastically reducing the time spent to initiate a ldap connection.

The pool has useful features like:
- transparent reconnection on server restarts, or timeouts/downtimes - configurable pool size and connectors timeouts - a context manager to simplify acquiring and releasing a connector - configurable max lifetime for connectors

The max lifetime feature is useful when you add a new server in a pool of ldap servers: if the pool is already filled with connectors, a server you will introduce will never have a chance to get a connector.
We avoid this problem by introducing a max lifetime for our connectors: the pool gets fresh connectors after a bit of time and can access all our resources.

The pool is used via a context manager, and is recycled when you leave the with block:
from ldappool import ConnectionManager

cm = ConnectionManager('ldap://localhost')

with cm.connection('uid=user,ou=logins,dc=mozilla', 'password') as conn:

      .. do something with conn ..

I have extracted it from our code based and released it on its own at pypi: http://pypi.python.org/pypi/ldappool/

Let us know if you try it !

Building Web Services with Pyramid

Tarek Ziadé — Fri, 21 Oct 2011 16:14:00 +0200

Using Pyramid for now on

Last year, when we started to write servers apps in Python at Mozilla Services, we've built a light micro-framework on the top of WebOb and Routes. That made a lot of sense back then because all our applications were pure JSON web services --except the reCaptcha view we display when you register in Sync--

The framework just needed to route a request to a function and let us do our work in pure Python from there.

Given the nature of our apps, we did not pick an async framework for the sake of code simplicity. And with Gevent, you still can boost a synchronous app that's waiting for some I/O, by making the socket layer cooperative. IOW, making your synchronous app do some work asynchronously transparently. And that works: The Sync server uses this feature for SQL and LDAP queries, and is able to handle much more concurrent requests that way.

When we made the choice of WebOb/Routes, Pyramid was on my radar and seemed like a good option too, but was still a bit new, so we just used WebOb, knowing that a move to Pyramid would be quite easy since it's based on it too.

Today, Pyramid is quite spread and mature, and for developing Sagrada components, it makes a lot of sense to adopt it. Having Ben Bangert and Rob Miller in the team also made that choice natural : they're involved in the project so they can sneak in our crazy patches ;)

Cornice - web service builder

Building web services for Sagrada, or for any project as a matter of fact, should be as simple as possible for developers. In all the frameworks I've used in the past, it was often requiring good chunks of boiler-plate code, or I had to use some tools that were making too many choices for me. For example, we want to build REST APIs, but sometimes our APIs are not strictly REST, just REST-ish I would say :)

In the past week, we've worked on a little extension called Cornice, that tries to simplify as much as possible the usage of Pyramid to write Web Services the way we like, with the experience we had writing Sync, Account portal, Easy Setup, and all the other services that are currently running in prod.

The result is a simple class that can be used to define a service on the server, then you can declare your APIs.

Here's an extract taken from demoapp,
from collections import defaultdict

from cornice import Service

user_info = Service(name='users', path='/{username}/info')

_USERS = defaultdict(dict)

@user_info.get()

def get_info(request):

    """Returns the public information about a **user**.

    If the user does not exists, returns an empty dataset.

    """

    username = request.matchdict['username']

    return _USERS[username]

@user_info.post()

def set_info(request):

    """Set the public information for a **user**.

    You have to be that user, and *authenticated*.

    Returns *True* or *False*.

    """

    username = request.matchdict["username"]

    _USERS[username] = request.json_body

    return {'success': True}

Please don't look at what the code does, it's crappy. The _USERS stuff should be in a Pyramid registry.

But look at how I've defined my service: a Service instance for a given path on our server, followed by two functions, one for the GET, and one for the POST. By default, the APIs are returning JSON, but you can change every option Pyramid offers when you declare a view.

And once this is done, I also get for free a documentation page at /__apidocs__ by grabbing the docstrings and also looking at the various Service instance that were declared.

[][]

Cornice is just syntaxic sugar, but we might add a few extra features late, maybe a fine-grained description of the params of each web services, so we could publish a manifest that could be introspected.

But it's good enough for people to start building Web Services for Sagrada in a standard, easy-to-read, fashion. They just need to call a config.include('cornice') in their Pyramid app, and use the Service class.

[]: http://ziade.org/cornice-apidocs.png

Mozillians, win a free pass for Pycon US

Tarek Ziadé — Thu, 13 Oct 2011 15:13:00 +0200

The 10th Pycon US conference will happen next March in Santa Clara, California. This is the biggest Pythoneers gathering in the world.

If you've never been to a Pycon and you like Python, you should consider coming -- It's a pretty intense week during which you can meet all the people that are behind the frameworks, libs or event the language itself. Some Mozillians will be there, no doubt.

It's an exciting time at Mozilla right now - in the WebDev team and the Services team for example, because we have a lot of folks who are very active in the Python community. Mozilla itself is active in the community and people are interested in what we build. They use our open source Python modules and we use theirs. And we'd love to have more of this.

Since Mozilla is a Gold sponsor this year, we get 5 free registrations to the conference, and we would like to give them away to some members of the Mozilla community. The pass does not cover the hotel and the travelling, but a registration costs from $200 to $450 (early bird), so if your budget is tight, it's nice to get a free pass !

I thought about picking people in http://mozillians.org/ with a silly script, but I think it could be more interesting to take this opportunity to find out what people are doing with Python in the Mozilla project.

So here's the deal -- if you are or have used Python to help out the Mozilla project, blog about it. Explain your project, how you've used Python in it. It can be on any topic, anything, really. Maybe you're more of a JS guy but you're using Python scripts for some specific tasks, or maybe you're working everyday with Django or Flask, etc.

Once the blog is published, send me an e-mail (tarek@ziade.org) with a "[pycon+mozilla]" header, and just the link in the body, before ~~December the 1st~~. February the 1st.

I will pick the five stories I found the most interesting and announce the winners shortly after.

OSDC Slides - High performance Python web apps

Tarek Ziadé — Tue, 04 Oct 2011 16:15:00 +0200

We gave a talk last month with Benoit in Paris about web servers.

The talk consisted of a first part were we've explained how a web server works, and we got into details like what's a backlog etc. and explained how the technology evolved from a single CGI process to modern setups based on event loops or such things.

We've also presented the stack I've set up for the Firefox Sync server, and gave an overview about stress tests tools.

You can grab the PDF translated in English here: High Availability Server Apps - benoitc + tarek

Or watch them on SlideShare here: High Availability Server Apps

The bottom line is : use Nginx, Gunicorn and Gevent

If you wonder why we have socks and shoes on our slides, it's because "socket" sounds a lot like "chaussette" (means socks) in French, so it's a (stupid but efficient) joke :)

"High Availability Server Apps"

The French Python User Group (Afpy) is 8 y.o.

Tarek Ziadé — Sun, 02 Oct 2011 20:55:00 +0200

Wow. I've realized this today. We've founded that user group in 2003, when I've proposed a Pythoneers meeting in Paris through my Python/Plone forum.

We've created Afpy.org after that, the non-profit organization, and we had many meetings since then, including various Pycon France.

Many people I've met through the Afpy are now people that I consider friends for life, even if I don't see them much these days.

And Python still rocks 8 years after we've started all of this, getting bigger and bigger every year, thanks to the awesome leadership of Guido and all Python core dev.

The bottom line is: if you like Python and don't have a user group in your area, create one.

Mozilla sponsors Pycon US

Tarek Ziadé — Fri, 30 Sep 2011 10:08:00 +0200

Mozilla is now officially listed as a Pycon US sponsor.

This is great news for Python and for the Mozilla Community: more and more projects in Mozilla are using Python. It's the mainstream language for all the work we're doing in the Services team on server-side, and numerous projects like SUMO or AMO are powered by Python.

So, if you are involved in the Mozilla community and in Python, you should consider submitting a talk or a tutorial. Or just attend to the conference in Santa Clara next March.

Pycons are always a lot of fun.

PyPI : CDN vs Mirrors

Tarek Ziadé — Thu, 29 Sep 2011 18:30:00 +0200

We had a few discussions in the last days about what would be a good way to make PyPI more reliable.

It think there were a bit of confusion about the mirroring protocol (PEP 381) and its goals, versus the reliability of the current PyPI main server.

Some people were basically saying (I am paraphrashing)

Just move PyPI to a CDN and be done with it, this mirroring thing is too complicated.

Well, ok. We could set up a CDN for our PyPI files and have all our archives at Amazon or elsewhere.

But since the mirroring protocol is implemented and available on server-side (We have 5 active mirrors), and since Pip already supports switching to a mirror, the shortest path to a better PyPI is simply to :

create a new mirror in a <put the name of the best provider> CDN and register it as a mirror at PyPI

And if it's so better, so reliable and fast, maybe we'll move it up in the mirrors list, as the first one so all clients should pick it by default.

And the day <put the name of the best provider> is down. (yeah it happens, remember EC2 a few months ago), you will all be thankful that we have other mirrors and a protocol to switch over them !

So, if you think a CDN is the magic solution, go ahead. Grab pep381client, set up your monster infrastructure, and let me know so I add it in the list or mirrors. And maybe we will never ever call another server again. Or maybe not.

For further info, here is a detailed summary of PyPI status we've built with Noah: http://wiki.python.org/moin/BetterPyPI

Related: I am going to submit a tutorial on how to work with PyPI, mirrors, private packages etc., to show how I do at Mozilla

Sagrada - Creation and Hosting of Web Services

Tarek Ziadé — Thu, 22 Sep 2011 16:01:00 +0200

tl;dr

I am building for Sagrada a web application that can be used to create and deploy web services through the web with a few Python functions.

See the prototype demo quick screencast -- If you can't view it in your browser, download it and use VLC -- The music is MC Jack IN the Box CC-AN 3.0

EDIT Here's a better demo, with Ace integration, where you can edit/create Python files in the browser -- http://ziade.org/redbarrel4.mpeg

Now read below for the long story :)

The Sagrada Project

We had a all-hands last week in San Jose, which gathered +600 Mozillians (yes, we're growing fast).

My team has started to brainstorm about Sagrada, the big project we're starting, and I've started to think about a few topics.

One long term goal of Sagrada is to let developers deploy on our servers their own web services. It's quite hard to put a strict definition for the term "web service", but the term is usually used these days to describe a server-side application that can be queried via the HTTP protocol, receiving and sending back Json objects.

So, the intent is to provide a Service Container for developers that need a server side for their applications.

The server side of Firefox Sync is a good example: It's a set of web services Firefox calls to power Sync. See the API definitions here.

For Sagrada the specific question I am interested to solve is :

What would be the easiest way to write and deploy web services into our infrastructure ?

To start off, here are a few assumptions I am making about what developers would probably expect from this kind of service -- but those are my own assumptions, and will change with the feedback I'll get in the process :

1. Building and deploying a web service should be easy to do whether you are a Python programmer or a JavaScript programmer.

While Python is our main target, being able to write JavaScript on server side should also be a goal -- And if the tool is extensible enough, why not other languages in the future ?

Moreover, most steps when building web services are not specific to a language. For instance, describing what HTTP method should be used, what's the web service URL, what kind of request body is expected, etc, is usually done in a specification document.

For example, when I've built the Easy Setup server for Sync, I've written this specification document : http://docs.services.mozilla.com/keyexchange/apis.html

It contains a description of each web service used by Firefox when you add a new device to your Sync account. The only thing that is not expressed in this document is the just the piece of code that does the job.

So, what if this document could be used directly by the server to run the application ? The only thing that would miss is a few functions.

The bottom line is that web services can be described in a Domain Specific Language (DSL) that can be used to generate the documentation automatically (and it stays accurate and up to date) but also to set up in the server things we usually do on the code side: the request dispatching.

In that case, the portion of code to write is reduced to building the response itself.

Even if we just support Python in a v1, if building a web service boils down to writing a specification document, and a few Python functions to build responses, I think that lowers the barrier enough for most developers.

And if we hide the DSL behind a nice user interface developers can use to build their apps, that's even better. That's what's happening in the screencast I've recorded. The forms generate portions of DSL.

2. The tool should be framework-agnostic if possible.

While tools like Node.js or Pyramid or <put your favorite framework here> provide great features to write web apps, web services people will build will be running in specific environments, where we will want to set up and control our own stack.

In other words, we'll want to isolate as much as possible the code written by developers.

Ideally, we'd just pass a request to a function and ask for a response. Python for instance has a CGI-like standard called WSGI, where the request is described in a mapping and the response is a sequence of string + a mapping of headers.

So telling developers: "Hey, your function will receive a WSGI request, send me back a WSGI response", sounds like a good basis.

I am pretty sure we can do the same in Javascript. And if we don't have such standard in Javascript, let's create it.

3. The tool should provide a Web UI, and CLI and a standalone server.

Web services should not be locked in our servers, developer should be able to edit them through the web, or via the console even if it's remote. They should be able to upload or download their apps in our environment, like what they would do with Google App Engine.

But they should also be able to create, run their apps on their own environment e.g. have a independent web server that can run their services. Not a toy server used for development only, but something they can really deploy in production.

For the remote console, Benoit was telling me this morning: why not iPython ? I dig this: editing web services through iPython would rock. If you don't know about this tool check it out it's amazing. It provides among other things, a way to build an interactive shell for a distant app.

A web service DSL

Back to our DSL. When you write a web service, it's always the same story no matter what framework you're using, you're basically doing these steps (I am over-simplying for now)
1. Define a route for your service on the server 2. Build the response 3. Send back the response

These steps can be described in a simple DSL.

Here's a basic example:

path hello_world (

 description "Simplest application: Hello World!",

 method GET,

 url /hello,

 use python:somemodule.hello

 );

With a function hello located in somemodule that can look like this

def hello(request):

return 'Hello World'

The application in that case is composed of
- a DSL file - a Python file with a single function

It's easy from there with the proper DSL parser to:
- deploy those two files in our infrastructure - run the app - provide auto-generated documentation for the service

Architecture

The prototype I've written for the demo does the following:
- The tool is a web application that provides forms to create Service Containers - Each Service Container has a unique root on the server - In each container you can add web services. - For each container, the DSL is built on the fly, then the corresponding AST is kept in memory. The web UI allow users to modify it on the fly - When a request comes in, it's rooted to the right Service Container depending on the beginning of the path, then the AST is used to find out what function(s) should be used. The function is then executed in a sandbox.

The prototype also provides a command-line tool to start a server by loading an arbitrary DSL file.

The DSL Parser

I will not go in to great details here, you can look at my previous posts mentioning RedBarrel.

The current DSL is implemented with PLY and can be found here.

Sandboxing the code

One thing we want to do when a developer uploads some code that is potentially going to be executed, is to sandbox it.

This is not really for security reasons, because we'll still need to protect our users by setting up VMs. This is mostly to
pre-configure what the user is allowed to do in the code and provide some useful feedback when he's doing things
we did not allow, like writing to the filesystem or using sockets -- I am not saying we will allow or disallow these
particular ones, I don't know yet.

In the current prototype, I've used pysandbox from Victor Stinner . It's used to load and execute the uploaded code.
see how I use it.

What's Next

Right now the web interface to build an application is simplistic compared to what the DSL can do. For instance you could chain several functions to perform pre- and post- processing for a request. So some functions can be reused in several services, like authentication.

Also, we want to provide our own Mozilla libs, like a way to authenticate against our own user database, or use a key-value storage. Basically, all the libraries we're currently building for Sagrada.

It's unclear at this point what imports we will allow in the scripts, and how we will publish our own libraries people will be able to use. I intend to clarify these in the upcoming days, and enhance the prototype to allow it to do more things.

Also, I'd like to write the Easy Setup server using this tool. I'll also try to organize a coding/brainstorming sprint since I have 5/6 people that worked with me on these topics to hack on this.

If you are interested or have some feedback, please comment !

Mozilla Services is hiring

Tarek Ziadé — Mon, 19 Sep 2011 16:18:00 +0200

My team at Mozilla (Services) is looking for talented people to work on Python-based server apps, such as Firefox Sync.

Currently, we're mainly writing web services. Firefox Sync and most projects we have, uses a micro-framework and stack I've initiated last year, based on WebOb and Routes, GUnicorn and NGinx.

For Sync, the code is a thin layer on the top of a big MySQL cluster (more than 100 servers) and some LDAP Servers for the user authentication.

We're responsible for writing the code & tests, and releasing it so the Service Operations team can deploy it.

That includes some packaging work, having a TDD approach, and caring about the beauty of the code.

I like my work for many reasons, here are a few:
- It's all open source - I work with very smart & talented people - We work on big scale projects - We're trying whenever it makes sense, to make it possible for people to run our apps on their own servers. - Anyone is welcome to contribute. In practice, we don't have a lot of contributions on server-side because the community uses our own deployed services, but there are still a few people out there that are running their own server and sending us back some feedback/patches. That's the interaction I like the most.

And check out the big project we're starting in Services:
- https://blog.mozilla.com/services/2011/09/15/introducing-project-sagrada/ - https://wiki.mozilla.org/Services/Sagrada

The only caveat of my job is that I am currently the only member in my timezone, so I feel disconnected sometimes. But this is changing: the major part of the server team is in PDT (California), but we now have someone in Australia (yay!) and it is planned to have some other folks in my timezone (or at least, close) soon.

I'd love to see more Europeans join the team. Click here to apply or contact me.

You can check my previous blog entries in the Mozilla category to see what I have been up to in this area.

https://blog.mozilla.com/services/2011/09/15/introducing-project-sagrada/

Pycon Japan Retrospective

Tarek Ziadé — Mon, 05 Sep 2011 16:42:00 +0200

I was back last week from my trip in Japan for Pycon Japan, and I wish I had spent an extra week there to see more places there.

Pycon Japan

Pycon Japan was a blast, really well-organized and from what I've heard they had more than 200 attendees, which is a good size for a single day conference. I took some notes for Pycon France.

[caption id="attachment_2029" align="alignright" width="406" caption="Some PyconJ organizers (Picture from Manabu Terada)"][][][/caption]

I gave a keynote session -- you can grab the slides here: http://ziade.org/pyconjp-keynote.html. The keynote was mostly about Packaging and a little bit on my (subjective ;)) opinions about the uptake of Python 3, PyPy etc.

I've spent the day chatting with people, and even with Guido himself, since the organizers set up a Skype-like video chat system were we could do live 1:1 chats with him. Guido spent quite some time with every attendee that wanted to speak with him. Thumbs up to Guido and to the organizers for this nice idea.

Most talks were in Japanese but lots of speakers used dual English/Japanese slides so I could follow a bit. The lightning talk session at the end was fun -- as usual for those, and the organizers ended up offering some Mozilla Goodies after a giant rock/scissor/paper game. Then we had a party in a pub and ended up in another bar late at night.

On a side note, I was amazed by the number of attendees asking me for an autograph on my book, Expert Python Programming, translated in Japanese.

Also, check out Ian Lewis' blog about the event.

Blockdiag

Komiya Takeshi showed me his tool called blockdiag, which is a DSL you can use to add diagrams in your documentation. The nice thing is that it provides a Sphinx extension so you can add diagrams in your documentation through simple expressions, and have Sphinx generate for you the diagrams on the fly. There's even an interactive online shell: http://interactive.blockdiag.com/

I've challenged Komiya to write a few diagrams I have for some Mozilla projects using his tool, and it took a few seconds for him to build them. So, I am going to use this in the future.

Tokyo, Kyoto, Nara

After the conference, we left with my wife to visit the city, then other places in Japan: Kyoto and Nara. Nara and its big park filled with temples is a very beautiful place. It was not the best time in the year to visit because it was very hot, but it still was really worth it. The best time is probably in March, when the cherry blossoms are opening.

We went to some beautiful parks and temples, and slept in traditional Ryokans. More pics here

[caption id="" align="alignnone" width="440" caption="Todai-ji in Nara"][/caption]

Japan has met my expectations : the food is great, people are extremely nice, and there are beautiful places to see.

A huge Thanks to the Pycon Japan Team for inviting me and taking care of Amina and I (Naokata Jay Hotta, Yasushi Masuda, Ian Lewis, Manabu Terada, Morimoto Tetsuya, etc..)

"pycon-japan"

[]: http://tarekziade.files.wordpress.com/2011/09/pycon-japan.jpeg "Todai-ji in Nara"

5 tips for packaging your Python projects

Tarek Ziadé — Fri, 19 Aug 2011 16:05:00 +0200

Next week I am keynoting at Pycon Japan, and one thing I will talk about is packaging of course. And in particular: what advice can I give my audience on how to package Python projects today ?

This is a hard task, because we are in some kind of transitional state.

Anyways, I wrote down a list of advices and removed everything that was dependent on the tools we did not release yet -- that's another part in my keynote.

Here's a list. Most of them are not controversial. If you see something missing or want to rant about one, please comment.

Tip # 1 -- Use a PEP 386 compatible scheme for your versions

Having several version scheme in our eco-system is pure madness. It breaks interoperability, and makes it impossible to write tools that handle versions properly. By using a PEP 386-friendly scheme now, you are making your project future-proof !

PyPI already rejects any Metadata 1.2 project that does not comply to this policy. You probably don't know this because no tools produces Metadata 1.2 packages yet. But that's going to be the default in Python 3.3 and distutils2.

So long "devdevdev123" and "3765-2011-test" versions !

Tip #2 -- try to make setup.py as dumb and simple as possible

setup.py is not your personal build system. I have seen crazy things in some projects. Remember that setup.py is used by installers for a lot of different tasks. Like getting the metadata fields of the project.

Here's a simple test: make sure "python setup.py --name" (double dash) eturns the name field without any external dependency, and without calling any function or method.

Remember that setup.py is going away in Python 3.3 and distutils2, replaced by simple options in setup.cfg. Don't be scared, you will still able to do complex tasks.

My advice: don't do anything else that feeding setup() with options in there. Put all your build things in another place, and if they need to be called by setup.py, make sure they are called only when needed.

Tip #3 -- Do not make any assumption about which installer will be

used

Make sure your setup.py can be run by a vanilla Python (==distutils). Even if you use setuptools or distribute, in most case you can manage to have it working in both tools. You can always tell the user to do extra steps manually if he needs to.

Forcing the installation of an installer, by using the ez_setup script for instance, without asking, is a bit rude to the end-user. It's basically forcing the end user to use a new installer. If you do this in your setup.py, ask first !

Or simply tell the user "This project only works with the XXX installer -- install it if you want. Aborting."

Tip #4 -- Do not release unstable releases at pypi

Our installers are not --yet-- smart enough to prefer stable releases when they are asked to get a project at PyPI. That's how PyPI is built: every project has a directory with all releases and it's up to the installer to decide which one is the "latest". The only tool out there that's smart about it is zc.buildout.

So when you push an alpha release or a rc release at PyPI, it's going to land in people environments unless they have mature processes to update their stuff -- or simply because they make the assumption that PyPI is where stable release go. So do not make assumptions about how your users are updating your project.

Prefer another explicit channel for your beta testers. All installers know how to install from any url or directory.

Tip #5 -- Be cautious about your data files

Distutils or Distribute or Python itself have no way to explicitly make a difference between a doc file or a media file or a configuration file. They are all data files. Worse, since they are no universal place for data files on the various OSes, people tend to treat their data files like Python modules so they are able to find them back on the target system without trouble.

Yeah that's broken, and we've fixed it in 3.3. But until then, that's unfortunately the most protable way to do this. So what you can do is document clearly how you handle your data files and create a single function or module that reads them. That'll help the downstream maintainers to handle your project.

Afpy Camp Wrapup -- RedBarrel + Pistil + 0mq

Tarek Ziadé — Mon, 08 Aug 2011 18:12:00 +0200

We had a lot of fun at the Afpy Computer Camp this year. Some people worked on RedBarrel with me and some others on various topics, including Pyti -- A GSOC project.

In RedBarrel, besides fixing a lot of bugs we did the following:

We've finished two socket.io based demos

a monitoring page that displays in real-time using Flot the CPU and memory usage of the server
an url shortener that provides a monitoring page where you can see on a google map a live stream of the location of the URL visitors.

We've integrated Pistil into Redbarrel

We started to integrate scaling features

Pistil is Benoit's new project. It's a project very similar to Gunicorn that allows you to define how a web server behaves via "workers" and "arbiters". Pistil is basically replacing all Gunicorn components except the command line script. One nice feature Benoit added to Pistil this week-end is the ability to define different kind of workers.

That's because in Redbarrel we need:
- some RedBarrel workers, each one running a RedBarrel wsgi server to build the responses -- see this code - One Flash server policy worker for the flash fallback in socket.io -- see this code - A Broadcast server -- I'll explain why later

Once you've integrated Pistil, you just need to add a custom command-line script in your app and call arbiter.run()-- see this code

From now on, RedBarrel doesn't need GUnicorn anymore to run and manage several processes. yay.

Scaling RedBarrel

The goal of integrating Pistil into Redbarrel was to make it possible to run several processes (==workers) to handle more requests.

RedBarrel is using Gevent underneath via gevent.socketio, so that gives a single worker the ability to run a lot of concurrent requests already. But still, if you're doing database accesses in your application, you will need to run several workers if you want to handle several hundreds of concurrent requests.

And that leads to another issue : if the application uses the socket.io features, we need the ability to broadcast messages to all connected clients no matter which process they're in. gevent.socketio implements this feature using Gevent queues in memory, so it's constrained to a single process.

If you have several processes, you need to be able to broadcast messages to all the workers in order to reach all connected clients. A inter-process communication protocol needs to be set up for this.

zeromq to the rescue !

zeromq integration

zeromq seems to be the best tool for this, because it hides all the complexity when you need to set up a simple communication protocol between several processes and servers. zeromq also allows you to use different transports, whether they are local to the same server (ipc) or between servers (tcp).

zeromq provides several behaviors to exchange messages : PUSH/PULL, PUB/SUB etc. see its nice doc.

For our use case, we need to be able to broadcast messages from any worker to all other workers. Instead of linking all workers to each other, a simpler pattern is to have a single server that is responsible for all the broadcasting. Workers subscribe to it to receive messages, but also can send messages to broadcast to the server.

Setting this up with zeromq is done by using a bi-directional communication through 2 channels:

A publisher/subscribers channel.

the publisher is a single zeromq server that is able to broadcast messages to several subscribers.
each worker becomes a subscriber and receives messages from the publisher.

A push/pull channel.

each worker may push messages to be broadcasted to the publisher
the publisher pulls messages and broadcasts them to all subscribers

That sounds like a complex setup, but is not at all ! In my first load tests it seems very efficient.

To implement this in RedBarrel, we used pymzq. We also needed to use[gevent_zmq][]. This small library simply makes pymzq green, so it's compatible with Gevent. The code I added is composed of a Broadcaster class and a Client class. It's very straightforward. You can read the code here.

Now when RedBarrel is launched, one Broadcaster is launched and every worker has one zmq Client, allowing inter-worker broadcasting.

Next steps

The next steps are:
- make gevent.socketio use our zeromq system. gevent.socketio currently uses Queues to push and pull messages to be broadcasted. We can keep these queues and just feed them via zeromq. What I need to do here is to hook the worker's broadcasting feature into the code that interacts with the queue. - allow inter-servers communication. this will simply be done by allowing a broadcaster to become a subscriber of other servers broadcasters, so it can broadcast locally to its own subscriber what was broadcasted in the other server. What I am unsure about yet is how the whole thing will scale under a very heavy message load.

Some other features ideas we had:
- provide a key/value storage using redis and a pure-python fallback and allow its definition via the DSL. - add an event system based on the zeromq integration. one event that comes in mind: broadcast to all workers an event everytime a key is changed in the storage.

And, yeah, we ate and drunk a lot..

"Dinner at the Afpy Camp"
"biere"

RedBarrel is now socket.io compatible

Tarek Ziadé — Wed, 03 Aug 2011 18:18:00 +0200

I got really excited by web sockets the other week-end, so I decided that RedBarrel should support them out of the box and provide a very easy way to work with them, like what socket.io can provide on the server-side.

The usual "Chat" application

I have a first version of sockets support in RedBarrel. Most web socket demos include a chat application. I ripped the one I found here: https://bitbucket.org/denis/websocket/src/tip/examples and changed it until it worked in RedBarrel.

The whole application is composed of the RBR file, a python module an an html page with its JS libs.

Here's the relevant parts of the RBR file:
# hooks the chat() function as a socket.io service

path socket (

    method SOCKETIO,

    use python:redbarrel.demos.sockets.chat

);

# that's the html file that displays the chat room

path chat_html (

    method GET,

    url /,

    use file:chat.html

);

And the whole app is contained in one module:
import json

_BUFFER = []

_D = json.dumps

def add(msg):

    _BUFFER.append(msg)

    if len(_BUFFER) > 30:

        del _BUFFER[0]

def chat(globs, request):

    """A Chat room using web sockets"""

    socketio = request.socketio

    sid = socketio.session.session_id

    def announce(message):

        socketio.broadcast(_D({'announcement': message}))

    if socketio.on_connect():

        announce(sid + ' connected')

        socketio.send(_D({'buffer': _BUFFER}))

    while True:

        message = socketio.recv()

        if len(message) == 1:

            message = {'message': [sid, message[0]]}

            add(message)

            socketio.broadcast(_D(message))

        else:

            if not socketio.connected():

                announce(sid + ' disconnected')

Nothing fancy here, gevent.socketio does all the magic. But I am very glad that I was able to integrate this in RedBarrel with the no-boiler-code-at-all philosophy I want to keep. That is 1/ define a RBR file 2/ do the coding

How the feature was added

The problem with web sockets is that their implementations may vary or may be nonexistent under some browser flavors. I've hacked something that worked, then looked at how I could make the thing work under Firefox, IE and Chrome and started to feel like back in the old days when we had to work out our Javascript code for all the different browsers. This may be still true but at least they are some decent libraries now that make most of the JS code cross-compatible.

socket.io

socket.io seems to do a very good job in this area for web sockets by providing a lot of different transports implementations for applications that want to have "websocket-like" features. A Transport here is just one way to send and receive data between the client and the server. It may be the shiny web sockets, it may be something else.

socket.io has a websocket implementation -- sorry I am not following the RFCs on this, but I know there are at least 2 versions -- but also some fallbacks in case the browser does not seem to be compatible with web sockets. That includes a flash component and various other techniques based on async calls, like XHR polling. And it works really well -- I don't know how the fallback algorithm works internally, but I was able to have a socket.io app running against the latest Firefox, an old one and Chrome.

gevent.socketio

On the server side, I wanted to implement in RedBarrel something that looked as simple as what socket.io offers. So I needed to:
- implement every transport protocol socket.io supports - provide an async layer for all the broadcasting work

But.. it turns out that the gevent.socketio project already offers all of this -- it implements requests handlers for all the transports and a socket object with what we need. It uses GEvent in the background.

So basically all I had to do was to :
1. extend the RedBarrel DSL so we can define "sockets" in applications 2. make the RedBarrel wsgi application use gevent.socketio to handle incoming requests 3. expose the socket object in the WebOb request object on the fly 4. let the code do whatever it wants with the request and the socket, hopefully something useful

was easy -- you can now define one path that has SOCKETIO as its method. That tells RedBarrel that the application will use socket.io and calls should be transmitted to the code pointed. The syntax might evolve since "path socket" (see the example at the beginning of this post) is not really needed. I might change it to something more straight forward.
was done by switching to gevent.socketio own wsgi handlers + gevent runner. Once gevent.socketio has done its prep work like the web socket handsake, it goes back to the wsgi application -- so in our case the generic RedBarrel application that runs the DSL.
When RedBarrel is called, it looks for the environ, where gevent.socketio adds a socket object and simply stick it to the WebOb request object.
the code that is called receives, like for classical calls, a WebOb object and can use the attached socket.

What's next

I am going to add a sexy demo -- I asked people on G+ what demo I should add and it looks like I'll add a real-time server monitoring demo (using Flot) with these features:
- people can look in real-time what's going on (CPU, Memory) via constantly updated diagrams - people can talk to each other on the page ("Hey bob, the server is melting down, don't you think?")

If you have a small websocket app in Python and would be interested to see how it could fit in RedBarrel, please let me know, I'd be happy to give it a shot

Pycon France and Pycon Japan this summer

Tarek Ziadé — Mon, 01 Aug 2011 16:56:00 +0200

It's awesome to see all those Pycon conferences getting organized throughout the world. http://www.pycon.org lists most of them.

For me, Pycon is also a good opportunity to travel and meet people -- I try to attend to Pycon US every year, and when possible to an extra Pycon elsewhere.

This summer I'll attend to Pycon France and Pycon Japan. I've submitted a proposal for the first one, that will happen in Rennes, France in September. If you're in France, you should attend. Rennes is lovely and while the conferences are in French, there will probably be some social events before and after, where english speakers won't be left aside !

I am also giving a keynote for the second one that's happening in Tokyo, Japan at the end of this month. If you happen to be a Pythoneer in Tokyo, speaking French or English, I'd love to meet there.

How to stress test your app using Funkload -- part 2

Tarek Ziadé — Thu, 28 Jul 2011 14:52:00 +0200

The first part of this blog post is here.

I've initially split it in two parts because I've encountered a few problems with the distributed feature, so I thought it was going to take Ben a few days to fix them. But he did fix them within the hour... \o/

So Hurra for Ben and here's the second part of my small tutorial.

Running a distributed test

If you want to hammer a server, you cannot run all the virtual concurrent users from a single box. You will end up eating all your resources and will not be able to simulate a very big load. If you need to run a couple of hundreds of concurrent users, one box is enough. But you'll need more boxes to get more load, obviously. Depending on the complexity of the tests, I usually end up having at the most 200 CU (Concurrent Users) per node.

In the past, I was using Fabric to do distributed tests for Funkload, see here. But this is not needed anymore because Funkload has now a built-in support to distribute the charge amongst several nodes. It uses a similar technique that I used in Fabric, by driving the nodes via SSH using Paramiko. The benefit though, is that it does all the heavy lifting for you: your test suite gets deployed on the nodes, and the XML result files gets downloaded for you on the master.

All you have to do once the test is over, is to merge the XML files and generate reports. And the merging is taken care of by a Funkload script so... Let's do this !

First of all, you need to install Paramiko in your environment:
$ bin/pip install paramiko

Also, move to a trunk snapshot version of Funkload, since there are a couple of fixes there for this feature:
$ bin/pip install -U -f http://funkload.nuxeo.org/snapshots Funkload

Next, select a few boxes that will be your nodes to run the tests, and make sure they have Python and virtualenv installed, and that you can access via SSH to them without having to type anything. The simplest way to do this is to copy over an ssh key with an empty passphrase. See this article if you don't know how to do it.

Notice: having a ssh key that does not require a passphrase is a potential security hole, so make sure these key are used only for this purpose, and safe.

The next step is to add a [distribute] section in the Simple.conf file:
[distribute]

log_path = distributed-simple-test.log

funkload_location=http://pypi.python.org/packages/source/f/funkload/funkload-1.16.0.tar.gz

These options tell Funkload which Funkload release should be used on the node, and where to put the logs. The script will install it in a virtualenv on every node, prior to running the tests.

Once everything is set up, you can run the test using all your nodes with the --distribute flag and the --distribute-workers option, that gives a list of the nodes. The script deploy the tests into the nodes, run them, and grab back the XML files.

In my environment I have a master, and two nodes (node1 and node2)
$ bin/fl-run-bench --distribute --distribute-workers=node1,node2 test_simple.py Simple.test_simple

========================================================================

Benching Simple.test_simple

========================================================================

Access our Demo app

------------------------------------------------------------------------

Configuration

=============

* Current time: 2011-07-28T14:11:47.720959

* Configuration file: /home/tarek/dev/hg.mozilla.org/funkload-demo/Simple.conf

* Distributed output: distributed-simple-test.log

* Server: http://master:5000

* Cycles: [5, 10, 20]

* Cycle duration: 10s

* Sleeptime between request: from 0.0s to 0.5s

* Sleeptime between test case: 0.01s

* Startup delay between thread: 0.01s

* Workers :node1,node2

* Preparing sandboxes for 2 workers...

* Starting 2 workers..

* [node1] returned

* [node2] returned

* Received bench log from [node1] into distributed-simple-test.log/node1-simple-bench.xml

* Received bench log from [node2] into distributed-simple-test.log/node2-simple-bench.xml

Once the test is over, you will find two XML files on your master, you can merge to produce an HTML report:
$ bin/fl-build-report --html -o html distributed-simple-test.log/node1-simple-bench.xml distributed-simple-test.log/node1-simple-bench.xml

Merging results files: ..

nodes: tarek-laptop, tarek-laptop

cycles for a node:    [5, 10, 20]

cycles for all nodes: [10, 20, 40]

Results merged in tmp file: /tmp/fl-mrg-JL62Bi.xml

Creating html report: .../home/tarek/dev/hg.mozilla.org/funkload-demo/html/

done:

/home/tarek/dev/hg.mozilla.org/funkload-demo/html/test_simple-20110728T141152/index.html

Congrats, you're now able to run distributed tests !

Monitoring the server

The last feature I want to show is the monitoring. Funkload with let you monitor:
- the network traffic on a given interface - the CPU load average - the memory usage - the number of concurrent users the server is handling over time

This is very useful to detect memory leaks or abnormal consumption of memory.

To do this, you need to run a monitor server provided by Funkload on the server you want to watch. Once Funkload is installed there, create a monitor.conf file.
[server]

host = server

port = 8008

interval = .5

interface = eth0

[client]

host = master

port = 8008

Here server is the server I am benching, and the monitor server will run on the port 8008. The interface parameter will just tell Funkload which one to watch. The client section tells the monitor server which server will call it, so our bench master.

Once this is saved, simply run the monitor server with:
$ fl-monitor-ctl monitor.conf start

Starting monitor server at http://localhost:8008/ as daemon.

use stop to stop it, obviously.

Last but not least, back to our master bench server, open Simple.conf and add these sections:
[monitor]

hosts = localhost

[localhost]

description = The application server

port = 8008

You're all set ! Simply run the benches as usual, and you should see a new section in the reports you're generating, with four new graphs.

[][]

That's all. I hope you found this mini-tutorial interesting, and that you'll give Funkload a shot. I've tried many tools, like The Grinder, Apache JMeter, and some proprietary things from Mercury etc. And Funkload, out of the box, beats them all because it let me create my own stress tests without any crazy interface of framework.

Thanks for all the work Ben.

"localhost_MonitorCPU"

[]: http://tarekziade.files.wordpress.com/2011/07/localhost_monitorcpu.png "localhost_MonitorMemFree" []: http://tarekziade.files.wordpress.com/2011/07/localhost_monitormemfree.png

How to stress test your app using Funkload -- part 1

Tarek Ziadé — Wed, 27 Jul 2011 13:07:00 +0200

EDIT: Part 2 is now published(distributed tests and monitoring)

Funkload is a functional and stress test tool that can be used on your web applications.

It's my favorite stress tool for these reasons:
- stress tests are implemented as PyUnit tests, so they can also be used as functional tests - the test runner is very light, it's dead easy to run a stress test on your box against a local app - it's simple to do a distributed test with a few extra options -- Funkload will drive the other boxes using SSH - Funkload provides nice reporting features out of the box: you can do trends and diff reports - You can monitor the server being tested

This blog post demonstrates how to use Funkload on a dummy app and generate report. A second blog post will explain how to run the test using several test servers, and how to monitor the server that runs the application.

The tested application

The dummy application we'll stress test for this demo is a very simple WSGI application that can be run in plain Python, or using Gunicorn:
from wsgiref.simple_server import make_server

import time

def application(environ, start_response):

    status = '200 OK'

    headers = [('Content-type', 'text/plain')]

    start_response(status, headers)

    time.sleep(0.1)

    return ["Hello World"]

if __name__ == '__main__':

    httpd = make_server('', 5000, application)

    print("Serving on port 5000")

    try:

        httpd.serve_forever()

    except KeyboardInterrupt:

        pass

Every time the application is called, no matter what's the request path or method, it will sleep for 100 ms then return a Hello World.

Let's run our application:
$ python wsgiapp.py

Serving on port 5000

The application is ready to get some hits !

Installing Funkload

Let's open a new shell and create a local environment in a directory, using Virtualenv, and install Funkload in it:
$ virtualenv --no-site-package --distribute .

New python executable in ./bin/python2.6

Also creating executable in ./bin/python

Installing distribute.................................done.

$ bin/pip install Funkload

Downloading/unpacking Funkload

...

Successfully installed docutils Funkload webunit

Cleaning up..

Once everything's installed, you will get in the local bin directory a few Funkload scripts:
$ ls bin/fl*

bin/fl-build-report    bin/fl-install-demo

bin/fl-record     bin/fl-run-test

bin/fl-credential-ctl  bin/fl-monitor-ctl

bin/fl-run-bench

The two scripts that interest us right now are fl-run-test and fl-run-bench.

A first Funkload test

Let's create a test_simple.py module in our directory:
import unittest

from random import random

from funkload.FunkLoadTestCase import FunkLoadTestCase

class Simple(FunkLoadTestCase):

    def setUp(self):

        self.server_url = self.conf_get('main', 'url')

    def test_simple(self):

        server_url = self.server_url

        res = self.get(server_url, description='Get url')

        self.assertEqual(res.code, 200)

        self.assertEqual(res.body, "Hello World")

if __name__ in ('main', '__main__'):

    unittest.main()

This test simply checks that the server returns Hello World and that the response status code is 200.

To run this test, Funkload needs a few options to run. These options can be placed in a configuration file. Let's create a Simple.conf file with this content:
[main]

title = Demo

description = Simple demo

url = http://localhost:5000

[test_simple]

description = Access our Demo app

[ftest]

log_to = console file

log_path = simple-test.log

result_path = simple-test.xml

sleep_time_min = 0

sleep_time_max = 0

[bench]

cycles = 5:10:20

duration = 10

startup_delay = 0.01

sleep_time = 0.01

cycle_time = 1

log_to =

log_path = simple-bench.log

result_path = simple-bench.xml

sleep_time_min = 0

sleep_time_max = 0.5

Those are defining options Funkload will use when it's running.

Now we can try out our Funkload script with the fl-run-test script, which will run the tests just once:
$ bin/fl-run-test test_simple.py

.

----------------------------------------------------------------------

Ran 1 test in 0.104s

OK

Victory ! The test is working nicely.

Let's now try a full bench, using fl-run-bench. The bench script takes an extra option which is the test method to use to run the bench:

$ bin/fl-run-bench test_simple.py Simple.test_simple

========================================================================

Benching Simple.test_simple

========================================================================

Access our Demo app

------------------------------------------------------------------------

Configuration

=============

* Current time: 2011-07-27T12:02:35.319172

* Configuration file: /home/tarek/dev/hg.mozilla.org/funkload-demo/Simple.conf

* Log xml: /home/tarek/dev/hg.mozilla.org/funkload-demo/simple-bench.xml

* Server: http://localhost:5000

* Cycles: [5, 10, 20]

* Cycle duration: 10s

* Sleeptime between request: from 0.0s to 0.5s

* Sleeptime between test case: 0.01s

* Startup delay between thread: 0.01s

Benching

========

* setUpBench hook: ... done.

Cycle #0 with 5 virtual users

-----------------------------

* setUpCycle hook: ... done.

* Current time: 2011-07-27T12:02:35.321906

* Starting threads: ..... done.

* Logging for 10s (until 2011-07-27T12:02:45.380090): .............................................................................................. done.

* Waiting end of threads: ..... done.

* Waiting cycle sleeptime 1s: ... done.

* tearDownCycle hook: ... done.

* End of cycle, 11.66s elapsed.

* Cycle result: **SUCCESSFUL**, 94 success, 0 failure, 0 errors.

Cycle #1 with 10 virtual users

------------------------------

* setUpCycle hook: ... done.

* Current time: 2011-07-27T12:02:46.986399

* Starting threads: .......... done.

* Logging for 10s (until 2011-07-27T12:02:57.117788): .............................................................................................. done.

* Waiting end of threads: .......... done.

* Waiting cycle sleeptime 1s: ... done.

* tearDownCycle hook: ... done.

* End of cycle, 12.86s elapsed.

* Cycle result: **SUCCESSFUL**, 94 success, 0 failure, 0 errors.

Cycle #2 with 20 virtual users

------------------------------

* setUpCycle hook: ... done.

* Current time: 2011-07-27T12:02:59.844273

* Starting threads: .................... done.

* Logging for 10s (until 2011-07-27T12:03:10.100680): ................................................................................................ done.

* Waiting end of threads: .................... done.

* Waiting cycle sleeptime 1s: ... done.

* tearDownCycle hook: ... done.

* End of cycle, 23.18s elapsed.

* Cycle result: **SUCCESSFUL**, 96 success, 0 failure, 0 errors.

* tearDownBench hook: ... done.

Result

======

* Success: 284

* Failures: 0

* Errors: 0

Bench status: **SUCCESSFUL**

The script runs three cycles of respectively 5, 10 and 20 virtual users, for 10 seconds each. This is configured with the cycles and duration options in the [bench] section of the configuration file (cycles=5:10:20 and duration=10), but you can also provide these options through the command line.

Let's say you want to run 2 then 5 users for 5 seconds each:
$ bin/fl-run-bench --cycles=2:5 --duration=5 test_simple.py Simple.test_simple

========================================================================

Benching Simple.test_simple

========================================================================

Access our Demo app

------------------------------------------------------------------------

Configuration

=============

* Current time: 2011-07-27T12:07:56.294745

* Configuration file: /home/tarek/dev/hg.mozilla.org/funkload-demo/Simple.conf

* Log xml: /home/tarek/dev/hg.mozilla.org/funkload-demo/simple-bench.xml

* Server: http://localhost:5000

* Cycles: [2, 5]

* Cycle duration: 5s

* Sleeptime between request: from 0.0s to 0.5s

* Sleeptime between test case: 0.01s

* Startup delay between thread: 0.01s

Benching

========

* setUpBench hook: ... done.

Cycle #0 with 2 virtual users

-----------------------------

* setUpCycle hook: ... done.

* Current time: 2011-07-27T12:07:56.297701

* Starting threads: .. done.

* Logging for 5s (until 2011-07-27T12:08:01.322419): ............................. done.

* Waiting end of threads: .. done.

* Waiting cycle sleeptime 1s: ... done.

* tearDownCycle hook: ... done.

* End of cycle, 6.28s elapsed.

* Cycle result: **SUCCESSFUL**, 29 success, 0 failure, 0 errors.

Cycle #1 with 5 virtual users

-----------------------------

* setUpCycle hook: ... done.

* Current time: 2011-07-27T12:08:02.574411

* Starting threads: ..... done.

* Logging for 5s (until 2011-07-27T12:08:07.637394): ............................................. done.

* Waiting end of threads: ..... done.

* Waiting cycle sleeptime 1s: ... done.

* tearDownCycle hook: ... done.

* End of cycle, 6.60s elapsed.

* Cycle result: **SUCCESSFUL**, 45 success, 0 failure, 0 errors.

* tearDownBench hook: ... done.

Result

======

* Success: 74

* Failures: 0

* Errors: 0

Bench status: **SUCCESSFUL**

Now let's check the reporting features..

Reporting

Everytime you are running a bench, an XML file is produced. In our case it's simple-test.xml

The XML file contains raw results and can be used to produce reports. fl-build-report takes these XML file and produce reports out of them.

For example, you can create an HTML report with the html option.

Make sure you have gnuplot installed, then run:
$ bin/fl-build-report --html --output-directory=html simple-bench.xml

Creating html report: ...

done:

/home/tarek/dev/hg.mozilla.org/funkload-demo/html/test_simple-20110727T120756/index.html

The result is a nice HTML page containing various diagrams, like the number of requests per seconds depending on the number of virtual concurrent users.
[][]

Another nice report is the diff report, which takes two already generated reports, and build a diff one -- if you get some errors, make sure you have the latest gnuplot installed.
$ bin/fl-build-report -o html --diff html/test_simple-20110727T123642 html/test_simple-20110727T123718

Creating diff report ... done:

/home/tarek/dev/hg.mozilla.org/funkload-demo/html/diff_simple-20110727T_123718_vs_123642/index.html

The diagram you get will provide a clear overview of the differences between the two runs. This is useful if you want to check for speed regression when you've changed some code.

[][]

The trending report has the same goal, but can be built using more that two runs:
$ bin/fl-build-report -o html --trend html/*

Creating trend report ... done:

/home/tarek/dev/hg.mozilla.org/funkload-demo/html/trend-report/index.html

That's useful to see how your application is doing over time.

[][]

In the next post we see two extra features Funkload provides:
- run distributed tests - monitor the benched server

Go to part 2

"requests_rps"

[]: http://tarekziade.files.wordpress.com/2011/07/requests_rps.png "rps_diff" []: http://tarekziade.files.wordpress.com/2011/07/rps_diff.png "trend_avg" []: http://tarekziade.files.wordpress.com/2011/07/trend_avg.png

Anatomy of a web service -- Part 3

Tarek Ziadé — Tue, 19 Jul 2011 17:15:00 +0200

I am pursuing the RedBarrel experiment, mainly by writing some documentation to show how things work.

If you don't recall what I am talking about, read back the two previous blog entries:
- Anatomy of a Web Service -- part 1 - [Anatomy of a Web Service -- part 2
][]

I added a full demo in the documentation, that explains how to write an URL shortener Web Service with RedBarrel (don't look at the code itself -- it's irrelevant): http://redbarrel.readthedocs.org/en/latest/demo/

If you happen to have some web services based on Pylons, Pyramid or anything on the top of WebOb, I'd be very happy to try to refactor them using RedBarrel as an experiment, to see how things work out. (not for you to use, but just for me to exercise the idea)

[Anatomy of a Web Service -- part 2
]: http://tarekziade.wordpress.com/2011/06/17/anatomy-of-a-web-service-part-2-redbarrel/

Firefox Sync Server is now 100% Python

Tarek Ziadé — Tue, 12 Jul 2011 12:36:00 +0200

Two weeks ago, we pushed the last bit of the Python Sync Server in production, and there's no more PHP.

For the client-side it's not changing anything, since the Python server is just a re-write of the existing PHP server.

Lessons learned

The first push we did of the storage part on that week went really bad and we had to rollback urgently, fix the problems and push it back a few days later.

The main problem we had in production was related to the MySQL driver we used in conjunction with Gunicorn and GEvent. We picked PyMySQL because we wanted GEvent's ability to monkey patch the socket module -- using MySQL-Python would have been useless for this since it uses C code.

When you use Gevent workers with GUnicorn, sockets become automatically cooperative and you can handle more parallel requests that are waiting for data from the SQL server. Read more about this here.

And that's exactly what the Sync server is: a thin layer of web services on the top of a database, sending requests and waiting for the results.

PyMySQL was working fine in our load tests and in staging. We were happily pushing the load and had slightly better performances than the PHP stack.We were not expecting a huge difference since most of the time (I'd say around 80%) is spent waiting for the SQL server and the Python server is using the same database.

But the main difference is that the Python stack stays persistent in memory, so we can pool connectors and avoid recreating TCP connections for every request. I don't have any hard numbers yet, as we're collecting them, but we've definitely reduced the time taken by our web services in those 20% spent outside the SQL server.

But.

But as soon as we pushed in production, everything started to lock. Some queries were just hanging and incoming requests were piling up until we were unable to cope with the load.

What happened is that PyMySQL is using socket.send() to send data to the MySQL server, without checking that all the bytes were really sent. And on high load, with Gevent, doing this will not work anymore because you're not necessarily sending all bytes at once. The API to be used is send.sendall() to make sure everything is sent.

Here's an extract of the doc for send():

socket.send(string[, flags]) ~ Send data to the socket. The socket must be connected to a remote socket. The optional flags argument has the same meaning as for [recv()][] above. Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data.

And for sendall():

socket.sendall(string[, flags]) ~ Send data to the socket. The socket must be connected to a remote socket. The optional flags argument has the same meaning as for [recv()][] above. Unlike [send()][], this method continues to send data from string until either all data has been sent or an error occurs. None is returned on success. On error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent.

As soon as we've changed the code in the driver, (PyMySQL's author was told about this, and the tip is now fixed. Also there's the same problem in MyConPy it seems...) everything went smoothly.

So the question you're probably wondering is: why didn't we caught this issue in our load test environment ? The reason is that our load test script was not asserting all the responses the web server was returning, and we did not detect those errors and the locked queries were basically timing out in a mass of normal behavior. They "came back" as valid. The load test infrastructure, while filled with hundreds of thousands of fake users' data, has less databases than in production so this kind of issue is not bubbling up as hard. While our load test infrastructure is very realistic, it will never be exactly like production.

The other thing is that the Grinder outputs raw data and we just used the Query Per Second indicator. I suspect we would have caught this issue with Funkload because it provides some results diagrams were you can see things like min and max.

So the main lessons learned here are:
- make sure the load test scripts assert all the responses (status + content) - make sure your load testing tools detect any abnormal behavior -- like a very very long request, even if it's a fraction in a mass of normal behavior

I am very thankful to the Services Ops team, and in particular Pete who drove the production push. These guys rock.

What's next

Now that everything works well, there are a few things we need to tweak in order to have a better system:
- Kill pending queries when a Gunicorn worker is restarted - See if we can cache a few LDAP calls - See if we can use several GUnicorn servers behind one Nginx -- the CPU is under-used.

But overall, I hereby declare the Python push as a success.

"socket.socket.recv"
"socket.socket.send"

Mozilla Services - Year 1 !

Tarek Ziadé — Tue, 21 Jun 2011 16:13:00 +0200

What’s this ? read this post.

What happened

Time flies... This is a special entry because today it's been exactly one year since I've joined Mozilla. So I'll try to summarize what I've done during the past year, what worked well, what needs more work.

I was the first "Python guy" hired in the Services team. Since the plan was (and still is) to do all future server-side work in Python, this gave me the opportunity to set up the basis for our eco-system. Since then, more Python devs have joined the team and others have gained some experience in the language, and we're now all Python guys ;)

I created a micro-framework for all our server-side applications. It's a thin layer on the top of WebOb + Routes, with some specific features we need (like a hearbeat page for all our apps). Nothing fancy but it makes it easy for someone to start a Server application sharing a common standard.

I re-wrote all the PHP code that is needed to run a Sync server (Registration + Storage) in Python, using the micro-framework. This was done months ago, and both the core and the apps have evolved since then, following the changes that were made in the PHP parts. Pushing the work in production was the hardest part because it supposes a lot of coordination between Devs and Ops. And Ops had zillions of stuff to do for the Firefox 4 launch, so we missed that launch. Since then, the Registration part was pushed, and we're about to push the Storage part.

I wrote the Easy Setup server, that is used for example when you set up your Android phone's Sync account by transferring your desktop Sync profile. This was really fun because it was the first project I did from scratch. I worked on the design with Phillip and Stefan and I really enjoyed it. A small, fun and interesting project.

I added a IP blacklisting middleware into the Easy Setup server, that rejects calls from an IP if it has done too many attempts in a short time window. This work led to a new project that is currently under development by Ryan: a blacklisting server that can work across several applications.

I've set up a documentation center at http://docs.services.mozilla.com/that is built using Sphinx. The sweet thing about it is that its content lives in a Mercurial repository, so developers can change the documentation directly. The goal of this website is to provide a single place for all our APIs specifications, and our development guidelines, with a minimal overlap with sites like MDN etc.

I've created pypi2rpm and we worked with Richard S. on deploying our Python applications using it. The tool is built on the new packaging tool I am working on for Python and let you create RPMs files for our projects. It's a mix of a PyPI crawler, a version sorter and a RPM creator. We've created a release process based on this: http://docs.services.mozilla.com/server-devguide/release.html

I've set up a Jenkins server for our server projects. Can't link it because it currently leaves in our intranet. The Jenkins server does the following for every server application:
- builds it - runs its tests - build all its RPMs under Centos5 and Rhel6 - Install all the RPMs under a Centos5 chroot and check that there's no dependency error

I've set up a PyPI Mirror in our intranet so Jenkins (or someone creating RPMs) does not rely on any external resource.

Also: we're having regular MoPy meetings now at Mozilla \o/

What's next

During the next months we will do a lot of work in the Sync application(s) to support new features and optimize existing ones. We'll also start new projects.

For the tooling/framework parts, here are a few paths where I'd like to see us going.

Continuous load testing: one goal I have is to make it possible for anyone to run a load test on one of our applications via Jenkins, on a given stage or dev server. Then see the results, compare them with some previous runs etc. The second goal is to make load tests an easy thing for developers to add early in a project. Writing load tests makes your application faster, just because you focus on it, that's a fact.

I've made good progress in this topic and some stuff are going to land into Funkload itself to make it easier to automate load tests and reports generation. The next task I have is to work with other teams involved with Services, and make sure their use cases in this area are provided by the tools.

A standard for our API specifications. This is a personal project I have, but it is directly inspired by our work at Services. I want to reduce the path from a server API specification to its implementation. For this I am working on a DSL that can be used to describe some APIs, produce their documentation and eventually, be used to run a server. If my experiment goes well, I'll introduce it.

Cross-team code review. Inspired by Jeff and some other people during a MoPy meeting, I started to write a small web site that could be used by a dev to ask for code reviews from people outside its team. The idea is that the website tries to find automatically someone that has the required skill to do the review. A review will cost you some credits and by doing a review you'll earn some.

On working remotely

I am a big fan of remote working. I am doing it for years. It works quite well within the Services team, and half of the team is doing it. The one thing that's hard to manage is the timezones: I am the only guy in Europe and I am 9 hours in front of Mountain View.

This means that if I need to work & chat with my colleagues, I have to do it on the evenings. Also, some events have to be scheduled on the PDT timezone. For instance, production pushes are starting at midnight for me. I had to adapt a bit my schedule for this, it was hard at first but now it's working well.

But... I would really love to see some more people in my team hired in Europe, so I can speak with them on daytime.

As a conclusion for year 1, I love my job at Mozilla. I'm working with a lot of smart guys and we get things done, mostly because everyone in this team is passionate. I a looking forward for year 2.

Anatomy of a Web Service, part 2 - "RedBarrel"

Tarek Ziadé — Fri, 17 Jun 2011 15:14:00 +0200

I was talking about web services the other day: read it back as an introduction to this post.

I am pursuing this DSL experiment as I have now finished a working prototype of a micro-framework. I've called it RedBarrel (Monty reference). I've called the DSL files "RBR files".

RedBarrel is a pure Python implementation of the DSL I've described in the previous post and does the following:
- loads the DSL file and run a WSGI web application (via rb-run) - Allows you to check the syntax of an RBR file (via rb-check) - generates a documentation page for the APIs at /__doc__ Note that description fields can be in reStructuredtext and are rendered in HTML - publishes the DSL file at /__api__ - runs the code pointed in the DSL and does the post- and pre- processing as described

Here's an example of a web service, that capitalizes the string you sent to it -- and requires authentication and json input.
define path capitalize (

    description "A web service with several post/pre processing",

    method POST,

    url /capitalize,

    use python:redbarrel.demos.capitalize,

    response-status (

        describe 200 "Success",

        describe 400 "The request is probably malformed",

        describe 401 "Authentication failure"

    ),

    request-body (

        description "Send a string in json and the server returns it Capitalized.",

        unless type is json return 400

    ),

    request-headers (

        unless Authorization validates with redbarrel.demos.auth return 401

    ),

    response-headers (

        set content-type "application/json"

    ),

    response-body (

        description "The string, Capitalized !",

        unless type is json return 503

    )

);

[caption id="attachment_1862" align="alignleft" width="542" caption="The /__doc__ page in RedBarrel"][][][/caption]

The one thing I am not entirely sure about yet, is if I want to provide helpers to instantiate some objects in memory when the server starts. That's useful when you want to keep a DB connector open or simply avoid initializing many things on every request. But that's very easy to implement with global variables... Maybe an "application context" could be created, for some functions to add objects inside.

Anyways, I've started to re-write a few web services to see how the DSL fits, and so far it looks useful: I have reduced a lot the boiler-plate code and the API is self-documented.

The code is still at Bitbucket and I am looking for some feedback or other people that are writing web services that want to experiment with it !

"The /__doc__ page in RedBarrel"

[]: http://tarekziade.files.wordpress.com/2011/06/barrel.png

Continuous Load Testing wint Funkload

Tarek Ziadé — Fri, 10 Jun 2011 13:08:00 +0200

There's one thing we're often neglect when we build applications: performance trending. In other words, are things getting slower over time because the code base grows, or because someone introduced a O(n) piece of code somewhere ?

In very big projects this is a major concern: the code grows and the performances are steadily decreasing. I read somewhere that this became a problem in the Linux Kernel at some point. Can't find a good link on this. If you know about this story, please let me know.

In smaller base code, it's still important to watch, and by following the changes you can detect such issues. But you can't focus all the time on everything. That's why doing it automatically and continuously is important.

For our Services at Mozilla, we're benching our APIs with stress tests, but what we miss is a bit of automation, so we can do continuous load testing and keep a history of our performances.

Some examples:
- I want to see after a new feature has been introduced, if my RPS average on my API remains the same. - When some indexes are changed in the Database, what's the impact on the overall performances ?

Getting good statistics from continuous load testing requires a stable environment with realistic data, so the trend means something, and that's hard to keep over long periods of time. But it's manageable for shorter periods, like weeks, I guess.

Funkload trending

I've started to work on this topic this week, to try to set up some automation in our stress tests, and I've the tool it takes to do this easily: Funkload.

[caption id="" align="alignright" width="531" caption="Disclaimer: These are not Firefox Sync performances, just a sample from Funkload docs ;)"][/caption]

Everytime you run a Funkload stress test, it produces its results in an XML file you can store.

Funkload then provides a script to generate reports for every bench your run, but is also able to produce trend reports, using Gnuplot.

So what I've started to do is:
- run Funkload via Jenkins against the server that has the deployed APIs - collect all produced XML files - refresh and publish the trend report

Grinder is nice, and I could do the same thing with the raw data. But why bother, Funkload does already all the reporting needed \o/

"Trend"

Anatomy of a Web Service

Tarek Ziadé — Thu, 09 Jun 2011 09:34:00 +0200

This is cycling in my head for a while now, and I think it's close to become something concrete.

Let me summarize the idea: web services are most of the time doing the same post- and pre-processing tasks over and over and there should be a way to describe them via a DSL.

Nothing revolutionary here, but what if Nginx could handle for you all the boring parts and let you just handle the meat of your services. Having a DSL to describe web services potentially allows such delegation.

Anatomy of a Web Service

A Web Service is basically doing these four steps:
1. [pre-processing] Check the request body and headers, and potentially reject it. Rejection can be due to a Basic Authentication failure, an unexpected value for the request body, etc. 2. [routing] Find what code or application should be called to build the response. This is usually computed with the path information and sometimes some headers. 3. [execution] Invoke the code to build the response 4. [post-processing] Return the response built and maybe do some post-processing or post-assertions like converting the content-type etc.

Steps 1., 2. and 4. could be delegated to a proxy as long as it has enough details on what should be done.

In Python, when you build web services using a WSGI framework like Pylons, Pyramid, or simply Routes + WebOb, all of these steps happen in your code. You define the routing using Routes descriptions, or using more clever dispatching systems like what Pyramid offers, then delegate the execution to a controller class or a simple function, after a potential pre-processing. Although the pre-processing part is often merged with the execution part because they are closely related.

For instance, if you have a web service that requires a JSON mapping in the request body, you could write something that looks like:
def my_webservice(request):

    try:

        data = json.loads(request.body)

    except ValueError:

        raise HTTPBadRequest("Unknown format -- we want JSON")   # this raises a 400

    ... do something ...

Of course you can always generalize this by using a decorator to clearly separate the pre-processing part:
@if_not_json(400)

def my_webservice(request):

    ... do something ...

Same thing for the post-processing step:
@if_not_json(400)

@convert_output('application/json')

def my_webservice(request):

    ... do something ...

Err... well, in some frameworks, the routing itself is expressed as a decorator:
@route('/here/is/my/webservice')

@if_not_json(400)

@convert_output('application/json')

def my_webservice(request):

    ... do something ...

It turns out that there are a lot of pre/post steps that can be pushed to a meta level.

Delegation of pre- and post-processing steps

A web application is most of the time accessed through a proxy. At Mozilla Services, we use Nginx for all our Python applications. NGinx is here --among other things-- to pool incoming requests and dispatch them to our Python application. The proxying job is pretty dumb right now, as everything that comes in is directly sent to the Python backend.

What if we were able to delegate all the pre- and post-processing we've seen earlier to NGinx ?

There would be some benefits, like a faster rejection of bad requests: no need to invoke the Python application anymore and spend CPU cycles in a backend for this. If some pre-requisites are not met, we can 400 right away.

Having all the pre-processing at the proxy level also make it simpler to modify them without touching the web service code itself. That can be a default as well of course in some cases : your application logic is split in two parts and this can be hard to follow. But as long as the full description of the web service is in a single place, I think it's fine.

Last, we've talked about Python, but each piece could be implemented in a different language, as long as NGinx is able to invoke it. Using Lua for all the pre-processing part is not a bad idea..

The DSL

The last time I've talked about this topic, someone talked about SPORE which is indeed quite similar to what I want to achieve. I guess the biggest difference is that SPORE focuses on providing a DSL to build clients that can interact with an existing set of server APIs.

What I want on my side is to provide a DSL API developers can use to create web services, and eventually have a proxy like NGinx use it to run the application.

A developer ideally would:
1. describe her web services in a DSL file 2. implement the execution part 3. test them in a development environment where a web server would load the DSL and the code 4. deploy the web service in production with NGinx

I could start off with SPORE but I want to experiment with my own DSL and build it little by little.

I started to build it the other week-end, and it looks like this:
define path hello (

    description "Simplest app",

    method GET,

    url /,

    use python:demos.hello

);

define path counter (

    description "A counter",

    method GET,

    url /count,

    use python:demos.counter

);

define path html (

    description "An html page",

    method GET,

    url /index.html,

    use python:demos.html,

    response-headers (

        set content-type "text-html"

    )

);

define path service (

    description "A web service that 400 if the body is not json",

    method POST,

    url /post,

    use python:demos.post,

    request-body (

        unless type is json return 400

    )

);

Some details:
- python:demo.post means here: the code to be invoked is located in the "demo.post" callable - that's the fully qualified name to reach it, so demo can be a package, or a module. - unless type is xxx return xxx is a full part of the DSL, a recognized structure. When parsing the file, it's loaded in an AST and executed on each request against the body.

Implementation details

I used PLY to read the DSL files, and it'll check many aspects of the DSL file like:
- make sure the method is known (GET/POST/etc) - make sure the HTTP codes used are valid ones - control that the URL path is valid - etc.

I wrote a small Python application that loads the DSL file at startup in an AST. Then it provides a web server that will do the post/pre-processing, and eventually delegate the execution to some Python code, by passing a request object using WebOb. The URL is for now using a simple regexp pattern backed by Routes.

I've also wrote:
- a script that validates a DSL file - a __doc__ web page in the small Python server, that displays the available web services - an __api__ page that just publish the DSL file for client-side discovery. (fwiw)

You can have a look at the ugly code of the prototype here: https://bitbucket.org/tarek/redbarrel

The next steps in this experiment will be to rewrite one of our small Services app with it, and see how it comes out.

All in all, I am having a lot of fun doing this, and it's eating some of the free time I have when I should be really doing some work on Python packaging... meh ...

Help us ironing Packaging

Tarek Ziadé — Thu, 02 Jun 2011 12:23:00 +0200

packaging has landed in the standard library, but the road to Python 3.3 is still filled with a lot of work. We've pushed the Documentation yesterday in the tip, and it now appears here: http://docs.python.org/dev/packaging/

There are a lot of stuff you can do to help us improving packaging. If you wish to help out, read up.

1. Install a Python 3 development environment

The first step is to install a Python development environment

There's a full dev guide here: http://docs.python.org/devguide/ but it basically boils down to run make on the tip:
$ hg clone https://hg.python.org/cpython (very long)

$ cd cpython

$ ./configure && make

Once this is done, you'll have a Python interpreter you can run:
$ ./python

Python 3.3a0 (default:94066c3e2236+, May 31 2011, 08:29:53)

[GCC 4.5.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> print('Python 3, yay !')

Python 3, yay !

2. Try out the pysetup script, as an end-user

This script is a global script people will be able to use to check what's installed on a Python installation, to install things, remove them, etc. The script has still a lot of rough edges, which is a shame since it's just the tip of a feature-rich system.

It's located in Tools/script/pysetup3 in a dev environment, and here's a demonstration of how to install the lastest Mako release, check that it's installed, look at some of its metadata, then remove it:
$ sudo ./python Tools/scripts/pysetup3 install Mako

Checking the installation location...

Getting information about 'Mako'...

Installing 'mako' 0.4.1...

[lots of output]

$ ./python Tools/scripts/pysetup3 list

SQLAlchemy 0.7.0 at /usr/local/lib/python3.3/site-packages/SQLAlchemy-0.7.0-py3.3.dist-info

distribute 0.6.17 at /usr/local/lib/python3.3/site-packages/distribute-0.6.17-py3.3.dist-info

Mako 0.4.1 at /usr/local/lib/python3.3/site-packages/Mako-0.4.1-py3.3.dist-info

Found 3 projects installed.

$ ./python Tools/scripts/pysetup3 metadata Mako -f Version

Version:

    0.4.1

$ ./python Tools/scripts/pysetup3 metadata Mako -f Author

Author:

    Mike Bayer

$ sudo ./python Tools/scripts/pysetup3 remove Mako

Removing 'Mako':

  /usr/local/lib/python3.3/site-packages/mako/parsetree.py

 [lots of lines]

  /usr/local/lib/python3.3/site-packages/Mako-0.4.1-py3.3.dist-info/RECORD

Proceed (y/n)? y

Success: removed 52 files and 2 dirs

So go ahead, play with this script, discover its features and:
- tell us what feels wrong - tell us what kind of features you wish you had in this script - found a bug, have a patch, tell us !

3. Make your project packaging-ready, as a developer

The sweet thing is that adding packaging support in your project is risk-free because it's just adding a few sections in your setup.cfg file. setup.py can stick around, and older installers will still pick it up.

So, here is how you can do: You can tollow the tutorial if you want to do something from scratch:http://docs.python.org/dev/packaging/tutorial.html

Or you can use the magic create command in your Project root directory, to create a setup,cfg out of your setup.py file !
$ sudo ./python Tools/scripts/pysetup3 create

A legacy setup.py has been found.

Would you like to convert it to a setup.cfg? (y/n)

    [y]: y

Wrote "setup.cfg".

If you're starting your project from scratch You can also generate a setup.py that will extract the options out of setup.cfg. Very handy to provide backward compatibility and avoid maintaining two files !
$ sudo ./python Tools/scripts/pysetup3 generate-setup

The setup.py was generated

So go ahead, learn how setup,cfg works, reads its specs at http://docs.python.org/dev/packaging/setupcfg.html and:
- tell us what feels wrong - tell us what kind of features you wish you had in this file - found a bug, have a patch, tell us ! - tell us if you were unable to convert your project

Once the project is packaging ready, you can even register and upload a new version of it at PyPI and check that pysetup knows how to install it

4. Give use some feedback

You can add new bugs/feature requests athttp://bugs.python.org/ under the Distutils2 component, that will be really really helpful.
You can tell us what's weird with our documentation, what misses, etc. That goes to the Documentation+Distutils2 components

Mozilla Services - Week 22

Tarek Ziadé — Tue, 31 May 2011 14:27:00 +0200

What's this ? read this post.

What happened

I am getting ready for an important push today, that will switch some services under Python. Things are looking good.

On a side note: my fishes kept on dying in my aquarium and I finally found the problem by having the water analyzed. Thenice little river in my small town (175 people), that looks so clean, is basically saturated with Nitrates. I used it for my aquarium because I did not want to use tap water combined with some products to remove the chlorine. I thought I was helping out those poor fishes with nice river water, but I was killing them. This is insane.

Cross-team reviews

During the previous summit, we discuss at the MoPy meeting an interesting idea. What if people could ask for a code review from anyone from another team, that has the skills to do the review. I first thought about doing some kind of plugin for Bugzilla and have for every registered user a set of skillx, then propose a reviewer in the patch UI.

But that means people need to use the Bugzilla review process, and sometimes they use something else. I also wanted this review to be just an extra review with a low-commitment from the reviewers. In Bugzilla, if you are asked to review something, it will stay there waiting for your review for ever even if you don't review it. I don't think there's a way to timeout a review.

Last, cross-team review could be something broader than Mozilla teams. What about getting a review from someone in another Python project ?

Anyways, I started "Bug Brothers", a prototype to do this. There's a demo running here: http://bugbro.ziade.org and the code is here: https://bitbucket.org/tarek/bugbro. This was a good opportunity to try Pyramid, and yeah <blush> no tests and the code is not very clean.

It's not finished but it already allows people to ask for reviews, provide a link to a diff. When you review something you get credits, and when you ask for a review you pay credits. Everything is email-driven.

The next steps are to add more features like Rietveld. a tighter integration to Bugzilla, github. etc -- but without introducing a dependency to any tool so it can work for every team.

Mozilla Pulse

Coming from the Plone/Zope/Python world, I miss my checkins mailing lists in Mozilla projects. That is, getting a mail everytime a commit is done in one of the projects you work on. You can always read the Atom feeds in the various Mercurial repos, but that's not the same.

What I want is a diff in a mail can quickly look at. This is very useful to get instant reviews from other people. You usually catch more typos or mistakes. It also help initiating coding discussions.

Christian Legnitto has started the Mozilla Pulse project, which is exactly what I needed: a way to get notified on every change in the Mozilla eco-system. I was waiting for Pulse to get hooked in all our repos and this is now done.

The script to send e-mails on commits is very simple: https://bitbucket.org/tarek/services-pulse/src/tip/consumer.py. I need to add a diff view in the e-mail and a few options, but that's basically it. For now, it keeps only events happening in hg.mozila.org/services, and it will send e-mail to our services mailing list.

Overall, Pulse is a good way for anyone to watch a particular area in the Mozilla project

Stop guessing encodings

We had a bug in our Services code, related to a password containing an non-ascii character. It's a shame that as a French I did not insist on unicode vs str before. So here we go.

In Python 2 we have two types to deal with strings. We can use the str type or the unicode type. The str type is basically storing bytes so a string is encoded using a particular encoding, By default the encoding is ascii. The unicode type encodes strings as 16 or 32 bits integers and covers the unicode table. The most common error is to make no assumption whatsoever on the type of the string you get. What will happen is that some functions that need bytes will simply try to decode unicodes using the ascii coded, or vice-versa:

import base64

>>> def encode(data):

...     return base64.encodestring(data)

...

>>> encode('I am oke')

'SSBhbSBva2U=\n'

>>> encode(u'I am oké')

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "<stdin>", line 2, in encode

  File "/usr/lib/python2.7/base64.py", line 315, in encodestring

    pieces.append(binascii.b2a_base64(chunk))

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 7: ordinal not in range(128)

One solution that comes in mind is to check for the type of the string in your function:

def encode(data):

...     if isinstance(data, unicode):

...         data = data.encode('utf8')

...     return base64.encodestring(data)

...

>>> encode(u'I am oké')

'SSBhbSBva8Op\n'

This is tempting but leads to another issue: if by default your program is able to deal with string or unicode for all your strings, there are high chances that you'll miss to check for the type somewhere or combine str and unicode in some places.

A much better approach is to use internally in your program only unicode and deal with conversions in inputs and outputs. In a Python web app it boils down to check that all inputs are unicode (beware of JSON).

The other issue is the encoding and the decoding. What codec should we use ? The asnswer is utf-8, because it's the most universal. To make sure there's no misunderstanding: a unicode is a decoded string, and a str is encoded. So you can decode() str and encode() unicode:

u'é'.encode('utf8')

'\xc3\xa9'

>>> 'é'.decode('utf8')

u'\xe9'

So, use only unicode in your apps, and when encoding, use the utf8 codec by default.

More on this here: http://docs.python.org/howto/unicode.html

What's next

More Python deployments
Some benches in the Sync server

The Architecture of Open Source Applications

Tarek Ziadé — Mon, 23 May 2011 22:26:00 +0200

About a year ago, Greg Wilson has started a very ambitious book project about the architecture of open source applications. The line-up was impressive, since more than 20 people have started to write about various topics. From the Battle of Wesnoth to Audacity, via Mercurial. And the project is, as of today, completed !

Everything is available online, but you should consider buying it at LuLu because all royalties are going to Amnesty International.

I can't really give a detailed review about the book yet because I am still reading it, but I already can tell you it's awesome :)

And I am proud to say that I've contributed a chapter about Python packagingin which I am trying to explain what we're doing to improve the situation. This is a good timing since I've just pushed packaging into the standard library. So if you want a bit of background in this area of Python, read up. The chapter should be pleasant to read thanks to the work of Greg Wilson and Amy Brown. They managed to convert an awful Frenglish chapter into something that is readable.

Again, you should buy a copy of that book.

"packaging" has landed in the stdlib

Tarek Ziadé — Sun, 22 May 2011 23:02:00 +0200

I've pushed in the standard library the new version of Distutils !

This new version, initially known as distutils2 was renamed to packaging (Tres Seaver came up with this name during the last Summit), and contains all the improvement we have been working on during the last 2 years or so on the distutils codebase.

There are still a lot of thing to do until Python 3.3 is out, but having packaging back in the standard library makes our work easier, and will boost its polishing for the upcoming release.

This work is still not very useful to the community since we need to backport it to Python 2. The plan is to run 3to2 and have a standalone version released under the Distutils2 name.

This tool has countless improvements I will explain later, whenever we push the documentation in the coming days.

Just to whet your appetite:
- for end-users Python gains a pysetup script you can use to install, uninstall projects, browse installed projects, browse PyPI, and many other things. - for developers, no more setup.py file. You just define everything in new setup.cfg sections. You can also define your data files in details. - for os packagers, you can relocate data files where they should be installed, without breaking the code.

~~nb : the push busted all the buildbots in the process, and I've been busy in the last days to fix them. I still have cryptic issues under BSD and Solaris, but most problems have been fixed. Thanks a lot to Victor, Ezio, Antoine, Ned, David and many more people that helped me there~~ Buildbot is now back to normal, everything was fixed \o/

nb 2: the commands and compiler parts are not really different from distutils, but we have numerous things to change before 3.3, and I'll try to get more people involved in that part because I don't know what's a compiler :). If you're doing sci stuff w/ Python, know compilers, and want to help us, let me know.

Mozilla Services, week 16/17

Tarek Ziadé — Mon, 02 May 2011 16:43:00 +0200

What's this ? read this post.

What happened

I took a bit of vacation and came back last week. I took a lot of pictures and I love this one taken at the Amneville Zoo near Metz. I was quite amazed by the quality of this Zoo by the way.

Share Server (F1)

We're continuing our work to get the F1 server ready, and Rob Miller, who joined last month (yay!) is helping me in this task. We're working on:
- removing the Pylons layer and using our own WSGI micro-framework like in other Services Apps -- since the F1 server is just a small oauth proxy, that should depend on nothing else than the library for every service (GMail, Twitter, etc.) - setting up the Services Status DB. This is explained here: https://wiki.mozilla.org/Services/F1/Server/ServicesStatusDB. Basically this will let us turn on and off the proxy for a given service - continuing the Grand Split. The static web pages are now living on their own.

The Services Status DB involved some research/brainstroming with Shane, on the best way to do it, and we decided we would:

1/ Use Membase to store the status of every service. Membase provides vbuckets and has a really nice web managment interface to add/remove/modify the cluster. Pylibmc and libmemcached do not understand vbuckets yet, but I am working on it.

2/ Reject calls at the Nginx level when the service is down. I first wrote a NGinx module in C but we will not use it. The problem with NGinx modules is that everytime you change them you need to recompile and redeploy Nginx. Meh. Instead, as I was suggested in the Nginx mailing list, we're adding the Lua module in Nginx, that will allow us to script the behavior we want from within the Nginx config files.

Documentation

Hey check this out : http://docs.services.mozilla.com.

This is the sweet documentation center I've set up for Mozilla Services. I have started to add more content about each one of our Service, basically extracting what we have here and there in our wiki. I am trying to follow a similar pattern for each one of them.

While wikis are awesome to work collectively on specs, I love Sphinx because it makes it so easy to consolidate documentation for a project, and share a glossary, terms, definitions etc. And the documentation is treated like code, all in an Hg repository here: https://hg.mozilla.org/services/docs/

There's a lot of content to gather from various places in the wiki(s) in order to build a complete list of our services, but the idea is to add documentation there once the API has stabilized.

What's planned

More Share Server work. I will also try to push on more docs.

Also there's a new side project I've started that might be interesting to share here, I will try to blog later in the week about it. In a few words: it's a tool that should help MoPy people to share knowledge and try to maximize cross-team reuse of code when possible.

Python Packaging Documentation sprint April 2-3

Tarek Ziadé — Fri, 01 Apr 2011 15:29:00 +0200

This is a short notice but here we go: We're having an online documentation sprint this week-end.

The goal is to clean up the documentation for packaging before I do the merge in the stdlib.

If you want to help please read this page : http://wiki.python.org/moin/Packaging/Sprints/DocumentationSprint and join the IRC tomorrow and/or the day after.

Anyone can help here, since the documentation aims to three targeted readers:
- Developers that want to package/release their projects - OS Packagers - End-users

I'll be sprinting myself Sunday since I am travelling today and tomorrow. Kudos to Kelsey, who's leading the sprint effort.

bdist_rpm is dead, long life to Py2RPM

Tarek Ziadé — Fri, 25 Mar 2011 16:39:00 +0100

One thing I've learned along the way while working on Python packaging matters is that it's quite impossible to maintain in the stdlib scope tools like bdist_rpm, which builds RPMs using Distutils' enabled projects.

I've asked around, and it turns out 9 out of 10 of the packagers are not using it for their daily RPM packaging work, unless they are also the developers of the projects to package. They use their own tools.

There are several reasons:
- bdist_rpm makes a lot of assumptions when creating the spec files. - there are no way to customize some sections in the spec file that will be generated. - the spec file is most of the time edited by packagers before it's actually used, so the automatic full translation from Python metadata to RPM metadata is a false good idea. And while packagers have templates, they eventually end up working manually in the spec. - What's good for RedHat 5 is not for the latest Fedora - Python sdtlib cycle does not match the distro cycles. So even if we update it, it would be soon deprecated again

So, when it was suggested a few months ago to add a bdist_deb command in Distutils, I told people that anything related to creating deb files should be a project on its own. And stdeb seems to have this role now.

For Windows, it's a bit different: tools like bdist_msi are easier to maintain and we don't have many flavors of packaging in the win32 world, just one. The release cycle of Python's stdlib fully works here.

Following this principle bdist_rpm has been removed from Distutils.

So for the RPM world I want the same standalone project, that provides RPM goodies for Pythoneers, on the top of Distutils2 a.k.a. the new packaging module in Python 3.3, to replace bdist_rpm.

As a matter of fact I've started this project a while ago for my work at Mozilla and called it pypi2rpm then. It's called like this because its main feature is to create for CentOS a RPM package out of a project released at PyPI.

Try it:
$ pip install pypi2rpm

$ pypi2rpm.py SQLAlchemy

...long output...

/tmp/ok/python26-sqlalchemy-0.6.6-1.noarch.rpm written

This script uses Distutils2 to browse PyPI and sort versions, then uses a custom version of bdist_rpm to invoke rpmbuild. This script can also create a RPM out of an existing project, by using an existing .spec file. It bypasses setup.py.

But this is just a quick script I've created for my needs. In the long term, I want to rename it to Py2RPM and make it the replacer of bdist_rpm. The features I want it to provide are:
- A standalone script that can be used to create RPMs, by using a configurable command-line call - A standalone script that can be used to generate a spec file out of a setup.cfg file - A way to read all options from setup.cfg, and to avoid to duplicate metadata fields like name, version etc - A way to be aware of the differences between the various RPM-based distros. - A feature to call via RPC the various CentOS, Fedora, etc repositories, to check if the name is already taken, if some files conflicts, etc - A distutils2 command, so the script can be called from the top-level pysetup script if wanted.

On a side note (but this is another blog post), setup.cfg is being spec-ed out and the spec will be versioned. This will allow other tools from the RPM world, {put a packaging system name here}, to consume it and provide similar features. And this file will soon be available at PyPI so we can work with it without downloading the tarball of the release.

If you're interested into Py2RPM, please get in touch with me. I am not a RPM expert at all, and I'd be happy to have more people involved.

Mozilla Services / Week 10/11

Tarek Ziadé — Wed, 23 Mar 2011 15:09:00 +0100

What's this ? read this post.

What happened

Well.. Firefox 4 is out and the number of downloads is impressive. Check this page: http://glow.mozilla.org/. It's almost 7 millions right now !

Back to my Services dev topics :D

It's been a while since I've posted a bi-weekly status update, because I was busy traveling to Pycon, then to Vancouver to work with the messaging team on F1. If you've never heard of it, check Bryan's post. Then go there, install the Add-on and enjoy it.

The work there was quite interesting as we've started to think about how to scale F1 so it can be used by millions of users. On server-side --the part I am the most interested in-- they have built a Pylons application that acts like a oauth proxy, used by the Add-On to send tweets, e-mails etc to third-party services like Twitter, GMail or Facebook.

The choice of Pylons made sense back when the project started because there was a database that stored some user data, and a few panels that are displayed when you want to manage your settings. But that database has been removed from the server and the code has shrinked to a few static pages and an oauth lib used to communicate with the various services.

One thing we will do is to move to a lighter web application and just use WebOb and Routes. Pylons became overkill for that application, and has now joined the cemetery of deprecated frameworks (Pyramid is the new thing).

On the scaling part, we're still working hard to come up with the best design, and we're trying to keep this wiki page up-to-date with that work: https://wiki.mozilla.org/Services/F1/Server/Architecture. Note that this is a work in progress, so nothing is settled in stone. The most interesting part is to decide if we do a synchronous architecture or asynchronous one. I personally think a synchronous architecture is simpler to start with, since we already have a working application that does what we want, and since we can always move to an async model later.

Another interesting part I am working on is how to do functional tests against our application, knowing that most APIs will call in turn a third-party server like Twitter or GMail. We need to mock those but in the meantime find a smart way to replay real sessions. I've worked on this topic in the past and blogged about it. I had created an initial version of a small mock tool but not really used it until this week.

Mark Hammond has been working on the topic as well and we need to synchronize our efforts next, but the general idea we've discussed and we'll use is to record a session that occurs with the real servers, dump it in a file, then allow the tests to reuse the recorded session instead of calling the real server again. This also needs to be done through a real TCP call to a third party server, and not mocked in Python, for a realistic behavior.

My first take on this is a simple proxy called recproxy which is a new version of my previous proxy. You can use it like this:
class TestSendController(unittest.TestCase):

    @recproxy('https', 'twitter.com', 'test_send.rec')

    def test_send(self):

       ....

This proxy does the following:
- A proxy server is launched on the localhost - Any call via urrlib2 which destination is https://twitter.com/xxxx is intercepted and sent to the local proxy - The local proxy reads the test_send.rec file that contains request/response pairs. If a request in that file corresponds to the request, the response is returned.

Of course, this means that the test_send.rec has to be created first. To do it, all you have to do is to run the tests with a PROXY environment variable set to 1. This will call the proxy server, but instead of returning values found in the file, it will proxy the requests to the real server and record the responses into the file.

Since we want to share the recorded files and obfuscate sensitive data, you can replace those data by XXX in the file. The proxy will match the incoming requests using a regular expression that will replace XXX by anything.

Here's an example of such a rec file:
POST /statuses/update.json HTTP/1.1

Accept-Encoding: identity

Connection: close

Content-Length: XXX

Content-Type: application/x-www-form-urlencoded

Host: localhost:65535

User-Agent: Python-urllib/2.6

oauth_consumer_key=XXX&oauth_nonce=XXX&oauth_signature_method=HMAC-SHA1&oauth_timestamp=XXX&oauth_token=XXX&oauth_version=1.0&status=%20http%3A%2F%2Fwp.me%2FpgWjI-7X&oauth_signature=XXX

====

200 OK

Content-Type: application/json; charset=UTF-8

Content-Length: 11

{"id": 123}

=======

This is basically a mock of a successful status update. And since the proxy works as a test function decorator, you can create records for bad behaviors as well, when you are fixing a bug that happens when there's a unexpected answer from the third party server.

You can also mock several servers (like bit.ly and twitter) by simply adding more decorators to the function.

The recproxy code is here. It's still an early version and will probably evolve a lot.

What's planned

In the next weeks we will continue to work on the architecture for F1, and also add more tests with Mark & Shane.

I am looking forward to synchronize my testing work with what Mark has done. Also, I'll probably make the proxy tool a lib, because that's what we want to use in other projects.

State of packaging (post-Pycon)

Tarek Ziadé — Mon, 21 Mar 2011 12:11:00 +0100

Language Summit

The language summit this year was less focused on packaging since the work is ongoing in this front. But we still made a few important decisions:
- Distutils2 will be named "packaging" in the standard library and released with Python 3.3 - A backport for 2.4 to 3.2 will be provided and be called Distutils2

The reason for this is to avoid any name clash in the future when people will have to deal with the two versions.

Conferences

I gave a talk about Distutils2 and people seemed to like it. At the end I had to hide behind the desk because people asked questions I did not want to answer ;)

The video is here.

Questions

I have run a Google Moderator before Pycon so people could ask questions. Here are the answers.

Q: In your long term view, should distutils2 completely replace pip?
A: No. But Pip will probably move to a thiner layer on the top of packaging, to provide its specifics (requirements freeze etc)

Q: What's the future of "entry points" ?
A: packaging will allow adding extra custom metadata. So the plan is to create a third-party project that provides a similar feature than entry points.

Q: Can distutils grow better support for compiling shared modules that are accessed via ctypes? This doesn't work on Windows because the wrong symbols are exported
A: Distutils is the poor's man tool for compiling shared modules. The plan is to make it easier too hook third-party tools to do this. We will still provide a minimal support, but nothing much fancier than what we already have. Although, your ctype problem sounds like a bug to me, we could fix.

Q: In PEP 345, why is license optional? I understand there are many packages today that don't declare their license (which sucks!), but has there been any talk of changing that?
A: It's not really planned since people can simply add a licence text in their releases if they want. If it was mandatory what would be the default license ?

Q: How can we get dependency resolution in an *offline mode* without running a local index (aka web server) and/or find links (aka clunky solution)?
A: By building a local cache of the metadata information

Q: Will distutils2 manage the dependecies with packages or will left that to pip?
A: Distutils2 will manage them

Q: How about a standard binary distribution format, with metadata like dependencies, like eggs but without the bad bits?"
A: What about the existing one ? (bdist)

Q: Is there a mechanism to provide more information about C-extensions, specifically, which are needed and which are cPython specific accelerators.
A: You can specify flags or environment variables, but nothing has changed here compared to the previous version

Q: Considering Python's "batteries included" philosophy, why should distutils2 not include pip and virtualenv in its scope?
A: Virtualenv is being added under the pythonv name --work in progress--. Pip2 will stay a third party project and just provides what Distutils2 does not in the future: developer tools that should not be placed in the stdlib.

Q: There are only one question - when? Python 3.3, I hope?"
A: Yes 3.3. The merge is imminent.

Sprints

We worked on porting Distutils2 into the stdlib, and it's almost ready here: https://bitbucket.org/tarek/cpython/ The merge is imminent !

Other interesting features we're adding:
- People will be able to add extra metadata in their projects. They will be published at PyPI and also installed and queriable. - setup.cfg will have its own specification and will be versioned. It will be published at PyPI as well so people can get all kind of information about project without downloading the tarball ! Also, other tools like Bento will be able to read the cfg file and use it to provide their build features. - lots of other stuff I have to remember -- I will add them here later.

Pycon 2011 - Ask questions on packaging

Tarek Ziadé — Thu, 10 Mar 2011 14:03:00 +0100

30 minutes is very short -- that's the length of my Python talk about packaging.

In order to give a chance to people that could not make it to Pycon, or for those who won't be able to ask them during the talk, I have created a Moderator here: http://bit.ly/pycon2010

Add questions there and vote for them -- I'll answer to them at the end of the talk if I have spare time, or in my blog.

Thoughts on Twitter noise

Tarek Ziadé — Thu, 24 Feb 2011 09:42:00 +0100

I was following a little more than 100 people on Twitter, and I realized I was mainly following my own technical tribe. By tribe I mean people gravitating around Python core, and since a few months a few Mozillians.

Well, what happens is that most tweets that are sent by people I follow are just circling around, and I get a lot of noise. And I am evolving in some kind of closed circle of information.

For Example, if the PSF account sends a tweet. It's retweeted by Jesse, Voidspace etc.. The most interesting Mozilla news will be retweeted by people like Chris Beard, or Alex Limi.

So why do I have (and we all have it seems) so many followers that are like me ? I think that's related to the fact that people follow friends or acquaintances they meet in conferences etc. We use Twitter a little bit like Facebook in this respect.

But I don't want to do this anymore, I think Twitter has much more value than just listing people you know or you like. I want to cut the noise there and try to get more value out of it.

Reducing my own noise

It turns out I am what Klout calls a Specialist (see here). But I am redundant. I mean, look at this:
1. Terry sends a mail at Python-dev about the fact that Python 3.2.0 was released twenty years after 0.9 2. I tweet it 3. Michael tweets it right after 4. Many other tweet it in turn.

So what it really means is : does it make sense to duplicate on Twitter news related to Python when they are already published in other medium ? I guess it made sense for this one, since it brings to a larger circle of Python fans this news. Guido himself tweeted it.

And I like to tweet those things -- And I have more than 1k followers that seem to be mainly interested in Python, so they want those tweets and consider that I am a specialized reporter.

One way to cut the noise on my side would be to check if someone has already talked about it, and eventually retweet it, like when you try send a link in Reddit and it tells you someone has already done it.

Or maybe just become more specialized -- stop talking about wine or random Python news I know they will be tweeted eventually. And also, maybe limit myself to 5 tweets a day.

Reducing other people noise

In order to cut the noise, I need to keep only a few "Specialists", for every domain I am interested in. I could probably just follow voidspace to get the latest news from the Python world.

The real value though, would be to follow other people that talk about other topics than Python or Mozilla. That does not necessarily means I'll follow random people, but rather people that are partially sharing my interests and opening a window to other worlds at the same time. People that are in my tribes, but also part of other tribes. That's a little bit what Malcom Gladwell calls connectors I guess.

For example, ogrisel is part of my Python user group but really focuses on AI. I enjoy his updates because they are food for thoughts. I also enjoy zooko updates because he has one foot in Python and one foot in security.

So, I guess the most logical thing for me to do with Twitter would be to:
- stop following Python or Mozilla people, except a very few "Specialists" - follow more "Connectors" like ogrisel

I am already down to 40 people I follow and will probably cut it down in half.

I already feel better today, my Tweetdeck backlog was greatly reduced. I wanted to tweet something about the weather but I held it this time ;)

QA portal ideas

Tarek Ziadé — Mon, 21 Feb 2011 11:25:00 +0100

GSOC is coming back. We'll soon have to select some projects for CPython, and I'll probably suggest a few Distutils2 topics -- it's an endless project ;).

There's a project I want to start, a QA portal. I am not sure how this could fit in a GSOC proposal, but I think there could be some interest in the community.

I could definitely use such a tool for our Mozilla Services projects.

But let's face it: I have no time, and this is far too ambitious. But let me dream a bit...

Level 1 - PyPI Watcher

For level 1, the idea is to create a portal that subscribes to all PyPI events (registering or uploading of new releases) and creates a QA report on specific projects I want to watch. The idea is not new, and we had a student on this last year -- But the project was not finished.

Basically I want a website like PyPants, but with more ambitious features. Maybe a good idea would be to approach the PyPants creators and see if the project could be extended.

Some tests I'd like the portal to perform:
- run a real install of the project in a VM, and returns a report on how the system was impacted with a tool like SystemTap. The VM part is mainly to avoid any security issue when running third-party code. While I supposedly trust the projects I select, you never know. - try to detect and run the tests, along with a coverage report. - run the usual metrics (pep8, maccabe, pylint, etc)

Difficulties:
- The VM part is not that easy to set up and scale, but feasible with Amazon for instance - What about Windows ? -- dropping its support for now seems a good idea :D

Level 2 - Clone detection

The next stage I would like to see the project take is clone detection. Tools like CloneDigger allows to detect similarities in code. It will tell you for instance that function A is very similar to function B.

What I would want to see is an global index of clones. The users select a project as being the master project, and a list of dependencies or other projects. Then the portal will report any similar functions, methods or modules it found.

Difficulties:
- Set up the right thresholds in the algorithm that detects the clones. There could be a lot of back and forth here before the results make any sense. - Scaling it. For instance, while it would be easy with CloneDigger to keep for each project the cluster of statements (see doc), so you don't have to rebuild everything every time, we are still doing a comparison work that grows at a logarithmic speed. I am not sure how to deal with it but that's an interesting issue to solve.

Level 3 - Hudson Jenkins integration

Once we have a shiny Level 1+2 QA Portal, I'd like to see it integrated in Hudson somehow.

One idea would be to create two web services in the QA Portal, a Hudson plugin can call. One to ask for a fresh report generation, and one to get the result and display it in a dashboard etc.

I am not sure how Hudson deals with running tasks asynchronously though, but I am pretty sure this is doable.

Ok let's stop dreaming -- back to work ;)

Do not upload dev releases at PyPI

Tarek Ziadé — Tue, 15 Feb 2011 17:15:00 +0100

The discussion we had on the SQLAlchemy mailing list, triggered by the release of 0.7.1b, made me realize that we need a heads-up on this problem. EDIT: a similar problem occurred for psycopg2 today.

Until our packaging ecosystem knows how to handle properly development releases, the best practice for mature projects is to avoid publishing anything that is not a final release at PyPI (or any download link that points to a development release)

By development releases, as opposed to final releases, I include:
- any release that is a snapshot of the trunk or tip of the project - any alpha, beta, or release candidate

And by mature project, I mean any project that already published one stable release at PyPI.

The reason is that our current set of packaging tools do not know how to make a difference between a final release and a development release.

Setuptools' easy_install script will scan the simple index page, order the releases number via its version sorting algorithm, then take the "latest" version.

So calling:
$ easy_install Foo

will install the latest uploaded release for the Foo project, even if it's a development version. If you use Setuptools' install_requires option in your setup.py, the same thing will happen.

To prevent it, you can tell the tool which version you want, and even provide a complex condition, like:
$ easy_install SQLAlchemy >=0.5, <0.7

But there's no way to tell it to get the latest final.

zc.buildout is the only tool in my knowledge that prevents this, with the prefer-final option.

How Distutils2 and PEP 386 fixes the problem

But uploading release candidates is a great way to get feedback from the community, and it's currently frustrating not to be able to push development versions at PyPI. Because depending on the installers our users will use, they might be unable to control if they want to opt in using non final releases.

And, well, what's a final release ? what's a beta ? Are we sure all tools agree on the versionning schemes ?

How can I make sure my beta version will be recognized as a beta release by all tools out there ?

PEP 386 solves this by defining a version scheme.

And guess what:
- The new Metadata 1.2 - PEP 345, implemented in Distutils2, recognizes only PEP 386 versions - PyPI will reject any project that uploads metadata 1.2 with a non-PEP 386 version. And this is already activated at PyPI. - The Distutils2 installer recognizes development releases and let you decide which one to pick.

Granted, it's going to take a while before all installers use the new standards, and all projects out there make the jump to Metadata 1.2, but at least we know the problem and implemented the solution..

AndPycon should be an important milestone since the first beta of Distutils2 will be released at... PyPI and will be usable by any project.

Mozilla at Pycon

Tarek Ziadé — Mon, 14 Feb 2011 15:52:00 +0100

Mozilla folks will be giving a few talks at Pycon US 2011, the biggest Python conference. That's happening in one month in Atlanta.
- Kumar McMillan on [Supporting all version of Python All the Time with Tox][] - Ian Bicking on Javascript for People who know Python. - Me on Firefox Sync and [Packaging, from Distutils1 to Distutils2][] - Any talk that I've missed (please let me know)

Brian asked me a few questions about my talks on the Pycon Blog if you want to have more details on them.

If you like Python or Mozilla, you should definitely consider going to Pycon.

And hey, Mozilla is a proud Lanyard sponsor of Pycon this year !

A light Description Language for REST web services ?

Tarek Ziadé — Mon, 14 Feb 2011 01:07:00 +0100

tldr: I have this idea that comes back and forth in my head since a few months now, working on the Sync, Easy Setup and Identity servers. I want to create a light DL to describe what REST web services an application implements, and use it to automate the request dispatching but also the documenting and testing of the services.

When we create web services, there's a lot of things we do that are quite systematic:
1. Describe the web services in a document. 2. Define what code should be executed. 3. Validate and transform incoming requests and outgoing responses. 4. Run functional and security tests on the web services.

1. Describe the web services (=API)

Documenting the web services the application needs to offer is the first step to build it. A document needs to list the URL paths, the methods to use, what goes it and comes out.

For the Easy Setup server (a.k.a the J-Pake server), I've documented an initial design of the web services here: [https://wiki.mozilla.org/Services/Sync/SyncKey/J-PAKE#Server_API][] and we worked with Stefan and Philipp to refine the design iteratively.

Once the application is built, that's a non implementation-specific description that anyone can use to build a new client that interacts with our server.

2. What code should be executed ?

The next step is to build the code that does the job. On a request, we need to define what piece of code should be called to build a response and execute it in order to return a response.

It can be a set of regexps managed by dispatcher tool, which output -- That's Routes. It can also be a simple function decorator -- That's Bottle. Some other approaches are consisting of using the code namespace to dispatch the request, like implementing the Root.get_index method for a GET on the /index url.

At Services, we use Routes and feed it with a list that contains a description of the URLs. See this example: [http://hg.mozilla.org/services/server-storage/file/78762deede5d/syncstorage/wsgiapp.py#l65][].

3. Validate & Filter Requests and Responses

Beside the feature code itself, the application usually do a series of validation or/and transformation on the incoming request -- like checking its headers, or extracting objects from a JSON body. This can also be done on the outgoing responses.

Those steps are usually generic enough to be reused for all web services. And those steps, most of the time, should not be implemented as WSGI Middlewares -- one simple question to ask yourself to know if you should create /use a middleware: does your application breaks if you remove that middleware ? If so, it means that your application may not work without the middleware, thus it should become a library on which the application has a hard dependency.

Some Examples:
- an authorization function that checks the Authorization header and set in the execution context a user name. - a function that controls that the body contains parseable JSON, and unserialize it in the execution context. - a function that serialize all response bodies into JSON

4. Create functional and security tests

Like I said in 1., building a new client against the server should be possible simply by reading the documentation.

That's how we build our functional test suite, which runs for each web service a series of requests, and checks that the responses are the ones expected. These tests are simply validating that the server acts as documented.

Depending on the security expectations, another series of tests can check for the server behavior when it receives unexpected responses.

A light, implementation-specific, DL

So, the idea would be to describe everything in a static file, that can be used:
- by the application to:
- automatically dispatch the requests to some callable - execute some functions before and after the main callable, to transform and filter data

by the testers to use a generic HTTP client powered by the DL file.
by the documentation to generate an HTML or Wiki version of that DL file.

After investigation, I found WADL and got scared. That's probably the XML effect :D

WADL is very close to what I am looking for, see an example. But while less complex than WSDL, it still seems a bit overkill for what I want to do. I am not sure for instance, that I want to fully describe the structure of the responses. And well, WADL is only documenting the web services, and not pointing the code that implements them.

I want to do both things in the DL, and keep implementation-specific parts light enough that they will not really annoy anyone. But they're still useful when your not in the application itself: for example, a wiki output could link to an online view of the code that's used.

Here's a (very quick) draft in pseudo-YAML out of my head:
POST "/the/webservice/is/here":

  id: cool_service

  description: The cool service does this and that.

  request:

    headers:

      Authorization: description of the supported token etc.

    body: explains here what the body should contains

  response:

    body: explains what the body contains

    headers:

       Header-1: description of that header

    content-type: application/json

    codes:

      200: explains here what getting a success means

      503: explains here when you might get a 503

      400: explains here when it's a bad request

  implemented_by: module.class.method,

  pre-hooks:

      authenticate

      extract_json_from_body

  post-hook:

      jsonify

The first part is really, documenting your web service.

But it's also detailed enough to be able to automatically create an HTTP client that can be the basis for tests. e.g.:
def post_cool_service(body, authorization):

    """The cool service does this and that.

   Arguments:

       Authorization: description of the supported token etc.

       body: explains here what the body should contains

   Returns:  code, body, headers

    """

    ... generic curl-y code...

Not sure about the generative aspect though, because it's hard to maintain. Maybe a dynamic introspection of the DL file is a better idea here...

The second part tells the server what code should be run, something that would be similar than:
def cool_service(request):

    """The cool service"""

   authenticate(request)

   extract_json_from_body(request)

   response = module.class.method(request)

   return jsonify(response)

In other word, I could strip all the boiler-plate code I have around the code that implements the features themselves, and just combine them with a few helper functions via the DL.

This approach is quite similar to Zope's ZCML glue, but without the ZCA layer --which I tend to find heavy and overkill--

http://hg.mozilla.org/services/server-storage/file/78762deede5d/syncstorage/wsgiapp.py#l65

Flake8 now Hg friendly

Tarek Ziadé — Sat, 12 Feb 2011 13:40:00 +0100

I apologize in advance to the authors of pep8 and Pyflakes, as Flake8 is a horrible hack on the top of those two tools. But it's really what we needed at Mozilla Services to check that anything that lands into our Python repositories follow these simple rules:
- PEP 8 compliant - No unused imports - No undefined names

So Flake8 is a simple script that calls pep8 and pyflakes and merges the output, so you get all warnings grouped by files, instead of having to run both tools one after the other. I've stripped all options for now and you just call the script over a single Python file, over a directory or by using a *.py glob-style pattern.

I added a few features on the top of it:
- Python files that starts with this header are skipped: # flake8: noqa - Lines that contains a # NOQA comment at the end will be skipped

In the 0.4 version I have added a Mercurial hook you can use to automatically control your code when you call the push or the qrefresh command.

Here's how you configure it in your .hgrc once Flake8 is installed:
[hooks]

commit = python:flake8.hg_hook

qrefresh = python:flake8.hg_hook

[flake8]

strict = 0

If strict option is set to 1, any warning will block the commit. When strict is set to 0, warnings are just displayed in the standard output. I use it in non-strict mode, so I just get a display of the issues my code has.

Stuff I might add in the future:
- A cleaner merge of pep8 and PyFlakes options - EDIT: Added in 0.5 ~~Acyclomatic complexity metric so I can print out "Hey what's up with all those loops ? flat is better than nested!"~~

A more ambitious feature would be to detect similar code patterns and warn the developer that the function he's adding looks a lot like function Foo in module Boo. This implies using a tool like CloneDigger, but also keeping in a local, centralized database, some kind of index of every piece of code that gets committed. This is mandatory to speed up the comparison work and avoid introducing huge lags as the code base grows. Mmmm that would be a good GSOC topic !

A simple self-upgrade build pattern

Tarek Ziadé — Thu, 10 Feb 2011 17:21:00 +0100

Bootstrapping is useful.

For example, we use it to build in-place our applications at Mozilla Services, to develop them. When the developer runs "make build", a Python script is invoked, and makes sure all the Python packages we are developing are checked out locally, and tied to the execution environment.

In the Python community you have similar tools like zc.buildout combined with mrdeveloper, or Pip's requirements. Although, our script does Mozilla-specific things, like allowing to update the dependencies at specific tags via command-line options, or at the "latest stable tag".

Anyways, back to bootstrapping.

Having such scripts is great, but maintaining them across all projects is painful. Since the script is the first thing the build runs, it cannot have an external dependency so it has to be completely independent. So you have to copy it around, in all your projects, and that leads to maintenance hell.

One solution is to add a self-upgrade feature to the script, so it tries to get the latest version online before it is run. zc.buildout has such a self-update feature, that will make sure it's at the latest version. Although, the implementation of this feature is a bit odd. 1/ you cannot run the bootstrap script if you're offline. 2/ on upgrades, zc.buildout restarts itself. I don't think 2/ a robust approach and I recall that it had some bugs under Windows in the past (updating files that your own program is using might fail).

So, since I was working in the tools-for-Mozilla-Services area, I added a self-upgrade feature and tried to isolate a simple mechanism for this. A pattern that would let us publish fresh versions of some scripts in our server and make it easy for our tools to use them.

An efficient bootstrap script should:
- be able to run offline, so the app can be built even if it's unable to check for the latest online version - be able to upgrade itself with a newer version - upgrade itself without having to restart itself - be completely independent -- besides a vanilla Python

The pattern I have used is as follows:
- The bootstrap script is an empty shell that is just responsible to update a second script that contains the real bootstrapping code, then dynamically import it via an __import__, and run it. - The server that publishes the real bootstrap script returns an ETag header with it, and supports the If-None-Match header. - The bootstrap script is able to ask the server if the online version is newer, by using the ETag value it has from the previous update in the If-None-Match header. If the online version is unchanged, the client receives a 412 code (Precondition failed).

The code has no dependencies besides Python.

Extract from the client-side code:
import os

import sys

import urllib2

def main():

    # getting the file age

    if os.path.exists('._build.etag'):

        with open('._build.etag') as f:

            current_etag = f.read().strip()

        headers = {'If-None-Match': current_etag}

    else:

        headers = {}

        current_etag = None

    request = urllib2.Request('http://moz.ziade.org/_build.py',

                              headers=headers)

    # checking the last version on our server

    try:

        url = urllib2.urlopen(request, timeout=5)

        etag = url.headers.get('ETag')

    except urllib2.HTTPError, e:

        if e.getcode() != 412:

            raise

        # we're up-to-date (precondition failed)

        etag = current_etag

    except urllib2.URLError:

        # timeout error

        etag = None

    if etag is not None and current_etag != etag:

        # we need to update our version

        _rename('_build.py')

        content = url.read()

        with open('_build.py', 'w') as f:

            f.write(content)

        with open('._build.etag', 'w') as f:

            f.write(etag)

    # we're good, let's import the file and run it

    mod = __import__('_build')

    project_name = sys.argv[1]

    deps = [dep.strip() for dep in sys.argv[2].split(',')]

    mod.main(project_name, deps)

if __name__ == '__main__':

    main()

Extract from the server-side code:
import os

import mimetypes

from hashlib import md5

from wsgiref.simple_server import make_server

# where the served files are located ?

_DIR = '.'

def application(environ, start_response):

    if_none_match = environ.get('HTTP_IF_NONE_MATCH', '')

    path = environ['PATH_INFO']

    path = path.lstrip('/')

    if path.startswith('.') or '/' in path:

        # looks like a bad path -- or a potential security thread

        status = '400 Bad Request'

        headers = [('Content-type', 'text/plain')]

        start_response(status, headers)

        return "Invalid Request"

    path = os.path.join(_DIR, path)

    if not os.path.exists(path):

        # unknown file

        status = '404 Not Found'

        headers = [('Content-type', 'text/plain')]

        start_response(status, headers)

        return "File not found"

    # we're good, let's look at the file age

    etag = '"%s"' % md5(str(os.stat(path).st_mtime)).hexdigest()

    if etag == if_none_match:

        # same file

        status = '412 Precondition failed'

        headers = [('Content-type', 'text/plain')]

        start_response(status, headers)

        return ''

    content_type = mimetypes.guess_type(path)

    status = '200 OK'

    headers = [('Content-type', content_type[0]),

               ('ETag', etag)]

    start_response(status, headers)

    with open(path) as f:

        content = f.read()

    return content

if __name__ == '__main__':

    httpd = make_server('', 5000, application)

    print "Listening on port 5000...."

    httpd.serve_forever()

The server is a plain Python wsgi script that is easy to hook in Apache or Nginx, to serve a directory of files. It should be safe enough as long as the served directory is isolated from the rest of the system. I did not try that hard to secure it, but it makes sure you can't leave the directory.

*EDIT: as noted in the comments, Nginx and Apache provides such ETag features, so you can just configure it to serve static files this way. I did a wsgi script as an initial prototype and to demonstrate what the server does, and I kept it because it's easy to run or hook into any web server. And I will probably add more features next there.*

Now the bootstrap script we use can live in its own repository, and get bug fixes and features, that will be automatically propagated to all our applications.

Do you solve this issue differently ? I am curious to get some feedback on this.

Distutils2 - Logilab Sprint Report

Tarek Ziadé — Mon, 07 Feb 2011 01:43:00 +0100

We had a great sprint at Logilab the other week-end, and this was by far the most productive sprint on Distutils ever. That's partially because all the core work we did in the last year starts to surface ! That's also because we had a good team of hackers, focused on specific tasks. I did not merge all the work yet but it should be done this week and an alpha released right after.

Now to the list of goodness..

Data files

We can now express data files in setup.cfg, and use a specific API to work with them. The goal is to let OS Packagers configure where those files should land at installation time, without breaking the code that uses them. The brainstorming we did last year at Pycon is here : http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst and it's now a reality. This is a good step torwards a cleaner packaging system.

Setup.cfg completion

setup.cfg lacked a few things to be able to fully replace a setup.py, like defining extensions to be compiled. This is now done and we're going to maintain a page were we list all projects that were successfully converted. The page lives here for now: [https://bitbucket.org/tarek/distutils2/wiki/Deployments_using_setup.cfg][] and I will announce it widely once the alpha is released, so people can start to convert their own projects. We did convert more projects, and I need to update that page later this week.

Installer / Uninstaller

A first version that installs a project and its dependencies, as well as an uninstaller, was completed. Among other things, it will rollback the installation in case it encounters any issue. This tool exists mainly as a implementation reference to exercise all the tools that are provided in Distutils2 to browse an installation, build a depgraph or query PyPI. There's still a bit of backward compatibility work to do in order to deal with Setuptools-based installations and projects, but most things are done. I expect this to be finished and included in the alpha.

From Distutils1 to Distutils2 and vice-versa

Since setup.cfg completely replaces setup.py -- which is ignored by Distutils2, you can make your project compatible with both. This implies that you have to maintain two files that contain duplicate information, like the project name or the version. We created a tool during the sprint to avoid this: a simple function that you can use in your setup.py file to convert setup.cfg options into Distutils1 options.

Of course, to avoid adding a Distutils2 dependency into that setup.py file, we inject that function into the module, and we'll provide a command in pysetup. to generate it:
$ pysetup createD1setup

setup.py successfully created.

And if you want to do it the other way around, we offer a new feature in the wizard that will run your setup.py file and generate a setup.cfg. This is the tool we'll promote in order to make it easy for people to add Distutils2 support.

Stuff left to do

Besides the merges, before I release the next alpha there's one big task I need to do: remove the fancygetopt.py file, a layer on the top of getopt, and add a simpler, isolated module to process the command line. This will clean up a lot the code base, and make the dist.py module stop dealing directly with this. It will only implement the Distribution class, and consume command line options without having to be the one that generates them.

I am not sure yet what I'll use, since both optparse and argparse are unable to meet my requirements here. A custom command-line parser is not the best thing to maintain though..

Thank you Logilab & Bearstech

Logilab is awesome. We had many people from their teams sprinting with us, and they paid our meals every day and covered Alexis' travel costs. Bearstech was awesome as well, as they covered several snacks expenses.

Thanks to all sprinters that made it there, or worked online. We should do these more often.

https://bitbucket.org/tarek/distutils2/wiki/Deployments_using_setup.cfg

Mozilla Services - Weeks 3-4

Tarek Ziadé — Fri, 04 Feb 2011 10:13:00 +0100

What's this ? Read this post.

What happened

The two last weeks where more about fixing small issues here and there and thinking about the tools we need to build good server-side applications. We finalized with Stefan the work on the Easy Setup protocol(Pake). The protocol is now able to work under flaky wifi condition. Typically, if your device is in a wifi environment that drops the connection often, instead of starting the transaction back again every time it drops. we've added a way to resume it.

The latest Firefox Home release includes this change, and the server is compatible with the hardened protocol as well as the previous one. That way it still works with older clients and with Firefox 4 itself, which do not have this improvement yet (yeah, Richard & Phillip are quite busy on fixing blockers, so they'll come to it later).

This flaky wifi issue, a few bugs we had with the Sync server when the LDAP or the MySQL server dropped, and the blog post about the Netflix Chaos Monkey gave me a idea of a project we should do at some point at Mozilla:

We need to make sure all our apps are robust when a back-end server (==ldap, memcached, mysql, etc) gets down, and in particular make sure:

all requests to our server return 503 and logs errors correctly, and nothing bad happens on client-side
once the back-end is back, the server should return to a normal behavior
the whole system should stay stable during the back-end blackout, and after

I've started to write down some ideas about this. Basically, I want each piece of our dev and stage servers infrastructure to get very slow or entirely down for a bit, at random times. Then, I want to observe how the application reacts during and after the event.

The Seism Project

There are many low-level tools that allow injecting delays and errors at the TCP level. Some of them can require BSD systems, like dummynet. Since Linux 2.4 TC can be used to do traffic shaping. But using those tools requires setting up a dedicated infrastructure/configuration every time we want to use such a tool for an app.

Since all our server pieces are TCP-based, another way of doing it is to write a TCP proxy that is able to manage the delays, shutdowns and errors, and to some extent simulate HTTP, LDAP or SQL specific issues when it makes sense.

I call this project "Seism"

The front server that is being tested can run one proxy per back-end and be configured to use them with no extra work.

One important feature is to log the behavior of the infrastructure when a perturbation happens, to see if the application behaves as expected.

One idea to perform this is to create yet another specialized proxy in front of the application to be tested to record what happens when a perturbation is simulated. The "back proxy" can send a signal to the "front proxy", describing what kind of perturbation it just did, and the front proxy can record how the application reacted by logging the response it sent (or the lack of response if the app dies).

The final log will tell us how the app is behaving depending on the perturbations. One nice feature would be a dashboard to display this. e.g. all green if the client receives 503s and 200s, red if it receives a 500 etc.

The behavior logger should also log the next N requests to check that the server is behaving normally after the perturbation. It can also log the CPU and Memory to check for stability.

I think this can be really useful to bullet-proof automatically every Server application we're building, in cunjunction with Hudson. A continuous integration system can run in a hostile, Seism-enabled infrastructure, and report failures.

Docs & Templates

Another thing I've been doing is starting to build a "Developer Guide" for the server-side, that would make it easy for people to get started in building server applications that are using our tools and environment.

We have this awesome tool in the Python community called Sphinx. It's used to document Python itself, but also tons of projects out there. It's based on a light text format (reStructuredText) and allow you to treat documents like code. In other words, everything you write is pushed in a repository, has an history, patches etc.. And Sphinx is able to generate a static HTML site out of it or a PDF.

So, the dev guide code is here: http://hg.mozilla.org/services/server-devguide/ and an HMTL rendering, update hourly, lives here: http://sync.ziade.org/doc/

It's an early draft, full of TODO and XXX. But you get the idea: I am trying to document everything a developer that wants to work with our code should know.

And one important thing I have started there is to build Applications Templates, so people that want to build a new app on the top of our tools, an app that follows our standards, can get started in seconds.

What's planned

So, the focus in the upcoming weeks for me is to make sure the Python Sync server gets into production, and second level tasks include writing more doc and polishing our server-side tools.

Data files in Distutils2

Tarek Ziadé — Sun, 23 Jan 2011 17:00:00 +0100

I am preparing next week's sprint at Logilab, trying to isolate several interesting tasks people will be able to achieve without having to dig in the whole code base.

One task we need to achieve in Distutils2 is a better description of the data files in a project, and in particular a way to categorize them so an OS packager knows how to make a difference between a documentation file and a configuration file. Also, we need to let the packager decide where these files need to land in the system at installation time, without breaking the developer's code.

So far in Python, people tend to include all files within their Python packages and access them using the __file__ variable:
import os.path

here = os.path.dirname(__file__)

images = os.path.join(here, 'images_dir')

This technique has its limits:
- If you use easy_install to install your projects, you might end up with a zipped egg, meaning that the directory is in a zip file. So this code will not work unless you've installed the project with the -Z option, or you use the zipimport helper to work with your files. - Installing data files in site-packages is not a best practice in most Linux distributions, and if the OS packager wants to move them in the "right" place, he needs to review all the code and patch it everywhere it makes an assumption on the location of the file. - If the developer does the right thing for Debuntu with his data files, he'd need to do it for every other Linux distribution, but also define a specific behaviour for Mac OS X and Windows.

Last year at Pycon we've worked on that topic and came up with an elegant design on the paper:
- Python will provide in the sysconfig module a list of default categories for data files, with a default location for each target system - The developer will define its data files in categories in setup.cfg, and add if needed new categories - Linux distributions will be able to change the paths for each categories via a configuration file global to Python -- sysconfig.cfg - Distutils2 will look at the paths of each categories at installation time, and install the files there. - The developer will use a special open() API to be able to locate the file no matter on what system the code runs, or even if the code runs on a development tree.

The draft of this feature is here: http://hg.python.org/distutils2/file/tip/docs/design/wiki.rst and starting to code this design for real is a perfect task for this sprint because it does not require a lot of knowledge about the Distutils2 code base.

If you are in Paris, or available online, and want to sprint with us, make sure you register to the wiki page !

"Sprint@Logilab"

The UI of my dreams, part 2

Tarek Ziadé — Sun, 23 Jan 2011 13:58:00 +0100

I had a lot of feedback on my previous blog post about the UI of my dreams.

Some people said that a tiling windows manager, like xmonad would solve this problem if every application like Firefox, Thunderbird, or the Terminal was managed like a regular window.

So I gave xmonad a shot during a week.

I really liked it ! It basically does with all windows what I am already doing within VIM in split mode: when a new window is open, it's added vertically or horizontally besides an existing window. It's easy to move the focus from one window to the other, and adapt the size of each window. The whole screen is effective and you don't waste time anymore by rearranging your windows with the mouse. And you can have up to nine desktops (==workspaces). Granted, you cannot save their states, but it seems that this will be the case in the future.

So the question is: can xmonad across 9 desktops replace Panorama ?

Greg commented by blog post:

To me, Panorama groups are equivalent to other workspaces; and tabs in other groups are equivalent to windows/documents on other workspaces.

Gnome are working on making the contents of workspaces persist, which matches your use of tab groups as persistent identifiers.

Scale (and presumably Exposé) can be used to show just “this” workspace, and it would be this mode you’d normally use to see just the tabs/documents you’re interested in right now.

But my own usage of tabs goes beyond this. For one workspace -- understand one working context --, I can have more tabs opened than the screen can fit. I don't want to have them all open in my screen. Tabs are more like active bookmarks.

For example, when I am working I can have:
- one vim session that displays 2 windows: the code and its test - sometime a second vim session for another pending work - two terminals, one to run the tests and one to manage my mercurial queues - up to 10 opened tabs in Firefox. Bugzilla bugs, technical doc, etc. - some Thunderbird tabs

By using xmonad, I can organize the screen to have one vim session, and one or two terminals to code. But as soon as I need to read documentation or bug descriptions, I need to display Firefox and browse some tabs.

And I don't want to use another Desktop for this. I have one for my personal stuff like managing my servers, I have one for Python development, one for organizing my trips, one for music etc. And they all use Firefox, VIM sessions, Terminals etc.

So switching from coding to reading when I work on something, should be done within the same desktop. In other words, I could use something like Panorama to display different organizations for the same desktop. I could organize windows in Panorama categories like "coding" and "reading" and define a desktop as work and another one a travels.

So the UI of my dream could be:
- xmonad or any tiling windows manager - One Panorama-like tool instance per workspace. - A global way to configure shortcuts. (VIM mode for me)

"Previous blog post"

The UI of my dreams : Firefox / Terminal / Thunderbird unified

Tarek Ziadé — Fri, 14 Jan 2011 11:00:00 +0100

I work in three environments the whole day:
- Firefox (now mostly Minefield) - Linux terminals - Thunderbird

[gallery]

Yeah you can tell I am not good at screenshots ;)

I switch from one screen to the other using <Alt+Tab> and the underlying Operating System does not really matter. While I am under Ubuntu these days on my MacBook, I'd say that if I had a Bash or ZSH under Windows or Mac OSX, I would not care that much about the OS -- Well I do because Linux offers the best environment when you develop server apps but that's not the point--. In the shell, 90% of my windows are displaying a Vim and most of the time Vim displays itself several files in split mode.

Every time I switch the context, I have this tiny ergonomic fracture in my head. I try desperately my Vim shortcuts in Firefox. Thanks to Vimperator, I was able to reduce this fracture and I'd love to see something similar in Thunderbird.

But there's still something Firefox has, I wish I'd find in all my contexts. Everyone of them has Tabs - But Firefox has ~~TabCandy~~ Panorama, which let you organize tabs in space. I'd also like to "Pin as App"some of my terminals, or do something similar in Thunderbird.

So the UI of my dreams would consist of having the same behavior for these three environments:
- Panorama - App Tabs - Same shortcuts everywhere (Vim mode for me ;)) - Eventually, have a Fusion mode where all tabs from all contexts are at the same level

I don't think it's that crazy and I think that's where we tend to go to in the future. So, who's in for a Desktop Panorama ?

Mozilla Services - Weeks 01-02

Tarek Ziadé — Tue, 11 Jan 2011 17:41:00 +0100

What's this ? read this post

I did not post a weekly update last week because things slowed down during vacations. I've also decided:
- to change the title to a less geeky one, that is easy to understand :D - to publish the post every two weeks instead of every week. It seems to be a more natural pace. We'll see

What happened

My talk at Pycon about Firefox Sync was accepted ! I've added a Wiki page containing the outline so you can help me if you see some missing things or want me to talk about - https://wiki.mozilla.org/Services/Sync/Pycon2010

Hudson now runs all kinds of tests over the Sync and Pake servers. The functional tests are running on a real Sync infrastructure, and that helped me find bugs that are happening only when the server has been running for a while. I could fix for instance a bug that was filling the pool of LDAP connectors without freeing them properly. We'll work on having such CI setups for every Services projects in the future.

The RPM-based release process for all the Python apps is working like a charm and I need to take it to the next stage. We will set up a in-house PyPI proxy to avoid calling pypi.python.org on every build. We will probably use collective.eggproxy. That'll be helpfull for the tests as well.

I also started to work with Stefan on hardening the J-Pake protocol for the Sync easy setup. The goal is to make sure it works well when the device is in a flaky network/wifi environment.

What's planned

I'll focus on finishing small bits in the Sync Server and make sure the Pake server meets all the requirements for the grand Firefox 4 release. Beside the work with Stefan, we need to add more logs in the server to make it easier to see what going on: successfull transaction, failures with reasons etc. I'll work with Richard on this.

Globally, the next two weeks are about "serrer les vis" :D

Distutils2 sprint !

Tarek Ziadé — Tue, 11 Jan 2011 09:24:00 +0100

Thanks to Logilab, our host for this, we will have a Distutils2 sprint in Paris from the 27th to the 30st January. I won't be sprinting the 27th but I will be, starting the 28th until the end.

The sprint will focus on:
- the install/remove scripts - polishing Distutils2 for the next alpha release - many other things...

I'll do a rehearsal of my Pycon 2011 talk, as an introduction to packaging and Distutils2. The sprint is open to anyone, but their will be no introduction to Python, TDD or sprinting and we expect participants to know those.

If you are interested, add your name here: http://wiki.python.org/moin/Distutils/SprintParis. You can sprint with us online if you can't make it to Paris. If you want to do so, add you name with that precision (online).

2010 recap, 2011 plans

Tarek Ziadé — Sun, 02 Jan 2011 11:53:00 +0100

Instead of the weekly recap I've been doing lately, I will do a year recap and list my plans for 2011. Happy new year to all of you !

tldr: 2010 was quite an important year for me. The most important event was Suki of course, six months ago ! In Python I am still working on packaging matters and I've joined Mozilla 6 months ago, working in the Services team. 2011 should be the Year of Packaging and more fun at Mozilla !

2010 recap

January: My work on PEP 386 and 345 is waiting for the final approval from Guido and I am back at work on PEP 376 and Distribute. PEP work involves politics and time. It also means making sure the solutions are good enough for a smooth evolution of the existing ecosystem. It was frustrating at first but after a while I got used to it. A PEP will eventually gets accepted when there's a clear need for it.

In February I gave a talk at PyCon and sprinted there. My talk was well received (it made the top #5 of all talks at PyCon with the token voting system). I think it was both because I was passionate about it, and because part of the audience followed the work we did in the last months. Thelanguage summit was held there as well, and was both frustrating and great. Frustrating because I was told to revert all changes to Distutils and do the work outside the stdlib for a while. That was the right decision but was a bit hard to accept at first. Great because Packaging was yet one of the main focus of the summit. In February, some of my PEPs were accepted, which was a very important milestone.

In March we started to develop Distutils2, which was basically the trunk of Distutils right before the Big Revert. Yannick and other fine folks from Montreal became regular contributors to the project. I've also started in March to think hard about my future, and found out I was really happy working remotely. I also started to focus on GSOC and build proposals for Distutils2 and other topics.

In April I wasnominated at the PSF. I also started to get some interest in Mozilla and in particular Sync (formely Weave). Beta versions of Distutils2 started to hit the shelve. I also added new features in shutil. Next, PEP 376 was accepted. That's the master piece for packaging interoperability ! Last, my GSOC proposal was accepted and we got no less than 5 students for Distutils2 alone !

In May, I tried and failed to merge Pip and Distutils2 efforts ;)

In June, Suki was born and I got a new job at Mozilla. Those events considerably slowed down my work on Packaging and my blogging of course.

The GSOC ended in August. We managed to keep a few contributors, which is the whole point.

In September I worked hard on Firefox Sync, and we hadour annual sprint at my house.

In October, while everything which is not related to work was slowed down because of the new-born baby, I was still finding a bit of time to work on Distutils2. But compared to Q1 and Q2, the activity on the project was very low. I was just hoping then that it wouldn't kill the momentum on Packaging.

November and December were mainly focused on work. See my weekly reports.

2011 plans

2011 will be the Year of Packaging, I am telling you ! Pycon will be a milestone because Distutils2 should be in a state where it's pretty much usable. The next tasks will be to promote it and try to gain contributors. I am also planning move it back to the stdlib once Python 3.2 final is out.

2011 will also be the Year of Python for Mozilla Services !

Firefox Sync will be running using my Python implementation in Q1, and Firefox 4 betas are already using the J-Pake server I've designed with the help of my favorite client developers Stefan and Phillip. I wrote it using the same stack than Sync. Other projects written in Python are coming in the pipe and we are building a set of reusable tools for this purpose.

I'll also try to pursue the MoPy (Mozilla in Python) effort. That is, sharing Python knowledge, tools etc between Mozilla projects.

All in all, the work I am doing in packaging for Python is helping us a lot at Mozilla: Distutils2 tools are now used to automatically create RPM releases of all our dependencies by fetching them at PyPI. I hope my Pycon talk about Firefox Sync will be accepted so I can explain how I built it !

$ rsync mozilla/services community #week 49

Tarek Ziadé — Fri, 10 Dec 2010 20:11:00 +0100

What's this ? read this post

Week 49

Easy Setup on new devices: I worked on doing distributed tests and blogged about it. Turns out using gevent does not speed up the application because it uses a memcached lib that works with sockets and it looks like it's locking Gevent. So I'll just use the regular worker and change some of the design later to be able to use several workers.

Firefox Sync in Python: Integrated some reviews from Ian and did some minor cleanup.

OpenID / Identity: I passed the project to JR, our new team member. (Welcome ;))

QA: I set up a Hudson server that runs the tests for the Sync and the Pake server and also builds the RPMs. This way the packages are continuously built and any error related to that part is also detected.

Next Week Plans

This week-end and next week, I plan to:
- Write some script to automate the deployment of the RPM built by Hudson - Start some documentation - Have an intensive work week in Moutain View - Prepare the 1/2 Python gathering we will have Tuesday there. - etc.

Funkload + Fabric = quick and dirty distributed load system

Tarek Ziadé — Thu, 09 Dec 2010 00:05:00 +0100

We're currently using The Grinder for our Sync load tests at Mozilla Services. Grinder will let you write the tests using Jython and run a distributed load test. Despite the Java and Jython environment layer which is always painful to set up, Grinder works well.

But do not expect it to produce nice reports, you're on your own for this. You need to work on the raw results to produce nice diagrams. Do not expect either to perform some operations on the test nodes before and after the distributed load test is executed. Tests are driven via an UI and while it's probably possible to script them via Java... it's Java. And well, the feedback when the tests are running is nearly inexistant.

The distributed load system of my dreams must:
- let me run arbitrary commands on every node before and after it runs the test. And retrieve results then merge them in a consolidated result. - let me write complex unittest-based functional tests that I can also use to check the health of a server. - provide basic reporting features - be written in Python ;)

Oh.. hold on.. I have it : Funkload and Fabric.

Funkload let you write complex functional tests, does load testing with complex cycles and produces nice reports.

And then there's Fabric, which allows you to run commands via SSH on a pool of servers.

Funkload is not distributed but it's fairly simple to drive it via Fabric. And when all nodes have finished the execution --whom you can watch via the Fabric stdout-- you can get them back and merge them in a single file.

So basically, the fabric script looks like this:

from fabric.api import run, get, env

def runtest():

    run('fl-run-bench testmodule TestClass.test_function')

    file = '/path/to/results.xml'

    get(file, env.host_string + '-results.xml')

This will run the command on the distant server, then download the xml file. All through SSH. It can be launched on every server with:

$ fab -H the.server.org runtest

Fabric does not provide parallel execution yet, so you have to create a batch to run the script on every server in parallel processes. But this feature should be included in Fabric 1.1. There's a branch for this: https://github.com/goosemo/fabric/commits/multiprocessing

Then you can merge the files in a single XML file like this:

from funkload.MergeResultFiles import MergeResultFiles

import os

def merge_results():

    files = []

    for file_ in os.listdir(HERE):

        if not file_.endswith('-results.xml'):

            continue

        files.append(file_)

    MergeResultFiles(files, 'results.xml')

And finally, create an HTML output:

$ bin/fl-build-report --html -o html results.xml

Granted, there's no control of the nodes during the tests, that feature would require a small TCP server that drives Funkload on each server. And that would obsoletes the need for Fabric I guess. But for setting up a quick distributed test, I am good with SSH for now.

I should also mention Benchmaster, which do something similar but seems complex to configure compared to a simple Fabric script.

Funbric ? :D

$ rsync mozilla/services community #week 48

Tarek Ziadé — Sat, 04 Dec 2010 16:16:00 +0100

What's this ? read this post

Week 48

Easy Setup on new devices: I have benched the server on stage using Funkload to make sure everything works nicely, and realized that the Nginx + Gunicorn setup is not currently optimal using a single worker. I need to use a single worker because I have coded an IP filtering system that relies on a memory queue for speed reasons. In other words, all requests on one server need to share the same memory. The solution is to use an async worker like gevent that can be used without having to change the application code (Thanks Benoit!). GEvent uses libevent and greenlets and will allow a single process to run an event loop to handle requests, ala Twisted. So that's my next task.

Firefox Sync in Python: We finally landed the Python Sync server in dev servers, using our full environment. The server itself works since a while now, and is successfully used by people in the community, but I didn't have the chance to get it running with our fully-fledge infrastructure yet at Mozilla. So I had a few bugs to fix this week, related to that. For instance the ACLs on the Ldap are set so the bind user cannot read all fields on users entries. Anyways, this is now looking good.

OpenID / Identity: The bench script is ready. There's not much left to do on the server at this point. Maybe promote the server a little bit to see if some people in the community have an interest in such a thing.

Next Week Plans

Next week I plan to:
- Test the gevent worker with GUnicorn - Start some documentation - Integrate code reviews for Sync, hopefully - Prepare the 1/2 Python gathering I have planned mid-december in Moutain View Offices - etc.

$ rsync mozilla/services community week #47

Tarek Ziadé — Tue, 30 Nov 2010 14:49:00 +0100

What's that ?

The various weekly updates the Mozilla projects are a good way to follow, from a ten thousands foot point of view, what's going on

I'd like to do my own contribution and launch my own weekly update, about the work we are doing in my team (Mozilla Services) and in particular everything related to Python. I'll try to keep it biased and say good things about Python and bad things about PHP and Perl ;).

More seriously, I think that's a good way to sync with the community for the ones who cares about Python and Mozilla, and get some feedback. Especially since the services we build are not direct end-user products for people out there. Although, they play an important role in the background. For instance, to store and retrieve your encrypted bookmarks via the Firefox Sync add-on, you are calling our Services.

I'll keep those reports short and synthetic and I'll leave it up to the readers to ask for more details on specific points. I will try not to use internal technical jargon.

Week 47

Easy Setup on new devices :
- The server-side, using memcached & Python is ready. It's now deployed on stage using Nginx and Gunicorn, and the security team is now reviewing it. - I worked with Richard to deploy it through rpms. The pypi2rpm script I wrote now works well and creates a RPM out of any project released at PyPI, with the right options for CentOS.

Firefox Sync in Python:
- Rysiek (from Poland) now successfully use the server with his LDAP and Postgres setup, after we worked on adding options for him. - Ian reviewed the code of server-core and I followed up with a few fixes. - Toby is now using it as well in the new project he works on, and gave me some feedback on the provided APIs. We fixed a few things to make his life easier

OpenID :

I have started a prototype of an OpenID server for Services. It works now with all websites I know of, that consume Openid. It uses Redis to store the association handles and the sites tokens. For the protocol implementation, I've first used python-openid but I ended up doing custom code. python-openid is great but a bit over-engineered for what I need to do on server-side. Although, one really useful feature in this project is in its examples/ folder: a client application you can use to test your server, and vice-versa.

The client part in Firefox is acustom weave-identity add-on that automatically performs the authentication on the openid server, and removes all manual steps you need to authorize a site. It replaces the input text where you usually type your open id identity, with a "sign in" button.

Next Week Plans

Well, this week as of yesterday :)
- Easy Setup: Will follow Richard and Michael work on deployment and reviews and help them in case there's an issue - Sync: more reviews planned, need to write doc for sync-core - OpenId: need to write the script for benchmarking the server using Funkload. - more things !

[personal] [fr] Comment je me suis fait avoir par Virgin Mobile

Tarek Ziadé — Fri, 19 Nov 2010 16:37:00 +0100

J'en reviens pas.

Il y a un an, j'ai pris le Forfait Paradyse 2h SL avec un HTC Hero. (L'offre a changé et a plusieurs limitations maintenant)

Mon objectif était clair: avoir un smartphone qui me pernette de faire du web sans me soucier de la bande passante que je consomme. Ce forfait était parfait, malgré une petite limitation pas trop gênante: au bout de 500 mo je recois un SMS me stipulant que je suis "trop gourmand" et que ma bande passante allait être réduite pour la fin du mois. Je trouvais ce message mignon, mais plus maintenant.

Tous les possesseurs de smart phone à base d'Android ou d'iPhone le savent bien: à mois d'être ultra-vigilant et de passer en wifi aussi souvent que possible, c'est assez facile de dépasser cette limite, sans pour autant avoir un usage extravagant du téléphone comme du tethering. Et justement, un forfait illimité c'est fait pour ne plus avoir ce souci de... limite...

Ce mois-ci j'ai fait pas mal de déplacements et mon téléphone ne passait pas trop en wifi.

Mais là ce matin grosse surprise: Virgin m'a suspendu ma ligne.

J'appelle le service client et j'ai d'abord le droit à cette explication fumeuse:

Nous avons coupé votre ligne pour raison de sécurite, car vous aviez consommé beaucoup de data et nous pensions que peut-être votre téléphone avait été volé. Nous attendions que vous nous appeliez pour rétablir la ligne... Elle le sera sous 3h.

Si mon télephone est volé la procédure normale est d'appeler l'opérateur pour qu'il bloque la ligne. Un operateur ne va pas faire du zèle de sécurité dans ce cas de figure ?! Au pire, il essaye de contacter le client pour vérifier. Bon, les techniciens au bout du fil ne sont pas très fiables parfois, alors je rappele et ce coup ci l'explication change un peu:

Vous avez consommé trop de data, et nous avons le droit de couper la ligne conformément à l'article 10.5. Votre ligne sera retabli d'ici 12h

Mince, j'ai pris 8 heures en rappelant.. et le fameux article 10.5 (que j'ai accepté me dit la dame) est assez incroyable. Je vous laisse savourer le titre et les passages que j'ai mis en gras.

10.5 Le Client s’engage à utiliser le Service en bon père de famille.
Lorsque le Client souscrit à une offre de service comprenant une offre d’abondance ou un nombre illimité de communications, de SMS, d’emails, de messages instantanés et de navigation internet via internet mobile et/ou via une application mobile illimités, le type des communications, de SMS, de messages instantanés, d’emails et de navigation illimités concernés, la destination et/ou la plage horaire sont précisés dans les documents commerciaux relatifs à cette offre. Lorsque les communications, les SMS, les messages instantanés ou les emails en abondance ou illimités sont possibles, ils ne sont autorisés exclusivement qu’entre deux personnes physiques et pour un usage personnel non lucratif direct.
Virgin Mobile se réserve le droit, en cas d’utilisation inappropriée ou non-conforme aux indications figurant dans la documentation commerciale d’une offre de services et/ou dans la fiche tarifaire, comprenant des communications, des SMS, des emails, des messages instantanés et de la navigation internet via internet mobile et/ou via une application mobile en abondance ou illimités, de facturer les communications, SMS, emails, messages ou navigation internet interdits jusqu’à la suspension au prix hors forfait figurant dans la brochure tarifaire, et/ou de suspendre puis de résilier l’offre, conformément aux dispositions de l’article 11. Constituent notamment des cas d’utilisation inappropriée d’une offre de service comprenant des offres d’abondance ou des communications, des SMS, d’emails, des messages instantanés et/ou de la navigation internet via internet mobile et/ou via une application mobile illimités :
- détournement des offres,
- l’utilisation des offres d’abondance, des communications, des SMS, des emails, des messages instantanés et/ou de la Data via internet mobile et/ou via une application mobile illimités à des fins autres que personnelles (notamment aux fins d’en faire commerce),
- une consommation anormale de communications, de SMS, d’emails, de messages instantanés et/ou d’un usage internet via internet mobile et/ou via une application mobile anormal dans le cadre des offres d’abondance ou d’offres illimitées par rapport à la consommation moyenne enregistrée parmi les clients titulaires d’offres d’abondance ou d’offres illimitées,
- l’utilisation à titre gratuit ou onéreux d’une telle offre de service en tant que passerelle de ré acheminement de communications, les appels depuis un boitier radio,
- l’utilisation ininterrompue du forfait par le biais notamment d’une composition automatique et en continu de numéros sur la ligne,
- la cession ou la revente, totale ou partielle, des communications, des SMS, des emails, messages instantanés et/ou de la Data via internet mobile et/ou via une application mobile illimités,
- l’envoi en masse ou groupé de communications, de SMS, d’emails de façon automatisée ou non,
- l’utilisation partagée du service Data et notamment d’associer la carte SIM à un équipement de type routeur à des fins de partage de l’accès entre plusieurs utilisateurs. Les usages de type peer to peer ; newsgroups (partage de ressources entre utilisateurs) sont interdits,
- l’utilisation à des fins de streaming concernant les offres Data via internet mobile et/ou via une application mobile,
- l’utilisation de la carte SIM avec un forfait navigation internet illimitée dans un terminal en tant que modem. Afin de pouvoir permettre une utilisation optimale à l’ensemble des Clients titulaires d’une offre illimitée de navigation internet via internet mobile et/ou via une application mobile, emails, messagerie instantanée ou d’abondance, une limitation du débit de l’accès au réseau internet au-delà d’un usage mensuel indiqué dans les conditions particulières des offres pourra être mise en œuvre.
Le Client reconnaît et accepte que dans la mesure où Internet n’est pas un réseau sécurisé, il lui incombe de prendre toutes mesures utiles pour se prémunir contre les virus ou les intrusions de tiers. Toute transmission par ce biais de données qu’il estime confidentielles se fera à ses risques et périls.

Donc je résume:

Si par rapport à la moyenne des utilisateurs de cette offre illimité, je suis trop au dessus, Virgin à le droit de me facturer un surplus ou de couper ma ligne. J'aimerais bien connaitre l'algorithme, mais en gros les plus gros consommateurs qui avant voyaient leur bande passante limitée jusqu'à la fin du mois, peuvent maintenant avoir leur ligne automatiquement coupée (=on ne peut plus vous appeler, le "numéro n'est pas attribué").

Donc l'offre de Virgin "internet illimité" est en fait "limité aux bon père de famille" selon les critères vagues de Virgin. Et moi, pauvre Geek naif, j'ai une consommation apparament anormale alors on me coupe la ligne... donc je suis puni de téléphone pour avoir cru a cette offre illimité il y a un an.

J'ai presque envie de contacter des associations de consommateurs ou la DGCCRF. Car il me parait évident que Virgin ne rentabilise pas ces anciens forfaits et essaye de limiter la casse avec des moyens pas très corrects. C'est vraiment dommage: impossible de récupérer le meme article 10.5 d'il y a un an pour voir s'il a été modifié, je serais curieux de voir ca.

Après, c'est pas forcément rose chez les autres opérateurs, mais je pars de Virgin dès que possible.

Si vous êtes dans le même cas ou un cas similaire, merci de commenter.

Packaging talk at Pycon 2011

Tarek Ziadé — Sat, 13 Nov 2010 11:47:00 +0100

I have submitted a talk for Distutils at Pycon U.S. 2011 under the "Extreme Talks" section. The idea is to do a deep-dive talk on how Distutils2 works and how to use it in your projects.

Here's the summary of what I am planning to present:
- Distutils2 presentation and goals

    - Framework

    - Command-driven packaging system

-  Changes from Distutils1

    - R.I.P. setup.py

    - The new metadata fields (PEP 345)

    - versions for your project (PEP 386)

    - PyPI goodies

     - browsing

     - uploading docs

    - What's installed ? what to install ? (PEP 376)

     - The Dependency graph tool

    - Extensibility !

     - commands

     - compilers

-  Pysetup, one command to rule them all

    - install !

    - remove

    - do other things

-  Examples

    - Example 1: A simple Distutils2 project

    - Example 2: Porting your project to Distutils2, and keep it working in Distutils/Setuptools/zc.buildout environments.

    - Example 3: Creating and releasing your own commands and compilers

    - Example 4: Developement process made simple with Distutils2

-  Conclusion

    - Roadmap

If you are thinking about something related to Distutils that you would like to see discussed during this talk, let me know here !

Distutils 2 alpha 4 - work in progress

Tarek Ziadé — Sun, 07 Nov 2010 23:53:00 +0100

Time flies, and we did not hit the first beta of Distutils2 yet. It's quite clear now that we won't make it for Python 3.2. But that's not a big issue because people will be able to install Distutils2 from Python 2.4 to 3.x and use it until it gets back into Python standard library.

We are going to have a sprint next week-end and try to wrap up a new alpha.

It should contain:
- The work done by Eric on the configure command during the GSOC. The configure command let you create a config file that can be used by the build or the install command. This fixes one issue people have: the install command will not have to run the build command again, just to get the options that were used to build the project. - The work done by Alexis on the installation script. We are adding a 'pysetup' script people can use to drive installations, un-installations and running any command or feature Distutils2 has. One script to rule them all. - The work done by Zubin to make Distutils2 Python3-compatible.

Other stuff I've added

You can now configure the sub command list the install or the build command uses in distutils.cfg, or setup.cfg. -- Or any other subcommands in fact -- This means that people will be able to provide a new sub command that extends install or build, without having to override the hardcoded list of subcommands they contain. They will also be able to define the ordering of all subcommands.

The sdist command has a new option called manifest_builders, where you can provide hooks that will be called just after the sdist command has created the list of files to include in the distribution, and just before the manifest is written. In other words, you can use it to add or remove files through code.

Other stuff I am working on

The compilers in Distutils are still messy and did not get any attention yet. I'd like to clean them up, to make it easier to register a new compiler, and to add a tutorial on how to write a new compiler.

Compilers are something I am not very familiar with. Jeremy Kloth, who worked on 4Suite and on Distutils2 should be of a good help in this area. I wished we would get some help from other folks from the Python scientific community since they have specific compiling needs. But according to David Cournapeau, ["Several extremely talented people in the scipy community have indepedently attempted to improve this in the last decade or so, without any success"][] .. reading his whole blog I am quite surprised because part of the problems he described with Distutils are getting solved right now in Distutils2 --some of them during the GSOC after his criticisms on python-dev-- and other problems don't sound like unsolvable issues.

Oh well, I can understand that people got fed up with Distutils, it's a very long process to improve it. And it's a good thing that other people build other tools. The Python community can have as many build tool as it wants. The more tools the better. As long as we all share the same standards when it comes to publish things at PyPI or install projects in Python. Interoperability matters.

And that's where Distutils2 has an important role to play. Besides the command system it provides to build and package stuff, it also provides implementation of all the latest packaging standards we have built through PEPs. The one that will be used in the future at all levels in PyPI and Python. So if you plan on building yet another build tool, you should have a look at them.

Help us !

Distutils2 support can be added today in your project without interfering with the existing Distutils1/Setuptools/Distribute support. You can add in your project's setup.cfg new sections that are specific to Distutils2 and that will make your project Distutils2 compatible without any harm. This would be of a great help for us to get some feedback from real-world projects on Distutils2. If you are interested and want to help, let me know.

And, if you have any frustration with distutils, or feature request, we want to hear them. You can drop by #distutils on Freenode or drop a mail in our mailing list.

["Several extremely talented people in the scipy community have indepedently attempted to improve this in the last decade or so, without any success"]: http://cournape.wordpress.com/2010/10/13/271/

Firefox Sync : deploying Python apps on RPM-based systems

Tarek Ziadé — Thu, 04 Nov 2010 18:43:00 +0100

The Python Firefox Sync server is going to be deployed at Mozilla in production early next year, on RHEL-based servers. Working on packaging matters in Python, I focused a lot on the best way to deploy our Python applications and libraries, and make every involved party happy.

Our environment

Unlike a fully-featured website, our various web applications for Firefox Sync don't have a lot of dependencies besides Python itself. This means that we don't have a huge list of dependencies to deploy to set up every part of the infrastructure. Another important point is that unlike smaller projects where a single server/VM might manage several applications whose dependencies can conflict, we have dedicated environments for each application we are deploying. Hey, I am happily running 5 small websites with 5 different environments on my own server, thanks to zc.buildout, but for Sync it's the other way around: many servers for the same app :)

In other words, tools like zc.buildout or virtualenv -- which provide application-level isolated environments -- are not really needed. We can happily install the different bits in the same Python environment.

Repository-oriented vs Release-oriented

One approach I have seen is to deploy applications in a directory using Git or Mercurial and just pull the code from a repository on a specific tag or branch. Dependencies can be fetch the same way by using a vendor repository, or simply installed locally using Pip or easy_install.

The biggest caveat of this approach is that there's no more static archive that freezes everything into a single object that can be manipulated by usual packaging systems the system provide -- The Python packaging system or the OS packaging system --

OS Packaging systems provide a lot of automation for sysadmins and more features for pre- and post- installation steps for the packager. Instead of using custom recipes to upgrade the application, they can use what they are using for everything else on the system, and with much more features like the ability to mark some files in the metadata as configuration files and tell the installer about it to avoid overriding existing files, etc.

The Python packaging system is also simpler than a repository-based system when you want to run the application in any Operating System. You just install the application that was released, and don't bother with source repositories.

Distutils-based and RPM-based

Sync is packaged in RPMs and in Distutils source archives. Every distribution has a .spec file containing the RPM metadata and also a setup.py file. Releasing the project into Distutils-based archive is done so people in the community that wish to deploy the server in a non-RPM based system can do it. In a near future, a simple "pip install SyncServer" will do the trick for them. They'll have extra manual steps, like hooking the app in the web server of their choice, but they won't have to checkout many repositories and create a dedicated environment.

For the RPM based distribution, I was a bit annoyed by the bdist_rpm command Python/Distutils provides. It's not good enough to easily tweak the RPM creation process and you always end-up writing your own .spec file. But build_rpm do provide a nice automation: it calls rpmbuild with good enough options, and for some distributions I don't need to write a custom .spec file as long as I am able to define a custom name for the RPM project. For instance, I prefix all my distributions with python26-.

Since we've removed build_rpm from Distutils2, I ended up writing an enhanced version in a project called pypi2rpm, that provides two new options: --name and --spec-file. I use them to build RPM on projects that have custom .spec files, or when I want to force the name of the RPM with a direct command-line option.

The pypi2rpm project also provide a nice command-line script that uses the power of Distutils2 to generate RPMs for the latest versions of any project, as long as it is published at PyPI. That's useful to complete my RPM collection when I cannot find a RPM for a project.

So, if I want to create a RPM for the latest WebOb release, I do:

$ pypi2rpm.py webob

I've pushed pypi2rpm at PyPI if you want to play with it. It's a work in progress and lacks of documentation, but is good enough for me for what I need to do right now. If you are interested in this small project let me know !

MoPy = Mozilla + Python

Tarek Ziadé — Thu, 04 Nov 2010 15:28:00 +0100

Python is gaining a lot of traction in many Mozilla projects. From the SUMO and AMO websites, to Socorro and the next-gen Firefox Sync server. I can fairly say that it becomes a mainstream language for building server-side software in our eco-system. This is making me really happy, since I am involved in the Python project.

[caption id="" align="alignright" width="381" caption="The MoPy fish - a 1997 cyberpet :)"][/caption]

I won't blog here about the pro's and con's of Python versus PHP or Java etc., there are already a lot of good articles out there about it. I'd be biased anyways ;). The least I can say is that it makes a lot of sense for us at all levels, to use Python and its great libs and frameworks. To name a few:
- Django, used by AMO and SUMO - WebOb and Paste, used by Sync (next-gen) to build wsgi-based Web Services - SQLAlchemy, used by Sync (next-gen) to work with our databases.

When I joined Mozilla, one thing that I really wanted to do was to start a "Python in Mozilla" informal user group. A space where people working on the various Mozilla projects could share about Python.

I started a mailing list here : MoPy Mailing List.

We also had a small meeting at Mountain View HQ. The WebDev team was in town so it was a good opportunity to do it. We talked about our various approaches to handle configuration files in our projects and that was quite interesting to share. We had people from WebDev, Labs and Services.

I'd love to share ideas about Python with the whole Mozilla community, and maybe try to organise more meetings in the future.

If you are involved in a Mozilla project, and use Python -- or simply curious about Python -- please join the mailing list and don't be shy to post.

Semi-related: I have submitted a talk for the next Pycon in Atlanta about Firefox Sync, and will try to organize a Mozilla BoF there as well.

"The MoPy fish - a 1997 cyberpet :)"

A quick glimpse at Distutils2 alpha3 - part 2

Tarek Ziadé — Sun, 03 Oct 2010 13:37:00 +0200

It seems that doing a little bit of teasing on the next release generates valuable feedback. So I'll do more :)

In the next version we have this really useful module called depgraph (full doc). It creates a dependency graph to study installed distribution, but also to-be-installed distributions. This is the basis of any installer script but is also useful for users who want to know what's installed.

Setuptools has a module called pkg_resources that would allow to create such a feature, but distutils2 used an enhanced version of the stdlib module pkgutil which now supports PEP 376 but also offers a compatibility mode to be able to browse packages that were installed by pre-PEP 376 installers like Pip or easy_install. The new pkgutil module lives in a _backport package in distutils2 but will be pushed back to the stdlib as soon as we reach a stable version for distutils2. It will make pkg_resources obsolete for the part that let you iterate on installed projects and will provide more. (Note that pkg_resources contains much more features besides.)

But enough talking, try to build your depgraph yourself ! depgraph can be called as a script to generate dependency graphs in the stdout, or as .dot (graphviz) files.

Here's a small demo:
# installing the latest tip

$ sudo easy_install http://bitbucket.org/tarek/distutils2/get/bdfaec90d665.gz

Downloading http://bitbucket.org/tarek/distutils2/get/bdfaec90d665.gz

Processing bdfaec90d665.gz

...

# what do we have installed ?

$ python -m distutils2.depgraph

Dependency graph:

 PasteDeploy 1.3.3

 virtualenv 1.4.9

 pyflakes 0.4.0

 ...

 Distutils2 1.0a3

 ropemode 0.1-rc2

   rope 0.9.3 [rope (>= 0.9.2)]

...

# let's create a dot file of the dependencies

# if a distribution don't have a dependency it's not added - so you don't get crazy graphs

$ python -m distutils2.depgraph -d

Dot file written at "depgraph.dot"

# let's create an image

$ dot -Tpng depgraph.dot > depgraph.png

Try it out and let us know how it worked for you !

"Depgraph example"

A quick glimpse at Distutils2 1.0 alpha3

Tarek Ziadé — Sat, 02 Oct 2010 15:44:00 +0200

We're busy preparing the third alpha of Distutils2 1.0. I am pretty excited about it because all of the hard work that was done during the GSOC is starting to surface.

The major feature in alpha3 will be the fact that setup.py is gone. Everything is now driven from scripts you can run using -m.

To describe your project you can now use setup.cfg, or better : use the mkpkg wizard which will ask you a few questions and create it for you.

In the demo below, I am installing distutils2 in a virtualenv and I am creating a setup.cfg for a project that contains the 'mycode' package. Then I am creating a distribution, installing it and testing it.
# creating a temp dir with a virtualenv in it

$ cd /tmp/demo/

$ virtualenv --no-site-packages .

New python executable in ./bin/python2.6

...

# installing Distutils2 from the tip

$ bin/easy_install http://bitbucket.org/tarek/distutils2/get/tip.gz

Downloading http://bitbucket.org/tarek/distutils2/get/tip.gz

Processing tip.gz

...

# creating a Python package to include to the project

$ mkdir mycode

$ echo print \"I am alive\" > mycode/__init__.py

# running the wizard that creates a setup.cfg for me

# Thanks god it takes care of the Trove classifiers for me now !

$ bin/python -m distutils2.mkpkg

Project name [demo]:

Current version number: 1.0

Package description:

   > short demo

Author name [Tarek]: Tarek

Author e-mail address [tarek@ziade.org]:

Project Home Page: http://bitbucket.org/tarek/distutils2/wiki/Home

Do you want to add a package ? (y/n): y

Package name: mycode

Do you want to add a package ? (y/n): n

Do you want to set Trove classifiers? (y/n): y

Please select the project status:

1 - Planning

2 - Pre-Alpha

3 - Alpha

4 - Beta

5 - Production/Stable

6 - Mature

7 - Inactive

Status: 3

What license do you use: GPL

Matching licenses:

   1) License :: OSI Approved :: GNU General Public License (GPL)

   2) License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)

Type the number of the license you wish to use or ? to try again:: 1

Do you want to set other trove identifiers (y/n) [n]: n

Wrote "setup.cfg".

# checking the result

$ more setup.cfg

[metadata]

name = demo

version = 1.0

author = Tarek

author_email = tarek@ziade.org

description = short demo

home_page = http://bitbucket.org/tarek/distutils2/wiki/Home

classifier = Development Status :: 3 - Alpha

    License :: OSI Approved :: GNU General Public License (GPL)

[files]

packages = mycode

# trying to build a source distribution

$ bin/python -m distutils2.run sdist

running sdist

# installing the distribution in the virtualenv'ed Python

$ bin/python -m distutils2.run install

running install

...

# is my project properly installed ?

$ bin/python

Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)

[GCC 4.4.3] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import mycode

I am alive

I'm hoping to have alpha3 released this week-end or early next week.

Twisted rocks !

Tarek Ziadé — Thu, 30 Sep 2010 18:14:00 +0200

When your LDAP server or your MySQL server starts to get really slow for any reason, you better make sure your web application don't wait too much for them. Otherwise you will just have more and more requests piling up, waiting for your HTTP server to kill them when they reach the server timeout. Apache default timeout is 300 seconds by the way, so depending on how many users your application serves, you might end up with a big bottleneck in your application and a potential disaster on the server resources.

One solution is to make sure your web application handles itself the problem, by configuring a tight timeout for every third party server it calls, like a SQL server or a LDAP Server. So whenever things gets too slow, you can return immediately a 503 to the client and free the thread. A Retry-Later header can also be added to inform the client.

For Sync, we have another header that is specifically looked up when things gets bad on server side, which is X-Weave-Backoff. But that header is used only on successful operations to gently ask the client to back off for some time.

Anyways, in theory it's quite simple to add some try..except timout: code in your application but in practice you better test it for real by benching your application with slow third party servers and check that the server does not melt in that case.

Not all servers (LDAP, SQL, etc) provide a way to make things slower and not all Operating Systems provide a simple way to slow down the network between two applications. netem is a nice tool but you might need to recompile your kernel to use it, and you have to be on some flavor of BSD or Linux for that.

Twisted excels for such tasks. I could write a port-forwarding script to simulate delays in less than 15 lines (that would take probably 100 lines using plain socket/asyncore).
import time

from twisted.internet import reactor

from twisted.protocols import portforward

class LoggingProxyServer(portforward.ProxyServer):

    def dataReceived(self, data):

        time.sleep(20)

        portforward.ProxyServer.dataReceived(self, data)

class LoggingProxyFactory(portforward.ProxyFactory):

    protocol = LoggingProxyServer

if __name__ == '__main__':

    fwd = LoggingProxyFactory('localhost', 389)

    reactor.listenTCP(390, fwd)

    reactor.run()

Once this script is launched, my application can use the port 390 to connect to LDAP, and deal with the timeouts.

From there, I am exploring different ways to delay the calls in a realistic manner. Like making the delay get bigger at every call until it reaches a max, then reducing it, etc. It's also a good way to log all the TCP conversations between the apps without having to set up a dedicated app like WireShark.

I already knew this, since Twisted was the framework we used in my previous company, but let me say it again: when it comes to write little network tools like that, Twisted just rocks.

Afpy Camp - Python Sprint Wrap-up

Tarek Ziadé — Wed, 22 Sep 2010 21:39:00 +0200

The Afpy Computer Camp was great as usual. We were fifteen of us sprinting at my house and we had a lot of fun watching zombies movies (Shawn of the Dead. Zombie Land) and eating that suckling Pig I bought.

[caption id="" align="alignright" width="315" caption="The packaging pig is dead "][/caption]

On the coding side I must admit I did not do as much as I wanted, which is a shame since I had around me great folks to work on testing, packaging etc. I had to take care of the catering and the train station shuttling.

I did find some time to brainstorm with Michael on plugins and some code should follow this week. The idea is to provide a entry-point like set of API in the version of pkgutil we have in distutils2, which complements PEP 376 and is supposed to update the module we have in the stdlib. Michael will use it for unittest2 once I ship a working version.

I also did quite some work with Holger, who helped me with is Tox tool to set up a Continuous Integration server for Distutils2 that tests multiple Python version. Yay! Its up and running but I need to add some ACLs before I publish it.

Next we talked about the best approach for people to start using Distutils2 amd Holger came up with great ideas.

We will provide as planned an alpha version that has a run module that can be run to execute distutils2 on a project. No more setup.py here, the script will just read the setup.cfg and read all options from there, including a section where all metadata are described.
$ python -m distutils2.run install

That's already committed and I have one more feature to add before I released Distutils2 1.0a3 : a small hook to allow people to run arbitrary code in the process. This hook won't be able to change the metadata so we don't break the "static metadata" paradigm we want to provide -- see PEP 345 --. It will be useful though to work on build options etc.

Projects will be able to keep their setup,py file for Distutils1 and Setuptools (that's the new idea :D) , and add sections in their setup.cfg to make the project Distutils2 compatible in the same time. IOW each Distutils version will not step on the other version toes. I am more concerned about the adoption of distutils2 in the various installers like Pip or easy_install but that's a good complement to have.

I had a great time with great folks, thanks to the PSF / Jesse Noller for the $250 grant we are going to use for paying part of the travel expenses for Michael and Holger. Thanks also to Logilab who sent us Pierres-Yves.

I am looking forward to read other people wrap-up, and also to next year Camp !

More Pictures

"Packaging Pig"
"Debian FTW"

Firefox Sync Server in Python - Take 2

Tarek Ziadé — Tue, 21 Sep 2010 17:11:00 +0200

It's been more than a month since the last update on my work on Firefox Sync. Time for a quick update.

The Code

The application grew quite well and was splitted in four separate projects:
- SyncCore: contains the authentication back-ends and various utilities like the CEF logger or various WSGI helpers. - SyncReg: that's the User application. Implements: https://wiki.mozilla.org/Labs/Weave/User/1.0/API. Can be used as a standalone WSGI application - SyncStorage: Contains the storage back-ends and implements https://wiki.mozilla.org/Labs/Weave/Sync/1.0/API (and the upcoming 1.1.) Can be used as a standalone WSGI application. - SyncServer: This is just a glue application that can be used to run in the same server both Reg and Storage servers. By default this application will run sqlite back-ends for storage and authentication, which means it can be launched with a zero-config environment.

I moved the code to bitbucket, and will clone it back to hg.mozilla.org once we set dedicated repositories for the Python server there. If you want to run your own Sync server, it's still very simple. Make sure you have the latest virtualenv installed, Mercurial and Make, then run:
$ hg clone http://bitbucket.org/tarek/sync-server Sync

$ cd Sync

$ make build

Then you can run your server on port 5000 by using the built-in web server:
$ bin/paster serve development.ini

Of course, a real setup should be done using SSL, a real web server like Apache/mod_wsgi and MySQL for the DB. But the default setup is useful and can replace the minimal-server Toby wrote.

Benching

One thing I want to make sure is that the Python server is as fast as possible, and faster than the PHP application. Since a Python web application can reuse the same interpreter in memory, there's a lot of room for improvements like connection pooling and light memory caching. I also wanted to bench out various configurations for the DB, like using postgresql instead of mysql etc.

The team is currently working on stress testing our Sync infrastructure and the tool that we use is Grinder. Grinder is a Java tool that uses Jython for writing tests, and provides a simple console to drive it. The results Grinder return are raw results, and there's quite some work left to do if you want to generate nice reports.

I used another tool to bench the server called Funkload. It's a Python tool that uses unittest classes to run benches, and provides a functional test tool to query a web server and do some assertions like WebTest. It produces HTML reports that are containing a lot of metrics. Some I don't use because they are specific to web sites. But it's good enough to stress-test the Sync server and compare PHP and Python speed. One caveat is that it cannot be distributed. There's a project called BenchMaster that adds this feature, that I need to try.

The stress test is the same than the Grinder one, and here are some reports using various configuration : http://sync.ziade.org/funkload/

[][]While Python already appears to be slightly faster than PHP, those were done on my MacBook with 100 users loaded in the DB, 6000 objects each, so don't mean a lot. Just that the Python application is not borked :D .

I'll probably run Funkload in the same environment we run Grinder at Mozilla, where we have a realistic setup. I also want to have this kind of reports generated every day, so I can keep an eye on how the Python server performs. Making sure the app does not slow down when it grows is one important part of continuous integration.

Caching: Redis vs Memcached

I used Redis to do a bit of caching in the Python app, instead of Memcached like the PHP app. See my previous post for the rational.

Redis was very stable during my benches, but I have heard from some other projects that they had quite a few problems with it in production [I might post more details here in another blog post]. I still think this is the tool we should use in Sync, and I also want to experiment writing a full back-end for Sync using it. But the first version of the Sync server we will deploy on our servers will probably use Memcached since it's proven to work well right now and since I don't really need all the extra features Redis offers if the usage is restricted to volatile caching.

Continuous integration

I am still working alone on the Python app, but a continuous integration server is something we really want to have. I am a big fan of buildbot but I wanted to give a try at Hudson. The management interface is brilliant and I could set up a Hudson server for Sync in an hour. I eventually moved it at https://hudson.mozilla.org/job/Sync with other Mozilla projects from the WebDev team. It contains Pylint reports, test coverage report, and of course Chuck Norris keeps the code safe.

What's next ?

The Python app is mostly done, besides a few things to clean up. The next big step will be to bench it alongside the PHP application on realistic data, fix any problem that will rise, then work on pushing it in production. The production switch will probably happen gradually since every node is standalone. And since the rest of the team is quite busy to make sure everything is ready for the upcoming Firefox 4 final release which includes Sync natively, switching to Python is not the #1 priority right now. I expect it to happen before the end of the year though.

In the meantime once the benches are done and the code is rock-solid. I'll start to play with different back-ends. A full Redis back-end and maybe something based on Riak or Cassandra.

"Funkload Report"

[]: http://sync.ziade.org/funkload "Chuck Norris"

Pycon France next week-end

Tarek Ziadé — Mon, 23 Aug 2010 16:48:00 +0200

Pycon France will be starting Saturday in Paris. Make sure you're there if you are in Paris. It's Free.

Here are a list of talks I don't want to miss (I want to see them all more of course):
- GUnicorn, the wsgi server - MongoDB and MongoKit - Python in Debian

The full program: [http://www.pycon.fr/#talksschedule][]

Distutils 2 - Summary of the GSOC

Tarek Ziadé — Thu, 19 Aug 2010 09:35:00 +0200

Man this has been an amazing summer. We had 5 students working on Distutils 2 and the work done was really great. Most important, I think we managed to find new contributors for the future.

Thanks to Josip, Alexis, Konrad, Eric and Zubin ! Also, thanks to their respective mentors: Michael, Fred, Titus and Lennart.

Let's have a very high-level summary of the tasks that were done. As a reminder, Distutils2 aims to provide two things: a toolbox for third party projects that want to provide packaging features, and a full-featured package installer/manager for Python.

What was done

Distutils2 has now an index package, containing tools to work with PyPI (or any PyPI-like server). This package contains classes to work with the Simple Index protocol as well as the XML-RPC APIs PyPI has. We think that the latter should be replaced by static REST calls, but that's another topic. If you want to build a tool that uses PyPI, that's the package you want to use. See this doc (temporary location) for more info.

Another thing we added is an installer script. This script is a very light script that uses the index package I've mentioned before, but also a dependency graph builder we now have. The graph builder allows you to analyze relations between installed projects, but also any project you want to install. See this doc. This installation script will be completed by an uninstallation feature very soon hopefully. You might wonder why we did this since there are existing scripts like Pip or easy_install. The reasons are quite simple:

1) we wanted to exercise all the modules Distutils2 provides to work with PyPI, metadata and installed projects in a high level script. The long-term goal is to have projects like Pip or Distribute use Distutils2-the-toolbox since they'll use the same standards in the future.

2) Distutils2-the-package-manager wants to provide a basic installer/uninstaller.

Oh by the way, the depgraph tool is PEP 376 compatible, as we have now a new version of pkgutil that supports PEP 376. See this doc. The next task will be to put this new version in Python 3.2, and that should happen fairly soon.

Speaking of which, we are not sure yet if we are going to include an plugin system like the entry points in Distutils2. Discussions on this has been controversial on Python-dev. In the meantime you can enjoy the extensions project, which is a quick hack to have entry points without depending on Setuptools or Distribute --and it's now Distutils2 and PEP 376 compatible--.

Distutils2 is almost Python 3 compatible. The student in charge is working hard to finish this task. See his repo. As usual, the last problems to be solved are about unicode, strings, bytes, you named it. But I expect to include this work in one of the alpha version of Distutils2 1.0.

Last but not least, we worked on the commands front of Distutils2:
- test was added -- a command to run the project tests, inspired from Setuptools - upload_docs was added -- a command to upload docs at packages.python.org. Originally created by Jannis Leidel and present in Distribute - check was added and improved -- a command that I created to check a project before releasing/uploading it. It checks its metadata, its description reST compliancy etc. - A command post/pre hook system -- similar to what RPM has so you can point code to be run before and after a Distutils command is run. Very powerful and useful -- but to be used with caution. You wouldn't want to add to the install process a code dependency you are not sure to have. But in the meantime, being able to run a code every time a project is installed on your system is sweet ;)

I am forgetting a million things, but I guess its not important, since all our students are blogging about it ;)

Thanks to Google, the GSOC is a great program. Distutils2 just got a really serious boost, we are very close to a good, useful release.

As a conclusion, to make things clear for the few people that still see all these efforts as reinventing the wheel since packaging has been kind-of-solved already elsewhere: Every extensible system, whether its an OS like Debian or Fedora, or a language like Python, will always provide its own custom packaging system, that is built on the top of the community experience and needs. The only thing that really matters is to have this work based on standards so inter-operability is manageable. That's the goal of some of the PEP we have written, like PEP 386 and PEP 345 and that will make OS packagers life easier in the future. An universal packaging system is an utopia.

Firefox Sync Server in Python -- Take 1

Tarek Ziadé — Tue, 10 Aug 2010 15:07:00 +0200

I have been working for a bit more than a month now on the next generation of the Firefox Sync server in Python and while the project is still in its early stages and subject to a lot of changes, I think it's a good idea to share now about what we are building here at Mozilla. Maybe that'll attract contributors !

About Sync

Firefox Sync (formerly Weave) let you synchronize your Firefox bookmarks, history, passwords, opened tabs etc. so you can have them on any computer, or even use them from your iPhone by using Firefox Home.

Clients that are syncing work with our servers at Mozilla by using the Sync and the User APIs defined in these documents:
- https://wiki.mozilla.org/Labs/Weave/User/1.0/API - https://wiki.mozilla.org/Labs/Weave/Sync/1.0/API

The User APIs manage the users accounts and tell the client which server holds the data of a given user. In other words, each user is tightly coupled to a single server when reading or writing data. This natural sharding is great for scaling Sync, and is possible because users don't share data (yet... ;))

Another important point is that the data are encrypted on client side before they are sent over. That's because one of the key concept of Sync is that your data should not be known by our servers, to protect your privacy. Well, we could probably still know how many bookmarks you have by counting the number of entries in the DB, or how often you use your browser. But as soon as you use a service like that you have to give away these kind of information, most of the time just because they are useful to make the service faster or understand any potential problem. Read our privacy policy here.

And the good news is that you can set up your own Sync server and even implement it yourself if you want.

So, a Sync server a pretty passive storage server, that is quite easy to scale while keeping data consistency across clients.

About the code

The current implementation uses Apache, PHP, LDAP, MySQL and Memcached. For various reasons I won't detail in this post --that might be another post-- , it has been decided to switch the Sync server to Python

Python libraries

The Sync server is composed of web services and a few screens used for the password reset process, so using a web framework would have been overkill. Although, writing a wsgi-enabled server made a lot of sense since it allows people to run our implementation on their laptop, or on any wsgi-compatible web server they wish to use.

So, I've picked :
- Routes, to dispatch requests to a few classes (controllers) - WebOb to process incoming requests and build responses - Paste. PasteScript, PasteDeploy, to group the configuration in an ini file and make it easy to run the application with a built-in server.

There are alternative routing systems, but Routes really fits my brain and make the dispatching quite simple. I really like the fact that you can optionally use regular expressions to validate URLs.

WebOb is quite a standard library and make our life simple to read requests and write responses. The code in our controllers stays KISS with WebOb when you have to read incoming data: they're all available in simple mappings. The response is also built by WebOb and you can forget about all the wsgi protocol details. We mainly return JSON dumps that WebOb wraps into responses.

Last, Paste is very handy to run the server locally, to initialize data, and handle multiple configurations. I should also say that my colleagueIan Bicking is behind the Paste and WebOb libs, and involved in the Sync project. So those were quite natural choices.

The authentication process is a custom function that reads a basic authentication header and checks it using an authentication plugin (more on plugins later in this post.)

For the storage, I've picked SQLAlchemy and python-ldap. I don't really use the ORM part of SQLAlchemy and write pretty raw SQL queries to avoid any extra overhead. The benefit of the ORM was null here anyways, since all storage I/O are contained in a storage class that outputs simple mappings. I have created the mappers though, as they are useful to initialize a DB on a first run.

But when the server runs, SQLAlchemy is mainly used for:
- its connection pooling abilities. - the nice parameters binding - the ability to switch to any DB system via configuration (as long as the SQL is compatible of course)

As for python-ldap (I didn't implement the LDAP part yet), it's the standard connector I have always used with various flavors of LDAP servers (OpenLDAP, ActiveDirectories, etc.). I don't think there is any competitor for this anyways.

Caching

The caching is currently done using Memcached. For instance, when clients are often asking for specific collection items, they end up in memcached to lower the number of queries made to MySQL. For the Python implementation though, I've decided to use Redis instead.

In terms of speed, Redis and Memcached are quite similar. Redis though has interesting extras:
- The data is saved to the disk, so you don't lose your cache. The speed stays almost the same as memcached since the disk syncs are done asynchronously from time to time. Since a Sync user is tightly coupled to a storage server, that's an interesting feature to have. And, hey, you can move data from a Redis DB to another, so migrating the cache to another server is even possible. - Redis provides built-in APIs to work with sets and lists, which authorizes more complex caching without extra code. This will allow us to do more caching in the future.

Storage

The storage itself will stay on MySQL but we will probably explore alternative storages systems in the future. One requirement of Sync is to be able to write data as fast as possible so all clients can have access to them as soon as possible. Right now, Sync provides immediate consistency, since all writes are done synchronously on a single server.

Plugins

The PHP application was built with extensibility in mind: the way Mozilla stores the data and authenticates users (a mix of LDAP and MySQL) might not work if the code is used by someone else. That's why the code was built using abstractions for the storage and the authentication part, and the Python version took back this good idea.

Basically, you can write a new authentication or storage class, and configure Sync to use it. See the documentation I am building on this: http://sync.ziade.org/doc/storage.html (temporary location)

Web server

The web server that runs the Python application will stay Apache (with [mod_wsgi][]) since it has proven to work very well with the current implementation. I might bench other servers in the future though, like Gunicorn + nGninx or uWSGI + nGninx. We now have a nice Grinder script that realistically mimics Sync users, so..

Doc and Code

I've started a documentation, the temporary location is at http://sync.ziade.org/doc and you can grab the code we are building at [http://hg.mozilla.org/users/telliott_mozilla.com/sync-server][]. You can already use the server with your Firefox / Firefox Home, but this is still at development stage, so use at your own risks.

I would love to get some feedback on that work !

Afpy computer camp 17/18 September

Tarek Ziadé — Thu, 05 Aug 2010 22:29:00 +0200

I have announced it already, and the dates have changed a bit. But let's say it again since the date is approaching: we will have a sprint at my house the second week-end of September (17/18), focusing on packaging and testing (and other topics if people want).

This sprint will be a special event like last year: we will enjoy wines directly from the producers place (Gevrey-Chambertin anyone ?) and do some restaurants in the area.

Now its a good time to tell me if you are coming, by adding your name here: http://www.coactivate.org/projects/afpy-computer-camp-2010/project-home (make sure you are registered)

http://www.coactivate.org/projects/afpy-computer-camp-2010/project-home

plugins system: thoughts for an entry points replacement

Tarek Ziadé — Sun, 25 Jul 2010 02:29:00 +0200

This blog entry was inspired by the discussion I just had with Michael on IRC, as he just added plugins in unittest2.

Setuptools' entry points feature is hated and loved by developers. If you are not familiar with them, you can read this post from Armin.

Hated because when you install a project that contains entry points (let's call them plugins), they can be used in another application without letting you know. So basically if a plugin sucks, it can break another application just by being installed in your Python. And it's not easy to have an overview of what plugins are installed and potentially active.The worst is that projects that provide entry points, usually provide many other things. But if you want to deactivate the plugin, you have to remove the whole project... Note that plugins are not loaded at Python startup. What happens is that any application can iterate over the metadata of installed projects, looking for plugins, and eventually loading them if wanted.

Loved, because from a developer point of view you can have a new feature added in a program with no extra configuration at all. Take Nose. Thanks to entry points, it's dead easy to create a plugin for this test runner, and tell people to pip-install this new project. Zero config. Nice. Another great thing is that it's global to Python. Any application can consume any entry point. Entry points are implicit plugins I guess.

Distutils has a plugin system as well: you can add new commands by adding in distutils.cfg the path to the Python package containing the command. That's an explicit plugin system since the end-user has to configure it manually so Distutils uses it. Mercurial uses the same technique: activating a plugin is done in .hgrc. I would call these explicit plugins.

I think we can get the benefits of entry points without their caveats really simply. And provide a generic plugin system for all. Let's summarize what we want:
- being able to list all installed plugins for every Python application - being able to remove a plugin or deactivate it. Without being forced to uninstall the project that provided it - have a plugin automatically installed and activated when the project that provides it is installed

Here's how we can do. That's a brain dump, please give me some feedback !

Global plugin registry

Let's have a .python-plugins.cfg file in the user's home (and one global to Python. The user cfg is merged with the global one at startup, and overrides the values -- thanks Mongoose_Q for mentioning this on Twitter). It's a simple ini-like file like .hgrc, where each section represents a python application and a group name. A group is just a family of plugins. For instance 'commands' can be a group for the 'distutils' application. In this section, each line is a plugin, represented by a pointer to the module or class, followed by a label as the value.

Here's an example for a distutils 'i18n' command. It's a MyClass class, located in the foo package, in the bar module:
[distutils:commands]

  foo.bar:MyClass = i18n

The link to the code comes first because some plugins could have no name:
[app:group]

  package.module:Class =

Accessing the registry

distutils can provide an API to read the file, iterate and load the plugins:
>>> from distutils2 import plugins

    >>> plugins.get('distutils', 'command')

    <iterator>

    >>> plugins.get('distutils', 'command').next()

    <Plugin "i18n" at foo.bar:MyClass>

    >>> plugin = plugins.get('distutils', 'command').next()

    >>> plugin.load()    # gets the code and loads it

    <MyClass Instance>

Installing the plugins

Last, distutils could provide a mechanism to automatically register a plugin.

Projects could describe their plugins in their setup.cfg:
[plugins]

  distutils.commands.i18n =  foo.bar:MyClass

Then distutils would automatically inject them at installation time in .python-plugins.cfg only if the end user agrees:
$ python setup.py install

  distutils has detected a "i18n" plugin for distutils:commands. Do you want to activate it (Y/n) ?

Random notes on Mercurial queues

Tarek Ziadé — Wed, 30 Jun 2010 17:21:00 +0200

Working on the various parts of the Mozilla project requires some patch fu.

Basically, everything happens in bugzilla.mozilla.org, where you upload a patch and ask for a review. Once you have started to work on several patches, maintaining several mercurial clones can be tedious. That's where queues are helping a lot.

Mozilla has a nice document about queues. I've also found this Sympy tutorialquite useful.

Here are my random notes about queues so far:
- a queue is a directory of patch files (in .hg/patches). You can qpush or qpull them in your local repo. qpush will apply the current patch and add an entry in the commit log. qpop undoes it. pushing and poping will move you up or down in the stack. - I enabled 'color' in mercurial and use "hg qseries" to know where I am in the stack - I reorder patches by editing .hg/patches/series. Pretty rough but good enough. how come there are no q* command for that ? - to delete a patch, I make sure there are no pending changes, then I do "hg qpop -a; hg qdelete the_patch" - to import a patch from bugzilla, I use "hg qimport -n xxxx.patch https://bugzilla.mozilla.org/attachment.cgi?id=xxxx" where xxxx is the bug number

How do you work with queues ?

Suki, Mozilla and Japanese book

Tarek Ziadé — Sat, 19 Jun 2010 21:28:00 +0200

June is a very intense month for me, and the rest of the year should be way more intense.

For once, this entry is not about packaging, and a bit personal. But I want to share this with the Python community because that's my second family.

First of all, Suki was born two days ago on June 17th ! That's my lovely little girl, my second child. I am so happy about this (you bet). Her aunt, who is creating short movies for a living, has made a small video to welcome Suki to the world. Check it out, she's done an amazing job.

I was looking for a job and I have found one. I have accepted a position at Mozilla Labsand I'll start next Monday. I am very excited about this for many reasons. The projects I'll be working on are great, and the Mozilla people & values match perfectly with how I feel about open source and the web. I'll start talking about this here in a few weeks I guess, once things will be really started. Thanks to all the people that helped me in my job seeking during these last months !

Last, my latest book "Expert Python Programming" wastranslated in Japanese. I know some authors are translated in other languages quite often, but for me it was a big deal, because I have worked around with the translation team and the Japanese book is actually better than the original book, thanks to them, their feedback, passion about Python, ideas and work. Now I need to go to Japan to meet them :D

Distutils2 vs Pip

Tarek Ziadé — Mon, 31 May 2010 12:21:00 +0200

Note: if you are not familiar with PEP 345, you might want to read it to understand this entry. It adds for instance "Requires-Dist" that is similar to setuptools' install_requires and provides a standard for dependencies description.

The GSOC has started and we are already working on a lot of tasks about packaging. The main difficulty is to make sure each student works without overlapping with others, and never get blocked. That's why we will have weekly meetings with (almost) everyone. In parallel, the nice posse from the Montreal user group is organizing Distutils sprints quite often now. That means that we now have an important manpower for Distutils and things are starting to speed up.

There's one controversial topic though, that we need to straighten up : do we want to add an installer in Distutils2 ? And since Distutils2 goal is to be back in the stdlib for Python 3.2, that means: do we want to add an installer in the stdlib ?

My answer so far is Yes. And that's what I'll be working on unless someone is able to change my mind :)

What is Distutils2 ?

Let me explain first what is the Distutils2 project, and what we want it to provide. Like its predecessor, Distutils2 wants to provide two things:
1. a toolbox for third packaging tools, whether they are simple installers or full featured package managers (PyPM, Pip, Enthought Installer etc..). This toolbox will include (if not already) reference implementation of PEP 345, PEP 376, PEP 386. In other words, if you want to create the next killer packaging system, you can use modules like distutils2.version (PEP 386) or distutils2.metadata (PEP 345) to build it, without depending on the "everything is a command" philosophy of Distutils. 2. a standalone tool that can be used to install or remove distributions. That's what Distutils is and that's what we want to provide in the future in Distutils2. The ability to install projects (and therefore its dependencies since this is a new metadata field we added in PEP 345).

The controversy is about 2. It's controversial to provide a script that installs dependencies via PyPI into distutils2 because some projects like Pip already provides this feature.

Our current packaging ecosystem explained

A few years ago, before Setuptools added the ability to install dependencies via easy_install, installing a distribution of a given project was as simple as running a python setup.py install. This was installing the distribution in the target system, in proper locations defined by the install command. That's it.

Setuptools grew organically on the top of Distutils to provide new metadata like the "install_requires" field, that lists dependencies. Setuptools provided two things:
- A new install command that triggers the installation of dependencies, by reading the setuptools-specific "install_requires" metadata, and fetching dependencies at PyPI and installing them recursively. - An easy_install script that can be used to install a distribution located at PyPI. That's just a bootstrap on the top of the new install command. In other words, it grabs the archive at PyPI, unpack it, and run "python setup.py install" on it.

In other words, your Python project setup.py is the installer itself because when you use setuptools, it calls its specific install command and triggers the installation chain.

That's when the mess started: people that didn't have setuptools installed couldn't install projects that was using it of course. So the solution that was provided was to propose an ez_setup.py script that you have to include in your project and to run when setup.py is used, to be able to run your installation. In other words, your setup.py is bootstrapping the utilization/installation of setuptools. And that turned out to be really messy since Setuptools has its own way for installing things. I hope I don't sound harsh here, Setuptools is the best thing that happened to packaging in years. And a lot of our current work is to bring back its features into the "main stream".

The result is that you, as a end user, do not control what installer is going to be used, and you end up with a site-packages that has projects installed differently, and that uses different installers.

I am strongly against this behavior because of the mess it creates. In my opinion a python source distribution should not embed an installer and force its usage like this. We need to separate concerns: a python source project should be a dumb container with the code, and with some metadata.

Then Pip showed up.

Pip is an installer script that grabs the project you want to install and run "python setup.py install" on it. That's all it does when the project is a plain Distutils one. When it encounter Setuptools projects, it blocks the installation of the project's dependencies I have described earlier, and installs it like a simple Distutils project. Then, it analyzes its dependencies and installs each one of them separately.

That's really the way to go because it breaks what setuptools is enforcing: projects are not installing other projects in the process anymore. And in the long term, it will allow us to get rid of setup.py (but that's another blog post). And I hope Pip will soon be able to install Distutils2 projects because it is providing unifi ed metadata (distutils+setuptools -> PEP 345).

Distutils2 vs Pip

So as I said before: it's controversial to provide a script that installs dependencies via PyPI into distutils2 because Pip already provides this feature.

But one Distutils2 goal (like Distutils) is to provide a command to install a Distribution of your system so it works. And the concept of "Distribution" has evolved, thanks to PEP 345. this means that it needs to install dependencies now, exactly the way Pip does.

We could just tell people to install Pip on the top of the stdlib. But the goal is to provide in the stdlib a working packaging environment, that provides a minimum set of features. The goal is to have something that works when you install Python 3.2, like what was provided when distutils was brought in (eg batteries included).

Mac OS X includes easy_install, I don't see any good reason not to include a package installer in the Python stdlib itself. At least, we will be able to have a control on what script gets installed by default with Python.

That's why I have proposed to include Pip in Distutils2 but Ian and Carl seems a bit reluctant for various reasons. One of them is that having Pip included in the stdlib will slow down their work. I don't think this is true as long as it's included carefully. If Distutils2 allows its installer to be replaced through configuration by another one, then Pip can have new releases independently from the version included in the stdlib and people can upgrade their system without having to wait for the next Python release.

In any case, we are working on the various bits that are composing an installer in Distutils2 during GSOC since one of the goal of the project as I said earlier, is to provide a toolbox. So if the merge does not occur, it's likely that we will start a installer/uninstaller script in Distutils2, and it will look a lot like Pip I guess.

EDIT: to make things clearer, when I am saying that both projects should merge, I am only referring to the raw "install with dependencies" features in Pip, and not all the other features.

Faking a server for client-side tests

Tarek Ziadé — Mon, 10 May 2010 11:59:00 +0200

Distutils makes some call to the PyPI server to register and upload projects. Distutils2 will also make some calls to packages.python.org to automate the upload of documentation. This feature was added by Janis a while ago in Distribute and is being backported in Distutils2 during the GSOC.

To test all these features, what I usually did in Distutils was to monkey-patch the code that calls the server and record in memory the exchanges to check them. The problem with this approach is that you have to be careful in the way you patch the APIs the client code use. A typical bug that can happen is to get a slightly different behavior making your code buggy or broken when it interacts with the real server. Of course you can run your test once with the real server then use some mock or stub techniques. But this work can be rather tedious and complex in my opinion.

What I tend to do these days is drop completely this client-side test fixture approach, and just run a local server that implements partially or fully the real server API.

This blog post is just to demonstrate how easy it can be to run your own test server.

For HTTP protocols, the standard library provides everything needed to write such a server in a few lines. The wsgiref module for instance, is great to get a web server up and running during your tests.

Here's a full example (working for Python >= 2.6 and Python 3):
import threading

try:

    from urllib.request import urlopen

except ImportError:

    from urllib2 import urlopen

import time

from wsgiref.simple_server import make_server, demo_app

class AppRunner(threading.Thread):

    """Thread that wraps a wsgi app"""

    def __init__(self, wsgiapp=demo_app):

        threading.Thread.__init__(self)

        self.httpd = make_server('', 0, wsgiapp)

        self.address = self.httpd.server_address

    def run(self):

        self.httpd.serve_forever()

    def stop(self):

        self.httpd.shutdown()

        self.join()

        time.sleep(0.2)

_SERVER = None

def run_server():

    """Runs the server."""

    global _SERVER

    if _SERVER is not None:

        # we suppose it's running

        return _SERVER.address

    _SERVER = AppRunner()

    _SERVER.start()

    return _SERVER.address

def stop_server():

    """Stops the server."""

    global _SERVER

    if _SERVER is None:

        return

    _SERVER.stop()

    _SERVER = None

if __name__ == '__main__':

    # 1. set up

    print('Launching the test server')

    url, port = run_server()

    print('Test server running at %s:%d' % (url, port))

    # 2. test

    try:

        print('Testing..')

        res = urlopen('http://%s:%d' % (url, port))

        assert b'Hello world!' in res.read()

    finally:

        # 3. tear down

        print('Stopping the test server')

        stop_server()

run_server() and stop_server() are driving a thread that runs a wsgi application using the wsgiref helpers.

In the main section you can see an example of a test fixture, and of running a client test on the demo_app wsgi application wsgiref provides. It should return a "Hello world!" page. These functions are making sure there's only one server running even if run_server() is called several times, so it's dead easy to use it from a unittest class.

From there, writing a test server is just a matter of implementing a wsgi application that will replace the demo_app one. I find this technique superior to all the stub/mockup work required when you don't run your own server.

For other protocols than HTTP, this work can be longer and more complex of course, unless the sdtlib or a third party project already has a server implementation you can use (like smtpd).

A Distutils2 Google Summer of Code

Tarek Ziadé — Mon, 26 Apr 2010 22:46:00 +0200

The GSOC list of accepted students for the PSF was released.

The Distutils2 project has 5 students that will work on various packaging tasks this summer.

Congratulations to :
- Konrad Delong - Alexis Metaireau - Eric Araujo - Zubin Mithra - Josip Djolonga

Also, a big thanks to the people that are going to mentor them:
- Fred Drake - Titus Brown - Lennart Regebro - Michael Mulich

Last, a second important project will be done this summer, to create a PyPI Testing infrastucture. Congrats to Mouad Benchchaoui and Thanks to Jesse Noller !

I'll be mentoring Alexis, and I'll help other mentors and make sure we all synchronize our efforts. I'm pretty excited about this upcoming GSOC because it's a true opportunity to speed up the work we need to do, in order to built the new packaging tools we all want to have.

PEP 376 is accepted -- What it means

Tarek Ziadé — Mon, 26 Apr 2010 12:30:00 +0200

PEP 376 has just been accepted. This is a very important step in the packaging work we have been doing during the last year.

This PEP introduces a database of installed distributions, and therefore a standard that allows interoperability among all tools.

To summarize, a distribution that gets installed will have to create a .dist-info directory in Python, containing these files:
- METADATA: contains the project metadata, as described in [PEP 345][], PEP 314 and PEP 241. - RECORD: records the list of installed files - INSTALLER: records the name of the tool used to install the project - REQUESTED: the presence of this file indicates that the project installation was explicitly requested (i.e., not installed as a dependency).

Python will provide in the pkgutil module, a set of APIsthat can be used to query installed projects. This ressembles a lot to what the Setuptools project currently provides with its pkg_resources module.

An interesting side-effect of the RECORD file is that package managers will be able to uininstall projects. As a matter of fact Distutils2 will provide a basic uninstall feature on the top of the pkgutil APIs, and I hope tools like Pip (that already provide this feature) will adopt the new standard.

This small PEP is the basis to a new PEP that is coming next, which will define a standard to describe and consume resource files in Python projects. The ultimate goal will be to be able to get rid of setup.py and describe everything in static configuration files in your project. But this is another story ;)

Thanks to all people involved in this PEP, and in particular, thanks to Philip J. Eby for his help (PEP 376 is massively based on what he has created in Setuptools.)

stdlib's shutil improvements

Tarek Ziadé — Wed, 21 Apr 2010 13:21:00 +0200

I have contributed a small feature to shutil in the past, and in my work on distutils I used it really intensively. So I took over its maintenance a while ago.

I had some feedback from the community that shutil shouldn't be used, because of some negative comments like this one in copytree: "consider this example code rather than the ultimate tool". Well guess what ? I removed that comment because this is a darn lie ! ;) . copytree() for instance, is really powerful and avoid you to write yet-another-os-walk loop. Use it.

The only limitation of shutil APIs are related to the fact that it's unable to copy all the file metadata when you move/copy files around. Those are too specific to each platform and it's impossible/very hard to come up with a portable metadata reader/writer.

But for most needs you have in your Python applications, shutil is perfect. Have a look at tools like Distribute or zc.buildout, they are using shutil all over the place.

Besides a few bug fixes I did, here's a list of major features I have added or I am currently working on, with their Python version availability:
- make_archive(): this function will let you create any kind of archive, given a tree of files. You can create tarballs, zip files, or anything you want as long as you register a function that knows how to create one. See the doc for more details. (Python 2.7)

unpack_archive(): this function will let you unpack any kind of archive. It has a registery like make_archive(). It's almost ready but too late for 2.7 inclusion. (Python 3.2)
copytree(): Two new options for this one (see doc). (Python 3.2)
copy_function: allows you to provide your own copy function. Used by copytree() it wants to copy a file.
- ignore_dangling_symlinks: if copytree() encounter a dangling symlink, and symlinks is set to False, it will ignore it and not raise an error anymore.

shutil groups all high-level files operations, that's why I have added new functions to work with archives.

If you have any feature suggestion, any file operation function that you think should be included there, please let me know !

Afpy Computer Camp #2 - 3/4/5 september

Tarek Ziadé — Thu, 15 Apr 2010 07:45:00 +0200

Last year at the end of the summer, we had in my country house (I am renovating) near Dijon/France, a nice week-end consisting of tasting Burgundy wines and coding, demoing etc.
[][]

You can see some more pictures [here][].

We had a great time, and I'd like to organize this event again this year. I am blogging very early about it to make sure that interested people will save the date !

If you are interested in joining the camp, join this CoActivate page : http://www.coactivate.org/projects/afpy-computer-camp-2010 and add your name in the wiki page once I have activated your account.

"Afpy Camp #1"

[]: http://picasaweb.google.com/ziade.tarek/AfpyComputerCamp2009Turcey30Aout

A Firefox plugin experiment. XUL, Bottle and Redis

Tarek Ziadé — Wed, 14 Apr 2010 22:36:00 +0200

I am experimenting Mozilla's XUL to write Firefox plugins. I never had a chance to give it a shot before, so let's roll.

To try it out, I decided to implement a very simple Twitter Firefox plugin that works a bit like TweetDeck: The plugin adds a new action in the contextual menu labeled "Twitt it!". When you click on it, a window pops, with a text box containing a short url for the current page. The user can complete the text and send a twitter about the page.

[][]

I am pretty sure there are hundreds of similar plugins out there, but it's a pretty simple use case to learn XUL :)

The shortener

Before I could start to write the Firefox plugin, I needed a url shortener. I had two options: a shortener local to the plugin or an online service. A local shortener is probably more efficient, but an online service is more fun and more interesting to code: it can be shared across clients and it's a first step to a more complex plugin that works with online services.

Services likes bit.ly provide such API, but you have to register of course, and you have a limited number of calls to the service.

It's not like I am going to make a zillions calls to such a service, but it's so simple to implement that I decided to write my own.

To store the URLs I used a Redis storage. This is of course overkill for an experiment, but it was a good excuse to try it out and see how Redis fits my brain.

Redis

Setting up a Redis server on my MacBook was done in a matter of minutes. You just have to compile the source and you get a redis-server binary you can launch and eventually daemonize:
$ ./redis-server

14 Apr 23:07:28 * Warning: no config file specified, using the default config. In order to specify a config file use 'redis-server /path/to/redis.conf'

14 Apr 23:07:28 - Server started, Redis version 1.2.6

14 Apr 23:07:28 - DB loaded from disk

14 Apr 23:07:28 - The server is now ready to accept connections on port 6379

14 Apr 23:07:28 . DB 0: 1 keys (0 volatile) in 4 slots HT.

14 Apr 23:07:28 . 0 clients connected (0 slaves), 294928 bytes in use, 0 shared objects

From there, you can use the redis Python client, which provides all Redis commands. The great thing about this client (I didn't try all of them though) is that it is self-documented. All you have to do is read the Redis Commands Reference here : http://code.google.com/p/redis/wiki/CommandReference. Lower-case them and you have the Python methods.

Redis can be used as a classical key/value storage, but also has convenient APIs to work with lists and sets. In other words Redis can handle many use cases out of the box.

To store URLs in Redis, I used the simple set/get APIs and stored two key/value per URL: long_url/short_url and short_url/long_url. This takes more room but makes all lookups efficient (that is, 0(1)).

Here's the code: http://bitbucket.org/tarek/urltotwit/src/tip/shortener/shortener/urlstore.py

I am pretty new to Redis, so if you think this is not optimal, let me know !

The web service

The web service is just composed of two methods:
1. /urls/short_id : when reached, redirect to the page corresponding to the short_id 2. /shortener: a GET method that returns a short url, given an url

This is a perfect use case for the Bottle micro-framework !

Nothing particular here, besides the fact that it took me 30 lines or so to write it !

The code is here : http://bitbucket.org/tarek/urltotwit/src/tip/shortener/shortener/main.py

One thing I have to add is a security layer once this is published on my server.

The Plugin

To start things out, I used the Extension Wizard located at http://ted.mielczarek.org/code/mozilla/extensionwiz that generated a skeleton of the plugin. The Firefox versions in install.rdf are a bit outdated, so I had to change the em:maxVersion value to 3.5.* so it would work with my 3.5.9 version. I am not 100% sure the layout is still fully compliant with the best practices but it worked.

From there, I created a new XUL file called send_twitter.xul in content. This file contains the window that is displayed by the plugin. It uses jQuery so its simple to call the shortener via Json.
<window id="twitit-window"

 title="Twit it !"

 orient="horizontal"

 xmlns:html="http://www.w3.org/1999/xhtml"

 xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"

 onload="centerWindowOnScreen();">

 <script src="chrome://global/content/dialogOverlay.js" />

 <script src="jquery-1.4.2.min.js"/>

 <script language="javascript"><![CDATA[

 function initTwit() {

   var current_url = window.opener.gBrowser.contentDocument.location.href;

   $.getJSON("http://localhost:8080/shorten", {url: current_url},.

   function(json) {

     document.getElementById('twit-text').value = json.result;

   });

 }

 function send(twit) {

   /* TO BE DONE */

 }

 initTwit();

 ]]></script>

 <box>

 <vbox>

 <label value="Enter your Twitter message for this page:"/>

 <spring flex="3" style="max-height: 5px;"/>

 <textbox id="twit-text" style="min-width: 100px;".

 rows="4" cols="80" multiline="true" maxlength="140">

 </textbox>

 <hbox>

 <button id="ok" label="OK" default="true" accesskey="t"

 onclick="send($('twit-text'));"/>

 <button id="cancel" label="Cancel" default="false" accesskey="c"

 onclick="window.close();"/>

 </hbox>

 </vbox>

 </box>

</window>

initTwit() calls the shortener via Json, and fills the text box with the short url. This version points to my local Shortener instance.

What stroke me when building this window was the need to rebuild the plugin and relaunch Firefox everytime I changed it. And working with XUL windows is not obvious because of the way the layout works (v/hboxes). I ended up working with a Live Xul editor: http://ted.mielczarek.org/code/mozilla/xuledit/xuledit.xul

The part that I still miss is TDD: how to set up a healthy Javascript TDD environment to work with Firefox Plugins ? Investigating..

The (unfinished) code is here: http://bitbucket.org/tarek/urltotwit/src/tip/urltotwit%40tarek.ziade.org

If you are a XUL coder, please comment !

Conclusion

I need to finish the work on the part that sends the twitter via the user account. So far I found XUL pretty nice. It feels like working in an enhanced HTML language. I also need to read some about the security model and the development best practices.

On Python side, I am pretty happy with the Redis+Bottle small stack, and I'll probably add a small web UI and make it available on ziade.org. I'll be the third one at my Python User group to have my own url shortener ;) (Gawel, David'Bgk)

"Twitter Plugin example"

[]: http://tarekziade.files.wordpress.com/2010/04/picture-36.png http://bitbucket.org/tarek/urltotwit/src/tip/shortener/shortener/urlstore.py http://bitbucket.org/tarek/urltotwit/src/tip/shortener/shortener/main.py http://bitbucket.org/tarek/urltotwit/src/tip/urltotwit@tarek.ziade.org/

a small Distutils2 foretaste

Tarek Ziadé — Thu, 08 Apr 2010 10:52:00 +0200

Notice: If you didn't follow what's going on in the packaging world, you can read this.

The first public version of Distutils2 will be 1.0a1, and is scheduled for April 30th.

We are working on implementing the various PEPs, removing the bad bits inherited from Distutils, keeping the good ones, and adding the work that was done in Distribute, like the Python 3 work (and grabbing the good bits of Setuptools as well, like the test command.) I have also proposed a GSOC project, and I hope it will be picked by the PSF, to boost the work.

In any case, lots of tasks will not be ready for 1.0a1, and will be shipped later. But I think its important to release what we currently have since it's also a result of over 1 year of work I have been doing within the stdlib. As a reminder: I reverted this work in Python, making Distutils there pretty similar to 2.6's, and I've forked this work to create distutils2.

Anyways, this blog entry is focusing on describing some of the features you will find in Distutils2 today.

The code is located in the new Python Mercurial repository (http://hg.python.org/distutils2).

Distutils2 philosophy

One of the key point of Distutils2 is to try to provide tools for third-party projects like Pip, PyPM, etc. without forcing them to adopt the everything-is-a-distutils-command philosophy. People have suffered from this in the past, and this problem was probably one of the reason the Setuptools project has to patch Distutils' code all over the place.

I am therefore trying to think about Distutils2 as two distinct things:

a python packaging framework with tools anyone can use
a python package manager based on CLI commands people can use to build, upload, install packages.

The way 2. is working today in Distutils has been very controversial, and they are other tools out there based on other philosophies. I'll focus on making 1. as useful as possible for all tools, without forcing them to adopt 2. But also to enhance/fix 2.

There are key features to add in Distutils2 to make the CLI command system more useful. For instance, adding a configure command that will take all compiling options and generate a configuration file that can be consumed by other commands (build or install for instance) will probably make people life easier. This has been done by 4Suite, and this will be done in Distutils2.

In the Framework part, we already have two interesting modules you can already use:

The version module

The version module is the implementation of PEP 376. It provides among other things:
- The NormalizedVersion class you can use to work with PEP 376 version - The suggest_normalized_version() function you can use to try to transform any version into a PEP 376-compatible version. This function is based on what we have learned by looking at the projects released at PyPI.

So, if you need to compare versions, you can create instances of NormalizedVersion using the string representation of the version, and use any comparison operator.

from distutils2.version import NormalizedVersion

>>> NormalizedVersion('1.2a1') < NormalizedVersion('1.2')

True

In case the version is not PEP 376 compatible, the class will throw an exception

NormalizedVersion('1.2-a1')

Traceback (most recent call last):

...

distutils2.errors.IrrationalVersionError: 1.2-a1

And for these "irrational versions" ;), you can use suggest_normalized_version to try to convert them:

from distutils2.version import suggest_normalized_version

>>> suggest_normalized_version('1.2-a1')

'1.2a1'

>>> NormalizedVersion(suggest_normalized_version('1.2-a1'))

NormalizedVersion('1.2a1')

The metadata module

The metadata module implements PEP 345 (the new metadata) but also works with old metadata versions. It provides a DistributionMetadata class you can use to read or write metadata files (those PKG-INFO file you can find in any release). This class is compatible with all versions of Metadata:
- 1.0 : PEP 241 - 1.1 : PEP 314 - 1.2 : PEP 345

The PEP 345 implementation supports the micro-language for the environment markers, and displays warnings when versions that are supposed to be PEP 386 are violating the scheme.

Reading metadata

The DistributionMetadata class can be instanciated with the path of the metadata file, and provides a dict-like interface to the values:
>>> from distutils2.metadata import DistributionMetadata

 >>> metadata = DistributionMetadata('PKG-INFO')

 >>> metadata.keys()[:5]

 ('Metadata-Version', 'Name', 'Version', 'Platform', 'Supported-Platform')

 >>> metadata['Name']

 'CLVault'

 >>> metadata['Version']

 '0.5'

 >>> metadata['Requires-Dist']

 ["pywin32; sys.platform == 'win32'", "Sphinx"]

The fields that supports environment markers can be automatically ignored if the object is instanciated using the platform_dependant option. DistributionMetadata will interpret in the case the markers and willautomatically remove the fields that are not compliant with the running environment.

Here's an example under Mac OS X. The win32 dependency we saw earlier is ignored:
>>> from distutils2.metadata import DistributionMetadata

 >>> metadata = DistributionMetadata('PKG-INFO', platform_dependant=True)

 >>> metadata['Requires-Dist']

 ['Sphinx']

If you want to provide your own execution context, let's say to test the Metadata under a particular environment that is not the current environment, you can provide your own values in the execution_context option, which
is the dict that may contain one or more keys of the context the micro-language expects.

Here's an example, simulating a win32 environment:
>>> from distutils2.metadata import DistributionMetadata

 >>> context = {'sys.platform': 'win32'}

 >>> metadata = DistributionMetadata('PKG-INFO', platform_dependant=True,

 ...                                 execution_context=context)

 ...

 >>> metadata['Requires-Dist'] = ["pywin32; sys.platform == 'win32'",

 ...                              "Sphinx"]

 ...

 >>> metadata['Requires-Dist']

 ['pywin32', 'Sphinx']

Writing metadata

Writing metadata can be done using the write method:

metadata.write('/to/my/PKG-INFO')

The class will pick the best version for the metadata, depending on the values provided. If all the values provided exists in all versions, the class will used metadata.PKG_INFO_PREFERRED_VERSION. It is set by default to "1.0".

Conflict checking and best version

Some fields in PEP 345 have to follow a version scheme in their versions predicate. When the scheme is violated, a warning is emited:

from distutils2.metadata import DistributionMetadata

>>> metadata = DistributionMetadata()

>>> metadata['Requires-Dist'] = ['Funky (Groovie)']

"Funky (Groovie)" is not a valid predicate

>>> metadata['Requires-Dist'] = ['Funky (1.2)']

Stay tuned for some more fortastes of Distutils2 !

And if you want to contribute, join us in #distutils (freenode)

python-weave released

Tarek Ziadé — Tue, 06 Apr 2010 15:42:00 +0200

I did more experiments on Weave last week-end, and released python-weave at PyPI, which is basically Mozilla's Python client for Weave, plus some work I did to make it a Distutils-based project so it can be installed easily.

If you want to play with it, make sure you have Swig installed, then run:
$ pip install python-weave

You can also use easy_install or a plain setup.py install call. Once this is done, you will have a weave package in your Python installation, containing the Weave client.

My first goal with this client was to publish my Firefox bookmarks in an RSS feed online in my Pylons website (ziade.org), so I could share them with others. I have now a cron job on my server that gets the bookmarks on my Weave account every hour, and creates a RSS2 feed out of it.

The result is here : http://ziade.org/bookmarks.xml

It simplifies my bookmarking workflow and make it dead simple for other people to grab them (I can drop delicious and use exclusively FF to mark pages).

I have added this script as a small example in the source release, see: http://bitbucket.org/tarek/python-weave-client/src/tip/examples/bookmarks2rss.py

The next steps in my experiments will be:
1. display the xml feed in a Pylons view, sorting the bookmarks with their tags/keywords 2. have a way to mark some bookmarks as private, and some as public (maybe through folder organization) 3. add some social features on the top of it. (enable comments, etc.) 4. let my friends register their own Weave accounts (pointing to any Weave server, not necessarily Mozilla's), so they can publish their own bookmarks.

I realize that 4. is potentially dangerous because it can allow a third party server to retrieve and store data that was protected by Mozilla's Weave server. But one of the goal of Weave is to allow several devices to get the users' data and Firefox is just one client. There's a lot of potential to build applications outside Firefox too. I have seen that Weave has an OAuth service (not sure yet if its implemented), so I need to experiment that. But I'd also need to display other people' bookmarks. So maybe the best solution for this would be to have a single Weave account all people share. And allow a single Firefox to manage multiple accounts: a personal one and a shared one accessed by several people. That would resolve 2. :)

Anyways, this whole experiment is about building something similar to what delicious provides, but with a tighter integration in Firefox and on an open standard.

http://bitbucket.org/tarek/python-weave-client/src/tip/examples/bookmarks2rss.py

Hello PSF !

Tarek Ziadé — Tue, 06 Apr 2010 13:57:00 +0200

I was nominated then accepted as a member of the Python Software Foundation. Thanks Marc-André and others.

Anyone can contribute to the Python community without being a PSF member of course, so what's the point ?

It's quite simple: being a member means that you can vote and participate in the decisions made by the PSF for Python. This means a lot to me since I am devoting a lot of my free time to contribute to Python.

And... well... I am just proud ! :)

Playing with Weave from Mozilla Labs

Tarek Ziadé — Sat, 27 Mar 2010 23:12:00 +0100

Mozilla Labs has launched an interesting project called Weave some times ago. The idea is to provide an online service where your Firefox instance can save personal data, like bookmarks or passwords. You can grab a Firefox Add-On here and get started : http://mozillalabs.com/weave/

[][]

The personal data is encrypted by Firefox using a scheme based on AES 256 and private/public keys. See https://wiki.mozilla.org/Labs/Weave/Developer/Crypto for more details.

The Weave protocol is quite simple and is documented here :https://wiki.mozilla.org/Labs/Weave/API, and I have seen some Python implementations around for the server-side. Although Mozilla's implementation for their Firefox service looks like it's in PHP right now.

Mozilla also provides a small Python client for the protocol, which can be used to get and set data to a Weave server:

I was curious about Weave so I started to play around with the Python client, and ended up cloning it on Bitbucket, to add a few things to try it out. The original code in Mozilla's repository is not packaged (yet) and you just get a few Python modules. Although it has a command line that will let you grab some data from the server, but I wanted to try it out from my code.

So I've made it an installable Python package in my clone so I could install the client with Pip or easy_install.

Then I wrote a small class to try it to get my Firefox bookmarks:
import json

from weave.client import WeaveStorageContext, WeaveCryptoContext

_DEFAULT_SERVER = 'https://auth.services.mozilla.com'

class WeaveClient(object):

    def __init__(self, user_id, password, passphrase, server=_DEFAULT_SERVER):

        self.storage = WeaveStorageContext(user_id, password, server)

        self.crypto = WeaveCryptoContext(self.storage, passphrase)

    def _deserialize(self, payload):

        return json.loads(self.crypto.decrypt(json.loads(payload)))

    def get_bookmarks(self):

        for bookmark in self.storage.get_items('bookmarks'):

            bookmark = self._deserialize(item['payload'])[0]

            if bookmark['type'] != 'bookmark':

                continue

            yield bookmark['title'], bookmark['bmkUri']

if __name__ == '__main__':

    client = WeaveClient('tarek.ziade', 'password', 'passphrase')

    for title, url in client.get_bookmarks():

        print title, url

Here, a storage context class and a crypto context class are created with my login information, and the get_items() API gives me back a list of encrypted bookmarks. Weave organizes its data into collections, and bookmarks is just a label for one collection. There are other collections ([see https://wiki.mozilla.org/Labs/Weave/Sync/1.0/API#Collections][]), but Weave could be used to store and retrieve any kind of data.

If you want to replay my example, or play with the client, you can install it using my bitbucket repository like this:
$ pip install http://bitbucket.org/tarek/python-weave-client/get/tip.gz

or

$ easy_install install http://bitbucket.org/tarek/python-weave-client/get/tip.gz

Beware that this is not a release, just my custom clone :). But I'll probably ping the author to see if he has any interest in packaging his modules like I did, so we get it at PyPI.

What I am really looking forward about Weave now is how the project is going to grow and securely handle millions of user's data. This will probably end up in a very interesting infrastructure, with some similarities with projects like Dropbox or Tahoe.

"Weave Logo"

[]: https://services.mozilla.com/

Another GSOC Idea : a PyPI testing infrastructure

Tarek Ziadé — Sun, 21 Mar 2010 21:38:00 +0100

Last year I've proposed a GSOC project that no student picked up. I think the fact that it was titled "A Distutils Regression System" was not really appealing. I should have called it a "PyPI testing infrastructure" because that's what it is really. And I know today exactly how the system should work, so I can make a clearer proposal. The proposal page is here: http://wiki.python.org/moin/SummerOfCode/PyPITestingInfrastructure

So basically, the idea is to run a service that will receive update notifications everytime someone uploads a distribution at PyPI, to perform a few tests on it and send a report to the author by email (or display it on a website).

The tests we want to do are quite simple:
- does "python setup.py install" works or throw out some errors ? - what files are modified on the system upon installation ? - does the tests pass ?

For the latter, it supposes that we are able to detect and run the tests. They are no conventions yet (but the TIP list is working on this) but we can try out a few things on the source tree.

EDIT: for all the tests we want to run on a distribution, we should look at the CheeseCake project (Thanks Richard for mentioning it in the comments)

Getting update notifications is fairly simple now that Martin added a PubSubHubBub service at PyPI.

The core part of the project is to make sure the commands are run on a clean system. The easiest way to do it is to use Virtual Machines through a service like EC2: we can run the test, then rollback the VM state so it's back in a clean state. There are a few libs in Python to drive systems like EC2. We also need to secure the VM so the code that is run doesn't make network calls. If the project tests are relying on a network resource, those are bad tests anyways.

Last, but not least, if we want to get back interesting reports, we need to probe what's going on. It's simple at Python level, but we never know what can happen in a setup.py module. If a C module or an external program is called somehow in there, it could perform operations on the system we wouldn't be aware of. I think SystemTap is a great tool to probe for all operations at the OS level. I am not sure how to do it under Win32 though..

But having a working story under Linux would be already a great thing, so I am not worrying about that at this point.

So, I've entered the topic in the GSOC list, and I am willing to mentor this if a student wants to do this. I know Titus is also very interested in this topic, so we might work together on this, as co-mentors, which would be great. That's a topic we've discussed several times in the past and I think Titus wants a similar system. Mmm.. wait, we all do, don't we ? ;)

Distutils2 proposal for GSOC

Tarek Ziadé — Thu, 18 Mar 2010 15:24:00 +0100

EDIT 2010-03-28 : Started a detailed list here http://wiki.python.org/moin/SummerOfCode/Distutils2
The Google Summer of Code will be happening again this year, and I am going to propose a project under the PSF : Distutils2.

If you are a student and you are interested in building with me the replacer of Distutils, this is an opportunity to contribute to an Open Source project !

A finite and detailed list of tasks will be defined just before the proposals deadline, to avoid proposing a project that is too vague or to make it impossible to rate the student at mid-term. This list will be built at the last moment, because I am unable right now to know exactly what will be the state of Distutils2 this summer.

But here's a taste of what the student will work on:
- work on the implementation of PEP 376 - finish the implementation of the static metadata (that is the "no more setup.py" subproject ;) ) - work on a "configure" command that will generate a configuration file other commands can read back and use. - write a tutorial on how to use Distutils2 - promote and evangelize Distutils2 - This will consist of helping python project to switch/adopt Distutils2. We will define a list of targets and the student will be in charge of helping them switching to Distutils2 if they want.

If you are interested, be sure to join themailing list and stay tuned !

4 simple tips for wannabe remote workers

Tarek Ziadé — Thu, 18 Mar 2010 01:08:00 +0100

I've been working remotely for the past 6 years now. I can't believe it has been so long already.

Quick explanation. I've started to work remotely for good reasons : I have a 7 years old kid that lives with his mother in Dijon, France. Since I am not with her anymore, I need to stick around Dijon in order to see my boy as often as possible. Until he's old enough to visit me elsewhere by himself ;)

6 years earlier, working remotely was not really possible in France. Companies were reluctant to do it because they thought they were unable to control their employees if they were not physically present.

But in the software industry, a well-organized company has no reasons to be reluctant to remote workers. As long as the proper tools and the proper processes are set, a remote developer (and under some circumstances a manager) is more efficient if he works from home. And it makes the company save some money in most cases (less office space to rent, etc.)

Some XP/Lean methodologies are organized around the fact that the whole team is physically present but it's quite simple to adapt the rules to integrate remote people. Skype and other tools are working fine these days.

Of course, I don't think a 100% remote work is doable: you have to meet people physically on a regular basis. That's important because a meeting in person is better in specific phases in software projects. (launch and design phases)

Anyways, here's a condensed list of tips if you have never worked remotely and might do it.

I hope you will find them useful. And if you are a remote worker and have other tips, please comment this blog post.

Tip #1 - Everything you do, should be done in a VCS and an issue

tracker

For all aspects of a software project, it's important to use a Version Control System (I use Mercurial) even for your design notes or documentation. Trust me, that's the only efficient way for other people that need to work on the same parts you are working on to stay informed. Listing the history of a file in a (D)VCS gives the reader a good understanding of what's going on.

The only problem with this approach is that everyone in the team needs to know how to use a VCS. Marketers and other people that are not used to developers tools might be reluctant with this, but there are GUI tools now on the top of every good VCS system (like Tortoise).

Like the VCS, using a good issue tracker to track every task that is done is very important. And issues should be linked to VCS changesets and vice-versa.

I use for instance Bitbucket with Mercurial, and they have a nice plugin that will add a comment to an issue with a link to a changeset, if you use the proper tag in the changeset comment when you push your change. All issues trackers do this these days: Trac, Jira, etc.

Tip #2 - A daily over-the-phone quick meeting is a good idea

The best way to follow what other people are doing in the project is to have quick meetings every day. I don't think it's a good idea to have the meeting in the morning, because people don't start their work at the same time and it's better to discuss what's hot (==what you have done during the day.). So I like evening meetings better.

A short meeting on Skype or over the phone, that doesn't last more than ten minutes, is perfect. It will "force" you to briefly summarize what you have done during the day before the meeting starts. Doing these brief summaries is a great exercise for yourself too: it helps you understand where you spend your time.

And don't do these meetings over IRC or chat: a voice meeting is faster, and is a good opportunity to "humanize" your work day a little bit. Spending a whole day at home working alone can be depressing sometimes, so hearing your colleague voices feels good.

Tip #3 - Your new office open-space is an IRC room

The best way to communicate instantly with your colleagues is a room in the Internet Relay Chat or any other IM system that allows several people to talk. I've also discovered lately Etherpad, which is the perfect IRC companion : a real-time collaborative text editor.

Tip #4 - Separate your private life from your work

Some people think that working at home means you work less, but that's a myth. Most of the time, you work more because it's so easy to finish something you have started, when you are at home. It's not fun to work alone in the evening in a big open space once other people have left .

But working late at home is a very dangerous habit for your private life. Your wife and kids will start to hate your work. So be sure to have office hours and to separate physically your office from the rest of the house.

Montreal Packaging sprint wrapup

Tarek Ziadé — Tue, 16 Mar 2010 19:42:00 +0100

After the Confoo.ca conference, Yannick Gingras from the Montreal Python user group organized two

[caption id="" align="alignright" width="343" caption="Sprinting on TurboGears 2"][][][/caption]

small sprints at the Brasseurs Numériques headquarters (Digital Brewers). Yannick's company is called like that because he brews his own beers ! As a matter of fact, he prepared a beer for the sprints, that was really good (and quite strong.). His beer would probably beat some good Belgium beers on a blind test.

Sprint #1 : Turbogears

Chris Perkins led a Turbogears sprint Saturday. I am not very familiar with this framework but I am using Pylons a lot, and its the basis of Turbogears. Alice Bevan-McGregor was present, and could confront his ideas on web frameworkery with Chris since he created his own tool : WebCore. I've heard that they worked on making the Turbogears dispatcher a standalone library so it could be used by both frameworks.

I worked on my side on small packaging issues TG has. We fixed the latest TG 2.x beta custom package index that was broken (it was generating incomprehensible errors when installing TG, and I found out that the index pages in the TG PyPI were broken)

Next, I've added to easy_install a --no-find-links option to prevent links added by projects in their setup.cfg. This will prevent projects like Pylons to implicitely add links that easy_install reads. The effect is that some old version of some packages like nose were installed. This is not released yet.

If you were present to the sprint, please comment to tell us what you've done !

EDIT: Chris added more details on the TG sprint -- see the comments

Sprint #2 : Packaging

Monday evening we worked on packaging issues. I started the sprint by presenting the current state of packaging on a board. That took quite a while because it is not obvious to understand the packaging eco-system (distutils, setuptools, distutils2 and pip.).

Then I've listed possible tasks and people started to work.
- Yannick Gingras worked a bit on the Hitchicker's Guide to Packaging then worked on Distribute on Issue #133. - Ahmed Al-Saadi was pretty new to packaging so he worked on the guide and tried to catch up with the state of packaging (that's a real work :) ) - Alexandre Vassaloti worked on porting distutils2 into Python3. So basically, like Distribute, Distutils2 will be installable on Python 2 and 3, using the same source tree, and a 2to3 call upon installation. - Nicolas Cadou worked on PEP 345 support. He created a sample project that will be used in a functional test to validate that everything works. He eventually fixed some code in Distutils2 so it works with the PEP 345 DistributionMetadata class I've built during Pycon. I need to merge his work asap. - Matthieu Leduc-Hammel worked with me on PEP 345 support for PyPI. I've changed the postgres database to add the new fields, and Matthieu worked on PyPI UI. For instance, you will have a nice box on the project pages now that displays links from the Project-URL metadata field. I need to merge his work asap.

As a global note: Mercurial was the perfect tool for this sprint. I am able to merge people work in Distutils2 and other projects without all the repository access issues we usually get when we start a sprint. I am looking forward for a full Mercurial switch of Python, because this will boost contributions.

Thanks Yannick, Nicolas for the Sprint and the Beer ! Thanks Ubity for sponsoring the Packaging sprint with pizzas ! (they are looking for developers btw)

"Les Brasseurs Numérique - Sprinting on TurboGears 2"

[]: http://lh5.ggpht.com/_BBU4XN71nJo/S5_Ly74ivlI/AAAAAAAAAoM/00wkIxnidaY/s512/IMGP7327.jpg

The fate of Distutils - Pycon Summit + Packaging Sprint detailed report

Tarek Ziadé — Wed, 03 Mar 2010 23:27:00 +0100

The summit

I quickly posted an entry right after the Language Summit we held before Pycon in Atlanta. Basically, all the work I have being doing in Distutils and the PEPs we've prepared for the "big refactoring" will not be done in the standard library. Distutils in the stdlib trunk will be reverted to its current 2.6.x state.

I was quite despaired right after the summit. All the work we did during in the past year would not land into the standard library for 2.7, and all the pre- refactoring work I did, like making the test coverage decent, was going to be useless for the stdlib. Having that work included in 2.7 was one of my goal and I worked hard on making sure most of the important PEPs would be accepted before the feature freeze for 2.7 happened (the first beta, freezing new features, is in 4 weeks.)

I was even more depressed because I started to pull out of Distutils the "sysconfig" module and simplified the code in distutils, while making sure that the backward compatibility was kept.

I had a twenty minutes meeting with Guido after the Summit to clarify the situation and he helped me understand why this was the right path and worked with me on what to do next in the stdlib front and outside the stdlib.

Basically, a package that comes in the standard library has a foot in the grave (I am paraphrasing Guido here.). Its APIs is frozen, and people don't really expect nothing from it, but small new features and bug fixes. Refactorings are dangerous, if not impossible.

I have hit that problem in the past, in one of the 2.6 bug fix release, where I broke Setuptools compatibility because of an internal change I have made in a private method. The breakage was partly because Setuptools overrides a private method and partly because a public method that was not clearly documented was affected.

A few weeks ago the problem happened again : someone complained on python-dev because a declaration (an exception class) was missing from Distutils. An exception class was imported from the errors module into another Distutils module, but not used anymore there. And the module it was imported in didn't have an __all__ attribute. A third-party tool was importing the exception from the wrong module, so when I cleaned it up the third party module was broken.

So basically, any change I make in Distutils, even a simple cleaning, and worse, even a private method change, potentially breaks third-party tools.

You could argue that they should be careful in how they use Distutils, and never patch it or change its internal etc., and for edge cases like missing imports, just fix them.

But hey, Python 2.7 is out of the door in five weeks, and the user experience will be that Python has broken third-party libraries.

And the worse part of it : some of these libraries like Setuptools are not really maintained anymore and expect Distutils not to evolve anymore. But Setuptools is used nevertheless since it solves some problems Distutils doesn't. So the end user is the one that will suffer from those regressions.

In other words, project like Setuptools slows down the work we want to do in packaging because the current eco-system depends on a big, monolithic, messy pile of code that is located in different projects with different maintainers.

At this point, I understood that the easiest way for Distutils to evolve was to get away from this pile and grow on another namespace called distutils2.

Welcome Distutils2

If you have followed what is going on with packaging since last year, you might think: "distutils, setuptools, distribute and now distutils2 ?, oh no!!!"

But that is going to be for the benefit of everyone. See the roadmap in image below.

[caption id="" align="alignnone" width="385" caption="State of packaging"][][][/caption]

So basically, I have forked Distutils and renamed its package into Distutils2. The project is located in http://hg.python.org/distutils2 and the goal is to put it back into the standard library as soon as it reaches a state where it starts to be used by the community. Distutils will just die slowly, probably pulling Setuptools and Distribute with it.

The Distribute project is still important because it can help us releasing bug fixes or Python 3 support things today.

Distutils2 will be 2.4 to 3.2 compatible and will get back from Distribute the good bits and implement the PEPs that were accepted lately PEP 345 and PEP 386.

And I am happily removing old code we don't want/need anymore without worrying about backward compatibility. Yeah !

The packaging sprint

After the conferences, we started a packaging sprint and I was surprised because many people showed up and worked on the topic.

[caption id="" align="alignnone" width="484" caption="Brainstorming on PEP 376"][][][/caption]

We created a few teams to work on PEP 376, mkpkg, the Hitchicker's Guide to Packaging (HHGP), and Distribute. I won't say the name of each person, I am too scared to forget someone :D.

PEP 376

Like last year, people from various distributions (Fedora, Ubuntu, Debian) and I worked on packaging issues. They worked on PEP 386 last year mainly (the versioning scheme) and focused on PEP 376 this year. This PEP is about setting up a standard for installed packages, and an installation index that allows to query what packages are installed, and get their metadata. In extend, it provides an uninstall feature. The goal is to have a standard for all package managers of course.

One part of the PEP is about describing the data files that are installed with the project (like configuration files or documentation) so they can be removed and maybe relocated. The group focused on describing the files a project contains in a static way (in setup.cfg) with variables that can be expanded an installation time (which values are provided by Python, but globally configurable by the OS packagers.)

We did quite some work and brainstorming on this, and even focused on removing setup.py ! A fully static description of a project (metadata+file list) is the key to a better packaging tool !

Expect a proposal soon on distutils-SIG, for PEP 376. If you want to have a look, the draft proposal is here: draft.

mkpkg and Distribute

We had two one-member teams at some point, so I can name them without being scared of forgetting someone ;)

Sean worked on a nice add-on for Distutils2, a script that builds a setup.py file after asking you a few questions. He blogged about it. so I don't need to get into further details :)

Noufal worked on fixing some bugs in Distribute. We should do a release at some point.

The HitchHicker's Guide to Packaging

Another group worked on the guide. The goal is to provide some help for people that want to package things today and are despaired with the sparse documentation they can find. Which tool to use ? how ? when ?

The work done was quite amazing, look at it : http://guide.python-distribute.org

I have spoken with Georg Brandl to see how we could move it to docs.python.org and make it grow there.

Distutils2 coding

Besides PEP 345, I worked on making Distutils2 work for 2.2, 2.5, 2.6 and this is now over. I have also almost fully implemented PEP 345 in there.

There's now a metadata module with a dict-like DistributionMetadata class that knows how to read and write PKG-INFO files. It also knows how to interpret the micro-language we've defined: the environment markers.

Last, I've added the PEP 386 version module : version.py. This one is used now by the metadata class to control versions.

More to come !

Next sprint at Confoo.ca

The next packaging sprint will happen in Montreal, where I am going as a speaker next week. We will continue the worked started, so stay tuned.

"State of packaging"

[]: http://guide.python-distribute.org/introduction.html#current-state-of-packaging "Brainstorming on PEP 376" []: http://lh3.ggpht.com/_BBU4XN71nJo/S45d_jRywdI/AAAAAAAAAl4/ggtwGaVkKwU/s720/IMGP7237.jpg

Pycon slides + answers to GM questions

Tarek Ziadé — Sat, 20 Feb 2010 14:18:00 +0100

I did my presentation yesterday, and it seems like people enjoyed it, from what I've heard in the halls and seen online. I am very glad about this feedback because packaging is not the sexiest topic in programming conferences in general.

Anyways, here are my slides : http://ziade.org/slides/pycon-2010-state-of-packaging.pdf

EDIT: And here's the video on blip : http://bit.ly/dlPB2X

And as promised, I've answered to all the questions people asked in the google moderator :[http://www.google.com/moderator/#16/e=4395 .][]Thanks to everyone that has participated.

I am now going to enjoy the rest of the Pycon event and I am looking forward to the sprints.

Python Language Summit -- Summary of the packaging track

Tarek Ziadé — Thu, 18 Feb 2010 19:03:00 +0100

Here are quick wrapup on what has been said during the packaging session in the language summit that is happening today at Pycon.

The four major points are:
1. The implementation of the accepted PEPs that have been done lately will not happen in Distutils but in a new package in the Distribute project (so logically in a "distribute" package). This resolves backward compatibility issues: new features will be under the "distribute" namespace. 2. Distribute will stay a third-party package and will be integrated in the standard library once it has enough support and feedback from the community. So this could happen in 3.3 (or 2.8 ;) ). Some part that are useful for the existing distutils package might be added in the stdlib today. But the idea is to stop adding features in distutils and focus on distribute. 3. The Hitchhicker's guide to packaging is going to be moved into the Python repository, so it becomes part of docs.python.org. It's not finished yet but it'll grow there. 4. Ian threw the idea to have virtualenv as a core feature, but he's not sure how yet. Some brainstroming on this should happen during Pycon.

Ask questions about packaging

Tarek Ziadé — Thu, 11 Feb 2010 22:20:00 +0100

I have started a Google Moderator series to gather questions people would like to ask about Python packaging. I will answer to the most popular questions at the end of my talk at Pycon.

Here's the link : [http://www.google.com/moderator/#16/e=4395][]

Please, add some questions, and vote !

PEP 345 and 386 accepted -- Summary of changes

Tarek Ziadé — Wed, 10 Feb 2010 10:24:00 +0100

Several PEPs were accepted this month by Guido, and among them PEP 345 and PEP 386, which are about Python packaging. I am summarizing in this entry the main changes we've made.

PEP 345 - Metadata v1.2

This PEP is about the Metadata that gets added in your project when you build a distribution using Distutils or a Distutils-based tool. Those are fields like "name" or "version" you pass as options in your setup.py file. They eventually land in a PKG-INFO file then at PyPI and in each Python's site-packages where the project is installed. They are also useful when your project gets repackaged by an OS packager.

New fields

PEP 345 adds some new fields :
- Maintainer : a string containing the maintainer's name - Maintainer-email : a string containing the maintainer's e-mail - Requires-Python : Python version(s) that the distribution is guaranteed to be compatible with. - Requires-External : Describes some dependencies in the system that the distribution is to be used - Requires-Dist : String naming some other distutils project required by this distribution. - Provides-Dist : String naming a Distutils project which is contained within this distribution. - Obsoletes-Dist : String describing a distutils project's distribution which this distribution renders obsolete - Project-URL : String containing a browsable URL for the project and a label for it, separated by a comma.

Maintainer and Maintainer-email were added because people were confused about the maintainer and maintainer_email options they have in Distutils. Before PEP 345, if you used the author and the maintainer fields, one was dropped and one was kept to fill the Author metadata field.

Requires-Python was added so people could list the Python versions their project is compatible with. We do have classifiers already for that in the Trove classifier, but this new field is more than a simple field : the version string that is used supports a syntax that makes it possible to describe any set of Python versions. See the version specifier of the PEP.

Remember the requires.txt metadata Setuptools introduced together with the install_requires option ? Requires-Dist is comparable to this and will let you define dependencies on other Python projects. Distutils had the Requires field, but it was defining dependencies at the module level and this was never really used by the community. So Requires is gone in 1.2. Last, Requires-Dist is using a version comparison scheme described in details in the version specifier section.

Provides-Dist gives you the ability among other things to reorganize your project names or to distribute a subproject in several projects. For instance, the project called ZODB used to include the project transaction that is also distributed as a standalone project now. If ZODB is installed, and if you need transaction, you won't have to install it again.

Obsoletes-Dist will let you make sure a project that is incompatible with your project is not installed at the same time.

Project-URL is useful to provide a list of URLs for your project like a browsable repository or a tracker. The goal will be to add a small box on PyPI project pages for these URLs, so developers can emphasize them. This will hopefully address the complaints PyPI had in these past months when the comment system was added.

Some small changes were made on existing fields. One important change is on the Description field (long_description in setup.py). Before PEP 345, once this value was written in the metadata file, its empty lines and spaces at the beginning of lines were truncated. In other words, if a tool was reading back this value, its reSTrucured syntax was broken. This is no longer the case.

Once I've finished implementing PEP 345, you will be able to read back a project's metadata usig the DistributionMetadata class. See an example.

Environment Markers

When Pip wants to install all the dependencies a project requires, it has to follow these steps:
1. download the project from PyPI 2. execute a Distutils command on setup.py, like egg_info, to get the metadata. And in particular the list of requirements

This is mandatory because the metadata are not statically defined and the developer might need to run some code in his setup.py to know what dependencies are required, depending on the target platform.

For example:
if sys.platform == 'win32':

        install_requires = ['pywin32']

    else:

        install_requires = []

In other words, there's no way to get the metadata without running third-party code. Environment markers fix this issue in most cases by providing a micro-language that can be used at the field level. At the end, you can write things like that in the metadata:
Requires-Dist: pywin32 (>1.0); sys.platform == 'win32'

And Distutils will provide a tool to parse and execute this expression, so you know if your target platform have to use this metadata field. Meaning that Pip or other tools will be able to read metadata of a project without running any third party code. For example to get all the dependencies of a project depending on the target platform just by querying PyPI.

See all the details in the section in the PEP.

PEP 386

Throughout all the PEP 345, we had to compare versions. To be able to perform this, we need to have a common standard for versions numbers. That's what PEP 386 was written for.

The idea is not to force people to version their project using this version scheme, but rather to provide a scheme that is good enough for interoperability and that is human readable.

PEP 386 provides this scheme (in pseudo-format) :
N.N[.N]+[{a|b|c|rc}N[.N]+][.postN][.devN]

The corresponding regular expression is:
expr = r"""^

    (?P<version>\d+\.\d+)         # minimum 'N.N'

    (?P<extraversion>(?:\.\d+)*)  # any number of extra '.N' segments

    (?:

        (?P<prerel>[abc]|rc)         # 'a' = alpha, 'b' = beta

                                     # 'c' or 'rc' = release candidate

        (?P<prerelversion>\d+(?:\.\d+)*)

    )?

    (?P<postdev>(\.post(?P<post>\d+))?(\.dev(?P<dev>\d+))?)?

    $"""

The scheme handles these cases:
- pre-releases - development versions - final versions - post-release versions - development versions of post-release versions

Here are some examples:

from verlib import NormalizedVersion as V

>>> (V('1.0a1')

...  < V('1.0a2.dev456')

...  < V('1.0a2')

...  < V('1.0a2.1.dev456')

...  < V('1.0a2.1')

...  < V('1.0b1.dev456')

...  < V('1.0b2')

...  < V('1.0b2.post345')

...  < V('1.0c1.dev456')

...  < V('1.0c1')

...  < V('1.0.dev456')

...  < V('1.0')

...  < V('1.0.post456.dev34')

...  < V('1.0.post456'))

True

This can look utterly complex to you, but in fact there are good chances that your version scheme is already compatible with this one. We've tested in on PyPI, and 88 % of the projects' distributions were recognized.

The suggest_rational_version function

Let's face it: there are hundreds of valid versionning schemes that are not compatible with what we've done. So we are adding a function called suggest_rational_version that can be used to transform a version that is not "PEP-386" compliant into one that is compliant.

This does a number of simple normalizations to the given string, based on an observation of versions currently in use on PyPI.

Given a dump of those versions on February 10th 2010, the function has given those results out of the 9066 distributions PyPI had:
- 8058 (88.88%) already match NormalizedVersion without any change - 795 (8.77%) match when using this suggestion method - 213 (2.35%) don't match at all.

And here's an extract of the 2.35% unrecognized scheme:
- 0.2-grigoropoulos - 0.1-alphadev - working proof of concept - bzr14 - 1 (first draft)

In other words, they are unusable anyways. If you want to try this on your own versions, grab the code at http://bitbucket.org/tarek/distutilsversion/. And if you version doesn't match at all and you think its a mistake, let me know so we can work your case.

Conclusion + my rant on packaging

I have started working seriously on packaging issues a year ago. And I kept on hearing complaints on how packaging sucks hard. That was frustrating since some folks and I were working hard to change things (and we still do). I kept seeing the same people ranting about packaging, and most of the time they were not willing to help around. I guess that's just me being naive, but let me say it one more time :)

If you don't like how packaging works in Python, stop complaining and come in the Distutils mailing list, or in #distutils on freenode and help us !

I am now really glad that these PEPs were accepted by Guido because they are the first milestone we had to reach to improve things for real in packaging. Python 2.7 is going to make a big jump forward in this area !

Next milestone we are trying to reach for 2.7 is to finish PEP 376 (uninstall feature, querying installed packages, etc)

improving Python's getpass module

Tarek Ziadé — Sat, 06 Feb 2010 23:51:00 +0100

UPDATED see the end.

The Python standard library has a module called getpass you can use to get a password from the prompt:

import getpass

>>> password = getpass.getpass()

Password:          <-- non-echoed typing here

>>> print password

worked

That's nice, and Distutils uses it to ask for your password when you register or upload a release at PyPI, if it's not found in your pypirc file. But this is annoying to type and type again your password, so you end up saving it in clear text in pypirc. Thats sucks. And the getpass module gets pretty useless if you want to store and retrieve passwords from other places than the user brain.

But wait... we have the Keyring project now.. what about making getpass use Keyring so you can safely read a password from your favorite keyring (Keychain, KWallet, etc..) ?

I've started to write a new getpass module that could do this. But instead of adding a keyring dependency in it and struggling for months (years) to get the addition of Keyring into the stdlib, I have made getpass pluggable.

In my improved version, you can define in a small configuration file (getpass.cfg) an arbritrary function that will be used by getpass for the getpass.getpass API. Here's such a file:
getpass

  getpass-backend = keyring:get_pass_get_password

Here I am configuring get pass to use the get_pass_get_password function from the keyring package. That's a function that gets installed in your Python once Keyring is installed.

This function has the same interface than the default getpass.getpass API and calls keyring.

The modified getpass module is here: http://bitbucket.org/tarek/getpass/

And works against the current trunk of Keyring.

What I would like to do now is to propose the small changes I've made in Python's getpass for inclusion in the stdlib. They are backward compatible changes and offers a simple, yet powerfull way to extend getpass without adding any other module in the stdlib. And maybe adding a setpass in there too would make sense.

Update from python-ideas

So I brought up the idea in the mailing lists and it turns out (thanks to the folks at Python-ideas) that the way I want to introduce this feature is not good for these reasons :
1. getpass is just a function that is used to get a password from the prompt. you can consider it as a potential, dummy backend for Keyring for example. Trying to make it extendable just denaturates its original purpose. 2. the only use case right now in the stdlib is for Distutils, so it doesn't really make sense to have a keyring in there. People can just use the Keyring project directly. 3. Now if other parts of the stdlib have the same need, it will be time to think about how it could be included in the stdlib level rather than in Distutils.

So, I'll work for its inclusion at Distutils level rather thah on getpass level.

Simple command-line vault : CLVault

Tarek Ziadé — Mon, 01 Feb 2010 17:26:00 +0100

I am pretty happy with the Keyring project. I use it now everywhere in my Mercurial-based projects, thanks to [mercurial_keyring][].

There's one other place I've started to use it: I needed a simple command-line based tool to save passwords and read them. The tool I've used so far was KeePass, but I need to run it then click on its UI. This is time consuming when I simply want to push a password in the clipboard to use it to unlock something.

So I've wrote these two very simple scripts that use Keyring to store and retrieve passwords
$ clvault-set blog

Set your password:

Password set.

$ clvault-get blog

The password has been copied in your clipboard

The code that copy the passwords in the clipboard was tested under Mac OS with its Keychain, but should work under Windows and Linux as well.

I think these scripts can be useful for people like me who spend most of their time in a bash prompt when they are not in Vim or Emacs. So I created a project called CLVault.. You can grab it at the PyPI: CLVault PyPI page

or install it like this with Pip:
$ pip install clvault

Let me know if it's useful to you !

Pycon packaging sprint topics

Tarek Ziadé — Mon, 11 Jan 2010 22:20:00 +0100

Pycon is coming soon. Here's a list of possible topics I would like to work on during the sprint:
1. adding the features in Distutils I've mentioned in my [earlier post][]. 2. work on the standalone release of Distutils, and make sure it works with 2.4, 2.5, etc so it can be distributed at PyPI. There are already installable nightly builds by the way. 3. Finish the buildbot work so Distutils is tested with more projects from PyPI 4. Continue the work on Distribute, and specifically the work on 0.7:
1. finish the develop command that would work with non-eggs formats 2. finish the configure/build/install command where all options are computed and saved by the configure command

Fix plenty of issues in the tracker
work on a geolocalisation feature for Pip, so the nearest mirror can be picked
...

Anyone interested in packaging sprinting at Pycon ? Let me know !

Fixing packaging terminology confusion

Tarek Ziadé — Thu, 07 Jan 2010 21:30:00 +0100

Edit: the discussion is still going on, so I've probably blogged that a little bit early (I was excited about it ;)). Stay tuned for the final output.

Brad Allen launched a thread in Distutils-SIG about packaging terminology confusion. In particular the usage of the word "package" in our community. Part of the confusion is because of the meaning of this word in Python (that is, a directory containing one of several Python modules, with a special one named __init__.py) and in some systems like Debian (there, a package is a distribution file for a library or an application).

This confusion was present in PEP 345 (which was started years ago, so that explains it) - and is present in Distutils documentation and also in PyPI (That is: Python Package Index).

I really like Tres Seaver's definitions, because they match prefectly the reality:
- package means a Python package, (directory intended to be on sys.path, with an __init__.py. We *never* mean a distributable or installable archive, except when "impedance matching" with folks who think in terms of operating system distributions. - distribution is such a distributable / installable archive: either in source form (an 'sdist'), or one of the binary forms (egg., etc.). Any distribution may contain multiple packages (or even no packages, in the case of standalone scripts). - project is the process / community which produces releases of a given set of software, identified by a name unique within PyPI's namespace. PyPI manages metadata about projects (names, owners) and their releases. Every real project has at least one release. - release is a set of one or more distributions of a project, each sharing the same version. Some PyPI metadata is specific to a release, rather than a project. Every release has at least one distribution.

And I really like Martin's proposal in the thread (in Catalog-SIG since it was cross-posted): "PyPI would then be the Python Project Index."

I'll fix Distutils documentation on my side accordingly, as well as the guide we are building. Let's promote these definitions :)

Possible new features for Distutils 2.7

Tarek Ziadé — Thu, 07 Jan 2010 06:46:00 +0100

While PEP 345 and PEP 386 are waiting for the final approval, I am back at work on Distutils code work, PEP 376, Distribute, and the HitchHicker guide to Packaging. The latter is growing faster than I have expected, thanks to the contributions of John Gabriele. It has quite some content already. I think the guide is an important task, and I'll try to focus on it in this first trimester.

Distutils 2.7 new features

Python 2.7 first beta version is around the corner, and once it's reached we can't add new features. So, besides the code that will be changed if the PEPs we worked on at Distutils-SIG are accepted, here's a list of small features I'd like to introduce in Distutils:
- a test command, that just uses the new [unittest discovery script][] to run unittest-compatible tests. - a new option for sdist called 'extra_files', that will allow to list extra files to be included in the distribution. These files will not be installed by 'install', just be part of the distribution. This will allow including files like CHANGELOG, etc.. without having to use a MANIFEST template. - a very basic pre/post commit hook for the install command. These hooks will be deactivated when any bdist_* command runs install to create the binary tree. Now for bdist_rpm own hooks, I guess the best way would be to make install consumes the same two options than bdist_rpm (pre-install, post-install) so a project will be able to define a hook that is used by RPM and/or python setup.py install

If you think about something that should be added in 2.7, speak up !

New year's Python meme

Tarek Ziadé — Mon, 28 Dec 2009 16:33:00 +0100

Here's a short, 5 questions, 2009 Python meme. Copy-paste the questions, and blog your answers !

1. What's the coolest Python application, framework or library you have discovered in 2009 ?

Twisted. I have used Twisted in small occasions in the past, but since I have changed my job, a fair amount of my daily work is around the Twisted framework. I also program in plain Python for some needs, and I use Pylons to build our website, but Twisted is the biggest part of our current application.

2. What new programming technique did you learn in 2009 ?

Network programming. I was a web programmer before, and now I am learning how to write programs that need to scale, to work using several computers. It's pretty exciting, and sometimes pretty exhaustive because when some bugs occur, they are often leading to snowball effects. The code needs to be more rigourous, and I need to know exactly what is going on when interacting with memory or I/O.

3. What's the name of the open source project you contributed the most in 2009 ? What did you do ?

Python. I am a commiter since December 24th 2008 (my birthday ;) ). And I have been working almost exclusively on Distutils. I did around 200 commits in the trunk (not counting the backports and forwardports on the branches), and I have been mainly correcting bugs and improving Distutils code coverage, which I raised from 18% to up to 75%.

I have also added some minor new features. The most important ones are or will be added in the What's new file hopefully. I also did a lot of work in the PEPs to prepare the future of packaging in Python, and wait for that work to be finished, to start to change Distutils more. While I did more commits in Distribute, Distutils was the most time-consuming project.

4. What was the Python blog or website you read the most in 2009 ?

Python Reddit. And I think I am not alone in that case. 90% of my blog hits come from Reddit :-)

5. What are the three top things you want to learn in 2010 ?

I'd like to learn how to program the C part of the CPython core, and eventually to try to contribute some patches for CPython. I would also like to write my own toy programming language, just for fun. Last, I would like to learn Japanese. And all this comes after Distutils, Distribute, etc, so I am pretty sure I won't find the time...

Read other people meme now (If you did one, let me know, I'd like to link it here):
- Daniel Greenfeld - Lennart Regebro - Martijn Faassen - Jorgen Modin - Chris McDonough - Arc Riley - Brian Jones - Corey Goldberg - Alex Clark - Thibaud "Teebes" Morel l'Horset - Matthew Wilkes - Rok Garbas - Michael Foord - ...

Starting The Hitchhiker's Guide to Packaging

Tarek Ziadé — Sun, 13 Dec 2009 13:35:00 +0100

DON'T PANIC !

Following-up various discussions we had about the need to provide a guide for people that want to understand our rather complex packaging eco-system, and how they can use it,

I've decided to start The Hitchhiker's Guide to Packaging. The goals are:
- consolidate in one single Sphinx-based powered website, tutorials and documentation on how to use the various packaging tools and how to package, release and distribute your Python application. - provide guidelines to the community that are up-to-date with the latest work done in packaging - see if it can be added in the Python documentation itself at some point when its mature, as Georg Brandl suggested.

If you want to join this documentation effort, by adding your existing documentation to this guide, by writing new one, or by helping on copy editing or reviewing, let me know (#distutils on Freenode, or by mail)

The project repository is hosted on bitbucket: http://bitbucket.org/tarek/hitchhiker-guide-packaging/

I've also created a google group: http://groups.google.com/group/packaging-guide

Distutils and Distribute status (part #1)

Tarek Ziadé — Wed, 18 Nov 2009 09:36:00 +0100

Someone told me on IRC that it's currently hard to follow what's going on in the packaging front. The truth is that it's almost impossible if you don't read all mails posted in Distutils-SIG.

So here's a quick wrap-up that can save you some time if you are not reading Distutils-SIG.

PEP 345 - Metadata 1.2

We are almost done with the update of PEP 345. This PEP is describing the Metadata fields for a distribution, that get added in the file named PKG-INFO. This file is inserted in all your distribution and also published at PyPI. It's the ID card of your project.

We are adding these fields in the metadata:
- Maintainer : the maintainer's name - Maintainer-email : the maintainer's email - Requires-Python : What Python versions are compatible with this distribution - Requires-External : A list of external dependencies, like "libpng", "libxslt" - Requires-Dist : A list of Python dependencies, from the names founded at PyPI. like "zope.interface" - Provides-Dist : A list of additional distribution names this distribution provides (as a complement to the one provided in "Name") - Obsoletes-Dist : A list of Python dependencies that are incompatible with the current distribution

Another important change is environment markers. An environment marker is a marker that can be added at the end of a field after a semi-colon (';'), to add a condition about the execution environment.

Examples:
Requires-Dist: pywin32, bar > 1.0; sys.platform == 'win32'

Obsoletes-Dist: pywin31; sys.platform == 'win32'

Requires-Dist: foo; os.machine == 'i386'

Requires-Dist: bar; python_version == '2.4' or python_version == '2.5'

Requires-External: libxslt; 'linux' in sys.platform

This will allow developers to define different conditions depending on the target platform. Moreover, this will allow tools like Pip to get a list of all dependencies for a given project and a given platform just by querying PyPI, and with no downloads or build required !

Last, for all the fields that manipulates versions, PEP 345 will use the version scheme described in PEP 386.

PEP 386 - Version scheme

We've designed in PEP 386 a version scheme that works with most Python software we know about. This version scheme comes with a new version comparison algorithm that will be provided by Distutils.

The scheme is in pseudo-regexpr (read the PEP for more details):
N.N[.N]+[abc]N[.N]+[.postN+][.devN+]

Don't be afraid ! It looks complex but it's not. The apparent complexity is due to the fact that we need to be able to work with development versions and post-release versions.

There are good chances that your project already works with this version scheme. If you want to give it a shot, there's a prototype you can play with in an external repo here: http://bitbucket.org/tarek/distutilsversion/

PEP 376 - Installation standard

PEP 376 is quite completed now. We have our "standard" for site-packages, we know how to query installed projects, and how to remove them.

The discussions are now focusing on the "data" problem. Which is : how to describe in Distutils, in a more elegant way, the data files you are using, such as images, man files etc.

This is required to provide to developers more control on how their data files are installed on the target system, and to the packagers more tools to re-package a Python distribution.

Wolodja Wentland has been doing a lot of work in this area and leads this "data" effort. You can follow the discussion on this work in the Python wiki, starting at: http://wiki.python.org/moin/Distutils/DiscussionOverview.

PEP 382 -Namespaces packages

Distribute comes with a namespace package system, that allows you to have packages under the same namespace, spread into several distributions.

That's what Plone and Zope use to be able to release all those plone.* and zope.* distributions.

Martin von Loewis proposed to implement it in Python, and this is described in PEP 382.

We are now waiting for Martin to implement it, and are ready to drop in Distribute 0.7.x the namespace feature in favor of supporting the PEP 382 one.

Distutils redesign discussions

One thing that makes Distutils a bit hard to work with, is how commands are designed. David Cournapeau (from the Numpy project) gave us an example of a use case that makes it hard. He basically needs to run the "build" command knowing the finalized options from the "install" command.

In other words, when you call something like :
$ python setup.py install --prefix=/some/place

The install command will use the prefix option to cook some other options. The build command that needs all the options needs in that case to look over the install command to get the values.

This is not optimal because it means that a build command depends on an install command to run. It also makes options redundants from one command to the other.

The solution we are going to try is to create a new command, called configure, that will be in charge of building a file with all the options that are required by the build command and the install command.

This is not new. It has been implemented years ago in 4suite, and it's the philosophy behind tools like scons, etc: a configure/make/make install principle applied to a Python project.

This redesign is going to occur in Distribute 0.7. Once it's ready, if the community has tried it and gave us positive feedback, I'll push it in Distutils.

It might happen before Python 2.7 is out, it might not.

virtualenv and zc.buildout now with Distribute included

Tarek Ziadé — Sat, 07 Nov 2009 01:36:00 +0100

We are still actively working in fixing all the remaining bugs in Distribute (our Setuptools fork).

But we have reached an important milestone this week: both virtualenv and zc.buildout now comes with an option to switch to Distribute.

In virtualenv:
$ virtualenv --distribute ENV

In zc.buildout, using its bootstrap.py file:
$ python bootstrap.py --distribute

Enjoy !

For those who may wonder why they should switch to Distribute over Setuptools, it's quite simple:
- Distribute 0.6.x is a drop-in replacement for Setuptools - Distribute is actively maintained, and has over 10 commiters - Distribute 0.6.x offers Python 3 support !

And if you still struggle with packaging issues, the place to hang around to get some help is the #distutils IRC channel on Freenode.

First Distribute mini-sprint (online)

Tarek Ziadé — Fri, 16 Oct 2009 19:36:00 +0200

While we are working at fixing bugs in 0.6.x, we are organizing a first online coding sprint for the 0.7.x series, and for various community tasks.

What are we going to do ?

We need to finalize the 0.7.x renaming and splitting work
We also need to define a more detailed roadmap for each splitted package (new features, deprecations, etc.)
We need to build a better test environment, and see if we can set up a buildbot for our work
We need to write a tutorial explaining how Distribute can be used in a project. (Python 3 support, Moving from Setuptools to Distribute, etc)

Who can join ?

There are no particular level required to participate, but we won't be able in this session to teach to participants how to work with Mercurial, Python, etc. But if you want to help in the Documentation part or testing, it's as important and useful as the code and we are welcoming you.

Last, if you are just interested in evaluating Distribute for your project, you are welcome to join, we will help you.

Where ?

Online, on IRC - Freenode - #distutils channel

When, how long and how ?

Possible times (all CEST +0200 UTC) :
- Sunday October 18th - 6 pm - Tuesday October 20th - 7 pm - Wenesday October 21th - 7 pm - Friday October 23th - 7 pm

The sprint will be held for a minimum of 3 hours, and people will be free to leave when they want.

The most important thing is to be present on time when the sprint starts, so we can make groups and kick it off. If you want to join, add your name here: http://doodle.com/bfgv3yi3pi48buuv

top-posting, mobile devices, mail threads and semantics

Tarek Ziadé — Sun, 11 Oct 2009 10:55:00 +0200

There's an interesting discussion on python-dev about how hard it is to follow a thread when people are starting to top-post, meaning they are answering by quoting the whole text and putting their answers at the top. I even got bitten by someone once because I was top-posting (don't get me wrong, he was right about it, I was just not fully aware of the problem)

First of all, if you use a mobile device, there are good chances that the mail application you are using doesn't give you the choice : it will quote the text for you and will let you answer at the top. That's how it works on my android (HTC) phone and I couldn't find a way to change it. I am expecting mail apps on mobile devices to improve on this.

But this problem reflects how hard it is to follow a thread with +100 answers. Worse, depending on the way people are quoting to provide an answer for a specific part of a mail, some people will just stop reading it.

Gmail is doing a pretty good job to reduce this problem, because it will automagically hide old content and you will only see new content on every new mail in the thread.

But some people are not using Gmail for good reasons.

The other problem with gigantic threads at python-dev is that they often end up in a tree of several sub-threads, making it very hard to follow what's going on if you don't sort mails by threads in your client. And again, this is not possible in some mail clients.

I was very frustrated about this problem on gigantic threads about Distutils because I was seeing people "lost" in a branch of the thread, asking questions that were answered at the other end of the tree.

So how could we improve on this ?

Imho, mail threads are not suited for design discussions. I think a way to improve the situation could be to link mails by keywords.

Everytime you answer a mail, instead of quoting some of its text, you provide some keywords related to the topic you want to discuss, and you just type a plain answer. Your answer and the original mail will then be linked through a semantical relation, like a RDF triplet. These keywords could be new headers in the mail that is sent.

Let's imagine all mail client are able to sort and browse the threads by keywords, and to list the used keywords on the side of the thread. Meaning that everytime you send a mail, you can pick one of the keyword that was already used, to limit the number of keywords.

Does that make any sense ?

Distribute 0.6.4 released - zc.buildout support

Tarek Ziadé — Sat, 10 Oct 2009 22:53:00 +0200

We've juste released Distribute 0.6.4.

This version is now fully compatible with zc.buildout, meaning that you can use Distribute in your buildout transparently as long as you use our special zc.buildout bootstrap file located here : http://nightly.ziade.org/bootstrap.py.

We've also added in [Jannis' upload_docs command][], that allows a project to upload its Sphinx based documentation to PyPI.

As a matter of fact, Distribute is using it now and you can reach its documentation at : http://packages.python.org/distribute.

If you have any feedback, or want to help around, drop us a mail at distutils-sig or come in our IRC channel in #distutils (freenode)

Python 2.6.3 and Distribute.

Tarek Ziadé — Sat, 03 Oct 2009 14:20:00 +0200

Python 2.6.3 is out, will a lot of bugs fixed. I had my share with Distutils and fixed quite a few, and 2.6.3 is looking very good so far !

Just a quick note for Setuptools users: you might bump into a problem if you provide a C extension. The setuptools code makes some assumptions on how and in which order the Distutils build_ext API are called. It also overrides some of these API to do some internal extra work. In other words, the way Setuptools patches Distutils makes it very sensitive to any internal Distutils code changes. In this particulare case you might have this bug:
File "...setuptools/command/build_ext.py", line 85, in get_ext_filename KeyError: 'xxx'

The fix is quite simple, it can be done by the end-user or in your package (which is better of course).
- In your package : use "Distribute >= 0.6.3" distribution instead of the usual "Setuptools == 0.6c9" distribution in you dependencies list. The code remain unchanged and you can still "import setuptools" and have it working fine. - As an end-user: just do a Distribute installation and your fine "(sudo) easy_install Distribute"

Hang in #distutils on Freenode, or drop a mail in distutils-SIG in case you have a problem.

Just to make things clear: The Distribute 0.6.x series is a mirror of Setuptools 0.6c9 code, with bug fixes.

Got Python 3 support ?

Tarek Ziadé — Sat, 26 Sep 2009 20:06:00 +0200

One thing that slows down the adoption of Python 3 is the low number of available third party projects. If your project depends on some other projects, you are pretty lucky if they are all available under Python 3.

I don't think that the problem comes from the Python 3 syntax adoption, because Python provides a pretty powerful tool to convert your Python 2 code into Python 3 code, called 2to3. (notice that the backward process is also available since this summer : 3to2).

The biggest issue in my opinion is the lack of packaging support. Distutils itself works fine on Python 3, but I am talking about Setuptools, which is widely used in the community and doesn't work under Python 3. So if your project, or one of its dependency uses Setuptools, you can't switch to Python 3.

Well, I am glad to say that this is not true anymore, thanks to Martin von Löwis, Lennart Regebro and Alex Grönholm that have been working on Distribute's Python 3 support lately.

The Distribute project is a fork of the Setuptools project and is now fully compatible with Python 3.

If you are using Setuptools, install Distribute 0.6.3, read its docs/python3.txt file and add Python 3 support to your project.

And if you need help in porting your distribution to Python 3, drop in Distutils-SIG, we will help you.

static metadata for distutils

Tarek Ziadé — Sat, 12 Sep 2009 00:25:00 +0200

In Distutils, every package has some metadata fields, defined in PEP 314.

The setup.py script is the place where you provide them, by calling the setup function, located in distutils.core. Each argument passed to the function can be one of these metadata.

So basically, you can describe your distribution in the setup.py file like this:

from distutils.core import setup

setup(name='MyDistribution', version='0.1', description='cool',

      packages=['my_package'], ext_modules=[Extension('foo', 'foo.c')])

Notice that packages and ext_modules in this example are not part of the Metadata fields. They are extra fields used by some commands.

From there, various distutils commands can be called using this script. They will get these options and act upon.

For instance, the sdist command will build a source distribution and create a static PKG-INFO file that contains the metadata fields. It will extract them from the arguments you've passed to setup.py.

The install command will install this PKG-INFO file in your Python installation alongside your packages and modules (since Python 2.5) and some tools like Distribute or Setuptools will let you read these information once the distribution is installed.

You can even get the metadata fields values by asking for them directly through setup.py:

$ python setup.py --name

MyDistribution

Another example : the register command can send the metadata or your distribution to PyPI. They will be made available on PyPI website and also through its XML-RPC interface:

>>> import xmlrpclib

>>> server = xmlrpclib.Server('http://pypi.python.org/pypi')

>>> server.release_data('distribute', '0.6')['author']

'The fellowship of the packaging'

>>> server.release_data('distribute', '0.6')['keywords']

'CPAN PyPI distutils eggs package management'

Limitations of metadata

Metadata are pretty handy, but there are some obstructing limitations we bumped into when we started to work on packaging matters during last Pycon.

Platform-dependant metadata

We wanted to extend the Metadata fields list in order to add a "requires" field that can be used to list the requirements (in term others python packages or modules).

For instance, if you want to define that your project depends on simplejson, you could write:

from distutils.core import setup

setup(name='MyDistribution', version='0.1', description='cool',

      packages=['my_package'], ext_modules=[Extension('foo', 'foo.c')],

      requires=['simplsjon'])

This is not a new proposal. It was proposed in PEP 345, but never really used.

Since then, Setuptools provided a similar field, called "install_requires" together with easy_install script that acts a bit like a package manager. easy_install reads the requirements and install them when you install a distribution.

But the limitation of those requirement fields is that they might be platform-dependant. For example, you don't need to install simplejson anymore under 2.6 since a json library was included in the standard library. In other cases you might have different dependencies depending if you run under windows or linux, and so on.

So to be able to get the metadata right, you have to work a little bit in your setup.py file:

from distutils.core import setup

import sys

if sys.version_info[0] == 2 and sys.version_info[1] < 6:

    requires = ['simplejson']

else:

    requires = []

setup(name='MyDistribution', version='0.1', description='cool',

      packages=['my_package'], ext_modules=[Extension('foo', 'foo.c')],

      requires=requires)

But the metadata will only be available at install time, when the install command will execute the code of setup.py on the target system.

Code-dependant metadata

In other words, once a field like requires is added in the Metadata, you will not know for sure if it's reliable when you look at the project page at PyPI. That's because the metadata you will see there will be the one created by the person that called the register command and sent the result. This result is tighted to his environment, not yours.

To be able to get the metadata for your environment you will need to run that code again, by downloading the package, then running a setup.py command.

Let's try to do it with the lxml source distribution. Let's try to get the name field :

$ python setup.py --name

Building lxml version 2.2.2.

NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' needs to be available.

Using build configuration of libxslt 1.1.12

Building against libxml2/libxslt in the following directory: /usr/lib

lxml

What happened here ? Frankly I am not sure. But asking for the name (that appears on the last line) called a bunch of code located in the distribution.

I could probably ask the lxml team to fix this output, and make sure setup.py can still be used to work with the metadata, but this was just to demonstrate a flaw in the way Distutils works : you need to run third party code just to get the metadata of a distribution you're not even sure you are going to install on your system.

The setup.cfg file

Part of the problem can be resolved by putting the metadata in a static file alongside setup.py. As a matter of fact, the setup.cfg file is already used by distutils to store some options. There's even a global section that can be used to set the metadata into the Distribution object Distutils creates when you run setup(). Using the global section that way is not documented and probably not intended. What's intended is to be able to set some global options like "verbose" or such things.

See [http://docs.python.org/install/index.html#syntax-of-config-files][]

But the code is a generic setter, that allows you to pass any field (so the metadata). Call it a bug if you want, but I was pretty excited to see that I could pass my metadata to Distutils through it. Unfortunately these values are not passed to the DistributionMetadata subobject in Distutils, so it doesn't work exactly like the arguments passed to setup(). Too bad ;-)

I could change this right away in the code, but we have better plans I think.

A new setup section

Instead of working in the global section which should stay specific to running options, let's create a new section and put the metadata in them.

[setup]

name: MyDistribution

version: 0.1

description: cool

The setup.py script stays, but is now not containing any metadata field, and does only contain what I would call "working arguments". e.g. argument used by commands that are not part of the Metadata:

from distutils.core import setup

setup(packages=['my_package'], ext_modules=[Extension('foo', 'foo.c')])

What about platform-dependant fields ?

In order not to require any third party code to read the metadata, we need a way to express platform-dependant fields in the setup.cfg file.

The proposed way is to have platform-dependant sections :

[setup]

name: MyDistribution

version: 0.1

description: cool

conditional-sections: py25

[py25]

condition: python_version == '2.5'

requires: simplejson

The py25 section is read only if the expression is true.

Another example:

[setup]

name: MyDistribution

version: 0.1

description: cool

conditional-sections: py25, py26

[py25]

condition: python_version == '2.5' or python_version == '2.4'

requires: simplejson

[py26]

condition: python_version == '2.6' and sys_platform == 'win32'

requires: bar

here, "bar" will be installed under Python 2.6 under Windows, and "simpljson" under Python 2.5 or 2.4 on any platform.

Distutils will provide a new function that is able to interpret the expressions provided in the condition, and calculate the metadata depending on the platform.

That's still some code we are running here, but:
- We are restricting the execution context to the bare minimum: python_version, sys_platform, os_name, and all values returned by os.uname() - The function will be vanilla Python: you will be able to extract the metadata without running a third party code, and knowing that the execution is restricted to a few string comparisons. - The code can be executed at PyPI without any potential security issue, meaning that the XML-RPC functions will be able to send you back the metadata of a packages depending on your environment. In other word, a package manager would be able to list all the dependency of a distribution for the target platform without downloading any of these distribution.

There will always be edge cases

For the 1% of distributions that need more work to calculate the metadata, setup.py will still be present and any option passed as an argument will override a value provided by setup.cfg. They'll just have to add a flag in the setup.cfg file, indicating that it does not provides all the metadata, and that running setup.py is required:

[setup]

name: MyDistribution

version: 0.1

description: cool

static-metadata: false

If this flag is present, people will now that running setup.py is mandatory to get the full set of metadata.

For example, if the web service provided at PyPI to get the metadata, will be able to return a platform specific set if we provide the target environment. Let's say we add in distutils an 'execution_environment' that returns the environment used to interpret the setup.cfg file:

import xmlrpclib

>>> server = xmlrpclib.Server('http://pypi.python.org/pypi')

>>> from distutils.util import execution_environment

>>> execution_environment

{'os_version': 'Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386',

 'os_name': 'posix',

 'python_version': '2.6',

 'os_release': '9.8.0',

 'os_sysname': 'Darwin',

 'os_nodename': 'MacZiade',

 'os_machine': 'i386',

 'sys_platform': 'darwin'}

>>> server.release_data('MyDistribution', '0.6', execution_environment)['requires']

['foo', 'bar']

>>> server.release_data('MyDistribution', '0.6', execution_environment)['static-metadata']

True

PyPI will be able to generate the metadata by interpreting the setup.cfg file with the execution_environment info.

What happens now ?

I won't write a PEP for this. I don't think it's necessary because this feature is backward compatible, and if people don't use it in Python 2.7 and 3.2, it will just fade away, like other things in Distutils.

But we need to reach a consensus at Distutils-SIG then inform about it at Python-dev. I just hope we will have this consensus real quickly, unlike most topics we are working on for a year ;)

Or maybe I should be a bit of a dictator for this feature and just go ahead and add it ? Because as Brett told me several times, it's impossible to make everyone happy about everything. And I'd like to see Distutils move on. There's so much left to do...

What do you think ? How do you like that feature ?

Gsoc is over, we have a Python Keyring lib

Tarek Ziadé — Tue, 25 Aug 2009 12:04:00 +0200

The Google Summer of Code is over and thefirst version of the keyring library was released last week by Kang at PyPI.

How Keyring works, the big picture

This library implements a simple plugin system. Each plugin has to implement a set of methods described in an abstract class and can wrap any underlying Keyring system. We called those plugins "backends". The nice thing about it is that you can implement your own custom backend and make it available through the Keyring configuration file.

Kang has coded various Keyring backends in C and C++ extensions, for KWallet, Keychain, and Gnome. We also have added a Keyring implementation that uses the Win32Crypto API so windows users can use the lib.

When the Keyring lib is used, all declared plugins, whether they are provided by the lib itself or by a third party package, will be loaded. Then they will be asked a simple question:

"Can you run in this environment ?"

The backend can answer one of these:
- "Yes, I could work in this environment" - "No, I can't" - "Yes and you should use me !"

The library filters out backends that can't work on the target, sort the remaining ones, and get one of the best backend. This doesn't happens of course if you explicitely define which backend you want to use, which is possible.

What's next

Keyring 0.1 is out and there will probably be 1 or 2 releases to stabilize the code.

The next steps will be :
- to use it in Distutils, with a soft dependency : Distutils will let you use it through configuration if it detects Keyring is installed. - to promote its usage and in particular see if projects like Mercurial could use it - to work on a PEP for its integration in Python stdlib, in the getpass module

Gsoc Keyring project - post-midterm quick status

Tarek Ziadé — Wed, 05 Aug 2009 21:24:00 +0200

Kang (my Gsoc student) work is going very well on the Python Universal Keyring Library. He's basically finished to implement back ends for major keyring systems:
- Apple KeyChain - Gnome Keyring - KDE Wallet

And for Windows users, there's an extra back end based on Win32CryptoKeyring, which stores a crypted version of the password in a file. Last, there's another File-based crypted back end based on the PyCrypto lib.

The library is extensible and there's an abstract base class that provides an interface you can use to write your own back end. Each back end has to return a "recommandation" value depending on the execution context (the platform, and anything the back end wants to check). 0 means the back end is not compatible, 1 means it will work, 2 means it's recommended to use it. This allows us to pick the best back end automatically depending on the execution context.

Check the code here : http://bitbucket.org/kang/python-keyring-lib/

Kang is tweaking the API names, modules names etc, and then will:
- add some documentation - write a patch for getpass.getpass - start a web page for the lib

For the latter we are brainstorming on the project name before we release it.

I've asked on Twitter, here are some proposals so far (thanks guys!):
- @nightlybuild: Vercotti ? ( the recurring Sicilian connected gangster, played by Michael Palin - http://tiny.cc/luigi - @pmclanahan: I like "Zoot". From The Holy Grail, the name of the naughty sister in the perilous castle Anthrax. - @tpherndon: Bridge Keeper - "Answer me these questions three!" - @gurneyalex: you could consider creosote too - @jessenoller: Call it bucket. From meaning of life. - @regebro: "Biggles". Because he chains an old woman to a wall. http://www.ibras.dk/montypython/episode15.htm - @gurneyalex: BrightSide

And you, what's your proposal ?

Once the name is picked, we will start promoting the lib. I'll work on distutils integration, and we will propose it to projects like Mercurial (it makes sense to allow the removal of those clear-text passwords from your hgrc when you are doing http auth).

Have I said it already ? being a GSoc mentor is a great experience, especially when you have the chance to pick students like Kang.

Words on : Distribute, Distutils, PEP 376, PEP 386, PEP 345

Tarek Ziadé — Fri, 24 Jul 2009 22:56:00 +0200

I am taking a break for a week starting tomorrow morning. My wife made it clear : my laptop and internet are totally forbidden during 7 days. So I thought it was a good opportunity to make a status of all the projects that are going on in the packaging world.

[][]Distribute

The setuptools fork is doing good. We've discussed the roadmap strategy with some people at Distutils-SIG and came up with that plan:

0.6 will be released the first week of August. It will support Python 2.x only and will provide a bootstrap to replace an existing version of setuptools, as described in my previous entry.

0.7 will support Python 2.x and 3.x and will refactor the code in three distributions. It will also rename all parts under new names, so they will not compete anymore with setuptools. Its development will start right after 0.6 is out. If you have ideas or feature requests, please add them in our issue tracker at bitbucket.

Distutils

They are some patches waiting for my attention, I need to work on when I am back. But globally, if you open Distutils code, you will notice that it has been PEP8-fied (besides a few places I need to finishe) and test covered.

Some small refactorings have been done, like removing duplicate code and using the tar module of the stdlib instead of the tar command on the system, thus allowing building tar archive under win32 without having to install the tar program for example. Another nice change was done on the upload command: it uses urllib2 now, meaning that you can use an http proxy with a environment variable transparently. Other changes are on their way, like fixing the mess with "compiler" option, and cleaning up some remaining duplicate behaviors.

But nothing is deeply changing yet on how Distutils works. It waits for the PEPs we are working on to be accepted...

PEP 376

After I thaught this PEP was ready to be accepted, I sent it from Distutils-SIG to Python-dev. Then we had very long threads on Python-dev about it, showing that it was not ready at all...

Man, I don't think I've picked the easiest topic ;)

Anyways, people gave a lot of thaught on the topic, and Paul Moore helped building a PEP 302-compatible version of the prototype, you can see at bickbucket. From there, we still need to fix the problem of the absolute/relative paths in the RECORD file, and try to have real world uses cases, with various package managers applications. But we will eventually have an acceptable PEP, I hope, within a few months.

This pep will add query APIs in pkgutils in the stdlib, so you know what's installed in your Python. It will also provide tools to uninstall installed ditributions.

It's a hot topic because we are dealing with the fuzzy boundary between Python the extendable language, where you can install distributions using distutils-based tools, and Python the interpreter, that gets extended with packages managed by an OS-based package manager like APT under Debian for example. So for the latter, all our tool are getting too much in the way and should not make it hard for system packagers to extract the metadata they need to re-package distutils-based distributions.

PEP 386

Man, this PEP would make our life easier. It's all the work we started during Pycon, to find an acceptable way to compare versions. By acceptable I mean that could be used in our community and workable by OS packagers. I said I would drop it, because there were too much controversy, but quite a few people would like to see it accepted.

I doubt Guido will accept a PEP that would enforce a version scheme though.

I had created this PEP, so we could move forward on PEP 345, where we want to introduce "install_requires", the field used in setuptools to list dependencies.

The prototype for PEP 386 is in bitbucket too : verlib

Whatever happens, I'll probably publish it. Its useful to build a package system, and it has a function to translate most distutils/setuptools version schemes.

Now about my addiction

I know that sounds stupid, but I don't know what is going to happen next week. It's been years since I've not been fully offline for a week. I already have right now so many threads to catch up on various mailing lists that I am scared of how it's going to be in 7 days...

But it's probably a good thing for me to do this :)

I am off, see you in 7 days !

"Internet Addicted"

[]: http://www.flickr.com/photos/9009139@N08/1068561077/sizes/o/

Vacations + a new job

Tarek Ziadé — Fri, 24 Jul 2009 16:38:00 +0200

I am officialy on vacations !

I've just finished my time at Alterway / Ingeniweb, where I was CTO for the last two years. Well CTO was a translation for "Directeur technique" in French, which is more like "head of developments". So I was like some kind of lead developer, but without a code project :)

These two years were a great time for me. It's time for me to summarize what I have done, what went right, and what went wrong.

Summary of my work at Ingeniweb

I won't talk about customer projects, but about what I did that served both for our projects and for the community. Ingeniweb is a company that does (did in fact) Plone websites.

I was hired at Ingeniweb with some clear goals (I had more, but these are the most interesting):
1. improve the release process 2. help Ingeniweb get back into the community 3. QA work

Improve the release process : when I started, Ingeniweb was not using zc.buildout, the de-facto Plone standard, so one of the biggest task I achieved was to make them switch to zc.buildout, and eventually make some minor contributions in zc.buildout. That was a hard work because the average developer was against such a big change in his habits. I eventually succeeded.

I've also worked on plone.org side to make it "pypi compatible" and eventually became a Python commiter to improve distutils, the mother of all packaging tools in Python. That was very positive for me and for the community, I think. And I am now involved in Python development for the future, trying to write some useful PEPs and so on. This is going to be one of the major community task I'll be working on for the upcoming years I think.

Help Ingeniweb get "back into the Plone community": this was quite
an interesting task. The people that hired me were frustrated because
their developers were not (besides a few exceptions) part of the Plone
community. So I worked on this by trying to understand why it happened.
It's quite simple in fact : that's because of the lack of communication: people in our company were not talking enough with the community and were trying to solve and work on their problems on their side. No judgment or disrespect on this : this is mainly because of the language barrier I think. And also because the people in charge were not sending our developers in conferences anymore. Being part of an OSS community requires from the company managers to send their developers in conferences from time to time.

The output of this task was not positive for Ingeniweb, but in the meantime very positive for me : I was able to become part of the Plone community and meet great people. Although I was unable to push my co-workers to follow me in this, neither to make the company contribute more on this. We organized one sprint, but given the size of the company at that time (20+) this was not enough in my opinion.

The worst part I think is were I realized that the people that asked me to do it, didn't care much about it anymore. The boss left us, and the "second" left right after :)

QA in customer projects : The third main task was to raise the QA in all projects, starting from scratch : beside a very few exceptions, people were not making tests neither buying the Test-Driven approach. The problem with TDD is that it cannot work if it's not done everywhere by everyone in the company. When this is the case, someone who doesn't practice TDD looks like he's trying to break a window of the house we are all building together.

But when you are working with developers that don't practice TDD, you need to first convince the project managers that it's better, otherwise it doesn't take. I have convinced some co-workers, but at the end, TDD didn't work out because some managers were just telling their developers to drop TDD to go "faster" in the customer projects. (faster... into troubles for big applications)

At the end, I think it's very hard to raise the QA in companies that sells development services, because it implies a lot of commitment from all people involved, from sells to managment. In an software editor company it's easier because you can put some guards around your software code base and keep the QA high.

But along the way I did some interesting projects to set up continuous integration. collective.buildbot is one of those. It allows you to set up automatically a buildbot into a zc.buildout-based project.

Don't get me wrong if you feel that this is a pessimistic summary : everything else was just great !

What's next ?

When I am back from my vacation, I am starting a new work in a startup in Paris, providing a Saas in the cloud fully written in Python.

I'll blog about it when I am back :)

Preparing to release Distribute 0.6

Tarek Ziadé — Wed, 22 Jul 2009 17:30:00 +0200

According to the poll, The name of the fork will be Distribute !

The code should not be changed anymore at this point, and I am working on the bootstraping so installing Distribute will work with an existing Setuptools installation and will replace it for the applications that requires it.

This is done by detecting an installation of Setuptools, and replacing it with a fake installation. This means that once you've installed Distribute, applications and especially installers will think that setuptools 0.6c9 is installed.

That's pretty strong and intrusive, but required for a simple switch : even if the programs you are using have a setuptools dependency, they will work without requiring any change on the code or in their setup.py files. Same goes for zc.buildout apps.

I still have a lot of work for this part :
- I don't detect and patch properly single-version-externally-managed setuptools installation yet (required for pip) - I need to fix a bug on a first run under jython - I need to fix a bug when virtualenv is not used with --no-site-packages - I need to do extensive tests on zc.buildout to see if it behaves correctly

If you want to help:
1. download [http://nightly.ziade.org/install_test.py][] 2. run it with the Python interpreter of your choice (possibly a virtualenv-ed one)

To uninstall, follow the Uninstallation instructions here : http://bitbucket.org/tarek/distribute/src/tip/README.txt

*Disclaimer: it might break your installation****

If the test end up with this line: **** Test is OK. It worked. Otherwise, please let me know !

The next "big" step will happen with Distribute 0.7 because the plan is to split the code in several distributions with renamed modules:
- Distribute : the core (setuptools package but renamed) - DistributeInstall : easy_install on its own, and renamed. - DistributeResource : will contain the pkg_resources.py module, renamed

I'll post more details on it in the upcoming days.

The strange world of packaging - forking setuptools

Tarek Ziadé — Sun, 19 Jul 2009 09:59:00 +0200

Again, like a year ago, people had enough of the fact that the setuptools project is not maintained since 9 months.

Phillip Eby explained that he doesn't have time to do it unless someone would pay him for that. But in the meantime, he doesn't bless anyone to do it. Well, he has blessed some people to do it (Ian Bicking and Jim Fulton), but unfortunately these people are not willing to do it because they have a lot of other projects going on. Other people that could maintain it, including me, fail in his "unqualified people" category :)

So we are all locked in a strange situation where tons of patches are ready to be commited in the setuptools tracker but are not making it. Several non-public forks have started to appear around of course.

So again, I decided with some other people to create a fork called "Distribute". It's a real fork located here : http://bitbucket.org/tarek/distribute.

By real I mean that this fork was not created with the purpose of forcing Phillip to do a release like we did last year for the 0.6c9 release, but with the intention to free us from that strange situation where we all depend on his wills and (lack of) time.

The plan is to release a first version next week, that corresponds to the setuptools 0.6 branch, with some patches applied.

Next, we are planning to start a 0.7 version where the code will be splitted in several distributions:
- a distribution for pkg_resources - a distribution for the setuptools package itself - a distribution for easy_install

A little bit of bikeshedding is going on to pick a name for that fork, and we ended up running a poll. (vote!)

Now, right after I have announced this plan on Distutils-SIG, Phillip reacted by annoucing a similar plan, e.g. splitting the setuptools project in several distributions. But since he previously said that he didn't have the time to do it, I doubt that it'll work out unless he's opening its development to a wider range of developers and maintainers.

That's the strange world of packaging...

Distutils nighlty builds

Tarek Ziadé — Fri, 03 Jul 2009 13:57:00 +0200

If you want to work (or try to find some bugs in it to help out) with the latest bleeding-edge Distutils trunk version that will be shipped with Python 2.7 and Python 3.2, you can do it now !

I am creating a nighlty build every day here now : http://nightly.ziade.org/

This distribution can be installed in your existing Python 2.6 installation, and will probably work with 2.5.

I'm not planning to publish standalone versions yet, except nighlty build, at least until 2.7 and 3.2 are starting to be released as alphas.

Dropping PEP 386 (versions comparison)

Tarek Ziadé — Fri, 03 Jul 2009 13:44:00 +0200

As you might know, we are working hard on Distutils side for Python 2.7 and 3.2 upcoming releases. The biggest work is on PEP 376, that will introduce among other things a uninstaller function and functions to query installed distributions.

The other "big" work is on PEP 345. We want to introduce a new metadata field called "install_requires" to be able to express requirements. That's from the setuptools project and is quite used by the community. Notice that there were several attempts to define requirements in the past in Distutils, but none of them really made it through.

For instance, if you want to define docutils as a dependency for your distribution but a version less or equal to 0.4, you can say :

docutils <= 0.4

But as long as you want to work with such dependencies and provide a way to express them with operators, you have to be able to compare versions. For instance if you want to compare an installed docutils distribution to see of it is compatible with 0.4.

That's another big topic we have been working on for the last few months, with people from various communities (Fedora, Ubuntu, etc). And I have started to write down PEP 386.

But comparing version appears to be a topic that cannot be generic. It seems that Distutils, therefore Python, shouldn't enforce any rule on this.

Furthermore, since we have said that Distutils should be a lighter package, it will not implement a complete package managment system, like setuptools or zc.buildout does.

So I've decided to propose to drop PEP 386, and stick on a very simple rule in PEP 345, saying that requirements can be defined, with :

distribution_name OPERATOR version

where OPERATOR is in >, <, ==, !=, >= or <=.

Last, so the work done at Pycon and in Distutils-SIG is not lost, I will publish the library we wrote. This could be a very good basis for packaging managment systems out there.

mod_ntlm in FreeBSD

Tarek Ziadé — Tue, 30 Jun 2009 13:39:00 +0200

We had to install [mod_ntlm][] under a FreeBSD 6.4 server with a colleague, but the port for this package seems broken at least for FreeBSD 6.2 and 6.4. It just doesn't work when it initiates an NTLM session through SMB, and it seems to be compiled without the log support so the last log you get doesn't give any useful hint on what's wrong.

So we had to recompile it to activate the log and try understand what the problem was. But Apache doesn't not support threaded mode under FreeBSD 6.x (or I couldn't manage to make it work howsoever), so mod_ntlm failed to compile since it uses Apache's thread mutexes to perform some locks on some operations.

We've deactivated them and recompiled the module, and now it works now like a charm. It's weird because it means that the binary package for this module is completely broken. I couldn't find any place mentioning this.

Anyways, I know this is an old software combination, but since Google doesn't give any hint on this problem, I wanted to blog about it and join the diff I made out of mod_ntlm2 at https://modntlm.svn.sourceforge.net/svnroot/modntlm/trunk here:diff file

So if you bump into this problem, you will hopefully reach this page and save a few hours.

Smoke testing in Distutils

Tarek Ziadé — Thu, 28 May 2009 15:38:00 +0200

I am not really sure smoke testing is the best name to describe what I am trying to set up.

Wikipedia says:

In computer programming and software testing, smoke testing is a preliminary to further testing, which should reveal simple failures severe enough to reject a prospective software release. In this case, the smoke is metaphorical.

But I think it describes well the need I have : being able to test the output of several releases of Distutils, without relying on unit tests for that. Simply because they did not exist on early versions of Distutils.

Why do I need "smoke tests"

Lately, I was working on Distutils, removing some old bugs and adding some tests. Working on an under-tested package is like opening a can of worms : when you change something you can introduce some regressions. It's hard to avoid for sure, because even if you add some tests subsequently for some features, you are not really doing proper TDD, where the tests and the code grow together. So there might be some uncovered cases left even if the coverage is improving slowly.

Regression also happens when you correct a behavior that was broken for years : if third party tools used a broken behavior without knowing it, fixing the behavior at some point becomes a regression from them.

But these pitfalls are unavoidable. Hopefully they don't last for too long in a big project like Python. They are quickly reported, either by buildbots, either by someone from the community.

The last behavior problem I had was on the build_ext command. There's a string option called "compiler", were you can force the compiler type. You can put for instance 'linux' and build_ext will use the UnixCompiler compiler class.

Although this option name is misleading. It should have been called "compiler_type" in the first place. And the build_ext command adds another problem : it turns the compiler attribute of the class where the option value is stored, into a compiler instance. No need to say a class attribute value should not have several types.

The first application to suffer from this is Python itself. It uses build_ext.compiler, as a compiler instance to build its modules.

This means that "compiler" is now doomed to be both an option and a compiler attribute. It's not easy to fix and will require a deprecation step.

So the idea of the smoke test is to make sure Distutils still knows how to produce releases of existing projects.

The Smoke testing buildbot

I have built a buildbot instance using collective.buildbot. The script the slaves run is quite simple: it gets some projects source and run their setup.py script, using various versions of the Python interpreter, then the trunk itself. The commands run so far are sdist and bdist.

Then it compares every file contained in the archive created to make sure produced archives are similar.

I will encouter some exceptions, when the package I am testing includes different files in its distribution depending on the Python version, but those exception will be managed.

Last, I cannot run this test on every package out there, for security reasons. Until I have the proper virtual environment to do that, I'll work on a white list of packages I "trust". So far, there's just a package of mine, and NumPy. But this list will grow.

If you are curious to know if your package builds under Python trunk, get in touch with me, I'll probably add it if I know I can trust it.

The last problem I have with this system yet is the fact that setuptools is not working anymore with the Distutils trunk, so I am unable to test setuptools-based packages.

The buildbot is located here : http://buildbot.ziade.org/waterfall

Monitoring a Zope 2 application

Tarek Ziadé — Tue, 19 May 2009 09:29:00 +0200

We have a simple need for a customer project that runs a Zeo server and a few Zeo clients : being able to check the status of every Zeo client, and monitor what they are doing.

DeadlockDebugger almost provides this feature since it is able to produce a dump of the execution stack for every thread a Zope instance is running.

Based on this tool, I have developed ZopeHealthWatcher, that provides a console script to query a Zope instance, and get back a status for every running thread. It tells you if the thread is idling or if it's running some code. The script also returns an exit code depending on the number of busy threads, so it can be used in tools like Nagios.

When there are 4 or more busy threads, the script will return the execution stacks for every busy thread and some extra info like the system load and memory info. The returned info will be extendable through plug-ins in the next version, but right now the provided info are enough for our needs.

I have also created an HTML version, so when the dump is requested from another tool than the console script (e.g. a browser), it displays a nice human-readable interface (check the PyPI page for more info and a screenshot).

Notice that DeadLockDebugger is hackish since it patches the Zope publisher at startup. But we won't change this part: we need this tool to run from the oldest to the newest Zope 2 version. And the patch just works fine, so...

The provided version should run out of the box in a buildout-based Plone 3 application, but requires manual installation steps on older Plone or CPS versions.

I didn't mention these manual steps in the documentation. I think I am the only person in the world interested in running this tool on the dead-but-still-in-production-in-many-places Nuxeo CPS.

By the way: kudos goes to Marc-Aurèle Darche, who is maintaining CPS for years now, making it one of the most bug-free and stable CMS solution out there. Ok it's probably easier to reach this level of quality since the platform is very stable and only evolves very slowly thanks to Georges Racinet.

Distutils state

Tarek Ziadé — Sun, 10 May 2009 15:30:00 +0200

Since Pycon, a lot of discussion and work has been going on.

During the summit, we made a few decisions (see http://mail.python.org/pipermail/python-dev/2009-March/087834.html) but this topic is so wide and so complex that a lot of discussion still needs to be done to have a clear and complete picture of where we want to go and how we are going to do it

But we have made significant progress and reached consensus on some key points. This entry is a short list of these key points.

Being able to compare project versions

If you take a look in Distutils code, there's a version and a versionpredicate module that provides a way to compare versions. This feature is barely used by Distutils itself, and a very few number of projects out there are using it (If you do so, please let me know).

This is probably because Distutils provides two different ways to compare versions: a "strict" one and a "loose" one. In other words Distutils clearly states that it is unable to provide a unique version comparison algorithm that can be used by anyone out there. Anyone means here : Python developers, OS packagers, etc.

Setuptools did a better job by providing an algorithm that covers most cases, but suffers from this universality: it's too heuristic.

So one of the main topic during Pycon was to try to find a version comparison algorithm that would just work for everyone and in the meantime would be strict enough to be useful. That's a pretty tough task, but I think we have reached something that is "good enough" for everybody. We had the chance to have people from Ubuntu, RedHat during Pycon to work on this task, and Trent Mick took the lead during the sprints to come up with a prototype.

It's described here : http://wiki.python.org/moin/Distutils/VersionComparison and I have put the code here http://bitbucket.org/tarek/distutilsversion.

Two more things to take care of:

Philip Eby came up with an edge-case : being able to do a development version of a post release.
Jean-Paul Calderone proposed to add a constructor that would take explicit arguments to describe the version, rather than a string representation

There's a branch (tarek-postdev) on bitbucket to work on these two cases. But basically, it seems that we have a consensus on a unique way to compare versions in Python ! This is a great step forward.

Another point that Toshio Kuratomi raised during a hall discussion was the fact that some Python developers are not having good practices when versioning their releases. So we agreed that a good "How to properly use version numbers for your projects releases" document will have to be delivered in Distutils documentation, besides the new algorithm. I am pretty confident that people will eventually follow it.

APIs to access installed project metadata, and an egg-info standard

Once a project is installed in Python, you don't have a way to get its metadata and to answer to basic questions like:

what are the installed packages ?
what is the version ?
how is the author ?

Of course you can dig in your installation and get back most of these, but depending how you installed your package (manual, easy_install, pip) it might be located in different places.

So we agreed on a standard for this, described in PEP 376 : http://www.python.org/dev/peps/pep-0376

An API will be introduced to be able to get the info for a installed package, as long as it was installed using PEP 376 standard.

An uninstall feature !

A feature that is claimed for a long time now should be introduced in Distutils : uninstallation !

Distutils will provide a basic, reference uninstall feature that will remove all files that where previously installed for a project. This will be doable as long as this list of files is recorded at installation time, and not used by another project.

See the details in PEP 376.

Standardize PKG-INFO

New fields will be added in the metadata of a project. The most important one is install_requires from setuptools, which let you list the projects your project depend on and their versions.

This is for informational purpose, and Distutils will not provide an install feature that will fetch and install dependencies. This can be done by third-party tools like easy_install as long as they use the installation standard described earlier.

We started this work with Jim Fulton at Pycon, and Tres Seaver took the lead on this task since then. The plan is to update PEP 345 . The work is done in a branch here : http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?view=markup

But the remaining work is focusing on practical details. Just remember that PKG-INFO will evolve and install_requires will be integrated amongst other changes.

Distutils code cleanup, test coverage

The test coverage is starting to look good and most modules are covered around 80%. I guess this is the average in most Test-Driven Development projects out there. So Distutils is becoming a good citizen of the standard library ;)

The remaining work for a good test coverage is mostly on compilers side and os specific commands.

My black list of untested (or nearly untested) modules :

bcppcompiler
cygwinccompiler
emxccompiler
msvc9compiler
msvccompiler
command.bdist_msi
command.bdist_rpm

I need to figure out how to properly test them. Last, I need to set up a buildbot that will try out Distutils trunk on a list of projects out there, like numpy for example.

Gsoc : Keyring library work started !

Tarek Ziadé — Sun, 03 May 2009 10:49:00 +0200

I am very proud to be a Gsoc mentor this year on a very interesting topic : an universal keyring library for Python.

About the topic

In Distutils, if you want to interact with PyPI, you have to register in the website and you get a login/password so you can register and/or upload your packages.

Before Python 2.6, the only way you could interact with PyPI was by storing these info into your .pypirc file in clear text. This was not the best solution. For example we have in my company some staging servers we share, and from whom we upload packages to various PyPI-like servers. So we have to store PyPI login/password info in them. This means that if Bob wants to push his package from that server, he has to put his password into a clear text file which is most of the time readable by everyone. It's not such a big deal in our company since we can trust each other, but it's a very bad practice.

So I ended up changing this in Python 2.6 so people could type their password in a prompt when working with packages, using getpass.getpass. So they wouldn't have to store them anymore.

But this is not enough : we need to provide something better. We need getpass.getpass to be able to interact with keyring libraries like KeyChain, Gnome Keyring, etc. So the login/password info are safely stored and can be reused.

This service will be useful for Distutils, but also for any Python application.

The idea of the GSOC task is to provide an unified keyring library for Python, and it's harder that it sounds. For instance, we need to find a way to provide something that works under Windows. So the whole work is quite challenging and interesting, and the goal is to end up with a keyring library we can use into Distutils and propose for inclusion in getpass.

About Kang

Kang Zhang is the student that was selected for this work. Congrats ! He has started to work on it. You can follow his work in his blog. I have a strong feeling that he will succeed in this work and come up with something good.

Take a look at the Python Soc planet too, where all students involved in Python GSOC are blogging about their ongoing work.

Basic plugin system using ABCs and the "extensions" package

Tarek Ziadé — Fri, 01 May 2009 20:59:00 +0200

I need a very simple plugin system for one of my projects. The project is a small WSGI application called mysysadmin that allows you to launch some commands on your system to manage some applications. It also allows you to view log files in your web browser.

It's similar in some ways to WebMin,

So in my application, every tab is a plugin that manages one application. I have a plugin for Apache, another one for MySQL, and so one.

Back to my plugin system. So every plugin that is registered becomes a tab in the WSGI application, as long as it provides all the methods my web application needs to interact with it. So I want to check that each plugin strictly provides the API needed by the main program.

The first tools that came in mind were :
- zope.interface, to be able to provide that each plugin meets the requirements. - setuptools entry points, so it's easy for a third party code to implement a plugin.

But I find both projects a bit complex to implement such a simple
plugin system.I could use the standalone Plugins package Phillip
provides instead of setuptools, but it still does too much things imho.
That's someting I am currently learning by working on packaging matters: one library should not provide too many features.

Extensions : a simple plugin system

So I have started to implement a light-weight plugin system called extensions, which reuses setuptools entry points principles but is more simple to use. The goal of this project is to provide very simple APIs to handle plugins, and to make it work without introducing a new argument into the setup.py setup method, like setuptools does.

For instance, if you want to define an apache function in your modules module, in your myapp package, you just call the register function :

from extensions import register

register('mysysadmin.modules', 'apache', 'myapp.modules:apache')

That's it !

And to use it, the mysysadmin application can use a simple API called get, that iterates over all plugins defined for "mysysadmin.modules" :

from extensions import get

for plugin in get(group='mysysadmin.modules'):

    # do something with the plugin

The magic is done by writing in the .egg-info directory installed for the package that contains each plugin, a PLUGIN file that contains the list of registered elements. It's an idea borrowed from setuptoools entry points. So get iterates over all .egg-info directories in your path and load the PLUGIN files it finds. Nothing new here. That's how setuptools does, and that's perfect.

If you have any feedback on extensions, let me know !

Strict plugins

The other need is to strictly check that every plugin provides the API needed, e.g. fulfill the requirements. This is what we could call Design by contract.

You can provide a base class for this, and ask the plugins to inherit from it. Or you can ask the Plugin to provide a marker to specify it implements a given behavior. zope.interface can do a nice job for the latter,and let you check that a given object implements an interface.

But I wanted to give a shot to the brand new Python ABCs and make sure anyone can write a plugin in plain Python, without having to rely on any kind of marker system. ABCs will let you check that a class implements some methods without requiring it to inherit from a specific class, to implement a specific interface or provide a custom marker. Pure duck typing !

So let's define for our application a Plugin class, that gives the signature every plugin will need to provide. It uses ABCMeta as its meta class, and the abstractmethod for every method that should be implemented by every plugin.

Here's an extract :
from abc import ABCMeta, abstractmethod

class Plugin(object):

   __metaclass__ = ABCMeta

   @abstractmethod

    def get_command_list(self):

        return []

    @abstractmethod

    def run_command(self, name):

        pass

    @classmethod

    def __subclasshook__(cls, klass):

        if cls is Plugin:

            for method in cls.__abstractmethods__:

                if any(method in base.__dict__ for base in klass.__mro__):

                    continue

                return NotImplemented

            return True

        return NotImplemented

The __subclasshook__ method is a class method that will be called everytime a class is tested using issubclass(klass, Plugin). In that case, it will check that every method marked with the abstractmethod decorator is provided by the class.

So basically, the application can discover and use the plugins, with:

from extensions import get

for plugin in get(group='mysysadmin.modules'):

    klass = plugin.load()

    if not issubclass(klass, Plugin):

        logging.info('sorry, not a suitable plugin')

        continue

    # do something with the plugin

    xxx

Abstract Base Classes are one honking great idea -- let's do more of those!

URLs in books

Tarek Ziadé — Sun, 19 Apr 2009 14:42:00 +0200

I received some complaints about the fact that some links in my books were dead by the time they were printed.

For the next book I am working on, I have proposed to my editor to set up a website to keep track of all references mentioned.

By using unique short ascii references throughout the book, it's easy to provide a simple redirect service to the target URL, and to fix it when it changes (just by setting up a mail alert if your redirect reaches a 404).

For example, if I am referring to mod_wsgi in my book, I can write this reference: #mod_wsgi, and provide a redirection to http://code.google.com/p/modwsgi into my website, through a unique, mnemotechnic permanent URL : http://ziade.org/urls/mod_wsgi.

This small service, à la Tiny URL is not a burden for the reader imho : he is using his computer anyway when he visits an URL mentioned in a book.

It's a simple idea I am sure a lot of people have thaught about before, but I fail to see it applied in the books I am buying these days. Is there any good reason I fail to see ?

new blog location, new design, update your bookmarks

Tarek Ziadé — Wed, 15 Apr 2009 15:51:00 +0200

I was thinking about doing this change for a while, and I took the time to do it last week-end:

My personal website at http://ziade.org is now powered by Pylons and Atomisator. It's both in French and English (the urls are translated too).

It contains:
- a home page with my latest twitter entries - a blog (this one) - some other pages like a résumé, a list of the books I wrote, a projects page, etc.

I am still using Wordpress to write my blog entries but they are now processed with Atomisator and displayed at http://ziade.org/blog. The feed is now located at http://ziade.org/blog/xml and I will make sure all Planets use this URL for now on.

Next I'll probably :
- move to another blogging tool, and my readers will not suffer from the change as long as they use the ziade.org feed url. - add some portlets into the new site, like the ones I have on Wordpress.

How do you like this new look ? Any comment / advice ? :)

Distutils: introducing the check command (reStructuredText control)

Tarek Ziadé — Fri, 10 Apr 2009 14:11:00 +0200

I am introducing the check command in Distutils. This command will check your package metadata, like the sdist and the register command already do (they display warnings).

But the new thing is that it will also allow you to check if long_description is reStructuredText compliant.

Its usage will be:
$ python setup.py check --restructuredtext

running check

warning: check: Title underline too short. (line 32)

warning: check: Title underline too short. (line 32)

warning: check: Could finish the parsing. (line None)

And there's also a strict mode, that raises an error in case something is wrong
$ python setup.py check --restructuredtext --strict

running check

warning: check: Title underline too short. (line 32)

warning: check: Title underline too short. (line 32)

warning: check: Could finish the parsing. (line None)

error: Please correct your package.

Last, this command will be used by register, and sdist, so you can stop the process in case the metadata are not correct. This is useful to make sure your PyPI home page is not broken for example, since it parses long_description to build it. So a good practice will be to use strict when registering a package:
$ python setup.py register --strict

running register

running check

warning: check: Title underline too short. (line 32)

warning: check: Title underline too short. (line 32)

warning: check: Could finish the parsing. (line None)

error: Please correct your package.

This feature will land in Python 2.7 (I am working on it and it should be commited this week end). Of course, it will not introduce a hard dependency on docutils in Python, neither it will change the current default behavior.

Until then, you can use collective.dist 0.2.3, that provides this feature for Python 2.4 to 2.6

Pycon hallway session #2: thoughts for multiple versions in Python

Tarek Ziadé — Mon, 30 Mar 2009 07:16:00 +0200

We had an excellent brainstorming session today in the hall, with Toshio, Georg, Martin, Thomas, etc.. (sorry we were so many I don't have the full list) with some insights from Guido and Brett. We tried to think about a way to handle multiple versions of a same package.

Here's the two most important concepts :
- Unicity: There should be one and only one instance of a Python package at a given version on a system - Combination: One Python application combines several packages to run

About unicity

A Python package is a component that can be installed on a system. If you use the standard Distutils approach, it will end up in the Python site-packages directory and be importable by the interpreter. This package comes with a version number and is unique.

This unicity is important for security and maintainability. For instance, if there's a security hole in a package, the fix is applied in one place and the system maintainer knows it can't be present elsewhere on the system.

This is the system administrator point of view

About Combination

What defines a Python application is the fact that is selects a list of packages it needs to run. And this varies for every application. So two Python applications might need a different version of a given package and that is normal.

What's important is to have the right list of packages when an application loads. Tools like zc.buildout or virtualenv are perfect for this need : they create an isolated environment for your application to run with the right set of packages.

So the simplest way to release an application is to ship it with everything required, regardless the unicity.

This is the application developer point of view

The idea

zc.buildout and virtualenv are a blast for developers, and another thing system packagers might dislike. This is because they break the unicity by allowing developers to ship their applications as black boxes. Of course one may say that this is perfectly fine since what's inside an application is not the problem of the system packager. But since this application is made of packages that may be shared on the system by other applications, that is redondant.

Forget site-packages for a moment. And let's think about a new loading system for packages. This approach is similar in some ways to setuptools' multiple version system.

Storing multiple versions

First of all, let's store the packages in a directory, and for each version of the package, under a sub-directory which name is the version.

For example:
- SQLAlchemy
- 0.4
- package code is here - package egg-info here

0.5
- package code is here
  - package egg-info here
jsonlib
1.2.6
- package code is here
  - package egg-info here
1.3.10
- package code is here
  - package egg-info here

Given this structure, some mechanism can provide to the interpreter the latest available version of a package, as long as Distutils knows how to handle version comparisons correctly (which will be the case in a near future)

Choosing specific versions

Back to our application. Let's call it MyApp, version 1.0.

It needs specific versions for some packages. Forget zc.buildout and pip for a moment. And let's think about a different way to express the packages (and versions) its needs.

Let's make a Python package for this application and let's make a few assumptions on some features in Distutils:
- setuptools' install_requires has made it into Distutils, as part of metadata. - metadata are defined statically in a package, apart from setup.py.

So basically, the application is mainly a static list of dependencies defined into install_requires. For example:
- SQLAlchemy > 0.4 - jsonlib == 1.2.6

When MyApp will get installed by Distutils, it will be added in the packages tree.

When it is used, it will need to load the versions of SQLAlchemy and jsonlib it needs. The ones that are inside its metadata.

To make it possible, the script that launches the application calls a built-in function called read_deps, that takes the metadata and reads them to know which versions fit:
# the script

read_deps('PKG-INFO')

import SQLAlchemy

import jsonlib

...

This call will load the right versions in the packages tree:
- SQLAlchemy
- 0.4 - 0.5

jsonlib
1.2.6
- 1.3.10
MyApp
1.0

What's next

I just dropped here the rough idea, and a lot of details are missing. But I think it's a good thing to share this idea here in its early stage.

We have decided we would try to write a prototype using importlib and sys.meta_path. Maybe Georg will start it during the Pycon sprint, and start to digg into the details. He was working on this when I left the open session at Pycon.

Stay tuned

Pycon hallway session #1: a keyring library for Python

Tarek Ziadé — Fri, 27 Mar 2009 14:55:00 +0100

Before I sit down and clean up my summit notes to send them to python-dev, I wanted to post an entry about a small project which I think could be a great task for a student at the Summer of Code (I doubt it can fill 4 months of work but it could be done amongst other tasks).

Yesterday, we did a late session with Martin von Loewis, Jim Fulton and Georg Brandl about PyPI and the fact that it needed a better way to handle passwords on client side. That is, the fact that you have to store your password in the .pypirc file if you want to upload your package to PyPI.

This is unsafe and unwanted. A few months ago, I have made a small change in Distutils so it would prompt for the password using the getpass module if it doesn't find it in the .pypirc file. (This was a contrbution of Nathan Van Gheem).

Anyways, this is not enough. Jim suggested to set up a SSH server on PyPI using Paramiko, so we could use a standard ssh connection and benefit from ssh-agent. But this is unfortunately not universal.

So let me get back to the idea I sent some time ago : http://mail.python.org/pipermail/python-ideas/2009-January/002465.html

What about having an option in getpass to store and reuse passwords in

system keyrings ?

    getpass(prompt[, stream])

would become:

    getpass(prompt[, stream, keyring])

where keyring would be a callable that can be use to retrieve the

password from a keyring system

and store it the first time.

The getpass module could provide some keyring support for:

- ssh-agent under Linux

- keychain under Mac OS X

- ...

ss

And let the developers use their own keyring system by providing a callable.

As Greg Smith said in the thread, the first task is to create a library that supports all standard keyring systems out there, including things like KWallet, Internet Explorer, Fireforx and so on...

I'll mentor this project if any student would like to do it.

http://mail.python.org/pipermail/python-ideas/2009-January/002465.html

Packaging Survey first results

Tarek Ziadé — Thu, 26 Mar 2009 03:35:00 +0100

Around 570 people answered the survey, which is a great number I didn't expect. Thanks again to Massimo for his help on this.

I have a lot of work to read all the open question answers, and all the comments that goes with the "other" answer, but I wanted to publish the results of the closed questions before the summit.

I don't want to comment the results yet. I will after I have studied all answers, so it'll be a little while ;)

Who are you ?

Professional developer using Python exclusively.

283

Professional developer using Python unable to use Python "at work".

Professional developer using Python sometimes.

196

Hobbyist using Python.

116

Where are you located ?

USA

212

Western Europe

268

Eastern Europe

Asia

Africa

Other

If you are a web programmer, what is the framework you use the most ?

Pylons

TG 2

TG 1

Django

184

Zope (including Plone)

137

Other

207

How do you organize your application code most of the time ?

I put everything in one package

171

I create several packages and use a tool like zc.buildout or Paver to distribute the whole application

137

I create several packages and use a main package or script to launch the application

198

I use my own mechanism for aggregating packages into a single install.

For libraries you don't distribute publicly, do you you create a

setup.py script ?

Yes

321

249

What is the main tool or combination of tools you are using to

package and distribute your Python application ?

None

setuptools

150

distutils

127

zc.buildout and distutils

zc.buildout and setuptools

107

Paver and setuptools

Paver and Distutils

Other

How do you install a package that does not provide a standalone

installer (but provides a standard setup.py script) most of the time ?

I use easy_install

241

I download it and manually run the python setup.py install command

139

I use pip

I move files around and create symlinks manually.

I use the packaging tool provided in my system (apt, yum, etc)

Other

How do you remove a package ?

manually, by removing the directory and fixing the .pth files

275

I use one virtualenv per application, so the main python is never polluted, and only remove entire environments.

154

using the packaging tool (apt, yum, etc)

178

I don't know / I fail at uninstallation

I change PYTHONPATH to include a directory of the packages used by my application, then remove just that directory

Other

How do you manage using more than one version of a library on a

system ?

I don't use multiple versions of a library

217

I use virtualenv

203

I use Setuptools' multi-version features

I build fresh Python interpreter from source for each project

I use zc.buildout

109

I set sys.path in my scripts

I set PYTHONPATH to select particular libraries

Other

Do you work with setuptools' namespace packages ?

Yes

178

344

Has PyPI become mandatory in your everyday work (if you use

zc.buildout for example) ?

Yes

228

294

If you previously answered Yes, did you set up an alternative

solution (mirror, cache..) in case PyPI is down ?

Yes

N/A

277

166

Do you register your packages on PyPI ?

Yes

239

281

Do you upload your package on PyPI ?

Yes

205

314

If you previously answered No, how do you distribute your packages ?

One my own website, using simple links

139

One my own website, using a PyPI-like server

On a forge, like sourceforge

N/A

251

Other

Pycon Language Summit is tomorrow

Tarek Ziadé — Thu, 26 Mar 2009 02:38:00 +0100

Tomorrow is the Language Summit, yeepee. :)

The package and distribution part of the Summit is going to be great since we have key people coming up.

zc.buildout and pip leaders (Jim Fulton and Ian Bicking) will be present, and many others. I'll be representing Distutils, since I am its current maintainer. Unfortunately Philip Eby (setuptools) can't make it, but he should be reachable via IRC (I am trying to set something up for tomorrow).

Anyway, one of the goal of the Summit is to validate the new features and enhancements we want to introduce in Distutils and PyPI. It's important to make sure they play well with third-party tools like zc.buildout, setuptools and pip. We also need to make sure these tools will evolve in the same direction in the future.

We have reached a point in Python where we need to concentrate all the packaging effort to build a common standard in the standard library, because it is badly needed.

Here's a draft of the slides I will present tomorrow, during the first 5/10 minutes, as a session leader:
- Packaging Survey results overview - Topics to discuss
- setting up an organized network of mirrors (see PEP 381) - discuss about other PyPI enhancements - improve the package installation / uninstallation (see [PEP 376)][] - discuss the package dependencies problem and see if we can come up with a PEP - discuss the multiple version problem and see if we can come up with a PEP - discuss the isolated environment vs the OS vendor approach and see what can be done to improve their coexistence.

Summit schedule:
- 13:20 -> 13:30 : presentation - 13:30 -> 14:30 : discussions/work in small groups - 14:30 -> 14:50 : "tour de table" - 14:50 -> 15:10 : break - 15:10 -> onwards: sprint !

I am not sure about the 'work in small group' part yet, because I don't know how many people will show up, and what people will want to focus on.

How to upload your package to Plone.org

Tarek Ziadé — Sun, 15 Mar 2009 13:35:00 +0100

We have been working hard in the past few months with my Plone friends to make it happen.

It's now available : Plone.org is acting like PyPI in its Products section, which means that you can use Distutils to register and upload your packages in both places now.

I have written a small tutorial on plone.org to explain how to use it: http://plone.org/documentation/tutorial/how-to-upload-your-package-to-plone.org

I will also explain during my Pycon presentation (http://us.pycon.org/2009/conference/schedule/event/44) how this new feature at Plone.org will help people that are working with Plone and Python packages.

http://plone.org/documentation/tutorial/how-to-upload-your-package-to-plone.org

Take the Python Packaging Survey

Tarek Ziadé — Mon, 09 Mar 2009 05:41:00 +0100

The Python Langage Summit is coming up. To prepare this event, I have put online a survey you can take to tell us a bit more about you and how you package your Python applications.
- Who should take the survey : any Python developer that packages and distributes his code, no matter how. - Take the survey: http://tinyurl.com/packaging-survey

Thanks to all the people that helped building the survey, and a special thanks to Massimo Di Pierro who created the application that runs the Survey and helped me set up the survey. It runs under web2py.

Raising Distutils test coverage : half-way

Tarek Ziadé — Mon, 23 Feb 2009 13:06:00 +0100

After the next commit I will make in Distutils (that adds tests for bdist_rpm), the test coverage of this Python standard library package will be at 41%. This means that I have doubled the test coverage over the past few months, from 18% to 41%.

My goal is to double it again, and reach 80% in the next 6 months.

This also means I am just half an idiot now ! (since people who don't have 100% code coverage are idiots ;)).

So does it make Distutils more robust ?

It would have probably made the latest Python 3 release looks better for this package, since we had a uncovered cmp() call left in Distutils by the time the release was made. In the meantime, as I said before, the "real" Distutils regression test suite is held by all the packages out there in the community, that are built and installed everyday.

Python trunk Distutils test coverage : 41%

Name Stmts Exec Cover

--------------------------------------

__init__               3      0     0%

archive_util          77     61    79%

bcppcompiler         185      0     0%

ccompiler            453    211    46%

cmd                  180    134    74%

config                73     59    80%

core                  93     50    53%

cygwinccompiler      161      0     0%

debug                  3      3   100%

dep_util              43     11    25%

dir_util             109     76    69%

dist                 581    386    66%

emxccompiler         118      0     0%

errors                49      0     0%

extension             97     28    28%

fancy_getopt         233    126    54%

file_util            124     77    62%

filelist             161    102    63%

log                   46     21    45%

msvc9compiler        408      0     0%

msvccompiler         370      0     0%

spawn                 93     28    30%

sysconfig            323     51    15%

text_file            112     61    54%

unixccompiler        160     64    40%

util                 255    157    61%

version               68     62    91%

versionpredicate      61     51    83%

__init__               3      3   100%

bdist                 61     35    57%

bdist_dumb            57     47    82%

bdist_msi            322      0     0%

bdist_rpm            252    198    78%

bdist_wininst        170      0     0%

build                 60     54    90%

build_clib            90      0     0%

build_ext            334    160    47%

build_py             213    178    83%

build_scripts         78     65    83%

clean                 35      0     0%

config               185      0     0%

install              251    156    62%

install_data          44      0     0%

install_egg_info      40     32    80%

install_headers       25      0     0%

install_lib           97     50    51%

install_scripts       33     29    87%

register             173     82    47%

sdist                228    180    78%

upload               112     38    33%

--------------------------------------

TOTAL               7502   3126    41%

Python 2.5.4 Distutils test coverage : 18%
Name Stmts Exec Cover

--------------------------------------

__init__               3      0     0%

archive_util          78     11    14%

bcppcompiler         185      0     0%

ccompiler            453      0     0%

cmd                  180     79    43%

core                  93     15    16%

cygwinccompiler      160      0     0%

debug                  3      3   100%

dep_util              43      4     9%

dir_util             106     50    47%

dist                 578    342    59%

emxccompiler         118      0     0%

errors                49      0     0%

extension             97      9     9%

fancy_getopt         233    121    51%

file_util            121     50    41%

filelist             162      0     0%

log                   46     15    32%

msvccompiler         365      0     0%

mwerkscompiler       140      0     0%

spawn                 93      0     0%

sysconfig            296     10     3%

text_file            146      0     0%

unixccompiler        159      0     0%

util                 235     69    29%

version               68     48    70%

versionpredicate      61     51    83%

__init__               3      3   100%

bdist                 59      0     0%

bdist_dumb            57      0     0%

bdist_msi            320      0     0%

bdist_rpm            248      0     0%

bdist_wininst        159      0     0%

build                 52     47    90%

build_clib            90      0     0%

build_ext            304      0     0%

build_py             213    143    67%

build_scripts         78     64    82%

clean                 35      0     0%

config               185      0     0%

install              220    120    54%

install_data          44      0     0%

install_egg_info      40      0     0%

install_headers       26      0     0%

install_lib           96      0     0%

install_scripts       33     29    87%

register             171      0     0%

sdist                204      0     0%

upload               118      0     0%

--------------------------------------

TOTAL               7026   1283    18%

What's new in Distutils ?

Tarek Ziadé — Sun, 15 Feb 2009 10:35:00 +0100

Since Python 3.0.1 was released this week, here's a quick wrapup of what is going on in Distutils.

Code work (since one month)

New features

Issue 2563 : now the manifest is embed in windows extensions
Issue 4394 : the storage of the password in .pypirc file is optional now

Fixed bugs

Issue 4524: distutils was failing to build scripts with the '--with-suffix=3'
Issue 5132 : build_ext command was failing under Solaris with '--enabled-shared'
Issue 5075 : bdist_wininst was depending on the vc runtime

Refactoring

Issue 2461 : added test coverage for util.py
Issue 3986 : removed string and type usage from distutils.cmd
Issue 3987 : removed type usage from distutils.core

Documentation

Issue 5158 : added documentation for depends option for extensions
Issue 4987: updated README info
Issue 4137 : SIG web pages were updated

Design work

The main topics that are being discussed are:
- Improving the console script story. thread starts here. - Publishing a Survey on Distutils before the Language Summit. thread starts here. - Adding an uninstall command to unistall packages in the sdtlib : thread starts here - Adding a get_metadata API in pkgutil to get the metadata of an arbritary package The same code can also be used for various lookups (uninstall, console script) : thread starts here - Having a new PEP to make egg.info a directory and clearly define PKG-INFO and its match with metdata. thread starts here

A Distutils Regression Test System ?

Tarek Ziadé — Sun, 08 Feb 2009 17:18:00 +0100

I am making some progress in Distutils. I closed something like 10 bugs last week, and I am reaching issues that were added 8 months ago. Not that everything is entirely cleaned up in the newest issues, but they're almost all being processed. Every commit comes with at least a test, to get the code base back into a state were it is easier to make things evolve without the risk of breaking it up.

It comes through tiny little changes, with tests and an eye on the coverage.

Now I am facing an unpleasant situation : since the test coverage is still low, I am always scared of breaking something in Distutils when I am fixing a bug or making a change.Buildbots are watching, and I run some of my own packaging work with the current trunk.

But still, this is an unpleasant situation, and I don't want to cause the package to be broken in the next Python version...

But the regression tests exists ! They are there, hidden, in the community. It's everyone package.
1. Joe adds an issue in the Python bug tracker, because Distutils didn't work as expected on his package because of a bug 2. At some point the bug is (was) fixed. 3. The test to make sure the bug is fixed is "Joe is running Distutils over his package again, and makes sure it is properly installed, compiled, etc". 4. The bug is closed.

So how can I get back this test to make sure Joe's package is still working properly, so he doesn't hate us at the next major Python release ?

A Distutils Regression Test Server

If Joe's package is on PyPI, we can set something up. A dedicated server that watches the PyPI changelog and triggers a buildbot when:
- a new release of Joe's Package comes out - we change something in Distutils code

The precise test to be run is still unclear to me but, I am thinking about some generic strategies and I think it's possible. Let's call this test a distutils regression test. (If you have a better name, I'll buy it)

Of course it doesn't have to be on all the packages that are uploaded out there at PyPI. Just Joe's one, because he came up with a problem we fixed. And we would be ashamed if the bug comes back on Joe's package.

This requires of course a server, and probably a vmware-like system if Joe runs Windows or Solaris, to make buildbot slaves etc. It also requires that Joe uses the right metadata in his package so we know if it works under Python 2, Python 3, etc. MvL added enough classifiers lately for this.

A Distributed Distutils Regression Test

But some package are not on PyPI, for privacy or conveniency in the packaging process of the person in charge. So, what if the distutils regression test is provided in a Distutils command ? It can run the same test the server runs, and come up with a report that is sendable or sent by mail to a special mailing list or so.

This supposes that the developer is cooperative. So maybe it can even be automatically triggered in case of any failure on any Distutils command, and ask the user if he would like to send a report ?

The good thing here is that it doesn't require CPU power on the test server, and that anyone can run that test.

So what ?

Well I am just throwing an idea here, because I am really concerned about the potential regression problems. Even if Distutils is 100% covered with tests, it's not possible to test all combinations. The real world environment is the only test that can be trusted at the end in the packaging area.

I'll throw this idea at the Language Summit in March, and if it catches people interest, maybe a Google Summer of Code task could be done for that topic ? Can't implement it myself, I am overwhelmed already in Distutils maintenance :D

Just out of curiosity, how do *you* test your packages to make sure they get installed correctly ?

Building a survey for Distutils (Pycon's Python Language Summit)

Tarek Ziadé — Wed, 28 Jan 2009 11:48:00 +0100

I have the opportunity to lead a session at the Python Language Summit at Pycon.

In order to prepare this event, I am currently building a survey because we need to know how people use Distutils, what is wrong with it, and so on...

The draft is here : http://wiki.python.org/moin/Packaging%20Survey

When it's ready (hopefully soon), I'll post it in a SurveyMonkey-like system online for people to take it, and synthetize the results, so what I am saying at the summit reflects hopefully the reality.

If you are a Python developer, an OS packager, come to the Distutils-SIG mailing list and help us build the best survey possible !

(while the draft is in a wiki, it's better to discuss the changes in the mailing list before it is applied)

Singletons (and Borg) are unpythonic (well.. imvho)

Tarek Ziadé — Thu, 22 Jan 2009 23:19:00 +0100

Alex Martelli wrote a review about my latest book (can't find a permanent link on this, just look at Amazon.com you'll find it).

Amongst the negative parts there's one noticeable part I'd like to discuss in my blog, because I disagree with Alex's analysis.

Alex says:

This also holds for the chapter on design patterns, with such egregious claims as "Singletons should not have several levels of inheritance" -- they should have as few as practical and feasible, *exactly like any other class*; the desire to limit the number of distinct instances (which is mostly about STATE) is quite orthogonal to the issues with subclassing (which is mostly about BEHAVIOR). From this original "totally missing the point" follows a classic howler (which I've seen repeated in a review above): "why not use a module?". I have news for you, Tarek: a module supports *ZERO* inheritance -- which is quite a bit stricter than even the unjustified "should not have several level" claim above. Having to completely give up the usefulness of inheritance just because you want to limit instantiation would be a very limiting engineering tradeoff! If there's no need for inheritance then *of course* you want to use a module - DOH! - but if there IS (or if special methods can really help you) then it's not an option.

I think that the Singleton (and Borg) pattern is totally useless in fact. That's not the philosophy of Python in my humble opinion. And I think my book is right to advise people not to use this pattern.

I don't see the point of bending down a class so it only has one instance, where you can simply create an instance of that class in a module, add a "_" prefix to that class, and tell the world that this instance is your singleton. I don't see why a class should deal with that kind of STATE.

Frankly, I doubt that this singleton/borg pattern is really used in the community.

The only place where I really had to use singleton classes was in Zope. But that was more like a marker than anything else, and the class was registered under a "single" name in a global mapping (eg. its id in its container, in the ZODB tree). And in that case, we were creating one mixin class that used a singleton class, and we called it a "tool", with all the desired BEHAVIOR inside of it.

And well, if we would had several instance of it, for sure nothing bad could really happen, because the real unicity was provided by the id of the object. And nowadays those "tools" are going away and they are now called "utilities", and I don't think any singleton class is still really present or used. Just simply because a class is not the right place to enforce this.

I'd also say that there's an architectural problem when you enforce things like this in Python. If a programmer tells me that he wants to use a Singleton on his class because it holds a DB connector he wants to be instanciated once in his application, I am asking him right away to review the way the program is structured.

So what is the closest element in Python that will let you mark an object as unique ? what is the most convenient way to mark an object with an id in a container ?

A simple variable in a module. (or a simple declaration in a zcml file if you are a zopish guy)

I love Python for this because it's multiparadigm unlike Java : you don't have to set up over-engineered OOP stuff for this kind of needs (and it is surely not a tradeoff to use well engineered OOP besides).

Last, when I am claiming in Frenglish, that "Singletons should not have several levels of inheritance". This is just to warn people that, since these patterns are trying to break the way classes work, you might get screwed at some point when singletons are subclassed. A descriptor or a metaclass or whatever can just break your singleton stuff because Python was not meant to be used like that. It's not robust.

Blog title changed - Fetchez le Python mes amis

Tarek Ziadé — Thu, 22 Jan 2009 21:49:00 +0100

I have too many search hits about Carpet Pythons on my blog, and too many book reviews on how bad my English is.

Welcome to "Fetchez le Python", the technical blog on Python programming language, in a pure Frenglish style !

Python standard lib : give me more withs !

Tarek Ziadé — Tue, 20 Jan 2009 22:11:00 +0100

I used to write in files like this :
open('somefile', 'w').write(content)

It's ugly for sure, and a more proper way is :
f = open('somefile', 'w')

try:

    f.write(content)

finally:

    f.close()

But since Python 2.6, the with statement is superior for this code pattern:
with open('somefile', 'w') as f:

    f.write(content)

This is so natural in fact that I am always thinking about with when I work with classes that have a start/stop or open/close behavior.

So, what about adding this behavior into imaplib.IMAP4, ftplib.FTP and smtplib.SMTP ?

So we can write things like this :
>>> from ftplib import FTP

    >>> with FTP('ftp.somewhere.com') as ftp:

    ...     ftp.login('someone', 'pass')

    ...     (some code)

    ...

I am working on a series of patches for this, and wondering if some other classes in the standard library could benefit from this as well..

Distutils : improved .pypirc for Python 2.7 and 3.1

Tarek Ziadé — Fri, 09 Jan 2009 08:30:00 +0100

When you launch such a command:
$ python setup.py register sdist upload

There's no way to give to Distutils your PyPI password in the prompt, so you distribution is uploaded to the server. You have to store your password in the .pypirc file:
[distutils]

    index-servers =

        pypi

    [pypi]

    username: <username>

    password: <password>

The password is stored in clear text, so it can be used by Distutils to authenticate. This is rather unsecure, since anyone who has a read access to your home can get your password.

I have detected this problem this summer while listing the possible enhancements in Distutils. Nathan Van Gheem sent me a mail a month ago to ask for that same feature in collective.dist; which is a port of the latest Distutils features into Python 2.4 so Zope can use them. So before having it into collective.dist, the first step was to introduce it into Python itself.

The idea is to be able to remove from .pypirc the password so it's asked at the prompt. Nothing fancy here : the Distribution object that is created before you launch any command is the place where you can share a context between commands.

So when you launch:
$ python setup.py register sdist upload

Here's what is happening:
1. register looks into .pypirc, if no password is found, it asks it to the user using getpass 2. register use it then store it in the Distribution instance 3. upload look into the Distribution instance to see if the password was stored, and use it

This is now available in Python 2.7 and 3.1, and heavily tested.

I'd like to go further and to think about a ssh-agent like system, so there's no need to enter the pasword everytime you work with PyPI in the same session.

Does anyone knows what would be the way to do it properly ? I think a ssh-agent like mechanism in Python's getpass would be a great feature itself.

2009 plans, part #1 : Distutils

Tarek Ziadé — Sun, 04 Jan 2009 19:52:00 +0100

Happy New year all !

I am going to make a few posts on the things I would like to achieve in 2009. Each entry will focus on a topic. This one is about Distutils.

I was granted a commit privilege in Python, specifically to work on Distutils maintenance. This is a huge privilege, and I try will do my best in this job. I have worked on a few tickets already and closed some. I learnt the Python development process, which requires to backport and to forward-port changesets in various Python versions. While this can be taken care of automatically by someone else if you don't do it, it's better that every commiter takes the time to merge his own work.

So what's next ?
- There are 132 tickets that are open in the Python tracker, that match the word distutils, and some of them are 5 years old ! - There's a Python language summit to be held in Chicago right before Pycon, and I volunteered to champion the task about Distutils, PyPI and packaging matters.

I am planning to :
- review and classify all the tickets in the tracker; - fix the maximum amount of them before the summit; - make Distutils a first class citizen in test coverage; - make Distutils code more modern.

Besides, I will try to build a roadmap for Distutils I will present in Chicago.

To build this roadmap, I will ask for input in the distutils-SIG mailing list in the coming days, and see what people will come up with. There's no crowd in this list these days but sometimes some threads are hot when it comes to the future of packaging in Python.

The roadmap I am planning to build will not address all the issues people have when it comes to distribute a Python application, since there is no consensus yet on the best practices. It will rather try to see if the current version of Distutils can be enhanced to adress some problems, and at least be the bridge to something new in the future. Maybe by including some best practices from third-party tools (the pre-condition for all of this imho is to make the Distutils code base healthier).

Anyway, I hope that the lead developers of: zc.buildout, pip, setuptools, paver (and those projects I forget about right now) will participate in this discussion, and that we will be able to find pragmatic enhancements.

Pycon 2009 talks

Tarek Ziadé — Tue, 16 Dec 2008 23:32:00 +0100

I have 2 accepted talks at Pycon, that is great. I would like to say that the Pycon review system is awesome because you can see what the reviewers have said, and understand why your talk was accepted or declined.

I was a bit frustrated that my Atomisator talk was declined, but I think it makes sense : this is a new tool, and beside my user group and a few people, it is not really used yet.

One reviewer said that it had to be picked, and another one answered :

I agree that PyCon should not restrict itself to well-known projects, but it should definitely restrict itself to projects that are (a) in production use, (b) under active development, and (c) likely to still be so in a year. There are so many projects meeting these criteria that for me, the bar is very high indeed to spend a talk slot on one that does not.

Ok, fair enough : I will present this talk at Pycon 2010 and they won't have any argument to decline it ;)

The talks that made it:
- How AlterWay releases web applications using zc.buildout - On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes

I will get into greater details later on.

Python Isolated Environment (PIE)

Tarek Ziadé — Mon, 15 Dec 2008 10:24:00 +0100

Here's a proposal I will send to the python-dev. What do you think ?

(Disclaimer : this proposal is highly inspired from the work done by people in various tools, it does not reinvent anything)

The problem

Python developers distribute and deploy their packages using myriads of dependencies. Some of them are not yet available as official OS python packages. Even sometimes one package conflicts with the official version of a package installed in a given OS.

In any case, the cycle of development of most Python applications is shorter than the release cycle of Linux distributions, so it is impossible for application Foo to wait that Bar 5.6 is officialy available in Debian 4.x.

Therefore, there's a need to provide or describe a specific list of dependencies for their application to work.

And this list of dependency might conflict with the existing list of packages installed in Python. In other words, even if this is not a wanted behavior from an os packager point of view, an application might need to provide its own execution context.

Right now, when Python is loaded, it uses the site module to browse the site-packages directory to populate the path with packages it find there. .pth files are also parsed to provide extra paths.

Python 2.6 has introduced per-user site-packages directory, where you can define an extra directory, which is added in the path like the central one.

But both will append new paths to the environment without any rule of exclusion or version checking.

The workarounds

A few workarounds exist to be able to express what packages (and version) an application needs to run, or to set up an isolated environment for it:
- setuptools provides the install_requires mechanism where you can define dependencies directly inside the package, as a new metadata. It also provides a way to install two different versions of one package and let you pick by code or when the program starts, which one you want to activate. - virtualenv will let you create an isolated Python environment, where you can define your own site-packages. This allows you to make sure you are not conflicting with a incompatible version of a given package. - zc.buildout relies on setuptools and provides an isolated environment a bit similar in some aspects to virtualenv. - pip provides a way to describe requirements in a file, which can be used to define bundles, which are very similar to what zc.buildout provides.

But they all aim at the same goal : define a specific execution context for a specific application, and declare dependencies with no respect to other applications or to the OS environment.

This proposal describes a solution that can be added to Python to provide that feature.

The solution

A isolated environment file that describes dependencies is added. This file can be tweaked by the application packager, or later by the OS packager if something goes wrong.

The isolated environment file

A new file called a Python Isolated Environment file (PIE file) can be provided by any application to define the list of dependencies and their versions.

It is a simple text file with a first line that provides :
- a list of paths, separated by ':', on line 1 - then one package per line, starting at line 2. each package can be prefixed by a `!`

For example:
/var/myapp/myenv

lxml

sqlite

sqlalchemy

!sqlobject

This list of packages might or might not be installed in Python.

Versions can be provided as well in this file :
/var/myapp/myenv:/var/myapp/myenv2

lxml >= 0.9

sqlite > 1.8

sqlalchemy == 0.7

!sqlobject == 0.6

The file is saved with the pie extension,

Loading an isolated environment file

A new function called load_isolated_environment is added in site.py, that let you load a PIE file.

Loading a PIE file means:
- for each package defined, starting at line 2, load_isolated_environment will look into the environment if the package with the particular version exists. The version is given by the package.__version__ value or the PKG-INFO one when available. If the package exists but the version is not available, the version 0.0 is used. - for packages without the ! prefix:
- if the package is not found, it will scan each path provided on line 1 of the file, using the site-packages method, looking for that package. - if the package is found, it is added in the path. - if the package is not found, a PackageMissing error is raised.

for packages starting with the ! prefix:
if the package is found, it is removed from the path

This function can be called by code like this:

from site import load_isolated_environment

>>> load_isolated_environment('/path/to/context.pie')

From there, sys.path meets the requirements and the code that is executed after this call will benefit from this context.
Another context can be loaded in the same process :

load_isolated_environment('/path/to/another_context.pie')

Limitations:
- if the new context brakes other programs in the process. It's up to the application packager to fix the context file. - it's not the job of load_isolated_environment to resolve dependencies issues : if the foo package needs the bar package, it won't complain. - it is not the job of load_isolated_environment to get missing dependencies.

Using an isolated environment file

Typically, an isolated environment file can be used into high-level Python scripts. For example, any script an application provides to be launched :
# this script runs zope

from site import load_isolated_environment

load_isolated_environment('zope-3.4.pie')

import zope

if __name__ == '__main__':

    zope.run()

Looking for beta testers for Atomisator

Tarek Ziadé — Sun, 14 Dec 2008 13:57:00 +0100

I am looking for beta testers, interested in customized rss feeds or email alerts experimentations.

Here's a list of services Atomisator can provide :
- You run a project and you would like to receive a daily summary in your mailbox on what is being said about it in blogs, tweets, etc - You have a list of feeds you want to aggregate, with specific filters and you can't manage to do with Yahoo pipes or any tools out ther, because it is too specific. - You want to annotate entries in a feed with extra information - etc..

What you get as a beta-tester:
- a custom Atomisator configuration that fills your needs - I am hosting the service, and you get
- either an url on my server to an xml file you can read in your aggregator - either one mail per day

What you are not getting as a beta tester:
- you don't get any guarantee on the output or the reliability, these are just experimentations. - if it's down I can't promise when it will be up again

Let me know by mail if you are interested

Pycon 2009 proposals

Tarek Ziadé — Fri, 12 Dec 2008 08:18:00 +0100

The proposal acceptance date is in a few days.

Here are the four proposals I have made:
- The state of packaging in Python. This discussion resumes the current options when it comes to distribute your packages. It also explains the pitfalls and the gap between the Python developers and the OS vendors and packagers. I think this talk will not be picked because the topic is wide and vague. So I proposed to transform it into a panel where lead developers from various framework could explain their usage of distutils and what is missing to make them happy. No feedback yet on this. - Atomisator, the agile data processing framework. This tool is starting to be useful, and I think it can be useful to others. Check http://atomisator.ziade.org for a quick overview. - How AlterWay releases web applications using zc.buildout. That is the same talk I gave at the Plone conf but I present it in a way people understand zc.buildout is not tied to Zope and Plone and can be used with any other application. As a matter of fact, it has become a standard here, and we use it for Pylons, etc.. - On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes. That's a long title. It presents my work on PyPI.

Last, I will go to the Python Language Summit the day before Pycon. I volunteered to be a "champion" on distutils matters.

How to make binary distribution of buildouts

Tarek Ziadé — Tue, 09 Dec 2008 01:26:00 +0100

The Problem

I need to distribute pre-compiled buildouts because some projects don't allow us to have gcc installed on the production system for security reasons.

Fair enough, we need to provide a pre-compiled buildout.

If you want to distribute your buildout-based Plone application in a binary form, so it can be installed without requiring any compiler on the platform, you need to compile all .c modules before you provide a tarball of your buildout folder.

This is easy : just run your buildout and all .so files will be created in the zope 2 installation. (.pyd under windows)

But this will work only if you compile in a directory that is located within the same path on the target machine, because zc.buildout uses absolute paths when it builds scripts.

Furthermore, if the python interpreter is not located in the same place, your buildout script itself is screwed.

Last but not least, plone.recipe.zope2install is not clever enough. It will remove your zope2 installation when it detects that the path has changed. This is pretty annoying even if you have gcc : what is the point of compiling the c extension again since they
are statically compiled in-place ?

The solution

I have changed plone.recipe.zope2install and added a new option called `smart-recompile` (in trunk right now, not released).

If you use it, the recipe will check for .so or .pyd files before trying to ditch your zope 2 installation and recompile it. Even if you don't use it to build binary distributions, it will make your buildout build faster if you already have zope compiled in there.

Next, I have created a special bootstrap.py, who is clever enough to rebuild the buildout script with the right path to the used interpreter, and with offline-mode capabilities. To make it short : boostrap.py works no matter if you have an internet connection or not. Grab it here : http://ziade.org/bootstrap.py

So now, basically you can compile your buildout and deploy it on any system, on any path, without any internet connection, like this:
$ python bootstrap.py # will rebuild the buildout script

$ bin/buildout

Of course this doesn't work if you have dynamically compiled extensions like python-ldap. For theses, the best pick is to rely on the system ones.

A PostRank plugin for Atomisator

Tarek Ziadé — Sun, 07 Dec 2008 11:54:00 +0100

Yesterday, I bumped into PostRank. This system is collecting data from various social systems like Twitter and provides a service where you can type in an url of a blog post or a entire blog. You get a PostRank depending on the popularity of the URL.

I wrote a plugin for Atomisator and ran it on my own blog. Here's the result: http://ziade.org/afpy/

And the Atomisator configuration for this is :
atomisator

sources =

    rss http://tarekziade.wordpress.com/feed/atom/

database = sqlite:///carpet.db

outputs =

    rss  public/rss.xml "http://tarekziade.wordpress.com/feed/atom/" "Carpet Python with PR" "Powered by Atomisator"

enhancers =

    postrank

How PostRank works

PostRank works with urls you provide, on their web interface or through their web services.

As long as these url are present in their big cloud-computing based system, they provide a rank that is calculated with the number of comments related to the blog, the number of tweet messages that refers to it, and so on. The complete algorithm they used is secret but this is not the point. I have secret algorithms too ;).

The point is that they are trying to categorize blog entries using social networks as indicators, and that they have a huge database.

Social indicators in Atomisator

This is one of the approach I have with Atomisator, when it is used to build a planet. For instance I have a Digg plugin that will inject in each entry the comments found on Digg if the entry was digged. It also present the number of Digg. Of course this is done live because I don't have a cloud-computing based system where I store data. I use Digg webservice on the fly. (On the fly here doesn't mean Atomisator make the calls to Digg from the Planet application of course. It means Atomisator calls them when it creates the merged feed on the system)

The benefit of this approach is that I can provide a social indicator on a post immediatly. Systems like PostRank will not work on entries that are too recent because their spiders have a lag of one week or so.

The pitfall of my approach is that I am unable to calculate trends because I don't store the indicators as they vary.

But if someone wanted to build a BtoC application using Atomisator, they could implement a set of plugins based on Amazon tools to make them store data in a more scalable way and in time.

Next steps

So I have this new PostRank plugin, and this is awesome because I have added a treshold parameter in it. Basically if a post has a high PostRank value, it will appear in the Planet. If it's low, it can be automatically removed. The fact that PostRanks are lagging for new entries is not a problem: interesting posts will eventually pop after a few days in the Planet.

This is perfect to reduce the number of entries in an aggregator.

But I do want to write my own PostRank that works live, with no storage at all. Because the whole point of Atomisator is to provide a framework where anyone can try out various filtering combinations.

So to be able to provide this power, it needs to work just by collecting data directly from the social services, like the PostRank plugin does with this PostRank "meta-service". The next step is therefore to see if I can query services like Twitter to list the twits related to an url, without having to store the twitter feed myself.

In any case, if my talk on Atomisator at Pycon 2009 is selected, the PostRank plugin will be shown besides the Digg plugin.

Expert Python Programming Book : typo sprint tonight !

Tarek Ziadé — Thu, 27 Nov 2008 21:03:00 +0100

I love Packt. As soon as I have told them that some people liked the book but complained about the typos, they proposed to go ahead and launch a new print cycle.

Basically it means that the next buyers will have a typo-free book. At least for all the typos that were reported on my Trac here, or at Packt's.

I am currently processing all the typos reported at Packt so I have a full list on my wiki, and will provide them the final list tomorrow.

Then, they will re-print it.

So if you already own the book, and you see a typo that is not listed, please let me know.

Python package distribution - my current work

Tarek Ziadé — Wed, 26 Nov 2008 00:16:00 +0100

I found a bit of time to work on distribution matters. Here's a status of what I am doing there.

There are two topics I am focusing on right now.
- clean up and enhance Python's distutils package - implement the mirroring infrastructure at PyPI

distutils work

Nathan Van Gheem proposed a cool patch in collective.dist, (this package is a port of the new features I have added in distutils so they are available in 2.4 and 2.5).

Nathan proposed a patch to be able to avoid the storage of the password in the .pypirc file. The prompt is used in that case. This is something that was in my pile for a long time.

I have added a few things to Nathan's patch, and a test, and proposed it to Python. I am now waiting for its integration in 2.7 trunk: http://bugs.python.org/issue4394. If it's accepted, I will backport it to collective.dist.

There are some other tickets I am waiting to be accepted:
- http://bugs.python.org/issue4400 - http://bugs.python.org/issue2461 - http://bugs.python.org/issue3992 - http://bugs.python.org/issue3985 - http://bugs.python.org/issue3986

I am not sure when those will be integrated. The average time for the integration of tickets in distutils in Python is between 6 months and 8 months. hihihi. :D

PyPI mirroring

The job I am doing in PyPI will be in three phase :
- Phase 1: implement the mirroring infrastructure in PyPI - Phase 2: promote it, and propose patches for the mirroring tools out there so they use the protocol - Phase 3: promote and propose patches for pip so it can use the mirrors efficiently (fail-over and nearest mirror infrastructure).

Phase 1: so far, so good.

With some insights from Richard Jones and Martin von Löwis, I am currently implementing the mirroring infrastructure for PyPI we have defined during the D.C. sprint (I still owe a blog entry about this sprint). The code lives in a branch on the python svn folder dedicated to PyPI.

The idea of the mirroring infrastructure is to be able to get a list of official mirrors for PyPI, that can be used as alternatives sources . (It is described here: [http://wiki.python.org/moin/PEP_374][]). A great behavior could be that the client application interacts with the nearest mirror location automatically, and switch to another if it goes down.

So, a list of mirrors will be made available at /mirrors, and the client applications will be able from there to use an alternative location for every package. The hardest part concerns the stats : we want to display in PyPI the download counts for each package by summing downloads from every mirror.

So every mirror will have to provide its "local stats" that can be visited by PyPI. That's the biggest part of the work I am doing. It will build the stats for PyPI by parsing its Apache log file. And hopefully, this code should be reusable by the mirrors themselve so they can build their stats the same way.

Of course this infrastructure could be used for any PyPI-compatible server even if is not a mirror of PyPI (like a private PyPI server)

Phase 2 will consist in promoting the infrastructure to the mirroring softwares out there. Maybe Pycon will be a good place for that.

Phase 3 is the most interesting one : make sure the client applications use the mirrors ! I think Ian Bicking's pip project could be the right place for these innovations.

Next topics in the pile:
- index-merging: describe in a PEP-like document the index-merging feature that would allow clients to merge several indexes with a content that differe. For example: PyPI + a private PyPI server. I have written a first draft of such a patch in setuptools in the past (http://bugs.python.org/setuptools/issue32) but I have lost all my hopes to see this project moving forward lately. - Brainstorming: try to understand the Python Packaging Paradox. That is = how come the community, which is composed of many briliant people, is unable to move forward in packaging matters. - Distribute the return :D

How to be disappointed with the "printed" in "printed book"

Tarek Ziadé — Sat, 22 Nov 2008 17:23:00 +0100

I feel really bad about this comment on my book : How To Be Dissappointed in Something You Recommend.

Just a quick word about the try, return finally code pattern, since I had some feedback about it. I would like to mention that this code pattern is perfectly right:
def function():

    try:

      return something

    finally:

      do something

I should have explained it better, because this pattern is not used a lot by people, so you can think that "do something" is called after the return of the function, which is not the case.

For the typos now:

The first thing I did wrong: when I started the book, I wanted, as I did in my previous book, to run unit tests on the book itself to avoid those mistakes. That said, the previous one was in Latex, which is quite simple to interact with, and this one is in OpenOffice, because that is how the editor works. I had to write a script to extract the Python code from the Ooo file, to unit test it. I didn't. I simply ran out of time, as usual when you have deadlines on books.

The second thing I did wrong: I should have told the editor to wait a bit, I didn't.

But Packt does Print On Demand, so I know that the Errata page I am maintaining here : http://atomisator.ziade.org/wiki/Errata, is being processed by the editor, and that the typos will be removed from the book at some point, without having to wait for a second edition.

I'll update this blog entry as soon as I know the status on this.

I am really sorry Calvin, and all the people that are suffering from these typos.

How to receive email alerts when someone talks about something - 6 steps tutorial using Atomisator

Tarek Ziadé — Sat, 08 Nov 2008 12:38:00 +0100

I like Google Alert, the idea of receiving a mail every day that summarizes all articles related to a given topic is really helpfull when you need to focus on a specific subject for a while.

But this is not enough. I want to receive a mail that points me to any mailing list or planet feed or blogs out there as well, that talks about the topic.

You can't do it with Google Alerts as far as I know.

Let's take an example:

I want to receive a daily mail that points me to any mail thread or blog entry, that is related to the word "buildout" or to the word "pycon".

Basically, to do it manually, I need to read Planet Python, Planet Zope, then take a look at the Python, Zope and Plone mailing lists. It takes at least 10 minutes, and more if you want to read all entries to make sure you won't miss anything.

Since online systems like Nabble provides RSS feed for mailing lists (don't find yours ? just add it there !), it is easy to read them as they where regular feeds.

From there, a script that reads all the selected feeds and sends a mail pointing to the entries that match the selected words is simple to write as well, and fill the need.

But don't code it : Atomisator will let you do this with a few lines of configuration.

Here's a step-by-step tutorial.

Step 1 - install easy_install
- download ez_setup : [http://peak.telecommunity.com/dist/ez_setup.py][] - run it with your Python interpreter

Step 2 - install Atomisator and SQLite

- run the command : easy_install Atomisator - make sure SQLite is installed on your system. If not, install it: http://www.sqlite.org/download.html*
*

Step 3 - create an "atomisator.cfg" file

The content of the file has to be:
atomisator

store-entries = false

sources =

  rss http://www.nabble.com/Python---python-list-f2962.xml

  rss http://n2.nabble.com/Plone-f293351.xml

  rss http://www.nabble.com/Zope---General-f6715.xml

  rss http://planet.python.org/rss10.xml

  rss http://www.zope.org/Planet/planet_rss10.xml

filters =

  buzzwords words.txt

outputs =

  email email.cfg

This file will look into Planet Python, Planet Zope and various mailing lists (Python, Plone, Zope). Of course you can add or remove feeds in the sources option.

Step 4 - Create the words.txt file

This file contains regular expressions, one per line, that will be used to match the entries. The file has to be saved besides atomisator.cfg.

For our example:
buildout

pycon

You can put any expression you want in this file, as long as you have one matching expression per line.

Step 5 - add an email.cfg configuration file.

This is where you define the target emails that will receive the alerts (tos option). You can also specify the from email, or the smtp server location. The file has to be saved besides atomisator.cfg.

In our case it can be:
[email]

tos = tarek@ziade.org

from = tarek@ziade.org

smtp_server = smtp.neuf.fr

Step 6 - Run it !

The command to be called is atomisator (installed by easy_install) followed by the configuration file:
$ atomisator atomisator.cfg

Reading data.

Launching worker for rss - ('http://www.nabble.com/Python---python-list-f2962.xml',)

Launching worker for rss - ('http://n2.nabble.com/Plone-f293351.xml',)

Launching worker for rss - ('http://www.nabble.com/Zope---General-f6715.xml',)

Launching worker for rss - ('http://planet.python.org/rss10.xml',)

Launching worker for rss - ('http://www.zope.org/Planet/planet_rss10.xml',)

Retrieving from rss - ('http://www.nabble.com/Python---python-list-f2962.xml',)

Retrieving from rss - ('http://www.nabble.com/Zope---General-f6715.xml',)

Retrieving from rss - ('http://n2.nabble.com/Plone-f293351.xml',)

Retrieving from rss - ('http://planet.python.org/rss10.xml',)

Retrieving from rss - ('http://www.zope.org/Planet/planet_rss10.xml',)

.................................................................................................................................................

Writing outputs.

Data ready.

Check your mails. This call can be put in a daily cron.

Tested under Mac OS X and Linux.

Plone Conference 2008 in Washington D.C. - summary

Tarek Ziadé — Thu, 06 Nov 2008 16:54:00 +0100

I am back from the Plone Conference in D.C., and the jetlag is gone. The jetlag is gone for weeks now but it's hard to find the time to blog these days :/

On the talks I have seen and topics I have chatted about

There were a lot of great talks in D.C., and it was hard to decide which one to look at. In any case it was easy to meet the speaker if I had missed the talk, because the Plone Conference, unlike big conferences like OSCON, is a place where everyone hangs around the same spot after a talk is over.

Here's a list of some topics I have seen or I have talked about with some people.

Deliverance - Ian Bicking

If you look at what Ian has produced in the past 5 years, he is one of the most prolific contributor of tools that become standards in the Python web development web community. Think about Python Paste or virtualenv, and many others. Deliverance might be the next big one.

Take a bunch of micro web applications you want to join to build a full web system, for historical reasons or just because you believe a particular feature just won't fit in Plone but will do great in Pylons.

Now ask a designer to glue everything together under the same look. He (or the guy that integrates his design) will probably hates you: he will have to learn how to integrate in heterogeneous environments. This is easy under some systems that let you stick a layout and a css in a simple way. This is not easy under Plone, unless you learn how to do it (but this will be improved in the future).

Deliverance is a proxy that let you skin any application that spits html content, by running some XPATH rules on the content and applying some changes to produce a new output. Basically, you have a simple html page that just provides the layout you want to have, without any content, and a xml file that explains how to extract some content from the page produced by the third-party application and where to inject it in your empty html page. The great thing is that you can call different third-party servers given the path you are in, and even call several servers to build one single page. This opens a lot of perspectives.

The first caveat of this approach is that you have to provide a
Single-Sign On feature to avoid people having to connect several times.
This can be a problem sometimes with some applications if they are not
open enough to let you do it. But most of the time, it is not a problem: if the users are all located in a LDAP it is easiy.

Furthermore, if you use only Python-based applications, you can use a WSGI envrionment and a middleware like repoze.who to glue together let's say, a Plone app and a Pylons app. Products.oopas is the PAS plugin that can be used for that on Plone side to grab the authentication context and use it.

The second problem I can see is about response headers. One example: if a page is composed of elements that comes from several pages, and if the page has a Last-Modifier header, I don't think Deliverance handles this correctly yet, to make sure to present the newest Last-Modified header from all third-party servers that where called to build that page. But this more likely to be a detail compared to the single authentication problem.

In any case this is a very promising tool !

Content Mirror - Kapil Thangavelu

I didn't see that talk, but I have talked about this tool with a few people. The idea is to serialize the content of a Plone instance into a relational database (eg Postgresql), as it happens, using events.

I need to give a try and check it deeper, to see how the overhead is dealt, and how the aggregator I have read about is doing (it collects mirorring operations to perform in a transaction, and optimize the calls at the end of the transaction to avoid redudant calls if I understood correctly). I don't know yet for example if there's a pool of jobs for the mirroring tasks to avoid a point of failure. But I am pretty sure this is taking care of. The other point I need to see if there's a round trip. e.g. if there's a way to apply a relational database change back into Plone.

But in any case I can already see various use cases for my customers. For instance, having a plone instance as a back office, with complex workflows for editors and contributors, and a lightweight Pylons application as the front application, that concentrates into displaying the relational database as fast as possible, makes a lot of sense in big environments. It just scales better.

So this is a interesting tool as well.

repoze.bfg - Chris McDonough

Chris gave a talk about repoze.bfg, which is a new web framework that takes back the good bits from Zope and push them into a WSGI world, using the Pylons approach I would say. That is : "here's the template engine you can use in repoze, but really, use the one you like".

Frankly, I am really seeing this new effort as one of the most promising one in the Zope community. Already, repoze.auth is a major middleware in WSGI : Zope's Pluggable Authentication Service outside Zope, usable with any WSGI application. This is a blast !

And people are starting to contribute a lot of interesting middlewares under the repoze namespace.

Now I didn't really try repoze.bfg itself yet, but given the people that are behind it, I am pretty sure this framework will meet success in the future. Having a MVC framework ala Pylons that let you use Zope packages with a "this zope package is repoze/wsgi compliant" label on each one of them is very cool.

collective.indexing - Andreas Zeidler and al

At the snow sprint, we worked with the Enfold crew that did a great work in integrating the Solr/Lucene system so it can be used from Plone. We replaced a few fields like the searchable text and indexed it on Solr side, just to give it a try. The snow work was really focusing on providing a buildout, a few recipes and a bench to say : "Hey, Plone community, this is a blast ! let's do more of it"

Later Andreas Zeidler and a few other guys continued the work on indexing matter and they delivered collective.indexing, which provides two things:
- a queue that collects all indexing to be done, and optimize the call to the catalog - a bridge to use collective.solr

I didn't follow the latest development and I didn't know how far the guys went, but I had the chance to hang around with Andreas and Tom Lazar in D.C., so now I know that this package is production ready :D

So in other words : I'll probably use it as a mandatory package for all the big plones out there.

The queuing part imho, should go into the catalog itself because there's no other way to make sure a third-party product is not calling the catalog during the transaction wile another product does the same.

Server-Side Include (SSI)

Tom Lazar worked during the Snow Sprint on lovely.remoteinclude to make Plone portlets accessible via unique URLs. From there, it is possible to push a page that contains a list of urls rather than the calculated page, to a front server that knows how to read SSI directive, and builds the page.

This is great for performances, and is a lot like ESI (Edge Side Include) we use to have in CPSSkins.

I am wondering if both could be implemented in the same tool in fact.

Tom told me that he will try to continue this work at the performance sprint in Bristol in december, so let's keep an eye on this !

I have seen many other talks and topics, but these few ones where the ones I really needed to talk about.

On the conference organization

I am helping in the organization of Pycon FR in Paris since 2 years now. I know what is means to organize such events : it is a LOT OF WORK.

You know when an event is well organized when you don't feel it is organized.

That was the case in D.C. Bravo Alex, Amy and all the others !

The only problem (wifi) was not the organizers fault, and I have never been to any event where it is not cahotic at some point (besides OSCON) so... :)

On the community

I love you all guys. It is an amazing community.

PloneConf'08 slides + screencasts : delivering applications with zc.buildout and a distributed model

Tarek Ziadé — Tue, 28 Oct 2008 15:37:00 +0100

I was totally drowned into some customer projects since I came back from the Plone Conference. But things are looking better now, so I can take a bit of time to start blogging about the conference. I'll probably do three blog posts: this one about my tutorial, the next one about the conference itself and last, an entry about the sprints.

So I gave a tutorial about zc.buildout. The length was a bit challenging, since I had 90 minutes. Enough time to explain more things than in a regular talk, but not enough time to get into great details, as a tutorial should be.

The other thing was about the topic: two talks were covering zc.buildout, Clayton Parker's one and mine. So my goal was to make sure they were not overlapping too much.

I had the chance to meet Clayton before we both gave our talk, since the Six Feet Up crew gave me shelter in the house they rented (nicest guys in the block). Even if we didn't exchange a lot on the slides themselves, I could figure what Clayton was going to present. So I.... started my slides from scratch two days before my tutorial and worked carefully on their scope :D

Alex Clark came and backed me up during the talk, since we are working together one plone.org for months now and since the talk presented the new plone.org that is coming up.

I think I did quite well during the talk, because we had a pause half way, and when we started back, the room stayed full ;)

This is the second time I record all the console work in small screencasts, to avoid live problems, and I think this is the best way to go if you need to do some demos, so I'll keep on doing it. Plus, it's nice to provide them to people after the talk.

Anyway, the talk was videotaped so you can to judge by yourself:
- The video - The slides .

And everytime you see a "get the screencast at http://ziade.org/ploneconf" in there, well get them :)

In detail:
1. Distutils demo 2. setuptools demo 3. collective.eggproxy demo 4. PloneSoftwareCenter installation 5. collective.dist demo 6. new.plone.org demo 7. collective.releaser demo 8. plone 3 buildout demo 9. Multiple target releasing demo

What's next ?

We need to finish the work with Alex on plone.org. It's not hard, it just takes time, and we both are quit busy in our jobs :)
I need to polish collective.releaser and collective.eggproxy. They are brut de fonderie, and the code suck a bit. If you are using them and want to help, or have some feedback/issues, please, pretty please, let me know.

Distribute sprint has started in D.C. (setuptools, distutils, PyPI)

Tarek Ziadé — Sat, 11 Oct 2008 15:05:00 +0200

Just a quick note:

We have started the sprint in D.C. on packaging matters. Our tasks for the moment:
- re-read all the threads in distutils-ML since mid-september. - write a short list of actions that can be done NOW in distutils, setuptools and PyPI - write a short list of writing that can be done to define a "new" tool

Please, join at any time in #distutils in freenode, and ping us, if you want to get involved.

Plone shirt for my funders

Tarek Ziadé — Thu, 02 Oct 2008 01:22:00 +0200

[caption id="" align="alignnone" width="400" caption="JJ shirt"][/caption]

So I promised I would create a shirt for the top #5 contributors of my fund raising.

I think I found an idea. My friend JJ is wearing a shirt I made on the picture you see. That's the phrase we wanted me to put on the shirt:

There's no place like 127.0.0.1

(except maybe ::1)

That is one of the geekiest shirt I know of :D

Anyway, I want to borrow the idea for the plone shirt, and write, beside the Plone logo:

There's no place like 127.0.0.1

(except maybe #plone on freenode)

That is, to express the fact that the Plone community is a really friendly one, and it's great to be part of it.

EDIT: I have better proposals from the two Alex:

$ whois plone

127.0.0.1

and:

There is no place like Plone

and:

Plone, I wish I knew how to quit you

and:

Did you mean Plone ?

I am hesitating now.... :)

"JJ shirt"

Simplifying the commenting system

Tarek Ziadé — Fri, 26 Sep 2008 16:05:00 +0200

They are many commenting systems out there for Plone, Zope 3, etc.

But let's take this simple statement : why a content provider like any Zope-based application should deal with comments ? A comment is a simple piece of text, that refers to a page and maybe to another comment. There are no workflow needed in most cases. Maybe an authorization required to add the comment but that's it. And how do you provide a comment on a page that is not generated by your web server, but that is displayed somewhere to your users nevertheless ? (like a static page)

So, looking at how Deliverance works, and because I want to have comments on Sphinx pages, I thaught of a simple way to deal with comments: A WSGI Middleware/Proxy.

The comment application stores comments in a SQL DB and use the pages id (the full URL) as references. I can even use services like Akismet to avoid spams. Then when the page is displayed, I can inject the comments associated to it on-the-fly, and provide a JS-enabled form for people to add comments (which can be caught when submited by the middleware too I guess, or sent through Ajax to a small JSON-enabled service that listens)

This is perfect for an hybrid environment where you have pages generated by various web applications.

Of course there are some caveats, like if the page is gone, or has moved. But dealing with orphan comments should be doable.

We might try that out soon for Afpy.org :)

Distribute, end of the fork -- or the start of a new hope

Tarek Ziadé — Fri, 26 Sep 2008 08:22:00 +0200

(This post is a work in progress, as things still evolve)

In a way, I am glad I have made a fork, and I am glad this is going to be the shortest fork ever. A lot of people reacted on my proposal and I could get a very clear picture of what is wrong in the distutils/setuptools world, and how I could help on this.

This is how I interpret it:

Guido explained that the mistake made was not to integrate setuptools in the core from the very begining. The current distutils code did not evolve in the meantime and Phillip Eby worked on setuptools to make it evolve. setuptools now reaches a point where it needs other contributors to fit all the needs and solve more problems. It had one contributor in the past, (Jim Fulton) but it needs more. It also became a mandatory package for many folks in the Python world because Phillip did a great work on it : it solves most problems.

From what I could see, all the people that gets frustrated at some point with setuptools, either don't use it, either try to see how it could be changed. But when they try that, they take the same paths others have taken before because there is a lack of info and documentation on setuptools current status and future. And these paths are quite long for the brain before you are able to provide a well-thaught idea or patch.

I have tried to help myself on setuptools, then I have started to blame Phillip saying that he did not have enough time to make setuptools evolve. But that was the wrong approach, because the only problem really, is a lack of visibility in setuptools development. And this is something that can be fixed quite easily I believe. The bug tracker added some months ago helped a lot. A roadmap and one or two page around it and that should be it.

Guido also stated that it was merely impossible to work on the next generation of distutils outside Python core. But in the meantime this means that we need some people from the core to help us in there. And they are really busy (I can understand that) on other matters in Python code. We can write patches for some fixes for sure, and there are hundreds of them to do in distutils.

But to boost it we would need a core developer that gives some love to that package. I mean, for example I still have a patch waiting for reviewing, wich only adds some unit tests that are lacking. There is no code there, just unit tests, and it is pending since 6 months... Guido said that I could try to become a core developer, that this was not impossible, if I started to contribute patches often to other parts of Python as well. But that's quite a challenge.

What frustrated me is that some people like Jim Fulton said they were willing to work on a new distutils from scratch, and in the meantime I understand what Guido says. he pointed us to Joel entry on rewriting from scratch.

My Plans

I don't have the brain to drive a fork of setuptools at this time. I knew it from the beginning, I just wanted to make people react. I think that setuptools can be the laboratory where we can experiment things, and distutils the place where we can apply them at some point, with Phillip help because he has a pretty big overview of all that. It is going to take years, but well, we are young.

To help on this I will:
- try to gather people at the PloneConf to talk about it, maybe even sprint - try to document setuptools and distutils at my level, and write a roadmap for people to know what is going on. Either through a Sphinx site either on Python.org wiki. - work on patches for distutils, and try to point things in setuptools that might be brought into distutils - try to see if I have the brain to understand and work on other parts of Python.

Expert Python Programming book, more details + sample chapter

Tarek Ziadé — Wed, 24 Sep 2008 16:26:00 +0200

It seems that my book is now officialy available, so I should give more details about its content.

Who should get this book ?

I think Shannon described the targetship of my book quite well in the foreword:
*If you're looking to progress from knowing Python to mastering Python, this is the
book for you. In fact, this is exactly the type of book I wish I had had ﬁve years ago.
What took me years to discover by steadfastly attending talks at PyCon and my local
Python users' group is now available in a succinct book form. *

What is means is that this book does not only focus on Python syntax, but also covers how to use Python in a professional environment. Beyond writing a program that works, a good Python developer uses continous integration principles and tries to think about the maintainability of his code. Taking care of choosing good names for instance, will naturally make the code better. Test-Driven Development make the code better too.

The book tries to synthesize these good practices and explain why they are good. So if you are using Python and know how to write a program, and want to push it further, this is a book for you.

On the code

All the code of the book lives here:http://atomisator.ziade.org. So it can evolve.

On my English

I am French. My english is far from being perfect. The Packt team did a great work on improving it, but you will probably feel my french touch in the book. I hope you won't mind.

On the structure

Depending on your needs, you might feel that the ordering of the chapters is not what you were expecting. The current ordering is the one I would use when writing an application, but this is my own vision. If you are a manager you probably have your own way. I worked this out with one concern in mind : every chapter is independant. So feel free to jump to any chapter when you have finished one.

Sample chapter

I am giving away Chapter 10, as an appetizer: chapter 10 of "Expert Python Programming"

This chapter is about documentation. It gives principles and good practices to document your Python projects. It is an invitation to use Sphinx and reStructuredText, together with a set of good practices.

More details on each chapter

Chapter 1, Getting Started. I took me a long time to decide whether I should drop or not this chapter. It describes how to install an environment to work with Python. Since the book is for people that already knows how to use Python a bit, it seemed out of topic.

A few things convinced me to let it in: I have a friend that works with Python under Windows for years, but don't know anything about distutils or setuptools and how to set the proper MinGW environment to be able to install packages without having any trouble. Secondly: I am making some assumptions there for the rest of the book examples to work and I am a setuptools fan. Last but not least: this is a small chapter compared to the size of the book (less than 10%)

Chapter 2, Syntax Best Practices—Below the Class Level, I had a lot of fun there. I am talking about topics like coroutines and contextlib. Pure Python joy !

Chapter 3, Syntax Best Practices—Above the Class Level, This chapter explains amongst other topics why super() is dangerous, how the MRO works, meta-programming, etc. Lots of fun too. I think people will enjoy it a lot.

Chapter 4, Choosing good names, Writing a program that works is a good thing. Writing a program that can evolve and that is comprehensible by other developers is harder. This chapter will give you some clues on choosing good names, beyond the PEP8 guide, and how to organize and build a modular application by working out the API.

Chapter 5, Writing a package. The main ideas here are: write and distribute all your packages the same way, so use templates and distutils.

Chapter 6, Writing an application. Same as chapter 5, but at the application level. This field is very different depending on what frameworks you use and what community you are part of. I tried to come up with something that seems to be the way all developers are tending to take. Maybe I am a bad visonnary, maybe I am wrong. I don't think I am. If you disagree, you will still find interesting stuff in there.I am presenting a micro case study to make things clearerild a modular application by working out the API.

Chapter 7, Working with zc.buildout. Buildout is widely use in Plone and Zope. There are other tools out there of course. But buildout rocks imho. This chapter show how it work and how to use it to work and distributeyour application.

Chapter 8, Managing code. Its starts with a state-of-the-art of version control systems and continuous integration principles. I am explaining how you can work using mercurial and buildbot.

Chapter 9, Managing lifecyle. Same as chapter 8, but focusing on software lifecycles, and why the iterative approach rocks. Also show how trac can be used.

Chapter 10, Documenting your projects, Just read it :)

Chapter 11, Test-Driven Development, This is my TDD manifesto :)

Chapter 12 and 13, Optimisation, Ever wondered how to calculate the complexity of the code ? how to benchmark and optimize ?

Chapter 14, Useful design patterns, Design patterns revisited, because the GoF did not know enough about Python :)

Feedback !

It is really frustrating in some way to write a printed book on a topic like Python. By the time I have finished the book and gave it back to the editor, I was already thinking on some changes and some improvments I could make.

I created a website for the readers : http://atomisator.ziade.org

You can add a ticket there for the v2 !

Distribute, a setuptools fork.

Tarek Ziadé — Wed, 24 Sep 2008 13:10:00 +0200

EDIT: Read this new entry

That's enough !

Setuptools is now a big part of our Python development and release infrastructure.

But we have been struggling with the setuptools development process for months around here. The project is run by one single man, who is really busy on other things as well. I am not blaming him, he's doing a great job. I am just saying that setuptools needs to be more open.

Yesterday, I had, like the week before, some developers complaining about some incompatibility between setuptools and Subversion 1.5.

It is not like it is a hard to fix, and as a matter of fact it is fixed now in the trunk of the project. But not released since quite a long time now. So I have to ask my developers to checkout the trunk on their buildout, and use the ==dev tag elsewhere. I proposed some help to do the release, but my mails were just ignored.

There are also a lot of improvments to do in this tool, so Python gets the distribution tool it deserves.

But having one single person with the commiter rights is not possible on such an important package for the community. It has to be community-driven.

Enough talking, I am launching a setuptools fork, which will be community-driven, and wich will remain 100% compatible with setuptools at this point.

You can react, of follow the reaction on the distutils-SIG I started here : http://mail.python.org/pipermail/distutils-sig/2008-September/010031.html

Maybe my attempt to make such a tool belong to the Python community will fail, I don't know. Maybe people will not like seeing someone forking like that and will continue to sleep and to unsubsribe themselves from the distutils-SIG.

But at least I am trying something because Python is suffering from a lack of a good, robust distribution tool at this point and there are a lot of people that could contribute if the development process is more open and the maintainer(s) more available and reactive.

I have contributed a bit in distutils in the past year, but the Python core development team has a lack of interest in this part of Python at this time. Thanks to some of the core team members, I could have some patch applied in the trunk, like the multiple servers pypirc. But we won't be able to create what I would call 'distutils 2' in Python itself at this point. And setuptools seems to be the best candidate for a distutils replacement in the future.

As long as the community can work on it of course ! We are the community, we are distributing large application to our customers.

So that's what "Distribute" is about.

http://mail.python.org/pipermail/distutils-sig/2008-September/010031.html

Annoucing collective.eggproxy, the smart PyPI mirror

Tarek Ziadé — Wed, 24 Sep 2008 09:15:00 +0200

I just wanted to announce the release of collective.eggproxy (http://pypi.python.org/pypi/collective.eggproxy/0.2.0)

collective.eggproxy is a smart mirror for PyPI, thaught and coded by my colleague Bertrand Mathieu.

It will collect packages on PyPI only when a program like easy_install or zc.buildout asks for it. In other words, unlike some mirrors that act like rsync and get the whole PyPI base (more than 5 gigas) collective.eggproxy will only get what you need.

At first run collective.eggproxy downloads pypi index and builds a page of links. When a software asks for a specific package, version, etc. collective.eggproxy downloads it if needed and stores it locally.

Want to give a try ? try it in two lines with easy_install:
easy_install collective.eggproxy

eggproxy_run

That's it !

The WSGI era is here

Tarek Ziadé — Fri, 19 Sep 2008 11:30:00 +0200

This is probably obvious for the people that uses Repoze or Pylons, or early adopters in the Plone world, but from a Plone or a Zope developer perspective, you could live without it until now.

Now WSGI is everywhere.

I remember when Martijn Faassen brought the idea in 2006, of hooking Grok and Zope 3 into the WSGI. Maybe someone else talked about it before but that was the first time I could picture what WSGI could bring.

Now with the work done by people like:
- The repoze team, that made it easier to run a Plone-based application in WSGI - The Paste Script / Paste Deploy team, that provided a simple way to describe a WSGI chain

And major WSGI middlewares like :
- repoze.who which allows you to deal with authentication separately - Deliverance, which let you theme any application and let this application focus on delivering a content - Things like Beaker, which let you use memcached for instance, to store session data and cache arbitrary things

From a CTO point of view, a WSGI environment brings me the ability to think about a web application and build it without having to stick into one framework and try to bend all technologies inside it.

For instance:
- I can write a Plone application and use Beaker to deal with sessions, without having to wrap Memcached into a custom plone package. - I can ask a graphic designer to work on a CSS and a layout without having to do it into Plone. It's not that Plone design tools are bad, but the learning curve of writing a rule file in Deliverance and apply it to any piece of application makes the designer more productive than becoming a specialist of one skinning tool.

If my customer use moinmoin as a Wiki, I can put it into my Plone site transparently by defining a composite section in my Paste configuraton file.
...

You could do all the mentioned thing without WSGI, just by importing the packages and/or dealing with proxies at Apache level. But that is not the point.

The point is that WSGI brought the idea of making all web frameworks and libraries interact together to build one web application.

It is not the silver bullet of course, but my gut feeling is that this will create some kind of reunification in Python Web development communities: people are starting to look at a wider range of package, beyond the framework they use everyday.

Paris Bobun Sprint

Tarek Ziadé — Tue, 16 Sep 2008 09:30:00 +0200

I don't have time to talk in detail about it at this time, but I wanted to say that the Bobun Paris Sprint was a great event.

Together with the sprint we have done a few month before here at AlterWay, this a great momentum for the French Plone community.

Those events will probably be recurrent from now on (the one we organized at AlterWay will be organize every year for sure, and I think the Bobun team wants to do it again as well).

Maybe we will be able at some point in a few years to organize the PloneConf in Paris ?

That would be a blast :D

zc.buildout recipe to build your Sphinx doc

Tarek Ziadé — Thu, 11 Sep 2008 11:41:00 +0200

Sphinx is now used by quite a few communities : Python, Django, Pylons, Grok (not sure about the current status), ...

No wonder, it's a blast.

We are now starting to use it to produce customer documentation for our buildout-based applications. Basically, a Sphinx structure is created in the buildout using sphinx-quickstart, and a few tweaks are made so the HTML and PDF outputs have a custom look.

Managing the documentation like the code makes life easier. This is one of the basic rule of agile documentation : separate the content from the layout, so you can provide documentation in any shape (html, pdf) with a single source.

To make things easier, I have released collective.recipe.sphinxbuilder.

This recipe:
- creates for you a Sphinx-based documentation in your buildout - creates a single script in the bin folder to build the documentation with one command - provides an extensive set of options to drive Sphinx from buildout

Adding it in your buildout is as simple as :
[buildout]

parts =

  sphinx

[sphinx]

recipe = collective.recipe.sphinxbuilder

I have also customized the look and feel of the output so it uses the Plone logo and a custom css. This is configurable from the buildout configuration file of course. By the way, if someone from the Plone community wants to improve the CSS, please dot it ! (I am not good at this :) )

[gallery]

Notice that if you use LaTex or PDF rendering, you will need to install pdflatex. Furthermore, the recipe script will not work under Windows unless you install a Linux-like environment, since it uses the Makefile provided by Sphinx. I guess MSYS+MinGW should make it work, but I didn't try.

Yet another Planet

Tarek Ziadé — Thu, 04 Sep 2008 21:20:00 +0200

Atomisator is a framework so it is hard to get an idea of its features until a real application uses it.

That is why I wrote a small application in Pylons called Yap (Yet Another Planet), that is basically displaying the XML file produced by an Atomisator instance. Since Atomisator does all the work, the Pylons apps is really small (one or two controllers, that's it).

My first use case was to produce a nice, smart Planet for our user group Afpy.

Here's a first draft: http://ziade.org/afpy/

You can play with 'j', 'k' and arrows to open and close posts, but I am still working on this, so it will also scrolling the window when you are on a post.

Anyways, it grabs various French sources for Python and uses these plugins from Atomisator:
- filter : reddit - filter : delicious - filter : doublons - enhancer : related - enhancer : digg

The result is basically following reddit and delicious links to display an extract of the page linked, and display digg comments as well. Duplicate are removed as well. A list of related entry are also added to each entry.

It is based on this configuration file, Atomisator uses to generate an XML file for Yap in a cron:
atomisator

sources =

    rss     http://del.icio.us/rss/tag/python+fr    Delicious

    rss     http://www.afpy.org/search_rss?portal_type=AFPYNews&sort_on=Date&sort_order=reverse&review_state=published Afpy News

    rss     http://feeds.feedburner.com/Baderlog/python Bader

    rss     http://www.biologeek.com/journal/rss.php?cat=Python Biologeek

    rss     http://www.gawel.org/weblog/rss/python/afpy/zope/zope3/rss.xml  Gawel

    rss     http://www.haypocalc.com/blog/rss.php?cat=Python    Haypo

    rss     http://jehaisleprintemps.net/blog/rss/  No

    rss     http://programmation-python.org/sections/blog/exportrss Tarek

    rss     http://api.blogmarks.net/rss/tag/python,fr  Blogmarks

# put here the database location

database = sqlite:///afpy.db

# this is the file that will be generated

file = /home/tarek/www/packages/Yap/trunk/yap/public/afpy.xml

# infos that will appear in the generated feed.

title = Planet Python Francophone

description = Le planet de l'Association Python Francophone, et des gens heureux.

link =  http://www.afpy.org/planet/

filters =

    reddit

    delicious

    doublons

enhancers =

    related

    digg

What's Next ?

Since now, there were no attempt to try to automatically classify entries. The next plugin I am working on will provide a Naive Bayesian filter to classify entries, together with a way to train it through the Yap web interface. basically a 'keep'/'ditch' button.

I will also set an english Planet Python to see how things go with more sources.

Thank you, Plone community !

Tarek Ziadé — Fri, 29 Aug 2008 17:25:00 +0200

Wow.

I have been really involved in Plone for about one year now. This community just rocks. I met incredible guys, and did tons of work and experiments with them.

I have asked the community for help to go to the Plone Conf with a fund raising. I raised the funds in 3 days... I am astonished, this is a marvelous gift from you guys. I alsmot cried when I saw all those people donating.

I am so lucky, and so proud to be part of the Plone community and to know all of you. I will work hard to produce the best tutorial I can for the Plone Conf, and continue of course my work in the community tasks I am currently involved in.

I would like to say a big, warm thank to (the order is chronological, from the latest donation to the first one):
- Andreas Jung - Veda Williams - David Glick - Jeff Kowalczyk - Youenn Boussard - Christian Klinger - Jesse Snyder - Alec Mitchell - John Habermann - Maurits van Rees - Jean-François Roche - Martin Aspeli - Alain Meurant - Aleksandr Vladimirskiy - Jon Stahl - Alexander Limi - Stephen McMahon

And a very special thanks to Calvin Hendryx-Parker.

This conference will be a blast for sure, if you can go there, do it !

Now I better find a good idea for those T-Shirts I have promised ! ... :)

Plone development: Nose or zope.testing ?

Tarek Ziadé — Thu, 28 Aug 2008 15:25:00 +0200

Python needs a better testing tool, hopefully this will happen some days.

Until then we, Plone developers use zope.testing for the best and the worst.
- the best because zope.testing provides layers and the Zope/Plone stack provides really nice things to work with them. - the worst because sometimes it just gives me some headaches to figure out how to set my test fixtures correctly. - the worst because I cannot easy-install zope.testing and just call it from the prompt to run some tests (this should get better sometimes see here)

That said, when you work with a zc.buildout based environment, it is easy to use zope.testing, thanks to zc.recipe.testrunner.

But since the eggification of Zope, we write more and more code that would benefit from a lighter test framework. Nose has this lighter approach, I love it to write simple tests without the Java-like heavy UnitTest framework. I can just write:
def test_something():

   assert 1 == 1

And call nosetests. Test fixtures are easy to set at all levels as well, using the with_setup decorator.

I tried to write some tests that could be launchable from both frameworks, by making some bridges, but I came up to the conclusion that a package should fully use either zope.testing, either Nose.

So now, I use two test scripts in my zc.buildout environments, and decide depending on the package. I have written another recipe for buildout to bind Nose: pbp.recipe.noserunner.

From there, I create two sections in my buildout.cfg file:
[buildout]

parts =

    ...

    nose

    ztest

[nose]

recipe = pbp.recipe.noserunner

eggs =

    egg1

    egg2

[ztest]

recipe = zc.recipe.testrunner

eggs =

    egg3

    egg4

These will generate two test scripts
- nose to run nosetests with egg1 and egg2 in sys.path - ztest to run zope.testing with egg3 and egg4 in sys.path

Here's my grid of choices when I am coding:
- zc.buildout recipes : zope.testing (doctests using zc.buildout testing framework) - Plone packages: zope.testing - Python packages and 'everything' else: Nose

Atomisator, visiting links

Tarek Ziadé — Wed, 27 Aug 2008 19:50:00 +0200

I am writing a plugin for Atomisator that detects when a post is a Reddit or a Delicious entry, and adds a sample from the page it links to.

On a Reddit feed for example, you will basically get a meaningful title, and summary that will look like this:
[link] [comments]

This is not really useful in a feed...

So basically, the plugin I am writing is detecting this kind of entries and is grabbing a sample out of the linked page, so the entry is transformed like this:
Extract from the link:

   ... blablablba

   blablabla...

[link] [comments]

The extracted text is a pure text, extracted using BeautifulSoup and the SGML parser from the standard library.

This is quite useful as long as the begining of the web page is meaningful, which is rarely the case... Most of the time the web page starts with a gibberish text, like a menu bar content for instance.

So I am trying to detect what is the "best" part of the web page pointed by the link.

To do so, I am using the title of the entry, which is suppose to make sense. Since there are good chances the text will contain the words used in the title, I am looking for them into the page.

I have tried several combinations, even by trying to find the smallest sample where I get the maximum number of words from the title by doing some cartesian products. But this was too slow.

Instead, I am trying to detect the real beginning of the post by trying some common patterns : most of the time the body of the post is a div tag with a class attribute like body, content, article-content, etc..

I am running it now over the Python feed at Reddit, and the results start to look nice so far (==you can understand what the page talks about most of the time). See here:http://www.ziade.org/atomisator/filtered.xml

Now I will try to run it over a fair amount of entries and with various sources to try to tune up the extractor.

This code will also be a useful base to visit links of any kind of entry, but it needs a lot of cleanup: I have spotted some quadratic complexity parts I need to get ridd of.

Try it yourself :
1. make sure you have SQLite installed 2. install atomisator with easy_install atomisator 3. create your atomisator.cfg file with the content below 4. then run it by calling 'atomisator' in the folder atomisator.cfg lives

atomisator.cfg content :
atomisator

sources =

    rss http://www.reddit.com/r/Python.rss reddit

database = sqlite:///atomisator.db

file = atomisator.xml

title = Meta feed

description = Automatic feed created by Atomisator.

link =

filters =

    reddit

Help me go to the Plone Conference !

Tarek Ziadé — Tue, 26 Aug 2008 21:52:00 +0200

I have been asked to be a "champion" last winter at the Plone summit.

My task was :

Improve release procedures for add-ons on plone.org: document a release process, and create release tools for packaging and uploading products from the command line.

Since then, the major steps that I have done for it were :
- change Python's .pypirc format so it can be used to interact with PyPI and Plone. DONE - provide a package to make the new .pypirc format available under Python 2.4 and 2.5. DONE - make PloneSoftwareCenter PyPI-compatible. DONE

The steps left are:
- continue the help on Plone.org Plone 3 migration, so it becomes reality - document the release process, based on PyPI, Plone.org and collective.dist

I am planning to go at the Plone Conference and to give a tutorial on all this work to deliver to the community the releasing process documentation. So these two remaining steps have to happen before the conference. I am also planning to sprint with people there.

My company contributes a lot to the Plone community. For instance we have organized the Paris Plone Sprint and we try to contribute as often as we can into the collective. But it has its limits and my airfare cannot be covered at this time.

This is the first time I am asking something to the community : please, help me go to the Plone conference !

I have created a page at ChipIn, and I am trying to raise the money to go to D.C.

What you get if you help me:
- You will contribute to plone.org and release procedures enhancements - Since I can create t-shirts, I will create a unique Plone'08 DC logo T-Shirt that I will ship to the 5 biggest donators.

EDIT : It seems that Paypal won't take your browser language settings into account. If you can't read french you can try this hack I have found to switch to english :

*when the paypal page is loaded add "locale.x=en_US&" into the url after "websrc?=" *

so it looks like this:
https://www.paypal.com/fr/cgi-bin/webscr?locale.x=en_US&cmd=...

Reload that link and it should switch in english.

EDIT 2: don't worry if Paypal tells you Amina El Kamel received the payment. This is my girlfriend Paypal account ;) .

Click on the cat :

Visual profiling with Nose and gprof2dot

Tarek Ziadé — Mon, 25 Aug 2008 09:19:00 +0200

Nose comes with a handy option to generate profiling stats.

To profile your code, create a test dedicated to this purpose and run it with the right options:
$ nosetests --with-profile --profile-stats-file stats.pf test_performance

This will run the tests that corresponds to the test_performance name and generate a stats.pf file.

Nose uses hotshot, so if you want to generate a file that can be read directly by the pstats module and all the statistics tools out there, you need to convert it using the hotshot.stats module.

From there, there is plenty of tools that can transform such a file into a visual graph. Most of the time, they use Graphviz to render a graph, by generating a file dot can read. This software is most of the time easy to install through a binary distribution on your system. If you need to compile it... good luck.. ;)

Anyway, from there, I use gprof2dot, which renders a nice graph with meaningful colors.

From the author:

The color of the nodes and edges varies according to the total time % value. In the default temperature-like color-map, functions where most time is spent (hot-spots) are marked as saturated red, and functions where little time is spent are marked as dark blue.

If you want to use it, I have created some console scripts for conveniency, you can install using [easy_install][]:
$ easy_install pbp.scripts

It creates a gprof2dot script you can use, following the author documentation, but also a hotshot2dot script that will convert automatically a statistics file and pass it to gprof2dot:
$ hotshot2dot /path/to/my/hotshot/file

This will print in the output a dot file, you can send to the dot program, using a pipe:
~~~~ {style="text-align:left;"} $ hotshot2dot /path/to/my/hotshot/file | dot -Tpng -o output.png ~~~~

You will get the visual result in output.png.

Atomisator, a framework to build custom RSS feeds

Tarek Ziadé — Wed, 20 Aug 2008 16:56:00 +0200

We are all overwhelmed by the amount of data in our feed readers. While this problem is unavoidable if you keep on adding new feeds in it, they could be automatically filtered and categorized to reduce the flow of data.

I wanted for a long time to try out some custom filters over my feeds to find for example related entries, by trying to understand the meaning of the posts, using tools like NLTK.

So I needed a playground for this, where I could play with feeds.

I think the closest tool for this is to use Yahoo Pipes but as far as I know, the only way to create custom filters is to run a web service and call it from Yahoo Pipes.

Anyways, I started to code a framework (at first it was an example for my latest book) that looks a lot like Yahoo Pipes in its principles. I don't have any User Interface at this time of course, but a simple plugin-based tool that will let me combine my code snippets with feeds.

It is called Atomisator (see http://atomisator.ziade.org).

[caption id="" align="alignnone" width="539" caption="The big picture"][]The big picture[/caption]

The process is quite simple:
1. Readers are plugins that know how to read a source and provide entries out of it. 2. Filters are plugins that know how to remove unwanted entries, or enhance them (change their title, summary, etc.). They can be combined. 3. the entries are then pushed in a database. This is useful to avoid doublons, and to keep track of past entries. 4. to create the feed, the entries are read from the database 5. Enhancers are plugins that will add to entries extra info. Typically info that can't be stored, like Digg comments if the entry is detected on Digg, or Google related searches, and so on 6. The feed is then generated.

Right now I am focusing on making it fast, which is not simple because the plugins can play with all entries in the database.

It is in early stage and undertested, but it kinda works. I pushed it at PyPI to see of it meets interest. If it does, I will document the process of writing plugins.

Make sure you have SQlite installed, and give it a try :
$ easy_install atomisator.main

$ atomisator -c atomisator.cfg

$ atomisator

You will have an atomisator.xml feed created. You can add other feeds in atomisator.cfg as well and try them.

Now with this environment, I can start to try out custom algorithms over my feeds.

I've been told the name doesn't sound right in Ehglish, but it does in French so I keep it ;)

iw.eggproxy, a smart PyPI mirror

Tarek Ziadé — Tue, 12 Aug 2008 09:04:00 +0200

Mirroring PyPI becomes a recurrent need for Zope development, because zc.buildoutmakes a lot of package downloads to build one application.

This is useful when you are working in an intranet with limited web access or when you want to speed up download times. It also makes things safer: if PyPI is down and if developers computers don't have caches, having a mirror will save your day.

While PyPI has proven its robustness (it is 100% up for months now as far as I can see), having mirrors makes a lot of sense.

We have created a small mirror application here at Ingeniweb, that we use for our buildouts needs. This work was thaught and created by my colleague Bertrand Mathieu.

It is a smart proxy that will download packages at PyPI everytime they have been asked by a buildout or an easy_install client. When the package is downloaded, it is kept in the proxy side for any new requests. This means that after a while, the proxy has its own collection of packages that corresponds to the real needs and will not query PyPI anymore.

This approach avoids having to download and synchronize PyPI with crons, which is a heavy process since PyPI weight several gigas. The caveat of course, is that it won't be able to get a new package if PyPI is down.

Take a look ! http://pypi.python.org/pypi/iw.eggproxy

By the way there is an interesting sprint coming up on all these topics, in Germany :

http://www.openplans.org/projects/black-forest-sprint/project-home

A new Python book : "Expert Python Programming"

Tarek Ziadé — Fri, 08 Aug 2008 14:57:00 +0200

The Packt team added a page about my new book, which will be out sometimes in September, so I guess it is the right time for me to announce it here !

So here comes "Expert Python Programming", where I explain how we work with Python every day to create software.

This is my first book in English, and writing in another language was quite challenging ;)

Anyway, this book is intended for developers that already have a background in Python and covers only advanced topics (see the editor details). But as I said, it explains how we develop our applications in Python so topics like continuous integration, documentation, testing, releasing, refactoring, etc. are covered.
Managers will also have a good overview of how a Python project can be run and managed, using modern tools like Distributed Version Control Systems (Mercurial for instance) or Buildbot.

Even if I am working on Zope and Plone these days, I have focused on writing a book that is only about Python, to make it useful to any developer. The fact that my friend Shannon did the technical reviewing helped on this : he doesn't use Zope too much so he was close to the target readership.

I will take more time when it comes out to announce it in the various mailing lists, and to get into greater details about the content that will be available online besides the book (some Python packages, etc)

I can't wait to see it out and hold it :D

OSCON report #1: the city of Portland

Tarek Ziadé — Tue, 29 Jul 2008 05:20:00 +0200

Covering OSCON when you are a speaker is quite hard. Until my talk was done, I was more in a mood of reviewing my slides and not really thinking about other talks or blogging.

[caption id="attachment_136" align="alignnone" width="225" caption="Convention center from the Bridge (the two twin towers)"][][][/caption]

I will try to write a few reports now that everything is over. This first report is about Portland, the city and is not technical at all.

This is the first non-technical entry in my blog, but let's this it for once. So if you are looking for technical content, skip this entry :)

Portland is an amazing place. The weather was nice throughout the whole week: sunny and not too hot. Some evenings were a bit chilly but that felt nice to me. The convention center where OSCON took place was located on the other side of the Willamette river, in front of the city center. It is a bit far from the center, but finding places were you can put so many people must be hard I guess.

The top six things I have noticed about Portland:

Local beers are great. You can find many places where they brew their own beers, that are comparable to Belgian beers in quality and taste, like the Lucky Lab (Where Jon, a member of the local Plone user group took us, thanks!). Some place have samples so you can taste several kinds of beers. I also went to the Oregon Brewers Festival right after OSCON was over, where you could taste beers from many places.

[caption id="attachment_137" align="alignnone" width="225" caption="Oregon Brewers Festival"][][][/caption]

Parking a car is a pain. I think the best way to travel in Portland is a mix of Tram, Bus and Bike. You can put your bike on the nose of the buses or inside the trams. Smart thing. This would be useful in Paris. We ended up parking our car far from the center and used the tram (called Max, and free in the center). Portland should set up a bike rental system like in Paris (Velib'), I am sure this would rock.

Organics: people here seem to be really concerned about sustainability and organics. I've been told there are a lot of organics farm around Portland. Supermarkets have a nice amount of organics stuff as well. This is nice.

Starbucks Coffee owns the streets: this is scary. there's a Starbucks almost on each block here. Anyway, local coffee shops have better coffees and they provide nice places to chill out, read a book or talk with people. I think this is where you can feel the real Portland way of living.

The food is good : I have to admit, when we, french people, travel, we feel a bit superior to some countries on food matters. We tend to show off about it :). But Portland has great places to eat. If you like donuts, the place you must go to is called Voodoo Doughnut. They are creative !

[caption id="attachment_138" align="alignnone" width="300" caption="Voodoo Doughnut"][][][/caption]

Big open spaces: I went to Mount Hood where people are skying in... July. I wanted to visit this place because it is where Stanley Kubrick did the outside shots of the famous motel in The Shining. I also had a walk at the Hood river sandbar, where people do kite surfing. One tip: don't walk there with shorts on, the wind is so strong that you get slapped by the sand. So if you like big open spaces and sports, Portland is the place to be: all those place are one or two hours drive from the city center.

[caption id="attachment_139" align="alignnone" width="300" caption="Mt Hood"][][][/caption]

End of the aparté ! The next entry will focus on OSCON

[]: http://tarekziade.files.wordpress.com/2008/07/img_3591.jpg []: http://tarekziade.files.wordpress.com/2008/07/img_3585.jpg []: http://tarekziade.files.wordpress.com/2008/07/img_3614.jpg []: http://tarekziade.files.wordpress.com/2008/07/img_3402.jpg

My OSCON slides online (zc.buildout)

Tarek Ziadé — Fri, 25 Jul 2008 00:46:00 +0200

After a few attempts to make my screencast look nice under Google Video, I decided I would just upload the original ones (.mov and .m4v files) and a PDF export of my presentation.

So everything is available here: http://ziade.org/oscon

Slides: http://ziade.org/oscon/oscon.pdf
- Screencast #1, playing with distutils: http://ziade.org/oscon/demo1.m4v - Screencast #2, playing with setuptools: http://ziade.org/oscon/demo2.m4v - Screencast #3, playing with zc.buildout: http://ziade.org/oscon/demo3.mov - Screencast #4, creating a recipe: http://ziade.org/oscon/demo4.mov - Screencast #5, creating a plone buildout and adding a develop package in it: http://ziade.org/oscon/demo5.mov - Screencast #6, collective.buildbot demo (no coding, just showing buildbot.ingeniweb.com): http://ziade.org/oscon/demo6.mov

The .mov files are streamed automatically in your browser I believe, not the m4v ones.

Thanks to Jim Fulton for the quick feedback.

Going to OSCON

Tarek Ziadé — Fri, 18 Jul 2008 12:00:00 +0200

I am leaving tomorrow, heading to Portland, OR, to OSCON. My talk on zc.buildout and Plone will be thurdsay the 24th, and I'll be there the whole week.

If you are going there and want to meet leave me a note, I am looking forward to meet other geeks there. :D

shutil.copytree small improvement

Tarek Ziadé — Tue, 08 Jul 2008 08:11:00 +0200

When you have to work with directories and files, there's a very common pattern: copying a directory into another, but filtering out a few files and directories.

For instance, if you want to copy a directory that contains source code, you will probably remove .pyc files and .svn directories if you work with Subversion. In that case, shutil.copytree cannot be used, so os.walk is the usual way to go (rough example):
import os

from os.path import join, splitext, split, exists

from shutil import copyfile

def copy_directory(source, target):

    if not os.path.exists(target):

        os.mkdir(target)

    for root, dirs, files in os.walk(source):

        if '.svn' in dirs:

            dirs.remove('.svn')  # don't visit .svn directories

        for file in files:

            if splitext(file)[-1] in ('.pyc', '.pyo', '.fs'):

                continue

            from_ = join(root, file)

            to_ = from_.replace(source, target, 1)

            to_directory = split(to_)[0]

            if not exists(to_directory):

                os.makedirs(to_directory)

            copyfile(from_, to_)

This is a lot of boiler-plate code, so I usually create a small function that accepts more arguments to filter out files and directory. But Python should provide this pattern in the standard library.

I have proposed a patch for shutil.copytree, to integrate filtering capability. It has been reviewed and commited in the trunk this week, so we will have it in Python 2.6. Now copytree comes with an ignore argument that has to be a callable. If given it will be called on each visited directory to decide what is copied and what is not.

There's a default callable in shutil called ignore_patterns, that can be used to filter out files with glob-style patterns. I have added this example to Python doc:
from shutil import copytree, ignore_patterns

copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*'))

Pretty straight forward ! :D
More info and examples here : [http://docs.python.org/dev/library/shutil.html#shutil.copytree][]

plone.org migration

Tarek Ziadé — Tue, 01 Jul 2008 22:03:00 +0200

Plone.org migration to Plone 3 is taking a bit longer than expected, but it should turn into reality soon.

There will be many improvements on the set of packages the website uses (I am thinking in particular about Maurits's work on POI that will speed up the trackers, but it is just an example), and blobs should be used for the products section (more than 700 projects are registered there).

I worked last week-end on the products section, by finishing collective.psc.mirroring, which will copy all packages that are uploaded at plone.org into a file system directory. This directory will be published directly by Apache so the website will become a new package location for zc.buildout (find-links section) and easy_install calls without invoking the Plone instance.

Now I am focusing on PloneSoftwareCenter (PSC) migration. It is a pretty interesting topic: for every project located in the products folder with releases, I am going to extract its "distutils ids". These ids are the name set in the setup.py file for each release.

I will then look at PyPI through XML-RPC if the package is also released, using the id. In that case, and if the author email is the same on both side, I will validate that the project on plone.org "owns" the given distutils id. From there PSC will act like PyPI and will reject uploads of packages if the user is not the owner of the project that owns the package id.

Of course there will be errors and some people might feel like their package has been hijacked if they cannot upload their packages. But this should be minor and should be OK after a while. A mail will soon be send to the community to ask people to check that they are synced between PyPI and plone.org.

I am really excited about this work because plone.org will then be compatible with distutils register and upload commands, which means that people will be able to update plone.org products section like they do with PyPI : through a single commande line.

Hey Sidnei, what you thought of several years ago is about to turn true. ;)

Edit: My apologies goes to Maurits, who did the work on POI, not Reinout, his brother ;)

Python 2.4: tarfile module is buggy, patch it !

Tarek Ziadé — Thu, 19 Jun 2008 00:44:00 +0200

I have encoutered really bad bugs in some Python 2.4 applications (Zope based) using the tarfile module. For instance, the TarFile.getmembers method that returns the files a tar file contains will just fail to return all files ...

Hopefully Zope will work under Python 2.5 sometimes. This is a work in progress in GSOC if I recall it correclty, but I am not sure of the current state.

Until then I just stick Python 2.5 module in my packages. It seems to work fine (tests pass :D) after a few changes as it is isolated from the rest of Python.

If you want to do it as well, there are only two changes to make it work under Python 2.4, to introduce 3 new constants in os module and to get rid of a syntax that does not work under 2.4:
$ diff /opt/local/lib/python2.5/tarfile.py tarfile.py

46a47,49

>

> os.SEEK_SET, os.SEEK_CUR, os.SEEK_END = range(3)

>

1070c1073,1076

<         self.name = os.path.abspath(name) if name else None

---

>         if name:

>             self.name = os.path.abspath(name)

>         else:

>             self.name = None

collective.buildbot mailing list

Tarek Ziadé — Fri, 13 Jun 2008 07:44:00 +0200

Interested in collective.buildbot usage or development ?

Join us in the dedicated google group : http://groups.google.com/group/collectivebuildbo

Reminders:
- collective.buildbot is a set of zc.buildout recipes and support for declarative configuration for Buildbot. - BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes, to set up a continuous integration system.

zc.buildout on-going work

Tarek Ziadé — Tue, 10 Jun 2008 09:20:00 +0200

Just a quick post about the work going on in zc.buildout.

Work published lately:
- Malthe and Mustapha have added a nice option to be able to add or substract values in variables that are inherited from an extended cfg file (already available, see the doc here) - I have added an allow-hosts option, that behaves like easy_install one, which can be used to restrict some accesses (see here) - Sidnei fixed some annoying bugs (missing quotes in some process calls)

On-going work:
- I am working on a timeout config option, to be able to set the socket timeout (see my previous post on this). Andreas added a command-line option a bit ago as well, but we need to refactor it a bit to have it as a config file option. - I work on a python API so the buildout can be driven from the code. This useful for instance to create tests that are not using a os.popen or os.system call to build a buildout: a separated process is hard to debug. - We are having some thaughts on having a multiple-index enabled buildout. Still brainstorming.

Eggs: releasing procedure and continuous integration

Tarek Ziadé — Sun, 08 Jun 2008 11:27:00 +0200

Disclaimer: this post is Subversion-centric

When releasing a setuptools based-package to the world, developers will eventually tag it as a stable version, then upload it to PyPI or to their own website. So typically they create a branch out of the trunk, fix the version, then create a tag for the release.

Now for continuous development releases, where people get for example a daily snapshot from the trunk, having a dev suffix to the egg version makes it possible to distinguish it from a stable release, so 'my.package-2.4dev' will not superseed 'my.package-2.4'. So a given package will need a dev prefix in its trunk, that gets removed in the branch where the next release occurs.

From there, easy_install will be able to distinguish them.

The question is: how should we do this ?

The Zope 3 way

The Zope 3 egg collection has a simple way to manage it, describe here: [http://svn.zope.org/*checkout*/Sandbox/philikon/foundation/releasing-software.txt][]. The packages have a dev tag suffix in the version metadata, that is removed in the subversion branch used to tag and release

This is simple and straight forward. Although, releasing the trunk has to be done manually. The problem is that any release of the trunk will have the same name for a given version (my.package-2.4dev) , and in some case people won't upgrade their environment because the archive keeps the same name.

The setuptools way

Setuptools comes with a continuous integration feature that allows people to push the dev tag in setup.cfg. Then it adds it automatically to the archive name when it is generated.

see: [http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion][]

Building a release will then consist of using the same process than Zope 3 one, except that the tag is removed from the setup.cfg file this time.

Now if you try to release the trunk using disutils sdist or bdist_egg command you will automatically get a dev suffix and the Subversion revision number sticked to it. This means that each new revision can generate a new version that will have a distinct name:
- my.package-2.4dev-r1245 - my.package-2.4dev-r1246 - ...

easy_install will grab the latest trunk revision when "my.package-2.4dev" is required, and handle upgrades the right way. This is better that a manual dev tag because when you re-release a new version of the trunk, it will superseeds and therefore upgrades previous revisions.

Another nice feature is to be able to provide to easy_install a subversion link directly, as long as you append the egg full name to the link:
easy_install http://my.svn/my.package/trunk#egg=my.package-dev

collective.releaser

The problem with setup.cfg is that if you forget to remove the dev tag from it when releasing a stable version (you bad boy), you will get the dev-r4565 suffix in the egg name.

collective.releaser takes care of this with a new setuptools command called release, by creating a release branch and removing the dev tag automatically. It upgrades the CHANGES.txt file and version.txt for Plone products.

It also registers and uploads the package to any PyPI-like server. To decide where the package should be sent, it looks at the release-packages variable into the .pypirc, to see if the package name matches it. To release all packages to PyPI a default file would be:
[distutils]

index-servers =

  pypi

[pypi]

username:tarek

password:secret

# regular expression-based variable

release-packages =

  .*

From there, making a release is as simple as:
$ python2.4 setup.py release

running release

This package is version 0.1.1

Do you want to run tests before releasing ? [y/N]: n

Do you want to create the release ? If no, you will just be able to deploy again the current release [y/N]: y

Enter a version [0.1.1]: 0.1.2

Commiting changes...

Creating branches...

...

Running "mregister sdist bdist_egg mupload -r pypi"

running mregister

Using PyPI login from /home/tarek/.pypirc

Registering iw.resourcetraverser to http://pypi.python.org/pypi

...

running mupload

Using PyPI login from /home/tarek/.pypirc

Submitting dist/iw.resourcetraverser-0.1.2.tar.gz to http://pypi.python.org/pypi

Submitting dist/iw.resourcetraverser-0.1.2-py2.4.egg to http://pypi.python.org/pypi

...

0.1.2 released

As described on its PyPI page, it has a plugin system to perform extra steps when a release is made. For instance, it is provided with a mail hook to be able to send mails everytime a release is made.

The tool is pretty new and needs to be smoothed up. For instance, we need to add a few extra controls like making sure long_description is reST-valid. But It works, and is being used by a few fellows in the Plone community already :).

Last, for continuous releases from the trunk, you can set up an alias:

$ python setup.py alias devrelease mregister sdist mupload

You will then be able to upload trunk releases with a single call:
$ python setup.py devrelease

The idea is to provide it as the standard tool to release add-on Products when Plone.org will be upgraded and able to interact with distutils commands. I will promote it through the work I am doing for the PSPS task I am championning (Improve release procedures for add-on products), but it has to live a bit.

Help and opinions welcome !

http://svn.zope.org/*checkout*/Sandbox/philikon/foundation/releasing-software.txt
http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion

DIY PyCon FR shirts

Tarek Ziadé — Wed, 21 May 2008 23:28:00 +0200

We made the shirts at home for PyCon FR and that was a LOT of fun, even if we had to work hard to make sure all shirt were ready by the time the event started. We used a silk-screen technique and the output looks really professional. The ink used makes the shirt design last longer than shirts that uses transfer. Although the number of colors is limited.

Have a look at the pictures : http://www.flickr.com/photos/82007723@N00/sets/72157605181699082/

Guess what: some people ordered us some t-shirts for some other events, so some more fun ahead ! :D

Google AppEngine sprint at Pycon FR

Tarek Ziadé — Wed, 14 May 2008 21:14:00 +0200

Late (but exciting) news: a sprint room will be held at Pycon FR, by the Logilab team, that published a GPL framework on the top of Google AppEngine, called Logilab Appengine eXtension (lax).

Participant will be able to learn how to play with AppEngine technologies.

All infos (in french): http://fr.pycon.org/programme/sprint-appengine

The new .pypirc format in Python / distutils

Tarek Ziadé — Mon, 12 May 2008 12:45:00 +0200

Thanks to the sprinters in Washington D.C., (Andrew who worked on merging it, Alex, etc.) and people that helped in the discussion (Fred, Martin), my patch to Python for multiple servers handling in .pypirc file has been integrated in 2.6 trunk.

This patch will allow using distutils register and upload commands with several servers, using the -r option and storing for each one of them the appropriate username/password in the configuration file. See http://wiki.python.org/moin/EnhancedPyPI for more details.

The collective.dist package provides the same feature for Python 2.4 and 2.5, through new commands: mregister and mupload.

The new format is also pretty convenient to store several profiles per servers. For instance, if you have several accounts on PyPI, one for your company (acme) and one for your own usage, you can define them like this:
[distutils]

index-servers =

    pypi

    acme

[pypi]

username:user

password:password

[acme]

username:acme_user

password:password

Then use them depending on the package you work in:
$ python setup.py register -r acme

$ python setup.py register    # default, which is pypi

When plone.org will go Plone 3 (this is happening soon), the pypirc file can be extended with this new target:
[distutils]

index-servers =

    pypi

    acme

    plone

[pypi]

username:user

password:password

[acme]

username:acme_user

password:password

[plone]

repository:http://plone.org/products

username:plone_user

password:password

From there, setuptools aliases can be used in each package to simplify things:
$ python setup.py alias plone_org register -r plone sdist bdist_egg upload -r plone

Which will allow you to upload the package to the website in one simple command:
$ python setup.py plone_org

The patch aslo corrects a few minor bugs in distutils, such as:
- .pypirc was not found on Windows - pydistutils.cfg was not found on Windows

PyPI usage statistics, the zc.buildout effect

Tarek Ziadé — Thu, 08 May 2008 20:59:00 +0200

It is interesting to look at the usage statistics at PyPI: [http://pypi.python.org/webstats/usage_200805.html#DAYSTATS][]

Besides the front page, the most visited pages are in order (same thing in April):
- zc.buildout - setuptools - zc.recipe.egg - plone.recipe.zope2instance - plone.recipe.distros - python-openid - elementtree - plone.recipe.zope2install

Then the packages in the top 60 are mainly zope eggs. This corresponds to the sequence done by a Plone or Zope being build in a buildout.

If you take a closer look at [http://pypi.python.org/webstats/url_200804.html][], the next top hits corresponds to zope.* then plone.* packages.

Pycon FR is coming up - may 17/18 - ask the program

Tarek Ziadé — Wed, 07 May 2008 13:24:00 +0200

Pycon FR will take place in Paris, may 17/18, at the Cité des sciences et de la Villette.

A very rich program will be presented:
- Learn Python first ! (Eric LEBIGOT) - Graphviz made easier with GvGen (Sebastien Tricaud) - Genes research with Python (Andre Espaze) - CouchDB (Benoit Chesneau) - PyPy explained (Victor Stinner) - Quality Assurance (Julien Jehannet) - Zope 3 overview (Christophe Combelles) - Scapy in action (Renaud Lifchitz) - E-sonoclaste, multimedia annotations (Vincent Rioux) - Why Django ? (David Larlet) - PLUIE project (Michel Claveau) - WSGI in practice using Paste (Gael Pasgrimaud) - Distributed Version Control with Mercurial (Michael SCHERER) - Create and deploy Python apps with zc.buildout (Tarek Ziadé) - Django everyday: quality and performance (David Larlet) - Python 3000 (Victor Stinner)

There will be lightning talks as well, and the French speaking user group (Afpy) will have its annual meeting on sunday morning. A second room will be available for sprints and BoF sessions as well. All the talks will be in French this year

If you are around Paris, please join use ! It free !

Otherwise, watch the talks in the live stream: http://fr.pycon.org/stream-live

Thanks to our partners, for making this happen:
- The Python Software Foundation - Logilab - Nerim - Ingeniweb - Resolver Systems - Pilot Systems - WingWare - Makina Corpus - Toonux - ActiveState - Gymglish - Eyrolles - Cité des Sciences et de la Vilette

Plone Paris Sprint wrapup #3, new.plone.org, collective.dist released !

Tarek Ziadé — Tue, 06 May 2008 12:49:00 +0200

The main task I worked on during the sprint with Alex and Matthew was making PloneSoftwareCenter ready for the new version of Plone.org. These guys rock. We did tons of things and the new plone.org website is coming up. Alex worked for quite a while on migrating plone.org to plone 3, but let me focus on the software center part.

First of all, let me explain what is the final goal of the work done in the software center.

The future of Plonistas: a 100 % egg-based world and what is means

for production

Since a few years, Zope code base was moved into a set of namespaced packages. Plone is following closely. From there zc.buildout is providing a simple way to pick up the right set of packages to build an application. This set is automatically chosen by recipes at Zope and Plone level. Then developers add their own packages and dependencies to provide custom features in their applications.

This mechanism means that each team has to:
- make distributions of packages (tarball, eggs, ..) - build the application with buildout, then release a source distribution - and provide a way for other developers to build the application on their own, by:
- publishing the buildout configuration files - pushing the packages the buildout uses into to a server that is reachable by developers - make sure Plone and Zope packages are also reachable

This means that each team is responsible at least to release packages.

Distutils and Cheeseshop

Distutils provides two commands to publish packages to the world: register and upload. These commands were intended to provide to the developer a way to push packages to any server that supports the protocol, by using the --repository option.

But in reality, the only server that is publicly available is the Cheeseshop (pypi.python.org). Furthermore Distutils is not providing everything needed to work with another server than the Cheeseshop, like I will explain later in this post. So all Plone and Zope packages are uploaded there.

Therefore, pypi.python.org became a single point of failure when you need to build a Plone or Zope application. With the growth of the community, this means that the repository will get bigger and have to deal with more and more request. PyPI weight around 4 gigas at this time, of zip files and tarballs.

While the actual server is really fast, it makes it a bit hard to work when it is down. It didn't happen often. Twice as far as I remember. But when it happens, the crowd that uses zc.buildout is frozen because they run buildouts several time per day. They can use cache of course, as long as those are up-to-date.

Some mirrors emerged, like http://release.ingeniweb.com/pypi.python.org-mirror. Some enhanced indexes were created as well, like http://download.zope.org/simple, which is still referring to pypi.python.org but performs a bit of crawling to speed things up when a buildout uses it through easy_install.

But all of these are just enhancement to a cheeseshop-centric model.

Furthermore, in case of a private application, you do not want to see packages in the Cheeseshop but you might also want to provide developers a similar way to manage and use them.

Plone.org software center is dying !

For public applications, given the fact that Distutils provides a simple way to push a package to a public location (one shell command), people have started to stop updating the Products section of Plone.org. This happens because updating it means doing a whole lot more than the simple register+upload call. You need to login into the website, then manually upload the packages, then change the front page of the product if needed etc..

So being able to push packages to plone.org with the same set of commands, is the right solution for developers.

Pushing a package means uploading a public archive but it also means updating the front page for the given package, with the register command.

That what I worked on, at PloneSoftwareCenter level, continuing Sidnei's work: making it act like Cheeseshop. Now the feature is ready and being alpha-tested at new.plone.org

Toward a distributed model

What we will be able to do from there, is to distribute packages (in blue) to the Cheeseshop (2) as usual, but also to Plone.org (1). Having such a tool also allows people to run other cheeseshop-like servers, wether they are public (3) or private (4). This is useful when you are working on customer projects and do not wish to make their packages available to the world, or even if it is a public project, you do not want to push them to PyPi.

Furthermore, it allows having a bit of redundancy : your packages become available in several places, which is better. Think about the mirrors at Sourceforge, same thing here...

This distributed model is used at Ingeniweb, and in other companies I am starting to list (If you do please let me know, this is important to promote the tool).

The new Plone Software Center (PSC)

If you are extensively working with packages and buildouts, you should consider trying the new PSC, read http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/ for this. Each project now in the software center can hold several eggs, and it makes it possible to provide a nice deployment model for your teams: developer push packages releases in such a server with the distutils standards, while customers or deployers can build their applications by picking them up through their buildouts.

And it is in Plone, so you can provide extensive features, like a bug tracker, and all the sweet things Plonecan provide.

The work left to be done in PSC is :
- making the storage for archives (tgz, egg) pluggable so new storing strategies can be provided in separate packages, under the collective.psc namespace, I started this work in a branch. - finish the collective.psc.mirroring package, that will be used in plone.org to fill a system folder, in order to provide an Apache direct view over the archives stored. This will make zc.buildout / easy_install use this stream rather than hitting the Plone instance. Although the final goal will be to transform it into a storage strategy, but time is running and this is useful now for plone.org. - extensive tests on new.plone.org

collective.dist

To be able to deal with several servers, I have released collective.dist after the sprint (previously iw.dist). This package provides two new distutils commands: mupload and mregister, together with a new pypirc format.

Follow the documentation, and you will be set in a matter of minutes. This package is already used by many developers to make their life easier, and will help you when the new plone.org goes live.

disutils evolution

collective.dist is an evolution of distutils I have been working at Python level for months. It is available as a Python patch here : http://bugs.python.org/issue1858. mregister and mupload are just modified register and upload commands that makes it possible to interact with several PyPi-like servers, that's it. PSC uses Distutils standards in any case, so it is possible to use the regular register and upload commands with it of course. It is just not convenient because you will need to have the same username and password on both CheeseShop and the third-party egg server (or change your pypirc file everytime :D)

So the patch, basically, just makes the -r option of distutils commands a 100% operational.

Now my job is to convince Python core developers it should be integrated into Python 2.6, and I am working on this. It is a logical evolution, but it sounds overkill to some people that does not have this kind of need: "Why changing that ? we have only one package server, which is the cheeseshop".

It can also sound to some of them like I am (together with the team of 20 developers we have ;) ) the only person on earth that wishes it, but as soon as the new Plone.org will be online, a lot of people will start needing such a feature. But having it in 2.6 will make it available as a standard in Plone/Zope world in... a few years.

In any case, most of them agree in the fact that this change is logical and that the current pypirc format is not to be kept. So I guess it is just a matter of time and patience.

http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/

Plone Paris Sprint wrapup, part #2: collective.buildbot released !

Tarek Ziadé — Fri, 02 May 2008 09:24:00 +0200

The Pimp my Buildbot project that was started here at Ingeniweb some time ago, to be able to set up a buildbot in a matter of minutes with zc.buildout, was continued during the sprint, and the guys did a great job on it.

It will be used here in customer projects with a Paster that adds buildbot support when a project starts, because it is a waste of time for the developers to set everything everytime.

Jean-Francois Roche, Kai Lautaportti and Gael Pasgrimaud added extensive configuration options (mail, scheduling), and made the SVN Poller works. This feature allows for instance to make the buildbot watch a SVN repository without having to add a hook in the server (post-commit hook for instance), when you don't own it (SourceForge, collective, etc)

The tool is released in the collective, and available at the Cheeseshop in one single package !

If you want to set a buildbot
- provide for each one of your project a buildout that has a test script - make sure the test script returns exit code (--with-exit-status with zope.testing) - create a buildout cfg file using collective.buildbot - run buildout, that's it ! - run the master, slaves scripts, and go to the /waterfall page

Just try out our own buildbot by running this sequence:
$ cd /tmp/

$ mkdir my_bot

$ cd my_bot/

$ svn co https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/buildbot/trunk .

$ python bootstrap.py

$ bin/buildout

$ bin/master start

$ bin/linux_debian start   (that's our slave)

You should have a buildbot running then at http://localhost:9000/waterfall

The tool, without the polling stuff, also works with Mercurial and Bzr, but probably needs more tests with these repositories. We also need to make sure the slaves works fine under Windows, and add a nice front page to the buildbot.

If you use it let us know !

Plone Paris Sprint wrapup, part #1: How do you use eggs and zc.buildout in your projects ?

Tarek Ziadé — Tue, 29 Apr 2008 13:02:00 +0200

This is the first post of the wrapups I want to do about the sprint that we had in Paris last week-end. We had a Bird of Feather about how people use eggs and zc.buildout in their projects, how they release and deploy them.

There were some people from Headnet (Anton, Mustapha) and Infrae (Kit, Sylvain) and Ingeniweb (Me), and we compared a bit how we are working with eggs, zc.buildout etc.

That is what I love in our community: companies can share their knowledge and grow up all together, technically speaking.

We all have internal recipes, command-line scripts, and are all relatively happy with zc.buildout. This discussion was very instructive.

From there, I thaught it would be a good idea to launch a new project in the community, on the top of zc.buildout (and maybe zc.sourcerelease), that would provide a common set of tools and deploying best practices, for people that works with the buildout, no matter which framework they use (Silva, Plone, etc.)

The first step for this project is to find the common needs and see if we can join forces to build common tools. To start it up, I decided to wrap up and release our internal set of tools into a single package called iw.releaser and publish it. This is the work I have done during the last months with the help of Gael, to help our team to work with zc.buildout in Plone Projects. It is Subversion dependent at this time.

I am expecting some feedback from Anton and Sylvain to see if we can make it a collective tool.

This package provides:
- a skeleton to build a project structure (buildout, packages; docs, etc.) so all projects have a standardized structure - a 'release' distutils command that releases a package, upload it to the Cheeseshop or other servers, and send a shout out mail - a set of command line tools, that can be used to deploy a buildout. These commands are doing many things besides launching a buildout building (which is a bit different from zc.sourcerelease)

This package is used at Ingeniweb for a few months now, and I tried to summarize how it is used in the docs. I bet a lot of bugs will be found if you try it, so consider this package as a non-mature package yet.

Join us all ! So you will be able to release and deploy your buildout-based apps with a few command-line calls, :D

Plone Paris Sprint going on, quick wrapup

Tarek Ziadé — Sun, 27 Apr 2008 10:51:00 +0200

We are working here at the Paris Sprint, and many things are going on. I wanted to do a quick wrapup before a longer and final wrapup.

Here's all the tasks going on (I might miss some):
- Youenn and Cartsen are working on plone.recipe.apache, that will let you build and/or setup Apache in a buildout - Michel presented a very interested talk yesterday on how insecure Plone can be (Man in the Middle attacks, etc), on the various existing solutions, and works on things to enhance it - Maik leads a Funittest group, teaching the tool to people, and enhancing i - Olivier and Christophe are working on Grok to learn it. Christophe worked yesterday on filling content to the new zope.org web site - Matthew, Alex and I are working on PloneSoftwareCenter and plone.org, so the website can be ready to move to an egg-enabled Plone 3. wOOt ! - Gael, Jean-Francois (and Kai yesterday) are working on "Pimp my Buildbot": configure and launch a complete buildbot system in a matter of minutes, using a simple buildout cfg file and running a simple command. I think this could be used for Plone repository and the collective as well ! The tool will be move to the collective at the end of the sprint. - Gilles, Cyrille, and others (sorry, can't keep up with all names), are working on iw.fss (FileSystemStorage for plone 3) - Mustapha and Anton are making sweet things in ZopeSkel - Anton worked also on finding the best way to do aysnhronous tasks in Plone

- There's a big PloneGov team that is working on making the tool friendly with various government systems. - Jean-Nicolas leads the work on kss/eventPush, so we all get a upload widget with live feedback (no flash inside ;)) - Godefroid and Lennart are Grok-ing as well; - Christophe is preparing [collective.kss.fly_guy][], a tool that provides drag and drop in folder_contents - Harito, Christian and Wim are making a sweet new theme for Plone: "Notre Dame". The theme is kept quite generic, with all the plone tools/portlets designed in blue tones, kept in mind to be easy adjustable and changed to other color schemes. It will be a table-less egg for Plone 3.

More blogging tomorrow, I need to get back to my task :)

Help Plone.org, help Alex Clark

Tarek Ziadé — Mon, 21 Apr 2008 12:18:00 +0200

Alex is trying to raise money for his travel expenses, to go to the Plone Paris Sprint.

Alex will work on making Plone.org go Plone 3, so this is important !

Help him here : http://www.aclark.net/Members/aclark/help-me-upgrade-plone.org

Pycon FR is not competing with EuroPython

Tarek Ziadé — Sun, 13 Apr 2008 10:59:00 +0200

edit: I have changed the blog title, which was not correct in English, thanks Maik ;)

I wanted to react on Martinj's blog to make things clear about Pycon FR

Background

First of all here's a little background about Pycon FR.

Afpy (Association Francophone Python) is a french Python user group created four years ago. The group counts around 100 members and is the only one of this size in french-speaking countries as far as we know. It has members from France, Belgium, Canada and North Africa countries.

Many meetings are organized throughout the year, mainly in Paris, and sometimes in Charleroi in Belgium.

An important event took place at the "Cité des sciences" in Paris last summer, called "Journées Python" We had around 80 attendees and a rich program was given during two days. Topics where presented by french speaking people from France and Belgium, and we had talks about AI, Django, Zope, TuxDroid, etc

Some videos are available on the web, you can probably reach them from here: http://video.google.fr/videosearch?q=journees.afpy.org

The conference was 100% free for attendees. We also decided to chose carefully the dates so the conference would not compete with other events like Europython.

This event was a *true* success.

This event is now called Pycon FR, to be in the same line of what happens in some other countries. This makes a lot of sense indeed, to have national Python events, with standardized names.

Attendees

Pycon FR attendees are mainly people that speaks english but for some of them not enough to easily follow English talks. Most of those people would not make it to Europython because of this language barrier, and a national conference held in Paris is something that was quite missing at this time, for them.

So PyCon FR is just a sign of the growing interest about Python, and should be seen as a 100% positive and constructive thing, believe me.

Together with Europython and Pycon US, the "local" PyCons are in my opinion, a very good thing to spread the good word. We will, like last year, promote Europython there. Maybe we could even do a survey this year to know exactly who goes to Europython, etc.

Pycon FR expansion plans

When I said in my last blog entry that we were working on making it less french-centric, and having international attendees, I should have explained it better to make sure no one thinks Pycon FR might be a bad thing for Europython.
- If we do invite international speakers, it will be done only if we are able to provide a live translation system. - The event date will always be picked carefully, as usual, so it does not compete with Europython.

Europython and local PyCons expansion

Here's my opinion about what Europython strategy could be: since it is an international english-speaking event that gathers a lot of people. It could be the seed to set up a local Pycon event in places where it does not exist yet, using the momentum of the event.

This could be done by working with the local user group this way, by organizing maybe two years in a row Europython there. So, maybe we could see "Pycon Lithunia" next year ? Then move EuroPython to Amsterdam and see "Pycon The Netherlands" in two years ?

Europython exists to promote the language at an international level, so that could be a good way to spread imho.

But please, pretty please, don't see local Pycons as Europython competitors, this is not true, this happens just because Python is growing.

Everyone should be happy about that I think. I disagree with the fact that each conference might fight to have the "best international speakers" for a very simple reason: the number of speakers is growing too, and I don't think either Pycon either EuroPython will ever miss of interesting talks.

The next smart move I think is to organize and synchronize all these events, and that is where the Python Software Foundation has a role to play in my opinion. They have started anyway, and I am really confident about what will happen next.

Pycon FR is coming !

Tarek Ziadé — Sat, 12 Apr 2008 07:31:00 +0200

PyCon FR is coming up ! We published yesterday the final program, and I must say that this second annual meeting of Pythoneers in Paris, looks very good.

Extracts:
- Why you should learn Python - GraphViz with GvGen - Gene search - CouchDB - PyPy - Quality Assurance - Scapy - Zope 3 - zc.buildout - Django - WSGI - etc...

This is going to be two intense days in Paris, so if you are around, please join us, it's free.

Paris - Cité des Sciences et de la Vilette - 16/17 May .

The talks will be in french, but the organization team works hard to have english talks as well next year. Anyway, you don't need to speak french to join us and have a good time ;)

zc.buildout monday trick :)

Tarek Ziadé — Mon, 07 Apr 2008 16:01:00 +0200

When a site used by your buildout is not responding, you can stare at it for ... ever

Add these two lines in your bin/buildout script:
import socket

socket.setdefaulttimeout(10)

With this, the buildout will go to the next link after ten seconds. This trick made my day :)

Doctests: specify the target readership

Tarek Ziadé — Sat, 05 Apr 2008 13:53:00 +0200

Ben Bangert was reacting about zc.buildout doctests, saying that they are hard to read from the PyPI page, and the examples hard to use and follow.

I agree with Ben as these doctests are very hard to read when you are not familiar with zc.buildout testing modules, which provides a set of API the doctests relies on.

But from a developer point of view, adding a feature to such a package is best done through doctests, using zc.buildout.testing goodies. And a developer that is familiar with this package, will find this doctest very useful.

zc.buildout in any case, is trying to structurize its PyPI front page, and push a maximum amount of doc for users, so ... kudos !

I think the problem is more about specifying the target readership.

I would like to point another example that comes in zc.buildout: dependency-links.

The main doctest that appears at PyPI as a light, human-readable section: [http://pypi.python.org/pypi/zc.buildout#dependency-links][]

And the very same section is continued in a specific doctest that does not appear on the main page: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/dependencylinks.txt?rev=81182&view=markup

So what happened here is that the developer specified two kind of readers:
- people that will reach the package through its PyPI page - people that will go deeper in how the package works, through recipes or tutorials

From there I think there's a simple guideline that could be applied to enhance the package documentation when adding a feature:
- resume the feature in the main page (long_description) with examples that does not rely on specific testing API. In other words it should be made of plain english and plain python when needed; - leave doctest that relies on internal testing API as complementary documentation. - define for each doctest its nature (recipe, tutorial, etc)

How could we help people doing such structuration ?

The distutils metadata could be a good place to do it, by adding an extra_doctests list for example, that would contain a list of doctests. From there, PyPI could display the long_description text as usual, and add a "more information section".

Let's take an example:
def _(f)

return open(f).read()

setup('my.package', ...

long_decription=_('README.txt'),

extra_description=

{'recipe': [_('create_this.txt'),

_('do_that.txt')],

'tutorial': [_('how_to_use.txt')

_('how_to_2.txt')]}

From there, PyPI could provide a Table of content, with a structurized documentation, and additional pages for the package, grouped by types (recipe, tutorial) etc. -> Maybe a Sphinx-powered PyPI ? :)

By the way, I have another post related, which tries to summarized good pratices in technical writing : http://tarekziade.wordpress.com/2007/02/23/technical-writing-the-seven-laws

http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/dependencylinks.txt?rev=81182&view=markup
http://tarekziade.wordpress.com/2007/02/23/technical-writing-the-seven-laws

Pimp my buildbot !

Tarek Ziadé — Tue, 01 Apr 2008 12:00:00 +0200

Edit : this post is a bit deprecated, the project is now called collective.buildbot. See: http://pypi.python.org/pypi/collective.buildbot

When a project team use Test-Driven Development to build the code (everyone should), the next step is to set up automate builds, as explained in continous integration.

This is where Buildbot is great. It is easy and flexible to install, even more since Twisted has been eggified. A few easy_install steps are now sufficient to create a buildbot waterfall. Configuring a buildbot requires writing a few Python scripts though, and this has to be done everytime a project starts.

In my work, I need to be able to set buildbots in a matter of minutes, and they are always similar. They just need a buildmaster, a buildslave, and a few rules.

A first attempt to make things easier is to write a Python Paste script that generates default files. This is not very flexible though, as upgrades are still a bit tedious.

A more interesting solution is to provide zc.buildout recipes that take care of buildmaster and buildslave generation, through a very simple configuration file.

We have started this project, in order to be able to launch buildbot within a buildout.

The project has three parts:
- iw.buildbot: a thin layer on the top of Buildbot that allows to drive it with configuration files, instead of Python code. In other words, it makes buildbot configuration based on declarative configuration file and dynamic Python code, instead of declarative Python code. - iw.recipe.buildmaster: a zc.buildout recipe that creates a buildbot instance together with configuration files.

iw.recipe.buildslave: a zc.buildout recipe that creates a buildslave

The result is that creating a buildbot is done in a few lines in the buildout.cfg file:
[buildout]

parts =

  buildmaster

  linux_debian

[buildmaster]

recipe = iw.recipe.buildmaster

project-name = Ingeniweb Public buildbot

project-url = http://ingeniweb.com

port = 8999

wport = 9000

url = http://buildbot.ingeniweb.com

slaves =

  linux_debian    xxxxx

projects =

  iw.recipes

[linux_debian]

recipe = iw.recipe.buildslave

host = localhost

port = ${buildmaster:port}

password = xxxx

[iw.recipes]

slave-name=linux_debian

base-url=http://ingeniweb.svn.sourceforge.net

repository=/svnroot/ingeniweb/projects/iw.recipes/

branch=buildout

build-sequence =

    python bootstrap.py

    bin/buildout

test-sequence =

    bin/test -v --exit-with-status

buildmaster defines the project name and url, the buildbot web port, slave port and url, and a list of slaves and projects.
each project has his own section, where the Subversion path is defined, as well as the build sequence and the test sequence.
each slave is defined in its section

From there other buildouts can be created to group packages to be tested, and to define a script that can be used for tests. For Zope applications, that would be zopectl of course, as long as it is used with --exit-with-status.

This is the case for iw.recipes: it grabs all iw.recipes.* eggs to hook them to a testrunner.

The result can be see here: http://buildbot.ingeniweb.com

The next steps are to make sure everything works fine under Windows, and see how things goes in various projects, then make it work with all kinds of VCS and schedulers.

I will probably propose a sprint task on this in Paris sprint and see if the package meets interest.

Plone Paris sprint D-30

Tarek Ziadé — Tue, 25 Mar 2008 09:02:00 +0100

This is a reminder if you wish to join us in Paris, April 25-27. (week-end)

There will be a Plone sprint, and we expect around 30 people so far.

All infos are located on the OpenPlans page here: http://www.openplans.org/projects/plone-3-paris-sprint/project-home

There's also a Grok sprint a few days after Paris one : http://wiki.zope.org/grok/GrokkerdamSprint
so this is a good opportunity to do a Python european tour ;)

http://www.openplans.org/projects/plone-3-paris-sprint/project-home

iw.recipe.backup and other zc.buildout recipes

Tarek Ziadé — Tue, 25 Mar 2008 07:20:00 +0100

A new recipe has been added to the iw.recipe.* family: iw.recipe.backup.

If you use zc.buildout to deploy Python applications in production, this one can be used to backup a buildout folder, before any upgrade is attempted, or for a daily backup. It creates an archive with a timestamp on its name, in a defined folder, with the whole folder content. Some directories can be excluded if needed.

This script is not doing fine-grained backups of the Data.fs like repozo.py does, but can be combined with it if this finesse is required. The recipe creates two scripts in the bin folder: backup and restore.

backup creates an archive and takes no parameters, whereas restore takes an archive name and decompress it into the buildout folder. The latter does a backup before it decompresses the archive for more safety.

Last, it will raise an error if the backup folder is located within the buildout folder ;), and logs all commands into a log file. This is a tiny recipe, but provides a simple way to backup your buildouts in a single call, no matter the platform you are in.

Other recipes we provide at this time are:
- iw.recipe.cmd: provides a way to execute shell calls, or inline Python calls. This is useful when you need to set a few things when a buildout is built, that does not worth creating a new recipe; - iw.recipe.fetcher: similar to wget. We use it to build python-win32.zip archive, to get all installers on the web; - iw.recipe.fss: will let you configure FileSystemStorage, so no extra step is required; - iw.recipe.pound: compiles Pound load balancer, and creates its configuration file, so it can be run directly; - iw.recipe.sendmail: set zope.sendmail as the default mail sender in a Zope 2-based application. This is useful if you wish to have a fast and robust mailhost service in your Plone sites; - iw.recipe.squid: This recipe installs all parts needed to run a Squid server dedicated to serve a Zope application, staying friendly with Apache and setting things nicely for Plone; - iw.recipe.subversion: recipe to checkout a svn location into a part. It is different from infrae.subversion because it creates a tarball out of the subversion checkout, and put it in the downloads folder. In other words, it works offline as well as long as the tarball was built once. It doesn't have the nice feature infrae.subversion provides which is a code checker that will raise an error when buildout tries to remove a changed file. But as we use the develop feature of buildout to create our code, we didn't need that feature. Subversion checkouts are useful to us when we want to use a bleeding-edge version of a third-party package that has not been released yet; - iw.recipe.template: this one provides a way to write files based on Cheetah templates, like Python Paste does, and is useful for micro-needs, like iw.recipe.cmd.

If you use one of those recipes and whish to make a feature request or a bug report, I have created a new space on plone.org to have a bug tracker for all recipes: http://plone.org/products/iw-recipes/issues

Sphinx as a doc builder for Python projects

Tarek Ziadé — Sun, 23 Mar 2008 09:08:00 +0100

Last year, I worked on a documentation builder based on doctest and reStructuredText format called PyCommunity. This tool is collecting doctests from Python packages and from special places on a source repository, basing its work on a set of project good practices I had presented and used in a Pycon 2007 tutorial (slides).

Basically, it uses the best practices described in Andreas Ruping book, called Agile Documentation, applied to Python projects using TDD and doctests. It can be called Document-Driven Development.

I have never found the time to finish the tool and I was looking forward to get back to it. As a matter of fact, George Brandl has released the tool that is used to generate Python documentation, called Sphinx, which does many things I still have in my TODO list, like pygments integration, and many other things.

See how Sphinx generates Python doc here: http://docs.python.org/dev

From there, it becomes dead simple to generate a website for a Python project, based on packages doctests and text files and on a specific doc/ structure.

Sphinx annoucement is really exciting, and it shouldn't be too much pain to bundle it in a buildout recipe to manage a project documentation. Since it is based on templates and configuration files, a default structure can be generated to startup a project documentation together with a code base.

I have to try and see if Sphinx allows me to set everything up. In other words, replay my Pycon tutorial with it.

zc.buildout and Plone at OSCON'08

Tarek Ziadé — Fri, 21 Mar 2008 18:51:00 +0100

Cool, my proposal for OSCON 2008 has been accepted, and will be about Plone 3, zc.buildout and the work on creating private PyPI servers using PloneSoftwareCenter.

It will present how we work at Ingeniweb with both public and private packages, to deliver Plone applications to customers, using eggs and buildout.

It's funny because since a few days, there are a lot of discussions around distutils, PyPI and setuptools, and about making things better at Python level on how to distribute packages and applications. So it seems to be a hot topic.

From my point of view, PyPI brought a lot in the past years in this area, and being able to deploy a pypi-compatible software center in a company helps a lot in using the same set of command line tool. (distutils/setuptools)

So, I am pretty happy nowadays with zc.buildout and setuptools (thanks to Mr Fulton, Eby and al), despite all the critical that has been made about setuptools in the last few days, and despite the fact that it is *so hard* to make a tiny little change make it to distutils trunk :'( ...

Anyways, if you do Plone or Zope dev, and if you are interested about software delivery, I'd be glad to exchange about it, to see how you work and deliver Zope apps, to get other point of views before OSCON.

Here's the abstract of my talk:
Software delivery for complex systems in Python/Zope used to be a little bit homemade: people usually used custom scripts to deploy their systems, or relied on generic installation tools. For Plone applications, most of the time a complex installation guide was provided to the customer, with a list of dependencies to install and system changes to take care of.

The Python Package Index (PyPI), formerly the Cheeseshop, brought a few years ago a new way to distribute Python applications, together with setuptools. It made it possible to install a Python library the same way package systems like apt or yum does. From there people started to deliver their software in separated components, called eggs. Since most applications in Plone are now egg-based, it is possible to install a software with a list of eggs.

zc.buildout provides a descriptive language to list all eggs needed for a software to run and a plugin system that allows to customize each steps.This talk will present a case study of a Plone application life cycle:

environment building - creating the buildout and its recipes
continuous integration with buildbot – running the buildout on target systems
deploying – preparing and packaging the buildout for an offline installation
updating – preparing and releasing an update

And will present a set of extra tool we have built on the top of zc.buildout to standardize our projects developments and help the developers:
- a set of templates to start a buildout-based project in subversion - a tool to create buildbot slaves automatically, given a buildout - a diff tool to ease the upgrade of a buildout that is in production.

how to run your own private PyPI (Cheeseshop) server

Tarek Ziadé — Thu, 20 Mar 2008 11:26:00 +0100

PloneSoftwareCenter 1.5 is heavily developed and not yet released, but the current trunk code is working well to use it as a PyPI-like server. It can be really useful for companies that develop Python software and are looking for a way to centralize their eggs internally. That's what we use now at Ingeniweb to work on customer projects.

This tutorial explains how to set a cuttting-edge PloneSoftwareCenter server, if you want to be an early-adopter of what will be the next version running on plone.org software center in a few months (but this code is to be used at your own risks of course ;))

Why a private PyPI ?

You have some python packages you want to treat the same way the Cheeseshop does. In other words you want to work with them with [easy_install][], zc.buildout, etc..

But these packages are private to your company...

The simplest way is to store your eggs on some network folder. But distutils and setuptools provide a nice set of commands to automatically build and upload eggs at PyPI or any server that implements PyPI apis.

How ? PloneSoftwareCenter !

The Plone community provides a nice tool to manage Python packages, and it is PyPI compatible. In other words it can act like Cheesehop to interact with your command line tools.

It also provides an extensive set of features to manage your releases, run your bug trackers, etc.

See the plone.org products section, it is powered by PSC.

4 steps to install

Thanks to zc.buildout, a Plone Software Center (PSC) is really easy to setup. There's no binary distribution yet though, so you need to compile a few things.

Step 1 - Pre-requests

If you are under Windows, grab this archive : http://release.ingeniweb.com/third-party-dist/python2.4.4-win32.zip

decompress it, and run "install.bat". It will install Python 2.4 together with a set of tools, and PATH updates.

If you are under Linux, make sure you have gcc, subversion and make installed. Then install easy_install and PIL,
like this:
$ wget http://peak.telecommunity.com/dist/ez_setup.py

$ python ez_setup.py

$ easy_install http://release.ingeniweb.com/third-party-dist/PILwoTk-1.1.6.4.tar.gz

Step 2 - installing PSC

Make a directory on your system called softwarecenter
Get into it and grab PSC code with the svn command line:

$ svn co http://svn.plone.org/svn/collective/Products.PloneSoftwareCenter/buildout/trunk .
run the buildout with this set of commands:

$ python bootstrap.py
```
$ bin/buildout
```

It will take some time, to grab all elements needed to build PSC.

run the server
$ bin/instance start

Step 3 - setting up PSC

Let's create a Plone website with a PloneSoftwareCenter instance:
1. Open a browser and go to http://localhost:8080/manage. The login/password is admin/admin. 2. On the left part, there's a dropbox, select "Plone Site" the hit the add button 3. In the form, set the id to "plone" and hit enter. 4. Go to [http://localhost:8080/plone/prefs_install_products_form][] 5. Check "PloneSoftwareCenter" on the left side and hit "Install" 6. Go to http://localhost:8080/plone 7. In the "Add new..." menu, Click on "software center" 8. In the form, in the Title, put "Catalog" 9. Check Use Classifiers to display Categories (with Topic :: *) under Classifiers 10. Hit the Save button

Your Software Center is ready and available at http://localhost:8080/plone/catalog

Step 4 - setting up the client-side

Now let's set the client-side so people can use your Software Center:
1.
install iw.dist:

   $ easy_install iw.dist

create a file in your home directory, called .pypirc with this content:

[distutils]

index-servers =

  pypi

  local

[pypi]

username:YOUR_PYPI_LOGIN

password:YOUR_PYPI_PASSWORD

[local]

repository:http://localhost:8080/plone/catalog/

username:admin

password:admin

Of course, the localhost value will differ if you are located on another machine..

iw.dist adds two new commands in distutils: mregister and mupload. These commands enhance register and upload to make distutils work with multiple servers. This should be merged hopefully in Python 2.6 very soon.

Let's use it !

Now, you will have two new commands in distutils, called 'mregister' and 'mupload' that will let you use either your PSC either PyPI.

Let's upload an egg into PSC:
$ python setup.py mregister sdist bdist_egg mupload -r local

Let's upload an egg into PyPI:
$ python setup.py mregister sdist bdist_egg mupload -r pypi

if -r is omited, pypi is the default one.

If you want to use PSC in zc.buildout or easy_install, you can provide http://localhost:8080/plone/catalog/simple as a find-links or index value:
[buildout] find-links = http://localhost:8080/plone/catalog/simple

Or:
$ easy_install -f http://localhost:8080/plone/catalog/simple my.egg

That's it !

http://release.ingeniweb.com/third-party-dist/python2.4.4-win32.zip

plone.recipe.cluster, one control script to rule them all

Tarek Ziadé — Thu, 06 Mar 2008 22:29:00 +0100

There is something that is a bit missing in a buildout environment to be able to drive it with no pain : a gobal script that allows starting, stopping or restarting every software that is bundled in it.

For instance, a buildout often has a Zeo server, a Zope instance and some other servers like Pound, Apache or Squid (just control scripts and sometimes a whole build. A global script can handle this by running a suite of commands for each action: start, stop, restart.)

Let's take an example: we want to run a Zeo server, a Zope client, and a pound load balancer. These actions can be listed in a buildout section like this:

[buildout] ...

[cluster]

recipe = plone.recipe.cluster

poundctl = ${buildout:bin-directory}/pound -f ${buildout:directory}/parts/pound/etc/pound.cfg -c ${buildout:directory}/var/pound.pid

start =

    ${buildout:bin-directory}/zeoserver start

    ${buildout:bin-directory}/instance start

    ${cluster:poundctl}

stop =

    ${buildout:bin-directory}/zeoserver stop

    ${buildout:bin-directory}/instance stop

    pid:${buildout:directory}/var/pound.pid

restart =

    ${buildout:bin-directory}/zeoserver restart

    ${buildout:bin-directory}/instance restart

    ${cluster:poundctl}

The zeoserver and instance scripts are daemons that just need to be called, and the pound script which is also running as a daemon, do not have any option to be stopped, but will let the user define a pid file. So having a way to kill the pid defined by this file is enough in this case.

The recipe can therefore build a control script that launches the set of commands for each action:

$ bin/cluster start

Starting Cluster...

$ bin/cluster stop

Stopping cluster...

Let's take another example: a buildout that runs a simple script that is not daemonized. In other words, it needs to be run in the background and its pid saved, so the cluster recipe will know how to stop it:

[cluster]

recipe = plone.recipe.cluster

start =

    background:${buildout:directory}/bin/script

    ${buildout:bin-directory}/instance start

stop =

    ${buildout:bin-directory}/instance stop

restart =

    background:${buildout:directory}/bin/script

    ${buildout:bin-directory}/instance restart

In this case, the background: prefix will indicated the recipe to execute the command in the background, and to keep its PID in a file. When the stop command will be called, it will be stopped automatically.

I think that all cases are covered by this recipe, and I have a prototype working for Linux/Mac here:

http://svn.plone.org/svn/collective/buildout/plone.recipe.cluster/trunk . I took back and adapted some work done by Blue Dynamics (bda.daemon). System programming is a bit over my head so it needs some more work in the coming days, but it is working already.

Then, the next step will be to make it win32 compliant, using NT Services for this, and hopefully releasing a first 0.1.0 version.

If you feel that there's some missing use cases, let me know ! (the full README is here)

http://svn.plone.org/svn/collective/buildout/plone.recipe.cluster/trunk

How to add rotatezlogs in your buildout

Tarek Ziadé — Sun, 02 Mar 2008 20:43:00 +0100

iw.rotatezlogs is a very useful package made by Gilles Lenfant to rotate log file in Zope. This is quite useful to make sure they don't get huge.

I have made the necessary changes in plone.recipe.zope2instance and plone.recipe.zope2zeoserver to make it available in a buildout environment. Since those changes are now released, here's the way to add the log rotator in your buildout if you wish to use it:
[buildout]

...

[instance]

...

event-log-custom =

    %import iw.rotatezlogs

    <rotatelogfile>

        path ${buildout:folder}/var/log/event.log

        max-bytes 1MB

        backup-count 5

    </rotatelogfile>

access-log-custom =

    %import iw.rotatezlogs

    <rotatelogfile>

        path ${buildout:folder}/var/log/instance-Z2.log

        max-bytes 1MB

        backup-count 5

    </rotatelogfile>

eggs =

    ...

    iw.rotatezlogs

[zeo]

...

zeo-log-custom =

    %import iw.rotatezlogs

    <rotatelogfile>

        path ${buildout:folder}/var/log/zeoserver.log

        max-bytes 1MB

        backup-count 5

    </rotatelogfile>

eggs =

    ...

    iw.rotatezlogs

The package is added in the eggs section to make sure the recipes adds it in the Python path when the script (zope or zeo) is loaded.

Gilles told us that the rotator will be integrated in ZConfig itself sometimes, so it will be even simpler to use.

plone.recipe.zope2zeoserver and Windows

Tarek Ziadé — Thu, 21 Feb 2008 09:27:00 +0100

The 0.11 release of plone.recipe.zope2zeoserver is out. If you are under Windows you will probably enjoy this upgrade. The zeoctl starting script was not working under win32 because it is based on zdaemon, which is Linux-specific.

I have added two scripts in the recipe, to be able to launch Zeo:
- at the command line, with bin\zeo.bat - as a service, with bin\zeoservice.exe install/start/stop/remove

Pylint installation made easier

Tarek Ziadé — Wed, 20 Feb 2008 17:12:00 +0100

EDIT: logilab.pylininstaller is now deprecated, since Logilab made Pylint easy_installable now.

So, just call : easy_install pylint.

I love Pylint.

Correctly configured, it is really useful to raise your code quality. But it can be really painful to install if you are not under a Debian-like or OpenSuse-like system, because of its dependencies that are not namespaced packages (logilab-common and logilab-astng).

In other words, don't try to easy_install the packages that are on PyPI, it will not work.

That's why I have created an egg, called logilab.pylintinstaller, that will let you install it as easy as:
$ easy_install logilab.pylintinstaller

It bundles all dependencies and grabs pylint 0.14, then installs everything.

Thanks to Sylvain Thénault from Logilab for his help on this.

Plone egg developers, please use iw.dist !

Tarek Ziadé — Wed, 20 Feb 2008 14:52:00 +0100

This is my first important step in order to fullfill the Strategic Summit task #7817 (http://dev.plone.org/plone/ticket/7817).

I have released the iw.dist package which is a replacement for register and upload commands. Theses changes will be pushed in Python 2.6 hopefully. Until then, iw.dist provides two new commands, called mregister and mupload, which are acting the same way.

Please, use them instead of register and upload ! So I can get some feedback ;)

It is quite simple to set up, see this page: http://pypi.python.org/pypi/iw.dist

It will not bather you at all since it does what the regular commands do, but are the first step to a tool that will let you upload the eggs to plone.org.

Plone Strategic Summit: Improve releasing procedures for plone.org add-ons

Tarek Ziadé — Wed, 13 Feb 2008 17:51:00 +0100

I have been suggested as a "champion" at the Plone Strategic Summit on this task:

# Improve release procedures for add-ons on plone.org: document a release process, and create release tools for packaging and uploading products from the command line.

This is great, because I have been working a lot in this area in the past month and I think I have a precise idea on what should be done in the Plone community to improve add-ons products visibility and releasing process. I am going to expose here the steps I think we should take, and how to do them, so people can give some feedbacks. Most of them were already explained on this blog on several entries.

PyPI vs Plone.org

The Cheeseshop (PyPI) is now playing an important role in Plone development. Everytime a Plone 3 application is built somewhere in the world, the Cheeseshop is serving hundreds of tarballs and eggs. Since Zope and Plone has been eggified, and since zc.buildout has been used as the standard way to build a Plone application, Plone developers are releasing all their eggs at PyPI.

This releasing process is really convenient, as a package can be uploaded, and shout out in just one command:
$ python setup.py register sdist bdist_egg upload

And an alias can make it even simpler:
$ python setup.py release

The problem is that many Plone.org add-on products pages that used to be up-to-date are not upgraded anymore. So Basically, Plone.org Software Center is dying because of the actual releasing process of eggs..

The SPOF problem

Another issue with the PyPI-centralized development process is that it becomes a Single Point of Failure. In other words, if PyPI is down, all the buildouts out there are stocked, unless you have a up-to date egg cache on your side.

PyPI though, together with distutils, was thaught as a distributed system: you can theorically call register and upload commands to any server that implements the PyPI Apis. But there are no other PyPI-like server yet in the community. The PyPI code is open source for sure, and anyone could take it and run his own PyPI...

PloneSoftwareCenter features

Another risk we have with a PyPI-centric approach is loosing the features that PSC provides at Plone.org. Those are great, and should be used by all add-ons out there. Milestones, bug tracker, etc.. Everything is provided at plone.org for someone to promote and work with his product.

The solution we should take

To avoid the problems mentioned, we need to:
- make PloneSoftwareCenter, therefore Plone.org, PyPI-compatible - make distutils command-line tools able to interact with several PyPI-compatible servers, besides the official one - provide a simple guideline for the Plone community to work with these tools

The steps

PSC

Sidnei has created 2 years ago a branch for PSC with an experimental PyPI support. I have taken this work and continued it on a branch that is almost finished. My goal is to finish it at the Paris Sprint, so PSC will be fully PyPI-compliant. I will soon blog on this to describe the work.

.pypirc and distutils

In order to be able to interact with several PyPI-like server, the .pypirc file need to evolve. I made a patch and a proposal (see: http://wiki.python.org/moin/EnhancedPyPI) I will try to push in the next Python Bug Day in two weeks, so it is integrated in Python 2.6. If it is accepted, I will release a library that implements the same patch, but for Python 2.4 and 2.5, through specific setuptools commands.

guideline

From there, I guess a guideline can be written, explaining :
- how to create a Plone 3 package (through skeletons) - how to release it to both PyPI and plone.org

Another point of interest will be to explain how to deal with several egg servers in a buildout.

Proposed calendar

Mid-March : provide a package to handle the new .pypirc format
End of March : submit the guideline, based on the PSC current branch, and a public instance of the new PSC so people can try it.
End of April : finalize PSC PyPI support at the Paris Sprint, together with Alex Clark, so it is available to Plone.org when it goes Plone 3
After that: submit a guideline on how Plone companies can use PSC to create a private PSC, and work together with PyPI, plone.org and their own PSC, in a buildout environment

I have also proposed an OSCON topic on this, in Portland, in July. So if my talk is accepted, this can be a good place to promote Plone.org's Plone Software Center.

Plone Paris Sprint, 25/27 April

Tarek Ziadé — Wed, 13 Feb 2008 16:10:00 +0100

Hey, fellow Plone/Zope developer, this is a quick reminder if you would like to join us at the Paris sprint in April. There will be a lot of work and fun !

If you want to come, please:
- join the openplans project here : http://www.openplans.org/projects/plone-3-paris-sprint/project-home - send a mail on the project mailing list if you would like to suggest a topic - send me a mail if you are sure to come, so I can add you in the attendee list. If you need a hotel, let me know. Let me also know if you are ok to spare a room.

Hope to see you there !

http://www.openplans.org/projects/plone-3-paris-sprint/project-home

Using RelStorage with Plone 3

Tarek Ziadé — Sat, 02 Feb 2008 23:10:00 +0100

Last week, Shane Hattaway made available RelStorage on svn.zope.org. This product replaces PGStorage and allows you to switch from FileStorage (the Data.fs file) to a Postgresql or Oracle storage for ZODB pickles. ZEO is not needed anymore in that configuration, and each Zope instance calls the same Postgresql server.

According to Shane, RelStorage handles high concurrency better than the standard combination of ZEO and FileStorage. So I have started to make a few tests with it to see how we can use it on some projects.

I have made a few changes to plone.recipe.zope2instance to be able to define a buildout that uses RelStorage, and created a buildout on the collective that runs a Plone 3 over RelStorage.

The buildout sets a relstorage configuration and patches the ZODB 3.7 code. Instead of a file-storage option in the buildout.cfg file, a rel-storage option is added:
rel-storage =

    type postgresql

    dbname zodb

    user postgres

    host localhost

    password postgres

If you want to try it with your Postgres or Oracle server, you can get it from the collective here: http://svn.plone.org/svn/collective/collective.relstorage/trunk/

Notice: the script that patches the ZODB code does a system call instead of using the difflib module, so you need to have the patch program available at the prompt. So I guess it will fail under windows.

Notice 2: The buildout installs psycopg2 egg, but not the oracle one.

Buildout: plone.org and pypi.python.org are acting like SPOF

Tarek Ziadé — Wed, 30 Jan 2008 09:20:00 +0100

Yesterday, plone.org was moved on another server. It was an horrible day for our people here that didn't have a local cache of eggs to build their instances. So plone.org was acting like a Single Point Of Failure(SPOF) for some packages.

A few developers, that are under windows, were even having permission denied errors on their buildout because when a package is badly downloaded is not correctly crushed before a new attempt (I need to add a ticket about this in setuptools tracker I guess).

Anyway, we decided to create a mirror here, (I am buiding it at http://release.ingeniweb.com/ this morning hopefully) to avoid such problems.

This makes me think that zc.buildout should introduce a high-level mirror mechanism in the find-links variable, that would let someone explicitely provide a list of mirror. It could look like this:

find-links =

  http://pypi.python.org/simple |  http://release.ingeniweb.com/pypi-mirror

  http://dist.plone.org |  http://release.ingeniweb.com/plone-dist-mirror

It could be used to switch the find-links values sent to setuptools when the primary url is down by attempting a simple call with a timeout.

Snow sprint report #4 : a new command in zc.buildout + a context-free grammar text generator

Tarek Ziadé — Sun, 27 Jan 2008 11:06:00 +0100

So this was the last sprint day here at the snowsprint, and a lot of work was done to wrap-up some of the tasks. On my side I worked on two topics:
- adding a new describe command to zc.buildout - coding a random-text generator library

Adding a "describe" command to zc.buildout

Godefroid came up with a nice idea about buildouts: when you are working with a recipe, it's sometimes hard to figure out what are the options it takes, which ones are optionals, what are the default, etc.

It means that you have to digg into the code, or get to the PyPI page. Hopefully this page will give you the infos, if the long_description variable was hooked into some reSTructuredText. (see iw.recipe.pound for example).

That make a lot of context changes for the developer, so basically, the idea of the new describe command is to be able to query for a given recipe help. This help will be displayed online as long as the recipe creator fills the Recipe class docstring. We checked with Jim that this would be a good idea, since he wants (and that's good) to keep the recipe as simple as possible (basically, any class with an install and an update commands). Since he liked the idea, we started to code it.

So basically, the command is called like this:
$ bin/buildout describe my.recipes

    my.recipes

        The coolest recipe on Earth.

        Ever.

It deals with recipe versions and takes care of multiple entry points:

   $ bin/buildout describe my.recipes:default my.recipes:second

    my.recipes:default

        The coolest recipe on Earth.

        Ever.

    my.recipes:second

        No description available

This feature looks quite simple, but was a bit tricky to implement, since we had to parse the working set of the current buildout to extract the infos. The version section is also taken care of.

All that work we did together with Godefroid and Dokai is in a branch, waiting for Jim's feedback.

Coding a random-text generator

When we worked on benchmarking Solr versus Plain catalog on the indexing task, we created a small script to generate random text, based on a chomsky algorithm. We were really excited about going deeper in this topic. Both Dokai and I worked on some generators. I have written on my side a Python port of nonsense, and the results were pretty interesting.

Anyway, we started a fun task for the last day with Dokai and Ethan: write a random-text generator library and a grok-based web app on the top of it. I worked on the core part, and we came up with this cool command line scripts that would generate som random text, given a file that would provide structure of sentences, and for each part of the sentence a list of choices.

The command is building the sentences picking the choices randomly. For example this file adapted from nonsense (extract):
[gibberish]

default =

    ${course}

name = college

annoucement =

    The ${university} class "${course}" has been cancelled due to lack of interest.

    Starting next year, incoming freshman at ${university} will be required to take "${course}."

    "${course}" will no longer be offered at ${university} due to lack of interest.

    Due to overwhelming popularity, an additional section of "${course}" will be offered at ${university} next semester.

    Not one single student signed up for ${university}'s "${course}" last semester.

course =

    ${adjective} ${noun} ${suffix}

    ${adjective} ${noun}: ${ending}

    ${adjective} ${noun} And ${adjective} ${noun} ${suffix}

    ${noun} & ${noun} ${suffix}

    ${group1} ${group2} ${life} ${suffix}

    ${group2} ${noun} ${life} ${suffix}

    ${group1} ${group2} ${life} Since {#1800-1970}

    ${group2} ${life}: ${ending}

 event =

    The African Diaspora

    The Harlem Renaissance

    The Civil Rights Movement

    The Italian Renaissance

    Westward Expansion

    Manifest Destiny

    Women's Suffrage

    World War I

    World War II

    The War Of 1812

    The American Revolution

    The French Revolution

    The Russian Revolution

    The American Civil War

    The Spanish-American War

    The Franco-Prussian War

    The JFK Assasination

action =

    Basketweaving

    Aquatic Ballet

    Synchronized Swimming

    Professional Sports

    The ${adjective} Pottery Experience

    Home Economics

    Cardplaying

    Birdwatching

noun =

    Diversity

    Globalism

 ...

will generate random, domain-specific text. A Grok application has been built on the top on this, allowing dynamic creation of such files, and online text generation. Check out Dokai's blog about this during the week, as he will present the Grok part. The code is in a Git repo here: http://repo.or.cz/w/gibberis.ch.git

Thank you Lovely Systems

Thanks to the Lovely team for this sprint !

Snow sprint report #3 : ZopeSkel refactoring

Tarek Ziadé — Tue, 22 Jan 2008 21:27:00 +0100

Today, while the zope instances we have prepared for the benchmarks where suffering from being fed with 40 000 documents, I have worked on a small task I wanted to do some times ago: refactor a bit ZopeSkel and add some tests in it.

I splitted the template objects that were located in separate modules, and added for each one of them a doctest that is running the template. This prevents the template to be broken because when you work in it, it is not obvious. As a matter of fact, at the time I did it, I found one template to be broken, so I think this is going to be useful to prevent regressions.

If you work on this package and change or add a template, please, pretty please, run the following to make sure nothing is broken:

$ cd ZopeSkel

$ python boostrap.py

$ bin/buildout

$ bin/test

Also the important thing to do for now on, is to keep accurate doctests in the docs/ folder. These files are simple to write, as I have added primitives to simplify the work (well, I have reused what Gael did in our skels at Ingeniweb).

For instance, let's have a look at the recipe doctest:

http://dev.plone.org/collective/browser/ZopeSkel/trunk/zopeskel/docs/recipe.txt

It actually launches the paster over the template, and also launches the freshly created recipe's own tests.

The next moves on ZopeSkel I can think of would be to:
- write more detailed doctests. Each one of them could become a recipe on how to use the given template; - gather in a top document all the doctest, to provide a detailed documentation in Zopeskel's frontal README.txt file, which is parsed and displayed at PyPI.

ZopeSkel is a very important product in my opinion, because it insuflates a standard way to write Plone code in the community. For that matter, the recipe template I have added a while ago was improved here at the sprint, and you should check on Dokai's blog about this. He's writing a wrapup about it right now ;)

http://dev.plone.org/collective/browser/ZopeSkel/trunk/zopeskel/docs/recipe.txt

Snow sprint report #2 : benchmarking

Tarek Ziadé — Mon, 21 Jan 2008 14:22:00 +0100

EDIT: The chomsky was somehow limited, and was creating very similar documents. Dokai worked on another text generator that generates more various document. It is based on various file and combine random texts that are quite nice, check it out ! (same place, but the method is called random_text() (I have updated the code extract as well))

Dokai and Tom are working hard on the best way to hook the regular catalog with the Solr utility. I was a bit aside on this task so I didn't catch up with it yet.

Anyway, I have prepared the field in order to compare a pure plone 3 with a solr-enabled one. I wanted to generate a Plone instance with many documents, which content would look realistic.

I found on ASPN a great recipe for a Chomsky-based random text generator: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440546

So I have just bundled it in a script that can be used to generate Plone folders with documents in it. When Dokai and Tom work will be ready, we will use this script to load several thoushands of documents in the catalogs, to start a few benchmarks.

Here's the script (used in an Extension, but straight forward to bundle in a class), you can also download it from here

""" Generates documents with realistic content,

    with a Chomsky random generator

    taken here : http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440546

"""

from Products.CMFCore.utils import getToolByName

import logging

import transaction

leadins = """bunch of lines"""

subjects = """bunch of lines"""

verbs = """bunch of lines""" objects = """bunch of lines"""

import textwrap, random

from itertools import chain, islice, izip

def chomsky(times=1):

    """Chomsky method of generating random text."""

    return ' '.join(chain(random.choice(part).strip()

                          for part

                          in (leadins, subjects, verbs, objects)

                          for i in xrange(times)))

def gen_documents(context, folder, numdocs=10, root_name='doc_'):

    wftool = getToolByName(context, 'portal_workflow')

    for i in range(numdocs):

        root = i

        id_ = root_name + str(root)

        while id_ in folder.objectIds():

            root += 1

            id_ = root_name + str(root)

        desc = chomsky(5)

        title = chomsky(2)

        sub = chomsky(1)

        context.invokeFactory('Document', id_, description=desc, title=title,

                              subject=sub)

        obj = context[id_]

        wftool.doActionFor(obj, 'publish')

        logging.info('created document #%d' % i)

        if i % 100 == 0:

            transaction.savepoint()

def gen_folders(context, numfolders=10, numdocs=1000, root_folder_name='folder_',

                root_name='doc_'):

    wftool = getToolByName(context, 'portal_workflow')

    for i in range(numfolders):

        root = i

        id_ = root_folder_name + str(root)

        while id_ in context.objectIds():

            root += 1

            id_ = root_folder_name + str(root)

        context.invokeFactory('Folder', id_)

        obj = context[id_]

        wftool.doActionFor(obj, 'publish')

        logging.info('created folder #%d' % i)

        gen_documents(obj, obj, numdocs, root_name)

        transaction.savepoint()

def gen_sample(portal):

    gen_folders(portal)      def random_text(data, num_words=100):

    """Source: http://www.physics.cornell.edu/sethna/StatMech/ComputerExercises/RandText"""

    # Read in the file and create a prefix mapping

    words = data.split()

    prefix = {}

    for i in xrange(len(words)-2):

        prefix.setdefault((words[i], words[i+1]), []).append(words[i+2])

    current_pair = random.choice(prefix.keys())

    random_text = current_pair[0] + ' ' + current_pair[1]

    for i in xrange(num_words-2):

        # last two words in document may not have a suffix

        if current_pair not in prefix:

            break

        next = random.choice(prefix[current_pair])

        random_text = random_text + ' ' + next

        current_pair = (current_pair[1], next)

    return random_text

Snow sprint report #1 : indexing

Tarek Ziadé — Sun, 20 Jan 2008 23:25:00 +0100

So we are here in Austria, sprinting on Zope and Plone (thanks to Lovely Systems). I have proposed a task on building an alternative indexer system for Plone. So, we worked with Dokai and Tom on this. Those guys rock, really !

Our goal was to create a plone 3 buildout that provides an out of the box solution.

Background

Let me give you some background about indexing in Zope before presenting our work. The default indexing system is quite effective, as long as your instance is not getting too big. Some years ago, we had to create an alternative indexer for CPS at Nuxeo, that would externalize the catalog because we figured out that :
- 50% of the size of the ZODB was the catalog (I am talking about gigas here) - 50% of the time on object creation was taken by indexing tasks, and was getting quite slow as the instance was growing.

Those values are approximate, but quite near the reality back then (I know some people worked on making indexing better on Zope lately).

Julien then wrote a XML-RPC server that would take care of the indexing tasks and reply to queries. The software behind it was Lucene, together with PyLucene. The overall solution was quite good, beside the pain we had to install it on some specific Linux back then.

Anyway. What did Julien some years ago exists now and is called Solr. I also had some experiences a while ago with Xapian (as Sidnei did too), which is quite efficient too, and easier to use from Python (see here)

Solr, Xapian

So the first task to do was to decide what to use. I called Alan from Enfold Systems because the guys over there have been working on the topic for years.

As a matter of fact, they have created a package for Python that bind a Solr server.
They also have a Plone integration that provides an utility to index content on Solr.
Since the guys are releasing all of this very soon as open source, we decided
to go with this solution for the sprint.

It is not a technological choice (Lucene) because Alan and some guys from
Lemur are actually considering a drop-in replacement for Solr based on Xapian.

In other words, the work done will be compatible with both Lucene and Xapian technologies. Xapian is pretty interesting since it avoids deploying Java ;)

The sprint task

The task was quite "simple" since the Enfold guys did all the hard work :)
So we worked on:
1. a buildout that builds a Solr server and launches it 2. a Plone integration to use Solr seamlessly

The buildout

The buildout done and usable (We tried it under Windows, MacOSX and Debian)
It uses new recipe we wrote:

collective.recipe.ant : build Java softwares using ant
collective.recipe.solrinstance : builds a Solr instance and provide a script to launch it

If you want to try it, here's (roughly) how (comment the blog entry in case of a problem)

$ svn co https://svn.enfoldsystems.com/public/enfold.solr/branches/snowsprint08-buildout buildout

$ cd buildout/plone-3.0.5/

$ python2.4 bootstrap.py

$ bin/buildout -v

$ bin/solr-instance &     <-- launches solr (python bin\solr-instance under Windows

$ bin/instance fg         <-- launches Zope

Then, on Zope, install SolrIntegration in the quick_installer. The next document you will publish will be indexed on Solr side, and searchable with the search box.

The portal_catalog remains though, so it is indexed twice ;) you can empty it to check
Solr is acting right.

Plone integration

The last part we need to work on is to make the SearchableText index 100% Solr based. Whit advices us to create a storage for TextIndexNG so that's where we are heading on (should be done tomorrow hopefully)

We would also like to do some benchmarks to compare the speed and ZODB size. We will
probably use Jmeter for this.

I would like to thank Alan, Leonardo, Sidnei for their work on this area, and for releasing it as open source: I really believe that it will become a great indexing solution for Plone in the next months. I was really waiting for this momentum in indexing in the Plone community.

An installer for a buildout-ready Windows

Tarek Ziadé — Sun, 20 Jan 2008 20:05:00 +0100

When you need to run a buildout under Windows, you have to take care of setting up quite a few things, like installing MinGW and linking it to Python, and setting up a svn command-line client for most buildouts.

We created a simple package that contains everything needed to make your windows buildout-friendly. It is a simple zip file that contains a batch script and third-party installers. When the batch is run, the environment variables are set as well, and win32-compatible buildouts should run without problems from there.

EDIT: the downloads urls have changed (thanks Sasha!)

You can get it here:

http://dl.dropbox.com/u/3265240/python2.4.4-win32.zip

and another version that adds developing tools like vim and tail:

http://dl.dropbox.com/u/3265240/python2.4.4-win32-dev.zip

This zip file is built itself with a buildout that we might publish soon so you can make a custom zip file.

distutils: multiple servers in .pypirc

Tarek Ziadé — Thu, 17 Jan 2008 11:06:00 +0100

Since I am working on PloneSoftwareCenter to make it PyPI-compatible, I have worked on distutils side to make the register and upload command more friendly when an egg has to be registered to several servers.

The problem with the actual .pypirc file is that it won't let you define many username/password for many servers: it is dedicated for one server. In the meantime, you can specify in a command line option which server you want to deal with:
$ python setup.py register -r http://my.server/pypi

But this will take the username/password in .pypirc. So if your username differs from one server to another, it won't work.

I have worked on an enhanced version for this, described here: http://wiki.python.org/moin/EnhancedPyPI

The patch is ready, and comes with new unit tests register and upload commands didn't have yet. The new .pypirc format was shaped with the help and feedback of catalog-sig people, thanks to Martin v. Loewis and Fred Drake and others. I am going to submit it for inclusion today. If it is accepted and integrated we will be able to deal with our eggs like this:
$ python setup.py register sdist upload # goes to PyPI

$ python setup.py register sdist upload -r plone.org # goes to plone.org ;)

The next step is to provide a patch for a permissive trove classifier in PyPI. Then all PyPI-like servers will be able to provide the same service for egg developers, no matter how they deal with classifiers.

sys.setdefaultencoding is evil

Tarek Ziadé — Tue, 08 Jan 2008 10:45:00 +0100

I have recently found some UnicodeDecodeError bugs on some products, that some people couldn't reproduced. The bug was due to a call to a CMF API that was doing a str() over the object, right before using it.

This is perfectly fine in that case, because the object is supposed to be a ZODB id, so it has to be full ASCII.

So the bug looks like this :

>>> id = u'éou'

>>> str(id)

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in

position 0: ordinal not in range(128)

The people that couldn't reproduced it because they use that ugly hack which consists of setting Python's default encoding to utf8:

>>> import sys

>>> sys.setdefaultencoding('utf8')

>>> id = u'éou'

>>> str(id)

'\xc3\xa9ou'

This will be applied to the whole process, and Python itself dynamically removes the method from the module at it first use. From the official doc:
setdefaultencoding(name)

Set the current default string encoding used by the Unicode implementation.

If name does not match any available encoding, LookupError is raised.

This function is only intended to be used by the site module implementation and,

where needed, by sitecustomize. Once used by the site module, it is removed from

the sys module's namespace. New in version 2.0.

I can't find the link back, but I have read once that this built-in was to be removed because it should not be used outside site.py

The problem is that people tend to add a sitecustomize.py in their environment, then work with str() and unicode() calls and forget about doing it right. The result is a major
misused of strings and unicodes and the code created will be buggy on other computers.

So never ever use this in your code. If you have a UnicodeDecodeError it probably means the function is waiting for a string. If you have a UnicodeEncodeError, it should be unicode. In the same way, do not guess the encoding in your code. You should work with one type (str or unicode) and know exactly what is its encoding.

I think this misued is partly due to a lack of warning here: [http://www.diveintopython.org/xml_processing/unicode.html][]

Because that's one of the first page a developer finds when he tries to understand why

he has such bugs.

See a similar entry on the topic 2 years ago here: http://faassen.n--tree.net/blog/view/weblog/2005/08/02/0

Planet and Wordpress buggy title

Tarek Ziadé — Tue, 08 Jan 2008 10:13:00 +0100

I have found out why my entry titles are removed in all planets. Wordpress recently added in their feeds a new tag in each item of the feed:
tarek

That's the one which get caught in feedparser, instead of the item real title. This bug was already noticed and added in the bug tracker: http://code.google.com/p/feedparser/issues/detail?id=83&q=wordpress.

So I guess it's just a matter of time for Lennart, me, and some other people, to appear right in all Planets.

PloneSoftwareCenter: news from the PyPI front

Tarek Ziadé — Mon, 07 Jan 2008 14:44:00 +0100

I have almost finished the work in the pypi branch for PloneSoftwareCenter PyPI support. There are a few things to polish but it works. (90% of the work was already done by Sidnei, so I have done mostly minor refactorings). I guess I'll polish it until Alex Clark merge it into the trunk.

If you want to give it a try, you can use the buildout I have added in the collective here:

[http://svn.plone.org/svn/collective/PloneSoftwareCenter/buildout/branches/pypi/

][]
I also have a running prototype here:

http://products.ingeniweb.com/catalog

were we are trying it. This will be our public PSC instance and will soon contain all our packages and some public packages mirrored or repackaged as eggs.

Current features

The features are:
- support of distutils and setuptools register and upload commands - automatic creation of projects and releases - support of PyPI's simple page, so it can be used by easy_install and zc.buildout

If you want to try it up ask me for a user account.

Trove classification

The current default categories uses PyPI to classify the packages, and everything is hooked to the register command. So when you upload a package, it will appear in the proper categories in the software center.

That said, you can change the categories to manage your own. When a package is uploaded, it will just ignore the unknown categories. I am working on PyPI side so the Cheeseshop itself works the same way.
(hopefully, it will be accepted, because the guys from the catalog team are helping me out in polishing my proposal)

see my document at : http://wiki.python.org/moin/EnhancedPyPI (see Making PyPI permissive for Trove classification)

Dealing with several PyPI-like servers

Last but not least, as a Plone developer, the final goal is to be able to register and upload packages to both PSC and PyPI. This is a bit tricky with the current distutils implementation and I am working on this so it can deal with several servers.

The final form will be to be able to do:
$ python setup.py register sdist upload -r http://example.com/repository # registering and uploading at example.com

$ python setup.py register sdist upload        # registering and uploading at PyPI

There are some default policies in PSC though, to avoid people uploading projects and file to easily: if the user is a simple member, a register command call will create a project and submit it, and the upload command won't work until the project has been accepted and published.

Next steps

I am really excited about having the same standard everywhere, and to be able to deploy our packages in the community through a simple command line:
- at the cheeseshop - in our private software center - in our public software center - hopefully, in plone.org when it goes Plone 3.x

The next steps will be:
- to get some feedback from the Plone community, and build a TODO list with it (I have to collect Wichert remarks from the ML to start to build it) - to polish the code - to add XML-RPC APIs, like PyPI has

[http://svn.plone.org/svn/collective/PloneSoftwareCenter/buildout/branches/pypi/

]: http://svn.plone.org/svn/collective/PloneSoftwareCenter/buildout/branches/pypi/

Plone sprint in Paris, April 25/27

Tarek Ziadé — Thu, 27 Dec 2007 16:19:00 +0100

I'm really excited about this: we are organizing a Plone sprint in Paris in April at Ingeniweb headquarters !

No topics are planned yet because it depends a lot on who will be able to come. This will be done throughout January.

Please join the project and its mailing list if you are interested:

http://www.openplans.org/projects/plone-3-paris-sprint/project-home

And do not hesitate to make some topic proposal on it

http://www.openplans.org/projects/plone-3-paris-sprint/project-home

Is __ prefix considered unpythonic ?

Tarek Ziadé — Wed, 26 Dec 2007 17:19:00 +0100

EDIT: Justus provided in the comments a great link where Jim Fulton argues that __ should be marked deprecated, folllowed by Neal Norwiz and Tim Peters answers. This helps a lot understanding what __ should be used for: http://mail.python.org/pipermail/python-dev/2005-December/058555.html

I am writing on Python OOP best practices and I was wondering what are the best ways to name attributes in classes. My main concern is about the distinction between private and protected attributes.
- private attributes are attributes that cannot be seen or used outside of the class, even buy subclasses. The Python parser calls the name mangling algorithm when it finds them to prevent name collision; - protected attributes are attributes that can be used and seen in subclasses and should not be used outside. Nothing is done on them and they can be used like public attributes.

When should we use them ?

If you read PEP8, it's clearly said that name mangling (using a '__' prefix) is the best way to protect an attribute from beeing accessed or overriden. So it should be used for all class internals that is not intended to be overriden.

But in the biggest open source code bases like Zope or Plone, '__' usage is very uncommon. The simple '_' prefix is often used instead, to mark attributes that are private to the class or to the module. So there are no real distinction between private and protected attributes. It seems that the 'private' concept is not even used, and people often cut their class code in two parts: public and protected.

In other languages (like Delphi) that define protected and private levels though, protected attributes are not used a lot, and people tend to cut their code in private and public parts and make the protected layer as slim as possible.

Practical rules

Based on these remarks, here's a tentative of '__' and '_' prefixes best usages in Python, for the use cases I know :
- use __ with property. since properties cannot use overriden methods and are tied to the class, the methods used with it should always be private; - use __ for methods that works with private attributes. If your methods works for private attributes, make them private too; - use _ on methods when they are clearly intended to be overriden; - use __ for all module functions and variables that are private. A protected level is not needed since a module cannot be overriden.

Following these rules would probably make 90% of class attributes private instead of protected, and change all base code conventions. So I am wondering: am I a bit unpythonic if I try to follow this standard in attribute naming ? My guess is that most base code are not clean enough in that matter. For instance, many of them use both new-style and old-style classes under Python 2.5, which lead to a MRO algorithm that differs depending on the classes !

I would love to hear how you people deal with these conventions.

http://mail.python.org/pipermail/python-dev/2005-December/058555.html%20

PloneSoftwareCenter Christmas mini-sprint

Tarek Ziadé — Wed, 26 Dec 2007 04:06:00 +0100

I made a mini-sprint on PSC for Christmas, since everyone around me was sitting watching christmas movies on TV and trying to digest.

Here's a wrapup for comment, and for upcoming work.

Current branch (pypi)

I've merged Sidnei's work into a new branch, with the current trunk since his work was done 2 years ago. I have made a few changes from his initial implementation:
- the PyPI API is now coded in a browser view instead of a persistent object, since it has no properties to keep at all; - when a release is uploaded, a new release object is created for the given version if it doesn't exists instead of raising an error and asking the user to manually create it inside the PSC; - the doctest was simplified and uses sample tarballs and eggs.

I need to finish up a few things and to add some features such as:
- automatic project creation. When a package is uploaded and no project corresponds to it, a new project is created using the egg name and provided metadata. This will make the PSC acts like the CheeseShop. (an option will be added in PSC to activate/deactivate this feature to prevent automatic creation of projects if not wanted). - trove web service the TROVE.txt file created by Sidnei needs to be replaced by a call to the categories; (see next section) - multiple uploads. Make sure everything works fine when several files are uploaded for one release; - more tests I need to write more tests from various clients and platform to make sure it works good. (by recording setuptools/distutils calls and creating tests with this).

This work should be done this week if everyone is OK with what I have proposed.

About the Trove classification

The Cheeseshop provides a Trove classification (see http://www.python.org/dev/peps/pep-0301) which evolves. For instance the "Django framework" category was added last week IIRC.

Obviously, Plone eggs should follow this classification but when they are uploaded in a PloneSoftwareCenter they might find specific categories defined locally (these categories might be specific to the project). I think we should let people freely define their classifiers in setup.py and let each server take the ones they have in their list.

The problem with Cheeseshop implementation is that it fails silently when a item in the 'classifiers' list doesn't exists on the server side. The package metadata seem to be lost after that. (this looks like a bug to me, I didn't digg the PyPI code yet). I need to ask over distutils-sig about this and see if we can come up with a Cheeseshop that will pick the categories it knows out of the classifier list, and let the other alone. This would allow PSC to deal with extra categories.

Then the PSC will have to implement the trove web services and serve its categories, so the "list-classifiers" option of the register command works.

Until then I guess we can leave the classification settings manual.

About the .pypirc file

This file that is used with the register command is working just for one server and will not allow having several set of login/password. This is not a problem when the login of your plone with your PSC is the same than the one in PyPI. Otherwise it won't work.

Furthermore, this command is using a hardcoded 'pypi' realm if you look at distutils/commands/register.py:
auth.add_password('pypi', host, username, password)

The real solution here is to make distutils evolve so the .pypirc file allows having several login/password for each server, using the host url and the realm (the realm can be queried automatically too). Until then we have to make the PyPI api return a 'pypi' realm on 401 error (and this was done by Sidnei's work).

To avoid maintaining several .pypirc files and forcing the realm on 401 errors though, we should create a new register command, that can work with a enhanced version of the file and allow adding passwords for several hosts. IMHO, the disutils package code is PyPI-centric but was primarly intented to work over any release server, so it has to evolve on that point.

About sdist and bdist_egg

We have discussed in my latest entry about having a new command to upload the package in the PSC:
$ python setup.py plone_upload

The idea is to be able to upload the release to plone.org and to the Cheeseshop in one step. People reacted over this because in my example I used bdist_egg instead of dist for the packaging. I think it's a false debate because it's up to the developer to decide how he releases his work, using a tarball that is compiled by the target system, and/or an egg.

So we can just define an enhanced upload command that replaces the original one, to automate the upload on several servers, and let the developer manually call sdist and/or bdist_egg.

Servers could be picked up by the user at the prompt.

Schedule & tasks

This are my plans in PSF this week and next week:
- finish my current work on the branch, so the basic implementation works; - provide an enhanced version of the register and upload command, for multiple servers uploads. This will be done in a new package, since it's more like a distutils enhancement; - implement the trove webservice using PSC categories.

Ideas on distributing and promoting Plone packages

Tarek Ziadé — Fri, 21 Dec 2007 11:10:00 +0100

Edit: Just after I posted this, I found Sidnei's work, so it is basically what I was thinking of. Good job :)

How to promote Plone packages today ?

Since eggs became a standard in Zope, distribution of Plone packages can be done directly to the Cheeseshop. This is quite nice since anyone can invoke an upload command like this:
$ python setup.py bdist_egg upload

This makes the package uploaded and available to anyone. This also made anyone able to promote his work without having to set up a web site or to study how the community works on that point.

I am pretty new to the Plone community, and I am trying to find the best way to promote Plone packages we do here.

My guess is that the Cheeseshop is perfect for Python package but not enough for Plone packages: in order to promote them, the best place is the plone.org software center. It brings a tracker, a front page and a release folder. In other words, anyone who wants to publish a Plone product that is seen by the whole community has to take care of uploading its packages into plone.org, or setting a link there, and to the cheeseshop.

If you have a better way to promote your Plone products, let me know !

I think this process could be enhanced a little bit through automation.

Making plone.org pypi-compatible

The Pypi index has two main features PloneSoftwareCenter does not afaik:
- it implements a [file_upload method][], that is called by setuptools when the upload command is invoked - it provides package index pages that allow [easy_install][] to look for a package

These two features are very simple, and could be added in Plone Software Center by:
- adding a file_upload view on the products page - providing an index-compatible view (PSC has a DOAP support though)

In other words, that would allow calling the upload command on plone.org like this:
$ python setup.py bdist_egg upload --repository=http://plone.org/products/

Another command could be added in setuptools to distribute a plone package automatically:
$ python setup.py plone_promote

plone_promote would invoke bdist_egg then make an upload on cheeseshop and/or plone.org. In other words, that would allow package developers to promote their work without pain.

Having such feature would also allow people to create their own Pypi-compatible private software center when they deal with private package they want to make available for instance to private project buildouts.

If people think it's a good idea, I am willing to code it in PSC (I made a proposal on the bug tracker).

The Python Papers volume 2, issue 4

Tarek Ziadé — Wed, 19 Dec 2007 08:11:00 +0100

The next issue of The Python Papers (2, 4) is now available, containing my blog entry "eight tips to start with Python", and many interesting articles, like an introduction to Test-Driven Code Generation. This article explains, after an introduction to test-driven development, how to generate skeletons that contains unit tests ready to run (like ZopeSkel in Plone) . Generated test fixture is a real advantage in complex frameworks to avoid developers to drop TDD.

Read it, it's free !

iw.* recipes release festival

Tarek Ziadé — Fri, 14 Dec 2007 10:02:00 +0100

A lot of buildout recipes have been created lately at Ingeniweb, you can list them here: http://pypi.python.org/pypi?%3Aaction=search&term=iw.recipe&submit=search.

Let me present them quickly:
- iw.recipe.cmd : yet another command line recipe. This is useful when you need to perform some pre or post installation steps that don't worth a recipe. - iw.recipe.fetcher : this recipe is a wget-like command, that knows how to downloads a file given by an url. - iw.recipe.fss : this recipe creates folders and configuration file when you use File System Storage in your Plone application. - iw.recipe.squid : automates the configuration of Squid, and is Cache-Fu friendly - iw.recipe.pound : compiles and installs Pound in your buildout, and creates its configuration file. - iw.recipe.subversion : this one is a bit like infrae.subversion but it doesn't make controls of modified files (which can be very long when you update a buildout), and it generates in download-cache when it is given, a tarball of the subversion branch. This allow us to perform offline installation with subversion branches without having to change the cfg files. - iw.recipe.template : this recipe knows how to render a cheetah template like Paster does for instance. It makes it easy to add a few custom files in our buildouts

If you use them please let us know !

Other recipes are beeing created, like iw.recipe.pen, which does like iw.recipe.pound but for Pen. I will blog sometimes about Pen, which can be used instead of Pound when you need an universal load balancer that can work on any platform.

http://pypi.python.org/pypi?:action=search&term=iw.recipe&submit=search.

Using ZopeSkel to raise Plone projects quality

Tarek Ziadé — Fri, 30 Nov 2007 11:05:00 +0100

At Ingeniweb, we have started to define standards for our Plone projects using ZopeSkel. IngeniSkel is a thin layer on the top of ZopeSkel, that was:
- injecting Archetype content within existing products - defining standard tests skeletons for all elements contained in products. - providing other templates like the recipe one, that creates a skeleton for zc.buildout recipes

The injection idea was previously proposed by Martin, and Mustapha started to implement it in a branch. Erik F. has added the archetype content injection yesterday.

It means you can now inject new archetype based content with the Paster directly with ZopeSkel:

$ paster create -t archetype my.package $ cd my.package  $ paster --help

usage: paster [paster_options] COMMAND [command_options]

options:

  ...

Commands:

  create       Create the file layout for a Python distribution

  ...

ZopeSkel local commands:

  addcontent   Adds plone content types to your project  $ paster addcontent --list

Available templates:

  contenttype:  A content type skeleton

  portlet:      A Plone 3 portlet

  view:         A browser view skeleton

  zcmlmeta:     A ZCML meta directive skeleton $ paster addcontent

This is great, now the only thing it misses so we can drop IngeniSkel in favor of this enhanced version of ZopeSkel is generating tests modules on all templates and local commands, and the same way everytime. I started such a work in another branch to backport what we have it and I will propose it.

Why ? Because having tests that are written the same way on all layers of a Plone project is important to:
- automate some QA and documentation tasks - make sure a newcomer won't get lost on how to startup a new piece of code, with the right test fixture. If he takes too much time to prepare the test fixture, he'll probably drop the TDD approach...

Edit: I have merged the the recipe template into ZopeSkel trunk already, as I've been asked to

Controlling Cache headers with zope.testbrowser

Tarek Ziadé — Sun, 11 Nov 2007 10:26:00 +0100

On a Plone website, setting the cache headers with CacheFu or other tools is quite easy, but when you need to check on a given page which headers are inserted, it's not always easy to know what will be called in the stack of rules you have set.

You can look at each page properties in you web browser to check this, but it is a pain and you won't be able to make sure nothing is broken when you change, for example, your CacheFu settings.

So, having a dedicated functional test to control your website cache settings is a good way to avoid regressions. zope.testbrowser allows you to write this on a doctest, so it's easy to gather all your cache headers control in one text file. To install it, you can use Easy Install:

$ easy_install zope.testbrowser

Here's an example of such doctest (cache.txt):

=============

Testing cache

=============

    >>> from zope.testbrowser.browser import Browser

This method is used to print a page headers::

    >>> def headers(url, login=None, password=None):

    ...     b = Browser()

    ...     if login is not None and password is not None:

    ...         b.addHeader('Authorization',

    ...                     'Basic %s:%s' % (login, password))

    ...     b.open(url)

    ...     print b.headers

Let's try on the python front page::

    >>> headers('http://python.org')

    Date: ...

    Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_ssl/2.2.3 OpenSSL/0.9.8c

    Last-Modified: ...

    ETag: "60193-3fd1-b811ca00"

    Accept-Ranges: bytes

    Content-Length: 16337

    Connection: close

    content-type: text/html; charset=utf-8

The page returns an ETag header, which tells the browser if the page has changed.

Let's try Plone's one::

    >>> headers('http://plone.org')

    Server: nginx/0.5.26

    Date: ...

    Connection: close

    Content-Language: en

    X-Cache-Headers-Set-By: CachingPolicyManager: /plone.org/caching_policy_manager

    Expires: ...

    Vary: Accept-Encoding

    Last-Modified: Sun, 04 Dec 2005 12:13:31 GMT

    X-Cache-Rules-Applied: yes

    X-Caching-Rule-Id: frontpage

    Cache-Control: max-age=0, s-maxage=3600, must-revalidate

    X-Header-Set-Id: cache-in-proxy-1-hour

    Content-Length: 44947

    X-Varnish: 783893351 783878475

    Age: 3364

    Via: 1.1 varnish

    Content-Type: text/html;charset=utf-8

    imagetoolbar: no

It has more complex cache information (CacheFu) used by Varnish cache software.

Notice the use of Ellipsis (...) that replaces changing data like Dates. It can be called using a python module that can look like this:

import os

import unittest

import doctest

OPTIONFLAGS = (doctest.REPORT_ONLY_FIRST_FAILURE |

                        doctest.ELLIPSIS |

                        doctest.NORMALIZE_WHITESPACE)

def test_suite():

    return doctest.DocFileSuite(os.path.basename('cache.txt'),

                                         optionflags=OPTIONFLAGS)

if __name__ == '__main__':

    test_suite()

This is not specific to Plone, and can be used on any website. Added to the stack of tests, this helps a lot on making sure the cache settings are doing what they are supposed to.

Scheduling tasks in Windows with PyWin32

Tarek Ziadé — Thu, 01 Nov 2007 12:23:00 +0100

I am posting this small entry because it took me quite a time to compute all informations to programmatically add tasks in Windows using pywin32, so this post should be an helpfull example.

The process to schedule a task is as follows:
- create an instance of the COM TaskScheduler interface - adding a task within the task scheduler - adding a trigger within the task

Here's how it can look in Python, to create daily tasks:
import pythoncom, win32api

import time

from win32com.taskscheduler import taskscheduler

def create_daily_task(name, cmd, hour=None, minute=None):

    """creates a daily task"""

    cmd = cmd.split()

    ts = pythoncom.CoCreateInstance(taskscheduler.CLSID_CTaskScheduler,None,

                                    pythoncom.CLSCTX_INPROC_SERVER,

                                    taskscheduler.IID_ITaskScheduler)

    if '%s.job' % name not in ts.Enum():

        task = ts.NewWorkItem(name)

        task.SetApplicationName(cmd[0])

        task.SetParameters(' '.join(cmd[1:]))

        task.SetPriority(taskscheduler.REALTIME_PRIORITY_CLASS)

        task.SetFlags(taskscheduler.TASK_FLAG_RUN_ONLY_IF_LOGGED_ON)

        task.SetAccountInformation('', None)

        ts.AddWorkItem(name, task)

        run_time = time.localtime(time.time() + 300)

        tr_ind, tr = task.CreateTrigger()

        tt = tr.GetTrigger()

        tt.Flags = 0

        tt.BeginYear = int(time.strftime('%Y', run_time))

        tt.BeginMonth = int(time.strftime('%m', run_time))

        tt.BeginDay = int(time.strftime('%d', run_time))

        if minute is None:

            tt.StartMinute = int(time.strftime('%M', run_time))

        else:

            tt.StartMinute = minute

        if hour is None:

            tt.StartHour = int(time.strftime('%H', run_time))

        else:

            tt.StartHour = hour

        tt.TriggerType = int(taskscheduler.TASK_TIME_TRIGGER_DAILY)

        tr.SetTrigger(tt)

        pf = task.QueryInterface(pythoncom.IID_IPersistFile)

        pf.Save(None,1)

        task.Run()

    else:

        raise KeyError("%s already exists" % name)

    task = ts.Activate(name)

    exit_code, startup_error_code = task.GetExitCode()

    return win32api.FormatMessage(startup_error_code)

You can see an example of usage here, in the iw.win32 package we have started to build, to gather all win32 specific Python things.

iw.recipe.fss : a recipe to install File System Storage

Tarek Ziadé — Tue, 23 Oct 2007 07:56:00 +0200

What is FSS ?

When you need to work with a lot of static files in your Plone website, you should consider using File System Storage (FSS). It's an Archetypes storage that can handle files like PDF, images or small videos. It prevents the growth of the ZODB. It's not like Blobs, because the files are not transactionals. This means you won't have to worry about network performances when you use ZEO: nothing will be copied from a node to another. You just have to use a NFS point to make sure all nodes uses the same files.

The missing part though, was to be able to easily deploy FSS using the standard way.

Deploying FSS

In Plone world, zc.buildout is now the leading project in the deployment area. Everyone should read Martin's tutorial on how to use it. It makes a typical Plone deployable in a matter of minutes with no pain. It is based on a configuration reader that instanciate recipes objects in charge of installing a part of the system. You have recipes for apache, ldap, etc. See existing public recipes at Cheeseshop.

iw.recipe.fss is a recipe that takes care of creating file system folders and the configuration file used by the Product.

To use it, just insert a section in your buildout file (it's called a part in buildout language) that describes each system storage, and the path to the configuration file:
[buildout]

parts:

    ...

    fss

    ...

[fss]

recipe = iw.recipe.fss

conf = ${zopeinstance:location}/etc/plone-filesystemstorage.conf

storages =

    storage_name /site/storage_path directory

See the recipe's README.txt for all options.

That's it. I love buildout.

More recipes to come

There are a lot of recipe available in the Python Index, but we still missing some to perform every kind of deployment. Besides the Plone recipes here are the recipes that I find realy usefull:
- infrae.subversion: clean, simple way, to checkout a piece of code from a Subversion repository. It's a perfect one to create developer buildouts. - zc.recipe.cmmi: the configure-make-make install recipe. Will perform a compilation and local installation) of Makefile compatible packages under BSD/Linux. - zc.recipe.egg: this one is very useful if an egg needs a special environment. For example if it needs a compiled library created with zc.recipe.cmmi.

We have more recipes that are being coded here at Ingeniweb, to cover our needs in deploying Plone instances with buildout. I'll try to blog on them everytime we update the Cheeseshop. If you create recipes yourself I encourage you to share them on Cheeseshop. Sharing recipes, in my humble opinion, is an important thing to do in a community: it helps standards to raise because it shows how people uses software in real infrastructures.

Unobtrusive benchmark and debug of Python applications

Tarek Ziadé — Thu, 18 Oct 2007 21:00:00 +0200

There are many tools available for Python to perform benchmarks and debugging. For example:

Hotshot is bundled in the standard library and provide useful data. Maybe you have to install an extra package on some linux distribution if I recall it correctly, because it's not GPL;
iPython provides a nice interface to perform live debugging, like automatic invocation of pdb on exceptions;
the standard module test provides pystone, that let you benchmark the computer in use before the timed tests. This is helpfull to bench the code on several computers: the measurements can be expressed in in pystones. In other words, you are able to have a reproducable measure of a piece of code and work on the code complexity to make it faster. In reality, any interference can change the results, but this is true as well for time measures.
all big python frameworks are using the logging module, so it's easy to hook in it if extra logging is needed.

But when trying to equip an application in order to find out why some functionalities are slow, or why something goes wrong, it's not always easy to set up precisely what you want to log if something is slow or what your want to hook if a bug appears. The simplest way is to call out all the mentioned tool from the code, but it too obtrusive. Another way is to use decorators.

To perform it, you'll have to:
- get and install iw.quality - create the benchmarking or debugging module

Get and install iw.quality

iw.quality gathers helpers for QA. It has an implementation of the Levenshtein distance discussed earlier, and now a decorator used for benchmarking and debugging purpose. Since it's available in PyPi, you should be able to install it like this:

$ easy_install iw.quality

See setuptools informations if you need to install easy_install itself.

Preparing the benchmark or the debugging

Whether you are about to benchmark or debug your program, you need to list all the places in your code where you need to hook a log or a pdb. Then you can create a specialized python module that can be used when needed. This module will simply decorate the functions you want to work with.

Benchmarking

Here's an example, let's equip sqlalchemy for benchmarking.

benchmarking.py file:

#

# benchmarking queries

#

from iw.quality.decorators import log_time

import sqlalchemy

def logger(msg):

    print msg

simple_logger = log_time(logger=logger)

sqlalchemy.create_engine = simple_logger(sqlalchemy.create_engine)

sqlalchemy.engine.Engine.execute = simple_logger(sqlalchemy.engine.Engine.execute)

The log_time decorator comes with a few parameters, like logger wich is called with the log message. By default it uses logging.info, but you can use your own like in the example. The chosen functions are then decorated.

Let's use it:

>>> import benchmarking      # applies the decorators

>>>from sqlalchemy import *

>>> db = create_engine('sqlite:///:memory:')

log_time::2007-10-18T21:50:52.352037::0.013::function 'create_engine',args: ('sqlite:///:memory:',), kw: {}

>>> db.execute('create table TEST(id int)')log_time::2007-10-18T21:52:13.761085::0.104::function 'execute', args:

(<sqlalchemy.engine.base.Engine object at 0x12e6e90>, 'create tableTEST(id int)'), kw: {}

<sqlalchemy.engine.base.ResultProxy object at 0x12e6ff0>

>>> db.execute('insert into TEST (id) values (1)')

log_time::2007-10-18T21:52:50.265860::0.000::function 'execute', args:(<sqlalchemy.engine.base.Engine object at 0x12e6e90>, 'insert into TEST

(id) values (1)'), kw: {}<sqlalchemy.engine.base.ResultProxy object at 0x12f50d0>

If you need to display more infos on the call, you can use your own formatter instead of the provoded one. Let's extend the benchmark file:

def formatter(execution_time, function, args, kw):

    return '%s = %.3f ms' % (function, execution_time)

simple_logger = log_time(logger=logger, formatter=formatter)

And rerun some code:

>>> from sqlalchemy import *

>>> db = create_engine('sqlite:///:memory:')

<function create_engine at 0x12328b0> = 0.014 ms

You can add a treshold on the function timing, to log only functions that are up to this treshold. This is useful to filter a bit.

Debugging

For debugging purpose, you can use the debug parameter:

def debug(e):

    import pdb

    pdb.set_trace()

simple_logger = log_time(logger=logger, formatter=formatter,

                         debugger=debug)

It will be called in case of an Exception:

from sqlalchemy import *

db = create_engine('gcckc')

--Return--

/Users/tziade/tests/benchmarking.py(14)debug()

(Pdb) c

/Users/tziade/tests/banchmarking.py(10)logger()

(Pdb) c

function create_engine at 0x12328f0> = 3.544 ms

Traceback (most recent call last):...

raise esqlalchemy.exceptions.ArgumentError: Could not parse rfc1738 URL from string 'gcckc'

Conclusion

By using this simple decorator, it's easy to group benchmarking and debugging in a specialized module, and generate custom reports. The best practice is to create a module per each use case. I didn't hook it on Hotshot or other utilities to let people use the tools they like.

Make your code base healthier: the anti-cheater pattern

Tarek Ziadé — Mon, 08 Oct 2007 19:20:00 +0200

I gave a few years ago some courses to college students. They had to write some small C++ programs and send them to me before the end of the course, so I could correct them and give some grades.

They were massively cheating :)

I created a anti-cheat tool to try to find the cheaters, mainly for fun and curiosity. To make it efficient, I tried to understand how people where cheating. They were cutting and pasting pieces of code from various people, and where arranging them so they would look original. They were changing the comments of course, and moving functions around so I wouldn't recognize the class similarities when I was glancing through the code. But when they were cheating, 80% of the code would look similar to the programs they borrowed. When they were composing a new program out of several sources, the atomic bloc was roughly the function: it was most of the time coming for one source. In other words, there were a few students zero (like patient zero in epidemics), that where the providing a function help-yourself copy-n-paste catalog for 30 fellows.

The Levenshtein distance worked great there in finding the cheaters: given two strings, it computes the number of permutations needed to get from one string to the other. A ratio can therefore be calculated between the two strings. Roughly, when its value is equal or up to 0.7, we can consider that the two are quite similar. This can be applied to function code as well.

It was quit funny to use this tool at the end of the course, as I could point the cheaters immediatly. I hurd that some college use such programs now to detect if students are doing plagias in their homeworks.

In software, developers acts the same: in the very same code base, as long as there is more than one developer involved, you will always find the same functions duplicated again and again. Sometimes, the duplication is not done on purpose: it's a natural use case involved by the APIs. In other words, a refactoring is needed to remove duplicate code. This is done most of the time by agile developers that smell the need: why this function is not made generic and moved into the base class ?

So it's a good practice to hunt for duplicates, to make the code smaller, thus more robust. But it's hard to see all duplicates, as it takes a lot of code reviewing time.

That's where the anti-cheater pattern is useful

The pattern can be applied in two steps:
1. Parsing the code and applying a bit of filtering. 2. Calculating the distance, and reporting similarities.

Parsing the code

The first step is to read the code, using the compiler module. This is the cleanest way to extract the functions because the module parses the code and renders an Abstract Syntax Tree (AST), that is browsable without having to import, thus compile the code. Regular expression could work, but would be painful to create. In the AST, each node represents a piece of the program, with a specialized class. For example, a function is a node of the Function class and a list of children that represent the content of the function. compiler also provides a visitor pattern that allows us to set some hook everytime a function, a module, a class or anything else, gets traversed by the parser.

Below is the visitor used to parse the code for our duty:
registered_code = {}

class CleanNode(object):

    def __init__(self, node):

        code = [str(el) for el in node.getChildren()

                if el is not None and not isinstance(el, basestring)]

        # too small

        self.small = len(code) < 5

        self.code = ' '.join(code)

        self.name = node.name

        self.filename = node.filename

        self.key = node.key

        if hasattr(node, 'klass'):

            self.klass = node.klass

class CodeSeeker(object):

    def __init__(self, filename):

        """compiles the AST"""

        self.filename = os.path.realpath(filename)

        self.node = compiler.parseFile(self.filename)

        res = compiler.walk(self.node, self)

    def _key(self, node):

        """calculates an unique key for a node"""

        if hasattr(node, 'klass'):

            return '%s %s.%s:%s' % (node.filename, node.klass,

                                    node.name, node.lineno)

        else:

            return '%s %s:%s' % (node.filename, node.name, node.lineno)

    def _clean(self, node):

        return CleanNode(node)

    def register(self, node):

        """register the node"""

        node.filename = self.filename

        node.key = self._key(node)

        node = self._clean(node)

        if not node.small:

            registered_code[node.key] = node

    #

    # compiler walker APIs

    #

    def visitFunction(self, t):

        self.register(t)

    def visitClass(self, t):

        for subnode in t.getChildren():

            if not subnode.__class__ in  (compiler.ast.Stmt, compiler.ast.Function):

                continue

            for f in subnode.getChildren():

                if f is None or isinstance(f, str):

                    continue

                f.klass = t.name

                self.visit(f)

def register_module(filename):

    """registers a module"""

    CodeSeeker(filename)

def register_folder(folder):

    """walk a folder and register python modules"""

    for root, dirs, files in os.walk(folder):

        if os.path.split(root)[-1] == 'tests':

            continue

        for file in files:

            if file.endswith('.py') and file != 'interface.py':

                register_module(os.path.join(root, file))

Each function, inside and outside classes, are registered in registered_code with a few metadata. There's a few filters as you can see. Some are Plone specific (like the omission of tests folders, and interface.py files), and some removes very small functions (when there's less than 5 nodes in the function, including its name, parameters, etc). This is the playground for our Levenshtein algorithm.

Calculating the distance

Now we can compare each function to each other, and get a ratio. When the value is up to 0.7 we can consider that the code is pretty similar. I have used David Neca's package: http://trific.ath.cx/resources/python/levenshtein/ for this, because it's fast and real simple to use:
from Levenshtein import ratio

def levenshtein(entry1, entry2):

    """returns the ratio"""

    return ratio(entry1, entry2)

Using it over our dictionnary will look like this:
def search_similarities():

    similar = []

    items = registered_code.items()

    done = []

    for key, value in items:

        done.append(key)

        code = str(value.code)

        for key2, value2 in items:

            if key2 == key or key2 in done:

                continue

            code2 = str(value2.code)

            ratio = levenshtein(code, code2)

            if ratio > 0.7:

                similar.append((ratio, value.key, value2.key))

    similar.sort()

    similar.reverse()

    return similar

Of course we could do some caching, and use an iterator to optimize the code (the sorting is not really needed)

Demo in Plone and Zope

Let's run this pattern over Plone and Zope. I have tried it on Plone 3 lib and Zope 3 lib (within Zope 2.10).

For example, addOpenIdPlugin in plone.openid.plugins.oid:
def addOpenIdPlugin(self, id, title='', REQUEST=None):

    """Add a OpenID plugin to a Pluggable Authentication Service.

    """

    p=OpenIdPlugin(id, title)

    self._setObject(p.getId(), p)

    if REQUEST is not None:

        REQUEST["RESPONSE"].redirect("%s/manage_workspace"

                "?manage_tabs_message=OpenID+plugin+added." %

                self.absolute_url())

was trapped to be similar to manage_addSessionPlugin in plone.session.plugins.session:
def manage_addSessionPlugin(dispatcher, id, title=None, path='/', REQUEST=None):

    """Add a session plugin."""

    sp=SessionPlugin(id, title=title, path=path)

    dispatcher._setObject(id, sp)

    if REQUEST is not None:

        REQUEST.RESPONSE.redirect('%s/manage_workspace?'

                               'manage_tabs_message=Session+plugin+created.' %

                               dispatcher.absolute_url())

This tells us that a common API could be written that way (if it doesn't alreay exists):
def addPlugin(container, klass, id, REQUEST=None, **kw):

    """adds a plugin"""

    plugin = klass(id, title=title, **kw)

    container._setObject(plugin.getId(), plugin)

    if REQUEST is not None:

        REQUEST.RESPONSE.redirect(('%s/manage_workspace?manage_tabs_message'

                                   '=Plugin+created') % container.absolute_url())

It would avoid having to write such boiler-plate code.

Another example, in Zope 2.10's zope.app folder. The class SimpleViewClass in zope.app.pagetemplate.simpleviewclass looks very similar to the one found in zope.app.onlinehelp.onlinehelptopic. Mmmm it's exactly the same in fact ! ;)

Last example. In zope.app.authentication.principalfolder, in PrincipalFolder class:
def search(self, query, start=None, batch_size=None):

    """Search through this principal provider."""

    search = query.get('search')

    if search is None:

        return

    search = search.lower()

    n = 1

    for i, value in enumerate(self.values()):

        if (search in value.title.lower() or

            search in value.description.lower() or

            search in value.login.lower()):

            if not ((start is not None and i < start)

                    or (batch_size is not None and n > batch_size)):

                n += 1

                yield self.prefix + value.__name__

This is very similar to GroupFolder's one in zope.app.authentication.groupfolder:
def search(self, query, start=None, batch_size=None):

    """ Search for groups"""

    search = query.get('search')

    if search is not None:

        n = 0

        search = search.lower()

        for i, (id, groupinfo) in enumerate(self.items()):

            if (search in groupinfo.title.lower() or

                (groupinfo.description and

                 search in groupinfo.description.lower())):

                if not ((start is not None and i < start)

                        or

                        (batch_size is not None and n >= batch_size)):

                    n += 1

                    yield self.prefix + id

This should be a common code as well, in the base class.

Conclusion

My example is not very clean though:
- the distance is calculated on the string representation of the tree of each function, and this is probably not optimal; - the ratio treshold was fixed by trying out the pattern on a few source code; - the whole thing is quite slow (I'm not Lundh).

So it's more likely to be a base for a better implementation, or maybe a CheeseCake addon ?

But this works for me at this stage, and let me find duplicate code. I'm thinking of hooking it in a buildbot, to analyze what developers commit and, when a similarity is found, send a mail with a few suggestions.

Testing code that calls third party servers

Tarek Ziadé — Fri, 05 Oct 2007 10:03:00 +0200

One of the fundamentals of unit testing is that the unit test should never depend on any external resource. This is true for all data that might be needed to run the tests, but also for third party servers like LDAP or SQL: they have to be faked.
- LDAP is quite painful to fake. The simplest way is to create all tests with a real LDAP server, then replace it with a class that returns explicit responses for each explicit request. This is managable when the LDAP layer is well done, and easy to patch. - Mailhost is also quite easy to patch in the test fixture, and printing back the mail sent instead of calling the smtplib will allow you to write doctests and unittest without depending on a smtp server through telnet. - For SQL, the simplest way, as long as you use a library that knows how to call different DBs through DBAPI, is to use a flat file DB system. I use sqlalchemy, and patching it in my test fixtures is easy as patching one line: the sqluri. For example, 'mysql://user:pass/server/base' will become **'pysqlite:///path/to/package/tests/data/test.db'. The tests then interact with a SQLite file, and as long as your code uses sqlalchemy APIs, everything should work like if the DB was MySQL or Postgres. The only difference I can think of is the DB unicode settings, that might be different in the production server, so be careful in your doctest when you test strings. - For other third party elements, Mocking can help !

4 tips to keep in mind about the ZODB

Tarek Ziadé — Thu, 04 Oct 2007 02:26:00 +0200

Plone is a great tool, probably one of the best CMS out there, that let you rapidly create an advanced web application. But there are some real pitfalls you might fall into if you are not familiar to the underlying technology. This post provides a list of tips that can help you out. I am not pretending to give you a set of solutions to make your website fast and scalable, but by following those tips, you will think the way experienced Zope developers think when they code an application.
1.
Ask you customer what will be the load.

It's very important to know it when you are starting a Plone
project. That might affect a lot how you are going to work: if the
application needs to deal with hundreds of hits per minute, and
store gigas of datas, you won't organize it the same way than if the
load is ridiculous, like a few hits a day. Project the data load in
the future also: how big will be the site in one month ? one year ?
If the data load grows, you might need to set up a cyclic purge to
make sure the stuff doesn't get huge.

Ask yourself what data will be stored in the ZODB

The ZODB is a great database, and provides a lot of features for CMS: each object (e.g. Document) is automatically saved on each change. This persistent layer, à la Hibernate, is nice. But this has a cost: if hundreds of objects are created by hundreds of users per minute, it won't work. It will technically work but will become very slow because the ZODB has to deal with concurrency changes on the objects. You might argue that this won't be a problem if you take care of who changes the data and how, but you won't be able to deal with how base objects work in the ZODB. For example, BTrees objects, that are supposed to be fast and to be able to deal with many items, are not really scalable on writes: on my Intel MacBook, on a BTreeFolder containing 1000 items, running 4 concurrent threads that are creating objects in a pace of 300 ms, will generate conflict error on 10% of the requests. You should really ask yourself if you need ZODB features on some data. Maybe they will fit better in a SQL database if they don't need versionning or sophisticated workflows.

The ZCatalog in its classical shape, is also a particular case. It can weight 40% of the ZODB total size, so if you index a lot of things, and a lot of features on your websites are based on queries, it's maybe a good idea to used a specialized database, like Xapian or Lucene. It's faster, and it won't make your ZODB grow. Hey, why indexes are stored in the ZODB anyway ? 3.
Don't hide your code behind caches

SQUID, memcached, CacheFu. All those tools are wonderful and mandatory when you put your site in production. But you should not hide your code behind them : that's the best way to code a crappy, badly architectured application. You should take care on how your application scales and on your code effectiveness before you think about caching. Be careful about the complexity of each of your function, and how they scale. 4.
Be careful of the pound/ZEO mirage

Making a cluster of ZEO nodes will not make your application faster, it will just raise the number of concurrent threads your application can handle simultaneously. Each node needs to be synchronized behind the scene everytime a data is changed. So the more nodes you have the more network traffic you will get. This can damage the overall performance of the application as well.

There's a lot of work going on in this area. I am looking forward for what people will do on this sprint for example: http://plone.org/events/sprints/copenhagen-performance-sprint

Extending setuptools: adding a new command

Tarek Ziadé — Sun, 30 Sep 2007 01:51:00 +0200

Before deploying a package with python setup.py install, it's a good idea to launch the tests with python setup.py test.

This command can be used as well to quickly launch tests within a package that is being developed. Since setuptools can be extended, other commands can be added to be launched from the setup script, while you work in your package.

I have created for example a qa command that launches pyflakes over the package code, to make sure I don't leave unused import. I could have used a direct pyflakes call, but my QAs test are going to grow so keeping the QA script details under a python setup.py qa call is a good practice. This will also make buildbot integration easier, as I can check for package QA through an unified serie of calls, that plays with the package setup.py script.

Commands are simple class that derives from setuptools.Command, and define some minimum elements, which are:
- description: describe the command - user_options: a list of options - initialize_options(): called at startup - finalize_options(): called at the end - run(): called to run the command

The setuptools doc is still empty about subclassing Command, but a minimal class will look like this:

 class MyCommand(Command):

     """setuptools Command"""

     description = "run my command"

     user_options = tuple()

    def initialize_options(self):

         """init options"""

         pass

     def finalize_options(self):

         """finalize options"""

         pass

     def run(self):

         """runner"""

         XXX DO THE JOB HERE

The class can then be hook as a command, using an entry point in its setup.py file:

 setup(

     # ...

     entry_points = {

     "distutils.commands": [

     "my_command = mypackage.some_module:MyCommand"]}

 )

This will add an entry point when the package is installed, so you can run your new command this way:

 python setup.py my_command

You can give a try of my qa example by installing my eggchecker package:

 easy_install http://programmation-python.org/pycommunity/eggchecker-0.1.tgz

That's merely a draft, but will show you how pyflakes is launched within the setup.py script.

A co-server for Zope

Tarek Ziadé — Fri, 28 Sep 2007 08:13:00 +0200

Zasync

A few years ago, when we hit with CPS on some big customers intranet scalability problems, we started to use ZAsync in order to perform some tasks in the background. That improved a lot the application overall performance. What ZAsync does is recording in BTrees within the ZODB tasks to perform, let's say Python scripts to simplify. Then a twisted client that runs independantly opens the ZODB to read the BTree and find the task to perform. It acts like another Zope thread in some ways. But there's something I never understood:

Why the job queue is stored in the ZODB database ?

When we talk about scalability, most of time, the infrastructure is more complex than a simple ZEO. It has Apaches, smtps, load balancers all over the place. It has cron tasks to perform a variety of things, link sending mails, creating images, or anything that can be done in the background. It has most of the time other piece of software that perform other things. Having a co-server that gives Zope code the ability to perform background tasks is good.

Having a co-server that gives any software the ability to program a task is better

Many applications, many different Zope instances, can benefit from a centralized task manager.

Quartz

In Java world, I have used a server called Quartz. It is an independant task manager, where you can register tasks and perform jobs, given a timing. It's like a smart cron. Using the beans technology, it can run code independantly, or run it within a Java Server application's context.

Why don't we have such a software in Python ?

Maybe we do, but I have never found it, so i ported part of the idea to Python in a tool called TaskManager, that I use for example on fr.luvdit.com which is a Django application. It sends mails, calculates neighbourhoods, etc.. Maybe I should release it but that's a packaging work I didn't find the time to do. Any piece of Python software can register itself as a task, in order to provide a service. The jobs are stored into a SQL Database, that is opened through an API by all the clients that want to perform a task, and by the co-server that reads the queue and actually perform the tasks. It has three queue in fact, for different priorities. The client-side APIs are really simple and are nothing but SQL queries.

lovely.remotetask

Back to Zope. Lovely systems works on a Zope 3 tool, which seems to be working a bit like ZAsync: it stills stores the tasks in the ZODB, but dedicates a Zope application to work as a web service provider if I understood well. It's the way to go in term of infrastructure but I think that it's overkill to use a Zope instance for that.

Why do we need to deploy a whole Zope stack to have a co-server ?

A dedicated, pure Python application, using a SQL database, fits better because several task runners can work in the same queue, to create a real producer-consumers queue. In their need to perform tasks on various platforms, having a centralized job queue and several executors is more scalable because the producer doesn't deal with several co-servers.

Furthermore, the XML-RPC layer is not a necessity, and not as robust as SQL: if the co-server is down, the Zope server cannot send jobs anymore, or check for job states and get them. Working with a SQL table prevent from this. You might argue that this is the worst scenario, but by experience, the more application servers an infrastructure has, the more potential point of failure you get. You might argue that the SQL server might go down as well, but it's not a code stack, and just holds data to be processed: all the functionalities, thus the weaknesses, are on the co-server side. You might also argue that it makes the solution Python-dependant, but it would be deadly simple to provide a client for another language.

Anyway, using the ZODB to store such things and a Zope to play with them is a small mistake in my humble opinion, even if it's based on PersistentQueue, which looks pretty robust. Let's keep this kind of database do what it was meant for: storing persistent objects that are publishable.

What I would love to have

The perfect co-server that I can think of, would be an independant Python software, like TaskManager that would look like this:
------- ------- <-> co-server instance 1 / win32

         | zope  | <->  | sqldb | <-> co-server instance 2 / linux

          -------        -------  <-> co-server instance 3 / linux

                           ^

 ----------------          |

| another server |<--------

 ----------------

sqldb is a database that store jobs;
each arrow is provided by a python API, that knows how to interact with the database;
a co-server is an independant, pure Python runner, that picks up some work into the DB;
each co-server instance is able to perform tasks, that are provided through a plugin system.
for zope-dependant tasks, a generic task provides an entry point to execute code through XML-RPC calls or through a direct ZODB opening to avoid eating a thread (eg à la ZAsync);

OK, this is exactly Quartz :)

In the last five years, most of the scalability problems I bumped into, were resolved by a good practice: let's be less Zope-centric when we talk about infrastructure.

I would be pleased to have a few comments from Lovely guys on this topic, and I thank them for their latest post, that helps a lot the community to think about scalable solutions for Zope.

"lovely-remotetask"

Using JMeter as a functional test tool

Tarek Ziadé — Fri, 28 Sep 2007 08:05:00 +0200

Today I made an audit on some customer intranet, and I used JMeter to perform stress tests. This tool is awesome, as you can get a whole lot of live statistics, and create a powerfull, distributed stress campaign.

There are some features to control that the output of HTTP calls are right, with simple but sufficient patterns (the output contains, the output doesn't contains, the header has.., etc..) and regular expressions can be used.

A high-level functional test is really nothing much more than that: it performs a user story and check for the result. Ok, maybe some tool like Selenium have more features, but they are not essential ones, and JMeter brings some better things.

What brings JMeter beside functionnal testing are:
- a powerfull reporting tool - the ability to stress-load your application with the user stories

A comparable tool, less powerfull though, is ben's funkload.

-> Creating you app through JMeter will give you the opportunity to tune it without extra work.

It won't make me drop zope.testbrowser tests, because those are merged within my code and explain how it works in doctests, but it will surely make my customers feel better with JMeter reporting capabilities, and myself calmer with its performance analysis:

"Hey, look at the screen, that's your functionality #123 running right now, and you can see its performance through this live performance graph"

[![graph_results.png][]][]

[![graph_results.png][]]: http://tarekziade.files.wordpress.com/2007/09/graph_results.png "graph_results.png"

Eight tips to start with Python

Tarek Ziadé — Mon, 24 Sep 2007 07:56:00 +0200

A friend of mine is starting Python. I tried to sum up some tips for him, that may be useful to others. Don't hesitate to comment it if you think something important is missing.

Get the best online documentation.

There are a few online documentation you must read:
- the official tutorial, that gives you a quite complete overview of Python; - the standard library module index. You can download it to simplify the search through greps. This is the documentation you get through the help command in the prompt. - Active State's Python Cookbook. There are thousands of code snippets that are created, ranked, categorized and commented by developers. - Dive Into Python online book, that makes you discover Python features through well thought examples.
Read PyCon, EuroPython and Pycon UK wrapups and slides.

They are the three main Python events, and a lot of things are happening there. You'll learn a lot by reading the talks slides. If you can go there, it's even better: sprints, bird of feathers and lighting talks are organized. To convince your boss to send you there, you could make a talk proposal "My first steps in Python" ;) 3.
Suscribe to the right feeds.
The mainstream is Planet Python. It gathers most of the blogs out there, so it is the best place to start.
- Reddit's Python is a great place to get the hot topics.
- Pythonware's Daily Python URL. Human-filtered feed. It used to provide several dozains of links per week, but it seems to have slowed down, and provides a few links a week now. I think it's better this way.
Learn and use the rising standards.

There are a few libraries that have a deep impact on the way people write and distribute their work:
- setuptools: helpers to build and distribute your code eggs. A public repository à la Perl's CPAN called Cheeseshop is wired with this library so people can distribute their code there. It's one of the major innovation of last years in Python world in my opinion. - sqlalchemy: The ORM that is now used by the majority of Python frameworks. Its flexibility is impressive. I think there is no equivalent tool in any other language (please let me know if there is); - Python paster. This tool allows you to create templates that can be used to generate skeletons for your code. It is used by many web frameworks to provide people a simple way to generate a standardized boiler-plate code canvas when they start up something. This is done in Java for quite a long time (you cannot do without it in Java, otherwise it would take you years to write any program ;)), and tools like PyDev and Eclipse would provide the canvas to do similar things. But the paster is independant from any IDE; - reStructuredText: learn how to use it. It's our LaTeX. Your code documentation should use it.
Ask for help. The three places you can get some help are:
the mailing list
1. the irc channel #python on freenode.
2. the tutor mailing list. Mihai Campean says: "This is a list specifically for those new to Python and those interested in helping people learn the language, and the atmosphere is very friendly. It’s probably a better place to start than python-list, in my opinion"
There are some talented guys that dedicate their free time to help newcomers. 6.
Try to adapt your successfull code patterns.

When I started Python, I tried to adapt what I used to do with the tool I mastered then (Delphi). Since There should be one-- and preferably only one --obvious way to do it. (try import this in a prompt), that helped me a lot to learn and understand all the subtles of Python on use cases I mastered.
The most pleasant thing about it is that you quickly drop all Python books and guide to work with the language, unlike Java for example, where you need to keep many reference books on your desk. 7.
Share on your experience and participate !.

A newcomer (yeah! fresh blood!) experience is a highly valuable material for the language advocacy: the discovering state of mind sometime reveals weaknesses or absurdities experienced users don't see anymore. Furthermore, fresh new ideas are often brought by people that comes from other communities. If you feel that something is absurd, unclear or wrong, you should start a thread on the language mailing list. If you have an idea onany kind of enhancement, maybe it worth a Python Enhancement Proposal. 8.
Watch what is being done in Python 3, PyPy and web frameworks

Python 3 is the next version of Python, PyPy is Python written in Python. Web frameworks like Django or Zope are large Python codebases. These three sub-communities have something in common: they form the R&D of the language.
Zope for example, has enhanced a lot setuptools and doctest through a massive feedback. Keeping an eye on them even if you don't use them will make you live and understand what rises in the language.
PyPy is an amazing project. Even if you don't understand everything (Python in Python ? what the... ;)), seeing one of Armin Ringo talks will give you an instructive high level view of Python. Now for Python 3, even if you cannot read and understand all threads in the dedicated mailing list, keeping an eye of Guido's wrapups and thread subjects will help you to do the jump on P3k, and probably make your Python 2 code look nicer.

How to generate proper reStructuredText titles with Python Paste

Tarek Ziadé — Fri, 21 Sep 2007 09:54:00 +0200

Python Paste and ZopeSkel are just great, if you don't know them and you are working on Zope packages, you should really take the time to look at them.
- Python Paste provides, besides other features, a engine to generate any kind of package structures based on templates. The templates can be written in Cheetah syntax; - ZopeSkel is a serie of templates that helps a Zope and/or Plone developer to start a Python, Zope or Plone package, taking care of all the boiler plate code generation, to make sure the packages are done in a egg-compatible, standard way.

Basing my work on ZopeSkel, I have started to create a serie of custom templates to speed up and simplify package coding bootstraps. The documentation in these packages are in reStructuredText, and I bumped into a small problem: when you create for example a "README.txt" in your template, which has a title that uses the project name, for example:
======================

$project documentation

======================

blablabla

the result may vary depending on the variable value:
======================

kool documentation

======================

blablabla

or even:
======================

i_like_long_names_for_packages documentation

======================

blablabla

It's ugly, and the reST file won't compile if you try to generate html or PDF with docutils.

Since Paste uses Cheetah, we can fix this, by calculating the length of underlines:
#repeat $len($project) + 13

=#slurp

#end repeat

$project documentation

#repeat $len($project) + 13

=#slurp

#end repeat

blablabla

This will create the proper length. You might argue it's a detail, but that saves me almost ten seconds for each new package now ;)

To blob or not to blob ?

Tarek Ziadé — Fri, 14 Sep 2007 16:08:00 +0200

Back to the storing discussion: at this time we have a quite complete tool to handle big files in Plone: File System Storage (FSS).

FSS Current features

The idea of this product is to provide a storage with a backup mechanism, to be able to do some restore. It can handle big file and store them on server-side with several strategies:
- Flat storage: All field values are stored in a flat structure. This strategy is the default one. - Directory storage strategy: All field values are stored in a directory structure. - Site storage strategy 1: All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure. - Site storage strategy 2: All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.

More information on this here : FSS Strategies

A Blob Strategy ?

ZODB Blob, that are beeing integrated into Zope provide similar features: it stores on the filesystem the file and provide access from the ZODB. To configure such a storage, the zodb_db section of zope.conf would look like this:

<zodb_db main>

    # Main FileStorage database

    <blobstorage>

      blob-dir $INSTANCE/var/blobs

      <filestorage base="1">

        path $INSTANCE/var/Data.fs

      </filestorage>

    </blobstorage>

    mount-point /

</zodb_db>

This tells Zope at startup to create a blob-compatible storage and store blobs in the $INSTANCE/var/blobs folder. Then, the subsection provide the regular Data.fs.

Using this kind of configuration will let us use ZODB.blob APIs in the code. To give a shot on blobs, I have created a new strategy on FSS, that uses blobs. A base class handles blob readings and writings:

class ImplicitBlob(Implicit, blob.Blob):

    def __init__(self, title='', path='',

                 name='', mimetype='text/plain'):

        """stores blob metadata"""

        blob.Blob.__init__(self)

        self.title = title

        self.path = path

        self.mimetype = mimetype

        self.name = name

    def get_size(self):

        """Return the size of the blob."""

        file = self.open('r')

        try:

            file.seek(0, 2)

            result = file.tell()

        finally:

            file.close()

        return result

    def updateMetadata(self, **kwargs):

        """fills the metadata"""

        for key, value in kwargs.items():

            if key in self.__dict__:

                setattr(self, key, value)

    def writeValue(self, value):

        """fills the blob"""

        file = self.open('w')

        try:

            file.write(value)

        finally:

            file.close()

    def getValue(self):

        """returns the blob whole content"""

        file = self.open('r')

        try:

            return file.read()

        finally:

            file.close()

The real version can be found in the collective repository in a branch. This Zope2-style code is not what I would have done in a new product, but my goal was to quickly add the feature in FSS to make some tries.

It took me quite a time though, to set up the test fixture to be sure to run the tests over a BlobStorage instead of a DemoStorage. You must create a custom_zodb.py file into your tests folder and make sure it's called before the TestCase imports. Here's my file:

from ZODB.FileStorage.FileStorage import FileStorage

from ZODB.MappingStorage import MappingStorage

from ZODB.blob import BlobStorage

from tempfile import mkdtemp

base_storage = MappingStorage("test")

blob_dir = mkdtemp()

Storage = BlobStorage(blob_dir, base_storage)

(Thanks ZODB guys for using doctests in your code, that helped me much)

Once the ImplicitBlob is made available, I just addded a new strategy in FSS that uses it this way:
- the tool becomes a BTreeFolder2 object and holds ImplicitBlobinstances in it; - each blob id is the file uid; - the strategy knows how to return a File object;

Now I can add FSSItem instances that are stored into blobs. And guess what, it seems to work ;)

What's next ?

Using blobs seems to be the future of FSS, because it has all the features needed to store files. Furthermore blob are transactional, and this is quite a difference with FSS's regular file storages, because in that case, it's not necessary anymore to set up a NFS to be ZEO-compatible. But all FSS strategies have use cases that we need to keep. For example, the Site Storage Strategy is really sweet: user can find back their file on the filesystem with the same name, etc. Even though we know this is not really important technically speaking, it can be reassuring for the customer.

For large sites though, I wouldn't use blobs and would set up a specialized co-server. Tramline fills the bill and we are thinking about providing a tramline-friendly strategy in FSS, in order to provide direct access to the files.

But all of this is a work in process, and the blob strategy still needs a lot of testing.

How to handle large files in Plone ?

Tarek Ziadé — Mon, 10 Sep 2007 08:42:00 +0200

The first time I had to help out a customer (a fifty year old lady that had no interest whatsoever about computers) on a Plone-based intranet, I had to explain to her why she couldn't upload really big files.

That was really tough, as she couldn't understand why the system had such a limitation. I think she was right. If we drop the technical point of view, a sophisticated system such as Plone should provide a transparent way to upload big files and handle them smoothly on the server side, so the user doesn't feel any difference. Think about it: what's the functional difference between a big file and a small file ? Merely none, except that it's longer to put or to get.

Uploading

The first problem when we deal with big files is the upload time. Browser doesn't provide any feedback, unless you install some Firefox extension or you use a dedicated protocol, such as FTP. In a web interface, there's nothing but an animated logo that is moving around just to say: "Hey, I am not dead !". Ruby On Rails guys came up with a great feature on this topic: they added in their publiser a few apis that would let client-side Javascript:
- display a progress bar; - cancel the upload at anytime.

I am not a technical guru on HTTP protocols but I am really wondering why this is not already available in Gmail... I tried myself to implement it a while ago on Zope 3 publisher, and came up with something that was working, but never had the time to polish it.

The idea is to maintain on server-side, for each ongoing uploads, some infos that can be read by the client, and a cancel API. The only important point is to provide a secure mechanism so no other client can read the infos of another client. The client-side JS code then can call asynchronously the server to display feedback, and provide a cancel button.

At this point, we may consider that it is better to use other piece of software, like Tramline, or some dedicated Apache plugin. But in a marketing point of view, I think it's really important to be able to provide this feature with an out-of-the-box Plone.

Server-side handling

OK, so our file is uploaded. My lady at the office now complains that the system is getting very slow. No wonder, her ZODB weights now several gigas... Again, functionaly speaking, that's a non-sense. Plone should be able to work with these big files without making the ZODB so heavy. The ZODB is a great thing for light objects, but was not meant to hold big chunk of data. A specific thing has to be done. In Java world, they have an advanced data storage for Content Managment System, called JSR-170, and implemented by the JCR. In this system, a tree of folders is maintained and big files are stored in blobs (Binar Large OBjectS), keeping the whole thing scalable, no matter what the users store in it.

In Zope, we don't have a lot of solutions, there's the File System Storage (FSS) product (and a few similar products) in the Plone Collective, and that's about it. The FSS provides a smart proxy over the file system, that let the ZODB breathe and the user handle his big files. But this is an extra product, created to provide a really missing core feature.

The 3.8 version of ZODB though, providen this feature now, through blobs. On Plone side, it seems that the latest Plone4Artists sprint in Boston has boosted the work in the topic and blobs will be available in Plone through ATBlobField.

This is great news, and I hope we will soon be able to make all files uses this feature transparently (big file, small file, what the difference ? they should all be stored out of the ZODB)

Downloading

One of my colleague had a tough issue last week on big files. The File System Storage was not acting right when the user was downloading PDF files from the Adobe Browser plugin: the file iterator internally provided by the Zope publisher was not acting right because it doesn't support the range feature. This HTTP feature used by the plugin make it possible to get pieces of files in non-sequential orders. Though, it's not possible with an iterator to provide pieces of files without having to rewind. So iterating is of no use there. The solution would be to provide a smart system that knows who asked the file and how, to smartly instanciate an iterator or a regular accessor, maybe with a cache system for called ranges. I need to dig on ZODB blobs, maybe they already provide such a feature. If you read this and know the answer, please let me know...

Conclusion

There's still a lot of work in the topic, and I think the Plone4Artists project is doing a great job to enhance things. The three topics to work on are:
- cancelable upload with live feedback; - transparent use of blobs to store files; - smart downloading capabilities.

This could make a great sprint I guess, to create some kind of Plone-FSS-NG, so my Lady can upload her movies in her Plone.

Plone 3 UI : omg !

Tarek Ziadé — Sat, 08 Sep 2007 08:10:00 +0200

The last time I *really used* Plone was back in the days I used to have a website called "zopeur.org", a few years ago. I used to work then with Plone 1.something, and was quite impressed by its features, but disliked a bit its interface.

I worked then at Nuxeo a few years with CPS, trying to enhance its interface through JS and other tools, until they quit Zope. Now, a year after, I'm back to Plone for my new work, and I must say that I am really impressed by the new Plone user interface.

The guys did a great work enhancing the user experience, through clever layouts and a smooth integration of KSS, a JS framework. I worked a little bit on it with Godefroid when it started but nothing compared to what it became: simple, yet powerful.

I didn't write to much on this blog lately, because it's hard to write for two blogs (one in french, one in english) but i'll try to do it regularly for now one, to talk about my Plone experience, since it's now becoming my every day work. I guess my next posts will focus on the tools I am discovering, and all those things that have evolved a lot in the past 4 years, focusing onthe UI of course, but also on Zope 3 integration, since I took a little bit part in its development.

The manual that was published yesterday is a good starting point for me: http://plone.org/documentation/manual/plone-3-user-manual

Indexation service with Xapian

Tarek Ziadé — Tue, 12 Jun 2007 15:39:00 +0200

In a previous post on my french Blog, I was explaining how to use Xapian from its Python binding, to index text content for documents. The idea was to use the indexing part of Xapian and not to store any data in the documents.When I started to hook my code into my Django application, I realized it was not very robust, because I had to open only one Xapian connector to index content. This was not easy to do and depends a lot on how the web framework loads the code. Furthermore, the website had to take care of launching threads to avoid waiting for the indexation to be finished.

I rewrote the tool to avoid these issues, by decoupling indexation and queries: when the website needs an indexation, it writes the data into a special table in a SQL database. On the other side, a thread in a separated process (eg a worker) reads this table and launch indexations when it finds lines in it. This producers-consumer pattern makes the solution very fast and simple.

It is better to perform this kind of pattern than a solution like a live server, based for example on XML-RPC, because it is more robust:
- if any part of the application crashes, the work in process is not lost in the SQL database; - if the indexer program crashes, the website can still query the Xapian database.

Queries are made with read-only Xapian connectors, which can be used concurrently with no problems.

These were the changes I made:
- A sqlite database is used to store indexation queries in a table (since it's based on SQLALchemy, postgres or mysql can be used as well); - a thread scans the table to pick its work and delete rows when its done; - the search API launches standalone and read-only Xapian connectors to perform each search.

When documents are modified and added on the website, a query is made on the database. This is done through the Django event framework wich allows to hook code on events like "A document was created". (explained here: http://www.mercurytide.com/whitepapers/django-signals/)

The package I wrote can be used this way:
- The thread in charge of indexations is launched via the run.py script - the searcher and indexer modules can be used to work with the API

Here's the complete doctest of my package, which provide examples on how to use these API:

{style="border:1px solid black;background-color:#efefef;font-size:10pt;padding:4px;"}

=======

indexer

=======

The indexer provides:

- client-side modules : API for client to ask for indexations and query the

Xapian database. When an indexation is asked, it is stored in a sql

database;

- server-side application: a standalone thread that indexes what has been

asked by reading the sql database.

Let's import the modules used by the client-side::

>>> import indexer

>>> import searcher

Let's reset the SQL DB first::

>>> indexer.reset()

Let's also reset the Xapian DB::

>>> from xapindexer import force_reset

>>> force_reset()

The Xapian DB should be empty now::

>>> searcher.corpus_size()

0

Indexation

==========

Each indexable content has a unique id, and a text to index::

>>> uid = '1'

>>> text = 'my taylor is not rich anymore'

Let's index it::

>>> indexer.index_document(uid, text)

Another one::

>>> indexer.index_document('2', 'pluto is a dog')

Let's start the worker that is in charge of asynchronous indexation::

>>> from xapindexer import start_server

>>> start_server()

Let's wait a bit so the worker has the time to read the SQL Database

and do the work::

>>> import time

>>> while indexer.is_working():

...     time.sleep(0.2)

`is_working` looks in the SQL DB if there is some work left.

The Xapian DB has two documents now::

>>> searcher.corpus_size()

2

Searching

=========

Let's search now, with `searcher`. Operator is AND by default::

>>> res = searcher.search('rich')

>>> list(res)

['1']

>>> res = searcher.search('pluto')

>>> list(res)

['2']

>>> res = searcher.search('dog')

>>> list(res)

['2']

>>> res = searcher.search('rich dog')

>>> list(res)

[]

Or operator::

>>> res = searcher.search('rich dog', or_=True)

>>> res = list(res)

>>> res.sort()

>>> res

['1', '2']

We have an API to detect if a document is present::

>>> searcher.document_exists('2')

True

>>> searcher.document_exists('ttt')

False

And another one to retrieve indexed terms::

>>> list(searcher.document_terms('2'))

['dog', 'is', 'pluto']

Reindexation

============

The document can also be reindexed::

>>> indexer.index_document('2', 'pluto is a cat')

>>> indexer.work_in_process()

([2], [])

Let's wait a bit::

>>> while indexer.is_working():

...     time.sleep(0.2)

Let's make sure the document has been reindexed::

>>> list(searcher.document_terms('2'))

['cat', 'is', 'pluto']

Then check the indexation has changed::

>>> res = searcher.search('rich dog', or_=True)

>>> list(res)

['1']

Or deleted::

>>> res = searcher.search('pluto')

>>> list(res)

['2']

>>> indexer.delete_document('2')

>>> while indexer.is_working():

...     time.sleep(0.2)

>>> res = searcher.search('pluto')

>>> list(res)

[]

The complete code is here : http://hg.programmation-python.org/browser/xap

It is not packaged yet, but can be used to provide an indexation service for a website or any other application.

RuPy is coming up - 14/15 april

Tarek Ziadé — Fri, 06 Apr 2007 22:46:00 +0200

This conference is a great idea: during two days, Python and Ruby experts will gather in Poznan, 14/15 april, to talk about their favorite language !

On Python side we'll have some speaks about:
- TurboGears - PyPy - Django - IronPython - many other interesting topics

On Ruby side, well..just take a look here: [http://rupy.wmid.amu.edu.pl/?page_id=15][], as I cannot say anything about Ruby (yet ;))

I'll have the chance to present a topic about Agile Documentation in Python, which heavily relies on my previous PyCon tutorial, but will focus on presenting how we, Python coders, play with doctests and enjoy it. (doctests, reST, Python, Beer = yepee). I have a few days left to digg on Ruby side to see how they work with tests and documenting, to conclude my slides.

There's something I don't understand though, why using Ruby ? Python is way cooler ! ... ;)

Anyway, I am looking forward to talk to people from the Ruby community to get an insight, and to see Poland.

dzień dobry !

"RuPy"

Slides from Pycon Tutorial

Tarek Ziadé — Sat, 24 Feb 2007 16:18:00 +0100

Here are the slides I have made for the PyCon tutorial:
- S5: http://programmation-python.org/pycommunity/pycon/PyCon07/slides.html - reST: http://programmation-python.org/pycommunity/pycon/PyCon07/slides.txt

Don't hesitate to send me feedback about them

http://programmation-python.org/pycommunity/pycon/PyCon07/slides.html
"S5 Slides"
http://programmation-python.org/pycommunity/pycon/PyCon07/slides.txt
"reST slides"

Technical writing, the seven laws

Tarek Ziadé — Fri, 23 Feb 2007 20:31:00 +0100

I am here at PyCon, enjoying the talks. I will blog about what I have seen in a few days. Until then, let me drop here the seven laws on how to write good technical documents.

These laws are the rules I have synthetized for the tutorial I gave yesterday, and that can be followed by anyone who write for softwares.

They are based on advices taken from:
- Writing With Power (Peter Elbow) - Agile Documentation (Andreas Rüping) - My own experience on the topic.

The seven laws for technical writing:

The Two-step writing process
Simple style
Targeted readership
Focused information
Realistic examples
"Light but sufficient" approach
Structured documents

The Two-step writing process

Writing should be done in two stages (Elbow, 1980):
- a free writing where ideas are written on the paper no matter the shape. - a review stage where things are structured and reviewed.

Each stage should take 50% of the time.

-> Split you writing time in two phases, no one can write it right the first time (except Mozart)

Simple style

Use short sentences and simple writing style.
Use simple vocabulary

Targeted readership

Focus on your target when you write a document (Rüping, 2003).

Assume their background knowledge to restrict the scope of the documentation.

-> A prerequest section can help a lot.

Focused information

A document is about a clear focus.

A precise title for each section and the document itself, and a summary can help.

-> If you cannot easily name the document or one of its section, there's a problem

Realistic examples

Drop the foo and bar habits. Examples must be real-life examples, and usable as-is.

Neh:

import graph

>>> foo = graph.calculateSquare(1, 1, 1, 1)

>>> bar = graph.renderSquare(foo)

Better:

import graph

>>> square = graph.calculateSquare(1, 7, 1, 10)

>>> square_view = graph.renderSquare(square)

"Light but sufficient" approach

A working software is more important than the best documentation in the world (Ambler, 2002).

Quality over Quantity is the best rule.

-> Spending too much time to find something in the documentation is a bad sign.

-> Think documents like code. Always limit the size of sections, examples, etc. Modularized documentation is the key.

Structured documents

Use a clear document portfolio to facilitate the reading. Document should use:
- an abstract with a readers guideline - a toc when there are more than one section - references - glossary - tables and diagrams

-> Never write a document that doesn't have a template

"id6"

PyCommunity 0.1 almost ready

Tarek Ziadé — Sun, 11 Feb 2007 22:53:00 +0100

I've been working a lot on PyCommunity packaging lately, so it can be ready for my PyCon tutorial. Version 0.1 will be released by then and will autogenerate html pages for a Python project that bases its documentation on reST files.

There are a few mandatory files that have to be present in the project for the tool to work, but it is not really restrictive. I tried a few runs on Zope codebase and it works fine, besides a few problems with epydoc (some Zope files are breaking epydoc, I need to dig on this) and a few failures on some malformed reST files.

PyCommunity is based on Cheetah templates, that can be customized to smoothly integrate generated pages into an existing website.

Features to complete for 0.1:
- Cheesecake index report for each package - test coverage report for each package, using trace2html - Crunchy frog integration - Pygments integration - SVN hook dameon with a queue, so the generation can be done smoothly - Folder and file skipping to avoid the process of unwanted content

The source code is in a Mercurial repository, right here: http://hg.programmation-python.org/repositories/public

If you follow this link, you will see a generated site based on a fake project used in PyCommunity unit tests.

"Trace2html"

a Python Developer Room at the FOSDEM

Tarek Ziadé — Fri, 09 Feb 2007 18:43:00 +0100

As a young PUG, we've missed the deadline last year. We're in this year !

The AFPY (Association Francophone Python) organizes this year, at the annual FOSDEM conference (24/25 February, Brussels, Belgium) a Python Developer Room.

The room schedule can be followed here: http://www.fosdem.org/2007/schedule/devroom/python

A lot of interesting Python-related conferences (and Zope as well) will take place, and a few sprints will probably be organized as well.

So if you are in Europe, not attending to PyCon, the FOSDEM is the place to go to !

"Python Developer Room"

Pycon: detailed outline for documenting projects

Tarek Ziadé — Thu, 18 Jan 2007 10:13:00 +0100

There's only a few days remaining before tutorial cancellation at Pycon, if you have interest in some of them, it's time to register !

I have completed my outline for the tutorial I'll be giving. If you consider to attend, you should bring your laptop and have this elements installed on it, to maximize your pleasure ;) :
- Python 2.4 or 2.5 - docutils - setuptools - pylint - cheesecake - subversion, client and server (optional)

(PyCommunity will be provided at tutorial day).

Updated outline:

- writing a document

  - mastering reStructuredText

    - presentation of reStructuredText

    - distribution of cheatsheet

    - exercises

  - writing techniques and tips

    - presentation of the ten laws, with examples

    - exercise in binomial

  - team writing

    - presentation: bad writer, good designer VS good writer, bad designer

    - exercise: doc review

- writing documents in Python

  - documenting a module

    - presentation : which module to document, how to split content into docs ?

    - exercise (on an existing package)

  - documenting a package

    - presentation: README.txt and friends

    - exercise: writing the minimum files, testing with cheesecake

- using documents for test-driven developement

  - mastering doctests

    - presentation: unittests ain't doctests

    - exercise: spliting tests between doctests and unittests

  - continuous documentation

    - presentation: designing through doctests

    - exercise: creating a module with its doctest

BREAK

- writing documents in a Python Project

  - writing tutorials

    - presentation: what's a tutorial

    - presentation: how to write a tutorial

    - exercice: write a mini-tutorial

  - writing cookbook and recipes

    - presentation: what's a cookbook

    - presentation: how to write a recipe

    - exercice: write a recipe

  - writing a project glossary

    - exercise: write a glossary

  - crosslinking everything

    - exercice: link the recipe, tutorial and glossary

- website autogeneration with pycommunity

  - presentation: pycommunity

  - exercise: setting up a subversion structure

  - exercise: setting up a hook for website autogeneration

  - customize pycommunity for an existing website

- conclusions

Pycon wiki updates

Tarek Ziadé — Thu, 11 Jan 2007 21:45:00 +0100

I have updated a bit the PyCon Wiki tonite :
- A more detailed class outline about my tutorial on documenting - A BoF about creating a content repository standard in Python

"DDD"
"PyJCR"

ZODB vs SGBD ? Give me a standard, it's about time !

Tarek Ziadé — Tue, 09 Jan 2007 14:16:00 +0100

I've been reading Carlos' post about Zope 3, and his thaugths on the role of the framework.

I want to react on this and give my opinion on Zope future and on Python Web developement as well, because I feel that people are not debating on the right thing.

Since a few months, a lot of people in Zope community are seeing Zope 3 as packages that can be used in Zope 2 or elsewhere. This is probably true. A lot of people are also using CherryPy, Django, TurboGears, .., and dropping Zope for some reasons. These reasons are probably right. (my reason to drop Zope would be the ZODB).

But beside this reasons, (every framework has pro's and con's) there's a missing brick in the Python Web Developement ecosystem that exists in Java world: a standard for document repositories. It's called JCR or JSR-170. Some people in Zope community have talked about it in the past. Nuxeo did a bind as well with nuxeo.jcr, then dropped Zope/Python for Java (which is totally logical given where CPS software is heading).

I strongly believe that the future and the maturity of all Python web frameworks relies at the first place on a common document repository standard, like DB API did with SQL.

Python web developers, it's time for all of us to cooperate and create such a standard in Python, and an implementation based on SQLAlchemy. This would let Zope drop the ZODB (yepee) and let every web developer use the best of all frameworks (the Zope Component Architecture, the Django approach, etc.) since they would all be able to work with a given repository.

I would like to see the PSF and all framework core developers support such a project.
edit: oups.. I just realized I have used a french acronym in the title (SGBD). The english one is RDBMS

"Carlos"

Document-Driven Development (DDD) in Python

Tarek Ziadé — Sat, 06 Jan 2007 23:04:00 +0100

Test-Driven Development is widely used in the Python community to create quality software. The benefits of such approach is not to be proved anymore. It creates better software by :
- preventing regression - making the developers actually think about what they write - providing a real help for newcomers : unit and functional tests are showing how the code was made and how it has to be used. - ...

Python has brought a real enhancement to test-driven developement with doctests. They allow developer to write documentation with embed code examples that can be run for real. This allows a team to integrate documentation in the development cycle and to replace most of their tests with documents that are always up to date: documents become tests.

In the meantime, documentation has always been the black beast of developers. They hate writing it and most of the time, a software project website is never up to date if there's no one dedicated to this task. For open source projects, the website is the most important media and should always reflect to what is happening in the code base.

doctests can resolve this problem by automating documentation creation at all level

I have used this principle to create a Document-Driven Developpement (DDD) approach, which provides to a project team a way to automate the update the project website : each commit on the code base calls a script that generates on the fly html pages for:
- the api - the glossary for the project - documents for all important modules - tutorials - recipes - ...

This method will be fully explained at my PyCon Tutorial (February, 22th).

I will published by then the script that automates the creation of the website, using a subversion repository that follows a few guidelines (in its content and structure).

Can't wait to be there !

"pycon"

Pycon 2007 tutorial about documenting Python projects

Tarek Ziadé — Sat, 02 Dec 2006 23:06:00 +0100

My tutorial proposal on how to document a Python project, has been accepted. I have quite a good idea on the things I need to present for people to understand how to integrate documentation writing in their development cycle.See the class outline for a summary

The main idea behind this tutorial is to explain how documenting can be done through coding and testing, in order to create lightweight but sufficient documentation instead of creating documents at the end of the project.

The technique is based on:
- literate programming, introduced by the father of Tex, [Donald Knuth][] - Agile documentation principles, which uses agile techniques for document writing . These have been [summarized in a book by Andreas Rüping][].

and provide simple processes to avoid the "programmers hate writing document" effect.

For open source projects, the final step of agile documentation is:
- to automate the update of the dedicated website everytime developers change the base code. - to provide a visible document landscape to anyone interested to the project - to increase the quality of the code base - to make developers become good writers, and make them love it

This tutorial is based on my work on zope-cookbook.org, which tried to follow a fast-moving technology : Zope, and some work on several Python open source projects, books and articles I have written.

Nevertheless, the conference is next February, and I would like to integrate more feedback on how people feel about documenting their application.

If you are a developer, using Python or any language out there, or a project leader, I would love to get a few words about your experience on the topic, in order to enhance this tutorial.
- What is the place of documentation in your project cycles ? - What are the different document types you use ? - What kind of formats do you use for you document ? (Word, OpenOffice, Latex, reST..) - Do you have dedicated people that write documentation ? - How do your developers feel about documentation ? - ...

Please don't hesitate to comment this entry, or to senf me an email (ziade dot tarek at gmail dot com)

"Pycon Tutoria"
"Outiline"
"Literate programming"
"Donald Knuth"
"Agile documentation"
"http://zope-cookbook.org/"

The dark side of the force

Tarek Ziadé — Wed, 29 Nov 2006 01:29:00 +0100

Remember the funny post about Python vs Perl on Python mailing list ?

I think it fits pretty well to TDD. Here's a slighty modified version:

EXTERIOR: DAGOBAH -- DAY

With Yoda strapped to his back, Luke climbs up one of the
many thick vines that grow in the swamp until he reaches the
Dagobah statistics lab. Panting heavily, he continues his
exercises -- writing test-driven code in Python, making
fakes, mock objects.

YODA: Code! Test! Yes. A programmer's strength flows from non-regression.
Never stop doing tests. The dark side of code maintainability
reach you would without tests. Easy to write without them when
package you create. If once you start down the dark path,
forever will it dominate your destiny, consume you it will.

LUKE: Isn't it simpler not to do tests ?

YODA: No... no... no. Quicker, easier, more seductive.

LUKE: So why I should do tests then ?

YODA: You will know. When your code you try to correct six months from

now.

"Yoda about Perl/Python"

How to prevent drowning in the huge rss daily feed you receive

Tarek Ziadé — Mon, 06 Nov 2006 19:32:00 +0100

Feeds, feeds, feeds everywhere. I can't keep up with all incoming data. I am drowned !

I thought digg and similar services would help me on this, but it just raised the problem on a meta-level.

It takes me at least 20 minutes per day to:
- Remove duplicates entries - Look over hundreds of entries to select the one that worth a reading, by looking at the title, the origin and the tags. (80% of the time) - Read my selection, and even though I know this will make it worse, add news feeds I've found from my selection.

I had to automate some of this daily activity. I had to cut off this huge amount of data and to get closer to what I needed to read.

But how ?

First of all, let's analyze a bit what a feed reader wants. It can be resumed in two things:
- She wants to get the best news on her field of interest, out of a huge list of rss feeds.

She wants to keep an eye on what's going on out there, and make her field evolve with some indexes.

Most of the work can be done by pieces of software. I think this is called webmining but I am not an expert. Webmining would be a part of Datamining. But let's cut off the big words: a couple of Python script can do the job.

That's "Atomisator" job !

Atomisator grabs multiple feeds out there, removes duplicate news by using the distance of Levenshtein distance, and create a new feed out of it.

It also provides a way to filter up entries on the fly, by looking at each entry content. Simple filters for instance, will validate an entry if it contains certain words. The filtering system is pluggable, and works as a transformation chain, so new filter can be written to fine-tune the entries. This makes it possible to create several custom feeds form the same pile of data.

Another field of investigation was to use a bayesian network to filter entries but it doesn't work well : pertinency moves too fast on this kind of news, and an inference mechanism would work on static news topics (Historic events maybe ?)

Last but not least, a special filter is used to collect statistics, and compute a buzz-o-meter report. This report indicates the top 50 most used words over all sources, and is filtered with an english common words dictionnary. It doesn't use the tags like most tag clouds, because people often use the same tags over and over, without really thinking about it.

A scan of the post content is way better: you get the REAL tags.

That's how I keep an eye on what's the most talked about, even though it's not on my custom feed: I can adapt it afterwards.

Ok now here's the feeds I use, updated every 30mn :
- "Meta Feed" Not filtered, but not redundant - Python/Zope oriented (lots of filtering) - Live buzz-o-meter

Get it all here in this page

This is pretty handy for my daily job. The buzz-o-meter is for fun, but I see from time to time new words that pops in the list, I can investigate on. It also shows that Ajax and Ruby lead the buzz-o-sphere.

The source code is GPL and available here, but still poorly documented and not packaged, should be better soon. The version running the feeds is a bit different from the trunk and some merging will be done soon. This was a project started some months ago I just wrapped and re-lauched yesterday.

N.B.: I had a few comments on how poorly written was this blog entry. If you find some mistakes or badly turned sentences, don't hesitate: tell me ! (i am french ;) )

"Levenstein"
"Metafeed"
"Python feed"
"buzz"
"Atomisator"
"Atomisator code"

Protecting a Python svn code base with the pre-commit hook

Tarek Ziadé — Wed, 01 Nov 2006 12:29:00 +0100

In a community project, opened to various contributors, there are a few thing to take care of in order not to break the code. I am not talking about code reviewing but about bad code editing that breaks it all.

Most frequent errors are:
- <tab> insertions, that get mixed with space-based indentation - carriage return insertion, before line feeds with some weird windows editors

Instead of tracking the developers that commited such things, and send them an army of white rabbits, an automatic code check is way better. It is not a good idea though, to clean up incoming code. The best thing to do is to block unwanted changes and warn the commiters, so they learn about it.

This is really easy with Subversion. I have grabbed a script on the web, and adapted it a bit for this task. Here it is:
#!/usr/bin/python2.4

# -*- coding: UTF-8 -*-

# adapted from:

#   http://blog.wordaligned.org/articles/2006/08/09/a-subversion-pre-commit-hook

# by Tarek Ziadé

from subprocess import Popen

from subprocess import PIPE

import re

import os

re_options = re.IGNORECASE | re.MULTILINE | re.DOTALL

class EOF(object):

    def findall(self, content):

        if content.endswith('\\n'):

            return []

        return ['\n']

tab_catcher = re.compile(r'^\\t', re_options)

windows_catcher = re.compile(r'\\r\\n$', re_options)

testers = (('found TAB', tab_catcher),

           ('found CR/LF', windows_catcher),

           ('no new line at the end', EOF()))

def command_output(cmd):

    """ Capture a command's standard output."""

    return Popen(cmd.split(), stdout=PIPE).communicate()[0]

def files_changed(look_cmd):

    """ List the files added or updated by this transaction."""

    def filename(line):

        return line[4:]

    def added_or_updated(line):

        return line and line[0] in ("A", "U")

    return [filename(line) for line in

            command_output(look_cmd % "changed").split("n")

            if added_or_updated(line)]

def file_contents(filename, look_cmd):

    """Return a file's contents for this transaction"""

    return command_output("%s %s" % (look_cmd % "cat", filename))

def test_expression(expr, filename, look_cmd):

    """test regexpr over file"""

    return len(expr.findall(file_contents(filename, look_cmd))) > 0

def check_file(look_cmd):

    """checks Python files in this transaction"""

    def is_python_file(fname):

        return os.path.splitext(fname)[1] in ".py".split()

    erroneous_files = []

    for file in files_changed(look_cmd):

        if not is_python_file(file):

            continue

        for error_type, tester in testers:

            if test_expression(tester, file, look_cmd):

                erroneous_files.append((error_type, file))

    num_failures = len(erroneous_files)

    if num_failures > 0:

        sys.stderr.write("[ERROR] please check your files:n")

        for error_type, file in erroneous_files:

            sys.stderr.write("[ERROR] %s in %sn" % (error_type, file))

    return num_failures

def main():

    from optparse import OptionParser

    parser = OptionParser()

    parser.add_option("-r", "--revision",

                        help="Test mode. TXN actually refers to a revision.",

                        action="store_true", default=False)

    errors = 0

    (opts, (repos, txn_or_rvn)) = parser.parse_args()

    look_opt = ("--transaction", "--revision")[opts.revision]

    look_cmd = "svnlook %s %s %s %s" % (

        "%s", repos, look_opt, txn_or_rvn)

    errors += check_file(look_cmd)

    return errors

if __name__ == "__main__":

    import sys

    sys.exit(main())

I've also added a new line at end of file control. This script has to be called in the pre-commit hook script (look up in SVN documentation)

The call should look like:
/chemin/vers/script/svn_check_source.py "$REPOS" "$TXN" || exit 1

You can then extend the controls made by this script, by controlling for example the quality of the commited code with tools like pychecker. But these extra controls should not block commits and should be quite light to perform because the commiter waits for the changeset to be validated.

Sending a mail to the commiter with suggestions when her code doesn't pass some quality checks is a better idea. Furthermore, for an extensive QA test, it is simpler to hook a script on a system like buildbot and create a nighlty digest over the whole code base. Pylint is very handy on this kind of controls, and can be fine tuned to generate a useful QA report that buildbot can send to developers.

Pycon proposal : How to document a Python open source project

Tarek Ziadé — Fri, 27 Oct 2006 22:14:00 +0200

I have sent a proposal for a tutorial at Pycon, covering the way to document a project through the developement cycle. It's based on Agile Documentation principles and provide a way to smoothly integrate documentation in the daily developer work.

Here it is:

Summary

Documentation plays a vital role in open sources projects.

Well designed, it attracts new people into the project,
help the users use it, and raises the quality
of the code base and the team work.

In other words, it builds the shared knowledge.

This tutorial is based on a talk i've made
at Europython about zope-cookbook.org, and go
further by giving a complete set of practical methods
that can help a team to create and maintain an up to date
documentation with no pain.

An example code base will be provided and used for all
exercises.

By the end of the tutorial, people should be able to
integrate documentation in their developement process,
the same way they probably did one day with unit testing.

Presentation outline
- agile documentation principle (short overview) - mastering doctests (overview + exercises) - mastering reStructuredText (overview + exercises) - writing techniques and tips (examples + exercises) - shapes of documents (overview) - building document templates (examples + exercises) - setting up documentation writing in the development cycle: - continuous documentation (examples + exercises) - team writing (exercises) - portfolio (overview) - setting up a help center (overview + exercise) - linking it all together (complete example (svn/trac/website/etc..))

* Intended audience

All, except the doctests exercises that will have a bit of coding

the zope-cookbook talk:
- http://indico.cern.ch/contributionDisplay.py?contribId=5&sessionId=53&confId=44

Now let's wait and see if it's accepted :)

"Pycon"
"Agile Documentation"
http://indico.cern.ch/contributionDisplay.py?contribId=5&sessionId=53&confId=44

Mini-book: a dozen recipes for Zope 3

Tarek Ziadé — Thu, 19 Oct 2006 09:59:00 +0200

I have started a few months ago to write a cookbook for Zope 3, both in english and french. The challenge was to be able to:
- follow the project code base wich is evolving (too) fast - provide transversal knowledge for developers that wants to create real world things with zope

The project was growing and doing well, but I had to stop, because of Nuxeo's drop of Zope: it became too hard to follow Zope 3 development exclusively in my spare time.

Nevertheless, Baiju copied the recipes in his wiki, and hopefully they will be reinjected in the work done their to construct a mainstream book.

However, this experience was really exciting, because I had to set up a writing process, explained last summer at Europython, based on agile documentation principles, to be able to provide up-to-date materials, and examples codes.

As a souvenir ;), I have generated raw pdfs in both english and french. They might be useful to people that are discovering Zope 3 out there, as they also introduces developing methodologies.

Here they are:
- Zope 3 cookbook in english - Zope 3 cookbook in french

Let me know of they are useful to you !

Important note: those recipes are not following a book plan, so the pdf table of content can look a bit strange to readers.

"Baiju wiki"
"Eurpython 2006"
"Zope 3 cookbook in english"
"Zope 3 cookbook in french"

Marketing Python is urgent, part #1: a Python Certification

Tarek Ziadé — Sun, 15 Oct 2006 16:45:00 +0200

Thing changes

For about three years, I had one of the best place a Python developer could dream of in France : one of the most Python-friendly company out there (Nuxeo), that was promoting the language everytime it was possible. In this context, even though you've told so, you don't feel how Python is small compared to Java in the business area. You just don't want to hear about it because you develop great software, and feel happy with Python.

But time are changing, and when your company decides to drop Python for Java for the main language, a lot of things goes through one developer's head.
- First of all, I felt like back in college, where people were laughing at me and my redhat box, because everybody was under Windows and thaught Linux was a piece of crap. I felt like I was pushed into some kind of minority. Weird feeling. - Secondly, I felt a little bitter because I understood, while all arguments of such a switch are quite understandable, that it's mostly a problem of money, because Python can do what Java does. If I would've been a billionnaire 10 years ago, Python would be the standard nowadays, and all arguments could be flipped ;). But the reality is here now, Java is the main stream language in OSS. Java and its related technologies are interesting, but I still look at them like I look at Ruby: what are their strengths ? How Python can learn of them ? This is a pragmatic Pythoneer habit. - Last but not least, with all the evangelism I've been doing these past years, in english or in french through my book or the AFPY user group, I still want to promote Python and be part of its community.

But how ?

This blog will throw ideas about marketing Python

All the talks about my company switch makes me feel like marketing Python is urgent, so it grows in companies, schools, etc. Looking at how Ruby does its marketing is quite interesting, but this language is still too young to be able to differenciate the reality from the buzz effect yet.

Looking at how Linux did, and still does its marketing, is more interesting because you know what worked and what didn't.

So this blog will focus on a few things that could be done. And this first post focuses on Python Certification.

Important notice: i have digged to find some infos on this topic, but if I missed something that is beeing done, please forgive me and comment this post.

A Python Certification

Linux certifications, brought for example by Dan York and the LPI, have became great tools to promote Linux in business.

The benefits are quite simple :
- Certification centers promote Linux and pay big money in advertisment to attract students - The LPI is a non-profit organization and can therefore create a standard in certification, that is followed by everyone - The students became in their own Linux evangelists, and there's a crowd of students out there that are just waiting to know the truth ;) - Companies can rely on it to hire - Developers can obtain it to get recognized

So what about Python ? Take the last chapter I wrote, change "Linux" by "Python", and "LPI" by "PSF" and you get the idea.

I really believe the Python Software Foundation should create a Python certification. That's probably what we are going to dig on our side at AFPY and maybe come up with some proposition at the PSF. And you, readers ? what are your opinion about it ? I'd love to hear it.
Next time I'll talk about the Python Program Certification, which is basically the same idea but applies to software standardisation and quality, and about cheesecake, which is a great idea.

"Nuxeo blog"
"My book"