Additional feature

So there's always been a flag to make the pool size of the bubble, and that was nice, but I wanted a way to make these more permanent as well as simpler to remember. So it's now an option in the runs_parallel decorator. To use it you'd just simply give it a size to use:

#!/usr/bin/env python

from fabric.api import *

env.hosts = ['host%2d.com' % x for x in range(20)]

@runs_parallel(with_bubble_of=10)
def poke():
    run('uptime')

This I feel makes a cleaner fabfile, and puts this information where it should be, in the code, and out of the writer's head.

How to use both ways, or just one

A big thing to note in using parallel tasks, is that anything put into shared variables, like env, is forgotten outside the execution of the one instance of the task. So if you don't add a @runs_once or @runs_sequential decorator to a task that say sets the env.hosts before an actual parallel task, the work done inside the env setting task is forgotten.

The reason adding these decorators addresses this, is that by adding them, the task isn't executed using the parallel bits. It is instead run inside the main fab process, and isn't creating a fork pool of size 1 and forgetting about it when the fork is finished executing.

So as an example, if one were to try and run a fabfile w/o setting up decorators for their functions, and running: fab -P set_hosts uptime

#!/usr/bin/env python

from fabric.api import env, local, run, sudo

env.hosts = ['somehost.com']

def set_hosts():
    env.hosts = ['web-0', 'web-1']

def uptime():
    run('uptime')

They'd get into an issue where the set_hosts not being specifically set to run sequential or once, would have the settings it made to the env.hosts var only apply inside the task, since it's been forked out. Which would cause the uptime to only run on somehost.com, and not both web-0 and web-1 as expected.

#!/usr/bin/env python

from fabric.api import *

#thanks to Eric who pointed this out, visit his site, it's neat
env.hosts = ['ericholscher.com']

@runs_once
def set_hosts():
    env.hosts = ['web-0', 'web-1']

@runs_parallel
def uptime():
    run('uptime')

Maybe it'll help

As a boon to people using both the parallel branch and Trunk, on a single fabfile, note that which one is being used can be determined at runtime using some silly introspection:

>>> from fabric import decorators
>>> dir(decorators)
['StringTypes', '__builtins__', '__doc__', '__file__', '__name__',
'__package__', '_parallel', '_sequential', 'hosts', 'is_parallel',
'is_sequential', 'needs_multiprocessing', 'roles', 'runs_once',
'runs_parallel', 'runs_sequential', 'wraps']
>>> "runs_once" in dir(decorators)
True

So one could just flip a Boolean and decorate/use things accordingly. Though I suggest using @runs_once on any tasks that are just that, single shots that do stuff local, or set vars for the fabfile, and to reserve using @runs_sequential for tasks that still need to have multiple hosts, but need to not run side by side.

Bug outstanding

Finally there is a outstanding bug with use of this branch on windows, https://github.com/goosemo/fabric/issues#issue/5, that'll bite people. I'll try and work this out, but I'm a bad developer and am dragging my feet on having to install windows to debug this. But it's the new year so I'll make it a resolution, and we all know people never drop those.

I've updated this a bit since the first push of the post

It's been a wish for some for a long time

Since at least 07/21/2009 the desire to have fully parallel execution across hosts, or something similar to that. I stumbled upon the thread around March of this year. Being that I'd been using Fabric for a number of months at this point, and had recently made a script set for work stuff that leveraged multiprocessing I decided to give the issue a go.

What I implemented, and why

The short of it being that I made a data structure, Job_Queue that I feed fab tasks into and it will keep a running 'bubble' (works like a pool) of multiprocessing Processes going executing the Fabric tasks as they are able to do so. The job queue having a pool size as set by the user, or 1/2 the size of the host list if not specified, or if the pool size is larger than the host list size, it will match the host list size.

---------------------------
[-----]--------------------
---[-----]-----------------
---------[-----]-----------
------------------[-----]--
--------------------[-----]
---------------------------

env.host_string = env.host = job.name
job.start()
self._running.append(job)

What this branch adds to Fabric

There are new command flags for fab:

-P, --parallel        use the multiprocessing module to fork by hosts
-z FORKS, --pool-size=FORKS Set the number of forks to use in the pool.

I have also added two decorators:

@runs_parallel
@runs_sequential

These will allow a fab file command to be set to be run either in parallel or sequentially regardless of the fab command flag. Without these commands switch when the flag is set.

What that means is that any tasks that are decorated are always run as either parallel or sequential. Tasks that omit these decorators though, are going to be able to switch back and forth between running parallel or running sequentially. Something the user would be specifying at run time with the -P flag.

With the new stuff comes a few caveats

If you are interested in the guts, the implementation is in the main.py file, and uses the Job_Queue class in the job_queue.py file. Note that this is only implemented in the fab command, as there is no way to determine how one will execute functions if they are using Fabric as a helper library.

If the runs_once decorator is being used on a function that is called from inside a fabric task, it won't be able to be honored. Because the states in the forks are separate, and every fork will think it's the first one to run the function. Simple solution being to make the call it's own task call.

Now to see it in use

Here is a little example of a fab file that is running some command on the server that will take 10 seconds to run. Yeah sleep is a bit of cheat for this, but it's good enough to show the benefit of forking out tasks that'd take a crap ton of time otherwise

from fabric.api import *
from server_list import servers

env.roledefs = servers.server_classes

@roles('servers')
def poke():
    run("sleep 10")

$ time fab poke -P -z 20
...

real   0m45.868s
user   1m7.928s
sys    0m8.425s

$ time fab poke
...

real   8m51.477s
user   6m3.239s
sys    1m26.637s

$ pstree -paul
...
│   ├─bash,20062
│   │   └─fab,21455 /home/mgoose/.virtualenvs/fabric-merge/bin/fab poke -P -z 20
│   │       ├─fab,21462 /home/mgoose/.virtualenvs/fabric-merge/bin/fab poke -P -z 20
│   │       │   └─{fab},21493
│   │       ├─fab,21463 /home/mgoose/.virtualenvs/fabric-merge/bin/fab poke -P -z 20
│   │       │   ├─{fab},21484
│   │       │   ├─{fab},21505
│   │       │   ├─{fab},21511
│   │       │   └─{fab},21517
│   │       ├─fab,21464 /home/mgoose/.virtualenvs/fabric-merge/bin/fab poke -P -z 20
│   │       │   └─{fab},21487
│   │       ├─fab,21465 /home/mgoose/.virtualenvs/fabric-merge/bin/fab poke -P -z 20
│   │       │   ├─{fab},21483
│   │       │   ├─{fab},21502
│   │       │   ├─{fab},21503
│   │       │   └─{fab},21504
...
(16 more fab lines)

Use it and let me know

I'd love to hear how people are using this, and if they find any holes in my implementation. I've got a few more things I want/need to add into this, and I've got them listed in the github issues just until this gets integrated into the Fabric mainline.

Presentations are hard

Not even just giving them, but the process of taking an idea and putting it down in a concise and clear manner, that will be simple to show and distribute.

I am not going to make a claim that I am good at giving or writing up presentations, but I do think I have a pretty slick setup going for making them simple to show and distribute with minimal effort.

reStructuredText

People may be familiar with it already, but it's a markup language that's well supported in python and used in the Sphinx project. It fits my brain well (in most places) and only requires a text editor.

Since it comes with some tools to take a simple plain text document and transform it into other well known formats (latex, html, and pdf), caused me to use it in a number of projects and for notes without much friction.

How I use reST

I mentioned in a previous post about my setup and how I leverage fabric to build and post my presentations, but I didn't really get too much into the specifics of how the rst document was formatted, and built.

I write a presentation in rst just like I would write any other document in rst. The goal I had was to write once, and have that document change into other formats without an issue. I do leverage a few classes and directives that aren't in the normal rst toolbox, to get my presentations just so. But these aren't out of line, and after a tweak or two in my pipeline don't break the other formats I build to.

s5

S5 is a slide show format based entirely on XHTML, CSS, and JavaScript.

And rst2s5 takes a reStructuredText document and complies it into the corresponding s5 representation. Giving back a plain html page with some JavaScript magic that is simple to post and host.

No need for server side scripting, or fancy Apache/lighttpd/nginx setups or any need for proxies or their kin. So using s5 alone will give me the goal of simple to show, since I can post a presentation and have accesses to it anywhere there is internet and a browser. I can even keep a copy on a thumb drive in case the internet dies, and browse the slides locally.

reST with s5

To meet the last part of my goal, I have to have a simple distribution medium. For a presentation that is a pdf. It's akin to a paper, even though it is much more broken down and split up into slides. Leveraging the handout class that the s5 and pdf converters from rst know, I am able to have parts of the presentation invisible in slide form, and show up only when the presentation is expanded, or complied into a pdf.

=========
GNU tools
=========
----------------------------
*mostly for text processing*
----------------------------

.. class:: right

    `Morgan Goose http://morgangoose.com`
    January 2010

.. class:: handout

    This work is licensed under the Creative Commons
    Attribution-Noncommercial-Share Alike 3.0 United States License.
    To view a copy of this license, visit
    http://creativecommons.org/licenses/by-nc-sa/3.0/us/ or send a letter
    to Creative Commons, 171 Second Street, Suite 300, San Francisco,
    California, 94105, USA.

Code blocks

Normal reST had code directives that will differentiate the code, and in most instances (Sphinx/Trac) will attempt to highlight the code accordingly. I ran into issues here because rst2pdf and rst2s5 had different ideas on what these should be named and neither really was highlighting the code. After searching a bit I found that pygments, a code highlighter in python, already had some docutils hooks that they mention on their site.

Using that as a stepping stone I added in code, code, and source code, directives to use pygments for the code they contained. In my presentations though I made sure to only use code because this is the directive that rst2pdf is expecting when it goes to format the document.

After that my code goes through the ringer as shown in the post I gave for fabric, in the line executing the rst-directive.py file and passing in the pygments css for the theme that I prefer.

the final rst-directive.py looks like this though:

# -*- coding: utf-8 -*-
"""
    The Pygments reStructuredText directive
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    This fragment is a Docutils_ 0.5 directive that renders source code
    (to HTML only, currently) via Pygments.

    To use it, adjust the options below and copy the code into a module
    that you import on initialization.  The code then automatically
    registers a ``sourcecode`` directive that you can use instead of
    normal code blocks like this::

        .. sourcecode-block:: python

            My code goes here.

    If you want to have different code styles, e.g. one with line numbers
    and one without, add formatters with their names in the VARIANTS dict
    below.  You can invoke them instead of the DEFAULT one by using a
    directive option::

        .. sourcecode-block:: python
            :linenos:

            My code goes here.

    Look at the `directive documentation`_ to get all the gory details.

    .. _Docutils: http://docutils.sf.net/
    .. _directive documentation:
       http://docutils.sourceforge.net/docs/howto/rst-directives.html

    :copyright: Copyright 2006-2009 by the Pygments team, see AUTHORS.
    :license: BSD, see LICENSE for details.
"""

# Options
# ~~~~~~~

# Set to True if you want inline CSS styles instead of classes
INLINESTYLES = False
STYLE = "fruity"

from pygments.formatters import HtmlFormatter

# The default formatter
DEFAULT = HtmlFormatter(noclasses=INLINESTYLES, style=STYLE)

# Add name -> formatter pairs for every variant you want to use
VARIANTS = {
    'linenos': HtmlFormatter(noclasses=INLINESTYLES, linenos=False),
}


from docutils import nodes
from docutils.parsers.rst import directives, Directive

from pygments import highlight
from pygments.lexers import get_lexer_by_name, TextLexer

class Pygments(Directive):
    """ Source code execution.
    """
    required_arguments = 1
    optional_arguments = 0
    final_argument_whitespace = True
    option_spec = dict([(key, directives.flag) for key in VARIANTS])
    has_content = True

    def run(self):
        self.assert_has_content()
        try:
            lexer = get_lexer_by_name(self.arguments[0])
        except ValueError:
            # no lexer found - use the text one instead of an exception
            lexer = TextLexer()
        # take an arbitrary option if more than one is given
        formatter = self.options and VARIANTS[self.options.keys()[0]] or DEFAULT

        print >>open('pygments.css', 'w'), formatter.get_style_defs('.highlight')
        parsed = highlight(u'\n'.join(self.content), lexer, formatter)
        return [nodes.raw('', parsed, format='html')]

directives.register_directive('sourcecode', Pygments)
directives.register_directive('code', Pygments)
directives.register_directive('code', Pygments)

from docutils.core import publish_cmdline, default_description

description = ('Generates S5 (X)HTML slideshow documents from standalone '
               'reStructuredText sources.  ' + default_description)

publish_cmdline(writer_name='s5', description=description)

And in combination with my fabric setup I can make new posts, publish to html and pdf, and republish with relative ease:

$ fab new:new_stuff
$ vim new_stuff/new_stuff.rst
$ fab upload:new_stuff