Planet Open Plans (dev)

[from rmarianski] Web to get more social as OAuth is sanctioned for use

2008-08-27T22:59:37+00:00

[from rmarianski] Ajaxian » Proxy issues with querystrings in path names

2008-08-27T13:46:22+00:00

Comment on Frisbee Design! by Williamveronica

2008-08-27T09:31:31+00:00

Had to leave my mark, Nice site! Rugzar

A protocol for volunteering opportunities? Part II

2008-08-27T04:15:40+00:00

Today, I again met with a member (Matthew Cooperrider) of the Web4Roots meetup group. Our goal was to pare down what a protocol for exchanging information about volunteering opportunities should look like.

First a quick summary of the problem:
1) Organizations need a standard way to syndicate volunteer opportunities they create.
2) Volunteers need an easy way to search for volunteer opportunities that match their location, schedule and interests.

During our first hack at this problem, we tried to treat volunteers and organizations as completely different entities. This time our thought process was to see if we could model volunteers and organizations in the same way. So what is fundamentally the same between volunteers and organizations? Both have something to offer (volunteers: time, money, experience, etc.; organizations: tax right offs, a good time, free food, etc) and both have something they need (volunteers: tax write off, recommendation, satisfaction, etc; organizations: labor, money, supplies, etc). So in effect, every time a person volunteers, he exchanges what he can give to the organization for what he wants from the organization. The reverse is true for organizations. So how can we model this?

Say we have a volunteer name John, who lives in New York City, enjoys cooking, is free on week nights, and wants to volunteer for 5 hours a month. Additionally, he is only willing to volunteer for an organization that can give him a $5000 tax write off. How would the protocol represent this character?

<exchange>
  <export>
    <type>Cooking</type>
    <location>Boston</location>
    <datetime>Weeknights</datetime>
    <quantity>5 hours per month</quantity
  </export>
  <import>
    <type>tax write off</type>
    <location>USA</location>
    <datetime>Now - Jan 1st 2010</datetime>
    <quantity>$5000</quantity>
  </import>
</exchange>

By the same token, say if we have an organization in New York City who is trying to redesign their website by March 1, 2009. Furthermore, it wants a volunteer who is willing to commit 20 hours a month to this project. In return, the organization is willing to write a letter or recommendation for the volunteer:

<exchange>
  <import>
    <type>Advanced PHP Web Develop</type>
    <location>New York City</location>
    <datetime>By March 1, 2009</datetime>
    <quantity>20 hours per Month</quantity
  </import>
  <export>
    <type>recommendation</type>
    <location>USA</location>
    <datetime>anytime</datetime>
    <quantity>1</quantity>
  </export>
</exchange>

Each “exchange” package can have 0 or more import and export blocks. Furthermore, each of these packages would have to name the web service that generated the data (and a unique identifier within that service). For example, if a client on Survv asked for help for a certain issue, Survv would create a unique URI for the volunteer opportunity before it syndicated the opportunity:

<unique id>http://survv.org/opportunities/19192</unique id>

Each “exchange” package could also have extra optional data such as keywords, tags, categories, and the name of the creator.

[from rmarianski] Firefox - Ubiquity

2008-08-27T02:26:55+00:00

[from rmarianski] Ride the City: safe bike routes made easy

2008-08-26T01:40:31+00:00

Comment on Hi, I’m the Opencore Release Manager by Pierre

2008-08-20T05:32:57+00:00

Hello Paul,

this is Pierre, looking to get in mail contact , from the midterm perspective of world social forum - ( see www.openfsm.net and www.openesf.net )

Comment on 5 Easy Steps to Tab Completion in the Interactive Python Shell by magicbronson

2008-08-19T20:26:37+00:00

if you’re used to hitting tab for indenting in your interactive python, either switch to using a single space, or change complete to some other keybinding, e.g. readline.parse_and_bind(’\C-J: complete’)

[from rafrombrc] Regular Expression to match email addresses according to the various RFCs

2008-08-18T19:56:09+00:00

yowza! what a doozy!

[from closedplans] Tarek's new python book

2008-08-14T16:11:40+00:00

Comment on Week 1: Xinha, no more popups by nicholasbs

2008-08-14T14:43:03+00:00

Your plugin was actually quite flexible and met all our basic needs better than the other plugins we looked at. The reason we didn’t use it was that someone else had already done exactly what we wanted to do (replace Xinha’s popup windows with lightboxes), and he had written all of the code from scratch, so the lightbox work was mostly done.

Good luck with nyroModal! I’m going to be working on a personal project soon, and I might be using it for that.

Finding the location of the current bash script

2008-08-14T00:41:53+00:00

In my work for TOPP, I’m the middle of some changes to our build system. We’re using an in-house build tool called fassembler. Considering that it’s completely specific to our needs, and was written mostly from scratch, it’s got some pretty great features (e.g. color coded output, database initialization). Our config files are stored in subversion, checked out, and then compared against when there’s an update. If they differ, you’re prompted to either replace, discard, view the diff, or merge the files. This is great for when you’re running a build.

As the Deployment Manager for openplans.org, however, I’m running tens or hundreds of builds. My goal is to make building and maintaining a deployment easier, and so I need to be able to run the build unattended, and not in a way that blindly discards or overwrites those changes.

Enter Gentoo Linux. Gentoo is a distribution of linux where all of the packages are built from source. On a system-wide level, or for each individual package, build options can be set before installing a piece of software. A fully installed Gentoo system, whether a server or desktop, can contain hundreds of packages, and users don’t have the time to sit interactively through the building and updating of each package.

Gentoo uses a script called etc-update to handle the merging of configuration files separately from the building of software. It works by saving the new configurations with a mangled name (e.g. httpd.conf would become ._cfg0000_httpd.conf), building the list of these files, and then allowing the user to diff, overwrite, discard, or merge any of the new configurations. It allows you to configure which tools to use, defaulting to diff, smerge, and nano. I’m a vi user, but I have that set at a system level, so that’s picked up by the script. smerge is just fine for me, but I prefer colordiff (some screenshots), because of it’s nicely readable output, and so I have that overridden in a configuration file.

etc-update is licensed under version 2 of the GPL, and so we will be redistributing it bundled with the rest of our build software. Where our situation is different, however, is that we can build in a myriad of locations, and the configuration files are specific to each build. In Gentoo’s version of the script, portage (their packaging system) is queried for the location of configuration files, but we don’t have the luxury of a system level tool to perform that work for us. I looked at a couple of possible solutions to the problem:

The command line: etc-update alread includes a way to pass directories on the command line, but this requires too much typing by the user.
Building a custom script: Easy to type, but it means installing modified versions of the script all over the place, which is just harder to maintain.
Reading from the environment: It requires the user to set the environment somehow, requiring extra steps, and is very hacky
Look in a path relative to the current script: Some magic involved, but if we at least use a configuration file relative to the script, it’s relatively straightforward, and the only magic involved is in expecting where the list of directories is saved.

Based on these options, I decided on the latter option. But this all hinges on knowing during execution where the script is located. Well, I know how the script has been called. That’s available as Arg0 ( $0 ) in the shell, I figured it would be pretty easy to go from there to the actual location of the script.

Being a python programmer, my first instinct was to code the logic in python, This wasn’t too tough. I took advantage of the fact that you can pipe a script to the python shell, but used bash string interpolation to pass the argument hardcoded into the script. Since it was a multiline program, I used a bash here document to make it readable. Here’s an example script (that just returns Arg0).

#!/bin/bash

RESULT=`python << EOF

print '$0'

EOF`

It took me about five minutes to put together a final script. It first checked to see if the script was called with any path information (e.g. relative: ../script.sh or absolute: /home/script.sh) If not, it looked for the script file in the $PATH command variable. Failing that, it tried to join the current directory to Arg0 to find the actual location. (Python’s os.path.normpath command will override the base path if the search path is absolute).

This script worked, and was easy to ready for python programmers. It bothered me a bit, however, because: 1) I was embedding a python script into a bash script, which could be rather confusing, and 2) it was 32 lines long, not exactly the shortest of solutions. This is that script:

#!/bin/bash

# Python script to figure out where this file is located.
HERE=`python << EOF

import os 
import sys 

# The path environemnt variable as a list.
path='$PATH'.split(':') 

# How the script was called    
arg0='$0'

# The current working directory.
working_dir='$PWD' 

# If the script was called in any way that includes path information (relative
# or absolute), we will not look in the system path.
search_in_path=(arg0==os.path.basename(arg0))

if search_in_path:
    for dir in path: 
        if os.path.exists(os.path.join(dir, arg0)):
            print os.path.join(dir, arg0)
            sys.exit(0) 

fullpath=os.path.normpath(os.path.join(working_dir, arg0))
if os.path.exists(fullpath):   
    print fullpath 
    sys.exit(0) 

sys.exit(1)
EOF`

My next thought was to re-implement the script algorithm natively in bash. Unfortunately, bash doesn’t have the python standard library at its disposal. Thankfully, however, there are a number of commands that allow us to achieve more or less what I wrote above. I use “readlink -f /basepath/../somepath” to convert two joined paths into a normalized path. The only problem with this is that when we executed a symlink to a shell program, it returns the location of the actual file and not the symlink. I’m not really sure if this is a problem that merits any worrying, but I could imagine having a single “source” script, and symlinking it into different environments. The second command I needed to replicate was os.path.basename (used to extract the directory from the scripts full path); luckily the basename program handles this identically.

I ran into one final problem in interpreting this algorithm in bash, and that was splitting the $PATH variable. Normally the for..in control structure in bash splits a string by spaces. We could use sed or tr to convert the colon seperated pathinto a space seperated path, but that’s going to run into problems when you have spaces in you directory names. HEre’s where the $IFS variable saves us. The $IFS variable is a variable that tells bash what characters to use to split up a string into a set. For our purposes, we temporarily save $IFS and set it to a single colon. This allows you to perform a simple “for DIR in $PATH”. If you’ve got colons in you directories, well hey, you could have used python… ;-) Here’s that script:

 
# The same algorithm implemented almost purely in bash
if [ "$0" == "`basename $0`" ]; then 
    # The IFS internal variable tells bash how to split a string into variables
    # for a list.  Since the PATH variable is colon seperated, we will
    # temporarily change this variable in order to interpret the path.
    export SAVED_IFS="${IFS}"; 
    export IFS=":"; 

    for DIR in $PATH; do       
        if [ -f "${DIR}/$0" ] || [ -L "${DIR}/$0" ]; then
            THERE="${DIR}/$0" ;
            break;
        fi;
    done;
    
    # We restore the saved IFS variable to return string handling to normal.
    export IFS="${SAVED_IFS}"  
else
    THERE=`readlink -f $0` ;   
fi

The same script is 20 lines in bash, which is an improvement. At this point I was happy enough with the result that I started to embed it into our local copy of etc-update. In doing so, however, I ran across a usage of the type built-in command that piqued my interest. It was being used to test for the existence of egrep on the system. It turns out that “type -p path” looks for a file-based command and prints it if it exists. I figured that this could be used in an even shorter bash only script, and wrote a test script to do so. In checking out the various permutations (in a symlink, from the path, etc.) I found out something interesting: when you invoke a script through in the path directly, bash sets Arg0 to the full path. “Great!” I thought, combine that with readlink from above, and I have a one-liner.

And then it hit me.

which

From the which man page:

Which takes one or more arguments. For each of its arguments it prints to stdout the full path of the executables that would have been executed when this argument had been entered at the shell prompt. It does this by searching for an executable or script in the directories listed in the environment variable PATH using the same algorithm as bash(1).

The captian obvious award of the day goes to me. “which $0” will always return the full path, as bash sees it, of the script file.

Comment on Week 1: Xinha, no more popups by Nyro

2008-08-13T22:45:46+00:00

Hi,
I just read this blog post. Glad you liked my plugin.
If I well understood, you finally choose to write your own code, right?
What was the problem with my plugin? Maybe I can work on it and write some code to produce a plugin more flexible?

And keep going with your WYSIWYG.

[from closedplans] Blockquote then and now

2008-08-13T20:06:03+00:00

A Glimpse From The Future - Hassle-Free Builds

2008-08-13T19:13:25+00:00

I’ve been working over the last week on fassembler’s conflict management for configuration files. As I mentioned in an email to the fassembler mailing list, I’ve been working on integrating Gentoo’s wonderful etc-update script. This script works by saving new versions of a config using a special mangled syntax (e.g. cabochon.ini becomes ._cfg0000_cabochon.ini), and then using standard system tools for diffing, merging, and hand-editing conflicts.

When embedding it into fassembler, however, I ran into a two major difficulties. The first was handling configuration on a per-environment basis. A few simple modifications later, and etc-update no longer queries portage, but looks for config relative to the installed script file. The second was stickier, as a number of fassembler projects contains steps not just for building the software, but also for performing data modifications to make it easier to go from zero to working for out of the box installs. In order for these steps to work, the config has to already be in place, or starting necessary services will faill in a spectacular fashion.

This led me to create the concept of phases for fassembler. In the etc-update branch of fassembler, every task has a phase, inherited from the base class, defaulting to ‘build’. A few tasks are overriden at the class level, so that they are automatically placed into the ’setup’ phase. (Running Zopectl scripts is a task like this). Finally, some tasks allow you to specify the phase as part of their init functions. (A good example of this is the Script task. It defaults to the build phase, but if your are running a script that populates a database, for example, you can specify phase=’setup’, and that particular script will be deffered to the setup phase.

The phases are left as a text field to allow more flexibility in the future. Right now, the build phase is triggered, then etc-update is launched, and finally the setup phase is triggered. In the future, it could be possible to add pre and post hooks for every phase, or allow custom phases, but I’ll let that happen if there’s ever a need.

I want to present an example run on my system of opencore 0.12.1. The first text below is what happens when you build using fassembler trunk:

Start
0:00
    douglas@cezan ~/topp/etc $ ./newbuild.sh openplans/releases/opencore-0.12.1

    ------------------------------------- SNIP -----------------------------------------

Prompt 1
+2:42
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini [y/n/d/b/m/?] d
    --- /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini (old content)
    +++ /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini (new content)
    @@ -6,7 +6,7 @@
     email_to = ianb@openplans.org
     smtp_server = localhost
     error_email_from = paste@cezan.openplans.org
    -base_path = /home/douglas/topp/etc/builds/20080807-1
    +base_path = /home/douglas/topp/etc/builds/20080813

     [server:main]
     use = egg:Paste#http
    @@ -16,7 +16,7 @@
     [app:main]
     use = egg:DeliveranceVHoster
     data_dir = /home/douglas/topp/etc/var/deliverance/hosts
    -custom_error_dir = /home/douglas/topp/etc/builds/20080807-1/deliverance/src/deliverancevhoster/errors/www.openplans.org
    +custom_error_dir = /home/douglas/topp/etc/builds/20080813/deliverance/src/deliverancevhoster/errors/www.openplans.org
     #debug_headers = true
     # To view bodies:
     #debug_bodies = true

    Overwrite /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 2
+2:50
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/deliverance.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 3
+3:30
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/cabochon.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 4
+4:12
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/twirlip/twirlip.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 5
+4:18
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/twirlip.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 6
+5:03
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/tasktracker/tasktracker.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 7
+5:09
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/tasktracker.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 8
+5:17
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/wordpress/httpd-wordpress.conf [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 9
+5:33
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/wordpress.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 10
+9:36
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/zope_etc/zope.conf [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 11
+9:43
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml [y/n/d/b/m/?] d
    --- /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (old content)
    +++ /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (new content)
    @@ -1,12 +1,14 @@
    -
    +
    +

      False

    -
    + 

    - OpenPlans
    + OpenCore Site

    - greetings@openplans.org
    + greetings@localhost.localdomain

    Overwrite /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml [y/n/d/b/m/?] n

    ------------------------------------- SNIP -----------------------------------------

Prompt 12
+9:51
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 13
+9:59
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore-zeo.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 14
+11:31
    Overwrite /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 15
+11:43
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/maildrop.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

Prompt 16
+13:16
    Overwrite /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/erroreater.ini [y/n/d/b/m/?] y

    ------------------------------------- SNIP -----------------------------------------

FINISHED
+13:37
    Installation successful.
    douglas@cezan ~/topp/etc $

As you can see, over a period of 13 and a half minutes, you are prompted to make decisions 16 times. Not only do you have to know which changes to accept and reject, but that’s time you have to spend dedicated to that build process. If you need to be doing other things, the build time quickly increases as you was time waiting for each step.

Below, is an example build using the etc-update branch

Start
0:00
    douglas@cezan ~/topp/etc $ ./newbuild.sh openplans/releases/opencore-0.12.1

    ------------------------------------- SNIP -----------------------------------------

Prompt 1
+11:58
    Scanning Configuration files...
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/cabochon.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/deliverance/deliverance.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/deliverance.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/erroreater.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/wordpress/httpd-wordpress.conf
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/maildrop.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/opencore-zeo.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/tasktracker.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/tasktracker/tasktracker.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/twirlip.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/twirlip/twirlip.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/supervisor.d/wordpress.ini
    Automerging trivial changes in: /home/douglas/topp/etc/builds/20080813/etc/opencore/zope_etc/zope.conf
    The following is the list of files which need updating, each
    configuration file is followed by a list of possible replacement files.
    1) /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/properties.xml (1)
    2) /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py (1)
    Please select a file to edit by entering the corresponding number.
                  (don't use -3, -5, -7 or -9 if you're unsure what to do)
                  (-1 to exit) (-3 to auto merge all remaining files)
                               (-5 to auto-merge AND not use 'mv -i')
                               (-7 to discard all updates)
                               (-9 to discard all updates AND not use 'rm -i'): 1

    File: /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml
    1) Replace original with update
    2) Delete update, keeping original as is
    3) Interactively merge original with update
    4) Show differences again
    Please select from the menu above (-1 to ignore this update): 2
    Deleting /home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml
    rm: remove regular file `/home/douglas/topp/etc/builds/20080813/etc/opencore/gs_profile/._cfg0000_properties.xml'? y

    The following is the list of files which need updating, each
    configuration file is followed by a list of possible replacement files.
    2) /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py (1)
    Please select a file to edit by entering the corresponding number.
                  (don't use -3, -5, -7 or -9 if you're unsure what to do)
                  (-1 to exit) (-3 to auto merge all remaining files)
                               (-5 to auto-merge AND not use 'mv -i')
                               (-7 to discard all updates)
                               (-9 to discard all updates AND not use 'rm -i'): 2

    File: /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/._cfg0000_config.py
    1) Replace original with update
    2) Delete update, keeping original as is
    3) Interactively merge original with update
    4) Show differences again
    Please select from the menu above (-1 to ignore this update): 1
    Replacing /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py with /home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/._cfg0000_config.py
    mv: overwrite `/home/douglas/topp/etc/builds/20080813/opencore/src/opencore-bundle/MaildropHost/config.py'? y

    Exiting: Nothing left to do; exiting.  

    ------------------------------------- SNIP -----------------------------------------

FINISHED
+12:53
    Installation successful.
    douglas@cezan ~/topp/etc $

The build was started. After twelve minutes, the system scans over the config files, determines which ones don’t need user attention, and prompts you for the only two files with any significant difference. Another minute later, and the setup tasks have complted.

When this completed last night, and I realized the amount of effort it would save me, it nearly brought a tear to my eye…

Now this post is titled a glimpse from the future, and that’s because the branch isn’t mature enough to merge it to trunk. There are still a couple of questions left before I’m ready to enable it for anyone. For example, fassembler keeps track of which projects are completed, and doesn’t rerun them when using the missing keyword. I’ve got to verify that the project isn’t marked complete until all of it’s phases have finished. Also, there isn’t yet a way to trigger a specific phase from the command line, so it’s an all or nothing approach. Finally, when running in non-interactive mode, the system stops after the build phase, which combined with the problem above, there’s no way to repeat the setup phase. Fortunately for me, this allows me to run builds without worrying about starting zeo on the same port, or modifying live data, but not everyone is building in exactly the same way as me.

All of my future builds will be using this branch (which you can acheive by switching fassembler over to the branch https://svn.openplans.org/svn/fassembler/branches/etc-update ). I hope that in the coming week or two, I’ll be able to polish it up enough to merge back to trunk…

Happy Building!

My first Django app

2008-08-13T00:15:11+00:00

I’m about to head off on vacation, so this seemed like as good a time as any to kick this out of my drafts folder…

As some of you know, I’ve been doing the brainpower project as a Django admin application.

The reasons for this decision were:

Django admin is touted as a very quick way to build CRUD applications, since it generates a UI from your model that in many cases is good enough for end users. No forms to write, maybe just a little tweaking. Brainpower is nothing other than a simple CRUD application, so this sounded like a perfect match.
Good excuse to learn a little about Django.
Get me to do something other than Zope for once in my freakin’ life.

So how did that pan out?

Well, in one month since our first requirements-gathering meeting with our “customer” (Liz and Robin from Streetfilms), in addition to all the other stuff I’ve been doing, I built something they said was good enough to start using. I did in really just a week or so of work, alone, from scratch, with almost zero advance knowledge of Django. I even spent some time testing and tuning performance (just enough that I feel confident we won’t ever have a problem with it). This also includes a full suite of Flunc tests; a random content generation script that I used for the performance tests; and a build script for development & deployment using Fassembler.

The core code is tiny: the bulk of it is in a 250-line models.py module that is reasonably clean.

As usual, writing the core code is only a small part of the story. A large portion of my time went to things like figuring out how to conveniently run external functional tests against django with a scratch database, writing and fixing the build script, and troubleshooting my initial attempts at deployment (tripping on a django bug.).

I do have some general early impressions of Django.

Things I liked

The admin interface is, indeed, pretty slick (with a few minor oddnesses like a pretty useless Time widget).
If I had another little CRUD app to build that seemed well-matched for Django, I could probably do it ridiculously fast.
The Django docs are pretty decent for the most part, much better than the current state of, say, Zope 2 docs, and more extensive and thorough than the Pylons docs. (Too bad I had to quickly un-learn a bunch of stuff when I had to switch to developing against more recent django checkouts.)
The stable release’s way to create an admin UI — by writing an inner class named Admin inside your Model — smelled really bad to me. Thankfully, this is gone in the newforms-admin branch. Similarly, you used to wire up custom validators inside your Model class and do cleanup in its save() method; now you can do custom validation in a ModelForm subclass, and you can do data cleanup in the same place. Newforms-admin is pretty nice!
Got a multiple choice field that you need users to be able to extend with new choices? Just add a foreign key field to your model referencing another model, and it just plain works in exactly the way you’d hope. Slick!

Gripes

I wish they had used an existing template language, I don’t see anything compellingly different about Django’s.
I wish Django’s setup.py had a “develop” command.
I wish the tools for syncing your models to your database were more developed. You seem to be entirely on your own for migrations. For example, if you modify the type of a field in a Model, you’ll likely need to either drop your tables and recreate them, or if you have production data to preserve, either do a dump-modify-restore cycle or hand-hack the database in place to keep your app working. At one point I ended up needing to do a dump-modify-restore using xml exports and lxml transforms; it took longer than I would have liked and I might try another strategy next time. For other model modifications such as adding a field, you might be able to get away with re-running manage.py syncdb; unfortunately I don’t know how to predict when this will just work and when it won’t.
I wish all Django manage.py operations could be performed non-interactively. For example, for my flunc tests I wanted to reliably create and destroy a scratch database with a test admin user. Django provides a way to do most of this, but the commands that create an admin user must be run interactively. I tried using a database fixture, but that didn’t work reliably — I’m wildly guessing there’s some salt that gets reset and I don’t know when? I ended up having my test script use pexpect to drive the interactive commands.
It seems I arrived at an inconvenient time. Django 0.96.2 may be the “latest stable release”, but it’s aging. The developers are all focused on a “newforms-admin” branch, which is cleaner and more extensible, but this work hasn’t landed on trunk yet. I don’t know when that will happen, or what else will change before another stable release. I opted to develop against the old stable release. I thought I was being smartly conservative :-p Unfortunately I soon hit a really irritating bug. While trying to understand the admin code enough to find a workaround (and feeling like I was on a familiar learning curve), I was advised to check if trunk still had the same problem, which meant an hour or two adjusting my code to work against trunk. Trunk turned out to fix only half of the problem. Then a week later the bug got fully fixed… on the newforms-admin branch. So much for trying to stick to a stable release! Maybe I was just unlucky. If I didn’t need to search fields from a join, I wouldn’t have hit this bug. (But isn’t that a common need?)
Customizing and extending the admin UI is commonly done in a way that also seems vaguely familiar: You just make a copy of the thing you want to modify and hack away on it. I’ve already had a hint of pain with this — my two modified template copies broke on both Django upgrades. That’s how things typically go in the Plone world: Every single skin override you did on a Plone site would add a drop of future pain to every Plone upgrade you ever tried to do.
This is a hard problem to solve. Just as Plone did, Django admin seems to be gradually adding more plug-in extension points so you don’t have to override the core templates as often; instead you just flip some configuration switch and/or add another template that magically gets slurped into the right place. Which has its own headaches, as every one of those bits of flexibility adds to the learning curve. Let’s hope Greenspun’s Tenth doesn’t grow a corollary substituting Plone for Lisp :-]

Opencore 0.13 released with Plone 3 support

2008-08-12T23:58:26+00:00

What’s New in Opencore 0.13

The biggest changes in this release are:

Plone 3 required!
Many things are noticeably faster, because Plone 3 is faster than Plone 2.5.
Lots of minor bugfixes and usability improvements

For more details, see CHANGES.txt in the opencore source.

This release also requires a data migration that is not backward compatible! Let me say that again:

This release requires you to run a data migration that is not backward-compatible.

Once you upgrade, you can’t go back.

Always Make Backups!

Ahem.

Also, if you have your own code that runs on top of opencore, you should plan on doing thorough testing and may need to modify your code to run under Plone 3.

How to Upgrade

First, make a backup of your ZODB.

Next, run a pre-upgrade migration as described here:

https://svn.openplans.org/svn/opencore/trunk/docs/Plone2.5-Plone3-migration.txt

Note that you need to be on Opencore 0.12.1 (or later, if we ever do an 0.12.2 etc.) to have that migration script available. So if you’re running an earlier version, you’ll need to upgrade to 0.12.1 first.

Then you’re ready to upgrade to 0.13. For this upgrade, I highly recommend leaving your old build alone in case something goes wrong, and create a new build from scratch in a parallel directory. As with previous releases, Opencore still doesn’t work via easy_install, you have to get it by checking out the tag from subversion: https://svn.openplans.org/svn/opencore/tags/0.13.0 and then install using python setup.py develop. You’d also have to install Zope 2.10 and our plone 3 Products tarball. Or use TOPP’s usual build tools which will do all that for you.

I highly recommend the latter approach, it’s a lot more convenient. We have requirements profiles ready for you to build:

For a pretty stripped down opencore build (just Opencore, Zope, Zeo, and supervisor; no Tasktracker, no Wordpress, etc.), you can use these requirements: https://svn.openplans.org/svn/build/requirements/opencore-minimal/tags/opencore-0.13.0
For a full-featured build like openplans.org, with wordpress and tasktracker and everything, you can use these requirements: https://svn.openplans.org/svn/build/requirements/openplans/branches/opencore-0.13

… although I’m not sure that’s the right place for that requirements set; Doug Mayle, the deployment manager, could overrule me and put it somewhere else. I’d suggest keeping an eye on https://svn.openplans.org/svn/build/requirements/releases where the final requirements are likely to show up once Doug does all his work to prepare for deploying to openplans.org. That would also be a good sign that we’re getting confident enough to deploy it ourselves

After you upgrade, you’ll need to run the post-upgrade migrations. Again, see:

https://svn.openplans.org/svn/opencore/trunk/docs/Plone2.5-Plone3-migration.txt

Start things up and you should be all set!

Special Thanks

The Plone 3 porting was almost entirely performed by Rob Miller. It was a big job, thanks Rob for moving us forward!

Support

As always, if you have any problems, contact us on the opencore-users mailing list or look us up on the #openplans IRC channel on freenode.

I will be on vacation until 2008/08/25. Good luck while I’m gone

– Paul Winkler

[from whitmo] Sam Ruby: Life after Bug Tracking Systems

2008-08-11T17:27:13+00:00

Interesting view on using your SCM to communicate.

[from whitmo] The Chandler Project Blog » Blog Archive » Chandler 1.0!

2008-08-10T22:22:57+00:00

omg... chandler beat opencore to 1.0!?!??!?! (insert pithy quote about the quantum reality of release numbers)

[from whitmo] Python Package Index : z3c.versionedresource 0.3.0

2008-08-10T22:12:40+00:00

Happy Birthday PotteryBlog

2008-08-10T19:15:15+00:00

A happy 4th Birthday Emily’s Blog!

Is there a good reason for this to be annoying?

2008-08-10T19:12:30+00:00

For some reason my imaginary version of SQL has some nice operations to take a random sample of rows from some table. In reality, the most common query that I’ve seen is something like:

SELECT * FROM sometable ORDER BY RAND() LIMIT samplesize

In MySQL, this appears to do just what it says: generate a random number for every row then sort them and return the lowest few — which is prohibitively slow for any reasonably big table.

If you know the identifiers of rows in the table and they form a gapless sequence, the story is somewhat better; you can generate samplesize random numbers from the range and select those identifiers. Of course, this is generally not the case (and almost begs the question). The best approach along these lines that I’ve seen requires maintaining a table that maps from a gapless sequence of integers to the identifiers of the table that you want to sample from using a bunch of triggers — http://jan.kneschke.de/projects/mysql/order-by-rand .

I’m also unsure of the right way to tackle this in databases like couchdb or the appengine datastore. Maybe a similar tactic with views/query indexes? Anyone implemented anything similar?

[from greebsnarf] virtualenvwrapper | And Now For Something Completely Different

2008-08-09T19:40:28+00:00

Commands for working with multiple virtualenvs (creating, removing, and switching active environments)

[from ianb] byCycle.org - Bicycle Trip Planner - Portland, OR, metro region

2008-08-07T17:12:51+00:00

The street data for this bicycle mapping seems to have improved (still just for Portland and Milwaukee though)

[from scrollie] nyc-travails-5016 on Flickr - Photo Sharing!

2008-08-07T01:44:22+00:00

I was skimming through a photoset on Flickr entitled "Rich People Rooftops NYC" and happened across this image of Lime HQ.

melkjug redesign concept

2008-08-06T15:53:05+00:00

Here’s some snazzy eyecandy from Phil’s work on sprucing up the melkjug user interface. Speaks for itself I think :)

[from cabraham] Malwebolence - The World of Web Trolling - NYTimes.com

2008-08-06T13:13:38+00:00

The Chicago GAE Hack-a-thon and App Engine wikis

2008-08-05T18:44:00+00:00

Last week I attended the Chicago Google App Engine Hack-A-Thon, which was a small day-long event to hang out at the Chicago Google offices and work on things related to App Engine.

I was hoping for something structured more like a coding sprint, where people would work together on small projects. I think some other people were also expecting something like this, but the day kind of wandered by without any clear “let’s start doing stuff together” moment, and the opportunity was lost. Some people followed a tutorial given by Marzia Niccolai, some worked on their own projects, and there was a bit of chatting.

I started working on a simple project I’d started to make a very (very) simple CMS/Wiki. After playing with Javascript for a little while I was chatting about wikis and MoinMoin came up. After a quick look at the code I realized it was very tightly bound to the filesystem, and lacked even the most basic abstraction layer for storage. MoinMoin should have an abstraction layer, but I wasn’t really inclined to work on that. Instead I thought: why not try a fake writable filesystem?

I knew from the start that the idea was a bit absurd. The App Engine data store isn’t much like a filesystem. Files are quick to read and write, okay for scanning, hard to query. The data store is okay to read and write, very slow to scan, and easy to query. But I figured there was some hack value to it.

I don’t know that I really accomplished my goal, but I did get some code in appengine-monkey. It kind of worked, but my strategy was probably wrong: I simulated an entire filesystem, except those places that were mingled with code (where templates would typically be stored). Instead I should have required explicit locations that would be handled by the data store (e.g., /wiki-data). Python doesn’t expect file operations to lead to a lot of Python routines, and there were some circular situations deep in the Python code as a result. I seemed to mostly work those out, but didn’t actually get MoinMoin to render anything, I only got it to work very hard on the slow process of setting up its files.

I’ve been a little annoyed with the App Engine environment from the start, because it left out lots of routines that are present in every other environment (e.g., os.utime). You can’t use these routines in App Engine because there is no file writing. But it would all be more sensible if every call to these routines just raised a permission error. This is the kind of error that existing code understands. Instead you can’t even import the routines, making App Engine incompatible with lots of existing code. That would be fine except the incompatibility is so trivial. appengine-monkey seeks to relieve some of this, but it would be much simpler if it was just there in the platform to start with.

Conclusion? Porting code is hard, but porting old, organically developed code like MoinMoin is really hard. And App Engine still needs a good wiki.

[from ianb] Python Package Index : EggFreezer

2008-08-05T16:33:35+00:00

Creates a bundle of all packages for an installation. Could be a nice complement to some other installation stuff we have.

moongifted

2008-08-05T15:25:35+00:00

Melkjug was featured on the japanese tech blog moongift yesterday — thanks guys! Here’s the original, and google’s translation. We probably aren’t as prepared for international visitors as we ought to be, but I hope everyone is finding the site to be usable. I put a quick patch through to fix up filtering using non latin parameters. Please let us know if you run into any problems! Suggestions ( and translations ^_^ ) are welcome.

[from closedplans] Label placement

2008-08-04T18:20:16+00:00

An interesting look at web forms, with an eye towards design and user interaction...

[from closedplans] Erlang services as middleware

2008-08-04T17:34:16+00:00

Getting data in and out of listen

2008-08-04T15:45:38+00:00

Robert Marianski and I have been working on adding the ability to import and export data from listen mailing lists^*. Specifically, we’ve been working on exporting mailing list subscribers to CSV files, exporting messages as mbox files, and importing mbox files into the mailing list archive.

By far the most interesting of these is importing mbox files into the mailing list archive. First (along with exporting) it allows for a new use case: Transitioning from one list to another. This gives users more flexibility and helps to avoid lock-in.

Second, it presents some challenging design problems. Importing can be seen in some respects as a relatively “dangerous” operation, as in the process it’s possible to accidentally import the wrong file or even import a corrupt file (this is in fact is something that’s gotten me in the past while migrating from one mail client to another; my “solution” involved manually deleting a lot of messages and then editing the source of the mbox file to fix the errors, hardly something that most users would ever want to do). In either case, your mailing list archive is now polluted with a bunch of messages you don’t want there. Worse, listen doesn’t currently support deleting messages from the archive, meaning once messages are in the archive, there’s no getting them out.

The question is, how do you prevent this from happening? One way to at least mitigate the issue is to give a confirmation warning before actually importing the messages into the archive. This may help in the event that the user accidentally selected the wrong file, but it’s hardly ideal and it’s still quite possible something could go wrong. Even better would be to show a preview of the messages that would be imported and let the user look at them to make sure they are correct. But this brings with it its own set of questions: How large should the preview be? Should the full messages be shown or just some of the headers? Do you make the user submit the file twice (once initially, once again when confirming) or do you store it on the server? What happens if the user never clicks anything on the confirmation page? What happens when the user confirms and then changes his mind?

Rob and I quickly realized that this was a case of using a warning when we really meant undo. What would be most useful for a user would be to be able to undo any import — at any time.

From a technical standpoint, this is actually not too complicated (save for a few nasty and hard to isolate Zope bugs, which I won’t bore you all with here…). All we had to do was store the message IDs, along with some other meta data (e.g., the time of the import), and provide an interface for removing just those messages from the archive. And this is exactly what we did: You can now remove any messages associated with any past import, even if you’ve subsequently imported other messages after them.

The branch we worked on is here, and here’s a screenshot to give you a better idea of what we did:

* There were a few days a couple weeks ago that I’ve missed chronicling so far. During those days Anil and I worked on the Feed Me plugin for WordPress, which makes it easy to incorporate feeds from Delicious, Flickr, and YouTube into any WordPress blog. The plugin has been used by Streetsblog for a while , but we generalized it and added a handy admin interface that lets you easily customize the number and type of feeds used. Check out the project page.

[from nicholasbs] Why Free Software has poor usability, and how to improve it

2008-08-03T16:35:29+00:00

A decent look at some of the challenges faced by the FOSS community in developing well-designed UIs, as well as suggestions of how we can do better.

A Doctest Wishlist

2008-08-01T04:11:11+00:00

Lately I’ve been doing most of my testing with doctest, primarily using stand-alone text files. I generally like it (otherwise I wouldn’t be using it), but it does make me frustrated with doctest sometimes. On my wishlist (roughly in order):

I wish output was always displayed, even when there’s an exception. I see no reason for the current behavior. Really exceptions could be treated like any other output (if ELLIPSIS was on by default).
I wish you could turn on options like ELLIPSIS from within a doctest, for all expressions. (# doctest: +ELLIPSIS on every line is beyond ugly.)
<BLANKLINE> is terribly ugly.
There’s no way of saying "I don’t care what this prints". You can’t do:
```
>>> some_function()
...
```
because the … is treated like a continuation.
Plugging in an alternate output checker is kind of tedious, and can’t be done from within a doctest (without horrible hacks).
I’d like to be able to easily jump into an interactive state from doctest. Maybe pdb can do this, but I’ve never figured that out exactly.
Getting nose to run .txt files as doctests is really hard, involving a combination of options I always forget.
There’s no way to abort the doctest. Sometimes I’d like to run some environment checks early on, and be able to stop the test if they fail.
I wish it was easier to apply to non-Python code. (I’ve adapted it via subclassing for Logo but I wouldn’t do that often.)
I wish I could copy and paste from doctests to consoles. But I don’t see any solution to this problem.
The integration with unittest is pretty hacky. Not that I’ve used unittest in years. But some other test frameworks build off this integration.
python -m doctest sometest.txt doesn’t do what it should do. Instead it runs doctest’s self-tests.

Comment on Moving to a new deployment structure by admin

2008-07-31T20:35:22+00:00

Yay. This also is a necessary prerequisite to running both stacks at once behind a load balancer, which we’ve been wanting for a while. RA is working on that.

OpenID Support is Coming

2008-07-31T19:54:41+00:00

A Melkjug user recently used the Feedback link on the site to recommend support for OpenID. Luke quickly forwarded the idea to the melkjug-dev list asking if anyone wanted to pick up the task. I stepped up.

Thankfully, there exists a very nice python-openid package. The documentation assumes some knowledge of OpenID specifications, so I had to do some digging there to understand how I was supposed to extract identity information from the response.

Here’s a nice screenshot of me logging in using OpenID:

The only tricky part to get working was Simple Registration. The OpenID specifications include a way to request Simple Registration information from the provider. In our case, we need a nickname and an e-mail address. Nicknames are used internally by Melkjug to find user’s jugs. For example, my full reading list exists at http://melkjug.org/tilgovi/full.

in some cases, the OpenID provider does not honor the Simple Registration request. In other cases, the user fails to supply the proper information. In these cases, Melkjug will ask for the additional required information before allowing you to sign up with your OpenID (see screenshot). If everything looks good, you’re in!

Look out for OpenID logins in the next release of Melkjug.

[ 10:29 Thursday, July 31, 2008 ]:

2008-07-31T10:29:27+00:00

trac is legos One of the first sentences on http://trac.edgewall.org is ...

melkjug 0.3 is released

2008-07-30T20:19:38+00:00

The 0.3 release focuses on lowering barriers to entry for new users based on the work Sonali and Bryan have been putting in. Most importantly, you can now check out the functionality of the site without creating an account and save what you make if you choose to sign up. We’ve also added a more informative “about” page, made the text more consistent overall and put better blank slates in place. There’s still a long way to go on this to get it where we want it, but I like how this first pass turned out.

For existing users, we’ve implemented some of the most requested features/fixes like sharing links via email / facebook etc, an option to keep you logged in after closing the browser, improvements to speed, and elimination of certain annoying duplicate articles. We have also moved all management of feeds back into the normal reading window and moved the detailed configuration of filters out of the sidebar and into lightbox dialogs.

Of particular note is Josh’s work to allow the tag filter to use multiple tags at once resulting from Bryan’s work trying to put together a decent demo. This makes tag filtering… actually kind of useful. Using this, you can group together related tags to represent a concept, or create clusters of synonymous tags to correct for differences between blogs. Enjoy!

try it out at melkjug.org — download it at melkjug.openplans.org

As always, feedback is appreciated

Making a proxy with WSGI and lxml

2008-07-30T16:52:22+00:00

You can use WSGI to make rewriting middleware; WebOb specifically makes it easy to write. And that’s cool, but it’s more satisfying to use your middleware right away without having to think about writing applications that might live behind the middleware.

There’s two libraries I’ll describe here to make that possible: paste.proxy to send WSGI requests out via HTTP, and lxml.html which lets you rewrite the HTML to fix up the links.

To start, we need some kind of middleware that at least is noticeable. How about something to make a word jumble of the page? We’ll use lxml as well:

from lxml import html
from random import shuffle

def jumble_words(doc):
    """Mixes up the words in an HTML document (doesn't touch tags or attributes)"""
    doc = html.fromstring(doc)
    # .text_content() gives the text without tags or attributes,
    # .body is the <body> tag:
    words = doc.body.text_content().split()
    shuffle(words)
    for el in doc.body.iterdescendants():
        # The ElementTree model puts all text in .text and .tail on elements, so that's
        # what we mix up:
        el.text = random_words(el.text, words)
        el.tail = random_words(el.tail, words)
    return html.tostring(doc)

def random_words(text, words):
    """Pulls some words from the list words, with the same number of words in
    the previous `text`"""
    # text can be None, so we need this test:
    if not text:
        return text
    word_count = len(text.split())
    try:
        return ' '.join(words.pop() for i in range(word_count))
    except IndexError:
        # This shouldn't happen, because we should have exactly
        # the right number of words, but just in case...
        return text

from webob import Request

class JumbleMiddleware(object):
    """Middleware that jumbles the words of HTML responses
    """
    # This __init__ and __call__ are the basic pattern for middleware:
    def __init__(self, app):
        self.app = app
    def __call__(self, environ, start_response):
        req = Request(environ)
        # We don't want 304 Not Modified responses, because we mix up the response
        # differently every time.  So we'll make sure all the headers that could call that
        # (If-Modified-Since, etc) are removed with .remove_conditional_headers():
        req.remove_conditional_headers()
        # This calls the application with the request, and then returns a response; this
        # is the typical pattern for response-modifying middleware using WebOb:
        resp = req.get_response(self.app)
        if resp.content_type == 'text/html':
            resp.body = jumble_words(resp.body)
        return resp(environ, start_response)

Well, you don’t really need to jumble up your own pages, right? Much more fun to jumble other people’s pages. Enter the proxy. Here’s a basic proxy:

from paste.proxy import Proxy
# We use this to make sure we didn't mess up anything with JumbleMiddleware;
# the validator checks for many WSGI requirements:
from wsgiref.validate import validator
import sys

def main():
    proxy_url = sys.argv[1]
    app = JumbleMiddleware(
        Proxy(proxy_url))
    app = validator(app)
    from paste.httpserver import serve
    serve(app, 'localhost', 8080)

if __name__ == '__main__':
    main()

If you look at the full source the command-line is a bit fancier, but it’s all obvious stuff.

OK, so this will work, but the links will often be broken unless the server only gives relative links. But you can rewrite the links using lxml…

import urlparse

class LinkRewriterMiddleware(object):
    """Rewrites the response, assuming the HTML was generated as though based at
    `dest_href`, and needs to be rewritten for the incoming request"""

    # The normal __init__, __call__ pattern:
    def __init__(self, app, dest_href):
        self.app = app
        if dest_href.endswith('/'):
            dest_href = dest_href[:-1]
        self.dest_href = dest_href

    def __call__(self, environ, start_response):
        req = Request(environ)
        # .path_info (aka environ['PATH_INFO']) is the path of the request
        # (URL rewriting doesn't really have to care about query strings)
        dest_path = req.path_info
        dest_href = self.dest_href + dest_path
        # req.application_url is the base URL not including path_info or the query string:
        req_href = req.application_url
        def link_repl_func(link):
            link = urlparse.urljoin(dest_href, link)
            if not link.startswith(dest_href):
                # Not a local link
                return link
            new_url = req_href + '/' + link[len(dest_href):]
            return new_url
        resp = req.get_response(self.app)
        # This decodes any possible gzipped content:
        resp.decode_content()
        if (resp.status_int == 200
            and resp.content_type == 'text/html'):
            doc = html.fromstring(resp.body, base_url=dest_href)
            doc.rewrite_links(link_repl_func)
            resp.body = html.tostring(doc)
        # Redirects need their redirect locations rewritten:
        if resp.location:
            resp.location = link_repl_func(resp.location)
        return resp(environ, start_response)

Then we rewire the application:

app = JumbleMiddleware(
    LinkRewriterMiddleware(Proxy(proxy_url), proxy_url))

Now there’s a fun little proxy for you to play with. You can see the code here.

[from whitmo] Wide Awake Developers: Beyond the Village

2008-07-30T14:39:38+00:00

150 is generally accepted as the dunbar number for homo sapiens (http://en.wikipedia.org/wiki/Dunbar's_number).

[from rafrombrc] Catalog query plan — ZCatalog performance improvements

2008-07-29T18:10:40+00:00

[from closedplans] Rich HTML Editing in the browser

2008-07-28T18:24:49+00:00

A great look at the state of web based WYSIWYG editors...

Me In Berlin & Amsterdam

2008-07-28T16:22:32+00:00

I’m going on vacation in a couple weeks to Berlin and Amsterdam, flying to Berlin on August 12, leaving from Amsterdam on August 27, and transitioning between the two sometime in between. Things Emily and I should see or do? Care to meet up? On vacations my evenings tend to be underscheduled.

Greylisting for comments

2008-07-28T06:56:09+00:00

Greylisting is an interesting idea that comes from the world of mail servers. It’s a system used to combat SPAM that’s quite ingenious, and at least on my mail server, is 99% effective. It’s very effective at blocking SPAM for three reasons:

The internet protocol used for sending mail (SMTP) is quite complex. Most spammers don’t have the time to write complete mail servers, they instead take shortcuts to cover the majority of cases.
Spam is about turning computer time into money. Spammers send out millions of mails per day, so if you increase the cost (in time) of sending mail, than you make spamming less attractive.
While both whitelisting and blacklisting require humans to maintain lists of good and bad servers, greylisting is completely automated. Since it’s automated, it’s easy to use.

They way greylisting works is by keeping a database of people sending mail to your server. For each mail it receives, it looks at three things:

The person sending the mail
The person receiving the mail
The computer performing the delivery.

If the server doesn’t already recognize all three of these properties, it responds with an error that tells the sender to try back a little bit later. Real email servers will try again shortly, usually in less than 15 minutes. A good number of spammers are stopped right here because their spam tools don’t handle this case. When the real server tries again, this time the mail will just pass right through and be delivered.

That’s it! That’s the magic of it all. For any mail coming from people that your users already know, there’s no wait; they don’t see any difference, and mail just keeps coming in. The first time someone sends a mail to your users, there will be a short wait, normally less than 15 minutes, and since mail isn’t guaranteed to be immediate, most people don’t notice the difference.

Now on top of greylisting, people often throw in Tarpitting. A tarpit in computers is something that slows down the server, so the server responds more slowly, as if it were under a heavy load. When combined with greylisting, this means that each mail coming from a new source costs the sender a whole lot more in computer time. In the case of someone who will be sending you mails regularly, this one-time cost is quickly amortized, costing the sender nothing in the long run. Spammers, however, who depend on sending millions of unique mails, see this cost with each email they send, and so your server becomes an unattractive target.

How does this relate to comments, you may ask? Well, I’ve written a a greylisting/tarpitting Django-app for this and patched the code for this blog to use it. For now, you can download it here: http://douglas.mayle.org/files/greylist.tgz

If you’d like the patch to enable this for your byteflow blog, it’s available at Byteflow Trac Ticket #93

[from whitmo] No Secret Software!

2008-07-28T02:56:42+00:00

right up our alley

[from paulhope] The brave new world of e-hatred

2008-07-26T03:59:07+00:00

The Economist rights about the dark side of internet politics

[from whitmo] Talks about Prophet and Sharecropping

2008-07-24T23:45:57+00:00

oops... now we are just another sharecropper...

opencore 0.12.1 released

2008-07-24T18:15:32+00:00

Hi folks, opencore 0.12.1 was released today.

This is a bugfix release with no user-visible new features.

As with the previous release, it still doesn’t work via easy_install, you have to get it by checking out the tag from subversion: https://svn.openplans.org/svn/opencore/tags/0.12.1 and then install using python setup.py develop. Or use TOPP’s usual build tools which will do that for you; however, we don’t yet have a requirements profile ready that uses this tag. You could modify requirements/opencore-req.txt if you have an existing fassembler build that you want to upgrade.

The biggest changes in this release are:

Force project content deletion to be done via POST; if it’s a GET, redirect to a confirmation form. Reduces chance of accidental content delete by non-javascript clients. Thanks to Alex Clark for being so helpful and patient with us on this one.
Added migrations/unmake-sites.py script to unmake the local sites at the app root and the Plone site root. Run this JUST BEFORE updating to a Plone 3-based version of opencore, which will be the next feature release, version 0.13. Do not run this migration under any other circumstances or you will break your site! For more info, see: https://svn.openplans.org/svn/opencore/trunk/docs/Plone2.5-Plone3-migration.txt

As always, if you have any problems, contact us on the opencore-users mailing list or look us up on the #openplans IRC channel on freenode.

I will be on vacation until 2008/07/31, I’m hoping my fellow TOPP employees can provide support while I’m gone.

– Paul Winkler

Moving to a new deployment structure

2008-07-24T17:20:12+00:00

While working on today’s micro-deployment, I decided I need a few diagrams to make it clear to myself what was really going. Since I went to the effort of making them, I thought I’d share them here with everyone.

Below is the deploy structure we’ve been using for quite a while. There are some more pieces to the puzzle, but for the purpose of this deployment, they’re irrelevant. Notice that we have multiple complete setups, but they all exist within an analogous space. Only one set can be running at a time, and squid is always pointing to whichever stack is running.

The current live deployment is a little bit messy, since we have two side by side setups, but with both zope instances pointing to the same instance zeo. While providing the side by side deployment that I prefer, the intertangling is messy and just not sustainable.

This is the deplyment we’ll have after today. There will be a separate zeo instance that both zope instances will point to. Care has to be kept to give zeo access to any code that performs zeo write conflict resolution. Currently, the only Product we use that does is the QueueCatalog…

[from slinkp] Palladion Software: Avoiding Temptation: Notes on using 'unittest' effectively

2008-07-24T16:02:11+00:00