The Promise
In preparation for a ramp-up in testing the OpenPlans Plone 3 upgrade efforts, I’ve been revisiting OSAF’s Windmill functional testing tool. The story is compelling; if you’re going to be launching a browser and clicking around to test a site anyway, why not turn on a recorder that will auto-generate test suites that can reproduce your actions? Even if it ends up making the original testing take a little while longer, that pays for itself the first time you re-run the tests.
With something concrete to accomplish, then, I sat down to see if Windmill delivers on this promise.
Bootstrapping
Getting started went pretty smoothly. Installing was easy, on Ubuntu Gutsy, anway; just regular setuptools stuff. I’m working from the Windmill trunk, so I created and activated a python 2.5 virtualenv, checked the code out from http://svn.osafoundation.org/windmill/trunk/, and ran ‘python setup.py develop’ to install it into the environment.
Windmill is then launchable with ‘windmill firefox URL’. This will open firefox, albeit without any of the customizations that you might have set up in your profile. It also opens a smaller Windmill IDE browser window, and starts a Windmill “controller service”, which exposes the API for manipulating the browser window.
Capturing Tests
The IDE is pretty simple. It’s implemented in HTML and Javascript, and it provides 4 primary features: a test recorder, a test runner, a DOM explorer, and an “Assert” explorer. The test recorder is what I’m first interested in. I click the record button, and the browser window jumps to the foreground. So I start clicking around, entering text and submitting forms, lo and behold, I can see what I’m doing being captured in the IDE. Good start.
To start, I make an incorrect login attempt, and, upon failure, try to create a new account, getting all the way to the “check your email” confirmation screen. I turn off the recorder, and click on the ’save’ link in the IDE window. Windmill can export the test suites as either python or JSON. I’ve chosen python, so I get another window with the following text:
# Generated by the windmill services transformer
from windmill.authoring import WindmillTestClient
def test():
client = WindmillTestClient(__name__)
client.click(link=u'Sign in')
client.type(text=u'bogus', id=u'__ac_name')
client.type(text=u'bobobobo', id=u'__ac_password')
client.click(name=u'login')
client.click(link=u'Create account')
client.type(text=u'testuser1', id=u'id')
client.type(text=u'Test UserOne', id=u'fullname')
client.type(text=u'test1@example.com', id=u'email')
client.type(text=u'testy', id=u'password')
client.type(text=u'testy', id=u'confirm_password')
client.click(name=u'task|join')
I paste this into an emacs buffer and save it as windmill_tests.py. Then I shut down the windmill process (I have to ‘kill’ it to make sure everything terminates correctly :P) and pass in the test I just created with ‘windmill firefox URL test=windmill_tests.py’. Sure enough, the browser launches, hits the site, and starts following links and entering text! Initial success makes much happy.
Controller API
The ‘client’ variable in the python code above is a handle to a WindmillTestClient object, which exposes the Windmill Controller API. The controller is what is driving the browser. One fun trick is that you can put a pdb in your test code and you’ll get an interactive prompt which you can use to control the live browser window. Now enjoy the power of a python command line interface to teh interwebs, while still being able to access all the rich multimedia that today’s users demand! It’s definitely more fun than it should be to fill out web forms and click on links by typing in ‘client.type’ and ‘client.click’ commands.
Assertions
Astute readers will have noticed that the second run of the user creation test would have a different result than the first. Indeed, when I used Windmill to run the test steps I’d captured, it ended with an error message informing me that the username was already taken. Real tests of course will want to make assertions, to ensure that the behaviour is really what you want. Enter the Assertion Explorer. At any point, you can click on the Assertion Explorer button, and then click somewhere on the page, and the IDE will generate a best-guess assertion for you for the element that you selected.
The generated assertions are sometimes spot-on. Whenever I clicked on a portal status message, for instance, it asserts (using XPath) that the PSM element exists on the page, and that it contains the specified text. When I click on regular page text, however, it only checks for the existence of the clicked node, when I want it to check the text as well.
The asserts are a part of the Controller API, and are pretty easy to understand. Here’s an example of a couple that I generated:
client.asserts.assertNode(xpath=u'/html/body/div/div/div/div/div')
client.asserts.assertText(xpath=u'/html/body/div/div/div/div/div', validator=u'Welcome! You have signed in.')
client.asserts.assertNode(xpath=u’/html/body/div/div/div/div/div’)
client.asserts.assertText(xpath=u’/html/body/div/div/div/div/div’, validator=u’Your changes have been saved.’)
client.asserts.assertNode(link=u’something else+’)
Nothing particularly mysterious there.
The Warts
Despite the early successes, it wasn’t long before I hit a couple bumps. Here’s an overview of the issues that came up for me:
-
Hard to extract info from the page
The Controller API works pretty well for controlling the page and making assertions, but there don’t seem to be very good hooks for extracting information from the page itself into the python environment. For instance, in order to complete the user registration, the test will need to log in as an admin and hit a special page which will return the user’s confirmation key. This will need to be extracted from the response and then used as part of the URL of a subsequent request. Even a simpler case, such as simply extracting the URL of the page that the browser is currently visiting, doesn’t come clear to me from a thorough comb of the documentation.
-
The ‘waits’ stuff seems to be broken ATM
The Controller API has a whole selection of ‘waits’ methods, which will tell the test running to not move on until some criteria is met, either a specific amount of time has elapsed, or the page has loaded, or an element shows up on the page, etc. For me, running FF2.0.0.14 on Ubuntu Gutsy, these were completely broken. Any time I try to use one of these calls, the test suite just stops right there. This is a show-stopper, since I very quickly hit false failures due to the tests running more quickly than the browser was responding.
-
Xinha typing didn’t get picked up by the recorder
Most of the actions I performed in the browser were dutifully recorded by the IDE. The only exception to this is any text I would enter into the Xinha editor. Whenever I would edit a wiki page, the recorder would skip directly from clicking the ‘edit’ link to clicking on the ’save’ button. I think we can work around this by just adding these commands by hand to the generated python. I’m not 100% on this, though.
Python is Nice
As I’ve mentioned, it’s possible to export the generated tests and assertions as either JSON or python code. Having such a straightforward way to drive the tests from pure python is a big plus for those of us who like that sort of thing. We can use try:finally to make sure clean-up code gets hit, and httplib2 to talk to the server to actually perform the clean-up. Control is easy in python. A quick peek at the code indicates that it does pass unrecognized options in to the test suites themselves, which means that we can code up test suites that understand additional control parameters as we need.
The Good News
While I did stumble on a couple of issues, I am very happy to report that the folks in the #windmill channel on freenode are very helpful. Even better than that, they’re very happy to get my bug reports, and are very responsive about fixing them. I spent about a day playing with Windmill a few months back. In that time I uncovered one bug and one usability weirdness. They were both fixed within days. The issues I’ve raised this time are already recorded as issues in their tracker, and I’m told that they should be resolved by the end of the week.
Conclusions
Takeaways? I’d say “cautiously wildly enthusiastic” best describes how I feel. Windmill is very close to delivering on its promise, making it ridiculously easy to generate robust, JavaScript-supporting test suites that can be trivially run on IE, Safari, and FireFox. If the pattern of developer responsiveness continues, and the issues that come up are either easily worked around or are resolved within Windmill itself, then I think this really hits the sweet spot. It didn’t take long for me to hit a couple of pretty big issues, however. If it turns out that there are a lot of similar bombs in there, that the problems stack up faster than the developers can deal with them, then it would probably end up being a headache for us.
I don’t really expect this, however. I know OSAF is using Windmill internally, and the developers that I’ve had contact with seem very enthusiastic about getting my reports and getting the problems resolved. I’m going to continue my exploration, and I hope to generate a thorough set of Windmill tests for the Plone 3-based OpenPlans stack over the next week.
