Lesson 4: Unit and Functional Tests

Teacher: Paul Carduner

Students: Brittney, Will, Preetam, and Linda

Paul:

Hello, and welcome to the fourth week of our Zope3 class. First we will briefly go over what we covered last time, and then get into some work.
Last week we made auto generated forms for the contact object and built the user interface into the ZMI.
Today we will learn how to automatically test the code that we have already written using both unit tests and functional tests. So far, we have been testing our code, but only manually - by clicking on links in the ZMI and trying it out for ourselves. Although this can work fine for a *really* small project, it is always best to write automated tests that can do more tests, more thouroughly, much faster.
Automated testing is also a vital step in the process of Test Driven Development, which in turn is a vital part of the practice of eXtreme Programming. So far we have not done any test driven development and we have really been very bad XP'ers. But it was important to sidestep testing at first so as to not overcomplicate the first steps in creating a zope application. In future projects, what we do in this lesson should be done before any of the other lessons.
An important aspect about testing in zope is that today people predominantly use what are called DocTests. As you can probably guess, DocTests combine both documentation and tests. Imagine that you were walking someone through how your program works. In essence, as you walked through it with the person, you would also be trying out every feature - and therefore testing it. Doctests just take this process and formalize it into something that can be automated. In zope3, looking at doctests is a fantastic way to learn about how something works and how to use it. But enough blabbing for now, let's get on with it.
Before we write the actual tests, we are going to have to do a bit of set up. The first thing to do is to create a file called tests.py. The file should look like this:
import unittest
import zope.testing.doctest

def test_suite():
    return unittest.TestSuite((
        zope.testing.doctest.DocFileSuite('README.txt',
            optionflags=zope.testing.doctest.NORMALIZE_WHITESPACE |
                        zope.testing.doctest.ELLIPSIS),
        ))
The main purpose of this file is to specify where our doctests will be located; specifically it says to look in the README.txt file. The one function called test_suite returns a TestSuite object. The constructor for the TestSuite object takes as its first argument, a tuple of tests, or other test suites. That is, test suites can be nested inside each other. We only pass a tuple with one test suite, a DocFileSuite object, to the constructor. All the tests will be contained in a file called README.txt. It is standard practice to put most unit tests in files called README.txt, although sometimes you will find them in other files - but always with a .txt extension. The optionflags keyword argument specifies some options for how the test should be run, we will talk about what these options do after we have written the actual tests.
The next thing to do is to create and edit the README.txt file. Here we will be testing the functionality of our objects. At the moment we only have one object - the Contact object, so the test won't be very long. In the README.txt file, write the following:
The Z Contact Application
=========================

This application stores contact information for people.

Creating Contact Objects
------------------------

A contact object stores a person's contact information.  The attributes of the
Contact class provide the ``IContact`` interface.
So far our doctest looks more like documentation that tests. At the top of every doctest you should always put a header describing something about the program or module being tested. The underlined sections are headers, one for the entire file, and one for each segement of the test. Depending on what editor you are using, it is possible to have syntax highlight for the headers, which makes them easy to find when you are looking for a specific set of tests. But now to write the tests. Continue editing README.txt by adding the following:
  >>> from zcontact.contact import Contact
  >>> from zcontact import interfaces

  >>> paul = Contact()
  >>> interfaces.IContact.providedBy(paul)
  True
  >>> interfaces.IContact.implementedBy(Contact)
  True

Linda:

And let's all not forget to separate python test code and text with blank lines.

Brittney:

How many spaces do we need before the >>> symbol? And when do we need to use it?

Paul:

I only added two, Brittney. In most cases, the documentation is not indented, but python code is.
As you should notice, what we have just written looks like a session from a python shell! Doctests work just like using the python shell, except that you get to write what the output should be, and you get to annotate what your commands with documentation.
So let's talk a bit about how these tests work. The doc testing module, which processes these kinds of files, parses the file looking for bits of python code (whatever follows >>> ). Then it will run these snippets of python line by line. If there is output underneath a snippet, then it checks to see if the output written in the doctest is what python actually would output had you written the command in a python interpretor shell. If they don't match, then an error is thrown and your test failed.
The first thing you should always test for your object is that it actually implements the IContact interface. We make sure that an instance of the object provides the interface, and that the class definition of the object implements the interface.
Now let's move on and add just a few more tests:
In providing the ``IContact`` interface, a ``Contact`` instance has the
following attributes which default to empty unicode strings.

  >>> paul.firstName
  u''
  >>> paul.lastName
  u''
  >>> paul.email
  u''
  >>> paul.phone
  u''
  >>> paul.address
  u''
Here we test that the attributes specified in the IContact itnerface actually exist in the implementation (Contact). They should be initialized to empty unicode strings. Remember that the constructor for the Contact class did not take any arguments to specify these values from the beginning.
Now let's try running these tests. To run the tests we will be using the zope test runner, which actually searches through a directory tree for tests and runs all of them automatically. To run the test, go to the top directory of your zope3 instance. For those following along with maddog accounts, it should be /home/yourname/zope3/. Then type the following:
./bin/test -uvvp -s zcontact
The u part stands for unit tests, vv part stands for "very verbose", and the p option gives a progress bar. The -s option allows us to search a particular package for tests. What the testrunner actually looks for are files called tests.py - not for README.txt. That is why we have to specify the README.txt file in the tests.py file. We often only want to test one python package (in the case that we have several in our instance), so this option is one to remember. The output of running the tests should look something like this:
Running tests at level 1
Running unit tests:
  Running:

  Ran 1 tests with 0 failures and 0 errors in 0.032 seconds.
Now try changing some line in the tests, or some line in your Contact class so that the tests fail. Then run the tests again and see what happens.
Now that we have tested our Contact object, we will start testing our user interface. Testing the user interface has historically been much more complicated because automating clicks on a screen is tricky business, and making sure the page looks a certain way is equally tricky. As of now, you can't have a super intelligent robot with eyes that does everything for you. But not to worry, doctests will come to our aid once again.
When the zope testrunner runs functional tests, it looks for files called ftests.py, so the first thing we will do is create and edit the ftests.py file. It should look like this:
import zope.app.testing.functional

def test_suite():
    return zope.app.testing.functional.FunctionalDocFileSuite("browser.txt")
We always put functional tests in a separate file from the unit tests, because they often take much longer to run and we often want to run each separately. So, as the ftests.py file suggests, create and edit the browser.txt file.
The browser.txt file is going to have the same format as the README.txt and will work just like typing stuff into a python interpreter shell. To start us off, type the following into the browser.txt file:
The Z Contact Application - Functional Tests
============================================

These are functional tests for the Z Contact application.  They test the user
interface.

Creating a Contact Object
-------------------------

  >>> from zope.testbrowser.testing import Browser
Zope provides us with a great way to write functional tests using a fake browser. The Browser class that we just imported works just like a real browser that allows us to go back and forward, click on links, enter information into forms, and so on. If you want to know how this works, and at the same time see a really excellent example of what a (unit) doctest should look like, look at /usr/local/src/Zope3/src/zope/testbrowser/README.txt, which is the README.txt file for the testbrowser package in zope.
Now let's move on with the test. The next thing to write is:
Here we set up the browser object and open up to the main ZMI page.

  >>> browser = Browser()
  >>> browser.addHeader('Authorization', 'Basic mgr:mgrpw')
  >>> browser.handleErrors = False
  >>> browser.open('http://localhost/manage')
We first instantiate a browser object. Then we log in by adding an Authorization header. When running functional tests, the default login is username: mgr and password: mgrpw. Next we set handleErrors to false so that when there is a problem with displaying a page, the test will throw an error and stop - rather than the zope server just serving up a page that says System Error. Next we open up the ZMI in our page. Note that we use the url http://localhost/manage to access the ZMI. When running the functional tests, the browser object does not connect to an actual zope server you are running, so the url is faked to always be localhost, on the default port 80.
Now we will write our first real test (which is going to fail). Let's say that once we've logged into the ZMI, and click on the Z Contact link in the add menu, instead of just creating an empty contact object right there, we want to be taken to a form that allows us to intialize the information to store in the contact. Rather than implementing this right off the bat, we will write a functional test that describes the behavior we would like. Now add the following to browser.txt:
Under the add menu, we should see a link for Z Contact.  Clicking on it takes
us to a form where we can enter information for a new contact.

  >>> browser.getLink('Z Contact Page').click()
  >>> browser.getControl('Last Name').value="Carduner"
  >>> browser.getControl('First Name').value="Paul"
  >>> browser.getControl(name='add_input_name').value="paulcarduner"
  >>> browser.getControl('Add').click()
So, if we were in a real browser, what we would have done to test out this feature is to find the link that says 'Z Contact' (this is the getLink part), and click on it (the .click() part). Next, we expect that the page this link takes us to has Controls (form input elements), with the labels 'Last Name' and 'First Name'. We can then type our name into these input fields by setting the value attribute of the Control. Finally we will type in the name of our object as it should appear in the ZMI, which is in the field add_input_name. When a form element does not have a label associated with it, we have to specify the exact name of the element. In this case, I figure out that it was add_input_name by looking at the html source generated from another add form. Finally we should be able to click on an 'Add' button to actually add the form.
Go ahead and run this test. To do so, type in the shell
./bin/test -fvvp -s zcontact
This time we have an f instead of a u, which stands for functional tests. Note that running this short test takes quite a bit of time because the functional tests start up a fake zope server and intialize a fake empty database. This test should fail because we have not actually implemented the code to make it work. This is called Test Driven Development, because the failing tests drive what we implement.
So now let's make this test pass! Implementing an add form is a lot like implementing an edit form, which we did in class last week. To do this, we will have to edit the configure.zcml file. So open it up for editing. Then add the following tag:
  <browser:addform
      label="Add Contact"
      name="addContact.html"
      permission="zope.ManageContent"
      schema="zcontact.interfaces.IContact"
      content_factory="zcontact.contact.Contact"
      />
The difference from the editform tag to pay attention to is that we do not have a for attribute, and we now have a content_factory attribute. The content_factory specifies the constructor for our object, which in this case is just the __init__ method of the Contact class.
Finally, we will have to change one more thing: the addMenuItem tag. You should add the following attribute to the addMenuItem tag:
      view="addContact.html"
This attribute tells the addMenuItem tag to point to the addform we just created. Now rerun the test and all should work well. Feel free to start up your zope instance and check it out manually for yourself by adding a contact through the ZMI.
Now back to more functional test writing. After adding the zcontact object with the addform, we should be able to click on a link to it that takes us to a page displaying information stored in contact. So let's write that as a functional test. Add the following lines to browser.txt
After adding the contact, we can click on the newly created object, which
takes us to a display page for the contact.  This display page shows the
information stored in the contact.

  >>> browser.getLink('paulcarduner').click()
  >>> print browser.contents
  <BLANKLINE>
  ...Z Contact...
  ...Name:...
  ...Carduner, Paul...
The three dots are called ellipses'. They are basically wild cards in functional tests. The browser.contents attribute holds all the html returned to the browser. We don't care so much about what the *exact* html is, so we use an ellipses as a wild card for any html.
Finally we will add one more test for the edit form. This should look like this:
You can edit the information for the contact by clicking on the 'Edit Contact'
link in the tabs along the top.  Here you can enter information for all the
fields you saw in the add form.

  >>> browser.getLink('Edit Contact').click()
  >>> browser.getControl('Phone').value = "703-xxx-xxxx"
  >>> browser.getControl('Email').value = "pcardune@polytope.com"
  >>> browser.getControl('Address').value = "Career Center, Arlington, VA"
  >>> browser.getControl('Change').click()

Once we have changed the information, we have to click on the 'View' link to
get back to the display page.  Here we should see the updated information.

  >>> browser.getLink('View').click()
  >>> print browser.contents
  <BLANKLINE>
  ...Z Contact...
  ...Name:...
  ...Carduner, Paul...
  ...Email:...
  ...pcardune@polytope.com...
  ...Phone:...
  ...703-xxx-xxxx...
  ...Address:...
  ...Career Center, Arlington, VA...

Files:

When you click on the links belove you will be presented with text versions of these files. If you choose to save these files from your web browser, be sure to put the files in the proper directory structure.

A tarball of all these files is located here: lesson04.tgz

Previous Lesson

Front Page

Next Lesson