Monday, May 29, 2006

 

My To-Learn List

Stress and other causes of unhappiness at work for the past few months have caused me to neglect "personal development." Through the years, I've always spent a lot of my free time learning more about my craft, but recently I've started hating computers so much that I can't bear to do any programming or study at home.

That needs to change. I'm making a list of things I'd like to explore that I can refer to when I get those "I'm so tired of C++ and Windows I could scream" rages:

  1. OCaml
  2. Haskell
  3. Eclipse
  4. AJAX
  5. MySQL internals
  6. OpenGL

A lot of this knowledge won't be "marketable." That's fine—I'm doing this for myself, not for my current or future employers. If all they want out of me is C++ and Windows, then that's all I'll give them.


Thursday, May 25, 2006

 

Early Bird

We have an entry field in our application that allows the user to enter a time of day. By default, it comes up with the current time. We received reports from QA that the entry field often came up with "garbage." After a few days of requestnig additional information, they gave us an example:

  72:6-

Unfortunately, we couldn't reproduce this in our development environment. We examined code, we tested the application extensively, but couldn't get it to show us this behavior. Every time we tried it, we'd get a good-looking time, like this:

  11:35

This remained an open issue for a couple of weeks.

Finally, last night while driving home from work, it hit me. The problem was that the field was not properly handling the case where the hour needed a leading zero. When it was displaying "72:6-", the actual time was "07:26".

Why couldn't we developers reproduce it during our own testing? Because we are never in the office before 10:00!


Sunday, May 21, 2006

 

Introduction to Fit and FitNesse

A few years ago, Ward Cunningham gave me an early version of his Fit (Framework for Integrated Tests) system, telling me he thought it was a great way to deal with software acceptance tests. At that point, it was a lot of code, with little documentation, and I just didn't have the time to figure out what it did. I looked at the simple tutorials and the simple examples, but I couldn't figure out exactly what it was or how I would use it.

Over time I've glanced at it a few more times, but still couldn't grasp the big picture. Recently, I discovered that Ward and a co-author had written a book about it: Fit for Developing Software: Framework for Integrated Tests (also known simply as "the Fit book"). Having nothing else to do this weekend, I read the book, and now I get it.

I've decided to write the introduction that I wish I could have read a couple of years ago. Maybe it will help somebody else with the same mental blocks I had to see how useful Fit is.

Unit Testing vs. Acceptance Testing

Most software developers are familiar with some form of unit testing, where one exercises a piece of code and verifies that it generates the expected results. A lot of people just do ad-hoc printf-style tests, or just run the code in the debugger to verify that it does the right thing, but enlightened people use a unit-testing framework that allows programmers to define automated repeatable tests.

While unit testing is useful for verifying that the software works like the programmers think it should, it is not useful for verifying that the software does what the customer wants it to do. This is where acceptance testing comes in. Acceptance tests take many forms:

In short, unit tests represent the developers' idea of what the system is supposed to do, and acceptance tests represent the customers' view of what they need the system to do. Bridging the gulf between the developers and customers is often difficult, because developers don't understand the customers' needs, and customers have trouble expressing their needs in sufficient detail and formality for developers to write code that meets those needs.

Fit's Approach to Acceptance Tests

The Fit framework addresses this problem by providing a means for customers to define their acceptance tests in a way that is formal enough for developers to create automated tests. While the test definitions have some formality to them, they are still simple enough that non-programmers can create the tests and run them themselves. The tests become the common language that customers and developers use to explore the requirements of the system, and become the measure of compliance and completeness.

Basically, tests are defined as tables, meaning in a 2x2 matrix. Test tables can be created in a spreadsheet, or using an HTML editor. There are several standardized formats for the tables, but Fit is generic enough that it can be used on anything that is in table form.

For example, say we are developing a cash-register system for a grocery store, and we want to define tests for a typical sales transaction. We can start by creating a table that sets up the starting inventory for the store:

Initial Inventory
Product CodeDescriptionPriceNumber in StockAdd?
001Tomato0.49200true
002Milk1.7950true
003Beer5.9950true

In Fit, this is known as a column fixture, containing a set of rows containing input data (Product Code, Description, Price, Number in Stock) and expected output. Here, the expected output of the Add() function for each item is "true", meaning we always expect the Add() function to work when this table is used in testing.

Next, the customer creates a table that illustrates a typical grocery-store sales transaction. The initial English description of this test case might be "Customer buys two tomatoes and a quart of milk, paying with a $20 bill and receiving the correct change." After thinking it through for a while, we end up with this:

Action Fixture
startCustomer Sale
enterproduct code001
pressproduct lookup key
checkdisplay descriptionTomato
checkdisplay price$0.49
pressquantity key
press2 key
checkdisplay descriptionTomato
checkdisplay quantityX2
checkdisplay price$0.98
enterproduct code002
pressproduct lookup key
checkdisplay descriptionMilk
checkdisplay price$1.79
presssubtotal key
checkitems total2.77
checktax0.20
checktotal2.97
checkdisplay descriptionSUBTOTAL
checkdisplay price$2.97
entercash tendered2000
presscash sale key
checkcash tendered20.00
checkchange17.23
checkdisplay descriptionCHANGE
checkdisplay price$17.23
checkreceipt printedtrue

Note how much more detailed this is than the original description. The customer has been able to specify which buttons get pressed, what should show up on the LED display at each point in the transaction, internal calculations by the register, and whether a receipt should be printed. We've even captured little details like allowing $20.00 to be entered as "2000" instead of requiring the clerk to enter a decimal point.

The customer wants to verify that the store's inventory data gets updated, so we'll add one more table to check the results:

Check Inventory
Product CodeDescription?Price?Number in Stock?
001Tomato0.49198
002Milk1.7949
003Beer5.9950

So, we now have a description of what the customer wants to see the system do. The customer could write a whole series of such tests, perhaps working with developers, quality-assurance testers, and other people to verify that all the bases are covered. It is written in the language of the customer, but it has enough structure and detail that a programmer might be able to figure out how to parse it and execute it.

Fixtures

The Fit framework knows how to parse tables. So all we need to automate this test is a fixture, which is a piece of code that can take the parsed data, execute operations against the system, and then return the result to Fit.

Fit includes a set of standard fixtures, which can be used as-is or subclassed as needed to meet particular needs. Developers can also write their own fixtures from scratch if necessary.

Writing a fixture can be very easy, especially for the standard fixture types. For example, we have to provide an InitialInventory fixture, with a function Add that will be called for each row in that table. Since our Customer Sale table checks that the "display description" is "Milk", we need to provide a CustomerSale fixture with members such as displayDescription and receiptPrinted to process the actions and checks. Finally, the CheckInventory fixture has to provide a numberInStock function to verify the correct numbers after all the actions have been executed.

Note that while non-programmers can create the tests, a programmer is needed to develop the fixtures. It is likely that there will be some back-and-forth communication between the developers and customers to further refine the meaning of the test steps and to resolve any ambiguities or inconsistencies. Communication is a good thing, and Fit encourages it.

Results

After the fixture associated with a table returns its results, Fit displays them in a web page. Calculated values are displayed in green if the result was as expected, or red if not.

For example, if Fit runs the above test case against the system while it is still in development, the output might look something like this:

Initial Inventory
Product CodeDescriptionPriceNumber in StockAdd?
001Tomato0.49200true
002Milk1.7950true
003Beer5.9950true
Action Fixture
startCustomer Sale
enterproduct code001
pressproduct lookup key
checkdisplay descriptionTomato
checkdisplay price$0.49
pressquantity key
press2 key
checkdisplay descriptionTomato
checkdisplay quantityExpect: X2
Actual: (empty)
checkdisplay price$0.98
enterproduct code002
pressproduct lookup key
checkdisplay descriptionMilk
checkdisplay price$1.79
presssubtotal key
checkitems total2.77
checktax0.20
checktotal2.97
checkdisplay descriptionSUBTOTAL
checkdisplay price$2.97
entercash tendered2000
presscash sale key
checkcash tendered20.00
checkchange17.23
checkdisplay descriptionCHANGE
checkdisplay price$17.23
checkreceipt printedtrue
Check Inventory
Product CodeDescription?Price?Number in Stock?
001Tomato0.49Expect: 198
Actual: 200
002Milk1.79Expect: 49
Actual: 50
003Beer5.9950

From this report, we can see that the item quantity is not being shown on the customer display, and the inventory data is not being updated. So we know what the developers need to be working on to make the customer happy.

Fit provides mechanisms for running multiple tests and providing summary reports. So you can run the entire application test suite at the touch of a button.

FitNesse

Fit is designed to work on HTML tables or spreadsheets. A group of people decided it would be really nice if tests could be created and edited in a wiki, to make it really easy for customers, developers, and everyone else to share in the process of creating and reviewing acceptance tests. And so FitNesse was born. See their web site for details on what FitNesse can do.

The Fit Book doesn't have much to say about FitNesse, but if you like wikis and hate HTML editing as much as I do, you won't want to use Fit without it.

Language Support

One of the reasons I was discouraged from learning about Fit early was the fact that it looked like it supported only Java and .NET, and I don't use either of those software development platforms. In fact, while Fit and FitNesse are written in Java, FitNesse invokes an executable called a "FitServer" to run each test fixture, so the fixtures can be developed in any language you want.

Of course, the job is easier if somebody else has already done the work of developing a FitServer for the language you want to use. I have to use C++ in my job, so I'll be looking into the C++ support for FitNesse. (It's there; I just don't know how good it is yet.)

Other Uses

While Fit and FitNesse were designed for acceptance testing, they are fundamentally just a framework for reading data provided in a wiki page, processing that data, and returning the results back to the user via that wiki page. This is very powerful and would be useful for a lot of automatable tasks other than acceptance tests.

Read the Book

If Fit sounds interesting, I suggest you read the book. It does a good job of separating the "what is this good for" material from the "how do we make it work" material, so customers/analysts/testers can read the non-technical stuff, and the programmers can read the technical stuff.


Wednesday, May 10, 2006

 

Status Report

A few people have asked why I'm not blogging anymore. I haven't stopped blogging, but I really haven't had anything to say. (Of course, it could be argued that I have never had anything to say.)

I spent three weeks on the road, and that trip took a lot out of me. Work has been very frustrating. My boss has ordered me to take a few days off. I was one click away from booking a trip to Nassau, but I don't have much fun traveling by myself, so I decided to just stay at home instead.

It's been just over a month since my parachute jump. My knee still hurts a little, but I think it will be fine in a few more weeks. I don't think I'll be pursuing skydiving again in the near future, but I may give it another try someday.

I've signed up for a basic motorcycle riding class, so maybe motorcycling will be my next "thing." Don't worry, Mom, during the class we won't go out in the street, and I'll always wear my helmet.

Every motorcycle rider I've talked to has a story or two about getting hurt. Pilots all have stories about times they were concerned about the possibility of a crash, but motorcyclists all have stories of actual crashes. It seems that if you ride a motorcycle, you are practically guaranteed to suffer some injuries. I'm not a big fan of injury, no matter how minor, so I'm not currently planning to buy one.

I'm taking the class just in case I need to flee from villains, and a motorcycle is the only means of escape. Action heroes like me find themselves in such situations all the time, so I need to be prepared.


This page is powered by Blogger. Isn't yours?