Posts Tagged ‘Acceptance Testing’

Agile Acceptance Testing: Mitigating the Risks of Enterprise Software Development

January 6th, 2012

Here is the video from my talk at the ALM Summit at Microsoft.


For more information about my courses check the  Effective AGILE Development site.

Tags: , , ,
Posted in Agile Engineering Practices, Agile Testing, Scrum | Comments (0)

Setup Fitnesse for .NET Testing

November 12th, 2011

When new pilots are in training one of the most unnatural things to learn is the repeating back of instructions to the control tower or Air Traffic Control.  Sometimes these instructions are pretty complicated and the pilot has to write the instructions down and read them back.  Obviously the reason this is so critical is that the controller needs you to carry out his instructions and the first step in this is that you correctly heard these instructions.  But that is not all there is to it.  The pilot must understand these instructions to carry them out and maintain the safety of flight operations.  After the pilot hears, writes and repeats then the training tells the pilot check his understanding of the instructions.  If they are not understood the pilot must ask for a clarification and the whole process starts over again.  In this domain, the pilot generally understands the instructions because their preparation and training make most of these instructions routine.

As software developers, when we get our “instructions from ATC”, that is our requirements, they are often written rather than spoken. How well we understand those instructions depends on a number of factors starting with how well they were understood by the person writing them and their skill in putting them into words.  However a document does not have the built-in feedback loop that is required of pilots!  It is also very common for us to understand every word in the document, yet not have a clue what they are talking about or even worse think we understand perfectly when we really don’t have a clue.

Using the process of Acceptance Test Driven Development is one of the quickest and easiest ways to increase the pace and quality of business value delivery from your software development team.  Stating back our requirements to check understanding, while valuable, would be expensive and boring if all we did was create a new document and rewrite the requirements in more technical terms.  However if we could create this early validation that we understand the requirements in such a way as to be able to check the product’s progress toward meeting those requirements, then we might have something valuable and not so boring!

There are a number of tools being used to create Agile Acceptance Tests, but by far the most well known and popular is the FIT/Fitnesse family tools that allow you to create Agile Acceptance Tests for a number of different languages and platforms.

Introduction to Fit

Setting up Fitnesse requires a little history. FIT: Framework for Integrated Test was developed by Ward Cunningham and several collaborators.  I found pages on from as early as 2001, so needless to say FIT has been around for quite a while.  In the introduction, Ward states that great software requires collaboration and communication.   FIT provided a basis for that two-way communication between the stakeholders and the team members.  Through the use of the various tables that were used to explicitly demonstrate the requirements, the team could create automated tests to check the current status of any requirement set up with these top-down tests.

While this is a great idea and a pretty good implementation, more was possible.  Bob Martin and his collaborators wrapped the FIT testing engine in a Wiki (which was a Ward Cunningham invention) and created FitNesse.

Here is how Wikipedia defines a Wiki:
A wiki (/ˈwɪki/ wik-ee) is a website that allows the creation and editing of any number of interlinked web pages via a web browser using a simplified markup language or a WYSIWYGtext editor. Wikis are typically powered by wiki software and are often used collaboratively by multiple users.

By wrapping the test engine in a Wiki, Martin and his collaborators enabled even more communication between the stakeholders and the development team.  Ideally the stakeholders would either record the requirements in the FitNesse Wiki and describe the scenarios with examples.

The original FIT was developed in Java.  Many developers added other platform implementations enabling the the use of FIT and FitNesse on the majority of application development projects.  This was a good thing, but there was a big problem.  Because of several factors including the desire to one-up other implementations, all of these implementations were different.  When I first started working with FIT, I downloaded it, installed it and then tried to use the included instructions to use the then current .NET implementation.  I tried for several hours!  After some searching, I found a nice article that discussed the setup for a .NET project.  This article was written by Gojko Adzic who remains a big contributor in the FIT/Fitness space.  My experience was typical and others spent days trying to get the FIT/Fitnesse port on their platform working.  The main problem was that all the implementations were different.  You could not always use what you learned on the Java implementations to set up the platform you were working on.

A few years later Martin and his team offered us an alternative to the FIT testing engine that makes this process much easier.  The Simple List Invocation Method is an alternative to FIT.  It is built into FitNesse.  Slim keeps the HTML processing, comparisons and the results output in FitNesse. In other words when you are using Slim, it does not call out to the FIT server to process the test results.  The big advantage is the fact that Slim is very slim.  It is easy to port because the calls to an external port are made with the same loosely coupled calls that are used with the internal Java implementation.  This allows all of the ports to be identical from an external API point of view.

Today FIT and FitNesse with Slim combined with the port for your platform provides a very powerful and capable set of tools to create and run top-down Agile Acceptance Tests.  FitNess and the .NET port of Slim and FIT are all in active development and you can use various online resources to get help and advice from other users and even the authors.

Setting up FitNesse

The place to start is the home of FitNess at   Not too surprisingly this is a FitNesse implemenation.  It contains links to download the Java Jar file, Java source code and links to all known plugins and accessories.

FitNesse Plugins

First you need to downlaod the fitnesse.jar .  Save it to a convienet location on your computer.

A Java Jar file is just a zip file that contains the folders and files of the application.  It is roughly analogous to a .dll.  Fitness.jar is an executiable Jar file.  It is run by calling the Java runtime engine.   If you don’t have a Java runtime on your computer you will need to download the latest Java runtime engine from

Next you need to decide where you want to put your FitNesse implemenation.  I like to keep my primary installation on the root drive of my computer.  I create a folder called FitNesse and copy the fitnesse.jar file there.

The next thing you need to decide is what port you are going to use for your FitNesse server.  Running the FitNesse.Jar file the first time unpacks the FitNessRoot which will be the repository of your wiki pages.  Then it starts the FitNesse server.

The command line looks like this…

java -jar fitnesse.jar

There are a number of optional command line switches, but the most important one is the -p which sets the port where the FitNesse server runs.  If you don’t use this switch it will try to run on port 80 where your web server probably runs.  It is a very good idea to include -p switch with an unused port like 8080.  So the full command line would be like this…

java -jar fitnesse.jar -p 8080

I don’t like to type that much, so I create a batch file to run the command put it in the fitnesse folder.  It would look something like this…

start java -java fitnesse.jar -p 8080

If all goes well you will get something like this is the command window that was opened by the batch file.

If the server does not start, take a look at the page on the site .

Then you can open your server with your favorite browser.


Mine is going to look a bit different, but you should get the Fitnesse server.

Configuring FitSharp

At this point you have the base Java system up and running.  To enable .NET testing there is one more component and some configuration to do.

On the plugins page you may have noticed the .NET Slim implementation by Mike Stockdale.  It is located at .  The site has both .NET 3.5 and 4.0 builds. Download the one that is approriate for your development environment.  The latest download has all the source and the entire binary history of the product.  Extract the files from the zip and copy the files from the latest binary into a folder on your system.  I put this folder in the fitnesse folder where you copied the Fitnesse.jar.  You can name this folder anything you want,but you will need to reference it in the next step.   I use dotnet2 with is a legacy name for me because of having tests set up from when support for .NET 2.0 was the latest.

When that is done, you are ready to go.  A future article will go into the details of how to use the system but for completeness on setup, there is one more step.

At the top level of a suite of tests ( or on any test page where the configuation is different ) we need to define four variables.  The first is the test system to be used.  This is how it would look in edit mode on the wiki for a slim test.

!define TEST_SYSTEM {slim}

Next is the command pattern which includes the full path to the fitsharp.dll.  Note  your path may be different.

!define COMMAND_PATTERN {%m -r fitSharp.Slim.Service.Runner, C:\Fitnesse\dotnet2\fitsharp.dll %p}

Next is the test runner.  Again your path may be different.

!define TEST_RUNNER {C:\Fitnesse\dotnet2\Runner.exe}

And last is the path to the .dll with the tests.  Put the full path and file name on the line.


Here is what it looks like in edit mode…

And after you save, the page will look like this…

That’s all for now!  Watch for a new article every couple of weeks or so.  The next article will be an overview of the Slim tables and how to use a simple Decision Table.

Feel free to post messages with your questions so others can share in your experiences!

Tags: , , , ,
Posted in Agile Engineering Practices | Comments (1)

Bugs, bugs, BUGS!

June 7th, 2009

bugI’m sure that you all know the story of how the term “bug” got associated with a computer failure. It was a moth in a computer which caused a hardware failure. However, since then the term bug has become synonymous with a software error. There is a bug in the program. Yes, there is a bug in the program, every program. Many software development organizations spend a large portion of their time and effort finding and fixing bugs. However the situation rarely improves.

This leads organizations to start tracking bugs and applying all sorts of analysis and pressure to fix them. How do we know which bug to fix first? The priority (how important we think it is) is assigned and the severity (who and what does it affect) is assessed. Then when we figure out what to fix based on priority and severity, then who should fix it? All too often the analysis is that the primary development team is too busy and important to fix bugs, so let’s create a bug team and have them fix all the bugs.

Quite often the situation gets worse again. More bugs are found by customers and the QA team. The backlog for the bug team gets bigger and the time from bug discovery to fix delivery to customers increases.

If you have been around software for any time at all, you will recognize this pattern. Lots of bugs being tacked separately from new requirements and an increasing slice of all development effort being allocated to the effort to attempt to slow the erosion of software quality.

I believe we need a different approach.

We have to understand what caused the bug in the first place. We have to make sure our goals are clear and that we are applying the right people and talent and money to reach our goals.

Should the goal of the software development organization be to deliver quality software?

Trick question. Of course we need quality software, but quality should not be the goal. The goal should be the delivery of business or customer value. That’s why we are building software. If we deliver poor quality then we will have to spend more time and effort fixing it so that we can deliver on the solution we promised. However the focus should be on the delivery of business value.

If we turn away from tracking and prioritization of bugs and turn towards the delivery of business value quickly and effectively, we can change the entire landscape of software development.

We need to understand where the bugs come from. Contrary to the belief of some, the code does not just rot. The gremlins don’t attack source control at night. Except for the occasional corruption of source and binaries from hardware failure, most of the time the bug was written by a developer.

I think there are two kinds of bugs. The first is just a mistake. Someone used the wrong variable, used a wrong value, called the wrong method. Stuff happens. This type is blocking functionality that was assumed to be working. Hopefully an Agile team using TDD will find these before the end of the iteration. If not, there are few things of higher priority. Track it if you have to, but get it on the next sprint backlog in the form of a story.

The other kind is when the team did not understand ( or was not given ) the full scope of the requirement. This one often includes unrecognized interaction with other systems. This type is much more common and the priority cannot be determined by the development team. The Product Owner must make sure that they understand the value of the missing or wrong functionality and present that story back to the team for sizing and prioritization.

We will probably need a system to track the errors that are found, but we need to quickly decide which type it is. The first type is the failure to really deliver what we said we delivered. The second is the failure of the Product Owner to deliver the requirements to the development team in such a way that they are able to deliver an appropriate solution.

In either case, I think that the work needs to be prioritized alongside the other stories. What is the business (customer) value? How much effort and risk is there in completing this story? If you must use a bug tracking system to be responsible and responsive, then do so. However your decisions to deliver business value should be contained in a story that the team can deliver.

This, by definition obviates the need for a “bug” team. The delivery of business value is the responsibility of the whole team. The team can and must learn from its mistakes. How did that error get into production (the mistake type)? What tests should have been in place to make sure that the code was correct. What about that requirement did we not understand ( the second type)? How can we better understand next time? How will we make sure that we understand it now? These are all questions that the team should investigate and try to answer.

The first type of bug is lessened by unit testing by developers. A test first approach using Test Driven Development is best. TDD all but eliminates this type of bug.

The second type of bug is lessened by an Acceptance Test Driven approach to requirements understanding. Virtually all requirements can be tested with Automated Acceptance Testing. Basically what we are doing is repeating back to the Product Owner the requirements in the form of a test that says, “when this happens, the code will respond with this behavior.” The Acceptance Test can be written before the object code is even named. The test can fail on day one of the iteration.

Scrum and Agile are about delivering business value quickly and repeatably. Prioritization of business value must drive the project. Prioritize all work together and drive the process with Acceptance Tests and Test Driven Development. Your team will deliver more business value quicker and will deliver higher quality software in the process.

Tags: ,
Posted in Misc | Comments (0)

SEO Powered by Platinum SEO from Techblissonline