“Testing is painful.”

“Testing is hard.”

“Testing is complicated.”

“Testing is not fun.”

I hear those sorts of things all the time when I talk to people about testing. I agree that sometimes testing can be all of those things, but if you choose the right tools, the tools that best suite you, testing doesn’t have to be. Let me give you an example of what I’m talking about, how choosing the right tools can make a huge impact on how you feel about testing.

When working for a client recently I came across the need for end to end integration testing. I needed to test, amongst other things, the flow of a user registering through the application in a few different ways. Because registration behaves differently based on where you come from and where you want to go, I needed a good way to test that entire flow, so simple unit and functional tests just were not going to cut it.

In the Ruby community there is a big push to use a testing framework called, Cucumber. Cucumber is a behavior driven development (BDD) tool that let’s you write user stories in plain English. Those stories then get translated into Ruby code that tests those stories against your application. Because of it’s popularity, and some of it’s quite amazing features, this was my first stop on the path to integration testing bliss.

Let me give you an example of a Cucumber script:

Feature: Registration
  In order to use My Great Application
  As a user
  I want to be able to register
  
  Scenario: 'Standard Registration'
    Given I am not currently logged in
    When I am on the signup page
    Then I should see "Sign Up"
    And I fill in "Name (required)" with "Mickey Dolenz"
    And I fill in "Email (required)" with "mickey@monkees.com"
    And I fill in "Password (required)" with "password"
    And I fill in "Password confirmation" with "password"
    And I press "Register"
    Then I should see "Sign Up - Confirm Your Account"
    Then I should be on the registration thank you page
    Then "mickey@monkees.com" should receive an email
    When I open the email
    Then I should see "Confirm my account" in the email body
    When I follow "Confirm my account" in the email
    Then I should be on the welcome page
    And I should see "Welcome to the Great Application"
    
  Scenario: 'Accepting an invitation'
    Given I am not currently logged in
    And the "Boys and Girls Club" invites "mickey@monkees.com" to join
    Then "mickey@monkees.com" should receive an email
    When I open the email
    Then I should see "Accept Invitation" in the email body
    When I follow "Accept Invitation" in the email
    Then I should be on the signup page
    Then I should see "Sign Up"
    And I fill in "Name (required)" with "Mickey Dolenz"
    And the "Email (required)" field should contain "mickey@monkees.com"
    And I fill in "Password (required)" with "password"
    And I fill in "Password confirmation" with "password"
    And I press "Register"
    Then the account "mickey@monkees.com" should be "activated"
    Then I should be on the accept/decline invitation page
    And I should see "Join the Boys and Girls Club"

That script tests the user registration flow through an application in a couple of different ways, first through ‘standard’ registration, and then through being invited to join. Now, the beauty of Cucumber is that these scripts are ‘human’ readable. Your product manager, or other stake holders, should be able to write these scripts themselves, and you, the developer, should be able to just plug them in and code until those scripts pass.

Unfortunately, while that sounds like a little slice of Heaven, the reality is far from it in practice. First, getting stake holders to actually write these ‘stories’, as their typically called, is a tough chore to begin with. If they do write them, they’re typically not going to be ‘plug and play’. Why? Well, when Cucumber reads these scripts it goes line by line and tries to find some code that matches the regular expression of that line and then execute it. If it doesn’t find matching code, then it fails. That means that your stake holders need to write these scripts in a very particular way or developers need to sit down and massage those stories to fit the correct regular expression.

Now, let me just take this opportunity to say that this is not a post about how much I hate Cucumber, in fact I think Cucumber is a pretty amazing piece of software, and does in fact have a lot of great uses. Instead, what I’m talking about it is how Cucumber turned out not to be the right tool for the job for me on a recent project.

So why wasn’t Cucumber the right tool for the job? Great question, glad you asked. Cucumber turned out not to be the right tool for a few reasons. The biggest of which was that I was the one who was writing the user stories. The stake holders had no desire to write these stories, which meant I had to write them. The I had to write the ‘steps’ that back each line of the script. In all fairness, Cucumber does give you some great steps right out of the box. After some fiddling I finally got the Cucumber scripts up and running and testing my work flow. But I definitely ran into some issues.

Because Cucumber isn’t pure Ruby I had a hard time doing something as simple as just printing out the request’s body and headers without having to write a step that did just that, then add that step to my story, etc… It’s overall fiddlyness and non-intuitive way of doing things caused me a lot of grief and time. And, most importantly, I wasn’t really getting the big benefit of Cucumber, stake holder’s writing the stories. So I was doing all this work and not getting the benefits of Cucumber.

So what did I do? I turned to a library called Steak. Steak allows you to write integration tests using pure Ruby and integrates directly in with RSpec, my preferred testing framework. With Steak I was able to write my integration tests in just a few minutes.

require File.expand_path(File.dirname(__FILE__) + '/acceptance_helper')

feature "Registration Steak", %q{
  In order to use My Great Application
  As a user
  I want to be able to register
} do

  scenario "register throught the standard registration process" do
    visit signup_path
    
    expect {
      fill_in_registration(:name => 'Mickey Dolenz', :email => 'mickey@monkees.com')
    }.to change(delivered_emails, :size).by(1)
    
    user = User.find_by_email('mickey@monkees.com')
    user.should be_unverified
    
    page.should have_content('You have signed up successfully. However, we could not sign you in because your account is unconfirmed.')
    current_path.should == root_path
    
    visit signin_path
    fill_in 'user_email', :with => 'mickey@monkees.com'
    fill_in 'user_password', :with => 'password'
    click_button 'Sign In'
    
    page.should have_content('You have to confirm your account before continuing.')
    current_path.should == new_user_session_path

    click_email_link_matching(/users\/confirmation/, delivered_emails.first)

    current_path.should == welcome_users_path
    page.should have_content('Welcome to the Great Application')
    user.reload
    user.should be_activated
  end
  
  scenario "register from an invitation" do
    mark = users(:mark)
    organization = mark.organizations.first
    signin(mark)
    
    visit new_organization_invitation_path(organization)
    expect {
      expect {
        fill_in 'organization_invitation_worker_emails', :with => 'mickey@monkees.com'
        click_button 'Send Invites'
        DJ.first.invoke_job
      }.to change(delivered_emails, :size).by(1)
    }.to change(organization.invitations, :count).by(1)
    
    invitation = organization.invitations.first
    
    signout
    
    click_first_link_in_email(delivered_emails.first)
    
    current_path.should == signup_path

    fill_in_registration(:name => 'Mickey Dolenz')
    
    current_path.should == organization_invitation_path(organization, invitation)
    
    user = User.find_by_email('mickey@monkees.com')
    user.should be_activated
    
    expect {
      click_on 'Get Started Now!'
    }.to change(user.organizations, :count).by(1)
    
    current_path.should == organization_campaign_path(organization, organization.campaigns.first)
  end
  
end

While my Steak scripts a bit more wordy and are definitely not ‘human’ readable and editable by stakeholders, they did achieve my goal of allowing me to write integration tests quickly.

So here you see I picked a very powerful tool, that has a lot of great benefits, Cucumber, but I picked it for the wrong reasons. I picked it because it was popular, and not because it would help me achieve my goals. If my goals where to have stakeholders write the stories and hand them off to development, than it would’ve been a better choice. But in the end my goal was to write integration tests and write them quickly, which is why Steak ended up being the right tool for that job.

This has all been a really long winded way of saying doing some research before choosing your testing frameworks, or any framework for that matter. Play with it, research it, make sure it meets your goals, not somebody else’s. If you choose the right tools then testing doesn’t need to be scary, complicated, frustrating, etc… Testing is a requirement and a must have, so why not make it fun?

When I’m talking with a potential client, well at least a client that has an existing code base, I always ask what their code coverage stats are. Now, I know at code coverage stats aren’t the be all end all of measuring how good your tests are, but they’re a basic enough metric to use as a guide. If they say they’re high, then I usually dig in more about how they’re testing; what frameworks, BDD, TDD, that sort of thing. Usually though I get a few minutes where they apologize and sheepishly give me their reasons for having little or no tests.

Here are few of those reasons:

• We don’t/didn’t have the time.
• We don’t know how.
• It was/is too complicated.
• It was/is too overwhelming.

Let’s talk about each of this points for a minute.

“We don’t/didn’t have the time.”

I never accept time as an argument against testing. Testing ends up repaying it’s time investment, and will ultimately give you more time than if you didn’t write code. It’s a win-win. Again see my previous post in how to get the time signed off on as part of t he project timeline.

“We don’t know how.”

Learn. There’s no better time than the present and no better way to learn than being thrown into the deep end. The web is crawling with documentation, screen casts, how to articles and tutorials, and there are plenty of books to get you going. In short the k knowledge is literally at your finger tips, and to be honest it’s easier than you think.

“It was/is too complicated.”

That usually means you’re doing it wrong. Take a step back and assess what it is you’re trying to do. You’re tests should be simple and concise. Don’t write tests that are hundreds of lines long. They’re tests, not entrance exams to MIT.

“It was/is too overwhelming.”

Certainly if you didn’t write tests as you went along it can get quite overwhelming thinking about all the tests you now need to write for your monolithic app. I’ll talk about how you can solve that problem in a minute.

Making It Happen

Ok, so now that we’ve identified a few of the excuses let’s talk about how you can starting writing tests today for your application. So, take a deep breath and let’s begin.

If you’re staring at an existing application, don’t try to tackle it all at once, you’ll just get overwhelmed, scared, and confused. Instead take it one file/class at a time. First start with your models, as this is where the majority of your application business logic should be. Alphabetically each day pick the next class (or a couple of them) in the list and start to fill our your test files.

What do I mean by fill out your tests files, I mean creating pending tests for each of the methods of your model. Here’s an example of a basic Ruby* class and what the pending RSpec spec file would look like:

# Class: class Entity def tax_id if self.person? # code here else # code here end end def person? # code here end end # Spec: describe Entity do describe "tax_id" do it "should return a Social Security number if the entity is a Person" do pending end it "should return a Tax ID number if the entity is a Corporation" do pending end end describe "person?" do it "should return true if the entity is a person" do pending end it "should return false if the entity is not a person" do pending end end end

Notice how the method that has the if/else statement in it has two pending tests for it. We need to test each variation of the method.

Now when you run your tests you’ll see that you have a bunch of pending tests. Great! Now you just need to fill them in, but at least you know what should be filled in.

I also recommend that you do this every time you create a new method. As soon as you give your method a name go to your corresponding test and create a pending test for that method. This way you know that you have to test that method later. (In a perfect world I would love to see you write your test before returning to your class to fill in the method itself, but baby steps for now.)

Once you have all your pending tests setup each day try to fill in the details of each pending test for a whole class. If that’s too much, then try to set aside an hour a day and fill in as many pending tests as you can. Alternatively you can also fill in the tests during the course of the day as you use one of the methods without tests.

Another great way to start filling in your test suite is each time you get a new bug, write a test to reproduce it. This is a great habit to get into as you’ll eventually have a great suite of regression tests in place to help prevent those nasty bugs from returning. Write the test, see that it fails, then fix your bug. When your teat passes then you know you’ve fixed the bug!

Finally, start small. Start by writing unit tests. These are the types of tests I just described. They test a very particular part of your code base to ensure that it does what it should do. These tests are typically easy to write and act as a great corner stone to your test suite as a whole. Don’t try to jump right in with full integration tests. The frameworks typically have a steep learning curve, and are more complicated to get up and running. This will lead to frustration and the old feeling of being overwhelmed. You can add these tests in later as you gain experience.

Well, there you have it, a few simple tricks to help you get started testing today. I know this post was a bit on the lengthy side, but I’m glad you stuck with me. Your life will be better for it. When you have a large and expansive test suite life is just better. Food tastes better. The sky is bluer. There will be a skip in your step. And you can use your incredibly high code coverage stats as a pick up line in a bar. On second thought, scratch that last thought. I wrote a test to see if that would work and it failed. It failed miserably.

* Please not that while I use Ruby as the example language here, the concept applies to whatever language you use.

The response to CoverMe has been great and through feedback from the community I’ve made a lot of improvements and fixed a lot of issues.

While quite a few things have changed under the hood, not much has changed in how you use CoverMe.

Installation

The following are instructions for how you would configure CoverMe for a Rails 3 project, adjust to your local environment accordingly.

In  your Gemfile add the following:

gem 'cover_me', '>= 1.0.0.rc1', :group => :test

Then run:

$ bundle install

After CoverMe is installed place the following line at the VERY TOP of your ‘test_helper.rb’ or ‘spec_helper.rb’ file (for Cucumber put it at the top of the ‘env.rb’ file):

require 'cover_me'

I can’t emphasize enough how important it is that the require statement is at the VERY top of that file!

Finally (and optionally) run:

$ rails g cover_me:install

This will simply install a Rake task that will wrap both Test::Unit and RSpec tasks with CoverMe and will launch the results at the end of the test suites. I would recommend it. It’s kinda the whole point.

That’s it!

Enjoy the release candidate, and of course, please let me know if you find any issues with it. Issues can be reported on here.

Fixtures, for those who don’t know, are YAML files that contain YAML-ized versions of objects that then get loaded into the test database when you run your test suite. These objects can then be pulled back from the database during your tests. Sounds great, doesn’t it? Well, not everybody thinks so. One of the biggest problems with fixtures is they can very quickly get out of control. Keeping track of all the different scenarios your tests needs can get very confusing and frustrating to deal with.

So how do we fix this problem? Well, most developers have turned to using factories. Factories allow us to quickly build the data we need for each test, now the building of the data you need for your test is right there in a setup or before method. Easy to manage and keep track of. Now there are a plethora of different factory libraries meant to make this task nicer, a few of the popular ones are Factory Girl, Machinist, and Object Daddy. The problem with this approach, however, is that it can slow down your tests as you are building database objects for nearly every test, and as we all know, object creation and database inserting can be expensive.

So, what can we do to help solve both of these problems? Well, we can use both of these technologies. Together. Yeah, that’s right I’m saying you should use fixtures as well as factories. Sound crazy? Not really. Let me explain.

Most Rails applications have most, if not all, of their functionality behind a login. So whenever we’re testing some controller action that sites behind a login we need a user to login with. If we were using factories we would have a setup or before method that would create a new User object and save it to the database, and it would do that for every variant of the test, as well as every other test in our suite that needs a user object.

Why not, create one user object and use that repeatedly through our tests? What I like to do is stick one or two users in my fixtures, so that they’re there whenever I need one. I like to do this with most of my major models. Then, when I need to have some custom scenarios, I can break out the factories and build those custom scenarios.

So what does this achieve? Well, I’ve sped up my tests by already having a few objects in the database, and not having to create them (and roll them back) with each single test. I’ve also cleaned up my tests significantly by eliminating a lot of setup and/or before methods where these objects were being created. I’ve also eliminated the biggest problems with fixtures, that they can get overwhelming, because we are only keeping one or two objects in them and using factories for the rest.

I hoped this helped you to understand that we don’t have to throw the baby out with the bath water when it comes to fixtures and factories, we can use both. Not go forth and test! Test like your life depends on it (because it does!!).

Enter CoverMe.

History

While working on a Ruby 1.9/Rails 3 project, and loving everything about it (except for the lack of RCov), I came across a post by Aaron Patterson (of Nokogiri fame). In this post he quickly outlined a very basic coverage tool using the new built-in Coverage module in Ruby 1.9.

After spending a morning playing with it, I was quickly able to grow the idea into something useful for the project. Later that day the company I was consulting for (BiddingForGood.com), and in particular their chief architect, Stuart Garner, told me to take a day or two and clean it up and release it for the world to use, and so here it is.

Features

Here is a brief overview of the features of CoverMe:

Index Page

• Sortable column headers (File, Lines, Lines of Code, Tested %).
• Searching/filtering by file name.
• Filtering by coverage percent.
• Color coded list of files to quickly see which ones are 100% covered, > 90% covered, or less than 90% covered.
• Large color coded average coverage percent, for quick reference.

Detail Page

• Line by line coverage report
• Color coded lines to quickly see which lines where executed and which ones were not.
• Side by side viewing with the corresponding test/spec file (if one exists).

See the README file for more information on installation and usage.

Thanks

I would just quickly like to give another quick thanks to Aaron Patterson for pointing out the Coverage module in Ruby 1.9 and inspiring this, hopefully, helpful little gem. Also another big thanks to Stuart Garner for pushing me to package this up and release it to the world.

Screenshots

During the past week or so I’ve spoken with many great companies and people. I’ve been fortunate enough to have a high degree of interest in what I can bring to the table. During those discussions I talked with a really nice guy at a what seems to be a really cool company, I won’t name names, because this isn’t about either the person or the company, but rather something the engineer said during our phone conversation that got me to thinking.

“We don’t have any tests because I couldn’t convince the company to allocate the time for them.” That statement really hung with me. After I got off the phone I started thinking really hard about that statement, and all I could think of was how testing is not an option and people shouldn’t need to be convinced to have time allocated to them.

As developers it is our responsibility to insist on testing. Always include testing in your time estimates. Never give the client (or your company) an option that includes a time estimate without testing. If a feature takes 2 days to code and a day to write tests, then your estimate is 3 days, never 2. You should never say, “Well, I can get it done in two days if I don’t write any tests.” That’s just an unacceptable thing to say. What you should be saying is, “That feature will take three days to code”.

I don’t feel I should sit here and tell you all the reasons why you should test, you should know them already, and frankly, they’re all very obvious! But, if you need a few bullet points to ‘convince’ your client, here are a few:

• Less bugs – The more tests you have the less bugs you will have. It’s just a fact. You won’t have 100% bug free code, that’s a nearly impossible goal, but you highly reduce the likely hood that as soon as you get your code into production your users will find all the breaking points of your code.
• Better maintainability, means faster feature turn around – When you have a large test suite it means adding, updating, or even removing features because a whole lot easier, which means it SAVES time! Why? Simple, you don’t have to go through and manually test every aspect of your code to make sure you didn’t break something elsewhere by adding that validation, or by refactoring that bit of code, etc… That translates into real $ savings.
• Test driven development saves time – this isn’t quite the same as my last bullet point. Imagine, if you will, you are writing a four step wizard in your application. If you write a few test scripts using something like Cucumber first before you write your code you can simply keep re-running those to make sure your code is working. If you don’t have those test scripts written then you continually have to keep going to a browser and entering all the information in each of the steps so you can test something in step four. Which one do you think takes longer, having a few test scripts you can run, or manually going through the four page wizard each time you make a change?
• It’s an investment – thinking of having test scripts like owning a house. When you don’t have tests and you just keep testing in the browser or the command line what you are doing is a kin to “renting”. There is money being spent, but at the end of the day you have nothing to show for it. You’ve spent hours “testing”, but tomorrow when you come in you have to do it all over again. When you spend those hours writing tests you are actually “buying” something. You have something to show for that time and money you’ve spent. Tomorrow, next week, next month, next year, those scripts will still be there, they’ll still be working for you, giving  you a return on your investment.

Well, I hope I have hopefully made a case to you the engineer as to why you should insist on testing. It’s the right thing to do, for you, for your application and for you client. If if anyone tries to give you grief about it, send them my way, I’ll sort em out!

Good news is that things seem to be working, at least going in the ‘up’ direction anyway. I still need to write the ‘down’ direction revert stuff. Hopefully, that shouldn’t take too long now that I have a good way of testing the task and the migration stuff.

Unfortunately, I feel as though as soon as I release this stuff, which should be by the end of this weekend, DataMapper 0.9.0 will be out, and I’ll have to re-write most of the DataMapper stuff because the API is changing so radically. I guess I’ll just have to deal with that when the time comes.

In the meantime I’ll let y’all know when migrations are done and out there in the wild.

I jumped on the spec test bandwagon about a year or so ago. I forced my whole team to start writing in nothing but spec tests. The team, is still not impressed. If I were to tell them we weren’t using spec tests anymore, they would be EXTREMELY happy with that.Spec tests are certainly prettier than ‘regular’ tests. I’ll give you that. They are more ‘human readable’. With that said I find assert_equal to be fairly easy to read.

So why do I use regular tests, and not spec tests? A couple of reasons. First, they’re there and built right in to Ruby, ready to go, no new gems or syntax to learn. They’re fast. They are faster than spec tests. I see it everyday at work. I can also, and this one is HUGE, run just a single test or a regex’d series of tests. I really love that last bit. I have yet to find a way to do that, easily, with spec tests.

Again, how people write tests doesn’t really matter so much to me, it’s just that they write them. Testing is too easy in Ruby, there’s just no reason for not writing them.

What makes this release so important is that it fixes, adds, or improves testing support for Mack applications.

The implementation of some of the testing stuff has changed under the covers, but it’s the new stuff that is really important. let’s take a look at some of the stuff:

• assigns(key) – This one should be familiar to all you Rails folks out there. This allows you to get an instance variable that was set in a controller, and access it in your test.

  def test_something
    get foo_index_url
    assert_not_nil assigns(:foo_list)
  end

• in_session – This methods allows you to wrap a block in the context of a session. This is great if you’re trying to do integration testing.
• set_cookie, remove_cookie – These methods allow you to either set or a remove a cookie in your tests. Great if you want to test an action that reads a cookie, or test to see what happens if you set a cookie, then remove the cookie and call something else. Also great to use with the in_session method.

There are a few other hidden treats in there, but those are some of the highlights. These things help bring Mack testing to a usable level.

The Mack Demo Blog application has been updated to demonstrate some these things. You can pull the demo using Git here: git://github.com/markbates/mack_blog_demo.git

Changelog:

• Improved testing support.

• Added assigns support to testing. This is allows for pulling instance variables that were set in a controller to tests.

• Added ability to do ‘session’ based tests.

• Added assert_difference method.

• Added ability to set/remove cookies before requests are made.

• gem: mack_ruby_core_extensions 0.1.4

$ sudo gem install mack