Meta Bates

“Testing is painful.”

“Testing is hard.”

“Testing is complicated.”

“Testing is not fun.”

I hear those sorts of things all the time when I talk to people about testing. I agree that sometimes testing can be all of those things, but if you choose the right tools, the tools that best suite you, testing doesn’t have to be. Let me give you an example of what I’m talking about, how choosing the right tools can make a huge impact on how you feel about testing.

When working for a client recently I came across the need for end to end integration testing. I needed to test, amongst other things, the flow of a user registering through the application in a few different ways. Because registration behaves differently based on where you come from and where you want to go, I needed a good way to test that entire flow, so simple unit and functional tests just were not going to cut it.

In the Ruby community there is a big push to use a testing framework called, Cucumber. Cucumber is a behavior driven development (BDD) tool that let’s you write user stories in plain English. Those stories then get translated into Ruby code that tests those stories against your application. Because of it’s popularity, and some of it’s quite amazing features, this was my first stop on the path to integration testing bliss.

Let me give you an example of a Cucumber script:

Feature: Registration
  In order to use My Great Application
  As a user
  I want to be able to register
  
  Scenario: 'Standard Registration'
    Given I am not currently logged in
    When I am on the signup page
    Then I should see "Sign Up"
    And I fill in "Name (required)" with "Mickey Dolenz"
    And I fill in "Email (required)" with "mickey@monkees.com"
    And I fill in "Password (required)" with "password"
    And I fill in "Password confirmation" with "password"
    And I press "Register"
    Then I should see "Sign Up - Confirm Your Account"
    Then I should be on the registration thank you page
    Then "mickey@monkees.com" should receive an email
    When I open the email
    Then I should see "Confirm my account" in the email body
    When I follow "Confirm my account" in the email
    Then I should be on the welcome page
    And I should see "Welcome to the Great Application"
    
  Scenario: 'Accepting an invitation'
    Given I am not currently logged in
    And the "Boys and Girls Club" invites "mickey@monkees.com" to join
    Then "mickey@monkees.com" should receive an email
    When I open the email
    Then I should see "Accept Invitation" in the email body
    When I follow "Accept Invitation" in the email
    Then I should be on the signup page
    Then I should see "Sign Up"
    And I fill in "Name (required)" with "Mickey Dolenz"
    And the "Email (required)" field should contain "mickey@monkees.com"
    And I fill in "Password (required)" with "password"
    And I fill in "Password confirmation" with "password"
    And I press "Register"
    Then the account "mickey@monkees.com" should be "activated"
    Then I should be on the accept/decline invitation page
    And I should see "Join the Boys and Girls Club"

That script tests the user registration flow through an application in a couple of different ways, first through ‘standard’ registration, and then through being invited to join. Now, the beauty of Cucumber is that these scripts are ‘human’ readable. Your product manager, or other stake holders, should be able to write these scripts themselves, and you, the developer, should be able to just plug them in and code until those scripts pass.

Unfortunately, while that sounds like a little slice of Heaven, the reality is far from it in practice. First, getting stake holders to actually write these ‘stories’, as their typically called, is a tough chore to begin with. If they do write them, they’re typically not going to be ‘plug and play’. Why? Well, when Cucumber reads these scripts it goes line by line and tries to find some code that matches the regular expression of that line and then execute it. If it doesn’t find matching code, then it fails. That means that your stake holders need to write these scripts in a very particular way or developers need to sit down and massage those stories to fit the correct regular expression.

Now, let me just take this opportunity to say that this is not a post about how much I hate Cucumber, in fact I think Cucumber is a pretty amazing piece of software, and does in fact have a lot of great uses. Instead, what I’m talking about it is how Cucumber turned out not to be the right tool for the job for me on a recent project.

So why wasn’t Cucumber the right tool for the job? Great question, glad you asked. Cucumber turned out not to be the right tool for a few reasons. The biggest of which was that I was the one who was writing the user stories. The stake holders had no desire to write these stories, which meant I had to write them. The I had to write the ‘steps’ that back each line of the script. In all fairness, Cucumber does give you some great steps right out of the box. After some fiddling I finally got the Cucumber scripts up and running and testing my work flow. But I definitely ran into some issues.

Because Cucumber isn’t pure Ruby I had a hard time doing something as simple as just printing out the request’s body and headers without having to write a step that did just that, then add that step to my story, etc… It’s overall fiddlyness and non-intuitive way of doing things caused me a lot of grief and time. And, most importantly, I wasn’t really getting the big benefit of Cucumber, stake holder’s writing the stories. So I was doing all this work and not getting the benefits of Cucumber.

So what did I do? I turned to a library called Steak. Steak allows you to write integration tests using pure Ruby and integrates directly in with RSpec, my preferred testing framework. With Steak I was able to write my integration tests in just a few minutes.

require File.expand_path(File.dirname(__FILE__) + '/acceptance_helper')

feature "Registration Steak", %q{
  In order to use My Great Application
  As a user
  I want to be able to register
} do

  scenario "register throught the standard registration process" do
    visit signup_path
    
    expect {
      fill_in_registration(:name => 'Mickey Dolenz', :email => 'mickey@monkees.com')
    }.to change(delivered_emails, :size).by(1)
    
    user = User.find_by_email('mickey@monkees.com')
    user.should be_unverified
    
    page.should have_content('You have signed up successfully. However, we could not sign you in because your account is unconfirmed.')
    current_path.should == root_path
    
    visit signin_path
    fill_in 'user_email', :with => 'mickey@monkees.com'
    fill_in 'user_password', :with => 'password'
    click_button 'Sign In'
    
    page.should have_content('You have to confirm your account before continuing.')
    current_path.should == new_user_session_path

    click_email_link_matching(/users\/confirmation/, delivered_emails.first)

    current_path.should == welcome_users_path
    page.should have_content('Welcome to the Great Application')
    user.reload
    user.should be_activated
  end
  
  scenario "register from an invitation" do
    mark = users(:mark)
    organization = mark.organizations.first
    signin(mark)
    
    visit new_organization_invitation_path(organization)
    expect {
      expect {
        fill_in 'organization_invitation_worker_emails', :with => 'mickey@monkees.com'
        click_button 'Send Invites'
        DJ.first.invoke_job
      }.to change(delivered_emails, :size).by(1)
    }.to change(organization.invitations, :count).by(1)
    
    invitation = organization.invitations.first
    
    signout
    
    click_first_link_in_email(delivered_emails.first)
    
    current_path.should == signup_path

    fill_in_registration(:name => 'Mickey Dolenz')
    
    current_path.should == organization_invitation_path(organization, invitation)
    
    user = User.find_by_email('mickey@monkees.com')
    user.should be_activated
    
    expect {
      click_on 'Get Started Now!'
    }.to change(user.organizations, :count).by(1)
    
    current_path.should == organization_campaign_path(organization, organization.campaigns.first)
  end
  
end

While my Steak scripts a bit more wordy and are definitely not ‘human’ readable and editable by stakeholders, they did achieve my goal of allowing me to write integration tests quickly.

So here you see I picked a very powerful tool, that has a lot of great benefits, Cucumber, but I picked it for the wrong reasons. I picked it because it was popular, and not because it would help me achieve my goals. If my goals where to have stakeholders write the stories and hand them off to development, than it would’ve been a better choice. But in the end my goal was to write integration tests and write them quickly, which is why Steak ended up being the right tool for that job.

This has all been a really long winded way of saying doing some research before choosing your testing frameworks, or any framework for that matter. Play with it, research it, make sure it meets your goals, not somebody else’s. If you choose the right tools then testing doesn’t need to be scary, complicated, frustrating, etc… Testing is a requirement and a must have, so why not make it fun?

In a previous post, Testing Is Not An Option, I talked a lot about why you should write tests, and the arguments you can put forth to your client, manager, or whoever it may be as to why you should write tests. What I didn’t talk about was how to start writing tests. So let’s talk about that for a bit, shall we?

When I’m talking with a potential client, well at least a client that has an existing code base, I always ask what their code coverage stats are. Now, I know at code coverage stats aren’t the be all end all of measuring how good your tests are, but they’re a basic enough metric to use as a guide. If they say they’re high, then I usually dig in more about how they’re testing; what frameworks, BDD, TDD, that sort of thing. Usually though I get a few minutes where they apologize and sheepishly give me their reasons for having little or no tests.

Here are few of those reasons:

• We don’t/didn’t have the time.
• We don’t know how.
• It was/is too complicated.
• It was/is too overwhelming.

Let’s talk about each of this points for a minute.

“We don’t/didn’t have the time.”

I never accept time as an argument against testing. Testing ends up repaying it’s time investment, and will ultimately give you more time than if you didn’t write code. It’s a win-win. Again see my previous post in how to get the time signed off on as part of t he project timeline.

“We don’t know how.”

Learn. There’s no better time than the present and no better way to learn than being thrown into the deep end. The web is crawling with documentation, screen casts, how to articles and tutorials, and there are plenty of books to get you going. In short the k knowledge is literally at your finger tips, and to be honest it’s easier than you think.

“It was/is too complicated.”

That usually means you’re doing it wrong. Take a step back and assess what it is you’re trying to do. You’re tests should be simple and concise. Don’t write tests that are hundreds of lines long. They’re tests, not entrance exams to MIT.

“It was/is too overwhelming.”

Certainly if you didn’t write tests as you went along it can get quite overwhelming thinking about all the tests you now need to write for your monolithic app. I’ll talk about how you can solve that problem in a minute.

Making It Happen

Ok, so now that we’ve identified a few of the excuses let’s talk about how you can starting writing tests today for your application. So, take a deep breath and let’s begin.

If you’re staring at an existing application, don’t try to tackle it all at once, you’ll just get overwhelmed, scared, and confused. Instead take it one file/class at a time. First start with your models, as this is where the majority of your application business logic should be. Alphabetically each day pick the next class (or a couple of them) in the list and start to fill our your test files.

What do I mean by fill out your tests files, I mean creating pending tests for each of the methods of your model. Here’s an example of a basic Ruby* class and what the pending RSpec spec file would look like:

# Class: class Entity def tax_id if self.person? # code here else # code here end end def person? # code here end end # Spec: describe Entity do describe "tax_id" do it "should return a Social Security number if the entity is a Person" do pending end it "should return a Tax ID number if the entity is a Corporation" do pending end end describe "person?" do it "should return true if the entity is a person" do pending end it "should return false if the entity is not a person" do pending end end end

Notice how the method that has the if/else statement in it has two pending tests for it. We need to test each variation of the method.

Now when you run your tests you’ll see that you have a bunch of pending tests. Great! Now you just need to fill them in, but at least you know what should be filled in.

I also recommend that you do this every time you create a new method. As soon as you give your method a name go to your corresponding test and create a pending test for that method. This way you know that you have to test that method later. (In a perfect world I would love to see you write your test before returning to your class to fill in the method itself, but baby steps for now.)

Once you have all your pending tests setup each day try to fill in the details of each pending test for a whole class. If that’s too much, then try to set aside an hour a day and fill in as many pending tests as you can. Alternatively you can also fill in the tests during the course of the day as you use one of the methods without tests.

Another great way to start filling in your test suite is each time you get a new bug, write a test to reproduce it. This is a great habit to get into as you’ll eventually have a great suite of regression tests in place to help prevent those nasty bugs from returning. Write the test, see that it fails, then fix your bug. When your teat passes then you know you’ve fixed the bug!

Finally, start small. Start by writing unit tests. These are the types of tests I just described. They test a very particular part of your code base to ensure that it does what it should do. These tests are typically easy to write and act as a great corner stone to your test suite as a whole. Don’t try to jump right in with full integration tests. The frameworks typically have a steep learning curve, and are more complicated to get up and running. This will lead to frustration and the old feeling of being overwhelmed. You can add these tests in later as you gain experience.

Well, there you have it, a few simple tricks to help you get started testing today. I know this post was a bit on the lengthy side, but I’m glad you stuck with me. Your life will be better for it. When you have a large and expansive test suite life is just better. Food tastes better. The sky is bluer. There will be a skip in your step. And you can use your incredibly high code coverage stats as a pick up line in a bar. On second thought, scratch that last thought. I wrote a test to see if that would work and it failed. It failed miserably.

* Please not that while I use Ruby as the example language here, the concept applies to whatever language you use.

In August I announced CoverMe a code coverage tool for Ruby 1.9. Well, today I announce that it has hit it’s first release candidate! I’ve very excited by the fact it’s getting close to an ‘official’ release.

The response to CoverMe has been great and through feedback from the community I’ve made a lot of improvements and fixed a lot of issues.

While quite a few things have changed under the hood, not much has changed in how you use CoverMe.

Installation

The following are instructions for how you would configure CoverMe for a Rails 3 project, adjust to your local environment accordingly.

In  your Gemfile add the following:

gem 'cover_me', '>= 1.0.0.rc1', :group => :test

Then run:

$ bundle install

After CoverMe is installed place the following line at the VERY TOP of your ‘test_helper.rb’ or ‘spec_helper.rb’ file (for Cucumber put it at the top of the ‘env.rb’ file):

require 'cover_me'

I can’t emphasize enough how important it is that the require statement is at the VERY top of that file!

Finally (and optionally) run:

$ rails g cover_me:install

This will simply install a Rake task that will wrap both Test::Unit and RSpec tasks with CoverMe and will launch the results at the end of the test suites. I would recommend it. It’s kinda the whole point.

That’s it!

Enjoy the release candidate, and of course, please let me know if you find any issues with it. Issues can be reported on here.

Testing in Ruby on Rails is incredibly easy. I mean stupidly easily. So easy that if you’re not doing it, you are a very, very bad developer and should re-evaluate your career choices. (Yes, I believe in testing that much!) One thing that is not all that easy, however, is object creation and populating your test database. Five years ago when I first started working with Rails the only options we had to get data into the database were fixtures, or hastily written ‘factory’-esque methods custom to each application.

Fixtures, for those who don’t know, are YAML files that contain YAML-ized versions of objects that then get loaded into the test database when you run your test suite. These objects can then be pulled back from the database during your tests. Sounds great, doesn’t it? Well, not everybody thinks so. One of the biggest problems with fixtures is they can very quickly get out of control. Keeping track of all the different scenarios your tests needs can get very confusing and frustrating to deal with.

So how do we fix this problem? Well, most developers have turned to using factories. Factories allow us to quickly build the data we need for each test, now the building of the data you need for your test is right there in a setup or before method. Easy to manage and keep track of. Now there are a plethora of different factory libraries meant to make this task nicer, a few of the popular ones are Factory Girl, Machinist, and Object Daddy. The problem with this approach, however, is that it can slow down your tests as you are building database objects for nearly every test, and as we all know, object creation and database inserting can be expensive.

So, what can we do to help solve both of these problems? Well, we can use both of these technologies. Together. Yeah, that’s right I’m saying you should use fixtures as well as factories. Sound crazy? Not really. Let me explain.

Most Rails applications have most, if not all, of their functionality behind a login. So whenever we’re testing some controller action that sites behind a login we need a user to login with. If we were using factories we would have a setup or before method that would create a new User object and save it to the database, and it would do that for every variant of the test, as well as every other test in our suite that needs a user object.

Why not, create one user object and use that repeatedly through our tests? What I like to do is stick one or two users in my fixtures, so that they’re there whenever I need one. I like to do this with most of my major models. Then, when I need to have some custom scenarios, I can break out the factories and build those custom scenarios.

So what does this achieve? Well, I’ve sped up my tests by already having a few objects in the database, and not having to create them (and roll them back) with each single test. I’ve also cleaned up my tests significantly by eliminating a lot of setup and/or before methods where these objects were being created. I’ve also eliminated the biggest problems with fixtures, that they can get overwhelming, because we are only keeping one or two objects in them and using factories for the rest.

I hoped this helped you to understand that we don’t have to throw the baby out with the bath water when it comes to fixtures and factories, we can use both. Not go forth and test! Test like your life depends on it (because it does!!).

Ruby 1.9(.2) is an amazing language to develop applications in. It’s faster, more powerful, cleaner, and a huge improvement over Ruby 1.8.x. Because of those reasons every Ruby developer should move to this exciting new version of our language.

Enter CoverMe.

History

While working on a Ruby 1.9/Rails 3 project, and loving everything about it (except for the lack of RCov), I came across a post by Aaron Patterson (of Nokogiri fame). In this post he quickly outlined a very basic coverage tool using the new built-in Coverage module in Ruby 1.9.

After spending a morning playing with it, I was quickly able to grow the idea into something useful for the project. Later that day the company I was consulting for (BiddingForGood.com), and in particular their chief architect, Stuart Garner, told me to take a day or two and clean it up and release it for the world to use, and so here it is.

Features

Here is a brief overview of the features of CoverMe:

Index Page

• Sortable column headers (File, Lines, Lines of Code, Tested %).
• Searching/filtering by file name.
• Filtering by coverage percent.
• Color coded list of files to quickly see which ones are 100% covered, > 90% covered, or less than 90% covered.
• Large color coded average coverage percent, for quick reference.

Detail Page

• Line by line coverage report
• Color coded lines to quickly see which lines where executed and which ones were not.
• Side by side viewing with the corresponding test/spec file (if one exists).

See the README file for more information on installation and usage.

Thanks

I would just quickly like to give another quick thanks to Aaron Patterson for pointing out the Coverage module in Ruby 1.9 and inspiring this, hopefully, helpful little gem. Also another big thanks to Stuart Garner for pushing me to package this up and release it to the world.

Screenshots