Selenium: Early Thoughts on Test Automation

I have recently been running a trial of Selenium to automate some of our regression and integration testing. I have only been looking into this for a short amount of time so I am by no means an expert but this post contains a few of my observations so far.

For those of you that are not familiar with it, Selenium is a browser automation system that allows you to write integration tests to control a browser and check the response of your site. An example of a Selenium script might look like this:

  1. Open the browser
  2. Browse to the login page
  3. Enter “user 1″ in the input with ID #username
  4. Enter “pa$$word” in input with ID #password
  5. Click the Login button and wait for the page to load
  6. Check that the browser has navigated to the users home page

Selenium as a framework comes in 2 flavours: IDE & WebDriver.

Selenium IDE

IDE uses a record-and-playback system to define the script and to run the tests. It is implemented as a FireFox plugin and is therefore limited to FireFox only.

We had run a previous trial using this version where we attempted to have our QA team record and execute scripts as part of functional and regression testing. We found that this had a number of problems and eventually abandoned the trial:

  • Limited to FireFox
  • Has to be run manually (i.e. Cannot be run automatically on a build server)
  • Often requires some basic understanding of JavaScript or CSS selectors to work through a problem in a script; this was sometimes beyond the technical knowledge of our QA team
  • Automatically-generated selectors are often extremely fragile. Instead of input#password, it might generate body > div.main-content > form > input:last-child. This meant that a lot of time was lost to maintenance and that the vast majority of “errors” reported by the script were actually incorrect selectors.

We decided that there we too many disadvantages with this option and so moved onto Selenium WebDriver.

Selenium WebDriver

WebDriver requires that all scripts are written in the programming language of your choice. This forced the script-writing task onto our development team instead of QA, but also meant that development best-practices could be employed to improve the quality and maintainability of the scripts.

This version of Selenium also (crucially) supports multiple browsers and can be run as part of an automated nightly build so seemed like a much better fit.

Whilst writing our first few Selenium tests we came up with a few thoughts on the structure

Use a Base Fixture for Multiple Browser Testing

This is a nice simple one – we did not want to write duplicate tests for all browsers so we made use of the Generic Test Fixture nUnit feature to automatically run our tests in the 4 browsers in which we were interested.

We created a generic base fixture class for all our tests and decorated it with the TestFixture<TDriver> attribute. This instructs nUnit to instantiate and run the class for each of the specified generic types, which in turn means any test we write in such a fixture will automatically be run against each browser

[TestFixture(typeof(ChromeDriver))]
[TestFixture(typeof(InternetExplorerDriver))]
[TestFixture(typeof(FirefoxDriver))]
public abstract class SeleniumTestFixtureBase<TWebDriver>
	where TWebDriver : IWebDriver
{
	protected IWebDriver Driver { get; private set; }

	[SetUp]
	public void CreateDriver()
	{
		this.Driver = DriverFactory.Instance
			.CreateWebDriver<TWebDriver>();
			
		//...
	}
}

This does have some disadvantages when it comes to debugging tests as there are always 4 tests with the same method name but this has only been a minor inconvenience so far – the browser can be determined from the fixture class name where needed.

Wrap Selectors in a “Page” Object

The biggest problem with our initial trial of “record and playback” automated tests was the fragility of our selectors. Tests would regularly fail when manual testing would demonstrate the feature clearly working, and this was almost always due to a subtle change in the DOM structure.

If your first reaction to a failing test is to say “the test is probably broken” then your tests are useless!

A part of the cause was that the “record” part of the feature does not always select the most sensible selector to identify the element. We assumed that by hand-picking selectors we would automatically improve the robustness (is that a word?) of our selectors, but in the case where they did change we still did not want to have to update a lot of places. Similarly, we did not want to have to work out what a selector was trying to identify when debugging tests.

Our solution to this was to create a “Page” object to wrap the selectors for each page on the site in meaningfully named methods. For example, our LoginPage class might look like this:

public class LoginPage
{
	private IWebDriver _driver;

	public LoginPage(IWebDriver driver)
	{
		_driver = driver;
	}

	public IWebElement UsernameInput()
	{
		return _driver.FindElement(By.CssSelector("#userName"));
	}

	public IWebElement PasswordInput()
	{
		return _driver.FindElement(By.CssSelector("#Password"));
	}
}

This has a number of advantages:

  • Single definition of the selector for a given DOM element
    We only ever define each element once
  • Page inheritance
    We can create base pages that expose page elements which appear on multiple pages (e.g. the main navigation or the user settings menu)
  • Creating helper methods
    When we repeat blocks of functionality (e.g. Enter [usename], enter [password] then click Submit) we are able to encapsulate them on the Page class instead of private methods within the test fixture.

We also created factory extension methods on the IWebDriver element to improve readability

public static class LoginPageFactory
{
	public static LoginPage LoginPage(this IWebDriver driver)
	{
		return new LoginPage(driver);
	}
}

//...
this.Driver.LoginPage().UsernameInput().Click()

Storing Environment Information

We decided to store our environmental variables in code to improve reuse and readability. This is only a minor point but we did not want to have any URLs, usernames or configuration options hard coded in the tests.

We structured our data so we could reference variables as below:

TestEnvironment.Users.AdminUsers[0].Username

Switching between Debug & Release Mode

By storing environment variables in code we created another problem: how to switch between running against the test environment and against the local developer environment.

We solved this by loading certain changeable elements of our configuration from .config files based on a #DEBUG flag

Other Gotchas

  • The 64bit IE driver for Selenium IDE is incredibly slow! Uninstall it and install the 32-bit one
  • Browser locale can – in most cases – be set using a flag when creating the driver. One exception to this is Safari for Windows, which does not seem to allow you to change the locale at all – even through Safari itself!

Summary

We are still in the early phases of this trial but it is looking like we will be able to make Selenium automation a significant part of our testing strategy going forward.

Hopefully these will help out other people. If you have any suggestions of your own then leave them in the comments on message me on Twitter (@stevegreatrex).

About these ads

Chrome Dev Tools & Inline Dynamic JavaScript

If you are using Chrome dev tools to debug your application then you might have come across this situation.  If you dynamically load some content, and that content contains an inline <script> tag, then annoyingly you can’t see that script under Source in the developer console.

Thankfully there’s a nice simple solution to the problem: insert the following tag at the end of your inline script:

<script>
    //...
    //@ sourceURL=MyInlineScript.js
</script>

This will make the script appear in the Sources list under the “No Domain” section:

InlineScript

Remember that if the inline script is part of a Razor view then you will need to escape the @

<script>
    //...
    //@@ sourceURL=MyInlineScript.js
</script>

Are Your Users “Sure”?

Are You Sure?

You’ve probably been asked the question a hundred times already today: are you sure you want to do that?

Cancel

Are you sure you want to delete that file?  Are you sure you want to log out?  To change that extension?  To update this setting?  Are you sure about anything any more?

One of my pet hates is working with an application that is constantly doubting me.  “Yes I’m sure – that’s why I clicked the button!”

Based on personal experience and absolutely no real data, I estimate that I answer “no” to that question…once a week?  A month?  Not often, certainly, and yet over and over again I have to keep expressing my certainty about every little decision.

Just Stop Asking

I’ve been working on a new version of an existing web application recently and one of the key UX decisions has been to stop asking unnecessary questions.  If the user says “jump” then don’t ask if they’re sure; don’t even ask them “how high”…just do it!  We want to trust our users to know what they are doing – wherever possible, we want to do the most obvious thing first and ask questions later.

But What About…

Ah, yes, good point.  Quite often, users don’t know what they’re doing.  And quite often, they’re going to do something wrong.  This, presumably, is the reason that we are constantly asked if we really want to do something: if we complain later then at least the developer can say “well you did say you were sure…”

So how can we account for the (hopefully rare) mistakes and still not get in the way of the average user?

Our approach is to make everything undoable.  Make the consequences of even the serious-sounding actions (“delete this forever?”) recoverable if they happen by accident.  In fact…

Make it as difficult as possible to seriously cock things up

When the user deletes something, give them a little notification saying “Great, that’s gone… unless you click this: [Undo]”.  It doesn’t need to be a big notification – just something that they’ll notice if they’re sat in a cold sweat thinking they’ve just thrown away the last 4 hours’ worth of work.

Undo Popup

Promote Exploration

There’s another up-side to this approach – it encourages users to play around and explore.  If they are confident that they cannot accidentally break something permanently then – hopefully – they will be happier to try something and see what happens.

In a lot of applications, “fear of breaking it” can be a pretty serious barrier to adoption.  It makes sense as well – if someone is not too computer-literate then they probably should be worried about doing something wrong.  But if, when they accidentally delete next months payroll, they have a big friendly button saying “don’t worry – just click here and everything will be back to normal” then you hope that they will worry less about clicking that button next time.

Keep IIS Express Running in Visual Studio 2013

Since upgrading to Visual Studio 2013 I’ve noticed a change in behaviour in IIS.  It still starts up when you start debugging a web project – same as it always has – but since the upgrade it automatically shuts down when you stop debugging.

As with most IDE behaviour, I was pretty familiar with the old way and so I found it incredibly frustrating whenever this happened.  The good news is that there’s a very simple solution: disable Edit and Continue for the project in the Properties dialog.

edit continue

Hopefully this will save someone else some pain!

3 Ways to Deal with SFOUC in KnockoutJS

What is SFOUC?

A Sudden Flash Of Unstyled Content, or SFOUC, refers to that irritating few milliseconds between when your web page loads and when all of your dynamic content pops into place.

The reason for this annoying phenomenon is that your HTML is being rendered a split second before your CSS & JavaScript files have finished downloading and running.  The un-augmented HTML sits there just long enough to catch the eye of your user, and then is whisked away as soon as all the lovely dynamic content is ready to replace it.

I’ve found that this is particularly evident when working with KnockoutJS because generally the last line of your startup code is ko.applyBindings(viewModel), and until you bind the view model you always see the unstyled content.

So how can we stop this menace?

Option 1: The “Classic” Approach

KnockoutJS is just a library on top of JavaScript so there is no reason why we can’t use the same approach as is suggested for non-Knockout apps.

There are a few flavours to this, but as a general solution:

  1. Add a class to the body element (e.g. no-js)
  2. Add a class to the dynamic content (e.g. dynamic-content)
  3. Add a style that hides dynamic-content within no-js

    .no-js .dynamic-content { display: none; }
    
  4. Add some JavaScript to remove the no-js class from the body once everything is loaded

    document.body.className = 
        document.body.className.replace("no-js", "");
    

This will do the job quite nicely, but still relies on the CSS loading quickly enough to hide the dynamic content before anyone notices.

Option 2: The “Visible” Approach

If the idea is to reduce the amount of time the basic HTML is visible then the fastest place to put the “don’t show this” instruction is in the HTML itself – using a style="display:none" tag.

The problem with this approach is that you then have to to go through and remove all of those styles once your JavaScript is ready to run; this is where the visible binding can help us out.

Under the covers, the visible binding simply sets the display style to none or to "" (i.e. not set) so we can use this to automatically remove the style tags created above:

<div style="display:none" data-bind="visible: true"> </div>
<!-- or -->
<div style="display:none" data-bind="visible: $data"> </div>

The first of these examples will display the div as soon as Knockout is bound; the second will display as soon as the current context has a truthy value.

Option 3: The “Make-Everything-A-Template” Approach

Instead of trying to hide rendered HTML content until we’re ready, what if we just stopped rendering it all together?  If we put all of our dynamic content inside script tags (which obviously won’t be rendered by the browser) then we can use the template binding to render the content from inside the script tags in the context of a view model.

<div data-bind="template: 'dynamic-content'">
    <!-- will appear empty on page load -->
</div>

<script id="dynamic-content" type="text/html">
    <h1>This will be rendered by Knockout</h1>
</script>

This makes for slightly more verbose markup than the other approaches but we can improve on this by switching to the containerless syntax:

<!-- ko template: "dynamic-content" --><!-- /ko -->
<script id="dynamic-content" type="text/html">
    <h1>This will be rendered by Knockout</h1>
</script>

A Note about Progressive Enhancement

Obviously if you want to employ progressive enhancement then these approaches will not work for you – they work by hiding the content until it is ready to be rendered.  If you are using progressive enhancement then I’m going to assume that the original HTML renders nicely anyway and that you don’t care about the SFOUC problem!

Protecting your CouchDB Views

If you work with a SQL or other RDBMS database you most likely have your schema backed up somewhere under source control.  Maybe it’s a bunch of SQL scripts, maybe it’s the classes from which you generated your Entity Framework schema, but you almost certainly have some way of restoring your DB schema into a new database (at least I hope that you do!).

couchdb

But what about CouchDB?

CouchDB, as anyone who has read the first sentence of a beginners guide will know, is a Non-Relational Database and so it does not have a schema.  All of the data is stored as arbitrary JSON documents which can (and do) contain data in a wide range of formats.

The problem is that whilst there is no schema to “restore” into a new database, there is another very important construct: views.

CouchDB Views

Views within CouchDB define how you query the data.  Sure, you can always fall back to basic ID-lookup to retrieve documents, but as soon as you want to do any form of complicated (i.e. useful) querying then you will mostly likely need to create a view.

Each view comprises 2 JavaScript functions: a map function and an optional reduce function.  I don’t want to go into a lot of detail on the map-reduce algorithm or how CouchDB views work under the covers (there are plenty of other resources out there) but the important thing here is that you have to write some code that will play a very significant role in how your application behaves and that should be in source control somewhere!

Storing Views in Source Control

In order to put our view code under source control we first need to get it into a format that can be saved to disk.  In CouchDB, views are stored in design documents and the design documents are stored as JSON, so we can get a serialized copy of the view definitions by just GETting the design document from couch:

curl http://localhost:5984/databaseName/_design/designDocumentName

Pass the output through pretty-print and you will see the contents of the design document in a JSON structure:

{
   "_id": "_design/designDocumentName",
   "_rev": "1-47b20721ccd032b984d3d46f61fa94a8",
   "views": {
       "viewName": {
           "map": "function (doc) {\r\n\t\t\t\tif (doc.value1 === 1) {\r\n\t\t\t\t\temit(\"one\", null);\r\n\t\t\t\t} else {\r\n\t\t\t\t\temit(\"other\", {\r\n\t\t\t\t\t\tother: doc.value1\r\n\t\t\t\t\t});\r\n\t\t\t\t}\r\n\t\t\t}"
        }
   },
   "filters": {
       "filterName": "function () {}"
   }
}

This is, at least, a serialized representation of the source for our view, and there are definitely some advantages to using this approach.  On the other hand, there are quite a few things wrong with using this structure in source control:

Unnecessary Data
The purpose of this exercise is to make sure that the view code is safely recoverable; whilst there is debatably some use in storing the ID, the revision (_rev) field refers to the state of the database and may vary between installations and shouldn’t be needed.

Functions as Strings
The biggest problem with this approach is that the map, reduce and filter functions are stored as strings.  You may be able to put up with this in simple examples, but as soon as they contain any complexity (or indentation, as seen above) they become completely unreadable.  Tabs and newlines are all concatenated into one huge several-hundred-character string, all stored on one line.  Whilst this is not a technical issue (you could still use these to restore the views) it makes any kind of change tracking impossible to understand – every change is on the same line!

As well as the readability issues we also lose the ability to perform any kind of analysis on the view code.  Whether that is static analysis (such as jsLint), unit testing or some-other-thing, we cannot run any of them against a string.

An Alternative Format

Instead of taking a dump of the design documents directly from CouchDB, I would recommend using an alternative format geared towards readability and testability.  You could be pretty creative in exactly how you wanted to lay this out (one file per design document, one file per view…) but I have found that the structure below seems to work quite well:

exports.designDocumentName = {
	views: {
		viewName: {
			map: function (doc) {
				//some obviously-fake logic for demo purposes
				if (doc.value1 === 1) {
					emit("one", null);
				} else {
					emit("other", {
						other: doc.value1
					});
				}
			}
		}
	},
	filters: {
		filterName: function () { }
	}
};

exports.secondDesignDocument = {
	//...
};

This has several advantages over the original format:

  • It is much easier to read!  You get syntax highlighting, proper indentation and the other wonderful features of your favourite code editor
  • There is no redundant information stored
  • jsLint/jsHint can easily be configured to validate the functions
  • By using the AMD exports object, the code is available to unit tests and other utilities (more on that below)

There is one significant disadvantage though: because I have pulled this structure out of thin air, CouchDB has no way of understanding it.  This means that whilst my view code is safe and sound under source control I have no way of restoring it.  At least with the original document-dump approach I could manually copy/paste the contents of each design document into the database!

So how can we deal with that?

Restoring Views

As I mentioned above, one of the advantages of attaching design documents as objects to the AMD exports object is that they can be exposed to node utilities very easily.  To demonstrate this I have created a simple tool that is able to create or update design documents from a file such as the one above in a single command: view-builder.

You can see the source for the command on GitHub or you can install it using NPM.

npm install -g view-builder

After installation you can run the tool like this:

view-builder --url http://localhost:5984/databasename  --defs ./view-definitions.js

This will go through the definitions and for each of the design documents…

  1. Download the latest version of the design document from the server
  2. Create a new design document if none already exists
  3. Compare each view and filter to identify any changes
  4. If changes are present, update the version on the server

The comparison is an important step in this workflow – updating a design document will cause CouchDB to rebuild all of the views within it; if you have a lot of data then this can be a very slow process!

Now we have a human-readable design document definition that can be source-controlled, unit tested and then automatically applied to any database to which we have access.  Not bad…

Other Approaches

Whilst this system works for me, I can’t imagine that I am the first person to have considered this problem.  How is everyone else protecting their views?  Any suggestions or improvements in the comments are always welcome!

Concurrently Testing JavaScript

You’ve heard of nCrunch, right?  It’s a continuous integration plugin for Visual Studio that tracks changes to your code and then automatically builds and runs your unit tests in the background whilst you type.  Once the results are back it decorates every line of code with a red or green light to indicate whether the tests are passing or failing.

image

I have been using nCrunch for a while now and cannot recommend it enough.  It has it’s faults – what doesn’t? – but all things considered it is probably the one add-on that I couldn’t live without.  It speeds up the red/green/refactor cycle to such an extent that working without it feels like you’ve lost a limb.

Where are you going with this?

Good question.  nCrunch is fantastic…for .NET development.  I do a fair bit of work in .NET but more and more I find myself working in JavaScript and desperately missing the constant feedback.

So how can we get something similar for our JavaScript tests?  By using grunt of course!

Testing with Grunt

GruntJS (which I’ve been playing around with recently) can be setup to run QUnit tests using the headless PhantomJS browser and to feedback the results.  In fact, this is all nicely packaged up in the grunt-contrib-qunit package for you, so it’s pretty simple to set up.

To get the tests set up I’m going to assume you have a project structure similar to this one…

image

Installing dependencies with npm

Life is much easier when you don’t have to set stuff up yourself, so let’s use npm to get everything installed.  Drop the following package.json file into the tests folder then open up a node command prompt in the same location.

{
  "name": "myapp-tests",
  "version":"0.0.1",
  "devDependencies": {
    "grunt": "~0.4.1",
    "grunt-cli": "~0.1.9",
    "grunt-contrib-qunit": "~0.2.2",
    "grunt-contrib-watch": "~0.5.3"
  }
}

Run npm install and wait patiently until all the packages are installed.

All done?  Great.

The Gruntfile

Now that we have grunt installed we need to tell it what to do.  Create a new gruntfile.js in the tests folder of the project and drop in the following:

module.exports = function (grunt) {
    
    grunt.initConfig({
        //configure the qunit task to load test-runner.html
        qunit: {
            files: ["test-runner.html"]
        },
        //configure the watch task to react to any changes to *.js
        //in either the src or the current (test) folder
        watch: {            
            files: ["../src/**/*.js", "*.js"],
            tasks: ["qunit"]
        }
    });
    
    //load the qunit and watch tasks from npm
    grunt.loadNpmTasks("grunt-contrib-qunit");
    grunt.loadNpmTasks("grunt-contrib-watch");
    
    //configure the default task to invoke watch
    grunt.registerTask("default", ["watch"]);
};

This file configures the tasks we want to be run by grunt:

  • The qunit task is configured to invoke test-runner.html
  • The watch task is configured to monitor the current and the ../src folders for any changes to a JavaScript file, and to invoke the qunit task whenever a change is made.

We also set up the default task to invoke watch so that it will be run automatically

Fire up a console window and run grunt; you should see something like this:

image

Now go and make a change to target.js

image

As soon as we save any change to a .js file our tests will automatically be run in the background.

True, it isn’t nicely integrated into the IDE, and the level of feedback is very basic compared to nCrunch…but it’s a start!