Saturday, March 22, 2014

Full automation of Protractor E2E tests using Grunt - Part 2

This is a continuation of the article series on automating protractor tests using grunt. You can see the first part here

In this part we will be doing the following tasks.

  • Adding the setup tasks to grunt to make the whole process easier to start. 
  • Adding an HTML reporter

Adding the setup command tasks


When you create a repo, you may want to clone this repo on several machines. Or, when someone else wants to use your repo to use in his/her project. When they do that, they also have to go through all the basic setup that you have done like installing selenium, chromedriver etc. We don't want that. 

We are doing all our setup on the command console. Then, why not automate it? Is there any grunt plugin for that? Yes. There is. We can use grunt-shell-spawn for running all the shell commands.

Lets add grunt-shell-spawn module to our project.



Now, we will add the shell-spawn task to Gruntfile.


   You can see that I have added two options. 'protractor_install' and 'npm_install'. It is better to add an npm install before we are installing chrome driver and selenium just to make sure that, everything is ready beforehand.

  Now we will add the install task. I have added both 'npm_install' and 'protractor_install' to the install task.




Now, if we do a grunt install on the command console, it will show as everything uptodate, since we have already installed selenium and chromedriver in the current project folder.


PS: You can see some warning. Please ignore that for now. This is because we haven't added the README.md file and repo to the project, which we will add at the end.

To demonstrate the usefulness of what we did now, lets make a copy of our project folder. Make sure that, you are not copying the node_modules folder. I copied the files to 'exampleCopy' folder. Go to this copy folder in the command console.

Now, if we do the grunt install, everything should work, right? Let's see.



It is saying that a local copy of grunt is required. So, we have to do an npm install before we call any grunt tasks. Let's do that.

1. npm install

2. grunt install


Now you are ready to do the protractor tests. So, any other person using this repo only need to run two commands "npm install" and "grunt install" and they are ready to go.

Adding the HTML Reporter

When we run the tests, we can see that tests are passing or failing in the command console. What if we have hundreds of tests and also want to keep a record of the test results  whenever we run the tests? We need test reporters for that.

Protractor gives the options to add onPrepare function property on its configuration file. This onPrepare function will be called before each tests. Jasmine test framework(which we are using inside protractor) provides the option to add test reporter. We will get the metadata about each test by using these two configurations.

There is a node module available for generating screenshots for each tests. protractor-screenshot-reporter. However this doesn't process the meta data and gives us the result in a readable format.

I couldn't find a node module doing this for protractor. So, I created a module for generating this HTML report, on top of protractor-screenshot reporter, protractor-html-screenshot-reporter. What it does is generate an HTML report with all the details of the test like status, browser used, message, link to screenshot etc. (If you are finding some other modules which are doing the same, please add in the comments. )

Lets install protractor-html-screenshot-reporter module in our project.


We need to add this reporter in the Protractor configuration file.

1. require the protractor-html-screenshot-reporter module and assign to a variable.


2. add this reporter on the onPrepare function.


onPrepare: function() {
      // Add a reporter and store screenshots to `screnshots`:
      jasmine.getEnv().addReporter(new HtmlReporter({
         baseDirectory: 'screenshots'
      }));
   },

3. Run the test using grunt command. It will run the default task in the Gruntfile.



   We can see that the tests are passing. Lets see if the reports are generated.

Screenshots folder is created.


Inside the screenshots folder, one json and png with a guid filename is available. This is the metadata and screenshot respectively for the single test. There is also a combined.json which combines all the metadata into a single file and a reporter.html file which shows all the test reports.


Opening the reporter.html looks like below.



Clicking on View link opens the screenshot image also.




So, now you have an html reporter ready. But, there is one issue though. Whenever you run the tests, the reporter files are getting overridden. If I want to keep a track of tests, this is not going to help.

protractor-html-screenshot-reporter provides the option of pathBuilder function property to give your own dynamic paths. We will add this in Protractor configuration file.



  onPrepare: function() {
      // Add a reporter and store screenshots to `screnshots`:
      jasmine.getEnv().addReporter(new HtmlReporter({
         baseDirectory: 'screenshots',
         pathBuilder: function pathBuilder(spec, descriptions, results, capabilities) {
          
            var monthMap = {
              "1": "Jan",
              "2": "Feb",
              "3": "Mar",
              "4": "Apr",
              "5": "May",
              "6": "Jun",
              "7": "Jul",
              "8": "Aug",
              "9": "Sep",
              "10": "Oct",
              "11": "Nov",
              "12": "Dec"
            };

            var currentDate = new Date(),
                currentHoursIn24Hour = currentDate.getHours(),
                currentTimeInHours = currentHoursIn24Hour>12? currentHoursIn24Hour-12: currentHoursIn24Hour,
                totalDateString = currentDate.getDate()+'-'+ monthMap[currentDate.getMonth()]+ '-'+(currentDate.getYear()+1900) + 
                                      '-'+ currentTimeInHours+'h-' + currentDate.getMinutes()+'m';

            return path.join(totalDateString,capabilities.caps_.browserName, descriptions.join('-'));
         }
      }));
   },


I am just using a basic function to give the folder name as the current date and time and browser.

Note that I am using the path module for creating the file path (path is a native node module for handling and tranforming file paths. ). Make sure that, you have required the path and assigned to a variable.





Lets run the test again using grunt command in command console. As expected tests are running successfully.

If we check the project folder we can see that, the report is now generated in the path 'screenshots/22-Feb-2014-5h-58m/chrome'. Great. Now, on seeing the folder names itself, you know when the test was run and on which browser.


Finally, lets make this project ready to add as a git repo. 

1. It is not a good a good practice to push node_modules directory to the repo. In order to avoid node_modules to be pushed to git, we need to add it to .gitignore file. 

Create a new file in the root folder with name as .gitignore. 



We don't want screenshots and node_modules folder to be added in git. So, I have added both names in .gitignore. 

2. We will also add the repository details to package.json. 



3. Also, create a README.md file in the root folder and add some description of your project. 

Finally, we are ready to push to the repo. 

More Requirements

 Now that we are in a good state, we can check the further requirements.

  1. Generating tests based on style guide
  2. Doing CSS Regression testing

We will cover that in the coming articles. Stay tuned.

Please share your comments.

All the changes are added on github repo - protractor-e2e-bootstrap


Tuesday, March 18, 2014

Full automation of Protractor E2E tests using Grunt - Part 1

Everyone now talking about Test Driven Development(TDD), Behaviour Driven Development(BDD) and all. I also got interested in the topic and tried to learn more about it. I tried several frameworks like Jasmine, Mocha, Siesta(for an ExtJS application), but couldn't go further on this. I always ended up in stopping after doing some sample tests

I think one reason for that could be that, I didn't see the practical use of Test
Then came a requirement in our project to refactor an existing application. Before I start into details of testing, let me give you some insights of the current app structure.

  • Half of the application is based on jQuery. 
  • Other half is the part which got converted to Angular recently, but still not completely. Some global jQuery plugins are still used as it is in Angular enabled pages 
Now the requirement is we need to convert two pages to Angular without breaking any existing functionality or styles. So, the first time I am seeing a great need of Testing.

Since ours is a web application, we are now mostly interested in browser testing. We have got an automation testing in selenium available. But that is for testing in the production site. How do we do the browser testing in localhost (local environment), with the new changes?

Here comes the Protractor for our rescue. There are already some content available in net on Protractor, setting up and all. However, most of the tutorials are mainly focused on Linux  or MacOS, not windows. I found it difficult to setup the protractor.

If we don't find it very easy to use, we will stop using and modifying the tests, because we are all lazy :-). 
I was looking everywhere to find ways to make it as easy as possible to setup and start testing. So, we should automate it. What I would have done if Grunt was not there.

So, Protractor + Grunt combination seems perfect. So, I decided to try that. I am going to write about all the steps of setting up Protractor in Grunt and the issues I have faced.

I will be also adding the samples to the following repository, so that, anyone can try it out.

Protractor-e2e-bootstrap Repo


Setting up the project

Requirements:
1. Install Node.js
2. Install Grunt-cli (Get more details here)

We will first setup a basic project by running the command 'npm init'





It will ask for a few questions and you will get a package.json created for you.



Create a folder 'specs'. We will put all our tests in specs folder. (You can give any name you want)

Installing Protractor


Lets first install Protractor.


You can install protractor globally also. Then you can run test directly from the command line using protractor command.

But since our goal is to automate the process, global installation is not required.

Notice the --save-dev argument given when we did the npm install. By giving an attribute like this, the installed module name will be added to package.json under 'devDependencies' automatically. (--save will add the module name under 'dependencies' property)



Adding Grunt


Now we will add Gruntfile.js to our root folder. This file is required for grunt to run the tests.

A minimum configuration Grunt file with jshint task added is given below.


module.exports = function(grunt) {

  grunt.initConfig({
    pkg: grunt.file.readJSON('package.json'),

    jshint: {
      files: ['Gruntfile.js', 'specs/*.js'],
      options: {
        // options here to override JSHint defaults
        globals: {
          jQuery: true,
          console: true,
          module: true,
          document: true
        }
      }

    }
  });

  grunt.loadNpmTasks('grunt-contrib-jshint');

  grunt.registerTask('default', ['jshint']);

};
For this to work, we need to install the jshint grunt plugin.




For the protractor to be run as a grunt task, we need to wrap it as a grunt plugin. There is already a grunt plugin for that, 'protracter-grunner'. We will install that now.



Adding Protractor configuration to Grunt


Now, we will add the protractor. We can put all the configuration of protractor in grunt file itself. But that will be very messy. So we will put all the configuration of protractor to a separate file.

You can get a sample protractor configuration file from the Protractor repo

https://github.com/angular/protractor/blob/master/referenceConf.js

We are going to use the following configuration file with basic settings.


// A reference configuration file.
exports.config = {
  // ----- How to setup Selenium -----
  //
  // There are three ways to specify how to use Selenium. Specify one of the
  // following:
  //
  // 1. seleniumServerJar - to start Selenium Standalone locally.
  // 2. seleniumAddress - to connect to a Selenium server which is already
  //    running.
  // 3. sauceUser/sauceKey - to use remote Selenium servers via SauceLabs.

  // The location of the selenium standalone server .jar file.
  seleniumServerJar: 'node_modules/protractor/selenium/selenium-server-standalone-2.40.0.jar',
  // The port to start the selenium server on, or null if the server should
  // find its own unused port.
  seleniumPort: null,
  // Chromedriver location is used to help the selenium standalone server
  // find chromedriver. This will be passed to the selenium jar as
  // the system property webdriver.chrome.driver. If null, selenium will
  // attempt to find chromedriver using PATH.
  chromeDriver: 'node_modules/protractor/selenium/chromedriver',
  // Additional command line options to pass to selenium. For example,
  // if you need to change the browser timeout, use
  // seleniumArgs: ['-browserTimeout=60'],
  seleniumArgs: [],

  // If sauceUser and sauceKey are specified, seleniumServerJar will be ignored.
  // The tests will be run remotely using SauceLabs.
  sauceUser: null,
  sauceKey: null,

  // ----- What tests to run -----
  //
  // Spec patterns are relative to the location of this config.
  specs: [
    './specs/*.js'
  ],

  // ----- Capabilities to be passed to the webdriver instance ----
  //
  // For a full list of available capabilities, see
  // https://code.google.com/p/selenium/wiki/DesiredCapabilities
  // and
  // https://code.google.com/p/selenium/source/browse/javascript/webdriver/capabilities.js
  capabilities: {
    'browserName': 'chrome'
  },

  // A base URL for your application under test. Calls to protractor.get()
  // with relative paths will be prepended with this.
  baseUrl: 'http://localhost:9999',

  // Selector for the element housing the angular app - this defaults to
  // body, but is necessary if ng-app is on a descendant of

  rootElement: 'body',

  // ----- Options to be passed to minijasminenode -----
  jasmineNodeOpts: {
    // onComplete will be called just before the driver quits.
    onComplete: null,
    // If true, display spec names.
    isVerbose: false,
    // If true, print colors to the terminal.
    showColors: true,
    // If true, include stack traces in failures.
    includeStackTrace: true,
    // Default time to wait in ms before a test fails.
    defaultTimeoutInterval: 10000
  }
};

We will be modifying this file in a bit. For now, lets keep it like this.

We need to add the protractor task in Gruntfile.
 },
    protractor: {
      options: {
        keepAlive: true,
        configFile: "protractor.conf.js"
      },
      singlerun: {},
      auto: {
        keepAlive: true,
        options: {
          args: {
            seleniumPort: 4444
          }
        }
      }
    }

  });


  grunt.loadNpmTasks('grunt-protractor-runner');

You can see that, I have added two options. auto and singlerun. For 'auto', I am giving keepAlive as true. This setting can be run if we need to run tests on each modification. 'singlerun' opiton will run just once and stop the task.

Finally, register the 'singlerun' task to grunt.
grunt.registerTask('default', ['jshint', 'protractor:singlerun']);

Now, that we have done the basic set up. For running protractor from grunt, lets add some test.

Adding the tests


Lets take a basic test which is given in Protractor  site and save in 'specs' folder.
describe('angularjs homepage', function() {
  it('should greet the named user', function() {
    browser.get('http://www.angularjs.org');

    element(by.model('yourName')).sendKeys('Julie');

    var greeting = element(by.binding('yourName'));

    expect(greeting.getText()).toEqual('Hello Julie!');
  });
});

Most of it is self explanatory. Protractor is a wrapper around selenium Selenium WebdriverJS with its angular flavour.

The runner exposes global variables browser, by and element. The browser variable is a wrapper around the WebDriver instance. We use the browser variable for any navigation or for grabbing any information off the page. element and by is used for selecting an HTML element, using various selectors. You can get more details on Protractor and WebdriverJS here

So, what this test does is given below:

1. browser opens the url http://www.angularjs.org
2. Gets the input having ng-model="yourName" and set the value as 'Julie'
3. Gets the element which binds the same model.
4. The expectation is that, the binding value is changed to 'Hello Julie!'



Installing Selenium and Chromedriver


Ok... So the test is ready. Now, lets try running this. Wait a minute.. we haven't installed Selenium. A standalone selenium server, and chromedriver is required to run Protractor. Luckily for us, Protractor npm module ships all the required items with it.



'webdriver-manager update' will install Selenium and Chrome driver and chromedriver. .

'webdriver-manager start' command will start the selenium server

Running Test


Keeping this command window open, open another console. Lets run the test using 'grunt' and see what happens.

You can see that browser is opening up and goes to angularjs.org.




Test is passing!!!

Improvements

However, there are some still some improvements needed.

1. I don't want to do the webmanager-update whenever I clone this project to another location. I want to make the installation easy.
2. The reports are just showing in command line. I want to see the reports in HTML with some screenshots also if possible.
3. And, some other improvements..


We will cover that in the next post. Stay tuned.

All the files are available in https://github.com/jintoppy/protractor-e2e-bootstrap

Update: Part 2 of this article series is available here.