Automating Tests

[180418 – updated to reflect move to ChromeDriver]

I feel like a fraud!

For months — no! Years! — I have been writing about testing, more specifically automated testing. I have shared my Site Infrastructure Tests github, I have spoken about it at the Adobe Summit in London, and I have mentioned multiple times that all of this should be integrated with the workflow of the developers, your workflow.

But I have never explained how to do that!

Worse: I have no idea how to do that!

I am not a developer (well, I used to be a C developer, in the days of Palm OS, but does that count? Not anymore, for sure), and I frankly don’t know what the current state of the art for testing is. I have therefore chosen to ignore this glaring hole, like a lousy trickster.

But not anymore!

Today, I’ll show you how to take a Raspberry Pi, install Jenkins on it and automate your tests.

That should add a considerable amount of protection against accidentally breaking any tracking, or at least it should detect breakage very quickly!

Feel free to build two, and put one on your friendly marketer’s desk, too.

And if we’re really lucky, you are using Jenkins anyway, so this little project neatly maps into your workflow.

Shopping list

For this project, you will need:

1. 1 Raspberry Pi (any version, but with network access),
2. 1 power supply and Micro USB cable
3. 1 screen, mouse, and keyboard, or familiarity with SSH and a bash — up to you
4. 1 MicroSD card
5. NOOBS

I’m not going to go into all the steps needed to setup your Raspberry Pi. The guide on the NOOBS page should get you going. Choose “Raspbian” as the operating system.

Make sure you set the timezone correctly! If the time on your Raspberry Pi is too far off, you’ll be unable to install packages over HTTPS. Been there, pulled my hair over that, felt foolish once I found out.

Once you have a working Raspberry Pi with Raspbian, and you have successfully logged into it, you’re ready to go.

You should update everything before moving on, plus teach apt HTTPS, for security reasons:

sudo apt -y install apt-transport-https
sudo apt update
sudo apt -y dist-upgrade
sudo apt-get -y autoremove
sudo rpi-update
sudo reboot

Step 1 — Install Stuff

You’re going to need some things on the Pi — Java, phantomJS, Chromium, ChromeDriver, Maven — the same things we needed on Windows and Mac, obviously. On top of that, you will install Jenkins.

Since this is Raspbian, which is a fork of debian, which is Linux, we’ll handle this via command line. Go ahead, open up a terminal.

Install Maven first:

sudo apt -y install maven

You’ll notice that quite a bunch of dependencies are installed along with Maven.

I’ll be totally honest with you: there is actually no need to install Maven at this point, as we will later configure Jenkins to install a version. But I’d do it anyway so I can run tests on the command line.

You’ll want to use Oracle’s Java implementation, not the openJDK. (Although, disclaimer: I don’t remember why. Feel free to try with openJDK.)

sudo update-alternatives --config java

Select the Oracle JDK 8. This is how it should look:

Now let’s find phantomJS somewhere. It is unfortunately not available in the normal Raspbian repository, but you can grab it from this github.

wget https://github.com/fg2it/phantomjs-on-raspberry/releases/download/v2.1.1-wheezy-jessie-armv6/phantomjs_2.1.1_armhf.deb
sudo dpkg -i phantomjs_2.1.1_armhf.deb
sudo apt-get install -f

Note that the proper way would have been to compile phantomJS from sources. I tried that once on a Raspberry Pi 2, despite warnings it would take days. I ended up downloading a binary, when I ran into missing dependencies issues and other hiccups.

I am fully aware that downloading binaries from sites I don’t trust is a bad idea. If you have a linux / dev person that could help you build phantomJS, please do so!

Install Chromium using the following command:

sudo apt install chromium-bsu

Selenium uses WebDriver to control Chromium, and the piece of software actually doing that is called ChromeDriver. It’s not part of the standard Raspbian libraries, but thankfully, the folk at the electron project have a pre-compiled version that we can use. Install it like so:

wget github.com/electron/electron/releases/download/v1.6.0/chromedriver-v2.21-linux-armv7l.zip
unzip chromedriver-v2.21-linux-armv7l.zip
sudo cp chromedriver /usr/local/bin/
sudo chmod a+x /usr/local/bin/chromedriver

Try whether you’ve done that correctly like so:

chromedriver --version

The result should be this:

Checkpoint

Ok. Let’s pause for a moment and make sure you’re on the right track.

Now that you have installed Maven and phantomJS, you can actually run a test. Here’s how:

mkdir dev
cd dev
git clone https://github.com/janexner/site-infrastructure-tests.git
cd site-infrastructure-tests
mvn test -Dwebdriver.chrome.driver=/usr/local/bin/chromedriver

Maven should now load a ton of dependencies, then run the default test. The output should look somewhat like this:

Awesome!

Step 1 — cont’d

With the right JDK and all other things in place, you can install Jenkins.

The version provided with the official Raspbian repository is quite old, which means you should pick up Jenkins from its own repository on jenkins.io. Add the repository:

sudo /bin/su -c "echo 'deb https://pkg.jenkins.io/debian binary/' > /etc/apt/sources.list.d/jenkins.list"
wget -q -O - http://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -

Worked? Good. Now install Jenkins:

sudo apt update
sudo apt -y install jenkins

Note: If apt tries to install jenkins 1.x, double-check the time on your Raspberry Pi. Yes, you read that right. It should install jenkins 2.x, really, but the jenkins.io repository is accessed via HTTPS, which doesn’t work if the time on your Pi is off too much. Happened to me.

After downloading a whopping 70M of packages, apt will install Jenkins and a service. Reboot your Pi, so the service can start. Give your Raspberry Pi some time, it’ll need it.

Step 2 — Setting up Jenkins

You now have a Jenkins continuous integration server. It wants to be configured.

Point your browser to localhost:8080 to start the configuration. (That is the browser on your Pi! If you know the Pi’s IP address, feel free to use the browser on your laptop instead, of course.)

The very first step is to unlock Jenkins.

Running this

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

will show you the password, which you can then c&p into your browser.

Next, it is time to customize Jenkins by adding plugins. Go for the “Select” option, and make sure you select at least “Dashboard View” and “Github Plug-in”. Click “Install”.

Jenkins works a little bit, then gives you an empty dashboard.

Now, tell Jenkins to install Maven as a tool.

Click the “Manage Jenkins” link in the menu on the left, then “Global Tool Configuration” from the list.

Scroll down to Maven, Add an installer. It should look like this:

Right, got it.

Step 3 — Creating a Job

You are now ready to create a job. That job will later be responsible for running our regular test. So exciting!

Give it a name, select “Maven Project”, then click the little “Ok” button that floats around in the lower left.

On the long page that you get now, you have to configure at least three things:

• the source of the code — where does it come from?
• the build triggers — when should it test?
• how to run the actual test

I guess the two things that need explaining on this long page are

1. the Build Triggers > Schedule box (“H H * * 7”), and
2. the Build > Goals and options input field

The former is in cron format. Unix/Linux-based systems have had this since the 70ties. Don’t want to go into details, but “H H * * 7” essentially means “at some time every Sunday”.

The latter, the “Goals and options” field, contains the exact same command line options that you would usually put right after the mvn command.

The example in the screenshot tells Maven to run a clean, then a test and pass the -Dtest.description.file=tests.webade.json (and -Dwebdriver.chrome.driver=/usr/local/bin/chromedriver, TBD) parameters into the testing tool.

Note that you can easily run multiple, different tests on your Raspberry Pi by creating multiple jobs. The only difference between them will likely be the “Goals and options” field, pointing to different test description JSON files.

Ok, you’re done!

From now on, the job will run every Sunday, as long as the Pi is powered on.

You can run the job from the Dashboard manually at any time.

Step 4 – Understanding the Results

Jobs can be successful, or they can fail.

The Jenkins website explains what those “Job Status” and “Build Status” icons mean.

The yellow circle in the first column, first row says that the status for the first job is “unstable”, meaning there have been failures recently. The blue circles in rows 2 & 3 say “all good here!”

The “W” column depicts the overall situation, the weather. All my jobs look pretty sunny.

If a job works, you’re happy, and there is nothing to do, really.

If it fails, you’ll want to know why.

My suggestion is to use the “Console Output” to diagnose the issue behind the failure. You get there via the job number (“#12” above).

The output you’ll see is exactly the same as the output you’d get if you run Maven manually. You can see which test(s) failed, and act accordingly. This will likely include a meeting with your friendly marketer!

What next?

Firstly, you should test your own site, build a JSON test description file for all the things on your site that are valuable for your friendly marketer.

The easiest way to do so would be to put your JSON file into a folder somewhere on your Raspberry Pi and refer to it in the test setup.

The good way of doing it would be to fork my github project, add the new JSON file to your own fork, then use that fork in the test setup.

That way you automatically get version control for your test descriptions, which is definitely a good thing!

You can even create different branches in your git, and test against those!

Maintenance

Do not forget that software grows old much faster than humans!

Every now and again (I do this every day, but weekly might be fine), log into your Raspberry Pi and do the updates:

sudo apt update
sudo apt -y dist-upgrade
sudo apt-get -y autoremove

If the updates include the kernel, you should reboot your Pi at that point, but otherwise there is no need. Just keep it running.

Feel free to comment. Does this work for you? Do you test differently? I want to hear what people do to keep data quality up!