What makes a good test?
| Monday 3rd June 2013 | Filled under Uncategorized |
I’m currently putting the finishing touches to a seminar on building simple automated tests for webpages with Selenium. As part of the session, I’ve got a section which talks about the elements of a good test. I’d actually finished writing it and was pretty pleased with the result when I suddenly realised that I’d pulled the whole thing out of my arse.
I’m pretty opinionated on QA matters. One of the biggest reasons I ended up leaving my last job was because I couldn’t get them to accept that I knew more than they did about testing and they ought to let me trash all their crappy existing infrastructure and spend lots of happy hours building them something newer and shinier. I’ve learnt a lot about tact, respect and incremental change since then, but I’m still a very outspoken advocate of continuous process improvement.
I’ve actually ended up in a fairly similar situation to last time, in that I’m surrounded by people who have been doing things in a certain way for a very long time and aren’t particularly interested in changing. It’s easy for me to see the huge glaring inefficiencies and unnecessary risks in the way that we work, but I’ve learnt that other people won’t take my word for it that things could be better, and that to all intents and purposes, if I can’t prove that the problems exist, they don’t.
Which is probably why I’ve become a huge fan of evidence-based debate instead of opinions. If I’m saying “I think we should have automated testing”, then there’s no reason why that’s more valid than another developer saying “I don’t think I want to”. But if I can send round a paper from ThoughtWorks on the subject of process maturity and demonstrate that we’re on the bottom rung of the model, that carries some weight.
So anyway, when I realised that I had no basis for my testing wisdom other than my own experience and (gasp!) opinions, it was obviously time to strap into Google and see what other people thought.
What I learnt is that there isn’t really any consensus on what makes a good test, and there’s very little research in this area. I found a few papers, most of which seemed to be just more opinions but from people with lots of letters after their name (and some of which I found myself violently disagreeing with, so obviously they were wrong anyway).
The acronym “SEARCH” (Setup, Execution, Analysis, Reporting, Cleanup, Help) is popular in some circles, although I personally feel that it sacrifices coherence for memorability (do you really need to separate analysis and reporting? And is the help system really part of the test, or even necessary if you wrote the thing properly in the first place?). I did find one description of a good test which I really loved:
The essential value of any test case lies in its ability to provide information (i.e. to reduce uncertainty).
Which is poignant, meaningful and inspiring, but of little practical help. However, with determination, pluck and a certain amount of fudging, I’ve put together a list of criteria which is not entirely just stuff I came up with myself. It’s my opinion, backed up by a couple of other people on the Internet, that a good test case should contain the following elements:
Setup / Initial State
Whether your test is manual or automated, it ought to be clear what state the system is expected to be in before the test starts. This helps to prevent differing results when the test is run under different conditions or by different people. This could include information about the tools to be used (e.g. which browser) as well as about the system itself (e.g. which version). I’m also including any required data for the test in this section.
Steps
This is the actions to be taken to carry out the test. They should be clear, unambiguous and repeatable in the case of a manual test. For automated tests, the steps should be well documented and clearly written. No matter how much you love your clever one-liner, when someone else has to figure out what the test does in 6 months’ time, they’ll hate it. Each step should have a clear purpose, and be necessary to get from the inital state to the expected state.
Expected Results
Without this, the test is pretty much pointless. There should be a simple way of telling whether the test has passed or failed, based on how the system responds to the test steps. Some of the worst tests I’ve ever seen simply printed out pages of numbers, and you had to look at the numbers and figure out whether they were ok or not. That sort of thing is a terrible idea – always go for pass/fail. In the event of a failure, your test should give a clear explanation of why the result is wrong and what it means, so that someone looking at the result can figure out whether it’s a showstopper or just an inconvenience.
Other Things
After scouring the Internet, I was forced to agree that there are a couple of other hallmarks of a good test:
- Purpose – a really good test has a single purpose, and will reveal some information about the system under test. If you can’t describe what your test is for within a non-waffly sentence, you may not have this.
- Meaningful Title – this is more applicable to automated tests, but a well designed test will have a title that means something, so that went it pops up in a list of failures, you can tell at a glance what it’s testing and what the failure is likely to mean. test_customer_3 is a useless title. test_add_customer is a bit better. But test_add_new_customer_to_contact_list gives you a good clue as to exactly what functionality is being investigated.
Adding email aliases to Alfresco folders via CMIS
| Tuesday 21st May 2013 | Filled under Uncategorized |
This took me a ridiculously long time to figure out, largely because there isn’t any documentation. So, in case anyone else out there needs to know how to do it, here’s how you set an alias on a node in an Alfresco repository in Java (via OpenCMIS and the Alfresco OpenCMIS Extensions).
try {
// folder/emailFolderName is the node path
AlfrescoFolder emailFolder = (AlfrescoFolder)session.
getObjectByPath("folder/emailFolderName");
} catch (CmisObjectNotFoundException e) {
// Folder doesn't exist - handle appropriately
}
emailFolder.addAspect("P:emailserver:aliasable");
Map<String, String> properties = new HashMap<String, String>(1);
// my-email-folder is the chosen alias for this node
properties.put("emailserver:alias", "my-email-folder");
emailFolder.updateProperties(properties);
And that’s all there is to it. I wish it hadn’t taken me nearly a whole day to work that out…
One thing I have discovered is the incredibly useful CMIS Workbench, highly recommended for working out the CMIS names for obscure properties. There’s a useful guide to getting started with it in the answer to this StackOverflow question.
Disclosure
| Monday 25th February 2013 | Filled under Uncategorized |
Disclaimer:
This story doesn’t have a happy ending. Bearing in mind some of the people who might read this, I thought I’d say up front that my tribe and I are all coping OK, and are still feeling delighted for our more fortunate friends. I don’t want there to be awkwardness, but I couldn’t cope with the secrecy any more. I’m not looking for sympathy, I just wanted everyone to know.
But first: kittens!
Kittens! See, don't you feel better already?
The first time I ever took a pregnancy test, it was because I really wanted a glass of prosecco.
I came off the pill in September 2012, but having been on hormonal contraception for more than eight years I was fully prepared for a long wait for my fertility to recover. In addition, I had no idea what my cycles would be like – I’d always been irregular, and didn’t expect that the various chemicals I’d been consuming to prevent babies would have helped. My first ‘natural’ period in nearly a decade came after only 25 days, but when my next one hadn’t turned up 30 days later I didn’t think much of it.
Which brings me to the test. For those counting along at home, we’re now up to November 2012. That was a crazy month for the inhabitants of Earth – Dan and I were still recuperating from the launch of Milestone Jethrik, a beast of an update to Three Rings which had involved many many late nights, but at the same time we were all rushing around preparing for Three Rings’ 10th birthday conference. We’d planned a drinks party at the end of the day as part of the celebrations, and after the countless hours of my life that Three Rings had by then consumed, I really felt like I deserved a slice of cake and a glass of fizz.
Mmm, cake
The day before the conference, my period still hadn’t come (at 34 days by that point). I decided to take a test, and then when it came up negative, I could drink my prosecco with a clear conscience.
It came up positive.
A positive pregnancy test. Yep, I peed on that! Factually speaking, this is not the pregnancy test I took before the conference - this is one I took later to reassure myself I hadn't imagined the first three. Incidentally, did you know that doctors don't do another test if you turn up and say you took a test at home?
I sat in the kitchen, staring at the thing. Then I got up and paced around a bit. Then I raised my eyes heavenward and said something like “I really hope you know what you’re doing…”. I gave myself a little excited hug and then paced some more. And then, because it was the day before the conference and I had a lot to do, I went about my day, sorting gifts for the speakers, collecting the AV equipment, and generally making sure everything was ready.
The day of the conference passed in a blur. Every time I took a loo break I found myself staring in the mirror and trying on the label ‘Mother’ for size. I didn’t get my glass of prosecco. But I did have a small piece of cake. Pregnancy diets are all well and good, but I earned that cake!
JTA and I decided not to tell people until we were past the first scan at 12 weeks. We told Dan and Matt, naturally, and a few family members, but mostly we kept a lid on it. The first trimester is a risky time and neither of us wanted to jinx things. I nicknamed the little dude Jethrik, and teased JTA by pretending I wanted the kid to have that for a middle name. Our due date was the 27th of July, Samaritans Day (24/7 – Geddit?), and for all the talk of jinxes and risks I found myself thinking a lot about what life would be like when a tiny person arrived in our lives in the summer.
I started reading up on the subject of pregnancy, and did all the things I could find to give Jethrik the best start in life. I gave up pretty much all unhealthy food, started eating loads of wholegrains and washing all my vegetables, and even changed my commute to avoid cycling along congested (and therefore fumey) roads.
Left: old route. Right: new route. Half a mile longer, approximately* 70% more park. (*Stats may shift in transit)
By the time Christmas came around, I was 10 weeks along and getting pretty excited about the whole thing. The extreme exhaustion I’d experienced in the early weeks was starting to pass off, and I still wasn’t having much trouble with nausea (except around fried bacon – the super-smelling skills I’d acquired weren’t always a blessing). It was weird spending the whole of Christmas sober (and having to try and explain my actions to all the various people we saw who weren’t in the loop), and not dipping in to the snacks with everyone else made me feel like a bit of an outsider. But my scan was only two weeks away, now – soon we could tell the world!
We got back from our annual Christmas Odyssey late on the 28th of December, and it was on that evening as we were all recuperating and gathering our energies for the Three Rings AGM on the 29th that I started to get some bleeding. It was only a little, and I knew that it probably meant nothing, so I dashed off a quick email to my midwife to let her know, and went to bed.
The AGM started just after lunch the next day, and as Managing Director I was chairing the meeting, so I put all thoughts of the bleeding which had worsened overnight out of my head and got on with the business of the day. Around 3pm I started to get cramps in my lower abdomen. I knew what that probably meant, but I also knew that there wasn’t anything that could be done to help, so I said nothing and we carried on with the meeting and the team Christmas Meal afterwards. Thanks to my voluntary first aid work with the Red Cross I knew the danger signs for haemorrhage, and I didn’t want to disrupt the meeting – it was the first time we’d got everyone together since recruiting a few very talented new volunteers, and we were getting a lot done. OK, so I’m a workaholic, but I still take a little faint pride in my professionalism at a time when I was miserable and scared.
After we got home from the meal (I don’t remember what time it was – everything’s a bit of a blur), I told Dan and JTA what was happening. I was in quite a lot of discomfort by then, and the bleeding was up to the level of a regular period. We went round to the John Radcliffe Hospital and after only quite a short wait I was seen by a doctor who took what felt like several pints of blood for tests and then sent me off to see the gynaecologist. I refused the offer of a wheelchair to get me over there, which left both the doctor and orderly confused – apparently, protocol is that patients are moved around the hospital by other people, not under their own steam – but I didn’t care. I was still clinging to the hope that there would turn out to be nothing wrong and I didn’t want to think of myself as an invalid.
The gynaecology ward was dark and eerie. It was the middle of the night by now and the residents were sleeping, so most of the lights were off while the minimal staff crept around trying not to wake anyone. We were sent into an examination room to wait for the consultant to come out of surgery, and someone stuck a pulse/oxygen monitor on my finger (mostly for something to do, as far as I could tell – my stats were fine).
When the gynaecologist arrived, he put me through a surprisingly uncomfortable exam and then told me that my cervix was still closed but in his opinion I was probably having a miscarriage rather than an ectopic pregnancy. Once I found out those were the options, I suppose I should have been glad it wasn’t worse, but mostly I just felt numb, and overcome by the unfairness of it all. I did everything the books said – why was this happening to me? It doesn’t work like that, of course, but it’s hard to be rational in the middle of the night when you’ve just found out you’re not going to be a parent after all.
I think it's time for more kittens (via Flickr)
The hospital staff felt that I was probably stable, so I was sent home in the early hours of the 30th and told to come back on the 31st when they would try and get me in for an ultrasound scan to find out what was happening with my insides. The boys took me home and I tumbled into bed for a good cry and a miserable attempt at sleep.
I have no idea what I did on the 30th. I don’t remember that day at all. Based on extrapolation from other times I’ve been sad and under the weather, I probably spent the day on the sofa watching Buffy, but I just don’t know.
On the 31st, I rose early and had a bowl of porridge. I was still sticking to my pregnancy diet, taking a kind of hopeless comfort in carrying on with the same routine I would have been following if everything was OK. In a stupid, superstitious way, I felt like if I pigged out on chocolate and then we lost Jethrik, it would be my fault.
The JR. We spent a lot of time here over the New Year. (Image from The Guardian)
JTA and I headed over to the hospital and camped out in the waiting room of the gynaecology ward. We’d been warned that they’d have to try and fit us in around all the scheduled appointments, so we were prepared for a long wait, but as it happened we were called up at 8 before the appointments started, thanks to one of the ultrasound operators arriving early.
The scan was the most difficult part of the whole thing. Sitting in the waiting room surrounded by excited couples, all I could think was that we were supposed to be like them – dammit, we were only ten days away! A short while later, we were ushered into a dark room and soon after that we were looking at Jethrik for the first time on the monitor. It was clear there was going to be no last minute reprieve. The little dude lay motionless at the very bottom of the uterus, and there was no heartbeat. It was the first time that the pregnancy had really felt real, and it was also the moment at which I realised it was really over.
Based on its length, our ultrasound operator estimated that Jethrik had stopped growing about two weeks earlier, never making it past the eight week mark. I’d had what’s called a ‘missed miscarriage’, where the embryo ceases to be viable but for whatever reason the mother’s body doesn’t notice. The operator was sympathetic, compassionate and understanding, but the simple facts were that the scrap of life which had been on its way to being our child had expired. I was handed a printout of the can results and sent back to the gynaecology ward for some more sitting around.
We made awkward conversation with another couple who were waiting for a scan. They also had a blue maternity folder, and I guess were in a similar boat. I wished the girl better news than us (and I still hope she got it). But mostly, I just read my book and waited to find out what would happen next. We spent what felt like a long time sitting in the waiting room (mostly uneventful apart from the brief period where I went to bits after a family with a young child came to visit someone on the ward), until finally a young doctor took us into the same consulting room we’d spent time in a couple of earlier. She explained the diagnosis to me – although I didn’t care much after “you’re not having a baby” – and then started talking about options. Apparently we had the choice between surgery (basically hoovering out the contents of my uterus) or waiting anything up to several months for nature to take its course. I didn’t at all like the sound of the latter!
Luckily, the JR were able to fit me in for surgery that same day to remove Jethrik’s remains. I was somewhat disturbed to find that the options for disposal of the embryo included taking it home with me (ick!) – we decided to donate it to science. Maybe our poor doomed offspring will still get to make a difference in the world, who knows?
And then it was all over bar the depressing phone calls.
It’s taken me a while to finish writing this, largely because I couldn’t come up with a nice conclusion to round it off. I’d like to be able to trot out a tidy moral; I drafted a couple of endings about getting stronger, how hopeful I am for the future and how grateful I am that we can at least try again at some point, but they sounded trite and irrelevant. The truth is, losing Jethrik sucked. I’m alright, most of the time, but it was a crappy thing that happened and I can’t pretend otherwise. Sometimes, I guess, things are just rubbish.
See also: JTA’s post on the subject
Gotcha of the day – Apache Basic Authentication on Windows
| Monday 25th February 2013 | Filled under Uncategorized |
I’ve just lost almost an entire day to this, so in case anyone else ever runs up against the same problem, I thought I’d document it here.
The issue
Trying to setup Trac on Windows (I know, I know – not my choice!), I found I couldn’t log in to my Trac project using Apache’s Basic Authentication. I started with the BitNami Trac Stack, which seemed to work fine, but my credentials simply weren’t being accepted when I followed the standard instructions for setting up Apache authentication for Trac.
The problem
As I eventually worked out, the default encyrption mechanism when generating a password file using htpasswd at the command line is crypt. And, as TFM will tell you, crypt is Unix only. After chasing my error through plugins, permissions, spelling and more, it turned out I’d written a password file that my OS couldn’t understand. Sigh.
The maddeningly simple fix
Re-generate the htpasswd file, passing the -m flag to htpasswd.
Rearranging Ticket Fields in Trac
| Wednesday 11th April 2012 | Filled under Uncategorized |
Trac will allow you to specify an order for custom fields, but there’s currently no way to change the order of inbuilt ticket fields, either out of the box or through any of the plugins on Trac Hacks.
You can still change the order, but it involves taking a copy of the ticket template source file and modifying it directly. It’s not hard, but might be a little scary for newbies. You’ll also need to be wary in case the original ticket template file is changed by the Trac team, in which case your modified copy will need updating.
Full instructions are available here.
Moving on
| Thursday 5th April 2012 | Filled under Uncategorized |
It’s over. I’m ending it. I’ve found something better.
We’ve been together in an on-again, off-again sort of way since I was a naive young student. We’ve had our good times, but recently it hasn’t been working for me. You’re not holding my interest, and I know I’m not making you happy the way I used to.
Yes, next week I’ll be leaving NAG and going to work at Oxford Brookes University. I hope we can still be friends.
Coincidentally, I’ve decided to launch a new blog. My old student one was a bit, well, studenty, and ending it after I got married felt very natural. I’m hoping to write more sober, more considered things on this one. Whether I’ll stick to that remains to be seen…
There’ll certainly be a greater number of boring opinion pieces on the technological questions which interest me (currently software development processes and DevOps). You have been warned.
Wait – I don’t know you!
That seems implausible. Don’t you have better things to do than read an unknown blog, hypothetical stranger?
But OK, here’s the synopsis.
I am a professional and voluntary software developer, living in Oxford with my husband and my boyfriend. I enjoy geek things, such as board games, well-constructed TV shows, and fantasising about having enough time to play computer games. I also like to do domestic stuff – baking, gardening, inviting people round and then overfeeding them. In my imaginary spare time, I volunteer with FESS, part of the Red Cross, and with Three Rings, a not-for-profit organisation that supplies administrative services to charities. It’s both cooler and more interesting than I just made it sound.
Recent Comments