UAT vs UT: It's a knockout

For years now I've been busily telling people, at conferences and events and in training courses, what UX (User Experience) and UT (User Testing) are. Part of that also involves telling them what it isn't - and at the top of the list of things it isn't you'll normally find UAT, or User Acceptance Testing.

I thought this was pretty clear to everyone but this week I had a bit of a shock when I had a company talk to me about how they didn't need to test - since they had UAT baked into the project. This was said with a smile and a straight face.

So, to that company (you know who you are!) and to anyone else who needs to know, here are the top five reasons, in reverse order of importance, as to why you can't spell UT with an A.

5. You're leaving it till the last minute

UAT happens at the end of a project, when the product is built. If all you get is good news then that's not an issue - but if you get bad news or a mixture of both (and a mixture is the most common outcome, in my experience) then you've left it till the last minute to put anything straight. Why is that an issue? Let's ask Bob.

Bob has started his first full-time job, and he's living away from home for the first time. He's got to balance his budget for that first month.

Now what's the best approach for Bob? His mate UAT would tell him to wait until the month was up, then check his balance and his bills. Hopefully, all would be well; if not, UAT would tell him that he'd failed to pay everything, and tell him where he'd failed. Bob would then have to put it all right, somehow.

Alternatively, his lovely Aunty UX would tell him that he should first plan his budget, and then check it - probably a couple of times during the month - to make sure he was on target. 

Personally I'm with his aunt on this one. A UX process sets the approach (the budget) and then measures trajectory throughout the process. If something is wrong there's time to adjust, and Bob can eat beans for a week to pay his rent. If he waits till the end of the month he may just find himself getting evicted.

4. It takes longer to fix

Bob spent some of his hard-earned money on an awesomely complex storage unit from IKEA. Bob being Bob, he's put it together without looking at the instructions, and he's now realised that one of the first pieces he put in is in the wrong place.

If he'd have realised this back at the start it would have taken him just seconds to fix. Now that he's nearly done, he's going to have to undo a whole heap of steps, correct it, then redo it all. Instead of seconds he's now looking at another sweaty sweary half-hour. 

The same is true for UAT and User Testing. If you user test a prototype design and find problems, the fix can be seconds, minutes at worst. You're correcting your plan to build something. However if you find that same issue at UAT stage then it can take hours or days to fix. In some cases I've seen issues that take several weeks of time to correct, since entire system components rely on it. Research shows that the cost of finding a change at the end can be up to 200 times the cost of finding that same issue early on, with most of that cost relating to the time it takes to correct it.

3. It costs more to fix

Just as it can take vastly more time to fix, so does it cost more. Not only is there the potential cost of redeveloping and fixing the issue, you can also be facing penalties for late delivery, knock-on penalties or lost work you're unable to move on to, increased costs in licensing and other ancillary costs incurred during the delayed timeframe.

And just like Bob, you can also face the horror scenario of complete failure. If Bob has screwed his budget then he loses his home. If your project completely fails at UAT stage then you can lose the project, lose the customer, and even lose your company. 

2. You're making your customer pay for your bad work

There are various approaches to UAT but one of the most common is that the end customer sets up a UAT team. Internal employees are engaged to become testers, since they are the intended audience. They then have to take time to create and review scripts, and to run the actual tests. They have to report and review all the issues uncovered, and review each change as it's applied. 

And the bill for all this comes out of the customers pocket. The more mistakes your team make in building the thing, the greater the cost on your customer. Having been quite heavily involved in the other side of the fence (talking to customers who've gone through this process), that often doesn't go down too well.

1. You're aiming for average

There is a fundamental difference between the approach of UX and user testing, and the approach for UAT.

User Acceptance Testing is a bottom-up approach to measure that the interface meets a minimum requirement. It is, essentially, a measure of mediocrity. 

The UX process takes the opposite approach - it identifies everything that falls short of perfection and helps to lift products and services from mediocre into the land of awesome.

Any system that passes UAT is technically fit for purpose. It will do the job. It might not do it well, it might not be easy or fun or even pain-free, but it'll do it. But just like a pair of clogs will technically get you round the race course, there's no guarantee of great performance. And here's an example of how that can work:

So if you want to win the race, don't ever look at UAT as a replacement - the winner is UX, by a knockout.


Gary BunkerComment