alot to agree with here, and a bit to debate.

First, I fully concur that a key challenge for marketers and analysts should be to identify and agree upon objectives for an optimization effort. This is the first nut that should be cracked. Without these key objectives (goals) in place, it’s just random 1-off’s with no real direction, and no momentum will be achieved.

I also agree completely that knowing what to test (e.g. which areas of the site, which creative elements, etc.) will save you alot of time and ultimately lead to superior results vs. pure trial-and-error. Drawing on historical web analytics data, voice-of-customer, usability feedback / reviews, and a dose of best practices will help with all of this. And of course you don’t get anything for free, so actual time & effort must be spent to move the needle in a meaningful way.

Where I’d offer some constructive feedback is in the ‘get over the tools’ message. (Full disclosure: as you know, but your readers may not, I’m the CEO of SiteSpect, a provider of multivariate testing & behavioral targeting tools and services.) I think it’s fair to say that each of the numerous tools out there each have strengths and weaknesses. Some tools are more suited to certain types of tests, certain types of sites, certain types of workflow, integration/connectivity with other tools, and so on. It certainly depends on what your optimization goals are (your first point), but selection of the “right” tool (your second point) then enables thorough, well-informed testing to take place (your third point.)

Ultimately, I think I’d paraphrase “get over the tools” as “work smartly to select the right tools so you can move ahead with the important and ongoing work of optimization.” Cheating the tool selection process is likely to lead to aggravation and/or additional time/effort spent (wasted) re-selecting and re-implementing the right tool. Of course, humans often learn best through the process of making mistakes, plus it isn’t always obvious which tool is best. So research is key – sites like whichmvt.com are certainly helpful – and buyers should always speak with other users, check references, and so on.

Thanks for the informative post!

cheers,
Eric

Let me go back in time. I was a competitive gymnast growing up. And, I spent hours watching videos of my routines, or listening to coaches, telling me to point my toes, or straighten my legs, or do whatever it takes to stick the landing. But, once all the mechanics were fixed and the execution was greatly improved, I didn’t score a 10. (In fact, I never scored a 10). But I did raise my scores significantly. The more experienced I got, the more I realized that all the collective pieces needed to work together, harmoniously. It wasn’t enough to do everything well in isolation. It was more important to make sure everything flowed and there was constant improvement. Did the video camera provide that? No, did the coaching provide that? No. It was only by a steadfast determination and passion to excel did the scores increase. So, to your point, it’s not the tool. It’s a steadfast commitment and passion for excellence that can move the needle in the right direction.

Next he chops off one of the legs and repeats the experiment. “Jump frog jump!” he says. The frog manages to jump 1.5 feet. So he writes in his lab book: ‘Frog with 3 legs – jumps 1.5 feet’.

He chops off another and the frog only jumps 1 foot. He writes in his book: ‘Frog with 2 legs jumps 1 foot’.

He continues and removes yet another leg. ” Jump frog jump!” and the frog somehow jumps a half of a foot. So he writes in his lab book again: ‘Frog with one leg – jumps 0.5 feet’.

Finally he chops off the last leg. He puts the frog on the line and tells it to jump. “Jump frog, jump!”. The frog doesn’t move. “Jump frog, jump!!!”. Again the frog stays on the line. “Come on frog, jump!”. But to no avail.

The biologist finally writes in his book: ‘Frog with no legs – goes deaf’

The point is test all you want but be careful you aren’t making erroneous conclusions from the data!!

All to often I find web analytic’s folks working off of too small sample sets and trying to fit their preconceived notions of what the answer should be versus what it actually is.