Behind the Scenes of TEC’s Vendor Showdowns

TEC’s Vendor Showdown series is the most popular regular feature that we run here at TEC. So far, we’ve run quite a few Showdowns, and usually things go smoothly, but sometimes there’s more that goes on than meets the eye—and then it’s usually problems, as you’ll see in a moment. I should know, I’m the person who writes TEC’s Vendor Showdowns.

For those who've never seen one, Showdowns compare three software solutions (it could be ERP, or CRM, or BI, or HR, or any of the over 20 different kinds of software that we cover) on the basis of pure functionality. We publish a series of charts based on the latest RFI data we collect from the three vendors, provide a bit of commentary, and let the reader see who placed first, second, or third, both overall and by main modules. (Here’s a link to the latest ERP Vendor Showdown featuring Infor, Epicor and Lawson, and you’ll see what I mean.)

Showdowns give our readers an opportunity to discover new vendor solutions, re-acquaint themselves with once-familiar ones, and come across solutions they find interesting enough to evaluate further or include in their shortlists. Showdowns give vendors the chance to expose their solutions to an interested group of people, some of whom may not even be aware that such and such a vendor offers this or that type of software solution.

It sounds pretty simple so far, but sometimes it doesn’t work out that way.

Firstly, Showdowns were never intended to be the final word on how good a piece of software is, or whether it’s better or worse than its competitors. Showdowns only provide rough, overall, rankings based on a purely quantitative calculation of how each vendor solution supports a long list of features and functions (is this or that feature or function supported straight out of the box, with a third-party add-on, in a future release, and so on.)

Showdowns also do not take into account the special needs of your organization, or anyone else’s, which is instrumental in how one determines which is the best software solution—and we take great pains to point all of this out in the accompanying text. But inevitably, some people miss the point—or fail to read the text that accompanies the charts—and send us nasty emails telling us that this or that piece of software is really the best, and how is it that we were too ignorant to realize it. Well, the results are based on information we poll from the vendors, and we can only hope that people who are inclined to send us an email missive take the time to read the accompanying text first.

Secondly, we like to give the vendors a heads up before we actually publish the results, so that they’re not taken by surprise. Well, this isn’t likely to surprise anyone, but while the ‘winning’ vendor is inevitably happy with the outcome, the ‘losing’ vendors are always a lot less than delighted. And, of course, I understand this. Vendors are highly competitive– it’s built right into their DNA, and that’s partly what’s helped make the software industry the tremendous success that it is. But as soon as the vendors get their advance look at the Showdown, the emails start to fly back and forth, with some vendors questioning the veracity of other vendors’ data, and other vendors just basically crying foul. Ouch.  The problem, of course, is that if there is a perception that if somebody ‘wins’, then there is a perception that somebody else must correspondingly ‘lose’. And the ‘losers’ are understandably never happy.

But in truth, there are no ‘winners’ or ‘losers’ in TEC Showdowns, simply solutions that are designed with differing purposes and end users in mind—no software solution fits everyone’s needs, and no software solution is perfect. Not only that, it is often only a very few percentage points that separates the leader from the second or third ranked solution.  One solution that we hope will make everyone happy is to feature only TEC certified solutions—which means that all of the vendor RFI data in the Showdowns will have been certified to be accurate by our TEC analyst team in conjunction with the vendor in question. That should help prevent vendors from being concerned that their competitor’s data in a Showdown is anything less than completely accurate.

Thirdly, we’ve had a couple of Showdowns that never even saw the light of day. I like to think of them as The Showdowns that Almost Were. One was an LMS Showdown with two vendors (it was going to the first of a new format that features two vendors instead of three). It turned out that both vendors felt their data was either slightly out of date or inaccurate or both, and in the end, the best solution was to put this Showdown in the vault. Will it ever see the light of day – I doubt it. We will be publishing an LMS Showdown soon, but I can’t imagine it will be with the same data.

The other was a Showdown that we actually started to run, but quickly had to pull down off the site. Everything went wrong on this one—we interpreted the data incorrectly, we published data that we weren’t supposed to, and we inadvertently made it possible to identify the client (this one was based on a real case study) and so on, till we managed to displease not only some of the the losing vendors (there were actually five in the running this time), but the winning vendor as well, and, if I recall correctly, the client too.

I suppose, in the end, that all of this is just the price of success. Or proof that Murphy’s Law is alive and well. But either way, keep watching—there will definitely be more Showdowns to come because you, our audience, likes them—and the vendors usually do too. And in the end, that’s what really matters.

comments powered by Disqus