During a vendor presentation last year a CIO turned to me and asked “So are those benchmarks 20 or 30% over priced?”
He succinctly summarized 3 truisms about “industry benchmarks”":
- vendors will use the most ‘convenient” benchmark no matter how old or how tangential to what they are pitching
- analysts who develop benchmarks are often “siloed”
- too many IT folks rely on a single “brand name” benchmark
While “speeds and feeds” benchmarks have been around for decades, in the last decade economic and t&c benchmarking has grown. Many outsourcing contracts allow for periodic benchmarking with related resets in pricing and SLAs.
Here are some thoughts on getting better benchmarks
a) Ask for the specific analyst or report the vendor quoted benchmark came from – and call them or read their reports to see how current and relevant the benchmark is. They may share other data points and color commentary about the benchmark.
b) Assess if the analyst or benchmarking firm is using contemporary data points. Markets are constantly in transition – a couple of years ago Hackett process benchmarks mostly reflected Western labor rates, even as significant processing was moving offshore. Today’s outsourcing benchmarks are outdated compared to emerging cloud benchmarks. ERP software benchmarks need to be adjusted for more recent SaaS benchmarks.
c) Assess if the analyst is “siloed”. Benchmarks and best practices “leak” from one market segment and from one world region to another but many analysts have a narrow market segment focus. MPLS can be benchmarked on a global basis. Software quality can be benchmarked against CMM and Six Sigma benchmarks coming out of offshore development world. Worse, some analysts suffer from Stockholm Syndrome – they lose their independence and start justifying vendor positions.
d) Ask your staff to seek out many sources of benchmarks. The worst answer often is a sole-source “I did check with Gartner” “Or TPI”. As I wrote before there are a “thousand points of influence” companies should be leveraging.
Benchmarking is a very useful part of any negotiation- and on-going vendor management. Ironically, the quest for precision is itself often an inexact art, rather than a well-defined process.
What are you reading?
I am a featured “reader” on BusinessWeek this morning for commenting on an IBM cloud story by Steve Hamm yesterday. Shirley Brady, the “Community Editor” at BusinessWeek informed me of that. In turn, I often cite (and often disagree with) Steve and other BW writers on my blogs.
On my own blog on IBM’s announcement, a couple of Gartner analysts commented. And in turn linked to their own blogs.
Tom Wailgum who writes for CIO magazine was a guest columnist on my Technology and My Hobby series.
What a difference in the four years since I started blogging. Much richer content as you get a convergence of analyst depth, journalist precision, blogger irreverence and checks and balances crowds increasingly provide. More customized to the consumer as RSS and other reader tools mature. More timely – some would say intensely so - with the spread of social tools like Twitter and Facebook.
Of course, media and analyst and blogger business models are still shaking out. Painfully so. But as this blog catalogs, that is the nature of disruptive trends.
In a garage somewhere, some kid is working on the next Twitter which synthesizes all these feeds and perspectives and counter-points – and changing its tag line
“What are you
doingreading?”March 31, 2009 in Industry Commentary | Permalink | Comments (0) | TrackBack (0)