During a vendor presentation last year a CIO turned to me and asked “So are those benchmarks 20 or 30% over priced?”
He succinctly summarized 3 truisms about “industry benchmarks”":
vendors will use the most ‘convenient” benchmark no matter how old or how tangential to what they are pitching
analysts who develop benchmarks are often “siloed”
too many IT folks rely on a single “brand name” benchmark
While “speeds and feeds” benchmarks have been around for decades, in the last decade economic and t&c benchmarking has grown. Many outsourcing contracts allow for periodic benchmarking with related resets in pricing and SLAs.
Here are some thoughts on getting better benchmarks
a) Ask for the specific analyst or report the vendor quoted benchmark came from – and call them or read their reports to see how current and relevant the benchmark is. They may share other data points and color commentary about the benchmark.
b) Assess if the analyst or benchmarking firm is using contemporary data points. Markets are constantly in transition – a couple of years ago Hackett process benchmarks mostly reflected Western labor rates, even as significant processing was moving offshore. Today’s outsourcing benchmarks are outdated compared to emerging cloud benchmarks. ERP software benchmarks need to be adjusted for more recent SaaS benchmarks.
c) Assess if the analyst is “siloed”. Benchmarks and best practices “leak” from one market segment and from one world region to another but many analysts have a narrow market segment focus. MPLS can be benchmarked on a global basis. Software quality can be benchmarked against CMM and Six Sigma benchmarks coming out of offshore development world. Worse, some analysts suffer from Stockholm Syndrome – they lose their independence and start justifying vendor positions.
d) Ask your staff to seek out many sources of benchmarks. The worst answer often is a sole-source “I did check with Gartner” “Or TPI”. As I wrote before there are a “thousand points of influence” companies should be leveraging.
Benchmarking is a very useful part of any negotiation- and on-going vendor management. Ironically, the quest for precision is itself often an inexact art, rather than a well-defined process.
Comments
Lies, Darned Lies and Industry Cost Benchmarks
During a vendor presentation last year a CIO turned to me and asked “So are those benchmarks 20 or 30% over priced?”
He succinctly summarized 3 truisms about “industry benchmarks”":
vendors will use the most ‘convenient” benchmark no matter how old or how tangential to what they are pitching
analysts who develop benchmarks are often “siloed”
too many IT folks rely on a single “brand name” benchmark
While “speeds and feeds” benchmarks have been around for decades, in the last decade economic and t&c benchmarking has grown. Many outsourcing contracts allow for periodic benchmarking with related resets in pricing and SLAs.
Here are some thoughts on getting better benchmarks
a) Ask for the specific analyst or report the vendor quoted benchmark came from – and call them or read their reports to see how current and relevant the benchmark is. They may share other data points and color commentary about the benchmark.
b) Assess if the analyst or benchmarking firm is using contemporary data points. Markets are constantly in transition – a couple of years ago Hackett process benchmarks mostly reflected Western labor rates, even as significant processing was moving offshore. Today’s outsourcing benchmarks are outdated compared to emerging cloud benchmarks. ERP software benchmarks need to be adjusted for more recent SaaS benchmarks.
c) Assess if the analyst is “siloed”. Benchmarks and best practices “leak” from one market segment and from one world region to another but many analysts have a narrow market segment focus. MPLS can be benchmarked on a global basis. Software quality can be benchmarked against CMM and Six Sigma benchmarks coming out of offshore development world. Worse, some analysts suffer from Stockholm Syndrome – they lose their independence and start justifying vendor positions.
d) Ask your staff to seek out many sources of benchmarks. The worst answer often is a sole-source “I did check with Gartner” “Or TPI”. As I wrote before there are a “thousand points of influence” companies should be leveraging.
Benchmarking is a very useful part of any negotiation- and on-going vendor management. Ironically, the quest for precision is itself often an inexact art, rather than a well-defined process.
Lies, Darned Lies and Industry Cost Benchmarks
During a vendor presentation last year a CIO turned to me and asked “So are those benchmarks 20 or 30% over priced?”
He succinctly summarized 3 truisms about “industry benchmarks”":
While “speeds and feeds” benchmarks have been around for decades, in the last decade economic and t&c benchmarking has grown. Many outsourcing contracts allow for periodic benchmarking with related resets in pricing and SLAs.
Here are some thoughts on getting better benchmarks
a) Ask for the specific analyst or report the vendor quoted benchmark came from – and call them or read their reports to see how current and relevant the benchmark is. They may share other data points and color commentary about the benchmark.
b) Assess if the analyst or benchmarking firm is using contemporary data points. Markets are constantly in transition – a couple of years ago Hackett process benchmarks mostly reflected Western labor rates, even as significant processing was moving offshore. Today’s outsourcing benchmarks are outdated compared to emerging cloud benchmarks. ERP software benchmarks need to be adjusted for more recent SaaS benchmarks.
c) Assess if the analyst is “siloed”. Benchmarks and best practices “leak” from one market segment and from one world region to another but many analysts have a narrow market segment focus. MPLS can be benchmarked on a global basis. Software quality can be benchmarked against CMM and Six Sigma benchmarks coming out of offshore development world. Worse, some analysts suffer from Stockholm Syndrome – they lose their independence and start justifying vendor positions.
d) Ask your staff to seek out many sources of benchmarks. The worst answer often is a sole-source “I did check with Gartner” “Or TPI”. As I wrote before there are a “thousand points of influence” companies should be leveraging.
Benchmarking is a very useful part of any negotiation- and on-going vendor management. Ironically, the quest for precision is itself often an inexact art, rather than a well-defined process.
March 31, 2009 in Industry Commentary | Permalink