Capability Maturity Model Appraisals

CIO has a fascinating story about Capability Maturity Model (CMM) appraisals with allegations of bribery and fraud. From the article:

As American and European companies stampede offshore to find companies to do their development work, they first need to understand what CMM ratings really mean. Yet few CIOs bother to ask crucial questions, say IT industry analysts and the service providers themselves. "Not even 10 percent of customers ask for the proof of our CMM," says V. Srinivasan, managing director and CEO of ICICI Infotech, an Indian software services provider that claims a Level 5 certification. "They inevitably take it for granted, and they don't ask for the details." ...

If this is true, why are American firms failing to press for verification of claims of CMM appraisals? Because they are naive about false claims? Or because they are interested in cheap labor and do not want to hear anything that would complicate their plans to shift work overseas?

Where CMM Comes From
The CMM was a direct response to the Air Force's frustration with its software buying process in the 1980s. The Air Force and other DoD divisions had begun farming out increasing amounts of development work and had trouble figuring out which companies to pick. Carnegie Mellon University in Pittsburgh won a bid to create an organization, the SEI, to improve the vendor vetting process. It hired Humphrey, IBM's former software development chief, to participate in this effort in 1986.

Humphrey decided immediately that the Air Force was chasing the wrong problem. "We were focused on identifying competent people, but we saw that all the projects [the Air Force] had were in trouble—it didn't matter who they had doing the work," he recalls. "So we said let's focus on improving the work rather than just the proposals." ....

This is a little off-topic from the article’s subject, but few Americans appreciate how much our government does to insure quality. In private industry if a project is over-budget and behind schedule, it is all just swept under the rug, unless the failure is spectacular, like the failed enterprise resource planning system that prevented Hershey’s from getting candy to market in time for Halloween.

The depth and wisdom of the CMM itself is unquestioned by experts on software development. If companies truly adopt it and move up the ladder of levels, they will get better at serving their customers over time, according to anecdotal evidence. But a high CMM level is not a guarantee of quality or performance—only process. It means that the company has created processes for monitoring and managing software development that companies lower on the CMM scale do not have. But it does not necessarily mean those companies are using the processes well. ...

Is it just anecdotal evidence? I would be interested in hearing from the Software Engineering Institute.

Truth in Advertising
Stories about false claims abound. Ron Radice, a longtime lead appraiser and former official with the SEI, worked with a Chicago company that was duped in 2003 by an offshore service provider that falsely claimed to have a CMM rating. "They said they were Level 4, but in fact they had never been assessed," says Radice, who declined to name the guilty provider. ....

Now that CMM has become table stakes for billions worth of business, some believe that providers should bite the bullet and get all their projects assessed if they are going to claim "enterprise Level 5 CMM."

"If I were a CIO and a company was telling me their entire company was CMM 5, I'd want all the people on my project to have gone through the assessment," says Margo Visitacion, a Forrester Research analyst and former quality assurance manager at a software development company. "[The service providers] are getting millions in business from their CMM levels. Why shouldn't they have all of their developers go through an assessment?"....

In all the meetings I have attended discussing CMM it is generally agreed that it is preferable to have one group go through the process and then show its gains to the rest of the company. It is easier to get the rest of the company on board if you can show results, so a pilot group makes sense. And I would think you would want to give them some sort of recognition, at least in the form of a successful appraisal they can show off. Perhaps on corporate web sites companies should be more specific on which part of their company has completed appraisal. There are plenty of ways to do that with maximum positive spin that are not misleading.

How Much for That Certification?
Appraisers continue to cheat too, according to their colleagues. The pressure on appraisers, in fact, is higher than ever today, especially with offshore providers competing in the outsourcing market. Frank Koch, a lead appraiser with Process Strategies Inc., another software services consultancy, says some Chinese consulting companies he dealt with promised a certain CMM level to clients and then expected him to give it to them. "We don't do work for certain [consultancies in China] because their motives are a whole lot less than wholesome," he says. "They'd say we're sure [certain clients] are a Level 2 or 3 and that's unreasonable, to say nothing of unethical. The term is called selling a rating." ...

Well what do you expect of a society who is content to roll tanks over peaceful demonstrators? I simply cannot believe the naiveté about China. Why does anyone suppose that those who run sweatshops, dump toxic waste into local rivers, are going to live up to promises of quality or respect business contracts?

More recently, the SEI toughened up the CMM itself and plans to completely replace it (as of December 2005) with a broader, more in-depth model called CMMI. In the process, it has increased the training requirements and controls on appraisers. According to Hayes, under CMMI, the SEI reviews each appraisal that comes in for irregularities. And under CMMI, appraisers have to file a report called an Appraisal Disclosure Statement that clearly states which parts of the organization and projects were assessed, as well as all the people who took part in the assessment (though assessed companies are not required to reveal that report publicly, either). The SEI, along with the lead appraiser community, is also developing a "code of ethics" for appraisers.

Sounds like a good beginning. SEI needs to respond to the issues raised by this article and I look forward to hearing their response.