Security Changes Everything

The last meeting of DC Spin featured two presentations. After Noopur Davis’s presentation, and a delicious buffet hosted by Booz Allen Hamilton, we heard from the grand guru of software process improvement, Watts Humphrey. There were almost 300 attendees and clearly Humphrey was the star attraction.

He began by saying he would walk through the basics and then explain how the Team Software Process and Personal Software Process address security issues.

In the early days of computing, we could assume that users were friendly, interested in results, and willing to help. The Internet used to be like that, but no more. (Technoflak recalls a client saying “The Internet was a great idea, until people got a hold of it.” ) Humphrey said the situation is serious and getting worse every year. Many successful crimes have not been publicized because of corporate embarrassment, and as Humphrey put it, “We haven’t even got to the terrorists yet.”

The way we work now is to hold our nose and ship product and fix it after defects become known. Sometimes we even charge for fixing defects (hearty laughter from the audience). But obviously the current strategy is failing. It accepts the cost of the initial attack, is impractical for system administrators, costly to suppliers, and has unknown litigation exposures.

Humphrey repeated Davis’s lesson that defective software is not secure. Over 90% of software security incidents are due to attackers exploiting known security defect types. The top ten defect types account for 75% of all vulnerabilities. Unfortunately, most software professionals do not understand that vulnerabilities are defects, do not know how to recognize and fix security defects, and do not view quality as their responsibility.

Humphrey said that security is an emergent property of the system, that security fails from the components up.

Humphrey joked that “if it didn’t have to work, we could build it pretty quick,” but went on to say that it is surprising how many defects software can have and still work.

Humphrey discussed the limitations of testing at length, joking that as long as people use it the way you test it, your software will work. He described writing a simple program with fifty-seven lines of code and coming up with one hundred and sixty-eight test cases. Obviously it is not economic to exhaustively test large and complex software programs.

Humphrey compared software development and testing to road repair. (Insert Microsoft joke here.) Suppose you were to build a road system but only repair pot holes on the most obvious routes. The roads would soon be impassable. Currently, Microsoft tests 1% of all test cases. (No, there was no derisive laughter here, possibly because the audience consisted of programmers and project managers.) Humphrey described the “universe of stress” and compared it to the much smaller “footprint of testing”. He pointed out that malicious hackers would always aim their attacks at the areas beyond the test footprint. Ideally, code should have no defects inside the test footprint.

Quality products are not built by mistake. We need development teams where all members know how to do quality work, recognize security defects, strive to find and fix every defect before testing, and consistently produce defect-free software. This requires properly trained and highly motivated development teams with supportive management.

Humphrey observed that a compiler will take whatever you give it and try to make code. In that sense, a compiler is “anti-quality.” Compilers will only find 50% of errors.

Humphrey described the best projects as an artful balance of conflicting forces. Project teams must consider business needs, technical capability, and customer desires. To build superior products, teams must understand the complete context for their projects. (One of Technoflak’s clients says that failure to understand business needs is the most common failing of developers.)

The best teams have skilled and motivated members who set aggressive but achievable (here Humphrey added deliverable) goals. Humphrey said that in his experience, management never overrules a team’s schedule, once a team has drawn up a plan. It was not clear to Technoflak whether he was speaking in general or only in cases of companies who have undergone Team Software Process training.

Humphrey described how altering the process of software development can improve quality. If A consistently produces better code than B, then have A teach B what A is doing. Not only will B’s code improve, so will A’s, because teaching his method will cause A to examine what he is doing and improve upon it. Humphrey pointed out that, “this is simply scientific method.”

Humphrey compared directing software professionals as herding cats. He said, “You must convince developers that these methods work.” As developers go through the Personal Software Process training, they gather data, and it is their data on what they did that convinces them of the worth of the method. Only three measures are used: time, size, and defects. After PSP training developers see a dramatic improvement in the reduction of defects.

Humphrey described the quality challenge as “pretty severe.” We do not have data on what part of software defects are security vulnerabilities. Presumably, if vulnerabilities were sufficiently reduced, the response strategy, fixing defects after shipment, would work. How many defects are too many? It depends on how much time a hacker has and the availability of the code: how exposed the code is, who is looking, how long they have to look, and the potential for serious damage.

We cannot acheive high quality by looking for defects, by “crawling through the code.” Simply “trying harder” will not achieve the needed quality or security levels. We need a quality management strategy that is as formal as is practical. Development teams must review and inspect all requirement designs and code. We need a strategy to record, track, and analyze every defect to fix the problem, to fix likely similar problems, to find similar problems, and to upgrade the process to prevent future problems.

Humphrey said, “People think I am against testing; I am absolutely not against testing. Testing is the only way to know what works.” He went on to discuss the component quality profile: standard design time, code review time, compile quality, unit test quality, and standard design review time. Humphrey said, “Saying the problem is requirements is tantamount to saying it is the customer’s fault.” (Technoflak was struck by this observation because one of her clients is firmly of the view that failure to get proper requirements is the most common cause of project failure. My client put the blame failure squarely on developers who do not understand business needs, fail to listen to clients, and thus are unable to develop proper requirements.)

Humphrey reminded the audience that our entire industry suffers from missed commitments, poor-quality and insecure products and unhappy users. He described security as a quality problem and said that to fix the security problem our top priority must be quality. To build quality software consistently, we must change the software-development process. We must also have skilled and motivated development teams who are dedicated to producing secure products.

Humphrey concluded by encouraging attendees to visit the TSP and PSP web sites. (Readers might also want to look at his book, Winning with Software, An Executive Strategy.)

A question period followed and the first question was about the importance of management support in the Team Software process. Humphrey said the Software Engineering Institute will not offer TSP without management training. Technoflak was deeply impressed by this requirement.

Humphrey went on to say that in the twenty-four teams they have studies for, the return on investment was 27% in the first year and the internal rate of return averaged 78%.

Another questioner wanted to know why hardware was so much more reliable than software. Watts said we could learn from hardware quality assurance and talked about the history of semiconductor manufacturing how when one plant reduced defective chips all the others had to meet the same standard of quality to stay in business.

Technoflak asked about employee turnover and its impact on quality and security. Humphrey agreed that high turnover contributes to quality problems and reducing turnover is desirable.