Having discussed the importance of security training in the July/August 2008 issue, we wanted to discuss next how to go about creating a training program. In this article, we will discuss in detail the creation of a program that is effective in improving the security of your applications, while still being time- and cost-efficient to roll out. With these objectives in mind, we will cover the following key aspects:

• Who needs to be trained?
• How do you scale across the software development team, especially if yours is a large one?
• How can you be effective and efficient in your training efforts?
• How do you measure success and effectiveness over time?

Who Needs to Be Trained?

Essentially the answer to the question of who needs to be trained is: every stakeholder in the development process. The developers are not the only ones who play a critical role in building software; there are a number of other stakeholders involved in the process, and all too often organizations tend to ignore or entirely forget about the impact they can have on software security.

Let's start at the beginning. The business - whether this comprises the product management teams in an independent software vendor, or the business units that need the specific application being developed in an IT organization - must be trained on why software security is important in the first place. Ultimately, in some way they pay the bills and decide the schedules - they will therefore need to accept the idea that during the development process the team will engage in security activities that seemingly provide no direct value, at least in terms of functionality.

Essentially, the necessity here is to convince them of the business value to be gained by investing in this effort.

Next are the system or business analysts, the requirements specialists - the people who write software requirements. It is essential that they understand how to document security requirements as well. They need to gather these based on the different drivers for security - whether these are government regulations, industry-specific standards or company policies, and then security features themselves. (Details on how to do this were covered in the July/August 2007 issue of this magazine.)

Following are the architects, designers, and developers. This is where the bulk of the training investment will be made. With this group, the sky is really the limit, and the training needs to continuously evolve as security research itself evolves.

This group needs to be trained in secure design and architecture principles and in secure coding practices (in each of the languages with which they work) as well as in performing threat models, doing code reviews and security unit testing.

The software quality testers are a community of key stakeholders that is often ignored, especially given their role in the security-enhanced software development cycle. After all, their charter is to test software, so the purpose of training here would be to build in them the right mindset, and to give them access to the appropriate tools. (Security acceptance testing was covered in an article in the September/October 2007 issue.)

Security testers, also known as penetration testers, are highly skilled individuals who focus on destructive testing rather than the more traditional quality testing that is already present in many organizations. Because vulnerabilities and attack techniques evolve on a daily basis, it is critical for these testers to receive a near-constant stream of training and education. If this role does not currently exist in your organization, you should consider creating it by extending the duties of your existing software quality testers, or by hiring third-party specialists.

Deployment and operations engineers are the next group. Because they're responsible for installing and maintaining the application in production, software security training is critical to teach them how to configure and run it securely. This should cover everything - including the need to patch servers and third-party components, the security aspects of the SSL configuration on the Web server, the secure configuration settings in the web.config or web.xml files, the service contexts and credentials. This is especially critical given the proliferation of application development technologies such as ASP.NET and J2EE - technologies that can blur the lines between development and deployment, thus taking deployment staff into areas with which they might not be fully comfortable.

Finally, consider the users themselves. Security awareness is also critical for users in general as well since, for example, even if a user's account is compromised due to his or her mistake (perhaps through providing a password to someone claiming to be from your help desk, or to a phishing website), he or she will most likely still blame the application and the organization ("Well, no one told me that I was not supposed to give my password out! And they did say they were from your help desk!").

How Do You Scale?

One challenge any company that is looking to implement a security training program encounters is how to scale. A lot of the development groups we work with have hundreds and, more often, thousands of members - and, as we said above, scaling generally tends to be an issue primarily with developers and possibly testers; the other groups tend to be smaller or don't require such a broad training regimen.

So how do we train each and every one of those developers while still ensuring we deliver products on budget and on schedule? Do we need to train every single one? Most training classes tend to be spread throughout the duration of a week or more; consequently, development managers often ask us how they can afford to take all of their developers or testers off their work tasks for that long. Not only does it cost them money, but it also means losing a week off the schedule

In our experience, this is best dealt with by creating a tiered training program. This is done by choosing, for each team (depending, obviously, on the size of the team), a person to take the role of software security architect. (See sidebar.) He or she should become the go-to person for anything and everything related to software security.

Once you have this type of organizational structure in place within your development group, the task of providing training becomes significantly easier. The software security architect is the individual provided with elaborate training and education. This could take the form of one to two weeks of training on security and the organization's development technologies, and how the two relate to each other. The software security architect's training will need to be refreshed every six months to a year, as security research is enhanced and newer development platforms and technologies are released.

The average developer, on the other hand, can be provided with four to eight hours of software security awareness training that covers the importance of security, security principles, and common patterns and anti-patterns/mistakes that development teams make and how to avoid them. These developers don't, for example, have to attempt to become experts at cryptography or authentication protocols. Any time they need help with such things, they can tap the knowledge of their software security architect. In fact, chances are that their software security architect may have built or could recommend a reusable and extensible API for all of these complex security functions.¹

A tiered program allows you to achieve the goal of having all of your development staff trained in software security, and indeed scale across the hundreds or thousands of developers you might have, without putting your entire business on hold while the training is being disbursed. Your organization might also consider prioritizing, based on internal risk assessments, which groups to focus on first. For instance, one would imagine that if you are a bank, then your online banking application development team would be more critical to train first than developers working on less critical applications.

Another successful strategy we have seen is to provide training as part of annual developer summits, or "geek clubs," if your teams already have those. This is a good opportunity to impart training, or at least awareness, when a large collection of representatives from your development teams happen to be in the same room, even if they're present for other purposes.

How to Be Effective and Efficient

Once the decision to disseminate training has been made, there are a number of things that can be done to make the entire process both effective and efficient. In this case, effectiveness is defined as truly increasing the average knowledge level from a software security perspective, and then seeing that increased knowledge used to improve the security quality of resulting software. Efficiency is based on achieving the desired level of effectiveness at the minimal cost and time commitment.

Firstly, for the shorter-duration classes (four to eight hours), consider self-paced computer-based training (CBT) classes. With CBTs, you gain the advantage of being able to start and stop at any time, as well as incurring minimal cost per student by avoiding using a human instructor. On the other hand, you lose the hands-on experience and interactivity that a live instructor affords. CBTs therefore work well for training that does not have significant hands-on components.

Instructor-led training is much better suited for the intricacies of implementing cryptography or an authentication protocol. You might also prefer such training for small development teams comprised of highly experienced individuals who are likely to have deep technical questions about architecture and development.

Most researchers who study learning techniques suggest that training is best reinforced by hands-on lab exercises, which should be considered when deciding between CBTs and live training. This is interestingly not as much due to the hands-on element as to the fact that a good live instructor can keep the student involved throughout. That said, however, it is possible to create a hands-on training environment using tools such as the free Foundstone Hacme series² or OWASP's WebGoat.³

Another thing to consider is how to position this training. All too often we see companies mandating training in response to a compliance objective or objectives⁴ and pushing it down to the employees in a similar fashion. It is always better to help employees understand the individual value propositions for security training.

For example, development managers will appreciate that training will help them meet project deadlines, because security is built into the development lifecycle rather than crammed in at the end. Business owners will want to avoid the media attention and reputational damage that accompany public security gaffes, as demonstrated by the Web Hacking Incidents Database.⁵

Another strategy is to sell training as an investment in the employees. Security is an increasingly popular component of the skill set for developers, and thus any formal or informal training they might get adds value to their career and helps in its progression.

How to Stay Fresh

One aspect of training that is often forgotten is this: The knowledge gained can and will be forgotten, at least in part! It is therefore vital to have at least yearly refreshers, where material is not only covered again but also updated to account for security research and development platforms released in the previous year. This is especially important, since the world of software security evolves at a rapid pace.

Problems that were once considered purely reliability issues suddenly allow an attacker to exploit them remotely. For instance, NULL checks after allocating memory were once considered optional, since the worst that could happen as a result was a system crash. However, recent research⁶ has shown that this is no longer a valid assumption.

Besides the formal training methods, there is also the opportunity to impart continuous or on-the-job training through other software security activities that the team engages in, such as threat modeling and code reviews. While this is an informal method, by uncovering bugs and flaws and discussing how to fix them, the lead developers/architects are training the development teams to avoid them in the future.

The quality of the training content is a major factor in the overall effectiveness of a software security training program. If you choose to develop the content internally, take advantage of the many excellent resources that are currently available. For example, OWASP has several projects⁷ that could be used for training developers, testers, and even business owners who negotiate software contracts. In the last five years, there have been numerous books published that cover a range of relevant topics, from using static code analysis tools to integrating software security with project management techniques.

If you consider bringing in an outside vendor for training, also consider having them customize their stock classes with examples, references to policies and procedures, and contacts for people within your own environments. This is also an opportunity to customize the classes given specific risk profiles, security requirements, and compliance goals.

Further, ensure that the vendor's instructors are developers in the languages they are going to be teaching. It is also helpful if the trainer has experience working with enterprise applications similar to your own.

Another effective strategy is to ensure that security awareness training, at least, is given to developers entering the company. This means that such training must be integrated with the general package of training for new hires. In this way, before a new developer even touches any code within his or her respective development team, he or she has already gone through at least the fundamentals of software security.

How to Measure Success

As Lord Kelvin⁸ once said, "If you cannot measure it, you cannot improve it." Without fail, organizations will ask how they should measure the success or effectiveness of their software security training. We have seen a number of mechanisms that are useful for answering that question. Perhaps the easiest way to do so is by tying the measurement into the training itself.

For example, a quiz or test can be given at the end of the training, or perhaps after each module or component. Multiple-choice quizzes, however, are not always the most effective at measuring success - they are more likely to measure the individual's memory rather than his or her true understanding of the concepts.

Of course, you could take this further and concoct more detailed assessments, where students have to actually write code that corrects a software security problem in an existing code base. It is also possible to do "before and after" assessments to measure the effectiveness of the training. The "after" assessments could be immediate, or spaced out by a few months to measure longer-term knowledge retention.

At perhaps the other extreme is measuring actual improvement on the ground. This is typically a much longer assessment, and at first glance it might appear that it would cost a whole lot more beyond the cost of training. However, if you think about the assessment as being tied not just to the training but to the larger software security program as well, it might not seem as heavy. To perform such an assessment, first create a baseline of the current state of affairs.

Consider performing threat models, code reviews, and penetration tests of a subset - perhaps five to ten of your most critical applications. The results from these security activities will form the current score. Then stakeholders go through the training program you have chosen to adopt. Once the first run of training is completed - perhaps a year to 18 months down the line - run a secondary series of assessments on the next versions of the same applications when possible.

If your training and software security program was truly effective, you will see an improvement in the results of your current score, and a genuine improvement in software security as opposed to the "quick fix" that we discussed in our previous issue.

If you see a slip downward, on the other hand, obviously you have a problem and have to question why there was "negative improvement!" Similarly, even if the improvement does not meet your expectations, you can question which activities are not having the desired effect and how they might be changed or tweaked appropriately. Consider this to some extent to be part of the longer-term learning process.

Summary

In conclusion, then, when setting up a security training program, or for that matter any kind of security program, it is vital to build a program that can allow you to be effective in truly educating the stakeholders but that also still allows you to be productive as a team. The only way to achieve this is to actually measure your performance and track metrics over time. Further, training has to be reinforced time and again so that it becomes a matter of habit.

Perhaps most importantly, training is just another piece of the puzzle called software security. Hence, just as relying purely on security testing is not sufficient to improve the security quality of your applications, neither is relying on just training. In future issues, we will further investigate other parts of this holistic, security-enhanced software development lifecycle.

Rudolph Araujo serves as a principal software security consultant and trainer at Foundstone Professional Services, a division of McAfee. He is responsible for creating and delivering the threat modeling, security code review, and secure software engineering service lines, as well as for content creation and training delivery for Foundstone's Building Secure Software and Writing Secure Code - ASP.NET and C++ classes.

Roman Hustad is a principal consultant and trainer at Foundstone. His responsibilities include secure software development lifecycle design and implementation, security code review, software architecture and design reviews, and threat modeling. He instructs Foundstone's Writing Secure Code - Java and Building Secure Software courses.

1 www.owasp.org/index.php/ESAPI

2 www.foundstone.com/us/resources/proddesc/hacmebank.htm

3 www.owasp.org/index.php/Category:OWASP_WebGoat_Project

4 It is still important to understand the compliance goals, and therefore the training might need to be customized and tailored especially for certain roles, like that of business analyst, which would need this knowledge to define security requirements, for instance.

5 www.webappsec.org/projects/whid/

6 tech.slashdot.org/article.pl?sid=08/04/18/0436232&from=rss

7 www.owasp.org/index.php/Category:OWASP_Project

8 en.wikipedia.org/wiki/William_Thomson,_1st_Baron_Kelvin