Traditional security tests have focused on penetration testing. However, in most software development life cycles we have a specific quality assurance (QA) phase. Moreover, very commonly now we see  unit tests and build verification tests in addition to the QA phase. Unfortunately, penetration testing is often performed too late in the life cycle - after the entire system has been built and deployed. It would be remiss for us to not leverage these earlier testing opportunities to catch as many problems as early as possible.

However, incorporating security into these testing methodologies is non-trivial. Part of the reason is that the typical security tester’s mindset is different from a quality assurance tester or developer. Both traditional software testers and security testers attempt to break down software, however, the former typically approach this problem by attempting to look for failures to meet a specific set of requirements and features. For instance, a classical software tester is concerned with whether the login feature works i.e. when they type in a valid user name and password that the specific user is logged on. Further, they will also test to make sure that if an incorrect username and password are entered the user is not logged in. A security tester on the other is not so much concerned with the login feature but with getting access to the application in an unauthorized manner. Hence, he / she are going to attempt brute forcing or SQL injection to access the account. Similarly, he / she will be paying special attention to the error messages displayed on a failed login. You can see a clear difference in the two approaches and similar examples can be drawn from all of the various features of the application. Hence, the first step in embodying security testing within the usual testing phases in a software development lifecycle is to work on developing the right mindset. This is primarily an issue of training and exposure. Using training tools such as the Hacme Series  from Foundstone, software testers can learn about the various types of vulnerabilities as well as the simple mechanisms used to test for the presence of such vulnerabilities.

The next step is to determine the level of effort of security testing that should be embodied within the development cycle. Most QA folks are used to estimating testing effort based on the number of features and software requirements. They do this by being intimately involved in the functional and design reviews. Along similar lines it is important to involve these stakeholders in the threat modeling process. This helps to not only identify security risks but can also help prioritize those risks based on business impact and thus help identify the list of test areas. Once this has been done, it is useful to simply integrate the relevant test cases into test plans that exercise those areas during a regular testing schedule.

Testing can then be split into a number of phases. In our experience, different parts of the security frame lend themselves better to some phases rather than others.

Unit testing is typically performed by the developers themselves and is often used as the exit criteria from the development phase to a formal testing phase. Testers here are looking for both coverage and pass percentages. A number of frameworks have been developed to help with this process and to essentially provide the infrastructure to make developers more efficient and effective. One such framework is provided within Visual Studio 2005. The framework can auto-generate unit test cases based on the APIs and functions exposed by your code. This is therefore an effective place to test data validation code. By resorting to fuzzing -style techniques which provide random data (“fuzz”) to the inputs of an application, exposed interfaces can be tested to observe how they react to different data elements both valid and invalid. It is possible to test here for issues such as buffer overflows, SQL injection and cross site scripting as well as other types of injection attacks. Unit testing can also be useful in testing method level authorization rules wherein the called method checks the authentication state and permissions of the caller.

In our experience the most effective manner to build such security unit test cases is to create an attack library  with commonly used attack patterns and then running through this library enumerating attacks against the API being tested. The results in turn can be compared against a known good value to determine if the attack was successful. It is often best if the unit test cases are defined within a peer development group. Thus, one developer defines the unit test cases and attack libraries for code written by a different developer.

Build verification testing is another area that could provide opportunities to test the security of an application. A number of security testing tools such as source code analyzers and web application penetration testing tools provide both the ability to integrate into build scripts as well as scripting interfaces that allow their core engines to be adapted to fit into existing testing processes. Further, these tools also often have rules engines that can be modified and extended to create custom rule sets based on risk and relevance. Organizations that have successfully deployed such tools will often define criteria for build acceptance that specify along with other more traditional QA parameters, the number and type of security bugs that maybe present in a build that is provided to the QA teams for rigorous testing.

One of the key lessons we have learned over the years having worked with QA teams is that they are very accepting of new testing techniques and methodologies. However, it is important to not cause any upheaval of the actual QA process. It is therefore vital that all security testing ultimately ties into the existing and often times tried and testing quality practices. One critical area this becomes relevant is in the bug reporting area. We have seen most success when security bugs prior to release are treated just like other software quality issues. This implies that firstly, all security issues are entered into the same bug tracking system and follow the same lifecycle. For instance, a number of organizations are used to thinking about bugs based on their severity and priority. It is therefore vital to convert the risk rating that is usually associated with security bugs into a severity and priority. These can vary based on the business risk that these bugs pose but as a general rule of thumb all high risk issues can be considered to have the highest severity and priority, medium risk to have medium severity but high priority and all low risk issues to have low severity and medium priority. It is however not atypical to see business rules that mandate all security issues be treated as highest priority. Once the security issues have been entered into the bug tracking system they should follow the same path other bugs do in being assigned to a developer, fixed, assigned back to the tester verification and being closed and potentially added to regression testing cycles. It is however recommended to tag security bugs under a separate SECURITY category in order to facilitate obtaining an aggregate view. Secondly this can also help to ensure that a security trained developer is assigned to work on these bugs rather than any developer. Doing this will improve the chances of a correct solution free from recurrent bugs. Further, given that most bug tracking systems allow for their schema to be extended it is also helpful to further classify security bugs based on the security frame. As mentioned above, performing statistical analysis across bugs or after a release can thus help identify root causes as well as techniques and tools to prevent such bugs from manifesting themselves again.

Finally, it is also helpful to classify security issues into four categories while entering them into the bug tracking system or while commenting on the fix:

·         Security Flaws: A security flaw is an issue that has come about due to an inappropriate design decision or implementation choice. They are generally architectural or design problems and have global implications. An example of a flaw would be a user authentication system that does not adequately protect passwords in the data store. Moreover this class of problems often stem from the fact that the design specification was itself incomplete or didn’t address the issue specifically. Thus, it is quite possible that a flaw might exist in an application even though the developers have implemented the design specification completely and correctly.

·         Security Bugs: A security bug is a code level problem. Often semantic in nature, they are usually the result of a coding error or bad implementation of a design decision. An example of a bug is a buffer overflow. These issues typically tend to be language specific.

·         Commendations: A commendation is a note of good security. Our experience is that this is as important to note the positive security attributes as it is the negative. Positive enforcement encourages repetition of good practices and can also help to highlight best practices that may have been implemented unbeknownst of their security value.

·         Recommendations: There are many ways of designing software. These findings list some considerations for future revisions that would facilitate enhanced or improved security. This category of issues is also used to convey any insight into better software development practices that the application developers can adopt. For instance, specific software or architecture design patterns might be recommended.

This classification like the security frame classification can provide valuable results that help measure the effectiveness and merit of performing security testing earlier rather than later in the lifecycle. They can also help reemphasize the need and value of activities such as threat modeling and peer code reviews.