Conventional Testing Challenges

Source
http://www.securityinnovation.com/holodeck/limitations.shtml

The problem:    Tens of thousands of stability, reliability and security bugs make it past current QA practices and into shipped software.  Although the testing tools available are "working", they are limited in their ability to test exceptional conditions, where the majority of functional and security bugs reside.

The Solution:  Holodeck was designed to fill the gap between current testing methods and what's needed to catch these bugs before they are shipped

ELUSIVE BUGS IN ERROR-HANDLING CODE

As a tester, you are tasked with determining whether your application functions correctly according to its specifications, which is generally under normal "production" conditions.  But do you know how it would behave with exceptional conditions such as low memory, corrupt inputs, or damaged load files?  Most testers can't answer this question because these environments are too difficult and costly to create and maintain, and it is nearly impossible to execute all statements and branches of code in an application.  As a result, there is a lot of untested code that gets released . This code (often error handling code) goes uncovered by conventional testing methods and tends to be riddled with serious reliability, functionality, and security defects.  Holodeck solves this problem via point-and-click environment simulation - making it the ideal automated functional and reliability testing tool for testers and GUI developers who need superior control for testing

IMPOSSIBLE TO PLAN FOR ALL SCENARIOS

A few years ago, we analyzed defects in hundreds of shipping products and found that the most of these had no chance of being caught. Why? Because the situations where the bugs surface were never considered as part of the test plan, which typically considers for testing under normal conditions.  At the conclusion of our study we decided to develop a tool that would have caught these bugs prior to release. Given the anemic toolkit available to testers, a revolutionary approach was needed - one that allowed for testing of the hidden or hard-to-reach branches of application logic and error-handling.  One that allowed you to create environments where applications misbehave and force applications into error conditions?  One like Holodeck.   :: more>> on How Holodeck Works

LIMITS OF TRADITIONAL TESTING

Hard-to-create environments like "out of memory", "invalid input/reply", and "corrupt registry entries" make applications behave poorly and existing automated tools can't force these condition - they simply test your application in "normal" environment and ignore the boundary conditions where many defects live.  These tools are typically  expensive and good for only one kind of testing: reliability, functionality, load/stress, etc.  Many of them require that you have access to the source code or work in "clean" environments. Further, when you test your application with conventional automated tools, there is a drain on the entire system because they are limited to testing the application _as part of_ an existing/overall system. With Holodeck. you never need access to source code, never need to have special builds, and you can test your apps in live production environments because it only affects the application under test – not the rest of the system. When you use it to deprive your application of memory, the rest of the system stays in tact -  it only 'fools' the application, not your whole system.

Finally, the problem of "cannot reproduce" is still prevalent with most automated test tools, but not the case with Holodeck. Because Holodeck intercepts all your application's system and API calls it gives you the power to examine low-level interactions and network packets - pinpointing and easily recreating bug-generating events with ease. Whenever there is an application crash, Holodeck creates a "minidump"  and provides the exact line of code and machine state when the crash occurred.