In the past, small businesses and independent consultants had to rely on freely available tools to aid in their security assessments of web applications due to the cost of commercial scanners. Tools such as AppScan (IBM) and WebInspect (HP) can run into the tens of thousands of dollars which is outside the budget of most independent consultants and SMBs. Freely available tools such as WebScarab and Burp have long been critical tools in the arsenal of the small consulting companies, and will remain so I am sure. However, as useful as these tools are, they lack many of the time saving features available in most commercial tools and frequently one must use a dozen or more open source tools to even come close to the features available in one commercial tool.
Fortunately, in recent years, two companies have developed commercial webapp scanners that rival the features, the speed, the usability and the accuracy of any commercial tool on the market. And they do it at a price point that just about any small business or independent consultant can afford. and are commercial webapp scanners that won’t break the bank. Both of these tools cost approximately $3000 for a one year subscription with unlimited scanning capabilities. The focus of this article is to provide a comparison of these two tools in terms of accuracy, features, speed, and usability.
For this comparison I selected two well known vulnerable web applications: the Damn Vulnerable Web Application () and the IBM AppScan demo site called . Both of these web applications were purposely developed with vulnerabilities in order to facilitate web application testing and research. The DVWA application was installed on a VM on my network and is running CentOS 5.5, MySQL 5.0.77, PHP 5.1.6 and Apache 2.2.3. Due to the way that DVWA works, it is important to note that the security setting was purposely set to the “low” setting. Additionally, I dropped and recreated the database after each scan to ensure that previous scans did not interfere with subsequent ones. The Testfire demo site is maintained by IBM and is running Microsoft ASP.NET. These two applications provide me with two different applications to scan, each using different platforms, and each containing different types of vulnerabilities. The versions of the scanners used in this test were Acunetix 7.0 (build 20110308) and Netsparker 1.8.3.3. Lastly, when scanning with Netsparker I used the “Full Scan” scanning profile and when scanning with Acunetix I used the “Default” scanning profile (which includes all checks Acunetix supports).
DVWA Scan Results
DVWA was developed with specific vulnerabilities (such as SQL injection, XSS and others) in specific pages of the application. The DVWA application has 9 specific vulnerabilities that were tested for. The goal of the test was to see how many of these specific vulnerabilities each tool would be able to identify, how many other vulnerabilities they could detect, and how many false positives each tool reported. The tables below show the results of the scans against the DVWA application. Acunetix correctly identified 7 of the 9 known vulnerabilities while Netsparker discovered only 4 of 9. Both scanners accurately detected the command injection, file inclusion, SQL injection and file upload vulnerabilities. Both scanners failed to detect the CSRF and blind SQL injection vulnerabilities. The three vulnerabilities that Netsparker failed to detect that Acunetix did detect were the brute force attacks and the XSS bugs (reflected and stored). Netsparker does not currently check for brute force attack vulnerabilities which explains its failure to detect this vulnerability. This feature is expected to be available in a future release according to MavitunaSecurity.
Both Netsparker and Acunetix identified a number of other vulnerabilities in the DVWA application that while valid, were not specifically part of the test. The table below provides a list of these vulnerabilities. In some cases the tools found the same vulnerabilities in the same pages, but in many cases they differed quite a bit. Some of those worth noting are:
- Acunetix identified XSS vulnerabilities in cookie parameters of nearly every page. Netsparker does not currently have a check to identify this type of vulnerability, but will in a future release.
- Acunetix identified that passwords are submitted via the GET method rather than POST.
- Netsparker was able to identify file inclusion vulnerabilities in a number of pages that were missed completely by Acunetix.
- Acunetix identified that session cookies do not have the Secure flag set, even though SSL is not enabled. While this is not exactly a false positive, it certainly does not make sense to have the Secure flag enabled when SSL is not in use.
- None of the vulnerabilities detected by either tool could accurately be classified as a false positive.
Testfire Scan Results
The tables below show the results of the scans against the Testfire application. Acunetix correctly identified 9 of the 16 known vulnerabilities while Netsparker discovered 8 of 16. Both scanners correctly identified SQL injection bugs in login.aspx and transaction.aspx. They also both correctly identified XSS bugs in customize.aspx, comment.aspx, and search.aspx. Both scanners failed to detect SQL injection in account.aspx, ws.asmx and / as well as XSS in disclaimer.htm. Acunetix was able to detect an Xpath injection vulnerability, a check that is not currently included in Netsparker. Netsparker, on the other hand, detected an LFI in default.aspx that Acunetix missed.
Both Netsparker and Acunetix identified a number of other vulnerabilities in the Testfire application that while valid, were not specifically part of the test. The table below provides a list of these vulnerabilities. In some cases the tools found the same vulnerabilities in the same pages, but in many cases they differed significantly. Some of those worth noting are:
- Acunetix detected that the application is vulnerable to password guessing attacks.
- Netsparker found a bug that allows for HTTP header injections.
- Both scanners detected that ASP.NET debugging was enabled and that the ViewState was not encrypted.
- Netsparker went further and detected that MAC validation was not used in the ViewState data.
- Again, Acunetix identified that session cookies did not have the Secure flag set, even though SSL is not enabled.
- None of the vulnerabilities detected by either tool could accurately be classified as a false positive.
The Skinny on the Scanners
Both Acunetix and Netsparker performed similarly at detecting vulnerabilities in the Testfire application. Acunetix did slightly better against the DVWA application. Neither of them discovered all the vulnerabilities in the applications, but they performed in a manner that is consistent with other commercially available tools. In fact, it is arguable that these two tools, which are two of the least expensive on the market, out-perform some of the better known, higher cost solutions. Several studies have examined some of these more expensive webapp scanners and the results are rather surprising (see the links at the end of this article).
As far as Acunetix and Netsparker are concerned, they each have particular strengths and weaknesses that make them more or less appealing depending on how one intends to use the tool. Let’s start with Netsparker. The most compelling feature of Netsparker is its ability to confirm vulnerabilities, thereby saving the tester time having to confirm them manually. The creators claim that Netsparker is false positive free which may be a bit of an over-statement. However, Netsparker does have an internal exploitation engine that will attempt to verify a vulnerability by safely exploiting it. If it is able to do so, then Netsparker will list the vulnerability as “confirmed” in the report. The screen capture below shows an example of this.
This is a very powerful feature indeed and one that can save the tester time when performing an analysis of a web application. Netsparker also has a very intuitive and simple interface. This can be both a positive and a negative. For the tester who wants to quickly run a scan and get reliable results without having to spend a lot time learning how to use the tool, Netsparker is a great choice. However, I found that it did not provide the same amount of flexibility and configurability as Acunetix.
The strength of Acunetix lies in its ability to quickly detect a wide variety of vulnerabilities with little need for advanced tuning and configuration. However, for those who desire more control over the tests and like to “get their hands dirty”, Acunetix provides the flexibility and built-in tools that even the most advanced pen testers will appreciate. For example, Acunetix has a built-in SQL injection exploitation tool that allows the tester to actively engage with a backend database through a vulnerable application. The screen capture below shows an example of this tool.
As can be seen, using the SQL injection tool I was able to list the databases via a SQL injection vulnerability and even manipulate them if so desired. Acunetix also includes an HTTP sniffer, an HTTP fuzzer, an authentication tester, a port scanner, Google hacking database queries and much, much more. Acunetix also allows you to create customized scanning templates.
If you are performing white box testing or have access to the application server being tested, Acunetix Acusensor is a particularly powerful tool for detecting bugs in PHP and ASP.NET. Acusensor works by combining black box scanning techniques with feedback from sensors placed inside the source code while the code is executed on the application server. This allows Acunetix to detect more vulnerabilities with fewer false-positives, provide source code line number and stack trace information related to the vulnerability, and improves its ability to detect SQL injection bugs without having to rely on web server error messages. When a scan was run against the DVWA application with Acusensor enabled, the results were quite different than when run without it. Acunetix with Acusensor detected a total of 156 vulnerabilities as opposed to 73 without it. Moreover, Acunetix also detected the specific blind SQL injection bug in /vulnerabilities/sqli_blind which it missed without Acusensor enabled. Thus, with Acusensor enabled, Acunetix detected 8 of the 9 specifically crafted vulnerabilities in DVWA. Below are screen captures demonstrating the effectiveness of scans with and without Acusensor enabled.
DVWA Scan Results Without Acusensor Enabled DVWA Scan Results With Acusensor Enabled
Summary
The world of web application security testing is constantly evolving and thankfully we now have tools that are affordable and provide a very strong set of features to aid the security tester in her work. In this article I have performed a thorough comparison of two such web application scanners (Netsparker and Acunetix) in order to determine their relative strengths and weaknesses.
Netsparker is a very good tool for point and shoot type testing. It does not require a great deal of knowledge to use the tool, it has a very good user interface, and it does a decent job detecting the most important vulnerabilities. It has good reporting features that are easy to read and intuitively designed. Moreover, its ability to confirm detected vulnerabilities is a very nice feature that is unique to this scanner. This feature can be a real time saver as the tester does not need to validate those vulnerabilities that have been confirmed by Netsparker. The downside of Netsparker is that it does not provide a high level of control and flexibility that more experienced testers are likely to require. It also does not include many additional tools that are useful for performing more manual testing. Finally, Netsparker does not appear to have the same number and breadth of checks that Acunetix has.
Acunetix is a very robust scanner with lots of features that will allow advanced users the flexibility and control that they will desire. While it’s user interface is not bad, it is not as streamlined as Netsparker’s and the learning curve is a little steeper. However, the payoff is a very powerful tool with excellent reporting capabilities. The Acusensor technology is especially useful and greatly improves the tool’s ability to find vulnerabilities in PHP and ASP.NET applications. This is a must in any situation in which the tester has the ability to install Acusensor on the targeted application servers. It combines the best of white-box and black-box testing to provide a very detailed report on the application’s security. Both Netsparker and Acunetix did a very good job of not reporting false positives. None of the reported vulnerabilities in my tests were discovered to be false positives.
Notes
- I would like to thank MavitunaSecurity for providing me with a Netsparker evaluation license in order to perform this review. They were very responsive to all my questions and provided helpful information during my tests.
- Both and provide free (limited) versions of their tools that can be downloaded from their web sites.
- No single tool is capable of finding every vulnerability that may exist in a web application. It is useful to employ multiple automated scanners if possible as this approach will be more likely to uncover potential vulnerabilities. It is also critical to supplement automated testing with manual tests in order to find issues in business logic and authentication which are often difficult to test with automated scanning. Automated scanners are just one tool in the security tester’s arsenal and should not be relied on as the sole source of information when testing web applications.
- Below are links to previous comparisons of webapp scanners by other researchers:
