Should we be more strict about analysis of reports from vulnerability scanners?

Question

The phenomenon

We regularly get questions where OP has run a vulnerability scanner on an application and asks for help with interpretation of the results. Usually they quote part of the scanner's report, and OP is uncertain about whether some particular finding is valid, what it means, or how to address it. The flagged issue then often turns out to be a false positive caused by some coarse scanning heuristics or other limitations of the tool.

A few examples:

• Remote file inclusion (RFI) found - vulnerability or false positive?
• Nexpose Vulnerability scanner
• JSON API response flagged as XSS by vulnerability scanner. Is this a false positive?
• VeraCode static code scan of django view reports "External control of Filename or Path" on render method
• Remote OS Command Injection - scanned by ZAP
• Sucuri giving false positive with their free online scanner because of an .htaccess RewriteRule (IMO well-written question, but probably still better suited for vendor support.)

Why it's a problem

Vulnerability scanning tools and services are becoming more widespread and their quality varies. Especially as some are advertised as complete security solutions, we'll probably keep seeing more questions by users who run automated scans, but lack the expertise to then assess and act on the reported results.

However, critical evaluation of the output and identification of false positives is a normal part of using a security tool. We should be able to expect some human analysis, not just an auto-generated scan report that's passed on to the community to do the "real" work. Similarly, we wouldn't accept a question that just lists open ports found via nmap or a dump of compilation warnings from gcc -Wall.

If a scanner doesn't seem to be working correctly, that's more a matter for the vendor rather than SE. The community shouldn't have to take on the role of compensating for poor performance or superfluous warnings by a subpar scanning service/software. And given that exaggerated or false alerts are a business model for some antivirus scareware already, this will probably also be a concern with vulnerability scanners in the future.

A suggested course of action

We could treat these like homework questions. That is, close questions which ask for analysis of a vulnerability scanner's report unless the asker also provides some assessment and reasoning of their own.

A question that boils down to just "Why has my scanner flagged this line of code? It seems fine to me." or "My scan report mentions this vulnerability. How do I fix it?" wouldn't be okay. In that case, we could point out that automated scans have limitations and may produce false positives – and recommend getting support from the vendor.

Bonus: We could establish a canonical question along the lines of "My vulnerability scanner gave a result that I don't understand. What now?" as a place to point to.

Happy to hear opinions!

Answer 1

I think it needs to lean more towards the "homework" model than a canonical question since we can help and sometimes the resolution is not clear-cut or Google-able.

And I would expand this concept to include log analysis, too. For homework, vuln scans, and logs, the problem questions are the same: "I have detailed artifacts that I do not understand. What do I do now?"

And for all these, a model that you suggest makes sense: the asker must be able to ask a question about their proposed response or their stated understanding. Not "tell me what to do".

But the trick here is to guide the asker to frame a better question, and that's tricky (as you have seen me struggle with that for years). People want answers to what they have asked, not to go through the work to better understand their own questions. It's easier for homework because students expect to have their questions challenged. Not so for non-students.

So, I agree with your approach. But it might require some very specific guidance that goes beyond just vuln scan questions.

Answer 2

I'm gonna give an answer counter to @schroeder's. This will be a ramble, but it'll come back, bear with me.

Programmers get a compile or runtime error they don't understand. What do they do? They type the error code or text into google, hit the first StackOverflow post that seems to be an exact match, scroll to the top answer, say to themselves "Make the class abstract? Alright I guess ... Hey! It worked!" and move on with their lives without having actually learned anything and likely made their code more brittle. I've done it. You've done it.

I worry that with security scanner results, the same behaviour will make people go "security.stackexchange says it's a false positive. Great!".

This Is Dangerous!

This will lead people to ignore real security issues in their products.

Security is subtle and tricky and highly dependant on the exact behaviour of the web app in front of you. The correct way to triage a security scan report is to pull out nmap or OWASP ZAP or whatever and try to manually reproduce the result until you're convinced that either the app needs fixing, or the scanner is wrong. If you don't have the skill to do this; then hire someone who does, throw the report back to the scanner vendor, or the app vendor, or something. Basically you're not qualified to analyze the output of this scanner; go find someone who is. Random people on the internet without access to the server you're testing cannot give you a complete answer. A detailed answer about the same error in a different web app does not carry over to yours.

---

When I first read @Arminius's question, my first thought was "If someone wants to spend time working for free, what's the harm?". No, there's definitely harm.

Rather than lumping them in with homework questions, I would lump them in with product recommendations or opinion-based questions in that the answers will not generalize beyond the exact situation in this question.

My vote is to do one of:

1. outright close the question (maybe this needs a community-specific close reason?)
2. Make it very clear on answers that this advice is unlikely to generalize to "similar situations". This will likely need community effort to put disclaimers at the top of top answers.
3. Keep to general education-type answers: "Here's what the tool thinks it found", or "Here's how you can re-test that manually", but stop short of declaring a finding Real or False Positive.