From: Robert E. Lee (robert@dyadsecurity.com)
Date: Thu Apr 22 2004 - 14:48:22 EDT
Mike,
I agree with you. The skills are indeed separate, but it helps to have
tools that know about the latest public problems.
That being said I'm finding myself cursing at every VA tool out there.
There are four things wrong with all tools that I have played with:
1) False Positives - No <insert silly tool here>, this machine is not
vulnerable to the "bitch slap" DoS :). This part takes a long time to
ferret out. You don't want to tell people they're broken when they're
not.
2) False Negatives - Even easy things are often missed by all tools
(which is why most people end up using 2-3 of them for complete
results). You can only test what you know about... and hope that you
wrote your test in a sane universal way.
3) Too much focus on "network" problems. Not enough "outside of the
box" thinking here. Granted a lot of this work may be through manual
steps the tester makes, but there should be a good way of tracking every
significant activity/discovery the team makes.
4) Broken AI - It's better to have a security analyst make conclusions
than a piece of software that can't possibly see the big picture.
In short, using these tools for a test of 1-50 systems is useful. Once
you cross the 50-100-1000+++ system mark it really becomes information
deluge.
These tools are being sold to non-security professionals in the hopes of
helping them improve InfoSec posture. This is unfortunate because it's
missing the mark badly.
My team is writing/using a data correlation engine that allows for a
team of testers to have the computer do the tedious work that humans
make get bored with and are error prone at, while allowing for human
interaction throughout. It is all based around the OSSTMM
(www.osstmm.org). This will make a good team of testers a lot more
efficient in their time and more complete in their analysis.
This represents a goal/design shift for "VA" tools, but I think it is an
important one that other companies might follow, especially as they
realize that companies that can afford $17,000 for VA software are
likely have an internal team of testers anyway :).
Robert
-----Original Message-----
From: Mike Murray [mailto:mmurray@ncircle.com]
Sent: Wednesday, April 21, 2004 8:33 PM
To: Shawn Edwards; clarke-cummings@columbus.rr.com
Cc: pen-test@securityfocus.com
Subject: RE: Why eEye Retina (was MBSA scanner)
Let me state up front: I work for a competitor in the VA market, so I'm
going to stay far away from any discussion on products, and try to stick
with a bit of philosophy.
I had one comment on something that Shawn said:
> I know for a fact that they have some very skilled persons
> doing dev there. ... Just check some of their development
> discoveries that's gotta count for something!
While this is definitely an argument for the fact that a company has
very smart people working for it (which is definitely not in question in
eEye's case), I question the validity of the argument as far as the
evaluation of a network VA tool. If the ability to discover new
vulnerabilities were the gold standard for a good VA tool, we'd all be
buying something that Dave Aitel wrote.
While it is often given as a reason that one tool is better than
another, it simply doesn't follow that an aptitude for discovering new
vulnerabilities in code is the same as an aptitude for discovering known
vulnerabilities in running services in the real world. IMHO, the skills
are related, but significantly different.
In my mind the analogy is similar to that of the difference between
medical research and surgery. People who practice one extremely well
don't usually practice the other to the same level, even though the
skills (though not necessarily the mindsets) required to perform both
are somewhat similar in many cases. One just happens to be focused on
discovering new techniques out in the world, and the other happens to be
focused on saving lives.
My $0.02.
M
-------------------------------------------------
Michael Murray
Director of Vulnerability and Exposure Research
nCircle Network Security
Office: 416-533-5305
-------------------------------------------------
------------------------------------------------------------------------
------
Ethical Hacking at the InfoSec Institute. Mention this ad and get $545
off
any course! All of our class sizes are guaranteed to be 10 students or
less
to facilitate one-on-one interaction with one of our expert instructors.
Attend a course taught by an expert instructor with years of
in-the-field
pen testing experience in our state of the art hacking lab. Master the
skills
of an Ethical Hacker to better assess the security of your organization.
Visit us at:
http://www.infosecinstitute.com/courses/ethical_hacking_training.html
------------------------------------------------------------------------
-------
------------------------------------------------------------------------------
Ethical Hacking at the InfoSec Institute. Mention this ad and get $545 off
any course! All of our class sizes are guaranteed to be 10 students or less
to facilitate one-on-one interaction with one of our expert instructors.
Attend a course taught by an expert instructor with years of in-the-field
pen testing experience in our state of the art hacking lab. Master the skills
of an Ethical Hacker to better assess the security of your organization.
Visit us at:
http://www.infosecinstitute.com/courses/ethical_hacking_training.html
-------------------------------------------------------------------------------
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:53:52 EDT