Penetration Testing considered harmful today


Early last year we presented at 44con with a talk titled: "Penetration Testing considered harmful today".

44con have just released the video so we figured it was worth a quick recap (for anyone not willing to tolerate the whiny voice!)

The original slides (in PDF) are available (here)

The central thesis of the talk is that penetration testing has established itself as a necessary activity for securing a network and is now pushed forward by a multi million dollar industry despite the clear signs that it is not helping all that much. (Read the annotated slides here)

Watch the video here:


Comments

  1. Andre Gironda17 March 2012 at 18:18

    The is a very great talk; a necessary talk; a talk that everyone in this field should hear.

    I find the conclusions in the talk to be unsatisfying (from my perspective at HP, which is where I work). I'm not going to say that more penetration-testing is necessary or even that you need to buy anyone's products per se, though. I'm not here to shill, but instead to attempt to say profound things.

    I think there is a lot more to be said about appsec risk management and penetration-testing-focused/oriented risk management, which is almost the point that you get to in your talk. In your view, it's a balance between a pen-test "Visio" and a playing-card game. Or worse, PTES. I'm not ready to jump to these conclusions without evidence, although I agree that it is likely more sound that what we already do in the pen-testing industry.

    My view of the "new" pen-testing industry is one of simplicity: "Trust, but Verify". The best way to drive costs down in the verification process is to:

    1) Standardize on 'secure-by-default' application, data, and infrastructure components
    2) Penalize cutting/bleeding-edge and legacy (i.e. non-'secure-by-default') components by compartmentalizing/segregating them to a nearly non-functional state
    3) Reduce the number of components to the smallest possible amount

    Verification costs a lot of money, but it must be done. Another avenue of verification cost reduction is proper maintenance of coordinated disclosure through 'outside-bug-reporter-friendly' incident response teams (at all layers of arch: app and data, too!) and bug bounty programs.

    "Knowing is half the battle" -- GI Joe wisdom that should be part of any information security and risk management program. Add the necessary intelligence. Add total intelligence, and especially add TECHINT (or technical intelligence, aka the study of one's adversaries' weapons and resources).

    Proper risk communication from subject-matter-experts to decision-makers should evolve modern information security and risk management programs to evidence-based conclusions. When it doesn't, then there is probably an issue with the core business decision-making process, which is often a greater risk to information assets than either one 0day or even a thousand.

    ReplyDelete
  2. mh18 March 2012 at 04:36

    Hi Andre,

    > The is a very great talk; a necessary talk; a talk that everyone in this field should hear.

    Thanks a lot, thats awesomely flattering..

    I agree re: conclusions.. In truth the main thrust of the talk is to point out that we need a reset, and to try to justify why i think we do.
    I offer some (possible) solutions, but im fundamentally saying: "we need more ppl thinking about this"

    "Trust, but verify" is gold and i think "TECHINT" will start making its way more and more into mature corps. All in all.. i think there are important conversations to have, and more deep thinking to be done..

    ReplyDelete
  3. Jeff Pettorino19 March 2012 at 10:50

    Great talk. Very thought provoking points. Not sure I agree with ALL of them, but many are absolutely valid. Thanks for raising the bar in our profession by promoting this dialogue.

    ReplyDelete
    Replies
    1. mh19 March 2012 at 14:10

      Hi Jeff.

      Thanks. I'm pleased that it didn't take too much of a beating, because I genuinely think the dialogue is necessary.

      Thanks for the kind words.. :>

      Delete
  4. Daniel Roberts19 March 2012 at 20:21

    Hi

    Very thought provoking presentation. I have been going back and forth with whether I agree or disagree with the article! Your presentation brings to the forefront a lot of problems (which needed to be done) but doesn't necessarily provide any great solution.

    The problem is of zero day's and your explanation and description really hits home the problem of just doing a standard pen test. However, information security is all about risk appetite and other great jazzy terms like risk tolerance and risk avoidance and risk acceptance. would it be too crazy to say that a lot of firms just accept the risk of 0-days and realise that if a hacker wanted to get to the crown jewels by either buying a 0-day or finding one themselves then they have accepted that risk.

    I guess one point I agree with you is that what use is the pentest nowdays? It probably is more a feel good factor for IT security managers, and at best a defense at script kiddies. Is that a great situation for IT security to be in, probably not.

    Another interesting point you bring up is browser vulnerabilities - these are definitely an issue, but I guess their place is not necessarily the pen test report. I guess it comes down to threat modelling again, and determining what, where are your major threats, and if one is a weakness in a browser then that needs to be classed as a threat and treated like any other threat.

    All in all, interesting article with a lot to think about.

    Good work.

    DR

    ReplyDelete
    Replies
    1. mh21 March 2012 at 11:48

      Thanks DR. I think if it get us all thinking about it again, then it has served its purpose.

      Delete
  5. Anonymous19 March 2012 at 22:26

    One thing that is left out of the talk is the business 0-day issues that pentest simply does not have time to address. Any sufficiently curious internal user who has put time into learning internal environment can and will be superior in results to majority of pentest teams.

    ReplyDelete
  6. Ian Tibble21 March 2012 at 07:51

    yes. Not sure who was saying "mission accomplished". Not me. There were plenty who were saying that there is no need for tech analysis of risk any more because it suited their agenda. This is the real reason for the mess we're in today. Most issues can be traced back to this.

    re: boards. Growing numbers of them suspect problems because they see the lack of confidence in their CISO. They don't know what to do about it though. None of them have really been well advised to be honest.

    I agree pen testing, even under ideal conditions, should feature quite low in the whole strategy, but for different reasons. Not that i disagree with the reasons given, just I see it in a slightly different way.

    Companies can use zero days in testing but do it in a balanced way...e.g. first zero day compromises the listening service which doesn't give us root. then the tester may have a zero day for a setuid root binary..so, easy, game over. However, they could go about privilege escalation by other means so returning more value for the client paying the bill.

    I cover some of these issues in Chapter 7 of my book, Security De-engineering.

    good stuff re: extrusion.

    Overall, good presentation - refreshing aspects on pen testing.

    ReplyDelete
  7. Daniel Roberts22 March 2012 at 12:39

    Just read this, which i think is your point exactly!
    http://www.forbes.com/sites/andygreenberg/2012/03/21/meet-the-hackers-who-sell-spies-the-tools-to-crack-your-pc-and-get-paid-six-figure-fees/

    ReplyDelete
    Replies
    1. mh23 March 2012 at 09:59

      The timing was perfect! (thanks for the link)

      Delete

Post a Comment

Popular posts from this blog

Simple Graphs with Arbor.js

Image
We recently released a tool at http://cc.thinkst.com to capture and collect infosec conference details. We commented on it [ here ]. One of the cooler components of it, is the ability to view the relationships between speakers/researchers who have collaborated. This post is a quick introduction to the library we used to build our graphs, with enough info to get you up and running in minutes. As I mentioned, we use ArborJS library which is a a graph visualization library using web workers and the popular jQuery . The API is really well documented [ here ] but like most people, I learn best by example (and there are precious few of these). Hopefully this post will fill that niche, and by the end of it you should have a basic understanding of how to use arbor.js in your project. Our Aim: We will be building a simple contrived example as seen in the image above. Project setup : Create a new html page and include script references to the following libraries (Download them from the links...
Read more

Small things done well¹

Image
Bad design is bad  In 2015 Moxie Marlinspike pointed out that the manual page for GPG is (now) 50% of the novel Fahrenheit 451. Any software whose man page approaches 20 thousand words better have a good excuse, and GPG can only gesture vaguely at decades of questionable design. GPG gets a bad rap but it isn’t really much of an outlier. Security software has a long history of crumby, unintuitive interfaces and terrible design choices. A deep dive into the factors behind awfully designed security software isn’t the purpose of today’s blogpost, but suffice it to say there is seldom pressure from the end users. Security software mandated by a security team is often rammed down users’ throats, so it doesn’t bother being pleasant. It’ll sell anyway. We’ve worked hard to buck this trend from our first version. It’s one reason why we are one of the few pieces of security software that customers actually talk about in terms of love: https://canary.tools/love Recently, we released a major...
Read more

Using the Linux Audit System to detect badness

Image
Security vendors have a mediocre track record in keeping their own applications and infrastructure safe. As a security product company, we need to make sure that we don’t get compromised. But we also need to plan for the horrible event that a customer console is compromised, at which point the goal is to quickly detect the breach. This post talks about how we use Linux's Audit System (LAS) along with ELK (Elasticsearch, Logstash, and Kibana) to help us achieve this goal. Background Every Canary customer has multiple Canaries on their network (physical, virtual, cloud) that reports in to their console which is hosted in AWS. Consoles are single tenant, hardened instances that live in an AWS region. This architecture choice means that a single customer console being compromised, won’t translate to a compromise of other customer consoles. ( In fact, customers would not trivially even discover other customer consoles, but that's irrelevant for this post. ) Hundreds of consoles runn...
Read more