Thinkst Blog

Two weeks ago we ran the second edition of our internal HackWeek, and it was fantastic. Last year’s event was great fun and produced projects we still use; going into this year’s HackWeek we anticipated a leveling up, and weren’t disappointed. We figured we’d talk a little bit about the week, and discuss some of the “hacks”. Our HackWeek parameters are simple: We downtools on all but the most essential work (primarily anything customer-facing) and instead scope and build something. The project absolutely does not have to be work-related, and people can work individually or in teams. The key deadline is a 10-minute demo on the Friday afternoon. The demos are in front of the rest of the team, and results count more than intentions. Everyone participated and everyone presented at the Friday demo, including sales, dev, support, back office and yours truly. We strive to keep Thinkst a learning organisation and this HackWeek is one way that we do it. For example, it’s great to see a salesper...

Dan Geer famously declared that security is “ the absence of unmitigatable surprise ”. He said it while discussing how dependence is the root source of risk, where increasing system dependencies change the nature of surprises that emanate from composed systems.  Recently, two of our servers “surprised” us due to an unexpected dependence, and we thought this incident was worth talking about. (We also discuss how to mitigate such surprises going forward). Background : Every Canary deployment is made up of at least two pieces. Canaries (hardware, VM or Cloud) that then report in to the customer’s dedicated console hosted in EC2. We’ve gone to great lengths to make sure that the code and infrastructure we run is secure , and we ensure that any activity on these servers that isn’t expected, is raised in the form of an alert . A few weeks ago, this real-time auditing activity tripped an alert on a development server. Servers are either built as production servers, which have been tested ...

aka: You know it’s supposed to hurt, you just don’t know which kind of hurt is the good kind One of the common problems when people start lifting weights (or doing CrossFit) is that they inadvertently overdo it. Why don’t they stop when it hurts? Because everyone knows it’s supposed to hurt. Hypertrophy is the goal, so the pain is part of the deal... right? Pain, Guaranteed In an old interview on the rise of Twitter, Ev Williams said something really interesting: in pursuit of the fabled startup we’ve gotten so used to praising the entrepreneurial struggle, and so often repeat the myth of the starving entrepreneur, that people tolerate the pain of a bad/unviable idea longer than they should. He said that seeing Twitter go viral, made it clearer how Odeo hadn’t. When Twitter took off, he just about said: “So this is what traction feels like.” This is an interesting problem. The cult of entrepreneurship is strong and there’s no shortage of glib one-liners pumping people up to worship the...

Moving backward is a feature too! We go through a lot of pain to make sure that Canary deployments are quick and painless. It’s worth remembering that even though the deployment happened in minutes, a bunch of stuff has happened in the background. (Your bird created a crypto key-pair, exchanged the public key with your console, and registered itself as one of your birds). From that point on, all communication between your bird and your console is encrypted (with a per-device key) and goes out via valid DNS requests. This makes sure that deployments are quick and simple, even on complex networks. Once your bird is successfully deployed, it’s completely configurable via your Canary Console. So with a few clicks, a user is able to change a deployed Canary from a Cisco Router, to a Windows Server However mistakes happen and, as anyone who has remotely configured network interfaces over SSH can attest, remote network changes aren’t kind to missteps. How does your Canary react if yo...

As a company, we are pretty huge fans of Open Source software. We use FLOSS extensively in our production stack and we make sure to give back where we can. One of the ways we do this, is by making our Canarytokens & OpenCanary projects open source and free to download. People needing Canarytokens can use the free hosted instance we run at Canarytokens.org , or they are free to download the docker images to run on their own networks. Literally hundreds of thousands of tokens have been generated online and the docker images have been pretty widely deployed too. Our paid Canary customers get their own hosted tokens server, the ability to trivially customize it, as well as some tokens that have not been ported over to the free-server yet. The relationship between OpenCanary(left) and Canary(right) is less clear. Marco and I recently spoke at LinuxConfZa where we discussed this. In the buildup to the talk, we added some new features to OpenCanary (mostly by backporting features from ou...

One of the things that surprise new Canary customers, is that we don't try particularly hard to keep customers looking at their consoles. ( In fact, an early design goal for Canary was to make sure that our users didn't spend much time using our console at all ). We make sure that the console is pretty, and is functional but we aren't trying to become a customer's "one pane of glass". We want the Canaries deployed and then strive to get out of your way. You decide where your alerts should go (email, SMS, API, webhooks, Syslog, SIEM app), set up your birds, and then don't visit your console again until a Canary chirps.. We have hundreds of customers who never login to their consoles after the initial setup, and we're perfectly happy with this. Their alerts go to their destination of choice and that's what matters. Of these, dozens and dozens of customers rely heavily on getting their alerts piped into a Slack channel of their choice. Getting your al...

Last month we downed tools for a week as we hosted a private, on-site version of the well regarded  “ARM Exploit Laboratory” (by Saumil Shah). The class is billed as “a practical hands-on approach to exploit development on ARM based systems” and Saumil is world respected, delivering versions of the class at conferences like 44con, Recon and Blackhat for years. It.absolutely.delivered! With a quick refresher on ARM assembly and system programming on day-1, by day-2 everyone in the class was fairly comfortable writing their own shellcode on ARM. By the end of day-3 everyone was comfortable converting their payloads to ROP gadgets and by day-4 everybody had obtained reverse shells on emulated systems and actual vulnerable routers and IP-Cameras. Without any false modesty, this is due to Saumil's skill as an educator much more than anything else. Pre-Class Preparation While our Canary is used by security teams the world over, many people in the team have backgrounds in development (no...

Security vendors have a mediocre track record in keeping their own applications and infrastructure safe. As a security product company, we need to make sure that we don’t get compromised. But we also need to plan for the horrible event that a customer console is compromised, at which point the goal is to quickly detect the breach. This post talks about how we use Linux's Audit System (LAS) along with ELK (Elasticsearch, Logstash, and Kibana) to help us achieve this goal. Background Every Canary customer has multiple Canaries on their network (physical, virtual, cloud) that reports in to their console which is hosted in AWS. Consoles are single tenant, hardened instances that live in an AWS region. This architecture choice means that a single customer console being compromised, won’t translate to a compromise of other customer consoles. ( In fact, customers would not trivially even discover other customer consoles, but that's irrelevant for this post. ) Hundreds of consoles runn...