Are Canaries Secure?

What a question. In an industry frequently criticised for confusing security software with secure software, and where security software is ranked poorly against other software segments, it's no surprise we periodically hear this question when talking to potential customers. We figured we'd write a quick blog post with our thoughts on it.

We absolutely love the thought of this question coming up. Far too many people have been far too trusting of security products, which is how we end up with products so insecure that FX said you'd be "better off defending your networks with Microsoft Word".

In fact, it's one of the things we actively pushed for in our 2019 talk on "the Products we Deserve":

(The Products we deserve)

So, how do we think about security when building Canary?

Most of our founding team have a long history in offense and we've worked really hard to avoid building the devices we've taken advantage of for years. From base architectural choices, to individual feature implementations, defensive thinking has been baked into Canary at multiple layers.

We're acutely aware that customers are trusting our code in their networks. We go to great lengths to ensure that a Canary does not introduce additional risk to our customers. The obvious solution here is to make it "more secure" (i.e. that it's a harder target to compromise than other hosts on the network). But that's not sufficient, a harder target is not an impossible target given enough time.

So the second part of "not introducing additional risk" is to ensure that there's nothing of value on the Canaries themselves that attackers might want.

tl;dr: Canaries should be harder to compromise than other targets and should leave an attacker no better off for compromising them.

What follows are some examples of our thinking. We've left out some bits (where prudent), but we (strongly) feel that customers should be asking vendors how they reduce their threat profile, and figure we should demonstrate it ourselves.

Implementation

All the important services on the Canary are written in memory safe languages and are then sandboxed. The Canary itself holds no secrets of importance to your network. Choosing memory safe languages has a performance tradeoff, and one we're happy to make. With that architectural decision, the only potential memory corruption bugs are in the underlying interpreter, which is well-tested (and harder to reach) at this point.

Network spanning

We also don't allow Canaries to be dual-homed or span VLANs. That's because it would violate the principle of not having anything valued by an attacker on the Canaries. Compromising a dual-homed Canary would allow an attacker to jump across networks, and we won't let this happen on our watch.

Cryptographic underpinnings

During their initial setup, Canaries create and exchange crypto keys with your console. From that point on, all communication between the Canary and your console is encrypted using these keys.

The underlying symmetric encryption library used is NaCl, which provides the Salsa20 stream cipher for encryption and Poly1305 MAC for authentication. Again, we could have chosen slightly more space-efficient cryptographic constructs, but we followed the best practice of selecting a cryptographic library which doesn't permit choices and removes all footguns.

Updates

Our birds are remotely updated to make sure they stay current, and that's a common subject of questions from potential customers. To maintain the integrity of our updates, your Canary will only accept an update that's been signed by our offline signing infrastructure. Furthermore, each update file is further signed (and encrypted) by your Console so your bird won't accept an update from another Console (even if it's a legitimate one). Lastly, the update is delivered via our custom DNS transport overlay which is also encrypted. An attacker wishing to push code to your Canary would need to compromise both your cloud Console, as well as the physical offline update-signing infrastructure.

Console monitoring

Your Console is a dedicated instance running on EC2. This simple architectural decision means that even if one customer-console was breached, there's no other customer data present. This single-tenant model also removes the risk of web-app bugs yielding data from other customers.

Aside from the usual hardening, we've taken other steps to further minimise "surprises". All syscalls across our fleet are monitored, and any server doing anything "new" quickly raises alarms. (We also make sure that the server only serves content we've expressly permitted).

By default, your Console won't hold any special data from your network. Alerts come through with information related only to a detected attack, and even though we support masking in the alert to make sure that you wont have an attacker supplied password lying in your inbox, its probably a good idea to cycle a password that an attacker has made use of. :)

(Password "masked" in email alert)

Customer-Support access

On the back-end, selected Thinkst staff need to jump through several hoops and jump-points before gaining access to your console. At every jump, they are required to MFA, and access is both logged and generates an alert. (Once more this means that such access can't happen under the radar).

(CS access to a Canary Console)

In addition to this, some customers request that no Thinkst staff access their console. These customers have the back end authentication/MFA link broken. The means that Thinkst staff will not be able to authenticate to the customer console at all.

Third-party assessments

We've also had a crystal-box assessment performed of both the Canaries and the Console by one of the leading app-sec teams in the business. A copy of their report is available on request, but their pertinent, summarising snippet is:

"The device platform and its software stack (outside of the base OS) has been designed and implemented by a team at Thinkst with a history in code product assessments and penetration testing (a worthy opponent one might argue), and this shows in the positive results from our evaluation.

Overall, Thinkst have done a good job and shown they are invested in producing not only a security product but also a secure product."

Wrapping up

So, is Canary an impossible target? Of course not, it's why we wrote "safer designs" above, not "safe designs". 

But we have put a lot of thought into making sure we don't introduce vulnerabilities to a customer network. We've put tons of effort into making sure that we limit the blast radius of any problem that does show up. And if a bird can get off just one warning before it's owned, it's totally lived up to its namesake and earned its keep...