On-screen Keyboards Considered Harmful

(aka: Shoulder Surfing: There's an App for that!)

We rarely talk about it these days, but shoulder surfing is a pretty old (but reliable) attack. This is why most password prompts are masked. Many modern mobiles (and tablets) however will highlight keys pressed on the keyboard making old style shoulder surfing attacks trivial (and reasonably automatable) again.

In an effort to (help) bring back the 90's we decided to do some fiddling and built a quick app(on top of the awesome OpenCV framework) to automate shoulder surfing against iPads.
(You can read more about it [here], download a short pdf on it [here] or just watch the youtube video below (but we think the pdf is more fun!))








There are a few more videos (available after the break)


This was an early version of shoulderPad. We assumed we had won by simply locating the blue key-presses.


One of the previewers asked: "Are you deliberately moving your fingers out of the way?"
We decided to answer by quickly typing A-E with normal cadence and normal movement.


All in all.. a fun time was had by all :>

Comments

  1. Federico13 July 2011 at 12:50

    This has been done on the native iPhone keyboard months ago, without requiring any blue key nor that the screen stands still. Both spying camera and victim's iPhone are allowed to move freely.

    Details: http://home.dei.polimi.it/fmaggi/downloads/publications/2010_maggi_volpatto_gasparini_boracchi_zanero_clearshot.pdf

    ReplyDelete
  2. mh13 July 2011 at 13:41

    Hi Federico.

    That paper looks awesome. I'll do a short follow up post and make sure to mention it. Interestingly, i dont think it negates searching for the blue key, since your attacks dont cover the password login screen (where the keys do not raise, and the discernable difference _is_ the color)

    ReplyDelete
  3. Federico13 July 2011 at 13:59

    Haron,

    thanks. However, in the native keyboards (at least on the iPhone) there are no blue keys and, even in the login/password dialogs, keys do raise in a predictable manner.

    ReplyDelete
  4. mh13 July 2011 at 14:05

    Federico.

    Ah.. Just spotted the big difference. My tests are against the iPad (with complex password) (blue - no raise), and iPhone (5.x) (blue - no raise).

    Color spotting is also interesting for the android devices, where a pattern is used for unlock (and tracks pretty well).

    Thanks again for commenting.. (and for the link)

    ReplyDelete
  5. Federico13 July 2011 at 14:21

    Haron,

    AFAIK Android also employs key magnification; also, the iPad keyboard on your video looks different (i.e., black) from the native one. Also, iOS 5 on iPhone has key magnification as well.

    -F

    ReplyDelete
  6. mh13 July 2011 at 14:34

    Hi Federico

    The android devices ive briefly examined make use of the unlock pattern.
    Not sure why the keypad looks different (nothing altered there).
    Heres a pic of my iPhone4 (iOS 5) with a button being pressed: https://img.skitch.com/20110713-et8a9ekuntuct6gk6b3uk47jyj.png

    ReplyDelete
  7. Federico13 July 2011 at 14:40

    Haron,

    ah! Now I see your point :) You target the unlock screen, whereas we target the full QWERTY keyboard. Unlock codes have just 4 digits. Is it really worth to automate the whole process?

    Cheers,
    -F

    ReplyDelete
  8. mh13 July 2011 at 14:42

    Fedrico..

    The black keyboard shows up for password unlock screens (the keyboard you are used to seeing is the regular one). (Just checked across iOS and iPad versions).
    My point there remains.. ie.. For that screen, apple is aware its a password, they are masking the text box, they can give us an option to not highlight the keys

    ReplyDelete
  9. mh13 July 2011 at 14:44

    (last 2 comments missed each other in flight) :>

    i think we have agreement now :>
    (will still go through your paper properly this weekend.. it looks great)

    /mh

    ReplyDelete
  10. Anonymous15 July 2011 at 07:37

    Android devices can have the unlock gesture or a PIN or password... Keyboards can also vary between devices as some use the standard Android one while others use Swype or other manufacturer supplied alternatives.

    You'll see more variety there, but the basic approach would be the same.

    ReplyDelete
  11. Ghostshell15 July 2011 at 15:35

    where is the download for the app!?

    ReplyDelete
  12. Anonymous8 August 2011 at 16:33

    Where is the Downloadlink? ;-)

    ReplyDelete
  13. Unknown5 October 2012 at 03:41

    Awesome to see your site. Keep up the great work. Thanks for all you could do.Lets stay in touch.

    Iphone 4 skal

    ReplyDelete
  14. ipad mini cases15 November 2012 at 11:01

    It''s quite impressive.

    ReplyDelete

Post a Comment

Popular posts from this blog

Simple Graphs with Arbor.js

Image
We recently released a tool at http://cc.thinkst.com to capture and collect infosec conference details. We commented on it [ here ]. One of the cooler components of it, is the ability to view the relationships between speakers/researchers who have collaborated. This post is a quick introduction to the library we used to build our graphs, with enough info to get you up and running in minutes. As I mentioned, we use ArborJS library which is a a graph visualization library using web workers and the popular jQuery . The API is really well documented [ here ] but like most people, I learn best by example (and there are precious few of these). Hopefully this post will fill that niche, and by the end of it you should have a basic understanding of how to use arbor.js in your project. Our Aim: We will be building a simple contrived example as seen in the image above. Project setup : Create a new html page and include script references to the following libraries (Download them from the links...
Read more

Using the Linux Audit System to detect badness

Image
Security vendors have a mediocre track record in keeping their own applications and infrastructure safe. As a security product company, we need to make sure that we don’t get compromised. But we also need to plan for the horrible event that a customer console is compromised, at which point the goal is to quickly detect the breach. This post talks about how we use Linux's Audit System (LAS) along with ELK (Elasticsearch, Logstash, and Kibana) to help us achieve this goal. Background Every Canary customer has multiple Canaries on their network (physical, virtual, cloud) that reports in to their console which is hosted in AWS. Consoles are single tenant, hardened instances that live in an AWS region. This architecture choice means that a single customer console being compromised, won’t translate to a compromise of other customer consoles. ( In fact, customers would not trivially even discover other customer consoles, but that's irrelevant for this post. ) Hundreds of consoles runn...
Read more

Good UNIX tools

Image
aka:   Small things done well  We spend a lot of time sweating the details when we build Canary. From our user flows to our dialogues, we try hard to make sure that there’s very few opportunities for users to be stuck or confused. We also never add features just because they sound cool. Do you “explode malware”? No.  Export to STYX? No.  Darknet AI IOCs? No. No. No..  Vendors add rafts of “check-list-development” features as a crutch. They hope that one more integration (or one more buzz-word) can help make the sale. This is why enterprise software looks like it does, and why it’s probably the most insecure software on your network. This also leads to a complete lack of focus. To quote industry curmudgeon (and all around smartypants) Kelly Shortridge : " it is better to whole-ass one thing than to half-ass many" . We feel this deeply. Most of us cut our teeth on UNIX and UNIX clones and cling pretty fastidiously to the original Unix philosophies ¹ : Make each...
Read more