Using WolframAlpha to Hack Text CAPTCHA

I’ve been playing around with the Text CAPTCHA demo page and wondered how well WolframAlpha is at logic questions.  As it turns out, Wolfram is pretty smart!  Although, since a CAPTCHA requires an exact answer, some of the results from WolframAlpha are logically correct, but are not exactly correct.  If someone wanted to use WolframAlpha to crack the text CAPTCHA technology, they could build in filters and such to narrow down answers to what the CAPTCHA is likely looking for.

Out of 10 demo questions, 3 failed and 7 were correct (although, 4 had the correct answer but would fail a CAPTCHA if the exact answer was not parsed out).  Here are the results:

Text CAPTCHA: What is seven hundred and forty four as a number?
WolframAlpha: NumberQ[744]
Result: ALMOST

Text CAPTCHA: The 7th letter in the word “central” is?
WolframAlpha: the word
Result: FAILED

Text CAPTCHA: Which word in this sentence is all IN capitals?
WolframAlpha: capitals IN
Result: ALMOST

Text CAPTCHA: Which word contains “z” from the list: zoologist, midwifery, spiderweb, crimps?
WolframAlpha: zoologist
Result: SUCCESS!

Text CAPTCHA: The 2nd colour in purple, yellow, arm, white and blue is?
WolframAlpha: yellow
Result: SUCCESS!

Text CAPTCHA: Of the numbers seventy six, 2, 50 or forty, which is the lowest?
WolframAlpha: or
Result: FAILED

Text CAPTCHA: What is the 7th digit in 9686561?
WolframAlpha: 1
Result: SUCCESS!

Text CAPTCHA: Which of these is a colour: monkey, bank or purple?
WolframAlpha: colour purple
Result: ALMOST

Text CAPTCHA: The day of the week in chips, house, bank, mouse, trousers or Friday is?
WolframAlpha: mouse
Result: FAILED

Text CAPTCHA: If a person is called Mary, what is their name?
WolframAlpha: called Mary
Result: ALMOST

Wolfram, you’re close… keep up the good work!  Text CAPTCHA, the demo page was easy.  Are the other questions harder?

Update: There’s a discussion going on over at Hacker News, if you want to check it out!

Update 2: WolframAlpha can generate a CAPTCHA image of each of these text questions, as to make it harder for a bot to decode AND answer the question!  Check it out:  http://www.wolframalpha.com/input/?i=CAPTCHA+What+is+seven+hundred+and+forty+four+as+a+number%3F

Update 3: There is more discussion going on over at Reddit for you guys looking for more insights…. :-)

Update 4: Looks like someone put together a script that knows the format of the Text CAPTCHA questions.  It was posted on Hacker News.

Share

Tags: , , ,

18 comments

  1. FYI if you refresh the demo page you’ll get randomized questions

  2. Using WolframAlpha to Hack Text CAPTCHA // Joel’s Blog…

    Text CAPTCHA: What is seven hundred and forty four as a number? WolframAlpha: NumberQ[744] Result: ALMOST Text CAPTCHA: The 7th letter in the word “central” is? WolframAlpha: the word Result: FAILED Text CAPTCHA: Which word in this sentence is all IN c…

  3. @jbw

    True. I didn’t fiddle with the demo page long enough to notice that. If you check out the discussion on Hacker News, you’ll see what some people are saying about the accuracy of WolframAlpha tested against more questions.

    http://news.ycombinator.com/item?id=1891375

  4. You could always take it a step further and render each textCAPTCHA question as an image before serving it. Then it would be considerably more difficult (though not impossible w/ OCR technology) to just plug it into a search engine.

    Really though, CAPTCHAs and anti-CAPTCHA measures are just another kind of arms race that will never cease.

  5. Hi Evan,

    I agree that making it an image would make it harder, but that also begs the question, are we trying to make the CAPTCHA’s easier for users to decipher or harder for bots to crack?

    Text CAPTCHA wants to address both, which makes sense, but the types of questions that are asked need to remain fairly simple for users and also have absolute answers. So there is some give-and-take there I think.

    I imagine that questions is the right direction, but they need to be structured in a more abstract way. Someone on Hacker News commented that a question about the contents of an image would make it harder for bots to comprehend questions and use detection algorithms on images to figure out the contents of the image.

  6. I find interesting that, in the ALMOST cases, they’re two words answers from WAlpha.

    The spammer just needs to have this into account, and, when receiving answers with two words from WA, try to use only one of them. If it fails, save the other in a database and wait for the question to appear again.

    Spammers would love to have this success ratio with traditional captchas.

  7. @Saiyine – Exactly. Using WolframAlpha directly is not going to be accurate. But understanding the types of responses they give and parsing the correct answer out of it can be easy. Adding a couple other algorithms to the process will definitely help… for the ones that WolframAlpha comes close to answering correctly.

  8. [...] So there was some buzz about this new CAPTCHA technique that doesn’t require users to decipher ungodly distorted letters and numbers. Instead it uses logic questions to see if you are a human. It’s called TextCAPTCHA (http://joelvanhorn.com/2010/11/10/using-wolframalpha-to-hack-text-captcha/). [...]

  9. @joel

    There’s another very important criterion for CAPTCHAs: they must be easy to generate computationally. If they’re not easy to generate, then you get a small number of questions that the bot can cache.

    It seems to me that a big limitation of text CAPTCHA is the limited range of query templates. You can write parsers for all of those templates pretty easily, defeating the system. Even if you only get 20% CAPTCHA hit rates, that just reduces the number of accounts you can get per IP address in a given amount of time. But you’re still “in”m fundamentally.

  10. @Matthew

    I posted a comment on the Hacker News thread about that. Questions will come in certain sentence structures that become predictable. I think that, if text CAPTCHA’s are to be harder for a bot to guess, the subject matter has to be much more abstract. For instance, instead of asking “what is the second color in the sequence blue, apple, banana, yellow, green, orange?” (which can easily be learned and guessed by a bot) you need to ask something along the lines of “how many people will have warm hands if you hand out 3 hats, 5 pairs of gloves, and 2 scarves?”…. Or at least, maybe that gets closer?

    Maybe a list of requirements/criteria need to be established against which each text CAPTCHA question is vetted. Then again, it’s just a CAPTCHA, so does it have to work 100%?

  11. Interesting how good Wofram is at decoding the logic. I’m the author of the textcaptcha service — I think its certainly true that textual captchas will never be as strong as their image counterparts, but they do serve a purpose in a middle-ground. I would hesitate about using them in a misson-critical situation as they form a weaker defence. On the most part CAPTCHAs are used as obstacles to spam rather than hard-line defence: whether this usage is justified is debatable and I believe there is a grey area.

    Rendering the questions as an image is not very helpful — this negates the reason to use logic at all (why not a word), you need to provide an audio alternative and is not much of an obstacle as OCR is easy.

    I’ll have to have a think about the question construction, but it is a delicate line between making questions that are simply too confusing (or take too long) for everybody to actually understand and solve. I have thought about trying to grade question strength — and allowing people to specify a question strength when they ask for questions: stronger questions would be harder to break but probably more difficult for real users to understand, and the decision would depend on your audience.

  12. @Rob

    I think you are right that CAPTCHA’s are not meant as an absolute means for protecting an application, but a way to slow down spammers and whomever else. I like the idea of text CAPTCHA’s making it easier for people to decipher and from an academic standpoint, playing more with sentence structure for increased strength could be interesting. However, the effort that may go into that might not be practical for your cause? Like you said, CAPTCHA’s are meant as an obstacle, not anything more.

    180 million questions is impressive! I look forward to seeing how the methodology for creating questions evolves. Good luck!

  13. [...] neuartige Captcha-Sicherheitschecks zu umgehen. Wolfram Alpha, das ergab der schnelle Test von Blogger Joel van Horn, kann (beinahe) Captchas lösen, die Antworten auf Rätselsätze einfordern. Etwa: “Welches [...]

  14. For most of these, it wasn’t really parsing anything. For example in the question:

    Which of these is a colour: monkey, bank or purple?

    Nothing mattered other than that the last word was purple.

    Change “a colour” to “an animal” and it will still give the same exact result about the color purple.

    So far the computational knowledge engine is a failure. But, it will improve as it gets more knowledge and relationships.

  15. @David – Yeah, it does some arbitrary guess work, but could be easily improved. However WolframAlpha have implemented it, it would be interesting to see if they use something like Text CAPTCHA’s database as “practice” in tuning their algorithms….

  16. The colour one is just a coincidence, it doesn’t work at all. Try changing the question to ask for the 1st, 3rd, 4th, first, or last colour in that list; it answers “yellow” to everything. It even answers “yellow” if you ask it “The purple colour in purple, yellow, arm, white and blue is?”!

  17. [...] because they are unusual and unknown to users. Secondly, computers can still break these CAPTCHAs. Joel Vanhorn points to Wolfram Alpha as an intelligence source strong enough to crack [...]

Leave a Reply

Your email address will not be published. Required fields are marked *


7 × three =

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Additional comments powered by BackType