Jun 12

Developing Apps with Environment Variables

This post is mostly a reminder to myself about my setup to deal with environment variables when working on web applications. I develop in Ruby on OS X, to give you the gist of my setup.

It wasn’t until I switched from ASP.NET and C# to Ruby that I started using Heroku. Heroku is awesome, but has its opinions of how apps should be setup. It was a little bit of a mental shift coming from Microsoft technologies, but I love it now.

Importing & Exporting Config Variables – The Heroku CLI

Heroku wants you to store all credentials and other settings in environment variables. This allows you to modify credentials and such without affecting the code. It also makes sure that the code doesn’t have sensitive information in it when checked into git.

Heroku’s command line utility lets you easily list and update environment variables on Heroku. However, it’s not very easy to get those environment variables loaded when developing on your local machine. There’s a plugin for Heroku’s command line utility that lets you export environment variables to a “.env” file. I’ve created a modified version of the plugin that lets you specify separate files for each environment (e.g. development.env, staging.env, production.env). These files can be modified and pushed back to Heroku easily.

Loading Environment Variables – Using Foreman

Now that we can manage the environment variables on Heroku, how do we use them locally? Well, that’s what Foreman is for. It was created by David Dollar, along with the original Heroku CLI plugin that I tweaked. The beautiful thing is that Foreman uses the same “.env” files and lets you specify the filename to load.

A quick step back: Foreman lets you easily start up multiple processes required to run your web app. You can specify what processes to start in a Procfile, which is the basis for Heroku web apps.

Now, you can just run a command such as “foreman start” and it will load up the Procfile in the current directory. You can also specify your environment variables to load such as “foreman start -e development.env”. Your Procfile might simply have “web: rails s” in it, so Foreman just runs that command with environment variables automatically loaded from your file.

The latest piece of the puzzle I put together was to get the environment variables loaded when I wanted to use the Rails console. Foreman has a nice little hidden command which is “foreman run”. The latest version as of a day or two ago lets you use your “.env” file. All you have to do is run “foreman run -e development.env rails c” and you’re good to go!

Simplifying Your Development Environment – Use Shortcuts!

Last but not least, I created some shell functions that really easily let me run any command with my environment variables loaded. Additionally, I wrap my command in “bundle exec” where appropriate to make sure the proper gems are loaded when running my command. Here they are:

function dev() { if [[ -f Gemfile && [email protected] != bundle* ]]; then foreman run -e development.env bundle exec [email protected] ; else foreman run -e development.env [email protected]; fi; }
function prod() { if [[ -f Gemfile && [email protected] != bundle* ]]; then foreman run -e production.env bundle exec [email protected] ; else foreman run -e production.env [email protected]; fi; }
function stage() { if [[ -f Gemfile && [email protected] != bundle* ]]; then foreman run -e staging.env bundle exec [email protected] ; else foreman run -e staging.env [email protected]; fi; }

All you have to do now is prefix any command with “dev”, “prod”, or “stage” to run the command with those environment variables. For example, I’m constantly running “dev rails c” and everything works flawlessly!

Update: Using Environment Variables in Your Gemfile on Heroku

I am hosting some private gems via git and to access them Rails needs a username and password. By now, you know where the credentials are stored. However, Heroku does not load environment variables when building the slug. To enable this, you need to install a Heroku “labs” plugin by running this command:

heroku labs:enable user_env_compile -a myapp

Hopefully all my bases are now covered…..


Mar 12

Pre-populating Shopify Checkout Forms and Other Undocumented Functionality


We’ve been using Shopify at ThankLab and chose them because they’re a hosted solution and have a simple API. However, we quickly uncovered many limitations that required us to come up with workarounds. Some of the issues we had to resolve were a result of integrating Shopify into a third-party website with existing user accounts.

As we dug into documentation on the official Shopify wiki and blog, it became clear that many of the solutions they provide are their own hacks around limitations many customers have run into. This realization was double-edged: Shopify clearly encourages hacking on their platform, but many capabilities or “features” are not really baked into their core product. In the end (frustrations aside), the challenge of making Shopify work for us became an exercise in creativity.

Seamlessly integrating third-party user accounts with Shopify checkout

Our users login to our website and checkout on Shopify. Shopify has these 3 options at checkout:

Shopify checkout options

  • Guest checkout – requires the customer to enter their email address
  • Customer login only – requires email and password to login, however each customer needs to be manually invited via email to create passwords for their account

Goal: pre-fill email address on guest checkout page (and hide email field)

Shopify limitations: Checkout page has a unique URL (is not known ahead of time), does not accept query string parameters to pre-fill form fields, and only allows you to customize CSS.

Get the Shopify checkout URL ahead of time

I kinda lied there a little bit. The checkout page CAN be determined ahead of time, but because the URL has a random looking token in it, it isn’t immediately evident. Here’s an example of my cart on the Penny Arcade store:

Shopify checkout URL

As it turns out, the ID in the URL is your shop ID (429942) and is always the same for all customers. The 32-character ID (c813f407436dc8cdda0c50d58e6d5e96) is your cart token, which is stored in a cookie called “cart”. To get the checkout URL for each customer, you’ll have to use JavaScript:

  1. Get your shop ID from your checkout page URL (simply click “checkout” in your shop)
  2. Create a JavaScript function that can read the “cart” cookie (here’s some code you can use)
  3. Create a JavaScript function that combines everything into a proper URL and do something with it

After combining these steps, you might have some JavaScript code that looks something like this:

var shop_id = "429942";
var checkout_url = "https://checkout.shopify.com/carts/" + shop_id + "/" + readCookie("cart");

We will use this URL to add some parameters to pre-fill the checkout form. Download my example script (shopify_hacks.js) from Github.

Pre-filling the Shopify checkout form

We are now able to determine the Shopify checkout URL ahead of time, but need to pre-fill the form. If you’re familiar with Ruby on Rails or web development frameworks in general, you know that you can typically take the names of form fields and add them to the URL as query string values. Those values are then pre-populated in forms that have fields matching those parameters.

In the case of the checkout form, if you view the HTML source of the page in your browser and find the “input” tags, you’ll find the field for the email address on guest checkout. The name is “order[email]“. Now, when you add that name as a query string value, the checkout URL looks like this:

Shopify checkout URL with email parameter

Unfortunately this didn’t work. The email address is still blank. However, if you submit the form with invalid information (e.g. blank email address), you’ll notice that the cart URL changes. The URL now ends in “create_order”:

Shopify checkout URL for populating form fields

 Now you can see that my email address is pre-populated in the form. You can pre-populate all form fields if you wish. In our case we are only pre-populating the email field and using CSS to hide the table row. This effectively lets our users checkout without re-entering the email address we already know.

Alternatively, use Mechanize to automate creating Shopify customer accounts with passwords

Goal: Create Shopify customer logins linked to third-party user database

Shopify limitation: Shopify customer logins can only be created by inviting a customer (manually) via email, at which point customers can create a password.

Originally wanted to create a Shopify customer account for each user in our third-party database, but Shopify doesn’t let you create customer accounts through the API. For some reason the only way to create a Shopify customer account is to manually “invite” the user via email, which sends them a link to create a password. We ended up automating this process using Mechanize:

  1. Create a Shopify customer using the API
  2. Create a Shopify administrator account that only has access to Customers
  3. Mechanize logs in to shop admin with administrator account
  4. Mechanize retrieves the invite link for a customer
  5. Mechanize visits the invite link and fills out the form to create a password

Pretty simple. However, we felt that this was less reliable than the guest checkout method. This process would also sometimes take up to 7 seconds to complete, and we decided it would add more complexity to our application to deal with this latency, as well as making sure that our users would be logged in to their Shopify accounts behind the scenes reliably (read: posting their Shopify username and password to the login form on Shopify in a hidden iframe).

Download the mechanize script (mechanize_activate_customer_login.rb) from Github. It’s a Ruby script that you can run from the command line. It accepts several arguments and outputs how long each step takes. Make sure to install the Mechanize gem before running the script.

Some other convenient undocumented Shopify functionality discovered

We actually have our own custom cart on a website separate from Shopify. So, when users click the “checkout” button, we need to “copy” the items we have in our cart to the Shopify cart. To do this, we first need to clear their existing Shopify cart and then post all items to Shopify.

Redirect after clearing Shopify cart

To clear a Shopify cart, you visit the “/cart/clear” page. After the cart is cleared, it automatically redirects you back to the empty cart page:


However, if you want to send your user to a page of your choosing, all you have to do is add that page to the URL as a “return_to” parameter:


This parameter is documented for adding products to the cart, but not any other pages. Presumably, you can add the “return_to” parameter to more pages that perform actions within Shopify.



Jan 12

Easily Transform JSON with json2json

I have been working on an auto-complete web service that searches Amazon’s Product Advertising API.  I built it in Node.js and using the APAC package made it really easy to query the API. The only thing that was extremely impractical was the JSON data returned by APAC.

Since Amazon’s API only returns data as XML, APAC uses xml2json to convert the XML to JSON. Unfortunately the resulting JSON is quite ugly. I wanted to be able to choose the data I needed and copy it to a new, clean JSON format. My solution was to create json2json.

json2json lets you create a template that describes how to transform the original JSON to a new structure. I wrote the Node.js package and example template in CoffeeScript because it has a much cleaner and simpler syntax than JavaScript. However, it is extremely simple to convert to JavaScript (click on “Try CoffeeScript”) and can easily be modified for use in a browser. Check out the (crude) documentation and example files and let me know what you think.


Dec 11

FBAPI.js: Facebook JavaScript SDK Simplified

Facebook Open Graph logo

I’ve been working on Rembly using several javascript libraries: Spine.js, Mustache.js, ICanHaz.js, and Facebook’s JavaScript SDK. These have made application development easier, but not always easy enough! I wrote previously about my enhancements to ICanHaz.js for loading Mustache templates. This time around, I wanted to use Facebook’s JavaScript SDK with less “overhead” and a simplified API.

I created FBAPI.js to handle the setup requirements that Facebook’s SDK requires, such as adding a “root” tag to the page before loading the SDK. Now FBAPI.js takes care of all the SDK requirements and lets you use the Graph API without worrying about the overhead. FBAPI.js adds helper methods for event binding and retrieving user data.  However, the best part of FBAPI.js is that you don’t have to wait for the page or javascript dependencies to be loaded before you can start using it! All methods use promises and callbacks. This lets you run your scripts in any order you want!

Check out the Github repository and let me know what you think!


Dec 11

Easily Overload JavaScript Functions with Optional Parameters


I just answered a Stack Overflow question from a couple years ago titled “Handling optional parameters in javascript” and figured I’d write about my solution here.

I’ll start by saying that the easiest way to handle optional parameters in javascript is to use an “options” object that allows a function to be called with as many or as few parameters (arguments) as you wish.

function displayOverlay(options) {
  if (options.alert) { 

However, if you need to use individual parameters, i’ve created a utility that acts as a proxy and lets you strongly type values.  It looks like this:

function displayOverlay(/*message, timeout, callback*/) {
  return proxy(arguments, String, Number, Function,
    function(message, timeout, callback) {
      /* ... your code ... */

I call my proxy arrangeArgs(). Here’s a clearer explanation of what’s going on:

function displayOverlay(/*message, timeout, callback*/) {
  //arrangeArgs is the proxy
  return arrangeArgs(
           //first pass in the original arguments
           //then pass in the type for each argument
           String,  Number,  Function,
           //lastly, pass in your function and the proxy will do the rest!
           function(message, timeout, callback) {

             //debug output of each argument to verify it's working
             console.log("message", message, "timeout", timeout, "callback", callback);

             /* ... your code ... */


I created the arrangeArgs() proxy to handle optional parameters for you.  It works nicely.  The code is in my Sysmo.js utility library on GitHub. Let me know what you think!


Dec 11

A Super-Charged Version of ICanHaz.js


I’ve been working on Rembly, which uses Spine.js as the core piece that ties all the functionality together.  I decided to use Mustache.js for my HTML templates.  And finally, I chose ICanHaz.js as a simple and lightweight way of managing my HTML templates.

Although ICanHaz.js is a great start, managing my HTML templates became unwieldy because I started having little templates everywhere.  Each part of a page that is dynamically updated needs to be broken out into its own template.  When you’ve broken a web page into small parts, it’s hard to keep track of what it looks like when put back together.  It also becomes hard to create the correct CSS styles when you lose track of the HTML hierarchy.

This lead me to enhance ICanHaz.js with a ton of new features.  The primary one being nested templates, which allowed me to keep my full HTML page template in tact, while designating specific HTML tags as “sub templates” or partials. You can also specify additional templates to load and replace script “include” tags with the loaded HTML.

Check out my fork on Github for more information about how to use my enhanced version if ICanHaz.js. Make sure to look at the javascript comments for details. And let me know what you think.


Jun 11

Announcing the ThreatSim Beta

I’ve been working with Stratum Security for the past couple of months on ThreatSim (@ThreatSim), which we are happy to announce to the world today!  ThreatSim is a web-based phishing attack simulator to help companies assess how vulnerable their network and internal assets may be to phishing attacks.  Not only does ThreatSim track who is clicking on phishing emails, but we’re also making an exfiltration agent available, which simulates transmitting sensitive data from the local network out to the internet.

Check out the website at www.ThreatSim.com, follow us at @ThreatSim, and check out the conversation on Hacker News.


Nov 10

Using WolframAlpha to Hack Text CAPTCHA

I’ve been playing around with the Text CAPTCHA demo page and wondered how well WolframAlpha is at logic questions.  As it turns out, Wolfram is pretty smart!  Although, since a CAPTCHA requires an exact answer, some of the results from WolframAlpha are logically correct, but are not exactly correct.  If someone wanted to use WolframAlpha to crack the text CAPTCHA technology, they could build in filters and such to narrow down answers to what the CAPTCHA is likely looking for.

Out of 10 demo questions, 3 failed and 7 were correct (although, 4 had the correct answer but would fail a CAPTCHA if the exact answer was not parsed out).  Here are the results:

Text CAPTCHA: What is seven hundred and forty four as a number?
WolframAlpha: NumberQ[744]
Result: ALMOST

Text CAPTCHA: The 7th letter in the word “central” is?
WolframAlpha: the word
Result: FAILED

Text CAPTCHA: Which word in this sentence is all IN capitals?
WolframAlpha: capitals IN
Result: ALMOST

Text CAPTCHA: Which word contains “z” from the list: zoologist, midwifery, spiderweb, crimps?
WolframAlpha: zoologist
Result: SUCCESS!

Text CAPTCHA: The 2nd colour in purple, yellow, arm, white and blue is?
WolframAlpha: yellow
Result: SUCCESS!

Text CAPTCHA: Of the numbers seventy six, 2, 50 or forty, which is the lowest?
WolframAlpha: or
Result: FAILED

Text CAPTCHA: What is the 7th digit in 9686561?
WolframAlpha: 1
Result: SUCCESS!

Text CAPTCHA: Which of these is a colour: monkey, bank or purple?
WolframAlpha: colour purple
Result: ALMOST

Text CAPTCHA: The day of the week in chips, house, bank, mouse, trousers or Friday is?
WolframAlpha: mouse
Result: FAILED

Text CAPTCHA: If a person is called Mary, what is their name?
WolframAlpha: called Mary
Result: ALMOST

Wolfram, you’re close… keep up the good work!  Text CAPTCHA, the demo page was easy.  Are the other questions harder?

Update: There’s a discussion going on over at Hacker News, if you want to check it out!

Update 2: WolframAlpha can generate a CAPTCHA image of each of these text questions, as to make it harder for a bot to decode AND answer the question!  Check it out:  http://www.wolframalpha.com/input/?i=CAPTCHA+What+is+seven+hundred+and+forty+four+as+a+number%3F

Update 3: There is more discussion going on over at Reddit for you guys looking for more insights…. :-)

Update 4: Looks like someone put together a script that knows the format of the Text CAPTCHA questions.  It was posted on Hacker News.


Nov 10

I Know Everything About You(r Garbage)

Working as a security consultant, I’ve come to be much more paranoid about my privacy over the years.  One thing that I do is shred anything that has my name on it.  I don’t need anyone rummaging through my garbage to write my biography or cash in on my identity!

I could go on, but why?  Read about someone else’s trash….


Nov 10

Change the Way You Sleep, the Way You Work, and the Way You Type

I definitely need more discipline in my life, and part of that is getting out of my home office into some real, collaborative office space.  Then I might start working on a routine that takes me more than a couple feet from my bed.

But check this out…. This guy talks about 3 different improvements he’s made to his daily routine…. I imagine attempting these might be a good exercise, but I don’t think I’m there yet.  But I am impressed.  And I should probably try it one of these days.