Building a modern web app, some learnings

I recently built a fairly rich web application from the ground up. Whilst I’ve being building web apps since the mid-nineties this little project had a different flavour to it and I thought I’d go over my learnings here.

Firstly why was this project different:

  • its a Facebook app and uses their JavaScript SDK. This means that 99% of the app runs client side with only a small bit of server side code.
  • I’m was the sole developer. Normally I work with 2 or more other developers, focus on the back-end work and leave the bulk of front-end stuff to others
  • rapid development approach. The app was built quickly with me working closely with a UX guy and a designer. Whilst the functionality was locked down early on, the user interface and interactions evolved rapidly and iteratively.

First observation: its so easy

OK that is slightly facetious, but modern CSS and JavaScript frameworks mean that you don’t have to be guru to build nice looking, smooth functioning, cross browser applications.

For this application I used the Twitter Bootstrap CSS framework and the jQuery Javascript framework.

Get a designer

Yes I know I just said that it’s easy and you don’t need to be a guru to create nice looking apps BUT whilst Twitter Bootstrap encapsulates a great design, you are still going to want to customise it.

Designers have the knack of adding the odd image, drop shadow or bit of whitespace that can make a perfectly ok looking web page just suddenly pop. Not to mention the selection of color a palettes and general pixel pushing.

On a similar front having a UX guy on hand is invaluable. If you don’t have one, then read Steve Krug’s Don’t Make Me Think book and then go and find yourself a UX guy  :) 

Facebook

The Facebook API is now pretty mature and is very stable (compared to the last time I used it a few years ago).

One annoying things is that the calls that you make to explore the user’s social graph (via the FB.api function) are very low-level and the data is returned in a callback. This can make your code very messy, since you often have to chain your app’s logic in the callback function, if your app needs data before it can proceed. For example:

FB.api('/me', function(response) {
  // do something with response here
  // make next api call (with nested callback) here
});

Additionally if you need to, for instance, retrieve friend profile data you can land up making hundreds of calls:

// grab all my friends
FB.api('/me/friends', function(response) {

    $.each(response.data, function(index, value) {
       // get profile for each friend (this triggers another HTTPS call)
       FB.api('/' + value.id , function(profile) {
         // do something with profile data
       });
    });

});

Luckily there is a better way, the FB.query and FB.waitFor functions. These let you run queries against a user’s data using a SQL like language (FQL) and then block waiting for the response. This greatly reduces the time it takes to retrieve data and keeps your apps logic a bit simpler:

var query = FB.Data.query('SELECT uid,name,current_location FROM user WHERE uid IN (SELECT uid1 FROM friend WHERE uid2=me())');
query.wait(function(rows) {
    $.each(rows, function(index, value) {
       // do something with friend data here
    });
});
// rest of app logic continues here

Firebug

Get it (hopefully this isn’t news to you). This tool is invaluable for web development. It allows you to debug JavaScript, view the DOM, check network traffic and tweak CSS in browser.

It’s also a good idea to test you app in different browsers as you go. During this app’s build I had FireFox, Chrome and IE 8 all running so I could see that everything was working. Luckily by using Twitter Bootstrap and jQuery I had no cross-browser issues at all other then the fact that IE didn’t get nice round corners.

Make sure it validates

The W3C provides a tool for validating web pages. It’s always good to be sure that your pages validate, not just because it is good practice but because it stops browsers from entering their “quirks mode” which often causes odd and quirky behavior.

The Web Developer plugin for Firefox provides a handle shortcut for submitting your local HTML to the W3C validator.

In my case everything validated except for the Facebook namespace declaration and the custom Facebook tags such as fb:like. Whilst there are ways around this I decided on not been too purist.

HTML5

Try use HTML5 for your app. It works very nicely. For IE browsers (before 9.0) use the HTML5 shim to get some HTML5 support.

However Twitter Bootstrap does not support IE6, so I used this trick to make any IE6 browsers redirect the user to a “browser unsupported” page with tips on how to upgrade their browser. In the __section add the following comment:

 <!--[if lt IE 7]><script type="text/javascript">window.location = 'notsupported.html';</script><![endif]-->

This uses the conditional comments feature of Internet Explorer to redirect IE6 browsers to the notsupported.html page, other browsers will simply treat it as a HTML comment.

Improving load times

It is best practice to load all the CSS files in the section of the page and all the JavaScript files at the end of the page (just before the </body> tag). This makes the page feel much more responsive as the browser can start to render and layout the page before everything is loaded.

Having said that, a complex web app tends to pull in a lot of CSS and JavaScript files and minimizing the number of objects that are fetched from the server is a key factor in having a speedy app.

Ideally you only want your app to load a single CSS file and a single Javascript file. To achieve this in my app I concatenate the CSS files and JavaScript files (in the correct order) and then compress them using Yahoo’s YUI compressor.

Since I wanted this to be a repeatable process I used Ant to create a build script to do this. Firstly the concatenation:

<concat destfile="${tmp.dir}/all.js">
  <fileset file="${www.dir}/jquery.min.js" />
  <fileset file="${www.dir}/jquery-ui-1.8.16.custom.min.js" />
  <fileset file="${www.dir}/jquery.masonry.min.js" />
  <fileset file="${www.dir}/bootstrap-modal.js" />
  <fileset file="${www.dir}/bootstrap-twipsy.js" />
  <fileset file="${www.dir}/bootstrap-alerts.js" />
  <fileset file="${www.dir}/app.js" />
</concat>

And then the compression:

<exec executable="java">
  <arg value="-jar"/>
  <arg value="yuicompressor-2.4.7.jar"/>
  <arg value="${tmp.dir}/all.js"/>
  <arg value="-o"/>
  <arg value="${tmp.dir}/all.js"/>
</exec>

Now during development you don’t want to be dealing with concatenated and compressed files so my PHP pages contain a snippet like this:

<?php
if ($config['mode'] != 'prod')
{
?>
    <script src="jquery.min.js"></script>
    <script src="jquery-ui-1.8.16.custom.min.js"></script>
    <script src="jquery.masonry.min.js"></script>
    <script src="bootstrap-modal.js"></script>
    <script src="bootstrap-twipsy.js"></script>
    <script src="bootstrap-alerts.js"></script>
    <script src="app.js"></script>
<?php
}
else
{
?>
    <script src="all.js?@buildtimestamp@"</script>
<?php
}
?>
</body>

With mode getting set in my app’s config file.

Of course there are a ton of other things you can do to tune your app. Get the YSlow addon for Firebug and run the report on your pages to see what can be done.

Cache busting

Ant can also very helpfully update parts of your files whilst copying them around. This can be used to burn in all sorts of things, like version numbers and build timestamps. It can also be used create a effective cache buster when a new version of your app is released….

Firstly you need to set up an Ant property with an appropriate value:

<tstamp>
  <format property="buildtimestamp" pattern="yyyyMMddHHmmssSSS" locale="en,UK"/>
</tstamp>

Then in your HTML you can append @buildtimestamp@ following to file references:

<link rel="stylesheet" href="all.css?@buildtimestamp@"/>
...
<script src="all.js?@buildtimestamp@"></script>

Then lastly apply a filter in Ant when you are copying the files around:

<copy todir="${tmp.dir}">
  <fileset dir="${www.dir}">
    <include name="**/*.php" />
  </fileset>
  <filterset>
    <filter token="mode" value="prod" />
    <filter token="release" value="${rel}" />
    <filter token="buildtimestamp" value="${buildtimestamp}" />
    <filter token="builddate" value="${builddate}" />
  </filterset>
</copy>

PHP config file

To keep things nice and clean, I created a config.php file looking like this:

<?php if ( ! defined('CONFPATH')) exit('No direct script access allowed');

// set this to '@' + 'mode' + '@' during development so that style sheets and javascript files are individually included (see index.php)
// ant build scripts will replace this at build time
$config['mode'] = '@mode@'

?>

Then in my other PHP files I pulled in the config file using:

<?php
   define('CONFPATH','config.php');
   require_once CONFPATH;
?>

As you can see from the comments and the Ant snippets above, I overwrite the mode value with _‘prod’_ during my Ant build process to ensure that my compressed .js and .css files are used.

Miscellaneous files

Finally make sure you add a favicon and a robots.txt to avoid annoying 404 webserver logs. For fun why not add a humans.txt too.

Summary

Well that pretty much covers it. Hopefully there are one or two new  ideas here that you can use in your apps. Feel free to post any questions in the comments section below.

The viral spread of my privacy check app for Facebook

Last week, interested by the noise on the interwebs about Facebook and privacy, I put together a small app using Facebook’s API that shows exactly what information you are giving away to Facebook enabled sites.

Thinking that others might be interested, I tweeted a link  and shared the app with my friends in Facebook. Now I don’t have a large social network about 220 followers on Twitter and 125 friends on Facebook, but the “like” counter for the app quickly started to grow first 10, then 20 and after a day close to 100!

A couple of days later, somewhat ashamed by how ugly the app looked, I did some work to pretty it up and I added a “privacy score”. I then retweeted the link, did a Facebook status update and went to bed.

The next morning (17 May), I was very surprised to see that I now had 200 “likes” and over 1500 page views. Over the next few days things started to  grow rapidly. I watched anxiously, concerned that my tiny little web server was going to explode…

I also started to wonder how people were actually out finding the the app. Here is what I can surmise from looking at my site logs and analytics.

Page hits and visitors

First off here are the graphs showing the page hits (dark blue) and unique visits (light blue)  per day:

As you can see the app was at peak usage on the 18th May. To date the app has had 1936 “likes” and 28845 page views.

Where did all the traffic come from?

The bulk of the traffic (64.55%) came from referring sites, 34.05% came from direct traffic and a measly 1.4% of the traffic was from search engines.

Of course a large percentage of the referring traffic was from Facebook, as my app allowed people to post their “privacy score” for all their friends to see. The post included a handy link back to my app so that others could then easily check their score.

A total of 11.8% of the total visitors to the app came from Facebook. The other referring traffic was from the large number of blogs and websites that posted articles about the Facbook Privacy issue with links back to my app. In fact there were over 100 sites that did link back to the app which is pretty staggering. See the end of this post for a list of some of them.

The direct traffic figure of 34.05% is interesting as it is highly unlikely that anyone was typing the URL to the app (http://www.rabidgremlin.com/fbprivacy) directly into their browser. I can only guess that that these visits must have come from people clicking links in various (non-browser based) twitter, RSS feed and (perhaps) mail clients.

Looking at the data I would suggest that over 45% of the total traffic could be attributed to social networking ! (the 11.8% from Facebook and the 34.05% direct traffic).

Riding the hype curve

I suspect that the primary reason that the app became so popular so quickly was it appeared right when there was the most Internet buzz about Facebook privacy.You can clearly see the strong build up and the equally sharp drop off in traffic as the Internet became enamoured with some other topic.

Did I WIN?

Now if my app was a commercial product then I would say NO. Around about the same time as my app went live so did several others such as: youropenbook.org, reclaimprivacy.org and Saveface.

I would say that reclaimprivacy.org would be a direct “competitor”. If you take a look at their site they have over  238000 shares and over $3000 in donations. So around 12100% more shares/likes and roughly 60000% more in donations!  BTW I must give a shout-out to Brian for his donation (and yes it is enough to buy a beer in New Zealand, thanks).

Clearly reclaimprivacy.org would be destroying me if we were competing businesses :)

So why did they do so much better?

I don’t believe that it is the “product”, several sites have mentioned that my app provides the most in-depth information.

I could see that perhaps it is because their app looks better but no one mentioned that my app was ugly or unusable.

I suspect that the answer is that of trust, reclaimprivacy.org has a nice domain name, the author released the source code and perhaps more importantly the author (Matt Pizzimenti) is readily identifiable. On the other hand, rabidgremlin.com sounds a bit suspect and I go out of my way to keep my identity secret (not that it would be hard to figure out who I am).

My suspicion is borne out by some of the descriptions of my app, such as this one:

The application is built by rabidgremlin, who doesn’t tell us anything about himself (I’m making an assumption here) and claims New Zealand as his home on his Twitter page. Can you trust the site?

I guess I’ll need to take this into account for future projects :)

Sources of traffic

The following list shows some the sources of traffic to my app:

http://sikkerhed.tdc.dk/publish.php?id=24892

http://youropenbook.org/about.html

http://www.f-secure.com/weblog/archives/00001952.h

http://www.net-security.org/article.php?id=1439

http://www.hwsw.hu/hirek/44627/facebook-adatvedele

http://www.infospyware.com/blog/facebook-privacy-c

http://news.ycombinator.com/item?id=1346003

http://webisztan.blog.hu/2010/05/17/teszteld_hogy_

http://www.simplyzesty.com/facebook/tool-lets-chec

http://themoderatevoice.com/73103/concerned-about-

http://www.guardian.co.uk/technology/blog/2010/may

http://bagtheweb.com/oembed/proxy/IBB2DI

http://www.taringa.net/posts/info/5534654/Facebook

http://37signals.com/svn/posts/2330-diasporas-curs

http://index.hu/tech/2010/05/21/kivonulas_a_facebo

http://www.informacija.rs/Vesti/Privacy-Check-apli

http://news.ycombinator.com/item?id=1354731

http://slo-tech.com/novice/t416348

http://flipthemedia.com/index.php/2010/05/facebook

http://wiredpen.com/2010/05/18/facebook-and-privac

http://www.cnis-mag.com/quand-les-navigateurs-parl

http://www.reddit.com/r/technology/comments/c3jyi/

http://caballe.cat/wp/eina-que-calcula-el-nivell-d

http://www.reddit.com/r/reddit.com/comments/c557m/

http://www.facebook.com/notes/f-secure-labs/facebo

http://www.cnis-mag.com/4456.html

http://www.taringa.net/posts/info/5515348/Facebook

http://taringa.net/posts/info/5534654/Facebook-Pri

http://twitter.com/mikebutcher

http://www.wir-muessen-twittern.de/blog/2010/05/18

http://www.techsupportalert.com/content/probably-b

http://www.matuk.com/forosmatuk/vida-digital/priva

http://livingwithoutfacebook.com/

http://giovannidepaola.nova100.ilsole24ore.com/201

http://blog.f-secure.jp/archives/50403236.html

http://youropenbook.org/about

http://socialbits.net/blog/facebook-and-privacy-to

http://www.lemagit.fr/article/google-france-facebo

http://news.ycombinator.com/item?id=1347982

http://www.websegura.net/2010/05/teste-a-sua-priva

http://www.plurk.com/bikerock

http://www.lemagit.fr/article/google-france-facebo

http://www.softcatala.org/planeta/

http://www.islamicaweb.com/forums/science-technolo

http://vulps.forumotion.com/discussion-f4/oo-er-so

http://twitter.com/theharmonyguy

http://twitter.com/harryadams

http://maketecheasier.com/4-tools-to-unscramble-yo

Facebook: Keeping things between friends

One of the most visited posts on my blog is titled “FaceBook:Do you know who is watching you?” which details how to keep your Facebook profile out of the public eye. In light of the MI6 chief/Facebook scandal (his wife obviously doesn’t read my blog), I thought I’d update the instructions so you can avoid suffering a similar embarrassment :)

  1. Log in to Facebook and from the Settings menu (top right of the page) select Privacy Settings
  2. Now select the Profile section (alternatively click on this link: http://www.facebook.com/privacy/?view=profile&tab=basic)
  3. Set all the options to Only Friends and click on the Save Changes button
  4. Now select the Contact Information tab (or follow this link: http://www.facebook.com/privacy/?view=profile&tab=contact)
  5. Once again set all the options to Only Friends and click on the Save Changes. If you don’t want your friends to know your email then you can select No one as the option for your email address too.
  6. Click on the Privacy part of the breadcrumb at the top of the page and the select the Search section (or follow this link: http://www.facebook.com/privacy/?view=search)
  7. For the Search Visibility option you have a couple of choices. If you are completely a paranoid set it to Only Friends. If you are slightly paranoid then set it to Friends of Friends. If you don’t care then select one of the other options such as Everyone.
  8. In the Search Result section you can control what people can see when they do search for you. I would suggest leaving only the A link to add me as a friend and A link to send me a message options selected. You can also turn on My profile picture but it all depends on how compromising your profile pictures get :)
  9. Lastly in the Public Search Listing section uncheck the create a public search listing for me… option because you don’t want your profile showing up on Google.
  10. Click on Save Changes and you are all done.

Happy Facebooking…