How to debug errors

There seems to often be confusion among developers when they encounter an error in the software or script they are working with. I've compiled a simple process for debugging without wasting lots of time (both yours and others')

1. Look at the first error and fix it. 
2. No really, don't consider all the warnings and other errors unless they can illuminate why that first error occurred. Once there is an error, all bets are off for the rest of the code/script/whatever. 
3. Seriously though: You tendency is going to be to look for something familiar that think you (or you think a coworker) can fix. You're going to try to fix the easy problem instead of the real problem and waste a bunch of time. Errors and warnings cannot go back in time, so the first one almost certainly was not caused by the later ones. 
4. After trying to fix the first one, you may discover reliable information about why that error can be ignored. Don't ignore it without a good reason or you are just wasting time. 
5. Don't fixate for too long. Seek someone else's perspective. Explain the problem to a ruber duck. 
6. After you've fixed the first error, run the code again and return to step 1. 

When asking others for help with your debugging, always be sure to include all the information, especially the first error. Don't assume that you know what is relevant and waste other people's time (and your own.)

Android - Reliable Alarms to Services

I love programming for Android. The APIs just make sense. You can tell that fluent Java developers designed this thing. Despite the fact that everyone's favorite criticism is that Android is "fragmented" I haven't had any significant cross-hardware issues. Follow the guidelines (What?!? I have to follow rules if I want my app to work? So much for freedom...) and 95% of apps should just work on any platform without much trouble. I'd say you're even a step ahead of iOS development because the platform has extremely simple ways to support multiple screen sizes, so your app can easily look good on any device. That said, there are occasional bugs, and this post is about one of them.

This may not even be a bug, but rather a case of poor documentation. The problem arrises when you use the AlarmManager to schedule a repeating alarm that is supposed to wake the phone and start a service. e.g.,

PendingIntent pi = PendingIntent.getService(this, 0, intent, PendingIntent.FLAG_UPDATE_CURRENT);
alarmManager.setRepeating(AlarmManager.RTC_WAKEUP, timeToStart, 15 * 60 * 1000, pi);

It will probably work great during most of your testing as the phone will be awake. However, once you stick your phone in your pocket and forget about it for a while, you'll realize, "Hey, wasn't an alarm supposed to go off?" You'll pull your phone out, turn it on, and suddenly it will go off. The first few times you'll think it's odd, until you realize that the alarms aren't going off when the phone has been asleep for awhile, but they are delivered as soon as it wakes.

Hopefully you find this post or something like it and don't have to spend days Googling to figure out that only a BroadcastReceiver can reliably receive a wakeup alarm and then spend another day learning that the Context that onReceive gets can't start services. The solution is pretty simple, but you'd never figure it out from the documentation.

EDIT: I may be totally wrong about the context being a problem as you can see in the comments. Try it without the using getContext first and if you don't have any problems getting the service to start, avoid the Cargo Cult code I'm using below. If you do have problems and the context trick helps, please let me know in the comments.

import android.content.BroadcastReceiver;
import android.content.Context;
import android.content.Intent;
import android.content.pm.PackageManager.NameNotFoundException;


public class AlarmProxy extends BroadcastReceiver {

    Context getContext(Context ctx) {
        try {
            return ctx.createPackageContext("com.yourpackage.ofyourapp", 0);
        } catch (NameNotFoundException ignore) {}
    }
    
    
    @Override
    public void onReceive(Context ctx, Intent intent) {
        Context context = getContext(ctx);
        Intent newIntent = new Intent(context, YourService.class);
        newIntent.setData(intent.getData());
        newIntent.putExtras(intent);
        context.startService(newIntent);
    }
    
}

The context your receiver gets is probably a ReceiverRestrictedContext which isn't connected to your application and can't start services, so you may need to use createPackageContext to get your app context and start your service. You'd then set alarms by sending them to your receiver:

Intent intent = new Intent(this,AlarmProxy.class);
// ...
PendingIntent pi = PendingIntent.getBroadcast(this, 0, intent, PendingIntent.FLAG_UPDATE_CURRENT);
alarmManager.setRepeating(AlarmManager.RTC_WAKEUP, timeToStart, 15 * 60 * 1000, pi);

Happy alarming!

EDIT: Don't forget to register the receiver in your Android manifest:

<receiver android:name=".AlarmProxy"> </receiver>

A Grails Porcelain Ajax Framework - Part I: Component Refresh

The Foundation

The foundation of a good porcelain Ajax framework is component refresh, i.e., refreshing an area of the page with new/updated information. Component refresh is equivalent to reloading the page, but provides a better user experience since it happens asynchronously, and reduces the amount of work the server has to do.  Because we are requesting a block of HTML from the server, we don't have to write any additional JS to update the page. jQuery.load does all the heavy lifting. With component refresh it becomes trivial to Ajax enable pagination, sorting, filtering, lazy loading, tabs and many other actions for a better user experience and better performance. In this post, we'll learn how to create a porcelain Ajax framework with component refresh using Grails.

The App

Porcelain frameworks are usually built around an application. For our purposes, lets say we have a simple app for creating, listing, viewing and editing products. The products have a name, price, and description. Additionally, each product can be viewed in four different colors. In our app, the only difference between colors will be the image that is displayed for the product, but the different colors could easily be different tabs of product information, or different configurations of the product.

All our work will be on the product list, which will look like this:

our made up products list

The Project

We start with a standard Grails project and add a Product domain class and generate the controller and views. How to do all that with Grails is out of scope for this discussion, but it's pretty easy.

By default, Grails lists products in a table. Typically products are listed as "cards" to give them more room, so we'll enhance the markup slightly:



At this point we haven't written a line of Ajax code, we're just creating a simple app that will work even with JS disabled. In anticipation of adding Ajax functionality though, we should break the list page up into the components that we plan to refresh. To do this, we'll make use of the g:include tag, which allows us to inline another controller action in our page. We'll create two new controller actions and adjust our templates accordingly:



Notice the original list code has moved into renderList and we return productIds instead of the actual product object.

And we replace the list in list.gsp with g:include controller="product" action="renderList". renderList.gsp is just:



and renderProduct.gsp is the product div above.

For the colors pages, we're going to reuse renderProduct:



and that's it. We now have all our functionality setup and it all works even with JavaScript disabled. Time for some progressive enhancement.

The Framework

The core of our component refresh implementation is going to be g:include. Since g:include is plumbing, we need to ensure that our porcelain covers it very thinly to make future extension easy. One option would be to  use Groovy meta-programming magic to intercept calls to the built in Grails taglib where g:include is. This  has the advantage of being transparent to the user. We don't have to worry about using the right tag, and if we already have a lot of existing code using g:include, this would be the way to go. However, it has the disadvantage of being complicated. If we're starting from scratch, it's much simpler to define our own tag. So enter the FrameworkTaglib:



Instead of g:include, we will use ajax:include. The new tag does two thing. First, it wraps the content of the component in a div with the class "ajax-component" so that out framework JavaScript will know that it's a component. Secondly, it sets the id of that div to the controller action and optional id for the include. That will allow us to call the correct action from our Ajax requests.

The second part of our framework is framework.js:



refresh is also a very thin wrapper, this time around jQuery.load. It has the nice feature of allowing us to use any element inside a component as a representative of the component. Also, we don't have to pass a URL, and can pass a data string or object, just like jQuery.load.

With these two components, our framework is ready! Our first task will be to Ajax enable the color links so we can view the different colors without leaving the list:



With our framework code in place, it couldn't be simpler. We strip the query string off of the link and pass it as data to refresh and we're done. The color links now refresh the individual products in place, for a greatly improved user experience. Also, notice we use jQuery.delegate so that when the component is refreshed we don't have to reattach event handlers.

So that was easy, but what about pagination? Surely it will be a bit more work to Ajax enable pagination.



Or maybe it's the exact same code. It looks like progressively enhancing links in this way might be something to add to framework.js. Notice that we can refresh the entire list as a component, but also have nested components that can be refreshed individually.

In very little code, we've built a framework that has allowed us to progressively Ajax enhance both pagination and content toggle in about seven lines of JS. But lets take an objective look at what this framework does well and what might be problematic.

You can download the complete project to this point from GitHub.

The Good

There is a lot to like about the framework as it stands.

• Very simple usage.
• We take advantage of progressive enhancement. Normal links become refreshes very easily.
• The whole framework is less than 50 lines of code at this point.
• Components can be nested inside other components.

The Bad

Using this code as is on a serious project is probably not the best idea for a few reasons:

• All components are wrapped as divs. That means refreshing something like a table row is impossible.
• In framework.js we assume that the default URL mapping of "/$controller/$action?/$id?" is used, which may not be the case, in which case refresh will fail.
• We have to pass all the parameters to refresh. For just progressively enhancing links, this isn't such a big deal since the link should have all the parameters, but as we get into more advanced features, it will become a pain point.
• Several features of g:include will not work. In particular, the model and params attributes will not be present in a refresh. For simple cases, that's not a problem, but a seriously application will quickly feel the pain of such limitations.
• Two nitpicks
• We use the namespace "ajax" for the taglib, which is not likely to be unique. Another poorly written plugin could cause problems.
• We use the id attribute to store the controller action information. A class might be more appropriate, if a little harder to parse.

The Conclusion

The core idea of a good component refresh is being able to pair together the view (GSP) and the code that generates the model (the controller action). This pairing allows us to treat each component as it's own entity, making refresh and other advanced features possible. Grails and jQuery provided us with some very nice pipes to start with, so creating the component refresh porcelain was perhaps easier than it would normally be, but I think the simplicity helps the underlying concepts to show more easily.

Next time we will build multi-component refresh and try to fix some of the problems with our framework. It's not good from much beyond basic enhancement at this point, but we're off to a good start.

Remember, you can download the complete project to this point from GitHub.

How to Build an Ajax Framework

Plumbing & Porcelain

I've built three or four Ajax "frameworks" in my career. I don't know if you would call them frameworks because there are really two kinds of things I mean when I say "framework". There is the extremely general purpose plumbing framework, which can do anything, but as a result leaves you to write a good bit of code and maybe pick and choose which parts you need. Some examples of plumbing type Ajax frameworks/components would include sever side frameworks like DWR, Ajax4JSF, etc. but also client side frameworks like dojo, Scriptaculous, jQuery, etc. Also most of the Rails type server frameworks have Ajax support built in: e.g., Grails and Django.

Then there are the porcelain frameworks. The problem with the plumbing frameworks is that they can do anything, so you're presented with a big toolbox full of neat things and a bunch of pipes, but you have to put it all together. A porcelain Ajax framework is usually built by an organization for their apps, or even for a single application. Sometimes they become an entire platform internal to the application and sometimes no thought is put into developing the porcelain and you end up with each developer doing Ajax in their own way.

Inner Platforms & Thoughtless Scaffolding

Some developers recoil at the thought of internal platforms. They tend to make a few use cases easy but then require horrible black voodoo workarounds when trying to do something the original architects didn't think of.  Other developers are completely frustrated with the thoughtlessly thrown together and generally rickety Ajax platform they've been trying to maintain.

An inner platform sometimes arises when otherwise highly competent enterprise (read: Java or .Net) developers first encounter front-end programming and dynamic languages.  They are used to structure and standards and ACID properties and so forth. This lack of static types and compilers instills a certain sense of fear and the only way to conquer that fear is to create "coding standards" and, then when those are not followed, to create an internal UI framework that forces everyone to code in a certain way, without closures, functional programming, overloaded operators, dynamic dispatch, and all those other ugly hacks that are bad for performance.

Other times, an inner platform is really just the result of well intentioned senior developers trying to make things easier for the more junior members of the team. Unfortunately, they often do so by hiding important information and you end up with a very leaky abstraction that is buggy and that the users (programmers other than the authors) are completely helpless to fix because they have no experience with the underlying framework. The points is that there are many ways to end up with an inner platform that does more harm than good.

Thoughtless scaffolding on the other hand is just the result of... thoughtlessness. Most often it's because a programmer without much experience and no real computer science training is writing all the code and whatever tool he's read about today is the perfect tool for the job. But it also happens when a new project is started and the technical leadership fails to identify the patterns that should be used. Maybe they start off with some vague notion like, "JSON is the answer," and become committed to a bad strategy early on and never have the time or the courage to go back and clean it up.

Both extremes have problems, but they also have advantages. For the tasks it was built for, the inner platform makes adding  new Ajax features extremely simple. On the other hand, the thoughtless scaffolding never gets in the way. Because very little of it is connected, it's easy to rip part of it out and build something new entirely. 

We want to find a middle ground between over engineered and thrown together, but where that middle ground lies can depend a lot on whether maintenance or new features are your priority. Instead we will focus on the core principles and features that any porcelain Ajax framework should encompass.

Core Principles

These are the core principles I have learned through trial an error over the past 5 years, building and using frameworks across the spectrum from somewhat thoughtless to a massive inner platform:

1. Remember it's a web app - It's not a desktop app. You're probably not build the next Gmail, so stick with what works. Use semantic markup, keep the pages simple and light, and use progressive enhancement so the app works without JS. For public facing apps this is a must obviously, but even for internal apps where you know that your users have a decent browser with JS enabled, you will stay out of trouble by keeping it simple.
2. Do everything you can on the server - Especially rendering HTML. JSON is overrated. jQuery's load method can do 95% of what you need in one line of JS. For the most part, JS should not be producing HTML. Things go much more smoothly when all HTML is produced server side because then the logic to generate the markup doesn't need to be duplicated. There are exceptions, like if you have a feature using 3rd party code, like a Google Map, that simply will not work without JavaScript. Then it might be necessary  to generate some of the UI in JS. If you must generate HTML in JS, use a templating framework. If you are concatenating anything more than 100 characters of HTML together, you're making a mess of your code.
3. Keep the porcelain thin - Do not under any circumstances create more than one layer of abstraction between the code and the core web technologies. Most of the HTML output of your application should correspond to HTML in a template file. Taglibs are fine, but try to use your templating language even inside of them. Do not create a system where the developer doesn't have to know HTML. The same goes for JavaScript and CSS. Do not write code to generate CSS ever. You may use a framework that generates CSS to reduce repetition, but try to only use the big ones with a good community. Do not generate JavaScript code ever. Generate inline JSON instead and have your scripts consume it to determine behavior dynamically. Use stateful CSS rules.
4. Cover all the pipes - When you realize that to make it all work you need every load() call to add some parameter, or check some condition, you better hope you can go to one place in your code and make that change. Hopefully the plumbing provides good hooks, or there is some AOP/IoC magic that can cover it. Otherwise, you may have to create very very thin wrappers around plumbing functionality. Not in the hopes of portability (you will never port this app to a different framework, it will be rewritten), but so that you can funnel all calls to that feature through one place.

Core Features

So what features does a HTML based, server oriented porcelain framework provide?

1. Component Refresh - This is the core. There needs to be a simple way to designate an area of the page as a component and an easy way to make a call to refresh its HTML. This may be as simple as using jQuery's load with a selector to get the desired page fragment, although it's nice if the server can do the minimal amount of work necessary to render the HTML you need.
2. Multi-Component Refresh - Sometimes you need to refresh several parts of the page whose only common parent is the body element. You could just reload the whole page, but for an action that only affects part of the page that's both inefficient and bad UX. You could also use multiple requests, but if there are more than 2 components, then many browsers will block the 3rd request until one of the others return. There is probably also a server part of the framework that allows the components to share data instead of making redundant calls.
3. JSON - Not to be consumed by Ajax calls, but to inline as script tags in the page. Most frameworks can take the same data model you pass to your templates and convert it to JSON automagically for your JS to consume too. You probably will want to include it inside your components' HTML so you can update the JS state when a component is refreshed.

Next Time Let's Build One

Soon, I hope to have some example code for a simple Grails porcelain framework. Actual code can be a lot more informative and illuminating than pithy maxims.

JSON is Overrated

JSON is great. It's an extremely concise, machine parsable yet still human readable format. It's the best thing to happen to data interchange since XML, and it makes XML looks like the big piece of dog crap that it is. (Seriously, angle brackets? And redundant closing tags? Who thought that was human readable?)

However, some of you may be surprised to know, JSON is not the only format that can be returned in an Ajax request. Don't get me wrong, if I make a request to the Twitter API or some other webservice, I'm not going to be happy if it returns XML. That stuff is a pain. But there are other uses for Ajax that just consuming raw data. Did you know you can return HTML?

For example, search results since the beginning of time have been paginated. The nice thing to do nowadays is to make an Ajax request when the user requests the next page so we can avoid reloading the header, footer, sidebar and whatever else in your page isn't a search result. Now you almost certainly rendered the first page of results on the server, using whatever templating language it is you use. Something like:



So when it comes time to make an Ajax call and load that next page of results, naturally you will write another controller method to render the results as JSON and then a bunch of hideous JavaScript to concatenate HTML together and replicate the logic you already have on the server. Because that's a lot easier than:


([/sarcasm] for the facetity impaired; it's a terrible affliction.)

When you're creating any sort of Ajax endpoint, just ask yourself one question, "Is anyone other than a script on this site going to consume this?" The answer is probably no, so I'm going to ask you a question in return, "What is wrong with you? Why would you want to generate HTML on the client by concatenating strings together instead of on the server with a nice templating language?"

If you're reading this blog, you probably already know that HTML concatenation is a code smell. You quickly realize that replicating all that rendering logic in JavaScript is going to be a mess. I bet you also know someone who's code stinks, so please find them and give them a gentle talking to about how their shiny new JSON hammer is not the best tool for screwing with Ajax.

OAuth and Android - Dropbox is wrong; never ask for passwords

EDIT: Not having an iPhone and not having any dev experience with it (yet), I wasn't sure whether or not this was possible on an iPhone. However, I've had an iPhone dev assure me that it is and that several apps (TripIt being an example) do it. So there really is no excuse. If OAuth is good on the web, it's good on iPhone and Android too.

Dropbox recently announced their API, which is pretty exciting because it's a great service. It uses OAuth for authentication, but under the Authentication For Mobile Devices section, there is this statement:

While Dropbox uses OAuth for the base of its authentication strategy, OAuth doesn’t work very well for mobile devices. It’s much too difficult and error prone to have a mobile device switch context to a browser, do the OAuth handshake, and then magically return to the original application with the tokens in hand ready to use. It’s bad for user experience and it’s bad for developer sanity.

 I emphatically disagree with this statement, at least as far as Android is concerned. I think OAuth vastly improves the user experience, even from a mobile app and this attitude that it's too hard or too much trouble is damaging to both the experience and security of users.

Lets take their statement piece by piece.

It’s much too difficult and error prone to have a mobile device switch context to a browser

This is trivial on Android. I'll use the Gowalla OAuth flow in my examples because I think it's an elegantly simple implementation of OAuth, but this works for any OAuth system:



That's it. The browser is launched and the user will see the Gowalla authorization page. If they're already logged in, they don't even have to type their username and password. Now there is an activity that is difficult and error prone. I hate entering login credentials on my phone, and I can copy and paste them from LastPass. Even with Swype, typing on a mobile device is neither easy or error free. Save your users the headache and use OAuth.

It’s much too difficult and error prone to ... do the OAuth handshake, and then magically return to the original application with the tokens in hand ready to use.

Again, this "magic" is almost trivial for an Android app. You simply specify that your activity can handle the custom data URI scheme you specified in your redirect link:



Once the user clicks "Allow", your activity will be launched and the Intent data will have a parameter telling you the access code needed to complete the OAuth handshake. This isn't any more difficult and error prone than implementing OAuth in a web application. But it's not that difficult, because there are plenty of great OAuth libraries out there to help you on the web server, and you can use them on your Android device too.

It’s bad for user experience

If this is true for OAuth on mobile devices, then it's even more true for the web and we should abandon OAuth completely. An app launching a browser is not an unfamiliar thing. Receive an email with a link on a desktop, you click the link and it opens your browser. This is normal, good UX. I click a link to comment on your site and you take me to Facebook? That's weird. As a user I'm not sure what is going on. Suddenly everything looks different... I thought I was commenting on your blog? Why are you on my Facebook?

The response to this (for web applications) is that users simply need to become familiar with the concept and then it wont seem weird at all, it will seem normal. That argument also works for mobile devices. In fact, if you are worried about the user not knowing what is going on, just tell them! "You will now be taken to Gowalla to authorize this app. You may be asked to log in. After you authorize myApp, you will be returned here."

OAuth is Good for Dev, and Good for UX

The idea that OAuth isn't suited for mobile isn't in any way grounded in the Android mobile experience, as far as I can tell. That might explain why Dropbox has a lot to say about it's Objective-C library, but for Java they only offer two words: "Coming soon."

Anything you can truthfully say about OAuth being bad UX or bad for dev in regards to mobile is equally if not more true for the web. But nobody is suggesting that OAuth be abandoned on the web and we go back to asking users for their passwords. So don't do it in your mobile apps either.

jquery.writeCapture.js - Now with 60% more magic!

I've gone ahead and written a jQuery plugin for writeCapture.js. I've reduced the problem to making one extra call in the chain and then the voodoo black magic takes over and you can't go wrong. Here's a kitchen sink example:

$('#foo').writeCapture().find('.bar').html(dangerousHtml).end().
find('.baz').load('returnsDanger.php').after(moreDangerousHtml).end().
addClass('loading');

In this context, "dangerous" HTML is any HTML that does or could contain a script tag that uses document.write, aka, the evil method. The implementation details are wonderfully complicated, but I've made the usage so simple that any code monkey can just chain whatever they like off of a call to writeCapture() and everything will be perfectly "safe". e.g., document.write will not destroy the application and all the content will show up where it is supposed to. So enjoy.

EDIT: the comments here aren't a good place to ask for help. You can get help through GitHub.