'Opt-In' Changes and the Importance of Testing Suites

A development mistake I see all the time is when a developer fails to make changes to core classes 'Opt-In' or backwards compatible. For example, suppose you find a bug in your table/grid widget that causes one of the several instances of that widget to fail mysteriously. You come up with a solution that involves an extra method call or two at a certain point, or changing an option/property to another value. Your first instinct might be to make that change the default behavior, but in most cases, that's wrong.

Minimize Affect

Summary: How many instances of that widget do you have in your application? Are they all failing? Why not? Were they functioning properly before? If they ain't broke, why fix 'em?

Most likely the change you are going to make really should have been the default behavior all along, however, you now have working code that was developed and presumably tested against the old behavior. If you change that behavior now, even if it is now the 'right' behavior and was previously the 'wrong' behavior, you now need to test every single instance and subclass of your widget in all possible scenarios to ensure you haven't introduced a bug. Do you have a comprehensive automated testing suite? If you do, great, but you probably don't.

Even if you do have a good test suite, why fix something that isn't broken? Suppose, instead of making that change the default, you instead add a useFooHack property to your base class. It defaults to false, but if it's true, you use your new behavior. When you discover that other instances have a problem caused by the old behavior, you just change the flag to enable Foo, and it's fixed. Easy as that. If you discover that every last instance has the problem, you can remove the useFooHack=true line from your instances and if(useFooHack) from your base class. Simple. The alternative is to risk introducing bugs into every instance that you thought was working.

Testing is a waste of time

Now some people will learn of problems like this and say you should have done more testing before committing. You know, you should have gone through all the major paths of your application to ensure that they're working before committing. Most of the time those people are managers who don't have to waste time doing the testing themselves, but expect you to do it.

Suppose a developer spends X time testing before a check-in and finds some bugs that take Y time to fix, it has cost him X+Y time. However if he instead verifies that his code does what he intended it to do and checks in, but someone discovers the bugs, emails him, and he spends Y time fixing it, it only cost him Y time. Now if X is small, say 30 seconds, because all the developer had to do was kick off the automated testing suite, then the developer will do the testing to avoid getting angry emails. However, if testing takes more than a few minutes, most people will realize they can get more work done by letting someone else hunt for bugs. You can argue about it all you want, you're not going to change human nature, and the optimal choice most of the time is to skimp on the testing.

In short, you have to work with people the way they are. No amount of training or lecturing will keep people doing things that are not in their own best interest. If your project's big problem is that people keep breaking the build, make not breaking the build the easy path. You may have to sacrifice some hours to developing and maintaining a test suite, but you will generally save more hours of lost productivity from broken builds.

Java Fun Continued - Groovy FTW

Here's the equivalent Groovy for my previous Java abomination that prints the Java wrapper methods for me:

new File("src/main/java/").text.
replaceAll(/(?sm)\s*public\s+(\S+)\s+(\S+)\((.*?)\)\s*\{/) {
s,ret,name,params ->
print """
public ${ret} ${name}(${params}) {
if("void" != ret) {
print "return "
print "foo.${name}("
// -1 hack avoids index out of bounds when params is ''
print params.trim().split(/\s*,\s*/).collect {
print """);

I could change the prints to concatenation and reformat the whole thing as a single line, so it only counts as one statement. Technically I cheated a little because this version doesn't look up the methods by reflection, but it doesn't make a difference in the result, and wouldn't significantly shorten the Java code. Definitely more readable than the Java. The JavaScript is a little messier since you don't have multi-line string literals, GStrings, and closures are function(it){ return; } instead of { }.

Java Fun: When all you have is a hammer...

I ought to be ashamed, but I was bored and lazy, so I thought I'd test my somewhat dormant Java skills. Can you tell me (without running it) what this code does?

Method[] methods = Foo.class.getMethods();
String source = FileUtils.readFully(new FileReader(
+ Foo.class.getCanonicalName().replaceAll(
"\\.", "/") + ".java"));
PrintStream out = System.out;
for (Method method : methods) {
Matcher matcher = Pattern.compile(
"public (\\S+?) " + method.getName() + "\\((.*?)\\)",
Pattern.DOTALL | Pattern.MULTILINE).matcher(source);
if (matcher.find()) {
String ret =;
String params =;
out.printf(" public %s %s(", ret, method.getName());
out.print(") {\n ");
if (!"void".equals(ret)) {
out.print("return ");
if (!StringUtils.isEmpty(params)) {
new Transformer() {
public Object transform(Object input) {
return ((String) input).split(" ")[1];
}), ','));
out.println(");\n }\n");

I'll post the equivalent Groovy (and maybe JavaScript) later. I imagine I can do it in a single statement function. I don't miss Java very much.

New (to me) Bike: Issues letting go

My friend John is letting me borrow his fancy Specialized bike that he doesn't ride anymore. I'd say it's about a third of the weight of Old Creaky, my bike that I built at the Yellow Bike Project, and is definitely nicer in every way, but I'm having some problems letting go.

Firstly, this thing is a true road bike, everything is built for speed. The problem is that the tires have little to no tread on them (by design) so if it's remotely wet or gravelly, I make myself slow way down because I'm afraid this thing will slip out from under me. Old Creaky had some decent tread, but not too much. Just something I need to get used to I supposed.

Secondly, the shifters are those neat kind that are built into the brakes, so naturally they're indexed. I've been riding for the past year with friction shifters, so I'm used to being able to go from 5th back to 1st in about 2 seconds just by pushing a little lever down. With these indexed things, I have to click, click, click, and then I'm not sure where I am because they don't have any sort of indicator, so I'm constantly looking down to see where the chain is. I suppose I'll get used to it, I just hope I don't get hit by a car in the meantime.

Lastly, John is about 6'5", and although I'm 6'3" Old Creaky was not built for a 6'3" person, while John's bike is built for John, so I'm bent over a lot further than my back is used to. I'm sure I'll get used to it too.

So, three things to beware of when you get a new bike. I wouldn't keep riding it if I didn't think it was a net positive, but I also never thought any of these things through. I'm working on my fear of change.


Cisco VPN Client Shell Script on OS X

I like to start and stop the VPN client from the command line, but sometimes, for whatever reason, the VPN service isn't running or isn't responding. I then have to go Google it to find the incantation to restart it. Rather than do that every time, I use this script to connect to the VPN:

CMD="sudo vpnclient connect MyProfileName user myUserName pwd myPassword"
yes | $CMD
if [[ $? -eq 8 ]]
# exit code of 8 means it couldn't connect to the vpnd
sudo /System/Library/StartupItems/CiscoVPN/CiscoVPN restart
yes | $CMD


Static vs. Dynamic: The Line

If you read many programming blogs, you've probably heard about the recent rekindling of the static vs. dynamic debate. In particular, it started with Steve Yegge's Dynamic Languages Strike Back, followed by Cedric Beaust's Return Of The Statically Typed Languages and finally Ola Bini's A New Hope: Polyglotism. Ola does a good job of summarizing the other 2 and then proclaims that the correct answer is somewhere in between, using both static and dynamic languages in different situations.

As a Java developer who knows JavaScript far too well, this is something I've believed for quite some time. I'm going to go one step further and say that I have a vague idea of where the line is where it is appropriate to switch from static to dynamic. If you've done much JEE development (if you're reading this, I know you have!), then you also know too, but you may not realize it.

Think about a typical web application. Say we're using Struts, Spring, and Hibernate, but it really doesn't matter. In the view we have this mix of XML/HTML and some sort of code, be it logic tags or inline code. The type of everything is a string. We produce a big string as the output, all the inputs are strings, we validate strings. We have to do conversion in varying degrees to pass things on to the service layer, but by the time we get to the persistence layer, you better not be passing a String when you really need a Long. Did you spot it? We get virtually no benefit from types in the view, which is why no one uses Servlets anymore. Declaring that everything is a String is kind of pointless, so a dynamic language is a good choice for the view. On the other side, types help us ensure the integrity of our persistent data and let us avoid nasty things like injection attacks. The persistance tier has to be fast and it has to be correct. A UI glitch is annoying, but if we commit corrupt or invalid data, we're in big trouble.

That leaves the middle tier, which is where the blurry line between static and dynamic lies. One option would be to use a language with optional static types like Groovy (i.e., persistence layer is pure Java, middle tier is Groovy, top is JavaScript or Groovy with minimal typing). Another would be to use a library like DWR that takes care of most of the type conversions so you can have a statically typed service later but a fully dynamic view. The JVM really shines here because it gives us so many options. What languages you choose are largely going to be affected by what people know (i.e., what's easy to support) but picking the right tool for the job is important too. I think one thing is certain though, the days where developers can be effective knowing just one language are drawing to a close.


RegExs for Common JavaScript Bugs

I use these 2 quite a bit:

This catches empty object and array literals in another object literal. Chances are you put this in a class/prototype definition like so:

myHash: {},
myList: []

That's roughly equivalent to declaring 'myHash' and 'myList' as static in Java. Unless you really want every instance of that class to share the same hash/array, that initialization needs to happen in the constructor:

initialize: function() {
this.myHash = {};
this.myList = [];

Catches [1,2,] and {1,2,}. The trailing comma causes IE to fail silently, which is always fun to debug. Since this is always wrong, you can change that to ,(\s*[\]}]) and do a find and replace with $1 as the replacement.

Post another regex findable bug if you know one.


A Java Developer's Guide to JavaScript - Functional Programming and JavaScript Mechanics

Homework Check

Last time we introduced the Prototype library. Hopefully by now you've at least looked over the Prototype API and are semi-comfortable with Firebug. If not, stop and take a few minutes to look around and write a short script or two.

Functional Programming

Functional programming is one of those things we don't really do in Java, since Java lacks first class functions. Essentially, when you can pass functions around like any other variable, you often end up sticking them together ("composing" them) to create new functions. The result of one function gets fed into another, etc. Programming effectively in JavaScript requires recognizing that many patterns that are explicit in Java are implicit in a functional language like JavaScript.

There are several 'classic' functions that are usually the basis of any good functional programming library:

A map function takes a value and returns a new value. It's complement is reduce/inject. Here's an example using Prototype's collect function:

var foo = [1,2,3,4,5];
var bar = foo.collect(function(a) {
return a * 2;
}); // bar = [2,4,6,8,10]

Takes two values (at least one is usually a collection, but they both can be) and combines them into one value. When called inject, the first value is usually the 'accumulator' and the second is a singular value. With Prototype, you'll use map/collect and inject on collections, but that's not the only way it is used. Map-Reduce is what powers Google searches. Here's an example:

var foo = [5,4,3,2,1];
var bar = foo.inject(1,function(a,b) {
return a * b;
}); // bar == 120

A little explanation for this one. The first call, the parameters are (1,5), the next call is (5,4), then next is (20,3), etc. The above is how inject works in Prototype, other versions of reduce can pass two outputs of reduce instead of the previous output and the next input. e.g., it might do (1,5), then (4,3), (2,1), resulting in 5,12,and 2, then apply the reduce function on the results and so on until we have a single value. To see how map/reduce powers Google, read Can Your Programming Language Do This?
Used to select/find a subset of a collection. e.g.,
[1,2,3,4,5].select(function(a) {
return a % 2 == 0;
}); // = [2,4]
all and any
Returns true if any or all of the elements in a collection match a given predicate. e.g.,
[1,2,3,4,5].any(function(a) {
return a == 4;
}); // == true

With Prototype, if you do not pass a predicate, then the value is evaluated as a boolean. You could use this to detect if a collection contains all (or any) non-zero, non-false and non-empty string values.
There are other common functions, but these cover the conceptual core. Most other functions will build off of these in some way. As you get to know the common functions, you'll discover uses for them that are extremely powerful.

Manipulating Functions in JavaScript

Before we talk about design patterns, we need to quick take a look under the JavaScript hood and see how we can manipulate the scope and arguments of functions. Remember that when we call a function as a member of an object, the this keyword lets us access variables in the scope of that object. But if you have only a reference to a function and you try to call it, this will refer to the global (window) scope. To allow us to set the scope of a function, we can use call. Here's an example:
var foo = { a: 1};
function aPlus(b) {
this.a = this.a + b;
return this.a;
},5); // == 6 && foo.a == 6
This would be almost equivalent to
foo.aPlus = aPlus;
However, the second example will overwrite aPlus on foo. If aplus happens to be an existing variable or method, we've now overwritten it.

can be used whenever you know the exact number of arguments you'll be passing to a function, but what if you don't? You might say, "When would that happen?" What if you have a function that takes a function as an argument, and returns another function? This is something we do a lot in functional programming, so we need a way to keep our function producing functions DRY.

Before we discuss how you would call a function without knowing the number of arguments, we need to talk about a special variable: arguments. Inside any function, the arguments variable is automatically defined. arguments is an array-like structure that has all the arguments passed to the function. In JavaScript, the parameters you declare fo a function are just convenient labels for indecies into arguments. The JavaScript interpreter doesn't care if you call the function foo(a,b) with 1,2, or 7 arguments. a = arguments[0] and b = arguments[1], and if arguments[1] is undefined, so is b. Like an array, arguments has a length, so you can iterate over it. e.g.,
function sum() {
var sum = 0;
for(var i = 0; i < arguments.length; i++) {
sum += arguments[i];
return sum;
Suppose we also have this function:
function double(a) {
return a * 2;
and we want to create a function that takes the sum of 2 or more numbers and returns twice the result. To reuse our existing functions, we need some way of creating a function that passes all of it's arguments to sum regardless of the number of arguments passed to it. apply to the rescue! Like call, apply lets you call a function and set the value of this, but instead of taking the arguments from the 2nd argument on, it takes an array (or array-like object) as the second parameter. e.g.,
function doubleSum() {
return double( sum.apply(this,arguments) );
doubleSum(1,1,1,1,1); // == 10
doubleSum(1,2,3); // == 12
doubleSum(); // == 0
We haven't made use of Prototype here yet because we haven't needed it. However, you may have noticed that we keep referring to arguments as an "array-like" object. That's because it's not a real Array and doesn't have any of the instance methods you can expect an array to have. Prototype provides a utility function for just this purpose, $A. $A(arguments) returns a true array with all of the arguments in it, so you can call all the standard Array methods and Prototype methods on it. We can rewrite sum like so:
function sum() {
return $A(arguments).inject(0,function(a,b) {
return a + b;
Using apply we can even write call:
function myCall(func,scope) {
var args = $A(arguments).slice(2); // creates a copy of arguments from index 2 on
return func.apply(scope,args);

That covers call and apply. Next time we'll see how we can use what we now know about functional programming and JavaScript mechanics to implement the design patterns we know and love from Java.

Techdirt: The Formula

If you read Techdirt very often, you know that they are the first to catch it when a corporation (or government, to be redundant) is trying to hang on to a business model that is losing relevance. While I often find Techdirt insightful, they too often use the formula "Why doesn't $company stop wasting resources doing $X and do $Y instead?" Where $X is something like filing a lawsuit, which often involves a lawyer on retainer or a few thousand in legal fees (i.e., a drop in the bucket of a corporate budget), while $Y is something like investing millions and billions in building new cable/telephone infrastructure. The implication is that $X is somehow keeping them from doing $Y, and that the two are comparable strategies, which is rarely the case. It would be like me saying: "Instead of whining about the flaws in corporate business strategies, Techdirt should fund startups to compete using their superior ideas."

That said, I like Techdirt most of the time and can generally count on them to be one of the first to break significant tech news, they could just use some fresh material.


jQuery Greasemonkey Template & Netflix Queue Shuffler

jQuery is quickly becoming my weapon of choice to do DOM manipulation/enhancement. I'm still a huge Prototype fan for general code, but jQuery really does the DOM well.

So, when I set out to write a Greasemonkey script to randomize the order of the movies in my Netflix queue, I needed to find a way to include jQuery. The end result is fairly brief:

// ==UserScript==
// @name
// @namespace
// @description
// @include
// ==/UserScript==

* @param {Function} func the function you want to execute
* with jQuery. The only argument passed will be jQuery, so
* your function should look something like this:<pre>
* withJQuery( function($) {
* $(".foo .bar")...
* } );
* </pre>
function withJQuery(func) {
if(!unsafeWindow.jQuery) {
var s = document.createElement('script');
s.src = '';

s.addEventListener('load',function() {
// lots of people like to have their own $ function

} else {

// your code goes here, use withJQuery when you need it

The trick was figuring out that in a Greasemonkey script, you can't do element.oneventname = handler (you have to use addEventListener) and that once jQuery loaded, it would be in unsafeWindow, not window.

If you came here for the queue shuffle script, sorry to have bored you. I'll do another 'JavaScript for Java Devs' chapter soon.


Grails, DWR, and the dynamic web

This post used to be titled "Using DWR with Grails", because the Grails DWR plugin lacked support for custom converters and creators. I wrote this patch (EDIT: Nabble screws up the patch formatting so that it doesn't take. You can download the patch here) adding this functionality and I was going to blog about how to apply and use it. However, before I posted this, I did a bit of searching, and apparently my patch got applied the same day I submitted it. It would have been nice if someone had told me they took my patch, but presumably this means that the next release will have the functionality. If you can't wait, I presume you can build from the trunk, or use my forum post. Enough of that.

I'm not big on making predictions, but this one seems pretty obvious to me. The reason there are so many Java web frameworks is that they all suck at being web frameworks. There is no clear winner in the Java space. We started with Servlets, which were terrible for web development because you couldn't tell what the heck the final HTML would look like. Then we had JSP, which is better, but once you start trying to apply DRY, get into a templating system and tag libraries, you're suddenly just as removed from the final result as you were with Servlets. In both cases, if I want to know what is being produced, I have to fire up my JEE container and do a request. Not to mention that creating templates and tags requires some lovely combination of XML and Java code. I don't know about you, but that feels off to me.

Why? XML doesn't give me much flexibility. DRY gets violated all the time because there is no mechanism in the design of XML for reasonable reuse. Java on the other hand lets me refactor and reuse to my heart's content, but as we've already established, it's crap for actually producing HTML.

And what about Ajax? JSP is fine if your site uses tables for layout and all links result in a full page refresh or occasional popup window, but if you want to build a responsive, dynamic site (using say, Ext) you'll end up having rendering logic in two places, WhateverCrapFramework and JavaScript. The best possible outcome is that you hardly use JSP (or whatever your 'controller' is) and rely on your Ajax toolkits.

Again why? Because static typing everything is not appropriate for user interfaces. A user only has one input type available to them: text. They can't enter FooBarDtos directly, they don't even know what one is. You have to take some text and convert and validate it. Granted, they can also click things, but HTTP seals the deal for the web because all data submitted must be in text form. Your types cannot help you until you have already converted and validated the data, at which point it's ready to pass on to your service layer. So why the heck do you need types everywhere in your controller? You don't, but Java forces them on you anyways.

I think Java is great for web applications by the way. Spring+Hibernate remains my first choice for implementing a solid service and data access layer (I like Guice too). Static typing is great for helping ensure that your services are solid, and your data remains valid.

Back to Ajax. I made a big investment in JSF in graduate school. I nearly killed myself and my wife building a framework for creating Ajax enabled components for JSF. It took me a few months after finishing to realize that JSF still doesn't make Java web development any less like wearing plate mail while figure skating.

When you get right down to it, the best language for developing a web UI is the language that was always meant for the web: JavaScript. DWR completely bridges the gap between Java and JavaScript. It converts all that pesky string data into nice typed data for your Java service layer and then turns around and converts those types that the user doesn't know about into pleasant little strings for web browsers to gobble up. You can expose your Spring service beans directly, secure them with Acegi (you better be doing that anyway, if you're relying on your controller to secure access, you're using a sieve as an umbrella), and call them from JavaScript without even having to know how HTTP requests work, much less Ajax.

In conclusion, we need real developers on the client side. Good designers/UX folk don't usually make the best technical developers, and the dynamic web (especially for enterprise) requires a solid, coherent client side codebase, and things like DWR make that easy. If you're a Java web application developer that's still afraid of JavaScript, you'd better learn to face your fears.

The IE script load event

This is just a short rant. I promise I'll post more about Grails latter.

I really hate IE. Take this code to create a script tag and fire a callback once it's loaded:

var script = document.createElement("script");
script.setAttribute("src", fileName);
Event.observe( script, "load", function() {...});

Works great in Firefox. But in IE6? Never fires. You have to do this instead:

Event.observe( script, "readystatechange", function() {
if(script.readyState == "complete") { stuff...
} );

It's not that there is extra functionality. It's that a decision was made to provide a complicated interface that can handle like two extra edge cases, but the much simpler, obvious and consistent 'onload' handler gets left out.


Custom DWR Converters

DWR is great. If you're full bore into Ajax development, it eliminates the need for a awkward controller like JSF or Spring MVC. You can write static DHTML that talks directly to your service layer, which keeps the UI code clear and simple. Secure your services with something like Acegi, and you're almost there.

I say almost, because the way DWR marshals your beans is either all or nothing. You can tell DWR, convert only these classes, and only these properties and that's it. What if you have different user roles that have different read permissions? e.g., detailed user info should be accessible to the admin and no one else. What if you want summary information when an object is nested inside another one? e.g., my Label BO has a name, description and a couple of flags associated with it. When I retrieve a labeled thing, say a message, I really only care about the label name so I can render a list of labels on the message. But when I retrieve a Label for editing, I need all the information about the label. Let's see if we can fix this.

Because DWR is open source, I can browse around and find the source for BeanConverter. It has a lot of hooks (including isAllowedByIncludeExcludeRules, which would be helpful for role based filtering), but what we really need here is a hook to control what converter is used. Because we have the source code, why don't we add one? I'll just create a subclass of BeanConverter and override convertOutbound like so:

public OutboundVariable convertOutbound(Object data, OutboundContext outctx)
throws MarshallException {
Map ovs = new TreeMap();

ObjectOutboundVariable ov = new ObjectOutboundVariable(outctx);
outctx.put(data, ov);

try {
for (Entry entry : (Set>) getPropertyMapFromObject(
data, true, false).entrySet()) {
ovs.put((String) entry.getKey(), getPropertyValue(
(Property) entry.getValue(), data, outctx));
} catch (MarshallException ex) {
throw ex;
} catch (Exception ex) {
throw new MarshallException(data.getClass(), ex);

ov.init(ovs, getJavascript());

return ov;

The code is pretty much the same as in the superclass, but with 1 key difference. My version calls getPropertyValue. In the template class, I keep the default behavior, loading the converter from the converter manager, but we'll create a subclass in a moment that takes a different approach.

First, let's look at what dwr.xml offers us in terms of configuration. It looks like you can set properties on converter instances using param elements. I prefer to use a concise format for configuration whenever possible, so our format will be something like "propertyName:includeExcludeProperty1,includeExcludeProperty2 otherProp:more,of,the,same". Supposing this was an include, then when we create the converter for the otherProp property, it will include the more,of,the and same properties and no others. Got it?

We need to provide parsing for our custom format in the converter:

public class SubBeanConverter extends TemplateBeanConverter {

protected Map subInclude = new HashMap();
protected Map subExclude = new HashMap();

public void setSubIncludes(String defs) {
this.subInclude = parseIncludeExclude(defs, true);

public void setSubExcludes(String defs) {
this.subExclude = parseIncludeExclude(defs, false);

private Map parseIncludeExclude(String defs,
boolean include) {
Map map = new HashMap();
for (String def : defs.split("\\s")) {
String[] split = def.split(":", 2);
BeanConverter bc = new BeanConverter();
if (include) {
} else {
map.put(split[0].trim(), bc);
return map;

TemplateBeanConverter is our original subclass of BeanConverter. Since we're using the same format for includes and excludes, the same method is used to parse them both. For each property we list, we create a BeanConverter instance that we will use in place of the usual converter.

The last step is to override getPropertyValue and use our special BeanConverter instances:

protected OutboundVariable getPropertyValue(Property property, Object data,
OutboundContext outctx) throws MarshallException {
Object value = property.getValue(data);
if (value != null) {
BeanConverter exConverter = subExclude.get(property.getName());
if (exConverter != null) {
return exConverter.convertOutbound(value, outctx);
BeanConverter inConverter = subInclude.get(property.getName());
if (inConverter != null) {
return inConverter.convertOutbound(value, outctx);
return super.getPropertyValue(property, data, outctx);

Easy right?

Next time I'll talk about my experience using Grails with DWR.

UPDATE: I mentioned role based filtering above, but I will leave that as an exercise to the reader. One hint: if you're using Acegi to control access to pages,
will get you the list of granted authorities (AKA roles, e.g. ROLE_ANONYMOUS) for the current user.