The Curious Schemer

I’ve been reviewing a lot of code lately, and I found so many abuses of eval(), that I wanted to blog about it. But I found that Eric Lippert has done excellent short articles about it right here, and here, and the abuses he blogged about there are exactly the ones I keep seeing in the code I’ve been reviewing. So instead, I’ll blog about something that baffled me a few months back, when eval() simply wouldn’t work when I used it to evaluate my JSON string.

The scenario was simple: we were getting back JSON strings that represent complex objects from the server side. Most of them were arrays that contained objects, and it worked perfectly, like demonstrated in my favourite tool jrunscript below:

js> var jsonEmployees = "[ { name: 'Ray', role: 'Fixer' }, { name: 'Rayzor', role: 'Cutter' } ]";
js> var employees = eval(jsonEmployees);
js> employees.length
2.0
js> employees[0].name
Ray
js> employees[0].role
Fixer

However when the JSON string is not an array, but an object, like this, it barfs:

js> var jsonEmployee = "{ name: 'Ray', role: 'Fixer' }";
js> var employee = eval(jsonEmployee);
script error: sun.org.mozilla.javascript.internal.EcmaError: SyntaxError: missing ; before statement (<STDIN>#1(eval)#1) in <STDIN>#1(eval) at line number 1

What happened? It says it’s missing a “;”. So maybe we need a ; at the end of the object literal:

js> var jsonEmployee = "{ name: 'Ray', role: 'Fixer' };";
js> var employee = eval(jsonEmployee);
script error: sun.org.mozilla.javascript.internal.EcmaError: SyntaxError: missing ; before statement (<STDIN>#1(eval)#1) in <STDIN>#1(eval) at line number 1

Hmmm. Obviously not. But why? I checked and rechecked the JSON syntax in the JSON site, it seemed to be correct. Putting the object inside an array (i.e.: “[{ name: 'Ray', role: 'Fixer' }]“) always works, but why shouldn’t eval() eval what seems to be a perfectly valid JSON syntax? (Actually, strictly speaking, no, it is not a valid JSON syntax although it is valid JavaScript syntax–I realized this later when I incorporated json.js into my app. But that’s not why eval() failed. Keep on reading.)

Ugh, Spec! Expressions! Ambiguity!

Have you ever read a language spec? I hate language specs (especially C++’s). But the answer to our question is buried deep somewhere inside the ECMAScript standard, 3rd edition, section 12.4, about Expression Statement: “Note that an ExpressionStatement cannot start with an opening curly brace because that might make it ambiguous with a Block.“

Ahhh, so when we do this:

js> var jsonEmployee = "{ name: 'Ray', role: 'Fixer' };";

eval() gets confused, because it is interpreting the “{” as the beginning of a block instead of an object literal! Fortunately, there are a few ways to get around this. The easiest one is to add parentheses around the expression. That is:

js> var jsonEmployee = "({ name: 'Ray', role: 'Fixer' })";
js> var employee = eval(jsonEmployee);
js> employee.name + ", " + employee.role
Ray, Fixer

Alternatively, we can make it obvious to eval() that the expression is an object initializer, by assigning it to a variable within the eval string:

js> var jsonEmployee = "var employee = { name: 'Ray', role: 'Fixer' };";
js> eval(jsonEmployee);
js> employee.name + ", " + employee.role
Ray, Fixer

A Better But Picky eval() That Won’t Evaluate My Object Literals

eval() works well when used appropriately. For instance, for the case of my app, using eval() is fine, not evil, because we’re only using it to parse JSON strings that come from a trusted server (that is, our own server). It’s different when the string you’re eval()-ing (JSON or otherwise) comes from some untrusted source. In that case, it’s better to use a real JSON parser.

(Remember that JSON is a subset of JavaScript–a JSON parser only parses JSON, and not the rest of JavaScript. Whereas eval() parses everything. That said, JSON parser is obviously slower since it has to check that only valid JSON can be eval()-ed.)

Using the JSON parser is simple. You don’t even need to add parentheses to get the string parsed. Just download Douglas Crockford’s JSON JavaScript library here, include it in your page (or load it if you’re running in jrunscript), and you’re set.

Note something about this parser though: it is REALLY strict. It won’t parse perfectly OK object literals like { name: “Ray”, age: 31 }. Eval parses this without problems:

js> var jsonRay = "{ name: 'Ray', age: 31 }";
js> var ray = eval("(" + jsonRay + ")");
js> ray.name
Ray
js> ray.age
31.0

But when we use the JSON library, only JSON-compliant strings are accepted. Let’s try it using jrunscript:

js> load("C:\\Documents and Settings\\schemer\\My Documents\\json.js")
js> var jsonRay = "{ name: 'Ray', age: 31 }";
js> var ray = jsonRay.parseJSON();
script error: sun.org.mozilla.javascript.internal.JavaScriptException: [object Error] (C:\Documents and Settings\schemer\My Documents\json.js#270) in C:\Documents and Settings\schemer\My Documents\json.js at line number 270

This string: “{ name: ‘Ray’, age: 31 }” is not valid JSON! Check the specification again. An object has to look like this to be JSON-compliant (properties names have to be quoted, and with double quotations marks too):

{ "name": "Ray", "age": 31 }

And indeed, now the string is parsed correctly into an object. Note that unlike eval(), we don’t need to put parentheses around the object initialiser–the parseJSON() method does it for us:

js> var jsonRay = "{ \"name\": \"Ray\", \"age\": 31 }";
js> var ray = jsonRay.parseJSON();
js> ray.name
Ray
js> ray.age
31.0

The fact that the new json.js adds parseJSON() as a method of String (instead of the old approach of passing the string to JSON.parse()) changes the way you code, because obviously you can’t call parseJSON() on a null or undefined variable. This JSON library also adds toJSONString() method to JavaScript datatypes, so we can do this:

js> ray.toJSONString()
{"age":31,"name":"Ray"}

The Conclusion

Use eval() when you can trust the string. Use parseJSON() otherwise. (I know, I know, it’s a lousy conclusion, like a fizzle after a pop or something. But I’m getting real sleepy, kay? Need to get some sleep.)

We just spent a few days tracking (and fixing) a horrible performance problem in our app. For the longest time everybody thought it was a JavaScript problem, too big a DOM tree, etc., etc. I tried tracking it down with IE Developer Toolbar, with Firefox Firebug (which is lightyears ahead of IE Dev Toolbar, I must say), to no avail.

The problem is like this: we have a few tabs, each of which contains a large table with a lot of columns. The requirement is to get a few first columns (first name and last name columns) of these tables to be sticky, so that when the user scrolls the table horizontally, he can still see whose details he’s editing.

The CSS-styling for the table is quite straightforward:

#tblCont th.sticky, #tblCont th.chkbox, #tblCont td.sticky,
        #tblCont td.chkbox {
    position: relative;
    left: expression(document.getElementById("tblCont").scrollLeft);
    z-index: 99;
}

You set the class of the th and td to be sticky, and voilà, you just made them sticky. They won’t scroll horizontally along with the rest of the columns.

Little did we know how much of a CPU hog a page containing a table styled this way, until we started getting complaints about “typing being too slow”, “tabbing between controls takes 7-10 seconds”, “moving the mouse pointer makes IE 6.0 use 100% CPU”, and so on.

And it’s particularly problematic because nobody thought it was the CSS (not even our web designer, who had come up with this sticky solution in the first place). I spent hours tracking down event handlers for mouseover, keypress, keydown… you name it. Basically the handlers for events that are supposed to be triggered when you type and move the mouse around.

Then my teammate tried removing that part of the styling–and the performance suddenly got a LOT better. I don’t know enough about CSS to give a detailed analysis WHY exactly is this such a performance hog in IE 6.0, but suffice to say, next time I see the same symptoms, I know one more potential suspect to investigate.

We ended up having to split the table into two to make the columns sticky.

Phew!

=== and !==. First time somebody told me about these two JavaScript operators, I thought she was pulling my legs. I was oblivious to these two operators for the longest time–after all, my JavaScript code had been working just fine with == and !=.

It was not until I found Douglas Crockford’s JSLint that I started wondering about these funny operators. Because JSLint just. Won’t. STOP. COMPLAINING. About almost every single use of == and !=. It gives out error messages like this:

Problem at line 3 character 11: Use '===' to compare with 'null'.

if(id == null) {

Very annoying. But I thought, hey, if Douglas Crockford says so, maybe there’s a good reason behind it, no? So I whipped out my trusty JavaScript: The Definitive Guide to find out more about these operators. (Come on, when was the last time you’re learning a new language and read the operators section?)

Here’s the difference between == and === : == is an operator to check for equality, whereas === is an operator to check for identity. What’s the difference? Well, if you come from a language like Java or C#, you know that just because two objects are equal doesn’t mean they’re identical. That’s what equals() is for–to find out whether two different (that is, nonidentical) objects are equal. Likewise,!= is a inequality operator, and !== is a nonidentity operator.

Fine, that was easy enough. But it ain’t helping. So what? Why is JSLint complaining about x == null? What’s the difference between 3 == 3 and 3 === 3?

The Silent Type Conversion

I dug deeper, and I found something that programmers coming from other languages may find surprising. Examples will illustrate this best (I’m using the jrunscript console that I blogged about here):

js> null == undefined
true
js> null === undefined
false
js> 5 == "5"
true
js> 5 === "5"
false
js> "true" == true
false
js> 1 == true
true
js> "1" == true
true
js> 1 === true
false
js> "0" == false
true
js> "0" === false
false
js> var x = { id: 123, name: "Blah" };
js> var y = { id: 456, name: "Not Blah, definitely" };
js> x == "[object Object]"
true
js> y == "[object Object]"
true
js> x == y
false

So you see that == (and != too) do type conversion under the hood before testing for equality. This may give some very surprising result, such as this:

js> 1 != true
false
js> 0 != false
false
js> 5 != "5"
false
js> var x = new Object();
js> x != "[object Object]"
false

Ahhh so. I finally figured out why JSLint was so fanatical in complaining about == and !=. Because JavaScript does type conversion under the hood when it checks for equality, which may surprise the heck out of programmers from other languages! We can find what the conversions are from ECMA-262 specification… but why? David Flanagan has done it for us in JavaScript: The Definitive Guide. Specifically, the type conversions are:

1. null and undefined are equal
2. if a number is compared to a string, the string is converted to a number, and the comparison is done again
3. true is converted to 1, false to 0
4. if an object is compared to a number or string, the object is converted to primitive, either by calling its valueOf() or toString() method.

Both === and !== don’t do any of these conversions.

Know What You’re Doing

So what was Douglas Crockford trying to say when his JSLint balked at the use of == and !=? (The latest version seems to be more relaxed in detecting this.)

I believe his point is that we should know what we’re trying to do. Consider the following example:

function doSomething(id, name) {


if(id == null) {

alert("id is mandatory");

}

}

Do you really want to check whether id is null, and do something different if id is undefined? Or you don’t really care whether id is null or undefined, as long as it has some values? If it’s the former, you’ve gotta use ===. For the latter you use ==.

Another example is comparing the contents of an HTML text box with a number. Let’s say you have a textbox containing a person’s age. With ==, it’s convenient:

if(document.getElementById("age").value == 20)

Using === will be troublesome in this situation. So the moral of the story is: really know what you want. The silent type conversion is there to help you, to make things easier for you. But you have to know what’s happening under the hood so you won’t get nasty surprises for cases in which you really mean === (or !==).

You know how sometimes you’re exploring a new Java API, and you don’t want to create a class (public static void main blah blah), compile it, (or create project if you’re using an IDE), typing System.out.println to see the output, just to find out how that API works? I really hate that. Especially if it’s a 3rd party library

Java SE 6.0 in this regard. It makes it really easy to explore an API without having to compile a class just to see how it works.

Trying Out Regular Expressions

For instance, let’s say we want to escape all single quotation marks, double quotation marks, and backslashes in a Java String (say because you want to pass it to JavaScript). A Perl guru can probably write the regex in 3 seconds, but my regex is rusty. I want to be able to try it out, and then paste the result into my code with just a few modifications.

Since I have Java 6.0, I just have to start jrunscript. By default it gives me a JavaScript console. Now it’s easy to try out different regular expressions:

js> var s = new java.lang.String("Testin' this thing: Test' test \" test \\ ");

The objective is to add an escaping backslash in front of ‘, “, and \ within the string. First trial:

js> s.replaceAll("[\\"'\\\\]", "\\$1");
script error: sun.org.mozilla.javascript.internal.EvaluatorException: unterminated string literal (<STDIN>#1) in <STDIN> at line number 1

Hmmm. That didn’t go so well. Oh, too many backslashes for the double quotation marks! Second trial–I just pressed the up arrow, edit the expression, and press Enter:

js> s.replaceAll("[\"'\\\\]", "\\$1");
Testin$1 this thing: Test$1 test $1 test $1

Meh. Third trial:

js> s.replaceAll("[\"'\\\\]", "\\$0");
Testin$0 this thing: Test$0 test $0 test $0

Ah, right, the $ needs to be escaped:

js> s.replaceAll("[\"'\\\\]", "\\\\$0");
Testin\' this thing: Test\' test \" test \\

Sweet! No class, no compiling, no System.out.println, just trying immediately the API you want.

Trying Out New Third-Party Libraries

Java’s Calendar and Date are quite… emasculating. I believe that using them too much shorten your lifespan and in all probability has disastrous effect on your sex life. Self-respecting developers who care about sex use Joda Time. Joda Time has been submitted as a JSR, and I’m wishing it every success. May java.util.Date and java.util.Calendar die a horrible death ASAP.

So, let’s say you’ve downloaded Joda’s JAR and would like to explore. Just run jrunscript this way (obviously change the location of the Joda’s JAR accordingly:

jrunscript -cp C:\apps\jars\joda-time-1.4.jar

and import the package:

js> importPackage(org.joda.time);

and we can start exploring right away:

js> var now = new DateTime();
js> now
2007-03-17T09:14:43.656+08:00
js> var oneMonth = new Period().withMonths(1);
js> oneMonth
P1M
js> now.plus(oneMonth);
2007-04-17T09:14:43.656+08:00
js> now.plus(oneMonth).plus(new Period().withDays(31));
2007-05-18T09:14:43.656+08:00

It’s easy to get all the static members of, say, Period:

js> for(m in Period) { println(m); }
days
months
millis
.
.
.

and the instance members:

js> for(m in (new Period())) { println(m); }
getMonths
withMinutes
withFieldAdded
.
.
.

Nice! Trying out Swing also becomes real easy with this. Frankly my Java development is all Web these days, though.

Don’t you hate JavaScript sometimes? You may miss a comma or a semicolon somewhere, and the error message that you get is something like “‘obj’ is not defined”, and you’re staring at the code and you know that that freakin obj is there, so how come it keeps insisting that it’s not there?

Enter JSLint. This is very useful JavaScript lint utility, written by Douglas Crockford–one of the most well-known personalities in the JavaScript world. This cool utility is very accurate–most of the time it can pinpoint exactly where the mistake is. Other than that, it also points out the best practices of JavaScript coding. Since I’ve found this utility, I never commit my JavaScript files before running them through it.

Check it out here: http://www.jslint.com/