Original Whatever

OMG you put Java code in the view!!

2012-02-11T09:55:00.000-08:00

One of the unique joys of developing anything with Java Server Pages is encountering the Scriptlet Police.

Scriptlets, as you probably know, are little bits of Java code embedded into the JSP. I think of JSPs as being servlets turned inside out. In a servlet, everything is Java by default, but you can throw in some HTML in the form of strings. In a JSP, on the other hand, everything is HTML by default, but you can throw in some Java in the form of scriptlets, like this:

<ul>
  <% for (Foo foo: listOfFoos) { %>
    <li><%= foo.getName() %></li>
  <% } %>
</ul>

But God forbid you do this and then admit it on a mail group or message board. Any hint that you're actually putting Java code in your JSP will attract a lecture from the Scriptlet Police, for example:

Question: "I am a complete JSP beginner. I am trying to use a java.util.List in a JSP page. What do I need to do to use classes other than ones in java.lang?"

Helpful answer 1: "Not to mention that this is not the best practice ;)"

Helpful answer 2: "FYI - if you are importing a List into a JSP, chances are pretty good that you are violating MVC principles."

and

Question: "I wish to access [a model attribute] in a scriptlet (within a JSP)."

Helpful answer: "It's stored as a request attribute. ... Unrelated to the concrete problem, I suggest to ask a new question wherein you ask how to achieve the functional requirement without the need to fall back to legacy practices."

The irony of course is that when the members of the Scriptlet Police write their own JSPs, they write something like this:

<ul>
  <c:forEach items="${listOfFoos}" var="foo">
    <li>${foo.name}</li>
  </c:forEach>
</ul>

Which of course is the exact same thing, just written in a slightly different "language" consisting of JSTL tags and JSP expression language (EL) expressions. So why is it verboten to write a loop as a scriptlet but not using JSTL?

If you're using Groovy and Grails, it becomes even more ironic. The code looks about the same, although the tag is called <g:each> instead of <c:forEach>. But the big difference is that the code inside the ${} is Groovy code, not some language that's been deliberately watered-down so as to make it difficult to implement "business logic" in the view.

The members of the Scriptlet Police think that they're enlightened. But in fact, they are confused. Someone told them a long time ago that they shouldn't put any "business logic" in the view, and somehow that got associated with "no Java in the view," because obviously Java is what you use to implement business logic, right? So in their minds, they are keeping business logic out of the view by only using JSP tags and Expression Language in their JSPs. In fact, instead of keeping business logic out of the view, they're only keeping Java out of the view, which isn't the same thing at all. But it's a simple criterion to apply (Expression Language good, Java bad) and enables them to avoid actually thinking about the difference between the view and... well, what?

What, because it's inevitable that there is going to business logic in the view. Applications exist in order to implement business logic. If you format one number as a currency and another as a percent, you're implementing business logic. If you only display the "Delete" button when the user is an administrator, you're implementing business logic. If you have hundreds of lines of JavaScript on your page, it's all but guaranteed that you have some business logic in there. If you have an N-tier application, you're going to have business logic in every single tier. So what's the point of all this worry about business logic in the view? What are we really trying to accomplish?

What's important isn't keeping business logic out of the view, but rather establishing some useful separation of responsibility among components. JSP is a view technology, so ideally we should limit its use to the view, and implement non-view-related logic elsewhere. For example, entitlement checks and form-saving logic are probably best implemented in a controller written in Java. If we tried to implement form-saving in JSP, we'd wind up with a JSP that had lots of logic paths and performed different functions depending on whether it was being called with GET or POST. It would no longer be a view and would be harder to understand and maintain than a simple view with two different Java controllers.

But as long as we limit ourselves to implementing view-related logic in the view, it doesn't really matter what language we use. Go ahead and write some scriptlets. Normal programming best practices still apply, of course; don't go cutting and pasting the same scriptlet into a dozen different JSPs. But don't worry that the scriptlets will pollute your view with "business logic." If a scriptlet is the best solution to your view problem, just do it.

Alex Papadimoulis also wrote about embracing business logic on his web site The Daily WTF.

Scala and Kotlin

2012-01-26T05:04:00.000-08:00

I've been spending the last three years writing code in Scala. In general I've been pretty happy with the language. It's much more compact than Java, and its higher-level features, such as closures (which have been around forever but which were popularized, I think, by Ruby), enable me to discard several of the Java tools and frameworks that I've had to drag around. Scala also comes with a very powerful collection library that I use on just about every line of code.

Lately I've been following Kotlin, which looks very promising. I'm old enough to realize that this may just be a case of the grass looking greener on the other side of the fence. Every programming language has its flaws, and Scala is no exception. There's always the sense that maybe the next language will be The One. The One that does everything perfectly. The One that incorporates all that is good about Java and Scala and C# and C and C++ (well okay, maybe not C++), introduces exciting new concepts, yet has no egregious flaws (like green threads, or no reflection, or different-just-for-the-sake-of-being-different keywords, etc.), and for that matter, no minor ones either. A language the bridges the divide between OO and functional, and between OO and relational, and between functional and relational, and between relational and non-relational, and... Well, you get the idea.

Back to Kotlin. I'm hoping that Kotlin will give us most of the benefits of Scala, but without some of Scala's more unusual features.

Consider operators. C++ offered operator overloading, which meant that every math geek could now write "m1 * m2" to multiply two matrices instead of writing (in C) "matrixMultiply(m1, m2)". It also meant that the C++ designers could create a completely new I/O library, in which you don't write data or read it but rather shift it into or out of streams. You can even shift formatting functions into a stream to change its state! Unfortunately they declined to give this new I/O library the obvious name, Shift Happens. But what can I say, I wasn't on the team.

Anyway, with C++, you were limited to overloading a subset of the operators supported by C++. You could overload "+" and "-" and even "," (the comma operator, which most folks don't even realize is an operator), but not "." (the member selection operator) and "::" (scoping operator). There were lots of advantages to sticking with the existing operators. No changes were required to the parser, which must have been a major consideration for Bjarne Stroustrup, the guy who had to write the damn thing. Nor did Stroustrup have to worry about how random new operators might fit into the precedence table. The existing operators also had certain meanings, and so when used judiciously, could actually improve code readability. While "matrixMultiply(m1, m2)" has a fairly obvious meaning, "m1 * m2" is better, and shorter too.

Of course people often did not use operators judiciously, and James Gosling decided to leave the whole business out of Java, and that's where things stayed for about 10 years.

The along came Scala, which eliminates the distinction between methods and operators entirely. Method names can have any character in them. Literally any character. If you want to call your method ";" (normally the statement delimiter in Scala), you can do it:

def `;` { println("Don't try this at home") }

You have to enclose it in backquotes every time, so that Scala know you're calling your stupidly-named ";" method and not trying to end a statement. But it's possible.

Naturally this leads to some questions about how exactly these strangely-named functions should work.

If the programmer writes "a*b," how does Scala know that's "(a) * (b)" and not a method called "a*b"?: The method name can't be completely arbitrary. It must follow a prescribed format. If it starts with a letter, then it cannot include a non-alphanumeric character other than "_" unless those characters follow a "_". So a method called "a*b" is not allowed, but "a_*" is okay. If a method name starts with a non-alphanumeric character, then all of the characters must be non-alphanumeric. So "*" and "**" are both okay but "*a" is not. (However, I should note that these rules only apply to method names not enclosed in backquotes. If you are willing to use backquotes, then anything goes.)
How to the precedence rules work for unusual operators?: Precedence is determined by the first character of the method name. Except when it isn't; Scala requires some special rules so that precedence works as expected. If the operator name ends in "=" then it gets the precedence of "=". Unless it's "<=" or ">=" or "!=" or also starts with an "=" (like "==").
If I define an method called "-", is that the infix subtraction operator or the prefix negation operator?: Infix. To define the prefix negation operator, you have to create a method called "unary_-". Also, the only prefix operators that are allowed are the usual suspects: "+", "-", "!", and "~".
If I define a method called "++", is the pre-increment operator or the post-increment operator?: Okay, actually this is a trick question. Scala doesn't recognize the "++" and "--" operators as such. You can define a "++" method, but it won't do what you expect, that is, assign back to the variable on which it's called. To increment you must use "x += 1".

Method names ending in "=" are special. If you write "a += 3" and there is no method called "+=" on "a" then Scala will try to interpret the expression as "a = a + 3". This only works if the method name starts with a non-alphanumeric character that's not "=".

By far the strangest aspect of Scala's method-operator unification is that if the name of the method/operator ends in ":" (a colon), Scala invokes the method on the object to the right of the operator rather than the left. So while "a*b" means "a.*(b)", "a*:b" means "b.*:(a)". As far as I can figure, this was done solely so that the "::" operator could be used to build singly-linked lists.

Now I have to admit, when I first read the Scala language specification, I thought that method/operator unification was pretty cool, even with the quirky bits described above. But after a few years, I'm not so sure.

Obviously, method and operators cannot be completely unified, at least not in a language in the C family tree. People want precedence to work the way it does in C, more or less, and also want certain conventions retained (like += performing an assignment), so Scala has to recognize a lot of special cases. And Scala had to do away with the increment and decrement operators, which just didn't fit into the generalized model.

But the biggest problem with Scala's strategy is that non-standard operators have no implicit meaning. When we see "+" in the code, we read that as "plus," and we expect it to do act something like addition, or concatenation, or union. We would be very surprised if it, in fact, performed multiplication, or closed a file. That's why operators work: they are symbols that identify specific concepts or specific operations.

But what does the "/:" operator do? Would you have guessed "fold left?" That is what it does, but of course only if the object on the right is a Scala collection. If it's something else, then who knows, it's whatever that object defined. Anyone who's writing in Scala is probably using the collections constantly and has probably reconciled with "/:".

But what if you're writing something with Dispatch, an HTTP client library for Scala? Dispatch implements a typical Scala "domain-specific language," or DSL. In the context of Scala, a DSL is a block of code that looks nothing like Scala and is nearly impossible to read, yet somehow conforms to Scala's syntax rules and produces some desired effect. Let's say you wanted to parse the headers from the HTTP response. You might go looking for a method called parseHeaders or getHeaders or something like that, but you'd be out of luck. The method that introduces a header handler is called ">:+". How do you even pronounce that?

Getting back to Kotlin again. Kotlin takes a much more pragmatic approach to operator overloading. When designing Scala, Martin Odersky recognized that making "==" be the equality method wasn't going to work, because Java had already established that the equality method was named "equals." So in Scala, the "==" operator actually invokes the "equals" method (and cannot be redefined). Kotlin just extends this concept to the other operators as well. The "+" operator invokes the "plus" method, "%" invokes "mod," etc. Notably, the comparison operators "<", ">", etc. invoke compareTo, in keeping with Java's Comparable interface.

In taking this approach, Kotlin loses some flexibility. You can't define a ">:+" operator in Kotlin. But that isn't such a loss, and in exchange, you get some potentially more-useful features that Scala can't support. For example, Kotlin maps ".." to the "rangeTo" method, and maps "[]" to the methods "get" or "set" depending on context, and supports the increment and decrement operators. I think that Kotlin's approach will prove more useful practical over the long run.

No-NoSQL

2012-01-19T05:06:00.000-08:00

Anyone who's been around for a little while knows that the idea of storing data outside of a relational database is hardly new. Unix, for example, stores essentially all of its data in flat files, often with keys. Have you looked at your /etc/passwd file lately? It's just a little database, and it certainly doesn't use SQL.

So I'm confused as to why the idea of a non-relational database is being treated by so many people as a novel idea. It's as if a generation of developers wrote a few web systems using the LAMP stack, noticed that MySQL wasn't ideal for everything, and concluded that the whole relational model was at fault and needed to be replaced. Is this increased attention simply due to the fact the this collection of non-relational data stores now has a name?

I think that we've all worked with a relational database that is big and slow. In my case, that would describe maybe half the databases that I've encountered. But those databases aren't big and slow because the relational model is bad. They're big and slow either because they aren't designed very well or because they are being used for something that's not a good fit for the relational model.

The fact that there are some relational databases that are slow and some use cases that don't call for a relational database doesn't mean that the relational model is inherently flawed. It just means that it's not the right solution for everything, which is something that can be said about every technology out there. Every technology is a compromise because, as is the case with almost everything that exists in the real world, optimizing the design to improve certain characteristics inevitably degrades others.

Even a cursory review of the available NoSQL databases strongly reinforces the notion that they are not overall better than relational databases, just different. They have been optimized for different use cases, and as a result, they are better-suited for some use cases than regular relational databases, but less suitable for others. And the optimizations tend to follow a pattern: almost all of the NoSQL offerings trade off functionality for speed. I can understand the motivation behind that, since relational databases are most often criticized for being slow. But it's important to understand what's being traded. MongoDB, for example, does not support multi-document transactions or queries. By omitting support for those features, MongoDB is able to deliver better performance for the features it does support. If you're basically storing independent documents and don't need to relate them in any way, then great. But if you want to make an atomic multi-document update at some point, you'll be out of luck. You'll have to roll your own transactions with version numbers or lock flags. VoltDB also trades off flexibility for speed. It's way faster than any traditional relational database, but that's because it runs entirely in memory and doesn't support locking or concurrency of any kind. If you're working with a relatively small data set and are only executing fairly simple queries, then you're golden. But it also has limitations that general-purpose databases don't.

I think it's also important to keep in mind that relational databases didn't spring up from nowhere. There were non-relational data stores like IMS long before there were relational databases. Relational databases were developed in response to the perceived flaws of those earlier database technologies. We've been here before.

So when I need to store some data, I'll probably think relational first. But since I might encounter a use case that would benefit from a different type of data store, I'll be happy to have NoSQL in my toolbox.

An Ode to My Macs

2012-01-16T20:08:00.000-08:00

About 18 months ago, Sara and I bought new Macs.

I know it was about 18 months ago because I just finished paying for them. Apple and Barclaycard were offering 0% financing for 18 months on purchases over some amount. An amount we could have reached with just one Mac, but since we were both feeling limited by our computers for various reasons, we decided to get two.

These two Macs are, without a doubt, the nicest and most useful computers I've ever bought. I suppose some of that is due to their being the newest, but there have been plenty of computers that I've bought new but that have merely been okay. That includes most of the Windows PCs that I've bought over the years.

The problem with all the Windows PCs that I've ever owned is that they've all been beige boxes. Sometimes they haven't been strictly beige, but they've all had a certain plasticky, generic feel to them. They've all come with generic keyboards, generic mice, generic power cords. Many of them have booted to a desktop overflowing with crudware. (John Dvorak once pointed out how absurd it was to spend a thousand dollars or more on a computer and then have to immediately run a program called "PC Decrapifier" on it just to get it into a usable state.)

I've had four Macs prior to these two--two laptops (including one PowerPC machine) and two iMacs, and they've all been quite good, certainly way nicer than any of the PCs I've owned. But I found the white MacBook a little underpowered, due to its relatively slow processor (it was an original Core Duo) and small screen, and the 24" iMac was limited to 4G of RAM, which was fine for a while, but once Sara started doing more photography and keeping Lightroom open all the time, our inability to increase the RAM became a real problem.

My new machine is a 15" MacBook Pro with a 2.66GHz Intel i7 dual-core processor, 8G of RAM, and a 128G SSD. For the first time in my life, my primary computer is a laptop. It's a beautiful machine and works flawlessly. The SSD is amazing; the system only takes about 10 seconds to boot, and programs open almost instantaneously. The memory is sufficient to keep all the programs I'm likely to use open at once. The battery lasts an exceptionally long time considering that my usage tends to be processor- and disk-heavy (compiling and running programs). The screen is gorgeous and has tons of contrast.

Sara's new computer is, for the first time, substantially more powerful than mine. She has a 27" iMac with a 2.8GHz quad-core i7, 16G of RAM, and a 1TB hard drive. Not even Lightroom can slow down this machine. The 27" screen is absolutely amazing.

But despite the enormous screen, the machine just doesn't take up that much space. It's a big screen, but that's it. Ever since I bought my first iMac (this is the third), I've been so happy not to have a big ugly box sitting on the desk or on the floor with a bunch of wires everywhere. And speaking of wires, this iMac has exactly one: the power cord. The keyboard and mouse are both wireless. And because this is an iMac and not a beige-box PC, the plug on the computer end of the power cord is specially designed to fit into a receptacle in the back of the computer such that nothing but the cord itself is left sticking out. The folks at Apple care how the back of the machine looks.

And the iMac is quiet. It's normal for a laptop computer to be quiet, especially one with an SSD. Their low-power processors don't generate a lot of heat in the first place, and spinning fans to dissipate it takes energy. The iMac is plugged into the wall and could spin fans 24/7 if it wanted. But it doesn't, because it's been designed to work as a little convection heater: heated air flows up through the case and out a vent at the top, while cooler air gets pulled in through a grille at the bottom. Most of the time the iMac is silent save for the slight whir of the hard drive. If I start doing lots of work in Lightroom, then the fans will spin up, but even then the noise is not obtrusive, and in any case I'm obviously doing something with the computer; it's not like it's making noise while I'm trying to watch TV, which is something that always irritated me with my Windows PCs.

Anyway, thanks to Apple for producing such great gear. Sara and I will enjoy these two Macs for a long time.

Best Album Closers

2009-05-31T04:51:00.000-07:00

I'm not a musician, but if I was one, I think that I'd want each album to stand on its own as a complete work instead of just being a random bag of singles. I'd want people to know the album ("Sgt. Pepper") and not a few songs from it ("the album with A Day in the Life on it"). I'd want to be recognized for having excelled at the difficult task of putting together an hour or more of music that people want to hear. Not to mention that being known for making great albums makes it more likely that those albums will remain available instead of disappearing as the singles are rolled up into "Greatest Hits of the 21st Century" collections.

By the way, this idea of an artist either being known for complete works or for singles is hardly new. Bach is known for complete works. Other composers are forgotten but for an entry in the encyclopedia and one or two pieces in the local symphony's "An Evening of Baroque" program (the classical music equivalent of those "Greatest Hits" collections). Search for Jean-Joseph Mouret on iTunes and you'll find dozens of performances of the same two-minute piece of music, which you might recognize as the theme from Masterpiece Theater.

I think that the last track of an album says a lot about how the composer approached the work. One random-bag albums, the first track is usually the strongest song, so as to pull in people who are only giving the album a 30-second preview, and the last track is usually nothing special, since nobody is expected to listen to the album all the way through. But when the composer is taking the album as a complete work, the last track is important because it's the last thing that listeners will hear.

So I like albums that have a good closer. I've put together this list of them. Please comment and add any that you know, so I can check them out.

Billy Joel, Piano Man (Captain Jack)

The Beatles, Sgt. Pepper's Lonely Hearts Club Band (A Day in the Life)

The New Pornographers, Twin Cinema (Stacked Crooked)

Bruce Springsteen, Born to Run (Jungleland)

Foo Fighters, The Color and the Shape (New Way Home)

Coldplay, A Rush of Blood to the Head (Amsterdam)

Rush, Signals (Countdown)

Jackson Browne, Running on Empty (The Load-Out/Stay)

Nirvana, Nevermind (Something in the Way)

The Shins, Wincing the Night Away (A Comet Appears)

Oasis, (What's the Story) Morning Glory? (Champagne Supernova)

Tori Amos, Under the Pink (Yes, Anastasia)

Here are some albums that have the title track in the last position.

Sarah McLachlan, Fumbling Towards Ecstasy

Sinead O'Connor, I Do Not Want What I Haven't Got

Stars, In Our Bedroom After the War

Steely Dan, The Royal Scam

Tori Amos, Little Earthquakes

How to make a YouTube video that no one will want to watch

2009-02-16T05:06:00.000-08:00

Let's say you want to make a YouTube video that no one will want to watch. How could you do that?

First, make sure that your video has lots of loud, obnoxious music, especially if your video depicts something that normally has nothing to do with music. People might like your video of your dog and cat playing on the floor together, so why not ruin it by replacing the sound with Ted Nugent? For extra points, dial the gain up until it distorts. It's important to choose the right music, too. You don't want something with a quiet intro. Ideally you want to lure people into clicking on your video, then hammer them with decibels so that they jump out of their seats and scramble for the volume control. As a side benefit, you'll help the economic recovery by exposing people who are browsing YouTube at the office when they should be working.

You also want to go for the worst quality possible. Find lots of videos that are already low-quality, then re-encode them at an even lower bit rate. Twice! You can also use this opportunity to screw up the aspect ratio: if the original video was wide, then squeeze it so it's narrow, or vice-versa. If you quickly cut between dozens of low-quality clips, then the viewer will never know what's happening!

Better yet, don't even use video. Just make your "video" a long slide show of still photos. Programs that generate slide shows, like iPhoto, often add some hokey pan, zoom, and transition effects, so remember to shut those off, since they may make the slide show (barely) watchable. Of course, just because you've successfully tricked the user into watching a slide show when they wanted a video doesn't mean that you should neglect the other points. You should still include inappropriate, high-volume music and only select low-quality photos for your show.

Using still photos instead of video works because it makes the user feel duped, which is a fantastic turn-off. You can achieve the same effect by using your video's thumbnail image and description to mislead visitors. Everyone knows that YouTube generates the thumbnail image from the middle of the video, so put a high-quality photo of a swimsuit babe on the screen for a few seconds, right at that point. For the description, enter: "This video is all about pussy!" Won't they feel fooled when they click on the video and get to watch your amateur reporting from the local cat show!

Another way to make users pay for clicking on your video is to use half the video for your own stupid title and credits sequence. There are many ways to drag this on. Users already know what your video is called, but that doesn't mean that it shouldn't be on the screen or five or ten seconds anyway. Lots of moving-text effects can waste a lot of time too. Have the title scroll in from the bottom, Star Wars-style. Then plug your web site, make some call outs to your friends, identify the video to which you're "responding," and give credit to the artist who unknowingly provided the sound track without compensation from you. There's no reason not to take as long as possible with the title sequence. Storage and bandwidth are cheap!

Now maybe you're thinking that this all seems too hard. It's tough to gather together all that loud music and all those crummy still photos and put it all together into a video with a long title sequence. Fortunately there's another way to make crappy, boring videos on the cheap. All you need to do is mount your camera on top of your monitor (iMac users can just use the built-in camera), look straight into it, and talk forever about something that's incredibly boring. Include lots of ums and ahs so we all know that you're not used to public speaking and will never be able to work in broadcasting. And provide your viewers with long-winded descriptions of things instead of actually using the camera to show them. Just because this is video doesn't mean that you can't treat it as radio!

The slow march of technology

2009-02-07T13:34:00.000-08:00

I had some dental work done recently. I had crowns put on four teeth in the back of my mouth, two on either side. Since crowning teeth is a one-way trip, I did some research before I went forward with it. While reading about crowns, I also found information about dental implants. Implants are fake teeth, but unlike old-fashioned bridges or dentures, which receive support from the patient's other teeth and gums, implants are actually screwed into the patient's jaw. The actual tooth that extends above the gum line is typically made of porcelain and is indistinguishable from a natural tooth.

If you think about it for a minute, implants are pretty amazing. They're completely artificial aftermarket parts for your body that work at least as well as the original equipment, and unlike natural teeth, they can't decay. Losing teeth will always be a drag, but now you can have them replaced.

When I was growing up in the 1970s, I read a lot of books and magazines that made predictions about what life would be like in the near future. There would be moon bases, and artificial intelligence, and supersonic airplanes cruising through the sky, and flying cars, and abundant cheap energy from nuclear fusion.

Obviously pretty much none of that came to pass, but it's not hard to fault the writers of that period for making such predictions. The world had changed drastically in their lifetimes. They witnessed the arrival of television, the proliferation of the automobile and the construction extensive road networks (including the interstate highway system), the breaking of the sound barrier and a reduction in intercontinental travel time from weeks or days to hours, and new skyscrapers reaching ever-greater heights, and the development of the atomic bomb and nuclear power. The United States was landing human beings on the moon. It was reasonable to think that airplanes would keep getting faster, that the scientists who mastered fission would figure out fusion, that buildings would continue to climb higher, and that that human beings would travel to farther planets and perhaps even to other stars.

As it turned out, there were practical issues that got in the way. Supersonic aircraft are loud and inefficient. Fusion turned out to be really hard. Mile-high skyscrapers are disproportionately expensive. Moon bases are useless, and other planets are just too far away. And civilization in general turned away from the industrial-age quest to build bigger, taller, faster. Instead, we spent the next decades focused on information and communication. We're not traveling to the stars yet, but if you're reading this, you almost certainly have your own Star Trek-style personal communicator that you can use to reach pretty much anyone in the world from anywhere you happen to be.

However, something that I've noticed over time is that people are still working on the bigger, taller, faster stuff. It's showing up. It's just taking a little longer.

Something that you may not realize is that for the past eight years, without interruption, human beings have been living and working in an orbital space station. Spacecraft launched from, well, let's call them spaceports, in Florida and Kazakhstan take new people and supplies up to the station and bring returning folks home. As of this writing, these spacecraft have made a total of 77 trips to the station and back. The space station has undergone continuous expansion and now looks like something out of a science fiction movie. Really, have a look at it. But it's real. That's pretty amazing. As for the science that goes on there, we don't hear too much about it, and maybe it isn't going to change anyone's life. But the research isn't the story; the story is that launching people into orbit and keeping them there for months at a time has become routine.

Nobody's built a mile-high skyscraper yet, but there's one that's a half-mile high called Burj Dubai. It dwarfs by a considerable margin anything previously built by human beings, neatly ending all the debates over whether it's more important to have the tallest building or tallest "free-standing structure" or tallest structure of any kind and whether or not antennas count toward building height. It's made of reinforced concrete instead of steel, so it doesn't have the problems with flexing in the wind that a supertall steel building would have. It's only recently that engineers have been able to build anything even approaching this height using concrete, but it's the future of construction. And Burj Dubai isn't an isolated case of extreme construction; in recent years, supertall skyscrapers have been built or at least started in Taiwan (Taipei 101), China (Shanghai World Financial Center), Malaysia (Petronas Twin Towers), Russia (Russia Tower), Chicago (Chicago Spire), and New York (Freedom Tower). Saudia Arabia is planning the first true mile-high skyscraper, and Kuwait is building a Burj Dubai-class structure. Could people eventually start building gigantic archologies, buildings housing entire neighborhoods or even cities? We're a long away from that. But at least after a half-century, the 100-story barrier has been decisively broken.

On the nuclear energy front, scientists have been diligently chipping away at the engineering challenges of fusion power. Just last year work was completed on the first fusion reactor engineered to generate more power than it consumes. The reactor, which has a name that only a bureaucrat could love--the National Ignition Facility--can only generate fusion power in bursts, but is nevertheless an important milestone on the road to commercially viable fusion power. Meanwhile work has begun on another reactor, one with an equally-bureaucratic name: International Thermonuclear Experimental Reactor, or ITER. ITER is designed to be an honest-to-God fusion power plant, producing 500 megawatts of power for up to 1,000 seconds. It won't actually generate electricity, but if the engineers can keep the reactor going for 16 minutes, it's easy to imagine the next reactor going for 16 hours or 16 days.

Something that all of these projects have in common is that they're all money-losers. The ISS, NIF, and ITER are government-funded research projects and therefore are expensive and wasteful almost by definition. The Burj Dubai is ostensibly a commercial enterprise, but it is clearly a monument to excess built at the height of a real-estate boom in a country where the line between private and government funding is blurry. It remains to be seen whether it will ever able to produce a return for its investors.

But on the other hand, the Internet was originally a government-funded research project too. Governments also used to be the only players in the rocket-launching business, and now there are several for-profit corporations that will put your satellite in orbit for you. So maybe we're just in an engineering lull. After all, when people watched Star Trek in the 1960s, they probably didn't think that inside of 50 years they'd have personal access to communication equipment that puts anything on the Enterprise to shame. (Captain Kirk's communicator can only reach his other Enterprise crew members. Lame.) Perhaps 50 years from now, people will be thinking, wow, that Mr. Fusion power plant from Back to the Future looks so clunky compared to what we have now!

Hibernate: Using proxies when classes have final methods

2008-12-13T06:08:00.001-08:00

If you use Hibernate, then you have probably encountered proxies. Hibernate generates and returns proxies for entities when (a) Hibernate knows the primary key of the entity, (b) it's not necessary to actually retrieve the entity from the database right away, and (c) the entity class is proxyable. Entities accessed through the Session.load method and through lazy references from other classes are all eligible to be proxied.

The Hibernate documentation states: "... you may not use a CGLIB proxy for a final class or a class with any final methods." But this statement seems a little doubtful, since every Java class has final methods, because Object itself has final methods. Obviously Hibernate and CGLIB are able to create proxies for classes with at least some final methods. But maybe CGLIB has special handling for the final methods of Object?

Actually its seems that classes with any number of final methods can be proxied. However, CGLIB cannot override the final methods in the proxy, so those methods may not work very well, and calling them will not initialize the proxy. But if the methods do not depend on any fields of the class, then they will probably be okay. (Fields defined in the entity class will be present in the proxy but, except for the ID field or fields, will not be initialized.)

Blurring thumbnails: ConvolveOp gotchas

2008-08-27T19:31:00.001-07:00

As part of my task of blurring thumbnail images, I wrote some code to use ConvolveOp. ConvolveOp is basically a filter that generates a new value for each pixel of an image by mixing the value of that pixel with the values of the pixels around it. The mixing depends on a "kernel," which is a matrix of weights. The ConvolveOp filter positions the center of the kernel over each pixel of the source image, mixes the values of that pixel and its neighboring pixels together, depending on their weights in the matrix, and then assigns the result to the pixel in the destination image. The size of the kernel determines the number of neighboring pixels that are used in the convolve operation. If the kernel is 5x5 pixels, for example, then the convolve operation will include the original pixel and those up to two pixels distant.

It seems simple, but ConvolveOp is actually the main (and sometimes only) building block in many image-processing operations. Basic blurring and sharpening can be done with ConvolveOp. The only difference between a blur filter, a gaussian blur filter, a sharpening filter, an edge-finding filter, and an embossing filter, is the values in the kernel.

However, there is one complication with doing convolves. What do you do with the edge pixels? When ConvolveOp positions the kernel at the upper-left corner of the image, for example, unless the kernel is only 1x1 pixel, then the top and left portions of the kernel will be hanging off the edge of the image. What do we do in that situation?

There are lots of different ways to handle the edges, but unfortunately, ConvolveOp only provides two, neither of which are very useful. EDGE_ZERO_FILL causes ConvolveOp to act as if the missing pixels are present and set to zero. EDGE_NO_OP tells ConvolveOp to simply do nothing.

A blur filter uses a kernel in which the all of the weights are more or less equal and total to 1. Since all the weights total to 1, the filter does not alter the overall brightness of the image. But using the default EDGE_ZERO_FILL edge strategy causes ConvolveOp to effectively mix a bunch of black pixels into the edges of the images, making them darker. That's usually not what the user expects. The user expects the image to be blurred but still have crisp edges. However, the other option, EDGE_NO_OP, just leaves the edge pixels unblurred, which is even worse.

Ideally ConvolveOp should provide a third option, EDGE_USE_NEAREST, which would use the value of the nearest image pixel to fill in the missing pixels. But there is no such option. Also, there is apparently no way to plug in a custom edge strategy. (This seems like it would be an incredibly useful feature, but I suspect that it doesn't exist because the actual convolve operation is implemented in native code.)

However, there is a way around this. Instead of relying on ConvolveOp to figure out the values if the edge pixels, just provide them yourself. This requires creating an intermediate image that is slightly larger than the original image--just large enough to include the extra pixels that you need. If your kernel is 5x5 pixels, then you need 2 additional pixels on either side of the original image, and therefore the intermediate image will be 4 pixels wider and 4 pixels taller than the original. Then, copy the original image into the center of the intermediate image, and fill in the outer pixels however you please. Run ConvolveOp on the intermediate image, then strip off the edge pixels that you added by copying the center portion (less the edge pixels) into the final destination image.

For the purposes of blurring, all I cared about was that the extra edge pixels be more or less like the edge pixels of the original image. So after creating the intermediate image, I simply scaled the original image up 4 pixels in either direction, placed it in the intermediate image, then copied the original image (at its original size) into the center of the intermediate image. That effectively made the edge pixels of the intermediate image the same as the edge pixels of the original image.

Obviously this strategy for handling edge pixels is only useful for small images. If the image was 50 megapixels, then creating another 50-megapixel buffer just to add some edge pixels would be a waste. But if you're working with 50-megapixel images, then you're probably going to want to write your own image-processing code anyway and not rely on Java 2D. For thumbnail images, creating an intermediate image is quick and easy.

Saving JPEGs: ImageIO gotchas

2008-08-27T18:31:00.001-07:00

While the first of the Java 2D classes that I used in my thumbnail-generation code was AffineTransformOp, in order to actually see the results, I needed to save the image. Fortunately, Java includes a library called ImageIO that can save images in various formats.

The code to write out a JPEG is pretty straightforward, though if you want to set a specific quality, you can't use the one-line version. But when I opened the resulting file in Preview (I use a Mac), it was completely black!

I've had experiences with weird JPEG variants before, and I figured that maybe Preview was lame and that a better program would be able to display the image. So I opened it in Photoshop. This time, I was able to see the image, but all the colors were wrong. According to Photoshop, it was a CMYK image. But the original image was RGB. How could Java 2D be loading an RGB image, scaling it, then producing a CMYK JPEG? And anyway, the file obviously wasn't really a CMYK image: the colors were all wrong. It was a regular RGB image that Photoshop was reading as CMYK.

I looked for information about CMYK JPEGs, so I could maybe figure out what ImageIO was doing wrong, but I couldn't find much real information about them. Instead I mostly found posts from other people having problems with them.

I wound up getting some code that I knew was able to generate a BufferedImage and save it out as an RGB JPEG successfully, and I started gradually converting it to my code in order to determine at which point it started creating the CMYK JPEG instead. That point turned out to be immediately after I added AffineTransformOp.

The only obvious difference that I found between the BufferedImage that ImageIO saved correctly and the one that it saved incorrectly as a CMYK JPEG was that the former had a type of TYPE_INT_RGB and the latter, which came out of AffineTransformOp, had a type of TYPE_INT_ARGB. In other words, the output of AffineTransformOp contained an alpha channel. An ARGB image contains four channels. A CMYK image contains four channels. A-ha!

I didn't investigate CMYK JPEGs much further after that, but my theory is that Photoshop interprets any four-channel JPEG as a CMYK JPEG. Or maybe the file that ImageIO writes doesn't contain a color profile, and Photoshop defaults to CMYK because it has four channels. In either case, Preview cannot display the four-channel image.

The solution was to create an intermediate BufferedImage of TYPE_INT_RGB, use Graphics2D.drawImage to copy the AffineTransformOp output to the intermediate image, then save it out.

But why was AffineTransformOp creating an ARGB image in the first place? After all, I passed in an RGB image. Shouldn't the output match the input? I traced into the source code to see what was going on, and I discovered that even if I passed in an RGB image as the destination, AffineTransformOp would ignore it and create an ARGB image. Why? It has to do with AffineTransformOp's ability to perform any kind of affine transform, which might involve rotating or skewing the image. In these cases, the transformed image may not occupy the entire area of the output raster. For example, if AffineTransformOp is being used to rotate the image by 45 degrees, then the output raster will still be rectangular--because all rasters are rectangular--but will contain a diamond-shaped image in the middle. The pixels of the raster that fall outside the image must be transparent, so the BufferedImage that AffineTransformOp uses for its output must have an alpha channel. If AffineTransformOp tried to use a regular TYPE_INT_RGB image, then those pixels outside the diamond-shaped image would come out black, probably not what the caller wants.

Once again, it seems like Java 2D does the right thing, even if its rationale is sometimes hard to figure out.

Update to using UUIDs as primary keys

2008-08-27T18:13:00.000-07:00

Some time ago, I blogged about using UUIDs as primary keys. To quickly recap, because programs can generate UUIDs directly, without having to consult an external ID-generation service such as a database, using UUIDs as primary keys enables developers to avoid having to deal with the differences between persistent and transient instances of the same class and allows them to develop classes that are completely independent of the database. And because UUIDs are globally unique, they facilitate the creation of system that can operate in a disconnected fashion and then sync up later.

After working with a system that uses UUIDs for primary keys, I come around to the idea that database-generated keys aren't so bad after all. But, I think that the key (pardon the pun) is to keep them in the database. To that end, I have been creating Java classes that have UUIDs and use them to implement equals and hashCode, but that simultaneously have an integer ID field that is assigned from the database. This strategy gives me the best of both worlds: The UUID enables me to maintain a database-independent identity for each object, but I can still use the smaller, faster integer ID for foreign keys within the database. My Java code doesn't use the integer ID at all, apart from mapping it to the database through Hibernate.

Generating thumbnail images: AffineTransformOp gotchas

2008-06-24T07:09:00.001-07:00

AffineTransformOp is a Java 2D filter that can scale (resize), rotate, and/or skew an image. Since I was generating thumbnails, I was only interested in its scaling ability.

To scale an image using AffineTransformOp, all you need to do is generate a new AffineTransform using the static method AffineTransform.getScaleInstance, then use it to construct an AffineTransformOp. When constructing an AffineTransformOp, you also need to pass a constant theat identifies the type of interpolation to use, either nearest neighbor, bilinear, or bicubic. Once you have the AffineTransformOp, you just call the filter method, passing in the original image and the destination image, which can be null, in which case AffineTransformOp will create a destination image for you.

After generating my first thumbnail image and saving it as a JPEG, I was able to view my generated thumbnail image in Preview, but it looked like crap. It looked blocky, as if AffineTransformOp was using the nearest neighbor interpolation method even though I had specified bicubic. I switched to nearest neighbor just to see if there was a difference, and there was--bicubic was definitely better--but the thumbnail looked bad.

Prior to now, I was generating thumbnail images in photoSIG using the Image.getScaledImage method. I looked for information on getScaledImage versus AffineTransformOp and found an article called The Perils of Image.getScaledImage that provided the answer. Bicubic interpolation doesn't work well when the scaling factor is very small, as it is when scaling from a 1,000-pixel-wide original image to a 125-pixel-wide thumbnail. The solution, according to the article, is to scale the image in steps, with each step scaling the image by no more than one-half.

I updated my thumbnail-generation code to use the multi-step method. I first created a AffineTransform to half the size of the image, specifying a scaling factor of 0.5 for both the height and width. As long as the width and/or height of the image was more than double the size of the thumbnail that I wanted to create, I applied the halving filter. Then, once I was within a factor of two of my desired thumbnail size, I created and applied a final, custom AffineTransform.

This seemed to work well enough, and the quality of my thumbnails was definitely better. But while it worked well for some images, others had a one-pixel-wide black line on either the bottom or the right side. Actually, I had noticed similar black lines on some of the other thumbnails that I had generated with the single AffineTransform, but I was so busy trying to figure out why the thumbnails looked like hell that I didn't pay much attention to them.

The black lines frustrated me for a while, and I started to think unflattering thoughts about Java 2D in general. Why can't Sun's stuff just work? But it turned out that Sun's stuff was working all too well. The answer came to me while I was in the bathroom at work. I don't know why I was thinking about it then, but I've often found that when I'm dealing with a vexing problem, my mind continues to work on it at while I'm doing other things, and eventually the answer comes to me. I almost always figure things out in the end. But sometimes it takes a while.

The black lines were caused by Java 2D trying to handle my request to scale the image by exactly 0.5--even when the width or hight of the image was odd. If the original image is 640x480, then no problem, AffineTransformOp returns a 320x240 image. But what if the image is 640x427? AffineTransformOp can't create a BufferedImage that's 213.5 pixels high, so it generates one that's 214 pixels high. I don't know exactly how it handles the bottom row, but I imagine that it uses the alpha channel to give the appearance of an image that is taller than 213 pixels but not quite 214 pixels. Then, when I convert the image to TYPE_INT_RGB, the bottom row of pixels becomes black.

I fixed the code so it generated a new AffineTransform each time through the halving loop with the exact scaling factors needed to make the resulting image an integer number pixels wide and high (which would have been 0.5, 0.49882903981 in the above case).

That fixed most of the problem, but strangely, I was still getting a black line every now and then. I finally figured out that these occasional black lines were due to my using floats to calculate the scaling factors rather than doubles. Why was I using floats? Because the Kernel used by ConvolveOp uses floats, and so I had gotten used to casting non-integer values to floats. Oops.

After doing all this work, I wound up getting rid of AffineTransformOp completely. What did I use instead? Graphics2D.drawImage. By passing a width and height to drawImage, I can make Java 2D scale the image without having to bother with AffineTransformOp. I can draw directly to an TYPE_INT_RGB image (instead of having to convert the TYPE_INT_ARGB I get from AffineTransformOp), and since I specify the target width and height directly, there are no scaling factors to miscalculate. If I'd used drawImage right from the start, I would have saved myself a lot of time. But then again, I wouldn't have learned as much!

Java 2D image-processing gotchas

2008-06-24T06:41:00.001-07:00

I've been working with Java 2D a fair bit over the past few weeks and encountered a fair number of gotchas, or least behavior that seemed strange to me at first but that eventually made sense, once I thought about it and/or UTSL and figured out why it was working that way.

By way of background, I've mostly been developing web applications for the past eight years, so my experience with Java 2D is minimal. I've been investigating the image-processing capabilities of Java 2D because I need to implement a new feature for my photography web site, photoSIG. The feature will blur thumbnail images of photos on the photo-browsing pages if the "content rating" of those photos is either R or X, and the page is displaying advertising. The purpose of this new feature is to recover some advertising inventory that I currently can't use because advertisers don't want their ads appearing on the same page as R or X photos, even if they're thumbnails.

At various points in the process of working through all of these issues, I was feeling pretty unhappy with Java 2D and started to view it as another example of a Java API that is ostensibly very powerful but that is clunky and cumbersome to use. But I pressed on, because Java 2D has been around a while, and I felt that it must be better than my initial impression would suggest. Also, I noticed that I was unable to find a lot of web pages, blogs, and/or public postings complaining about the sorts of problems that I was having, which led me to think that maybe I was doing something wrong. It turned out that in most cases, I was doing something wrong, but Java 2D wasn't giving me much in the way of diagnostic assistance. After figuring out the solutions to my problems, I decided to post them here, in case other people run into the same difficulties.

Fighting words on Ruby

2007-04-26T06:37:00.000-07:00

Is anyone but me dismayed by the current state of Ruby? It's a great language, and it has a lot of promise--but there are also major issues that make Ruby and related efforts seem more like a set of academic projects than serious production-quality programming tools.

The main thing that prevents me from using Ruby is that the 1.8 interpreter is too weak, and the other alternatives are too immature. I can use Ruby for scripting in the same way that I'd use Perl or PHP, but I wouldn't want to write a major application using a technology that doesn't support native threads, Unicode, or some kind of GUI. Yes, I know that you can roll some primitive Unicode support into Ruby, and I know that there are several GUI toolkits out there, but overall, Ruby's support for anything beyond basic scripting is spotty.

So what are the alternatives? I guess the front-runner is JRuby, but JRuby is really just a scripting language that works with Java, not something that you'd use to write an application. And compared to MRI, it's really, really slow.

And anyway, why would I want to write something using Ruby 1.8, when I'd just have to change it as soon as Ruby 2.0 comes along? I'm skeptical that Ruby 2.0 will ever arrive; the Ruby 1.9 branch seems like a free-for-all experiment with language features that's not converging on anything coherent. I've seen posts from Matz about Ruby 2.0 on comp.lang.ruby from as far back as 2001. But the fact that Ruby 2.0 is out there puts the the wisdom of developing Ruby 1.8 code into question. I can't even develop Ruby 1.9 code with any confidence, because there are features in 1.9 that might not make it into 2.0.

My feeling about Ruby is that it will never enjoy the success of C, C++, Java, Perl, or even PHP. But I am hopeful that someone will take some of the concepts and syntax from Ruby and use it to create a viable post-Java application development langauge.

JetBrains TeamCity 1.0 beta

2006-09-15T19:47:00.000-07:00

I decided to give the JetBrains TeamCity 1.0 beta a try. JetBrains is the company that makes IntelliJ IDEA, the best Java IDE by far, and TeamCity is their new continuous build tool.

You can download TeamCity as either a WAR or a ZIP, which includes Tomcat. I downloaded the ZIP and unpacked it into a directory on my iMac. It looked like a pretty standard Tomcat layout. I ran the startup script, and (after giving me a warning about a missing work directory) it started up. So far, so good.

I accessed TeamCity on its default port of 8111. The main page appeared and invited me to create a new administration account. What could be easier than that? I didn't have to configure anything beforehand; the software was smart enough to know that the first account created has to be an administrator account.

After logging in, I headed to the Administration page and created a new project. I use Perforce for source control, and TeamCity supports Perforce directly. I created a new client for TeamCity and configured TC to use that client. All I needed to enter was the standard Perforce connection information (server, port, username, password) and the Perforce-style path identifying the sources.

TeamCity supports a lot of different build systems. You can build using an Ant or NAnt (if you are running on Windows) build file, Visual Studio 2005 project (again, Windows only), Maven project file, or IDEA project file. The ability to build using IDEA project files is particularly welcome, since it allows the build to remain completely in sync with an IDEA project. JetBrains promises support for Eclipse project files in a future version. I already had an Ant build file, so I selected that option.

TeamCity keeps track of a build number that increases by one for every build. You can use the build number to identify each build. Unfortunately there doesn't seem to be a way to use any identifier other than the build number. I use Perforce change numbers as my build numbers--a build of some branch is always called somebranch.xxxx, where xxxx is the number of the last change submitted to that branch (obtained using p4 changes -m1 -i //depot/path/to/the/branch/...). JetBrains could improve TeamCity by enabling it to get the build identifier from an external script or a Java method.

TeamCity itself is just a web application; it doesn't run any builds. To run builds, you must first download and install a build agent on some machine. There is a link within the TeamCity application for downloading an agent. After you configure and run the agent, it registers itself with the TeamCity web application and can start running builds. You can have as many agents as you want, so a single TeamCity installation could theoretically support hundreds of projects.

You can trigger builds manually, periodically based on elapsed time, whenever code is delivered to the source control system, or some combination thereof. I chose to build the application whenever code is delivered to source control.

When I ran the build the first time, I encountered an error with the build. TeamCity displayed the error message, but strangely, it did not display all of the output from Ant, which I would have expected. The problem was that my build file assumed that the current directory was the directory containing the build file, which was always true when I ran Ant from the command line, but when the build ran under TeamCity, the current directory was the directory containing the agent run script. I fixed that by using a property in my build file to specify the directory of the files that the build file needed. Then I set that property in the project configuration in TeamCity. When I delivered the new build file to Perforce, the build agent started up almost immediately, built the project, and reported success! TeamCity also included, on the build results page, a link to the changes that had been submitted to Perforce and included in the build.

TeamCity will also perform code coverage tests using EMMA and can run IDEA inspections. There is also a TeamCity plug-in for IDEA. I have not explored any of those features yet.

Overall, I am impressed with TeamCity, especially its ease of use. Without reading any documentation, I was able to install the product, create a user account, create my project, and enable builds on delivery, all in less than an hour. Apart from ease of use, TeamCity's other strength is its integration with IDEA, particularly its support for IDEA project files and inspections.

There are a few areas in which TeamCity could use some improvement. Although it can scale to hundreds of build agents, there is no way to organize projects into groups or folders. Something like that would be essential for a large company with hundreds of projects. There is also an assumption, in the user administration pages, that each user will only use one source code repository of each supported type. A complex environment may include multiple repositories of a one type. Of course, the organization could configure one TeamCity installation per repository, but that just forces the user to access more than one application to figure out what's going on. And as I mentioned before, I'd like to be able to use a shell script or Java method to generate the build identifier.

Who needs DAOs when there's Hibernate?

2006-09-04T20:22:00.000-07:00

I'm sure that everyone is familiar with the data access object or DAO pattern. The basic idea of a DAO is that you encapsulate the nitty gritty details of saving and loading data to and from a persistent store in an object so your application logic isn't riddled with SQL or filesystem access or whatever it is that your DAO does.

In the two years that I've been working with Hibernate, the DAO pattern continues to bother me. I always seem to be writing the same code over and over again: save this, load that, run some query, etc. Whenever I have to repeat myself a lot, I start to think that something's wrong. I eventually put some of the common stuff into a general-purpose DAO class that I used as the superclass for all my other DAOs, but then it seemed that my DAO superclass was just a less-functional wrapper around the Hibernate Session interface. I also kept having to write code to do things that Hibernate already supports, for example, query by example.

A deeper problem, though, is that it's very hard to hide Hibernate behind a DAO. Hibernate is very powerful and exerts an influence on the application even if the Session interface is not visible.

One major difference between Hiberante and an alternative data access strategy such as JDBC is that Hibernate automatically persists changes to objects back to the database when the application--or the container--commits the transaction. JDBC doesn't do that; the application must invoke save methods on the DAO in order to persist changes. So right away, the fact that the DAO is implemented using Hibernate influences the design of the calling code: the callers don't have to explicitly save anything.

Also, objects returned from Hibernate DAOs may be full of lazy-loading proxies that may trigger additional database queries as they are touched by the application. Objects returned from a JDBC DAO definitely don't do that (unless you go to great trouble to implement your own lazy-loading scheme). I realize that the DAO methods could be written to initialize any proxies that were needed by the caller, but now the DAO has to know what the caller wants, and there may need to be multiple versions of the same method, each of which initializes different proxies.

In fact, it's possible to design Hibernate DAOs so that they acted like JDBC DAOs, by initializing proxies and by disconnecting objects from the session before returning them and forcing the application to pass any updated objects to a save method, but what would be the point? You'd have a Hibernate DAO that stripped the application of much of the benefit of using Hibernate in the first place. On the other hand, if you're going to let the application take advantage of Hibernate, then what's the point of hiding Hibernate behind the DAO interfaces? All you're doing is forcing your application to use something less powerful than the Hibernate Session interface and writing a lot of extra code.

Here's my disruptive thought. I think that the Hibernate Session is the ultimate DAO. It does exactly what you'd think a DAO should do: you tell it what objects you need, and it returns them. That's exactly what a JDBC DAO that you might write yourself does. Your application doesn't want to deal with Connections and PreparedStatements and ResultSets, but it does want the business objects, and that's what your JDBC DAO does, convert those ugly JDBC objects into business objects. After you've written a few dozen JDBC DAO objects, you may start thinking that what you really want is one DAO that can give you anything you need, so you don't have to write more JDBC code whenever you add more business objects. That ultimate DAO is the Hibernate Session.

I can think of a couple of other arguments against using Hibernate straight, without DAOs, but I don't think that they are very compelling. Someone will undoubtedly mention that DAOs can be mocked. True, but you can also mock the Session interface. Another argument might be that the DAOs restrict database access patterns to those that have been optimized through the use of indexes or whatever. I suppose whether or not that's a good argument depends on who is going to use your DAOs. If it's yourself, then you can probably trust yourself to use the database properly. If it's another group of people who are not the regular application developers, then I suggest that you probably want to expose some kind of service interface to those folks instead of giving them your DAOs.

I've pretty much convinced myself that wrapping Hibernate in another DAO layer is a waste of effort, but I've not yet put my idea to the test. I'll give it a try over the next few days and write about my experiences.

Does Maven actually make things easier?

2006-08-12T12:50:00.000-07:00

I've been using Maven 2.0.4 for the last month or two and am discovering that I spend so much time dealing with Maven that I am wondering if I should give it up completely and go back to Ant.

I used Maven archetypes to set up my project modules, and that worked pretty well. But almost immediately, I found myself on the Maven treadmill.

One of the improvements that Maven 2 has over Maven 1 (which I have never used) is support for transitive dependencies. That means that Maven will download not only the artifacts (usually jars) that your project needs, but also any artifacts that those artifacts need, and so forth, to any depth. It sounds cool, but it's not so simple in practice.

For one thing, in order to figure out the transitive dependencies, Maven has to download the POM (project object model) files for all the dependencies, but many artifacts in the Maven 2 repository do not have POM files. In that case, I have to work out the dependencies myself and add them to my project's own POM file. Meanwhile, Maven will try to download the missing POM files every time I build the project and raise the same warnings over and over again.

Even when an artifact has a POM file, it sometimes causes more problems than it solves by referencing artifacts that you don't actually want in your project. One such case is when artifacts reference different, incompatible Spring Framework components; for example, one artifact may reference spring-core-1.2.7 while another references spring-web-2.0-rc3. And sometimes you just don't want the artifacts in your project because you know that you're not going to use them; as an example of this, Spring WebFlow pulls in Struts, even though I'm not developing a Struts application. Another annoying example is the tendency of applications to pull in Log4J, which, when present in the classpath, causes the behavior of Commons Logging to change. Maven allows me to correct all these problems in the POM file--I can force Maven to use version 2.0-rc3 of the Spring components, and exclude Struts and Log4J--but I have to add a lot of extra lines to my project's POM file.

So far I've focused on missing and/or inappropriate POM files, but sometimes it's the artifact itself that's missing from the repository. Specifcially, none of the Sun jars are in the repository, due to restrictions imposed by Sun. I was also unable to find the Cache Taglib from the Apache Taglibs project. In each of these cases, I must download the jar myself and manually add it to the local repository. And if another person works on my project, then that person must also download the jars.

So I'm spending time downloading and installing jars and sorting out dependencies. Tell me again what Maven is doing for me?

Oh yes, all those plug-ins, like the automatic project web site builder, and the IDEA project generator. Those features are nice, but they are hobbled by a lack of good documentation. The documentation for the IDEA project generator plug-in, for example, doesn't make any mention of the available options; I found out about the options from a discussion board. And configuring the plug-ins requires a excessive amount of XML. I had to write all this just to use Java 5:


<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.5</source>
                <target>1.5</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-idea-plugin</artifactId>
            <configuration>
                <jdkName>1.5</jdkName>
            </configuration>
        </plugin>
    </plugins>
</build>

What I really want Maven to do, more than anything else, is manage the dependencies and build the project. The fact that I have to spend a lot of time doing those things myself is what makes me wonder if maybe Maven is one of those tools that solves 80% of some problem but then makes the other 20% a lot harder.

Getting more complicated

2006-08-11T19:25:00.000-07:00

I was reading about C in Wikipedia (under C programming language). The article first presented the ubiquitous "hello, world" program written in K&R C:


main()
{
    printf("hello, world\n");
}

and then the same program in ANSI C:


#include <stdio.h>

int main(void)
{
    printf("hello, world\n");

    return 0;
}

Now I know that ANSI C is an improvement over K&R C. But in this case, the "improvement" has produced a program that needs two additional lines to accomplish the same thing.

Wouldn't it be better if technologists spent time making technologies simpler rather than more complicated? It's true that simpler technologies are always arriving. But unfortunately the tendency for any single technology is to get more complicated rather than less. I think that the only real reason why C hasn't gotten significantly more complicated since ANSI C is that all the extra features got tacked on to C++ instead, making it an insanely complicated language.

Java is undergoing a similar...complificiation. Generics do cut down on source code volume slightly and increase type safety, but they are anything but straightforward. For proof, all you need to do is look at the interfaces for the collection classes, which have suddenly sprouted lots of >, <, and & characters. It's a lot easier to use generics-enabled classes than write them. Meanwhile the new enum type creates a whole new set of rules. And annotations blur the line between code and data. All of these features have uses, but they all also increase the total amount of knowledge required to effectively work on Java applications, and that doesn't bode well for the language over the long run.

java.util.Date must die

2006-08-06T14:44:00.000-07:00

I find java.util.Date to be most of the most annoying class in the entire Java class library. It is essentially a wart left over from Java 1.0, but the fact that it is used in many APIs makes it difficult to remove.

The most insidious aspect of Date is that despite being nothing more than a wrapper around a long, Date is mutable. Anyone can call the setTime method and change the internal long. That causes problems with methods changing Date instances that are passed to them and code accidentally changing Date instances that are members of collections. What benefit comes from Date being mutable? Java understands the benefits of immutable objects. String is immutable. Long is immutable. Why not Date?

What's worse is that the SQL date types specified by JDBC inherit from java.util.Date, making them mutable too! That's in addition to the general inconvenience imposed by having them in the first place.

The inconvenience of Date might be acceptable if it actually did something, and in fact, in Java 1.0, the Date class did have useful functionality. But since Java 1.1, Date does nothing. Nothing! All of the useful methods of Date were deprecated and replaced by java.util.Calendar. But for some reason, setTime survived the purge.

I'd like to see Date.setTime deprecated and the Calendar-like accessor methods reinstated. Why did those other methods have to be deprecated? They could have been respecified in terms of Calendar and could have operated on an internal thread-local Calendar instance, allowing developers to write


int year = someDate.getYear();

rather than


Calendar calendar = Calender.getInstance();
calendar.setTime(someDate);
int year = calendar.get(Calendar.YEAR);

And while I'm there, didn't anyone think that was odd that the method to pass a Date into Calendar is called setTime? How about setDate! Then the method that passes a long into Calendar could have been called setTime, just like the same method of Date.

UUIDs as primary keys

2006-08-06T12:17:00.000-07:00

I wrote in my last post that I preferred to let the database just store data and and implement all the business logic in my Java code but that I usually let the database handle primary key generation.

After thinking about it for a few days, I decided to experiment with using UUIDs
as primary keys. Java 5 has a convenient java.util.UUID class that I used to
replace the integer identifiers in my entity classes.

I found a bunch of blog entries about using UUID versus integer primary keys. UUIDs do have some disadvantages:

UUIDs are larger than integer IDs. A UUID, when stored in its most compact form, consumes 128 bits, or 16 bytes. That's four times larger than a regular 32-bit integer.

UUIDs are binary. While they fit in a 16-byte binary column, when expanded into a string, they typically consume 32 or 36 characters, depending on whether the string is the human-friendly 00112233-4455-6677-8899-aabbccddeeff format.

UUIDs are computationally harder to generate than integers.

The disadvantages really boil down to space and speed. But the more I thought about it, the more I thought that there were significant higher-level advantages:

UUIDs can be generated by the application itself, without having to write the object to the database or consult the database in order to determine the range of integer IDs already in use. That simplifies unit testing. And since the UUID is immediately available, persistent classes can implement equals and hashCode in terms of it, rather than falling back on object identity.

A UUID is unique not only within a single class but across all classes and all JVMs.

This allows relationships that were previously implemented as <many-to-one> to be re-implemented as <one-to-one> when that is really the nature of the relationship.

This also allows the construction of tables that support references to "any" object without having to include explicit type information in the table. For example, an audit log table could have one column for the UUID of the object that was changed without having to maintain track of what kind of object it was. That in turn simplifies the interface to access the audit log entries.

Heterogenous distributed systems can exchange information about objects without identifier conflicts.

Using UUIDs in URLs and in web forms makes applications more secure. If a user sees a URL that looks like http://www.something.com/editmessage?id=50 then the user may be tempted to change 50 to 51 to see if he or she can edit someone else's message. If the message identifier is 14809d2d-8c71-473f-a5cd-123649cd9624 instead of 50, then it's unlikely that the user will be able to find another identifier that identifies a real message. This is the same reason why your MasterCard or Visa has a 16-digit account number even though there are certainly less than 10,000,000,000,000,000 cardholders.

It's true that UUIDs consume more space and require more computing power to manage than integers. But while this is the kind of argument that I would have considered back in 1994, computers today are roughly one thousand times more powerful than they were then, and I don't think that slinging around 128-bit keys rather 32-bit keys is going to make a difference in the overall performance of any but the most speed-critical applications. Throughout the history of computing, we have repeatedly taken advantage of ever-increasing performance to abstract away complexity and solve higher-level problems. Java itself is an example of this trend. It is undoubtedly more efficient in terms of computing power to skip the JVM and interact with the operating system and memory directly, but we pay the price of using the JVM because we see benefit from the tradeoff.

Unfortunately Hibernate doesn't know how to persist the java.util.UUID type out of the box (though I have to believe that a future version will), so I wrote a couple of Hibernate UserType implementions. One of them persists the UUID to a 16-byte binary column, while the other persists the UUID to a 36-character char column. I used 36 characters instead of 32 because UUID has toString and fromString methods that generate and consume UUID strings in the 00112233-4455-6677-8899-aabbccddeeff format.

In my persistent classes, I'm using UUID.randomUUID to generate the UUIDs. I don't do it in the constructor, though, because then I'd be generating UUIDs even when loading objects from the database. Instead, I initialize the UUID lazily, in the getUuid method. Since I'm using property access for the UUID property, this lets the object construct a new UUID when needed but not incur the cost of creating UUIDs that are just going to be overwritten with different UUIDs loaded from the database. I benchmarked UUID.randomUUID and found it to be pretty fast--I created 100,000 UUIDs in about 750ms--but there's no need to accept the cost when the workaround is so straightforward.

On the database side, I'm using Oracle, so I created the UUID column as RAW(16). Oracle provides to functions for working with RAW columns, HEXTORAW and RAWTOHEX. You can probably guess what they do. Oracle also provides a SYS_GUID function that will create a UUID for you, so you can define the UUID columns with DEFAULT SYS_GUID() in order to avoid having to create UUIDs yourself when manipulating the database directly.

Update (May 7, 2007): Zach Nichter has done some analysis of how UUIDs (he calls them GUIDs) perform as primary keys with SQL Server.

I switched from using Oracle to PostgreSQL, and I'm now using bytea columns to hold the UUIDs. It's unfortunate that there's no UUID type in PostgreSQL, especially since the database supports a wide range of types. A UUID type would be more efficient than bytea, because bytea columns maintain the length of each value on disk, which is always 16 in the case of UUIDs.

Property methods: just shoot me now

2006-07-31T21:33:00.000-07:00

How much time I spend, writing crap like this:

class Something
{
    private int id;
private String name;

public int getId() { return id; }
public void setId(int id) { this.id = id; }

public String getName() { return name; }
public setName(String name) { this.name = name; }
}

Any decent IDE will generate the property methods for you, but why is this necessary? Way back in ancient history, Visual Basic knew how to do the right thing: If you defined a setter for some property, then VB would call it when the application set the property. Set it how? With the equals operator, of course. (The equals operator was how software developers assigned values to variables prior to the release of the JavaBeans specification.) And if you defined a getter, then VB would call the getter when the app used the property in some non-lvalue context. If you didn't provide a getter or setter, then no problem, VB just treated the property as a regular old field.

In an anemic domain model, whole classes may exist with nothing but private fields and a public getter and setter for each. What's the point? Why not just make the fields public and be done with it?

Someone's going to say, well, we don't know what's going to happen in the future, and someday we might want to add actual business logic to one of those property methods. That seems like a great example of doing a lot of work now in order to avoid some potential work that may need to be done in the future, which is exactly what we shouldn't be doing. And anyway, if you really want a struct then you should just say so:


/**
  * This class is a structure used to pass A from X to Y.
  * All fields are public. Do not add logic.
*/
class Something
{
    public int id;
    public String name;
}

I think that my biggest gripe about the proliferation of property methods is that Java doesn't do anything to make it easier for us. Java forces us to decide, up front, whether we want to expose the field or "hide" it behind trivial methods, and then makes us actually write the trivial methods if we want them. VB got it right. Why can't Java?

Hibernate events

2006-07-31T19:40:00.000-07:00

I'm always having an argument with myself about how much intelligence to put in the database. I have a strong preference: none. I would rather work in Java and just let the database store data. But I always make an exception for database-generated identifiers, and then I start thinking that maybe I should write a trigger to let the database take care of timestamps, or recalculate a photo's total rating in photoSIG when someone ads a new critique, or...

And then I start thinking that maybe I should do the opposite, remove the database-generated IDs, use UUIDs for everything, and really use the database just for storing data.

For lots of domain objects I maintain two fields, "created" and "updated." You can probably figure out what they do. Throughout the photoSIG code I am forever updating the updated field. I decided to investigate whether I can use Hibernate to do this for me. I only really care about when the object was written to the database, not when it was last updated in the code (I only keep these objects around for a fraction of a second, the duration of a web request), so I figured that I could use a Hibernate event to wait for the object to be flushed and then update the updated field at that time. Or so I thought.

I've never had any love for the Hibernate Interceptor interface. It seems very low-level. It provides you with the object under consideration, but it also sends you the fields from the entity in parallel arrays. If you want to update anything, you have to update the arrays, meaning that your Interceptor implementation not only has to know a lot about Hibernate but also has to know a lot about the structure of your object too. It's not pretty.

And so it was with relief and anticipation that I approached the new Hibernate events API. But if anything, the Hibernate events API seems even more low-level than the Interceptor interface. The way that Hibernate works now is that essentially every method you call on Session produces an event that gets handled by a chain of event listeners. You can add your own listeners, but the last listener in the chain better be the Hibernate default listener for that event, otherwise Hibernate will behave very strangely or just won't work at all. The "events" API is really an implementation of the strategy pattern. This is unfortunate, since I would have preferred an API that specified a useful contract between Hibernate and an application, rather than an internal contract between Hibernate and Hibernate.

Regardless of my gripes about the API itself, the thing that really got in my way was the lack of documentation concerning when the "events" are produced and where I should put my code to achieve some desired effect. The JavaDocs aren't much help. The class description for LoadEventListener is "Defines the contract for handling of load events generated from a session," and the documentation for its single method, onLoad, reads "Handle the given load event." Is this called right when someone calls load, or after Hibernate instantiates the object, or after the object is populated, or what? There is also a PreLoadEventListener and a PostLoadEventListener. What's the difference? The quality of the Hibernate documentation overall is excellent, but in the case of the events API, it's just not there.

I pressed on. I created an implementation of FlushEntityEventListener and, in onFlushEntity, set the updated field. But when I tested this implementation by creating a new object, Hibernate attempted to insert null into the updated column in the database. Huh? It turns out that Hibernate schedules the insert as soon as the application calls Session.save. If you update the object in the onFlushEntity method, then Hibernate schedules an update. Meanwhile the original insert fails.

Obviously I had to intercept the save "event" and update the object before Hibernate scheduled the insert. I created an implementation of SaveOrUpdateEventListener that updated the updated field and registered it using EventListeners.setSaveOrUpdateEventListeners. But when I ran the test again, nothing changed! Now I was really confused. I had to USTL to figure out that the setSaveOrUpdateEventListeners sets the listeners that are called when the app calls Session.saveOrUpdate, not Session.save or Session.update. I had to call setSaveEventListeners, which still accepts a SaveOrUpdateEventListener.

Now Hibernate inserted the row with a non-null value for updated. But I still had some problems. The updated value that I set in my SaveOrUpdateEventListener wasn't the time at which the object was written to the database but rather the time at which it was handed to Hibernate. A fine distinction, but either you implement the requirements that you've defined for yourself or you don't. FlushEntityEventListener still updates the updated field when the object is flushed, so the "save" time is immediately overwritten by the "flush" time, but Hibernate has to issue an update in order to do it.

After poking around some more, I started thinking that the Session-interface "events" that I was using were the wrong way to go and that I should look into PreInsertEventListener, which is apparently called just before Hibernate inserts something into the database. Unfortunately the corresponding PreInsertEvent holds an array of field values, just like the methods of the Interceptor interface. It's too late to change the object there.

At that point, I abandoned my attempt to get Hibernate to maintain the updated field.

I understand the motivation behind the design of the events API. It's not that I don't get it. But I wish that the Hibernate team had given us something simpler and well-documented. All I really want is a callback from Hibernate saying, "Hey, I'm about to write this object to the database, so if there are any last-minute updates you want to do, this would be a good time!" In other words, what I want from Hibernate is a trigger.

Or maybe it was Philip Greenspun

2006-07-30T15:53:00.000-07:00

Philip Greenspun is another prolific developer and author. He created the immensely popular photo.net web site and developed database-backed web sites back when that was cutting-edge, even writing a book about it, called Database-Backed Web Sites. I don't know if Philip came up with the subtitle himself, which is "The Thinking Person's Guide to Web Publishing," but it sounds like something that he would say. It's interesting that back then, someone was willing to publish a book just about interfacing web sites with databases. These days, of course you're going to integrate your web site with a database. The days of static web sites are pretty much over.

Using code that he developed in the early days of the Internet, Philip founded ArsDigita Corporation and produced the open-source ArsDigita Community System, beating pretty much everyone in the race to develop a reusable, extensible web framework. Interestingly, ArsDigita also suffered through a something-to-Java rewrite project following their decision to convert ACS over from Tcl. The Tcl version of ACS is now maintained as OpenACS. Meanwhile, Red Hat bought out ArsDigita, which, like every other Internet startup in 2002, was in dire straits, and the Java version of ACS became Red Hat CCM.

I gave Philip a credit on the photoSIG About page: "Though he doesn't know it, Philip inspired me to develop the photoSIG system and indirectly, through his books and writings, influenced the development both of that system and of photoSIG."

Matt Raible made me do it

2006-07-30T14:53:00.000-07:00

I'm a software developer. I've been developing web applications of varying degrees of complexity for about six years or so, mostly in Java. I investigate a lot of new technology in the Java space, and I've been noticing for a while now that whenever I google for information about some bit of technology, I always find something on Matt Raible's blog. I've never met the guy and, as far as I know, don't use any of his code, but I feel like I know him pretty well at this point. Which got me thinking: maybe there's something to this blog thing. So here is mine.

I've been writing software for pretty much as long as I can remember. I can remember a time when I wasn't very good at it, which always seems to have ended about five years ago. But I can only barely remember a time at which I wasn't involved with computers in some fashion.

My folks bought a computer, an Atari 800, when I was about 11, but even before that, I had taken some programming classes. I wanted to get a Commodore PET, because that's what I'd used in the classes, but my parents bought the Atari instead. That was probably a better choice. I mostly worked with BASIC and some 6502 assembly on it. I ran a dial-up bulletin board system using software that I'd written myself. I upgraded to an Atari 520ST in the late 80s, but it was a pain to develop software for that machine, and I didn't do much work on it. Soon after I graduated (sort of) from high school, in 1990, I got a job writing software in C under MS-DOS. I bought a PC (a 33MHz 80386) because I figured that I should have one if I was going to make a living from it. I worked on the Mac for a few months and even on the Philips CD-I, then moved on to OS/2 for a while. In 1995 I found a job writing in C++ under AIX and have been on the Unix side ever since. In 1999 I started working mostly in Java, and that's where I am today. But I am troubled.

In 2001, when I was on vacation for a couple of weeks, I built a web site that allowed users to upload and critique photos. I called it photoSIG. I wrote it in PHP because it seemed like an easy development path and because I could get cheap shared hosting. I was able to write the app in no time with PHP. I didn't worry about mixing the view with the business logic or anything like that because it was five years ago and, as I mentioned before, I didn't know about those issues then. The site became amazingly popular, and before I knew it, I was on a dedicated server, and then I was on two dedicated servers. photoSIG went from zero to a million pages per day in about eight months. At that point, PHP started showing its limitations. I used a lot of caching in order to deliver the performance that I needed, but that led to a lot of data-freshness issues. I also had scripts that ran from cron to do things that were too expensive to do in the context of a request, for example, to increment view counts in the database, but sometimes those would fail. It was becoming a mess.

So in 2002, I set about rewriting the system in Java. This would fix my performance problems, I thought. I could use more intelligent caching, write threads (rather than cron jobs) to take care of housekeeping issues, and so on. I did rewrite the system, but after deploying it, I started thinking that I'd only swapped one set of problems for another. The Java system, for one thing, was not noticably faster than the PHP system. It was much larger and took much longer to write. The code had better structure and was more reusable, but the reusability, at least, was an academic concern because I had nothing to reuse it with.

At least I didn't use EJB. I'd already had experience with EJB when I started the photoSIG rewrite, having led the development of a big commodity derivatives risk management system that used EJB pretty extensively for a couple of years prior. That system wasn't completely done in by EJB, since I recognized when I started it that entity beans were lame, but even stateless session beans were a big bother. For the new photoSIG system, I developed regular POJO service classes and managed transactions myself.

In 2004 I started converting parts of the system to Spring and Hibernate. I like both of those technologies, Hibernate in particular. But here's the thing: I don't feel that they're making my life any easier. They offer ways to solve problems, but the solutions themselves are pretty complex. I find that I'm spending an awful lot of time trying to work out the best way to apply these technologies to each situation and editing configuration files rather than working on the actual business problem at hand. It's no fun spending a lot of time dealing with infrastructure issues when you'd rather be writing an application. I'll get into some specifics in a later entry.

So I'm troubled. And I'm starting to think that something is wrong here. It doesn't seem like things should be this complicated. Maybe all this dependency-injection and object-relational mapping stuff is overrated. Or maybe Java just isn't up to the task at hand?