Guarding Logger Statements In Ruby

December 9, 2009

Whether you are a Java or a Ruby programmer, I'm sure you are familiar with this idiom:

require 'logger'

log = Logger.new(STDOUT)
log.level = Logger::INFO

log.debug("hello")
log.info("Done")

That's a simple logger where the log level is set to info, so the debug statement isn't logged, but the info statement is. One gotcha to look out for is something like this:

require 'logger'

log = Logger.new(STDOUT)
log.level = Logger::INFO

def fib(n)
  if n < 1
    0
  elsif n < 2
    1
  else
    fib(n-1) + fib(n-2)
  end
end

log.debug("fib(30) => #{fib(30)}")
log.info("Done")

This also just logs "Done", but it take more than a few seconds to do so. The reason why is that even though you aren't logging the string that gets passed to debug, the ruby interpreter still has to incur the cost of generating the string and passing it to debug, where it gets ignored.

If you are an old Java programmer like me, you'll probably know you can fix it like this:

require 'logger'

log = Logger.new(STDOUT)
log.level = Logger::INFO

def fib(n)
  if n < 1
    0
  elsif n < 2
    1
  else
    fib(n-1) + fib(n-2)
  end
end

if log.debug?
  log.debug("fib(30) => #{fib(30)}")
end
log.info("Done")

That works, but it's not the Ruby way of doing it. It's the idiomatic way of doing it in Java, but that is due to the fact that Java doesn't have anonymous functions nor a concise syntax for creating them. The Ruby way of doing it is:

require 'logger'

log = Logger.new(STDOUT)
log.level = Logger::INFO

def fib(n)
  if n < 1
    0
  elsif n < 2
    1
  else
    fib(n-1) + fib(n-2)
  end
end

log.debug { "fib(30) => #{fib(30)}" }
log.info("Done")

The difference between this version and the original is that instead of passing a string to debug, we pass a block that returns a string when it is called. We don't have to wrap it in an if statement because the block can be conditionally evaluated based on the current log level.

The difference between the if statement and the block is admittedly minor. That being said, prefer the block syntax. :)

The important thing to remember is that if you have a debug statement that does any kind of calculating, pass it a block instead of just a string to avoid the overhead associated with unnecessarily building the string.

Posted in Technology | Tags RubyOnRails, Ruby, Java | 2 Comments

Infinite Recursion

September 2, 2009

A few days ago I posted an article on Tail Call Optimization. One really quick way to determine if any language/VM/interpreter performs Tail Call Optimization (TCO) is to write a function that calls itself, in other words, creating an infinitely recursive function. If the function just runs forever and doesn't return, the interpreter is doing TCO, otherwise you will get some sort of stack overflow error. So I decided to test a variety of languages to see what happens when you write an infinitely recursive function. First up is ruby:

def forever
  forever
end

forever

It was no surprise to me that running this results in this error:

run.rb:2:in `forever': stack level too deep (SystemStackError)
    from run.rb:2:in `forever'
    from run.rb:5

Ruby doesn't have TCO, I knew that. Next up, Python:

def forever(): forever()

forever()

This has a similar result, but the stack track is a little more revealing:

Traceback (most recent call last):
  File "run.py", line 3, in <module>
    forever()
  File "run.py", line 1, in forever
    def forever(): forever()
  File "run.py", line 1, in forever
    def forever(): forever()
...
  File "run.py", line 1, in forever
    def forever(): forever()
  File "run.py", line 1, in forever
    def forever(): forever()
RuntimeError: maximum recursion depth exceeded

This shows pretty clearly what is going on, the stack frames are pilling up and eventually it gets to the point where the Python interpreter says enough is enough. Interestingly enough, this mailing list thread shows that it's completely feasible to add TCO to Python and Guido just doesn't want to add it.

JavaScript is no surprise either, but we can write our infinitely recursive function with an interesting little twist:

(function(){ arguments.callee() })()

Yes that's right, it's an anonymous recursive function! Running this with SpiderMonkey results in InternalError: too much recursion. So how about Java:

class Run {
  public static void main(String[] args) {
    Run run = new Run();
    run.forever();
  }

  public void forever() {
    forever();
  }
} 

The stack trace for this one looks a bit like the Python one, we see a stack frame for each iteration:

Exception in thread "main" java.lang.StackOverflowError
    at Run.forever(Run.java:8)
    at Run.forever(Run.java:8)
    at Run.forever(Run.java:8)
    at Run.forever(Run.java:8)
...

So that means no TCO on the JVM. So Scala doesn't have TCO, right?

def forever : Unit = forever

forever

Wrong. This one runs forever. Scala does TCO with some bytecode tricks which Nick Wiedenbrueck explains really well in this blog post.

Clojure is a functional Lisp saddled with the problem of no-TCO on the JVM, but it gets around it in a slightly different way than Scala. If you write a tail recursive function like this:

(defn forever [] (forever))

You will get a java.lang.StackOverflowError just as you do in Java. Instead, Clojure provides a language level feature to do recursion:

(defn forever [] (recur))

Calling recur in the function makes it call the function again. recur also checks that it is in the tail recursive position, which ends up being an interesting feature because you have to explicitly say "I expect this to do TCO". In other languages that do it transparently, you might think TCO is happening, but it might not be and you won't find out until you get a stack overflow error. Also, this construct allows us to have recursive anonymous functions:

((fn [] (recur)))

Erlang, being a function language, has TCO as well:

-module(run).

-compile(export_all).

forever() ->
  forever().

Now one thing all of these functional languages that have TCO; Scala, Clojure and Erlang, all have in common is that while the infinitely recursive function runs, it essentially does nothing, but it will peg your CPU utilization to nearly 100%. This next language, Haskell, being the mother of all functional langauges, of course has TCO. Here's the function:

forever = forever

Yes, that's a function. And if you call it with forever, it just sits there and runs forever. But here's the crazy thing, it uses no CPU. My guess is that it has to do with lazy evaluation, but I'm not sure. Any ideas?

Posted in Technology | Tags TCO, Javascript, Scala, Haskell, FunctionalProgramming, Python, TailCallOptimization, Ruby, Java, Erlang, Clojure | 19 Comments

In Search Of Sharper Tools

July 21, 2009

After reading the comments in my last post, one thing I realized that I neglected to do is define what I mean by metaprogramming. Rubyists probably already know what I mean, but people coming from other programming languages might have different ideas about what I mean by metaprogramming.

I've actually mentioned this in a few talks I've given about Ruby. I don't really like the word Metaprogramming, it's a little bit nebulous. I think a better term is dynamic code generation. I like that term because I think most programmers will have a pretty good mental model of what I am talking about when I say that. There are several features of Ruby that when combined together allow you to do almost anything to bend the language to your will. To understand how to do that, I recommend reading the book Metaprogramming Ruby, which is in beta right now.

I'll give a short, real world example of what I'm talking about. I'm working on a Rails app that uses a database that has bit field columns. I want to treat the boolean flags in the bit field as though they are regular boolean attributes throughout my code. So I wrote a Ruby gem has-bit-field, which generates all the methods necessary to work with the bit field columns. You define what is in the bit field in the most clear, simple, elegant way possible by just stating which column has the bit field and then what attributes should be generated for each bit:

class Person < ActiveRecord::Base
  has_bit_field :bit_field, :likes_ice_cream, :plays_golf, :watches_tv, :reads_books
end

This is the kind of abstraction that the metaprogramming capabilities of Ruby afford you. I threw this together, with tests, in an hour or so. Can you imagine the amount of nonsense you would have to go through in Java to create an abstraction equivalent to this?

This type of abstraction is what I think makes Ruby a great language, but I realize you are definitely walking a fine line with this kind of thing. It's possible for this sort of thing to go wrong quickly. The first way these things go wrong is when the abstraction is not intuitive and not documented. First of all, a good abstraction should be almost intuitive, requiring other programmers to be able to guess what is does. This is commonly referred to as the Principal of Least Surprise. This doesn't mean that you are excused from providing some sort of documentation explaining how it works, especially for more complex abstractions.

The reason why it's important that the abstraction is clear is that most of the time the code that defines the abstraction is dense at best, and downright ugly at worst. This isn't a problem specific to Ruby, as anyone who has worked with Lisp macros can attest to. But in the end I'd rather have a small chunk of code that is tested and documented that I don't really need to look at that enables me to make the code where the business logic is defined as clear as possible. If other programmers are constantly having to dive into the guts of the definition of these abstractions just to understand how the code works, you have officially created a mess. And this is no ordinary mess, this is meta-spaghetti, and is a mess on an order of magnitude not possible in statically typed languages.

So does this mean you shouldn't use Ruby? Not at all, and I think Glenn Vanderburg sums it up best:

Weak developers will move heaven and earth to do the wrong thing. You can't limit the damage they do by locking up the sharp tools. They'll just swing the blunt tools harder.

I think developers often associate "blunt tools" with static typing, because really they associate static typing with Java. I'm not sure that static typing is in fact a blunt tool. If static typing means I can't create these kinds of abstractions, then yes, it's a blunt tool. But can you do this kind of thing with Scala Compiler Plugins? How about with Template Haskell? What about with MetaOCaml? If you can, are those tools then sharper than Ruby? Or is there a way to define abstractions like these without metaprogramming at all?

Posted in Technology | Tags Scala, DynamicTyping, MetaOCaml, OCaml, Haskell, StaticTyping, Metaprogramming, Ruby, Java | 0 Comments

Emulating Haskell's Maybe in Java with Checked Exceptions

July 17, 2009

Haskell is one of the interesting languages that I've been looking into off and on for a while now. I think the type system is really interesting and if you don't know Haskell, you should check it out. I've found that the best introduction to the language is Learn You A Haskell. Once you've got the basics down, you should tackle Real World Haskell, which is a much more in-depth book. I'm still working through it when I have some spare time.

One interesting concept in Haskell is the Maybe type. The purpose of the Maybe type is to declare in the type signature of a function that a function might return a value or it might return nothing. In Java, the return type of method can't indicate that. For example, if you have a method with this signature:

Object lookup(String key)

You have no idea if that will actually give you an object or not. The lookup method may or may not be implemented in such a way that prevents it from returning null. If the method does return null, you are risking the possibility of encountering the dreaded NullPointerException.

Haskell doesn't have this problem because it doesn't treat null in the same way that Java treats null. As we just implied in the previous paragraph, this is a valid implementation of the lookup method from the Java compiler's perspective:

Object lookup(String key) {
  return null;
}

Haskell's equivalent to null is called Nothing, but the difference is that Nothing cannot stand in for all other types, as it can in Java. So if you actually had a type called Object in Haskell, a function that returns an Object cannot return Nothing.

A practical example of this would be looking up a value in a Map. When you try to retrieve the value for a key in a map, you might get a value or you might not find one. Here is an invalid Haskell program that tries to do that:

import Prelude hiding (lookup)
import Data.Map

zipToCityMap = fromList([("21230", "Baltimore"), ("21046", "Columbia")])

main = do
  putStrLn "What is your zip code?"
  zip <- getLine
  putStrLn $ "You live in " ++ (lookup zip zipToCityMap)

If you try to run this (which you can do on a mac by first installing ghc through macports with the command sudo port install ghc, then saving the contents above to city.hs and running the command runghc city.hs), you will get this compilation error:

city.hs:9:32:
    Couldn't match expected type `[Char]'
           against inferred type `Maybe [Char]'
    In the second argument of `(++)', namely
        `(lookup zip zipToCityMap)'
    In the second argument of `($)', namely
        `"You live in " ++ (lookup zip zipToCityMap)'
    In the expression:
          putStrLn $ "You live in " ++ (lookup zip zipToCityMap)

What the compiler is trying to say is that the lookup function returns a Maybe [Char], not a [Char]. [Char] is a type that is a list of characters. In Haskell, the type String is just an alias for [Char]. By wrapping the [Char] in a Maybe, the lookup function requires it's callers to unravel the value out of the Maybe in order to use it as well as explicitly handle the case that no value was found. If we modify our program to look like this:

import Prelude hiding (lookup)
import Data.Map

zipToCityMap = fromList([("21230", "Baltimore"), ("21046", "Columbia")])

main = do
  putStrLn "What is your zip code?"
  zip <- getLine
  case lookup zip zipToCityMap of
    Nothing -> putStrLn "I don't know where you live"
    Just city -> putStrLn $ "You live in " ++ city

This will compile and run. The program will ask you for your zip code and it will print the city you live in, but if it can't find it in its "database", it prints a friendly response rather than blowing up. Now here's a Java version of this program:

import java.util.Scanner;
import java.util.Map;
import java.util.HashMap;

class City {

  String name;
  String state;

  public City(String name, String state) {
    this.name = name;
    this.state = state;
  }

  public static City lookup(String zip) {
    Map zipToCityMap = new HashMap();
    zipToCityMap.put("21230", new City("Baltimore", "MD"));
    zipToCityMap.put("21046", new City("Columbia", "MD"));
    return (City)zipToCityMap.get(zip);
  }

  public static void main(String[] args) {
    System.out.println("What is your zip code?");
    Scanner in = new Scanner(System.in);
    String zip = in.nextLine();
    System.out.println("You live in "+lookup(zip).name);
  }

}

If you compile and run this, if you enter one of the chosen zip codes, it will tell you what city you live in. But if you enter an incorrect zip code, you will get the dreaded NullPointerException:

$ javac City.java
Note: City.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
$ java City
What is your zip code?
21230
You live in Baltimore
$ java City
What is your zip code?
90210
Exception in thread "main" java.lang.NullPointerException
        at City.main(City.java:26)

Now to get around this you would normally just check to see if the city is null and then print an alternative method if it is:

City city = lookup(zip);
if(city != null) {
  System.out.println("You live in "+.name);  
} else {
  System.out.println("I don't know where you live");
}

We know to do this only because we know the lookup method might return null. It would be nice if when defining the lookup method we could make it explicit to the callers of the method that the method might return null and have the compiler generate an error if the caller does not explicit check for the condition of a null result. If this were the case, a NullPointerException would be impossible. This is exactly what Haskell's Maybe type does.

A few other people have attempted to define a class in Java that mimics the Haskell Maybe type. That appears to work somewhat, but actually seems fairly convoluted to me. Another way to achieve the same goal would be to use a checked exception. To do this, put this in a file called MaybeNullException:

public class MaybeNullException extends Exception {}

Then add throws MaybeNullException to the end of the method signature for the lookup method. With just those changes, you will get a compilation error:

City.java:26: unreported exception MaybeNullException; must be caught or declared to be thrown
    System.out.println("You live in "+lookup(zip).name);

Which is exactly what we want. We have indicated in the method signature of our lookup method that it might return null, therefore the caller of our method is required to do something about it.

To get this to work, first we need to make the lookup method actually throw the exception if the city isn't found. This is semantically equivalent to returning Nothing in the Haskell version:

City city = (City)zipToCityMap.get(zip);
if(city == null) {
  throw new MaybeNullException();
} else {
  return city;
}

And then in the method where we call lookup, we need to check for the Exception:

try {
  System.out.println("You live in "+lookup(zip).name);  
} catch(MaybeNullException e) {
  System.out.println("I don't know where you live");  
}

Now we get the desired behavior:

$ javac -cp . City.java
Note: City.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
$ java City
What is your zip code?
21230
You live in Baltimore
$ java City
What is your zip code?
90210
I don't know where you live

The problem with this solution, at least from a Java perspective, is that many people, myself included, consider checked exceptions to be a failed experiment. Making use of checked exceptions all over your code results in having to add a lot more boilerplate and try/catch statements. But is this a small price to pay for eliminating the NullPointerException? Or does the Maybe type impose too much overhead in Haskell, making it doomed to fall out of favor the same way checked exceptions did in Java? My thought is no, that packing as much information as possible into the type of functions is the essence of Haskell. Also in Java, checked exceptions are something programmers learned to essentially get rid of by wrapping them in unchecked exceptions, where Nothing is very core to the way Haskell works. Nevertheless I thought this was an interesting thought experiment and might serve as a good analogy for Java programmers learning Haskell.

I should point out one more thing on this topic which is that the Maybe type is definitely more powerful in Haskell because it is a Monad. Because of this you can chain function calls together without having to deal with the possibility of something like a null pointer exception. The Java Checked Exception doesn't provide us a similar mechanism.

Posted in Technology | Tags NullPointerException, Maybe, Haskell, Java, MaybeMonad, CheckedExceptions | 18 Comments

"Write Everything Twice" and "Because What If I Need It Later"

April 1, 2009

Programmers are concerned with the number of lines of code they write in their programs. There is this idea that the fewer lines of code you write the better. This is a hold over from the days of tiny hard drives measured in Kilobytes, a far cry from today's standard of measuring hard drives in hundreds of Gigabytes or even Terabytes. We no longer need to convolute our programs with abstractions simply to save on the number of lines. This is why I'm advocating the "Write Everything Twice", or the WET principle. The next time you have a method or function in your code that does something and you want to use that function in another place in your code, but have it function slightly different, just simply copy and paste the original method and alter the copy. You've heard it said time and time again. Disk space is cheap, which in turn means lines of code are cheap.

Another closely related principle is "Because What If I Need It Later", or BWIINIL (pronounced ba-WIN-null). Many programmers are lazy in that when they develop an application, they sometimes only write the code for that features that they think they need at that moment. What they don't realize is that there is significant mental cost of switching back into the context of that part of the code a later date. Any time you are writing code you should say to yourself "What else might I need later?".

One particularly good way to apply the BWIINIL principle is to get the book Design Patters: Elements of Reusable Object-Oriented Software, which is more commonly know as the Gang of Four book, and attempt to apply every pattern in the book to every method you write. This will allow you to insert layers upon layers of abstraction and indirection that may seem like overkill at first, but will come in handy some day when you are adding new features.

Here's an excellent real word example taken from Uncyclopedia. We'll all familiar with this trivial implementation of HelloWorld in Java:

public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, World!");
    }
}

This kind of code is often included in books about Java, which is understandably watered down for programmers first learning Java. But if you really want to graduate to that next level and write a professional, enterprise-ready implementation of Hello World, you need this:

interface Printer {
    void print(Message message);
}

class Message {
    private String message;

    public Message(String message) {
        this.message = message;
    }

    public void print(Printer printer) {
        printer.print(this);
    }

    public String toString() {
        return message;
    }
}

abstract class AbstractPrinterFactory {
    public static AbstractPrinterFactory getFactory() {
        return new SystemOutPrinterFactory();
    }

    public abstract Printer getPrinter();
}

class SystemOutPrinterFactory extends AbstractPrinterFactory {
    public Printer getPrinter() {
        return new SystemOutPrinter();
    }
}

class SystemOutPrinter implements Printer {
    public void print(Message message) {
        System.out.println(message);
    }
}

class HelloWorld {
    public static void main(String[] args) {
        Message message = new Message("Hello, World!");
        AbstractPrinterFactory factory = AbstractPrinterFactory.getFactory();
        Printer printer = factory.getPrinter();
        message.print(printer);
    }
}

Also make sure to include UML class diagrams that shows how these components interact. Once you learn to embrace the WET and BWIINIL principles and apply them to your everyday coding, I'm sure you'll quickly see an increase in the quantity...er, I mean....quality of your code.

Posted in Technology | Tags AprilFools, Java | 4 Comments

  Older Articles >>