Blackdrag's View

The invokedynamic API

2012-01-08T12:55:00.000+01:00

I thought I should write a bit about the invokedynamic API, since the API is very powerful and flexible, but sometimes needs a bit getting used to it. I won't write about everything or even in detail, just some hints for thinking on a few elements - the ones I used for Groovy mostly and are most probably the ones you will use as well.

Let us say we are in the situation, that we have the method we want to call as handle already and we have the target type of our call site. I call that method handle SM (for selected method) and the target type TT - for obvious reasons.

Now normally in Groovy you would do the following: you transform the arguments into a form you can use for calling the method. If you for example have an int, but want to call an long taking method, then you have to transform the int into a long. You normally do this by using a general type transforming method, that takes the argument and a goal type, inspects the argument and then goes through quite big code parts for all the transformations that are allowed. We do this not during method selection itself, but for each call of course. What we do is kind of like wrapping the method object into something new, that we can call in our call site and that for each call will test if it has to apply the big standard transformation - and well, apply it too.

If you read that big transformation code, then you will have a direct transformation of the arguments to whatever the SM needs. In invokedynamic we work a slightly bit different. Instead of having only one transformation we have many small ones, that we combine into a specialized one. We do this by taking our SM, apply transformers to it and use the result for our method calls. Since the main work is now no longer correcting the big transformer code, but the combination of the many small transformers the workflow feels kind of reversed to me. You now want to apply transformers to make out of your SM, one that accepts the target type TT.

Some notes for better understanding:

One thing to remember when working with MethodHandles for example is that it has no receiver. In a method call foo.bar(x), we say normally that foo is the receiver (well the class of foo), bar the method name and x the argument. For a MethodHandle it itself is the receiver of course. So if we work on its arguments, then the first argument is the receiver of our method call. A MethodHandle for above would maybe have this MethodType: (Foo,X)Object - taking the foo receiver and an x argument, returning an Object.
The classes you work with mostly... You have MethodType to describe a method's parameters, including return type and receiver. There is MethodHandle of course, the core of it all, representing your SM. And there is a collection of helper methods in the class MethodHandles. Note the additional s. Well, and SwitchPoint, but I haven't used it really yet... that is still to come soon.
Type matching... your SM has to be changed by the usage of the transforms to fit the requested target type of your call site. For this you use for example asType

But without further delay...

MethodHandles#dropArguments is one quite useful method, that at the same time shows very much the reversed thinking that needs to be applied. This is a transform, that will drop arguments, it does not do it right away. In the old thinking we have for example arguments a,b,c and if we would drop there, we would for example get a,b. This transform *will* do exact this, but what we look at gives a different impression. We have a handle that takes A,B (A being type of a and B being type of b) and dropping then means we will take that and produce a new handle that can take A,B,C. This new handle will then ignore the argument c of type C and in doing so, it does exactly what I described above, but if we debug the code producing the combinations of transformers we see a method handle that gets the dropArguments transformation applied and now takes one more argument instead of one less. So again.... assuming you want to get rid of an argument you start with the handle, that is without it and have to transform it into a handle that takes it... for me that is like reversed thinking and really sometimes produces problems for me. You may want to apply this one if your SM takes less arguments than provided.

MethodHandle#bindTo is also one I use often. Assuming we have a handle taking A,B,C and A will be the same for each call, we can bind it to produce a new handle taking B,C... but only for the first argument. So for example if you have a method call that will be done through the meta class system, then this results mostly in calling a method MetaClass#invokedMethod. But the receiver there is not the receiver from my callsite, so I bind the first argument, the meta class. A changed meta class would require a new method selection so it is kind of constant for this selection. You may want to apply this one if your SM has one more argument than the ones provided and it is the first one

But what to do if we have more than one extra argument our SM would like to take? In this case you use MethodHandles#insertArguments. For example your SM would like to take A,B,C, your TT is A,B and you want to bind a c. Then you can use newHandle = insertArguments(oldHandle,2,c).If it where A,B,C,D, and you want to bind for C and D, then it is insertArguments(oldHandle,2,c,d). If you have A,B,C,D and you want to bind B and D, then it is for example (there are two ways): newHandle = insertArguments(oldHandle,1,b); newHandle = insertArguments(newHandle,2,d). You may have noticed the chaining of transforms in this one. The first makes A,B,C,D into A,C,D, moving D from position 3 to position 2. The second makes A,C then. You may want to apply this one if your SM has more arguments than the ones provided.

MethodHandle#asCollector is also a nice one. There are several invokeMethod methods in Groovy that take an Object[] as argument to contain all the arguments and then delegate calls to builders or do other things. The callsite you start with though may not provide the Object[], but the arguments one by one. So your TT may look like this: A,B,C,D and your SM may accept this: A,Object[]. Meaning we somehow have to wrap B,C,D into an Object[], if thought from the perspective of the arguments. And that is why the method is called asCollector. What you see on your method handle is that with handle = handle.asCollector(Object[].class, 3), your A,Object[] handle is now a A,Object,Object,Object handle. You will need asType to get to the final form. Should you want to collect the arguments at a different place than the last one, you may have to permute the arguments. The opposite way is asSpreader, but I didn't use that one yet. I may do so when I extend the implementation to include the spread operator. You may want to apply this one if your SM has arguments you want to collect into an array

Sometimes we have to make an transformation, that changes the runtime class of a reference type. This is of course not possible, because the JVM is strong typed - but we can create a new object with the desired result. For example Groovy has GString, which is a kind of String. A method call done with an GString argument is supposed to be able to call a method that takes a String. Now GString and String are not in a subclass relation, meaning that we will have to do a real type transformation here and MethodHandles#filterArguments will help us with that. A filter takes one argument and returns the transformed value. For our case it should take a GString and return a String. Let us say our SM takes A,String,String,B; TT be A,GString,GString,B and our filter MethodHandle takes GSTRING. Then we can simply do newHandle = MethodHandles.filterArguments(oldHandle, 1, GSTRING, GSTRING) to produce a handle for A,GString,GString,B. Again I had here some problems with the reversed thinking: The filter's return type must match the type in the SM and the argument type the type of our TT. If you think about it, it is quite clear and obvious, but when I work on a program I always have to stop here and rethink. Anyway... You may want to apply this one if your SM argument differs form the provided argument in a way that boxing and casting alone don't do the transformation requried.

And the last one I want to mention: MethodHandles#guardWithTest. In many situations you have to handle the invalidation of your call site to some extend. One way is for example (if you use a MutableCallsite) to have a guard causing exchanging the target for your callsite by a new method selection. Let us assume we have a call foo(a,x) and the methods foo(Object,Object) and foo(Object,String). Now x may be a String and we want to call foo(Object,String) then, or we want to call foo(Object,Object) if it is not String. Let us assume there will be three calls: The first done with an Integer, the second with String and the last one with an Integer again. We will have to exchange the method each time. A guard takes a number of arguments (0-N) and returns a boolean, which will be used to determine if we call our normal target or a fallback handle. For this case here I have used a handle called SAME_CLASS. It takes a class and an Object argument and tests if argument.getClass() is equal to the provided class. Since the first argument, the class, is fixed I bind it upfront, leaving me a guard handle which takes an Object argument only. guardWithTest does not support any position, but we want to check the second argument to the method, not the first... and there is the receiver too. If you did not jump to this section, then you may find we are in a situation in which we have less arguments than given... only it is not the SM, but our guard now. But that doesn't matter. Still we drop the first two to get a handle Object,Object,Object, which will ignore the 0 position argument, the receiver, and the argument at position 1, but test the one at position 2. Then it is simply newHandle = MethodHandles.guardWithTest(guard, oldHandle, fallback) and be done with it. If you want to have multiple guards, you continue to apply this kind of transform. I haven't found a way to combine guards themselves to have only one guardWithTest call. But maybe that isn't needed too. You may apply this one if your call site needs to be invalidated based on the given arguments.

This is just a short overview and by far nowhere near complete or a replacement for reading the javadoc. Just a little guide that may be of help.

The Java Way, simple Type Inference and Flow Sensitice Typing

2011-11-03T14:50:00.003+01:00

In "Groovy static type checker: status update" Cédric gave one of his favorite type checking examples. Even though the example made some things clear that I was not so clear about in my last blog post, I still think we need to look at this example a bit more in detail.

Perfectly fine Groovy code like this:

class A {

        void foo() {}

}

def name = new A()

name.foo()

name = 1

name = '123'

name.toInteger()

is a problem for a static type checker and I want to explain once more in a bit more detail why. There are currently 3 ways to approach this code...

The Way of Java
In this version we try to be as much Java as possible, but obviously we have to give the def a meaning. In standard Groovy this is basically equal to Object. As I wrote in my last blog post, the view on types is in Groovy a tiny bit different compared to Java. Anyway, if def is just an alias for Object and we compile this code with Java's type rules, then the code will not compile:

class A {

        void foo() {}

}

def name = new A()  // name is of type Object

name.foo()          // error: foo() does not exist on Object

name = 1            // assigning Integer to Object typed name is allowed

name = '123'        // assigning String to Object typed name is allowed

name.toInteger()    // error: toInteger() does not exist on Object

While the assignments would all work just fine, the method calls will not pass, since those methods are not available on Object.

Simple Type Inference
This seems to be the way groovypp goes. If I am wrong about it, feel free to correct me. Again we have to give def a meaning and this time we use right hand side of the assignment to do so. For the remaining code we stay more or less with the Java rules. The result is then this:

class A {

        void foo() {}

}

def name = new A()  // name is of type A

name.foo()          // no problem, foo() exists on A

name = 1            // error: assigning Integer to A typed name

name = '123'        // error: assigning String to Object typed name

name.toInteger()    // error: toInteger() does not exist on A

Instead of the two problem from before we have no 3, 2 of them at a different position in our code. If we want that piece of code to compile then those two approaches won't do it.

Flow Sensitive Typing
In this third version we still have to give def a meaning, but this time the meaning is not fixed:

class A {

        void foo() {}

}

def name = new A()  // name is of flow type A

name.foo()          // no problem, foo() exists on A

name = 1            // name becomes of flow type Integer

name = '123'        // name becomes of flow type String

name.toInteger()    // no problem, toInteger() does exist on String

With this we reach our goal - who would have guessed that ;)

The difficult thing for a Java programmer here probably is, that name is not of a fixed type. In fact, looking at many papers in the area of formal semantics, most type systems out there use the flow type only for type checks, not to actually give a variable a type. On the other hand, if you look at the Java way, you could say, that simple type inference is just an enhancement. Instead of letting the user write the type, the compiler will set the type. There are actually many old languages that support that kind of logic. This is really nothing new. Still, if we see that as an enhancement, then saying we let the compiler set the type of a variable automatically at more than one place can be considered as just the next step.

But I am getting side tracked... I only wanted to show why exactly this example is causing a problem or why not. That is all.[UPDATE: I had to reformat the article a bit, because I had problems with my java script based syntax highlighter and with the line length of some code examples]

Flow Sensitive Typing?

2011-10-26T16:34:00.002+02:00

While we (Guillaume, Cedric and myself) had a meeting in Paris, we talked about the typing system of Grumpy a bit.

Coming from a dynamic language and going static often feels quite limiting. For me the main point of a static type system is to ensure the code I just wrote is not totally stupid. Actually many would say the purpose is to ensure the correctness of the code, but maybe I am a more dynamic guy, because I think this is impossible to achieve for a poor compiler. So a static compiler usually checks

method calls, to ensure the method I want to call really exists
fields/properties, to ensure they exist
check assignments, to ensure right-hand-side and left-hand-side have compatible types
check type usage, for complying with the type system (including generics, class declarations and so on)
and others...

So usually if a compiler detects a violation it will cause a compilation error and if it cannot check things the code will probably include runtime checks.

Optional Typing

Now Groovy has, what we call optional typing. In Groovy the compiler won't check fields, properties or methods on their existence, since the compiler cannot be totally sure we really mean some entity, that exists at compile time. Groovy allows you to create/remove/add/replace methods at runtime and attach them to classes via their MetaClass. A program that would not compile statically, may run just fine with those additions. What the Groovy compiler does though is to check type usage to some extend. So you can for example not extend an interface. The Groovy compiler has to do this, because the JVM is also typed to some extend, and doesn't allow directly for arbitrary type systems. sure there are ways around, but that always means to reduce the high integration with Java, and we don't want that.

Another aspect is for assignments. If you assign a value to a typed variable in Groovy, then the compiler won't ensure this works at runtime, but it will add a cast, that may fail. Effectively this means for a typed variable in Groovy: We guarantee you, that if the assignment works, the value assigned to the variable, will be at least of the declared type.

This implies for example for a String typed variable, that you assign a non String to, that its toString() method is called. We call that way of casting a Groovy cast, and the rules for it are actually quite complex.

Still there are enough cases we could actually check, if we would know the type of the right-hand-side. In general we don't know that type, because for example a method call is done there and then we cannot ensure the type. By the time we actually reach that point in the code, the method might have been replaced.

Strong Typing
If you follow the discussions about typing, then you will most probably see very fast, that dynamic and static typing might be kind of defined, but beyond that, there are often conflicting definitions for other terms. For example some say that Groovy is not strong typed, while Java is. In my definition strong typing means that an value cannot change its type at runtime, without creating first a new value. In Java we have this situation for example for boxing. You can assign an int to an Object, but not without the value being boxed, thus a new value being created. Now in Groovy this is just the same. An int cannot become an Integer or a String, just like that. We depend on the type system, enforced on us by the JVM, and the JVM is strong typed... well that may change in the future, but for now it is strong typed. In Groovy you can add methods for example and with it changing the interface a value provides, but there is no way for a value of a certain class to become the same value with a totally different class, without a new value being created and that one used instead.

Flow Sensitive Typing
Flow Sensitive Typing is not unknown in the world. It is normally used to for example find the type of a complex expression, to then check that with the actual allowed type in an assignment. Now in Groovy we want to go a bit a different way. Basically we want to have not a fixed static type, instead each assignment can specify a new one. If you defined a variable using "def", then in normal Groovy all assignments to it are allowed. Basically we see "def" as Object in Groovy. But if you want static method checks, you still want something like "def str = "my string"; println str.toUpperCase()" to work. This case can so far be solved also by type inference. But in Groovy you can do also this: "def v = 1; v = v.toString(); println v.toUpperCase()". Even though we start out with an int, we assign later a String to v. If we work only with a simple inferencing system, this will not compile. But since it is of course our goal to make a wide range of Groovy programs available even in a static checked version, we would of course like to allow this. And a simple flow analysis can indeed give us the information, that the flow type of v change to String and thus the toUpperCase() method exists. In other words, this would compile. Taking into consideration, that "def" in Groovy doesn't mean much more than Object, we don't want this being limited to "def" only. We want also to allow this: "Object v = 1; v = v.toString(); println v.toUpperCase()" Java would not allow for this. Sure, you can assign the 1, you can even call toString() and assign it to v, but because v is declared as Object, the compiler would start barking at you for the toUpperCase() call. Our thinking is, that there is not really a need to limit the type system like this. As in Groovy, we would again give the guarantee, that v is at least an Object. But the specific type is in Groovy depending on the runtime type, on Grumpy on the flow type. Something Grumpy would for example still not allow is "int i = new Object()"

But till now this flow sensitive type system is not approved of by the community.

Feeling Grumpy?

2011-10-26T15:32:00.003+02:00

Grumpy, might have been mentioned a few times here and there already, but what is it?

It is a little project for Groovy we are currently working on and the project title is for now Grumpy. Grumpy is no final name, we use it really just until we found a final and more serious name.

The goal of Grumpy is to have a static type checker for Groovy driven by an annotation. This will be optional and not cause any changes to normal Groovy. We see this as part of the Java migration story, in which people may not want the dynamic type system of Groovy everywhere. Grumpy is also no static compiler, it will be normal Groovy under the hood. I will not exclude a future static compiler based on Grumpy, but that is not the primary purpose of Grumpy. also we don't want to compete with Scala here. For my taste their type system is too complex. We may go beyond Java's system though, if we think it makes sense.

Basically there is a static type checker with Groovy++ (and a static compiler), but integrating that into Groovy just for type checking will mean either to integrate huge parallel structures, that no one but the Groovy++ people do understand. Or it means to invest a lot of time to transfer everything over, probably more than it takes to write the type checker new. Thus we decided to make a new type checker and try to involve the community as much as possible on every step.

How will it work?
So far you will be able to annotate methods or classes and an AST transform will then perform some type checking on that code. We don't want any new syntax constructs for Grumpy. This means only a subset of Groovy will be accepted by Grumpy.

Can I mix dynamic typed code and Grumpy?
Yes you can. If you use the per method level annotations you can just use a different method for the dynamic or grumpy code. So far we have no annotation that will turn dynamic typing on again for the case you used the class level annotation, but that may follow in the future.

When will it be available?
Grumpy will be part of Groovy 1.9, the next beta will already include a pretty advanced Grumpy.

What will Grumpy do about the GDK?
For those, that don't know... the GDK is a set of methods we use to enhance standard Java classes. For example we have an "each" method on Collections, to iterate over the list using a Groovy Closure. Grumpy will support all GDK methods, but to enable type checking in those Closure blocks, there will have to be much more work.

Much more work?
In for example "int i=1; [1,2].each {i += it}" you want to ensure nothing is added to i, that is not compatible with int. Since we cannot possible let the compiler know about how each of those GDK methods is working by code, we will have to find a way to do that more automatically. The Java type system with generics is not really able to express our needs here. For example if you iterate a Map using each, you have one variant, that takes a Map.Entry, another one, that takes key and value. If you just declare everything as Object, you won't gain all that much in terms of type checking. most probably we will have a second annotation for this kind of thing, in which we will store a String with extra type information. The goal here is to let the compiler add this annotation on its own for Groovy code, but for predefined Java code we will of course have to depend on the user doing that, or not having the type information.

Anyway, I am sure Grumpy will be an interesting project. Feel free to suggest names

About primtive optimizations - Come into being

2011-09-07T23:02:00.001+02:00

New idea?
The idea to those kind of optimizations is by far not new in Groovy. In very early days the compiler had two modes of operation, where one was doing a more or less direct compilation as static language. But this compiler mode was not paid attention to for years and then finally removed by me. The normal compiler evolved far beyond that one and the language itself had a long history of semantic and syntactic changes back then already. As some may remember, it took Groovy 5 years to get to an 1.0 version.

Even if we ignore those first failed attempts, my optimizations are still not really new. Many other languages know the concept of slow and fast paths, ways of switching between them and so on. Going to the JVM language summit for several years is going to influence you.

So what is the idea?
The idea is to have some kind of guard, which decides if we want to do a fast path or a slow path. Of course we prefer the fast path, but it is not doable if for example we want to do a 1+1 and the integer meta class defines a new add method, making your fast path resulting in a wrong value compared with the slow path.

Synchronization is bad
The first big problem was thus to find a way to recognize meta class changes and react to them. This guard has to be as simple as possible. If we compare ourselves here with Java for example, then every little extra does cost you badly. The classic way, as call site caching does it, is far from ideal. to ensure no category is active we have to test - in the end - a volatile int. That may not be much you think, but this volatile int represents a big barrier for inlining and internal compilation by hotspot. Volatiles are something we have to avoid.

But if our guard is not using volatiles and not using any kind of synchronization, lock or even fence, then this means meta class changes done in one thread may not be seen in another thread. It is not that they would be invisible, there are normally enough synchronization mechanisms involved during execution to give a piggyback ride (see piggybacking on synchronization from Java Concurrency in Practice). If the user wants better controlled synchronization the user would then have to add synchronization code of his own, that enforces such happens-before-relationship... for example waiting with a BarrierLock in the Thread that is supposed to see the change and do the change in the other Thread, to later then reactivate the waiting thread. the change will become visible to the other Thread, even though we didn't synchronize the guard directly. In most cases though I dare to say this is not really needed.

Simple boolean flags
The next problem with checking, if a meta class is unchanged is to actually get the meta class. Since we are talking about for example int here, we would have to go the ugly way and request the current meta class from the registry. This again involves a lot of synchronization and many many method calls on top of that. If our guard should be as cheap as possible, then this is bad. Maybe even so bad, your guard is eating up all the performance gain of the fast path. I decided for a different way then.

I plug a listener into the MetaClassRegistry, which reacts to set meta classes for Integer and then set a boolean according to that. The default of the flag indicates, no custom meta class has been set and since MetaClassImpl does not allow for modification later on, we are safe in our assumptions for the fast path. The first guard is therefore a boolean flag in DefaultMetaClassInfo and some easy method calls to see on this flag. I implemented a logic that allows to enable the flag again, after a custom meta class was used. Another way for custom meta classes is the meta class creation handle. Setting that one will thus also disable all the fast paths. For this I use a different boolean. This boolean then represents: no active category && no meta class creation handle set. This normally means a globally enabled ExpandoMetaClass will not allow for the fast path changes to play out.

My branch point for slow and fast path for int+int is then guarded by two boolean checks, one for the standard meta class usage in general, the other for Integer specifically.

Piggybacking
For the reader this solution may look easy and simple, but I must confess coming up with that one took quite some time on my side. I was facing the volatile problem and looked into fences for a while before I found it is nothing for Groovy. Only then I started wondering what may happen if I don't synchronize at all and stumbled upon piggybacking, which fits my needs well enough.

Now that we know when to take the fast path, it is time to actually implement one. The theory is that int+int in Java is quite fast compared to Integer+Integer, because the JVM has special instructions for operations on primitives. That means we need to use the IADD operation.

Object operands
People knowing internals of Groovy may realize at this point, Groovy has primitives in the fields, in method signatures, but not for local variables. Even worse, every primitive gets boxed right away. This has partially historical reasons, but in the end it boils down to having less trouble with emitting bytecode. Not only comes there a big amount of additional instructions for primitives, some of them take also 2 slots (long and double), making basic operations like swapping the last two operands on the stack a little problematic. Of course, if I want to benefit from the fast operations on primitives I have to have them primitive. Of course without losing the ability to handle them as objects if needed.

So I had to rewrite the compiler to trace the operand stack in a way that allows me to see if I need to do boxing or not. From the first tests to actually having that for Groovy 1.8.0 this took a big part of my time. I used the opportunity to refactor the gigantic AsmClassGenerator into many smaller parts and establish some logic, allowing everything to be visited twice (one time slow path, one time fast path), with some more abstraction what to use here and there.

Type extraction
Another problem to be faced was: When do I actually have an int? If I have a local variable or field declared as int, then well, then I know I have. If it is an int constant, I know I have, but beyond that? My first tests also showed another problem: If I check the guards for each expression, then checking the guards consumes too much time.

So he decision was to guard statements, or if possible blocks of statements. For the statement there are then two versions, the fast and the slow version, where each expression in the fast version, may be handled as part of the fast path. If I have now i = a+b+c+d+e, I still have only one int guard - assuming i,a,b,c,d,e are all ints.

In this there is yet another part we have to look at. Is int+int an int? Only then we can guard using an int and safely do a+b+c. Luckily Groovy does not have type promotion, so there is no danger of a+b giving a long as result. a+b will stay int, even in the overflow case. Staying in the same group are also minus and multiplication, devision not, because int/int results in a BigDecimal. Other allowed operators are the shifts of course, the binary operations and modulo. The last mathematical operation is not allowed, the power operator ** has type promotion, so I cannot safely assume it working on int.

Stupid Fibonacci
A typical test I used, that concentrates on not too many operations, is a Fibonacci function:

int fib(int n){

  if (n<2) return 1

  return fib(n-1)+fib(n-2)
}

But if I am going to only implement n-1 and n-1 as fast path elements, I will not gain pretty much performance. The function is dominated by one compare, 2 minus operations, 1 plus and two method calls. The two minus operations are therefore only a small part in this. To also optimize the plus I need to know the return types of the fib method calls. In ordinary Groovy I will not know the return types.

This led me to want to optimize the method call in this case as well. In general this is a problem not really easily solvable in Groovy, but if I limit the optimization to only calls on "this", then I have a chance, because then I need to check only the meta class of the current class and since the current class is a Groovy class I have some freedom. I implemented another flag, that indicates, the current class is using the default meta class.

The result was that I now could emit optimized code for fib(n-1)+fib(n-2) guarded by thee guards now. Testing the result was not encouraging though. I gained maybe a third in performance. Compared to Java this was by far too slow.

Groovy standard compare is slow
By using a profiler I found that a lot of time is actually burned in the compare. The n<2 branches of into a gigantic internal function for all kinds of cases. I imagine hotspot simply going on strike for this one. That means I have to emit specialized code for the compare as well. Actually the principle is the same as before. int+int normally results into int.plus(int), and n<2 in n.compareTo(2). Of course I don't want to call the compareTo, just the same as I didn't want to call the plus. There are special VM instructions for this kind of thing and I should use them. And I did. I also did a small optimization in this code. Since now the first statements is guarded by 2 flags and the second statement by 3, of which 2 are identical I made all 3 flags be checked for both statements. In the AST they are in a BlockStatement, so I kind of guarded that one instead of the single statements. That the BlockStatement has nor representation in the bytecode should not bother us. in the end it is some checks and some jumps only. The shortened bytecode looks then like this:

  public fib(I)I

   L0

    INVOKESTATIC test.$getCallSiteArray ()[Lorg/codehaus/groovy/runtime/callsite/CallSite;

    ASTORE 2

    INVOKESTATIC org/.../BytecodeInterface8.isOrigInt ()Z

    IFEQ L1

    GETSTATIC test.__$stMC : Z

    IFNE L1

    INVOKESTATIC org/.../BytecodeInterface8.disabledStandardMetaClass ()Z

    IFNE L1

    GOTO L2

   L1

    /* slow path ... */

   L2

    /* fast path ... */

    ILOAD 1

    LDC 2

    IF_ICMPGE L5

    ICONST_1

    GOTO L6

   L5

    ICONST_0

   L6

    IFEQ L7

    LDC 1

    IRETURN

    GOTO L7

   L7

    ALOAD 0

    ILOAD 1

    LDC 1

    ISUB

    INVOKEVIRTUAL test.fib (I)I

    ALOAD 0

    ILOAD 1

    LDC 2

    ISUB

    INVOKEVIRTUAL test.fib (I)I

    IADD

    IRETURN

Quite visible the BytecodeInterface8 calls for the check for categories and the original int meta class, as well as __$stMC, the guard for the default meta class. Compared to normal Javac code:

   public fib(I)I

   L0

    ILOAD 1

    ICONST_2

    IF_ICMPGE L1

    ICONST_1

    IRETURN

   L1

    ALOAD 0

    ILOAD 1

    ICONST_1

    ISUB

    INVOKEVIRTUAL X.fib (I)I

    ALOAD 0

    ILOAD 1

    ICONST_2

    ISUB

    INVOKEVIRTUAL X.fib (I)I

    IADD

    IRETURN

We can see that those two version are not all that different. The compare looks a bit different, some LDC should be ICONST, but all in all, quite similar. The result for fib(38) on my machine is 940ms in the groovy version and 350ms in the java version. In Groovy 1.7 we would have had something over 14s and most probably a stack overflow. And I think that is a pretty good result. If I would have only one guard, I would be almost at Java speed.

Invokedynamic and others
I mentioned earlier this technique is not really new, but I have actually never seen it being used in this way. Normally you have some kind of interpreter, representing the slow path. At runtime you find the parameters for the path and then emit a specialized fast path for these. You have to check the parameters after of course, but it is similar to what I do with the guards. Only in the interpreter you are working with actual runtime information and you can go far beyond what I did. I optimized for example method calls on "this". I have this limitation because I know of no good way to check the metaclass of the receiver in general. In the interpreter this is most probably no problem to care about so much and you get that call optimized as well. But Groovy has no interpreter that could be used for this, so that was no option. And while this approach here avoids the generation of classes at runtime it also bloats the bytecode quite a bit. Which means methods can now contain less Groovy code, a drawback I have not yet encountered in real life, but it probably is one.

The of course a word on invokedynamic here. This work was done for JVMs of the pre Java7 era. Java 5+6 will be around for at least another 2 years I imagine and thus we wanted to have something for those cases. Also the problems with the guards is one for invokedynamic as well, so I thought back then. Now I know that in the cases I would normally guard, I will simply invalidate all call site on demand and be done with it. What stays is the static code analysis for the types. If I know I will get only ints then I don't have to check for other types. If I have for example a+b and now nothing more of a and b, then they might be Integer in one case and String in another. If I make a callsite target for the Integer I will have to check a and b for each invocation to ensure they really are Integer. If it then turns out they are suddenly Strings, I will have to make a new callsite target and check for Strings.

Conferences I am going to this year

2011-03-19T00:10:00.003+01:00

I decided I could at least my update my blog with the conferences I am planing to attend this year.

So first is JAX 2011 May 2 till May 6 2011 in Mainz. The JAX is a pretty big German conference about Java and other things, including Groovy. I will speak on May 3 11:45.

Following tightly is GR8Conf Europe 2011 May 18 to May 19 2011 in Copenhagen. This conference is surely not as big as the JAX, but in exchange it is about all the great (gr+8) technologies around Groovy. I am especially looking forward to the Hackergarten there again. I will speak there on May 19 12:10. My third time for this conference.

Next and last that I am planing to attend is the JVM Language Summit. Well, at the moment the link still leads to the 2010 page, a final date is not up yet. I strongly hope there will be another one and that I can visit it the third... or was it forth? time. This conference was for me as language implementer on the JVM always extremely interesting. Also being able to bug the different JVM engineers about the dark corners of the JVM is just plain fun.

Besides that I did not yet plan anything more, but who knows ;)

git again

2010-11-15T11:46:00.002+01:00

For a while now I am using git locally and I am starting to like it a little. Coming from CVS and SVN many things just behave not right and it takes quite some time to make the transition to the new system. Along the way I found some very helpful pointers, that I want to mention here too. There is http://flavio.castelli.name/howto_use_git_with_svn giving a good overview of how to use git and SVN together. But the most helpful had been the documentation at kernel.org. There I discovered for example the rebase command, allowing me to reorder, split and edit my local changes. This truly helps keeps you to commit often and always having a backup of your work. With SVN I often did not commit, since my code would break trunk. For some time I had a local SVN repository that I used to save my changes. But the reediting capabilities in git are so much better.

The next step for me would probably be to put my git repository on github.com. Now the question is of course how to do that? How to have a git personal git repository on github, that I can use to commit to Groovy trunk (SVN) and that I can use for my local work. Can I have like two remote branches? I guess for this I will have to look around more.

Next steps with git...

2010-02-17T12:49:00.002+01:00

So after doing a checkout of the repository using git-svn for over two hours I started looking for an eclipse plugin for git.

What I found was EGit, which installed fine... but... well none of the git options appeared. Neither can I import from a git repository nor can I use git commands on the ready project. I tried to install about 5 times and failed 5 times. Maybe it is an error with the plugin, or something with eclipse is making a problem - I really don't know. The error log is clean.

So for me the one and only existing Git plugin for eclipse I found is unusable for me atm.... Next minus for git.

In fact I am now wondering about how to use something equal to for example RapidSVN. A diff tool, that allows me to remove different changes is very important to me - doing that only on the command line is not very effective.

Trying out Git

2010-02-16T18:42:00.002+01:00

My current work requires a lot of changes and experiments so I thought it might be good to try out git with the subversion back end.

So the first step is to make a local repository and then get the current svn repository... uhm, yes, the complete repository. Well ok, I am chekcing out only for trunk here, but that is the branch with the most traffic. NOw trunk is atm around revision 19250. A lot of those are not relevant for trunk, since it is about making branches or the test system making commits. But even if a third of those are normal revisions, we still talk about over 6000 revision. Revision git will all have to download.

Now after 45 Minutes I am at about revision 2350... 17000 to go... ehm... this will take hours it seems.

Ok, you have to do that part only once, but still... This I take as a minus for git vs. svn

Static Groovy - about calling methods

2009-11-10T17:19:00.007+01:00

After a long time I am now here writing something in my blog again. In the past I talked mostly about new features and possibilities like for example http://blackdragsview.blogspot.com/2006/09/groovy-on-speed-fast-mode-for-groovy.html

I want to talk about a static mode for Groovy here, since this is a reoccurring thing on the lists. I get the feeling people don't understand exactly what I mean and I want to try and explain it here a bit more. Of course I have my knowledge and others have their knowledge and maybe I will tell something that is wrong here. Well, if that happens tell me, I am aware of the fact that I know only a small fraction. And maybe one of the readers here knows just the way I was searching for and unable to find.

One basic part in Groovy is the method call. Almost any operator results in a method call and many structures are simulated through method calls. Now Groovy is a language with what I call instance based multi methods. Let me give an example:

class A {
 def foo(Object x){ bar(x) }
 def bar(Object o){1}
}
class B extends A {
 def bar(String s){2}
}
def a = new A()
assert a.bar("x") ==1
assert a.foo("x") ==1
def b = new B()
assert b.bar("x") ==2
assert b.foo("x") ==2

Now looking at that code you might realize, that Java would fail in the last assert for multiple reasons. First foo takes an Object typed parameter and that is never going to call a String based method, because the static type of the parameter is used as type for the argument, which then influences the method preselection process the compiler does. So Java would always select the bar(Object) method. The other aspect is that bar(String) is not declared in A, but in B. If you want another method get called in Java, you have to override one from the parent class, but bar(String) is not overriding in the Java sense of inheritance. So in Java you would in class B write a bar(Object) method, that checks if the given argument has the runtime type String and then dispatch to the String taking method else to super.bar(Object). In Groovy this is not needed. If bar is final, then in Java you cannot do that additional dispatch, for Groovy this makes no difference.

I should maybe also mention that often people talk about dynamic dispatch or dynamic method calls. What some mean is that from a given set of methods of a given class a method is selected using the runtime types of all involved arguments and the receiver. They often don't realize that this is multi methods. Coming from a Java thinking they assume only methods from class A can be taken, since foo is defined in A and so a bar of A must be used. In Java you might add that this is always true unless a subclass is overriding the method, but no new method signatures will be added to the set. Only, that if you use the dynamic type of the receiver anyway (yes, Java does this!) why make methods of it invisible? That was a decision made by the Java people and I am sure they had their reason to do so, but this is no requirement for a static language, especially if the return types cannot differ Before Generics were introduced this was the case for Java. Even for current Java I think it would be possible to ad. And if you take a look at MultiJava for example, you might realize that it does not have to be a dynamic feature either.

Another aspect of Groovy and method calls is the invokeMethod/methodMissing logic. Basically the method is called if the goal method couldn't be found. The parameters for this search are based on the dynamic types and multi methods, but a static language could have a methodMissing as well. But what would that mean?

A static mode for Groovy is something all those people like to have, that either think the compiler should do more checks or that Groovy should have more speed. Let us concentrate on compile time checks here. A common example is the misspelling of a method name or a method name gets changed. Let us take the example Chris Richardson in his Community One East 09 talk gave http://www.slideshare.net/chris.e.richardson/communityoneeast-09-dynamic-languages-the-next-big-thing-for-the-jvm-or-an-evolutionary-dead-end On slide 30 executeRequest got renamed to executeEc2Request and he complains about his test not picking up the problem, where it should have been a job of the compiler anyway.

Now let us imagine a language in which every method call, that does not point to a specific method, the compiler will create a call to method missing. In such a language a change of the signature of one method will most probably not lead to a compilation error. Instead the method call will now call method missing. In other words: One of the most important features, the check of a method call, will not work.

Still it wouldn't be worthless since in refactorings you could still have the method being changed automatically you might say. But this is an IDE-thing and not a compiler thing. As such it is only partially concern of the language and more of the IDE being able to do something in such cases.

class A {
 def bar() {1}
}
def a = new A()
a.bar()

Let us assume this is the code in your IDE and you want to refactor A.bar into A.foo. An IDE could through type inference know that "a" will be of type A and as such know that the call a.bar() would normally have called A.bar and that if I rename that method, the IDE should change a.bar() into a.foo(). Also if the example would have been

class A {
 def bar() {1}
}
def exec(x){x.bar()}
def a = new A()
exec(a)

the IDE would have had a much tougher job. But an IDE knows all the code that is involved, so it could try to identify all types that are used to call exec and then distill a common super type for it. So in the end it could still find out, that exec is called with A and since we refactor A.foo it could still find that it should be changed to A.bar. It is difficult, but not impossible for the IDE to set things right here as we can resolve the issue with just type inference. Let us go even one step further

class A {
 def bar() {1}
}
class B {
 def bar() {2]
}
def exec(x){x.bar()}
def a = new A()
exec(a)
def b = new B()
exec(b)

Here A and B have no common super type that contains the definition of bar, still a name change for A.bar would affect the program. The right thing here to do would be to also change the name of B.bar. If the IDE knows how exec is called and with what types these calls are made, it can find this out. There might be cases in which the IDE won't be able to find the right types, but I am positive that in most cases the IDE will be able to just do that. If such type inference is available, then auto completion for method signatures will work too. So the biggest use cases in an IDE, renaming and completion, can work out just fine in most cases even with Groovy.

So if method missing has no positive affect and multi methods are restricted, then what is left is the normal method call logic Java uses. And the question here is if that is worth it.

Java 5 features in Groovy

2007-10-13T21:32:00.000+02:00

Groovy 1.1 RC-1 has been released yesterday and I thought this is a perfect time for giving a complete list of all the Java 5 features that are available in Groovy and how much of it. I will not explain the Java 5 features itself, I will just show what you can do and what not. So let us start right away:

Variable Arguments

Groovy is able to use this for a long time now. Since Groovy does not know exactly which method it will invoke at compile time we use a vargs (variable arguments) when selecting a method for invocation. but not only the vargs Java has...I mean these things with the funny little triple dotted type as last parameter, no Groovy just requires an array there. so if you want to use a vargs method from Groovy just do it like in Java. vargs methods defined in Groovy can be used as such from Groovy too. Currently Groovy does not add the special modifier for vargs to the array, so it won't be a vargs method for Java, just a method taking an array as last parameter. I just filled an issue for that and it will be fixed before the 1.1 release. Ah yes, before I forget... of course you can use the triple dot notation instead of using an array in Groovy too. The tripple dot notation will be exactly the same as an array as last parameter.

To use vargs no java5 vm is needed from the groovy side.

New For-Loop

Groovy had always a for-loop in that style, only we don't use only the ":", we use allow also the usage of the keyword"in" at that place. The Groovy for-loop asks for an iterator by calling the iterator() method, which is available on any object. Groovy adds a dynamic iterator() method to the MetaClass for every class as long as the class does not define it on its own.

To use the new for-loop no java5 vm is needed from the groovy side.

Covariant Return Types

It took a while before Groovy was able to use these (available since 1.1 RC-1). Originally we wanted to be able to overload a method if the return type differs, not override it. But it seemed to make no sense to differ here from Java. So Groovy follows here Java now and if you write a subclass, that has a method that matches a method in the parent class in name and argument types, but has a different return type, then covariants kick in. If the the new method has a return type derived from the old method, then the new method overrides the old method. If it does not derive from that, then you get a compile time error. As an example:

class A{
Object foo(){1}
}
class B extends A{
String foo() {2}
}
def b=new B()
assert b.foo()==2

To use covariant return types no java5 vm is needed from the groovy side.

Annotations

Again these are available for some time now. The syntax is equal to Java. Even though you can define interfaces in Groovy you can not define an annotation in Groovy yet. But using Annotations in Groovy is basically the same as in Java.

If annotations are used Groovy needs to produce java5 bytecode.

Generics

Well... originally we didn't want to add them. They are complicated and look kind of surplus in a dynamic world. But some applications can make use of generics. for example List<book> books contains much more informations for a persistence layer like for example JPA. But maybe I should first explain that we do not support checks where type erasure is used. For example if you define a script like

List<String> list = ["a","b"]
list << 1

then Groovy won't complain. This is simply ignored. But if you use generics for a method, a field or in the class header (class X<A,B> extends Y<A> implents Z<B>), then Groovy will write the information in bytecode the same way as Java would do. That means if you use the Groovy class in Java, you have the same pain... ehm pleasure... what ever... as with a class defined in Java. With the 1.1 RC-1 release Groovy now also checks the header information, so you won't be able to write class X extends Map<String> anymore and get a compile time error now.

If generics are used Groovy needs to produce java5 bytecode. Groovy needs to be build with >java5 for this (only important if you build Groovy by yourself).

Generics+Covariant Return Types

I already covered these two, but I think the combination is worth an extra word. If you have a program like this

class A<T> {
T foo(T t) {1}
}
class B extends A<Long> {
String foo(Long l) {"2"}
}

then Groovy will do as Java and throw a compile time error. If you use raw types you will see that the following script for example works:

class A<T> {
T foo(T t) {1}
}
class B extends A {
String  foo(Long l) {"2"}
}
def b = new B()
assert b.foo(new Object( )) == 1
assert b.foo((Long) 1) == "2"

If used with raw types the method A#foo returns Object and takes Object. Since B#foo takes a Long Groovy will not care about the covariant return types and compile it. So you end up with two version of foo in B, one derived from A returning 1 and one from B returning 2. If the usage is like in the first case above, then A<Long> "kind of" defines a foo method taking a Long and returning a Long. I say "kind of", because the class does not really define that method, it is just the view B gets of A. Anyway, since B defines also a foo(Long) the covariant return type function kicks in, but finds that the return types String and Long are incompatible. So it will give a compile time error. The following script would for example compile:

class A<T> {
T foo(T t) {1}
}
class B extends A<Llon>> {
Long  foo(Long l) {2}
}
def b = new B()
assert b.foo((Long) 1) == 2

The foo defined in B will now override the foo defined in A.

Enums

Groovy now supports with 1.1-rc-1 also simple enums. With simple enums I mean you can declare them in Groovy, you can even declare them inside a nnormal class as long as you don't use references from the surrounding class. You can also define methods/fields/properties that will be in all enums. This should cover almost all of the features you need for enums. What we do not support is writing code that is special to one enum value. For example overwritng a method in the enum value, or addding a method/field/property. I have never seen such enums in real live, so I think it is not too important.... On the other hand I haven't seen much enums at all and my picture might be wrong. Anyway, Groovy is most possibly not going to support these enums with additional methods.
Groovy enums will differ a little from Java enums, they will implement GroovyObject and thus you can have a meta class per enum value. That relativates the missing feature with additional methods a bit I think.

If enums are used Groovy needs to produce java5 bytecode.

Asking The User!

I hope all the people out there, that like Groovy so much, will help us in finding bugs for these features. Some features are quite new, sometimes the tests lack fantasy (especially if written at 3 o'clock in then morning) and do not cover all needed cases. Sometimes something is simply misunderstood. I encourage all users to find the bugs and enter issues for them so we can deliver a pleasant 1.1 release to you.

About SwingBuilder

2007-07-18T21:20:00.000+02:00

Disclamer: I will use the term "closure" quite often here and experts will say they are not closures. I still call them closure in the sense, that they are instances of groovy.lang.Closure. So If I say closure I don't mean that functional thing ;)

This time I thought I should write some things about Groovy SwingBuilder and assumptions people seem to make about it.

groovy.util.BuilderSupport

First thing you need to know is that SwingBuilder is a builder... that might be obvious, but it implies, that if I do a method call in the builder structure, then the builder will handle that call and map the method names to certain actions.

Now in Groovy we have this class BuilderSupport, that you can use to map structures in a builder. Personally I don't like that class much, because the logic looks more complicated than needed, but it fits very general cases. Anyway, the class tries to map method calls in the builder structure to calls of createNode in the builder class. There are several of them, each responsible for a certain case controlled by your method call. The most important fact here is that if your last argument is a closure, then this closure will not be part of the creatNode call, instead the closure will be used by the builder directly. I guess it is best to show examples:

def builder = new MyBuilder()
builder.start {
 methodWithClosure {
   methodWithMap(foo:"bar")
 }
 methodWithNormalArgument("I am a argument I guess")
}

methodWithClosure a normal method call with one argument, that is the closure containing the method call with methodWithMap. methodWithClosure is now mapped to createNode(Object), the object there is the method name "methodWithClosure" as String. methodWithMap is mapped to createNode(Object,Map), where the first is again the method name and the map is our [foo:"bar"]. If you combine one normal argument and a map entries, then you get createNode(Object,Map,Object), where the last one contains your normal argument. And if there is no map and no closure, just a normal parameter, like with methodWithNormalArgument, then createNode(Object,Object) will be called. This logic supports only 1(!) normal argument, but that is enough in general.

After the createNode call of your choice is made the return value of that will be hold, I will call this currentNode. To connect the currentNode and its parent, which is done by setParent(currentNode,parentNode). now what is parentNode? Remember? we still have a closure to call. When we do, then our currentNode becomes parentNode and the new currentNode will have a parent. So the first time setParent is called we are not in a closure that belongs to the builder, which means the parentNode is null. If you would build a tree using this logic, then you would build the tree starting with the root and then adding node by node in I think it is called preorder traversal.

Architecture of SwingBuilder

SwingBuilder is making use of these methods in BuilderSupport. For each method call SwingBuilder creates a new instance of a bean we specified with the method call. So

frame(title:"I am a JFrame")

will create a new JFrame instance and set the property title. And as we just learned

frame(title:"I am a JFrame"){
 label(text:"I am a JLabel")
}

will also create the JFrame, the property title will be set again, then the closure will be executed causing the label method to create a JLabel and the text property on that label is set. After that setParent is called with the first parameter being the JLabel and the second parameter being the JFrame.

The logic we stored in setParent will connect our frame with the label by frame.getContentPane().add(label). SwingBuilder#setParent knows several cases and handles adding a JMenuBar to a JFrame different from adding a JLable. This method and helper are around 100 lines, about 20% of SwingBuilder source.

So basically SwingBuilder is a builder that maps method calls to bean creation actions, using map arguments to init the beans and the closures to connect the created beans. It using a mapping method name -> bean class and contains itself nearly no methods you call when using SwingBuilder.

Names supported by SwingBuilder

If you are not sure if SwingBuilder supports a swing widget, just remove the J, keep the next letter in lower case and try it. For example JEditorPane becomes editorPane, JSplitPane becomes splitPane (both supported). But SwingBuilder does not only know widgets, it does also know layouts. there is usually no 'J', so just use the next letter in lower case, as in gridBagLayout, flowLayout or others. You can use the layout as normal method causing the layout property of the container to be set. I have often seen code like:

frame(layout:new FlowLayout()) {
 label(text:"1")
 label(text:"2")
}

but you can write that also as

frame() {
 flowLayout()
 label(text:"1")
 label(text:"2")
}

I like this version much better, because you do not need to import FlowLayout and can give the layout some options while keeping the frame call simple. Groovy supports all the normal layouts, even Box layout. Another special thing is maybe the method gbc, which is the same as the method gridBagCosntraints, which maps to GridBagConstraints. Maybe I should also mention TableLayout, which tries to implement the layout you know from the table tag in html. It needs tr and td calls to place the componentes... really just like in html. Take a look at alpahbetic widget list to get an ideas what you can do.

Another important link is extending SwingBuilder. It does not mention the possibility of simply subclassing the class SwingBuilder, but that should be obvious and was done for example by SwingXBuilder.

All in all it is a bit difficult to provide a documentation for SwingBuilder, because you still need to learn swing, SwingBuilder doesn't help you with that. And then it is just connecting instances of classes... For example when people ask how to attach an action to a JButton and I tell them to assign a closurey to the actionPerformed property, then I am not talking about a special property, actionPerformed is defined by the bean specification and assigning a closure to that property is a normal thing in Groovy.

New things in 1.1-beta2

We got complains that if I do

def frame = swing.frame(...) {
...
}
frame.pack()
frame.visible = true

that the resulting gui will not be constructed in the EDT thread, but in the normal main thread. Now I am no Swing expert and I always assumed it makes no difference, but it seems that future changes in Java will need you to change in the EDT. And of course it is more clean that way too. So we eneded with adding two methods, the first is edt, which causes the attached closure to be executed while in EDT. the code looks then like

swing.edt {
 def frame = frame(...) {
  ...
 }
 frame.pack()
 frame.visible = true
}

unlike many other methods available in SwingBuilder the edt method is a real method and no registered widget. the other method is static and called build. It will automatically create a new SwingBuilder instance and call the attached closure with that instance as parameter

SwingBuilder.build {
 def frame = frame(...) {
   ...
 }
 frame.pack()
 frame.visible = true
}

build uses the edt method, so we build the GUI while in the EDT thread, just like before.

Future Plans

I think SwingBuilder is already a nice piece of work, but its evolution might not stop here. Currently I am thinking about integrating a Binding framework. That would some update logic to SwingBuilder, something you have to do all by yourself atm. For example imagine a label and a button and each time you press the button the label text should show a higher number. What do you do? You use a closure as actionPerformed for your button that increases a number and sets a new text for the label...

import groovy.swing.*
import groovy.swing.impl.*;
import javax.swing.border.EmptyBorder
import javax.swing.WindowConstants

def numClicks = 0
def state = {"Number of button clicks: $numClicks"}
def label
SwingBuilder.build {
def frame = frame (
    title: "SwingBuilder Label Update",
    defaultCloseOperation: WindowConstants.EXIT_ON_CLOSE
){
    panel (border: new EmptyBorder(30, 30, 30, 30)) {
       gridLayout(rows: 2, columns: 1, vgap: 10)
       button (text: "I'm a button!",
               mnemonic: "I",
               actionPerformed: {numClicks ++; label.text = state()})
       label = label (text: state())
   }
}
frame.pack()
frame.visible = true
}

note the closure state, that is called at different places? that's quite ugly I think. With a binding framework the code might become

import groovy.swing.*
import groovy.swing.impl.*;
import javax.swing.border.EmptyBorder
import javax.swing.WindowConstants

def model = new BindModel(numClicks:0)

SwingBuilder.build {
 def frame = frame (
     title: "Binding and SwingBuilder Test",
     defaultCloseOperation: WindowConstants.EXIT_ON_CLOSE
 ){
     panel (border: new EmptyBorder(30, 30, 30, 30)) {
       gridLayout(rows: 2, columns: 1, vgap: 10)
       button (text: "I'm a button!",
               mnemonic: "I",
               actionPerformed: {model.numClicks ++})
       label (text: bind("Number of button clicks: $model.numClicks"))
   }
 }
 frame.pack()
 frame.visible = true
}

as you see the need to keep an external closure vanished along with the need to keep a reference to the label. the way it will be used in the end is not yet sure, this is just a sketch based on a simple implementation I made.

While doing my "research" in this area I noticed that binding frameworks are not that well known. At last by the people I know. I myself didn't here about that before, but I am usually not doing much with swing. So I got a bit puzzled why people don't know these things if they can save so much code... but then I saw it. Some of these frameworks are producing rather cryptic code, that makes sense for the framework but is just plain hard to read. Your former models are now hidden in abstract constructs, but you still need to connect the things. So I guess SwingBuilder could find a more "natural" way by hiding all these things.

We will see.

Joint Compilation in Groovy

2007-07-04T18:16:00.000+02:00

Note: This is not implemented since today, it is already some weeks old, but there was no information about it on the net... So I wrote this

When working with Languages like Groovy you naturally mix Groovy and Java all the time. But the problem then arises to compile the resulting monster. Especially the Eclipse plugin for Groovy makes you sometimes think your project will compile outside as nice as inside Eclipse. But it might not. Consider for example this case

class A {
  B b;
}
class B extends A {}

And think A is written in Groovy, but B is written in Java, or the other way, A is written in Java, but B is written in Groovy. It is clear, that when you compile B, you need class A as well, because the class might be final or abstract or other things that need to be checked. But when you compile A, you will see that it refers B, which means you need to compile B as well. Now if this is one compiler we have no problem, but A and B are written in different languages with compilers not sharing their class information. That means we are stuck. And while this example looks a bit artificial, you get very fast into this situation in a larger project. Who would keep track of not referencing Groovy classes from the Java side to get the compilation done? It's disturbing.

Solutions?

Now the one solution would be to let the compiler share their class data. The Groovy compiler is able to do that, the eclipse compiler is able to do that, we might see a bright future for this in the future. But for example JavaC is not able to do this. At last I don't know how.

Another way is to create stubs for the Groovy classes, run the Java compiler with these stubs, then run the Groovy compiler and overwrite the generated stubs. Alex Tkachman was so free to show us Groovy people how to do this and provided a patch, we could us as base to get this version of the compilation running.

How it Works:

The details of the stub generation are not so important I think, they are created as Java files from the parse tree the Groovy compiler provides in a temporary directory and then feed to JavaC along with the normal Java files. You don't need to start the compiler yourself, the Groovy compiler will do this for you right after the parse phase and then continue with its normal compilation process.

Controlling JavaC:

Since we now have a combined compiler we of course want to use it in our build, but the problem is that javac would by default created a wrong bytecode version for us, we need 1.4 compatible bytecode, so source and target options are needed at last. I then decided to forward the options to the compiler from the command line of Groovy. So if you do

groovyc *.groovy *.java -j -Jsource=1.4 -Jtarget=1.4

You will get the java files compiled for 1.4. -j turns the joint compilation on , the -J parts are gving key-value pairs to the compiler. Using Options without value is also possible using the -F option, just without the equals part like -Fdeprecation. Anything the JavaC compiler supports can be dropped in there... Of course some special options like the VM memory size would not make sense since the VM is already created.

For the GroovyC Ant task the picture is a bit different. first I thought about a way to generically define attributes for the GroovyC task I can forward to JavaC... But me not being the Ant expert I gave this up and decided to do the following work around

<echo message="Groovyc of test code."/>
<java classname="org.codehaus.groovy.ant.Groovyc" fork="true" maxmemory="128M">
  <classpath>
    <pathelement path="${mainClassesDirectory}"/>
    <pathelement path="${testClassesDirectory}"/>
    <path refid="testPath"/>
  </classpath>
  <arg value="${testClassesDirectory}"/>
  <arg value="${testSourceDirectory}"/>
  <arg value="-j"/>
  <arg value="-Jsource=1.4"/>
  <arg value="-Jtarget=1.4"/>
</java>

Oh, that reminds me that the GroovyC task needs a fork ability. Anyway, that's when using the ant task from the command line. If you want to use it normally, then

<groovyc
  srcdir="${mainSourceDirectory}" destdir="${mainClassesDirectory}"
  classpathref="groovyMainCompileDependencies"
  jointCompilationOptions="-j -Jsource=1.4 -Jtarget=1.4"
/>

can be used. Same game as on the command line.

What this solution can't do:

Yes, there is a downside. Ok, I think it is already a downside that we have to use a temporary directory, but another one is that we need to know all files we want to compiler before compilation. I don't think that is a problem when running a ant or maven based built, but for the typical usage on the command line, where you just compile your main class and the compiler will get a hold on all further classes will not work. To be more specific, it will not work when the Groovy compiler needs to get an additional Groovy class and the Java compiler would need that class too. That's because in this case no stub will be created and thus the java compiler will fail telling you it can't find a that class. On the other hand GroovyC works with the resulting class files, so if a Groovy class refers a Java class and JavaC did not compile it, then GroovyC won't be able to compile it either... well, ok, just give the compiler all needed files ;)

Future Work:

The current implementation uses JavaC directly a nice framework would be nice here to have more than just this compiler. And there is for example JCI, but JCI seems not to support options... well we need to take another look at it, maybe it supports enough. On the other hand I am thinking about integrating the JavaC task from ant. in that case we could maybe use the normal task as nested element (with some tweaks) and have all the abstraction to different compilers ant allows. Of course by directly using the Eclipse compiler (it is usable outside the IDE) we could let the compiler share class data and then compile files that are not part of the file list given at runtime.

But I think the ant task version will make it. the work around with the "-j -J -F" options might then vanish.

But none the less, have fun with the upcoming

Groovy 1.1 beta 2

Visitor Pattern in Groovy

2007-03-27T00:19:00.000+02:00

Shame on me, such a long time without update. I was too busy it seems. I just wrote a small article about how to use the visitor pattern in Groovy. Enjoy the Visitor Pattern in Groovy and ignore the misspellings ;)

If you think I should mention other patterns as well, tell me please.

AST macros and mixins

2007-01-06T23:47:00.000+01:00

Hi all,

I know my last article was written some time ago, but I was busy with Groovy 1.0 bug times. Anyway. This is another part of my "beyond Groovy 1.0" series, only that we are now officially in that era and I think I no longer need to prefix the title with that.

So my next wild idea is about a macro mechanism for Groovy. But not the kind of macros you know from C or such, no I think of macros rewriting parts of the AST used by our compiler. So the idea is not really something new, it is something the compiler does already provide. The new thing is to let the compiler do the integration steps for you automatically. No need to add a PhaseOperation to the compiler or such.

No, I have not really thought of a syntax yet, let us think about this as a case study and not as a final draft or something. Ok, back to title. let us say, there is a statement like the import statement and let us call it macro. So when we want to use a macro we simply tell the compiler this by:

import macro Foo

or by

import macro Foo as bar

and let us assume the compiler can use them like a method:

bar {
  // a block containing code
}

looks much like the dynamic features we already provide, but in fact the compiler recognizes "bar" as macro and does not produces a method call with a closure instance for this, no, instead the compiler takes the logic provided by our Foo class and applies it to this part of the code.

Why should we do that? One thing I always disliked in Groovy is the huge amount of keywords. Do they really have to be keywords? One example for this is synchronized. Usually you do something like:

synchronized (object) {
  // code using object
}

But isn't that almost our macro "bar" from before? It is, not only almost. In fact it is really alike, the compiler would transform that into

synchronized (object, {  ...t })

just like the bar example

bar ({...})

it is no different. So that would mean we could remove the keywords "for", "while", "assert" and "synchronized". Ok, not much... but still it is a step forward. That would still mean that the do-while loop is not supported, but maybe there is a solution for this too.

How would Foo possibly look? Well I guess something like:

class Foo {
  static phase = Phases.CONVERSION
  def visit(ASTNode[] contextPath, SourceUnit source)  {
    // ... transformation code here
  }
}

The compiler would take a look at the Foo class ensure that there is a default constructor and a static field named phase. Then the compiler would create a new instance of the class and add it to the phase operations it already provides, plus a filter to ensure only the correct classes are affected by this. So if the compiler compiles the file it goes along the AST until it finds the macro and then does execute it.

That's pretty easy to implement, really no big deal, but the impact might be big. I think such a construct would be the first step to unify the Groovy grammar parts and to remove special constructs. Well, the thing really needed for this is a way to exchange/remove statements in the AST. That is currently not possible. It is only possible to transform expressions, but not statements. But well, that is no problem we can't solve, is it?

The mentioned keywords earlier could be seen as macros themself. That doesn't mean that we can remove these statements from the AST, but it would mean a programmer could define additional keywords as he likes. For example a macro transforming a closure into a SQL statement. We already have this, but wouldn't it be exciting to have that at compile time? I mean with different checks and without runtime impact. the macro could use any expression, for example

  sql {
    select row1,row2
    from tx
    where id==26
  }

I am also thinking about SODA queries or queries for GORM. The above can't be interpreted using a normal closure, because I used 3 statements that are interpreted as method calls. There is for one select(row1,row2), then from(tx) and where(id==26). So to correctly interpret this we have to use the getClassNode method in MetaClass, but that relies on runtime analyzes of the source file atm. Yes, we are going to change that, but it means to store the source inside the closure in compressed form or else we wouldn't be able to ensure the correct interpretation. With these AST makros we are able to compile that directly into the goal construction. I mean even different SQL dialects could be covered by this.

But SQL is just a example. I used it only to illustrate the abilities a little. I know that this article does only give a little idea about how it might work and writing this doesn't mean it will go into any version of Groovy. But I think the idea is quite simple and yet powerful.

But this is not all... AST handling is complicated and not really something that would make your day. so I think about going a step further, in fact I am planining to adding this pretty soon. I am talking about AST level mixins.

"Ok, what's that again?" you may ask. I think of defining a chunk of code as normal groovy script and a defined way ofr a phase operation to use that code to modify the AST. This chunk of code does itself not contain any AST handling code, it will just be used as archetype for the compiler to produce the real code. This might be a simple thing as

class X {
  long id
}

causing the compiler to add a field named id to the currently processed classNode, or something more complex. Well we have a problem when it comes down to bytecode instruction. For example if we want to replace a for loop we would have to define jumps. So, as long as there is no mechanism to represent this, there is no way this transformation by example will work for all kinds of macros too. But well, this is like doing research. At the beginning you have only a vague idea of how things might work. you do experiments to see if your expectations are correct and you develop new ideas if they do not or if the experiments are uncovering no facts.

But still, the idea is exciting somehow. I am sure another language does already have a construct like that... well I am thinking about LISP here. I wouldn't dare to compare this to the macros in LISP (I am talking about Common LISP), but possibly they will become alike in the future after doing same practical work. LISP does have a more easy part here, because LISP does have a much simpler syntax than Groovy. Doing structural work in such a language is naturally more enjoyable than in a complex language like Groovy. But I still think it can be done.

Btw, this is no replacement for DSLs looking like natural language, but that is more a syntax issue, than an issue of AST macros.

Groovy 1.0 is here!

2007-01-03T16:06:00.000+01:00

16 hours ago Groovy 1.0 was released. The work of 3,5 years. So many discussions, so many code was produced, removed and reworked.

In my eyes this is not the final version of Groovy, nor is it perfect. But it is important to give a stable version to the world out there. I hope many people will like that Groovy. But that doesn't mean Groovy wouldn't evolve anymore. There are many ideas we can try out now since we do not any longer have to focus on the release. In the release notes you will find not much this time, we fixed a number of bugs, but these are more like minor adjustments. But Groovy In Action and GroovyGrail.com do show increasing interest.

The next months will be spent to improve the documentation and to really start the JSR process. Then Groovy will be really like a standard and get a good documentation.

New Layout

2007-01-03T14:25:00.000+01:00

Since the changes on blogger.com my blog wasn't working properly, so I decided it is time to get a new layout. Making custom layouts here is not the most nice thing to do, because I always have problems with layout support pictures. But putting them in a post and reusing the link to the image allows me to have layout pictures as the two on top, that are nearly invisible. The layout is not perfect and my javascript for keywords is no longer working, but I think it is a good start. A multicolumn layout made in css can really be difficult to do. This page currently consists of about 4 columns, 2 for images, 1 for the sidebar and 1 for the content itself.

ChitChat with the Groovy Compiler

2006-11-08T16:50:00.000+01:00

Ever wished to be able to influence the compiling process in Java?

To do this you usually don't influence the compiler directly, you use AOP tools with strange syntax rules and extra files for configuration or you are working on the bytecode level and sue the instrumentation API. But these technics have their limits. To be able to instrument a class you need a ready compiled, the verifier passing class. What if you can't produce such a class?

But ok, let us go some steps back. Programming by convention is in all mouths, so let us look at adding a method depending on the class name. I guess people would use a inter-type declaration in AspectJ to add methods to a certain class. The problem here is, we don't know the class before. We react to the name of the class and only if the name fits, we add the method.

Another thing is adding imports. There is no point in adding imports in a class file, all class names in a class file are already resolved against the imports. Adding another import would be pointless. But if you want to let people write a small peace of code where they can forget about importing every class they use, then we are talking about adding something to what the compiler gets as input. The most easy solution would be to use some kind of macro system. Expand the macros and give the result in the compiler. But then, what is about the line numbering? Exceptions will show useless information about the position in the source file, compiler errors will point to invalid places. That's a bit bad.

Compiler are normally working on an AST, an abstract syntax tree. Such a tree represents the source as a tree, each node is a part logic created by the source. For example, the expression "foo.bar()" is in Groovy a method call expression with foo as the object the call is made on, bar the name of the method and an empty number of arguments. Such a method call expression is a node in the AST. I am afraid that explaining all the expression types and statements in Groovy would be too much for this small article. I suggest to look into the packages org.codehaus.groovy.ast.expr, org.codehaus.groovy.ast.stmt and org.codehaus.groovy.ast. Of course having around 35 expressions and 19 statements doesn't make this task very nice, but it is needed. Groovy uses the visitor pattern to visit the classes. That makes it a bit problematic for people how are not used to that. but it really is nice to have that.

Each phase of the compilation process after the AST is created consists of a visitor travaling along the AST replacing or modifying nodes. Btw, the Groovy AST is for example used in Eclipse for the outline.

Now imagine you could change the AST by yourself. Rename methods, add code, add methods... yes, right, we wanted to add a method if the class name fullfills a certain convention.

Now the Groovy compiler offers here some hooks you can use to change what the compiler will produce. What I will explain now counts only for the case that a file is compiled using the GroovyClassLoader, for example because a script is found in the classpath or because we use the GroovyShell to execute a script. The basic process to creat a compilation unit, that will hold the AST and other information, add a source to the compilation unit and then start the compilation. By subclassing GroovyClassLoader you can overwrite createCompilationUnit(CompilerConfiguration, CodeSource), the very first step. The CompilationUnit is really our compiler. It holds all the logic. So if you return a subclass of CompilationUnit here, you would return a custom compiler. But I don't want to go this far here.

I talked about hooks, but subclassing CompilationUnit would be a bit big possibly. I talked also about the phases a compiler goes through and that's where CompilationUnit offers hooks. The addPhaseOperation methods do allow you to give the compiler additional logic for a phase of your choice. In Groovy these phses are:

initialization: open files setting up the environment and such
parsing: use the grammar to produce tree of tokens repersenting the source
conversion: make a real AST out of these token trees
semantic analysis: check for all things the grammar can't check for, resolv classes and other things
canonicalization: complete the AST
instruction selection: choose isntruction set, for example java5 or pre java5
class generation: create the binaries in memory
output: write the binaries to the file system
finalization: cleanup

not all these phases are really filled, the canonicalization step is currently empty, but that is a implementation detail. If we want to operate on the the AST then we need one, thus the conversion phase is the most early phase to do that. You have to decide if your operation is about changing a certain source file or a certain class. Each file may contain multiple classes, so this is not equal. For example if we want to add an method, then it makes not much sense to work on the source (SourceUnit) but on the ClassNodes instead.

In our example we would call the addPhaseOperation(PrimaryClassNodeOperation, int) method. You may wonder about the name primary class node operation. A class node represents a class, but in the compilation process there are two kinds of classes: precompiled classes and classes we want to compile. These primary class node are classes we want to compile.

If you know the AST then the next steps are quite easy. For example:


class MyOperation extends PrimaryClassNodeOperation {
        public void call(SourceUnit source, GeneratorContext context, ClassNode classNode) throws CompilationFailedException {
               classNode.addMethod("foo", OpCodes.PUBLIC, null, null, null,code);
        }
}

This would add a method named foo to every class we compile, foo takes no arguments, is public and has the return type Object. The contents of the method are stored in "code". I haven't specified that here to keep the article not totally complicated. Ok, again let us think back to the orignal task of adding a method if the class name fullfills a certain convention. For example if a class is XyzFoo, then add a method xyz, if the class is AbcFoo, then abc. Ok, that is a bit crazy, but well, just for the demonstration of the flexibility:


import org.codehaus.groovy.ast.ClassNode;
import org.codehaus.groovy.ast.stmt.BlockStatement;
import org.codehaus.groovy.classgen.GeneratorContext;
import org.codehaus.groovy.control.CompilationFailedException;
import org.codehaus.groovy.control.CompilationUnit;
import org.codehaus.groovy.control.CompilerConfiguration;
import org.codehaus.groovy.control.Phases;
import org.codehaus.groovy.control.SourceUnit;
import org.codehaus.groovy.control.CompilationUnit.PrimaryClassNodeOperation;
import org.objectweb.asm.Opcodes;

import groovy.lang.GroovyClassLoader;

class MyGroovy extends GroovyClassLoader {
    protected CompilationUnit createCompilationUnit(CompilerConfiguration config, CodeSource source) {
        CompilationUnit cu = super.createCompilationUnit(config, source);
        cu.addPhaseOperation(new PrimaryClassNodeOperation() {
            public void call(SourceUnit source, GeneratorContext context, ClassNode classNode) throws CompilationFailedException {
                String name = classNode.getName();
                if (name.endsWith("Foo") && name.length()>3) {
                    name = name.substring(0,name.length()-3);
                    BlockStatement code = new BlockStatement();
                    classNode.addMethod(name,Opcodes.ACC_PUBLIC,null,null,null,code);
                }
            }
        },Phases.CONVERSION);
        return cu;
    }
}

Grails does use this mecahnism to add the toString and hashcode methods as well as the id field in domain classes.

Another easy example is adding a import. But since we don't add an import to a specific class, but to all classes in a file, we are operation on the SourceUnit (representation of the source file) so we use a SourceUnitOperation.


        cu.addPhaseOperation(new SourceUnitOperation() {
            public void call(SourceUnit source) throws CompilationFailedException {
                source.getAST().addImport("User",ClassHelper.make("my.company.UserDAO"));
            }
        },Phases.CONVERSION);

here we add an import for UserDAO and make the class known as User inside the file we compile. That is comparable to "import my.company.UserDAO as User" in Groovy.

As you may see this is much more than aspect oriented programming could do. You can place arbitary logic inside the compiler and modify the ASTs. If you want to do more complex operations or more local ones, then you might have to really go into the AST. I suggest to use the LabelVerifier as entry point for this. A little debugging helps you to understand how the AST looks like and how the visting works. I plan to write a big documentation about these things, but this will be on the Groovy wiki then. The purpose of this article is just to give you a insight in the internals of the compilation and how it can be influenced from the outside. Without knowing the AST there is not really much you can do. It is not that the AST is very complicated or bad documented, but learning about over 60 classes isn't done in a short form. The biggest exmaple of AST traversion is AsmClassGenerator, the part in the compiler creating the binaries. But it is a big class, not too good to learn.

My plans for the future are to add an mechansim that allows to write some text that gets mixed into the classes the compiler produces. In the import example above you would maybe write something like

cu.mixinPerSource("import my.company.UserDAO as User")

. Or in the method example:

cu.mixinPerClass {"def ${it[0..-3]}(){}"}

. But it looks like this will ahve to wait for post 1.0

You could also use the AST to transform any DSL containing binary operations such as "&" and "<" into a builder or a method based version. You could transform it into a database request or whatever. There are many possible usages of this. And in fact something like that is already used in GroovySQL - but well hidden ;)

So might I ask again: Ever wished to be able to influence the compiling process in Java?

Concurrent Loop Excetion - Comparing Java and Groovy

2006-10-19T23:39:00.000+02:00

Neal showed in his blog a new example of how closures in java could look like, in real world usage.

If I would program this in Groovy (which runs on the JVM btw), I would maybe write this:


def getAttendees() {
  return getResponses()
         .findAll{it.mayAttend()}.collectAsync(threadPool) {it.attendee}
}

Right, that's nearly a one-liner. Of course that code is not fully equal to Neal's program, because mayAttend() is executed synchronous, and Groovy doesn't have collectAsync. But we could write such a method and use it in a category. Maybe the method would look like:


def collectAsync(threadPool, Collection col, Closure action) {
  def result = []
  def tasks = col.collect {
    try{
      synchronized (result) {
        result << action.call(it)
      } catch (Throwable ex) {
        LOGGER.log(Level.SEVERE, "Uncaught Exception", ex);
      }
  }
  threadPool.invokeAll(tasks)
  if (!result) return null
  return result
}

For the non Groovy people some comments.. Groovy has an dynamic typing system, native literals for lists (see result), operator overloading (<<), closures and of course runs on the jvm with very good integration. Anyway, I am wrapping the original closure in another closure for the logging. This removes some code from the usage. If we where funny, we could use an additonal closure for the logging. "it" is a shortcut for the current value given to the closure. "it.attendee" is equal to "it.getAttendee()", Groovy does have logic for beans.

So how does it work? I use findAll to filter out all EventResponses that don't attend. findAll returns a new list containing only the important responses. collectAsync creates a list of closures, each closure representing a task, and gives that to my threadpool as suggested in the lower part of the article - of course it wasn't with closures there. Anyway, during execution each task calls the action closure and writes in "result". The nice thing is, that this function could be a library function used somewhere. There is no reference to something special besides the logger and threadPool. In fact the code is short only if the method is a library function. It is no use to tell people how well closures might look, if I compare library functions and their usage with raw code. The Groovy version is short because of the lack of typing we are using and much other factors. If I would have to add types then it would become much larger. For example even if there where a findAll method working on a collection and able to take such a closure, I still would have to define the list element type... possibly findAll<EventResponse>{it.mayAttend()} Then, "it" is not typed. Ok then it might look like: findAll{EventResponse it -> it.mayAttend()}. the Groovy syntax for defining parameters is a little different form the closure proposal, don't care to much about that. Next is collectAsync.. collectAsync<Principal>(threadPool){EventResponse it -> it.attendee} That makes:


getResponses().findAll{
  EventResponse it -> it.mayAttend()
}.collectAsync<Principal>(threadPool){
  EventResponse it -> it.attendee
}

Could still be seen as a one-liner, but it is a bit too big for my taste. Of course collectAsync would have to be changed too:


  Collection<R> collectAsync(threadPool, Collection<T> col,
                                Closure<T,T> action) {
  List<R> result = []
  List<closure<Void>> tasks = col.collect<> {
    try{
      synchronized (result) {
        result << action.call(it)
      }
    } catch (Throwable ex) {
     LOGGER.log(Level.SEVERE, "Uncaught Exception", ex)
    }
  }   
  threadPool.invokeAll(tasks)   
  if (!result) return null
  return result 
}

And this version is just a wish... I mean [] must be replaced by new ArrayList<R>(), !result with isEmpty(). Then we still are not in java, because we need to redefine our generic closure type to Closure1 for the closure taking one argument and Closure0 for closure taking no arguments. Next downside is that findAll and collectAsnyc might not be defined on Collection to keep compatibility (that is no problem in Groovy, as Groovy does have a special mechanism to handle this). So findAll might be a static import, and collectAsync is possibly defined in whatever threadPool is. Then the code may look like:


return threadPool.collectAsync<Principal>(
  findAll(getResponses()) {
    EventResponse it -> it.mayAttend()
  }), {
    EventResponse it -> it.attendee
  })

which is much worse to read due to the new ordering...


List<EventResponse> filtered = findAll(getResponses()) { 
  EventResponse it -> it.mayAttend()
}
return threadPool.collectAsync<Principal>(filtered){
  EventResponse it -> it.attendee
}

looks much better again.. Of course we are not using the syntax suggested in the closure proposal.. then it would look like:


List<EventResponse> filtered = findAll(getResponses(),
          EventResponse(EventResponse it) {it.mayAttend()});
return threadPool.collectAsync<Principal>(filtered, 
          Principal(EventResponse it) {it.getAttendee()})

And a version using inner classes would like:


List<EventResponse> filtered = findAll(getResponses(),
   new Closure1<EventResponse,EventResponse) {
     EventResponse call(Eventresponse it) {it.mayAttend()}
   });
return threadPool.collectAsync<Principal>(filtered, 
  new Closure1<Principal,EventResponse) {
    Principal call(EventResponse){it.getAttendee()}
  });

So we really gain something from the closure proposal - if compared to the current usage of inner classes, but then if I compare that to the very short Groovy version... Java could be shorter here if more type inference was used. It is not the shortness of the Groovy version I like so much, it is the clear code.

But another thing should be made clear with this: the programming style in a language like Groovy is different from the style normally used in Java. I mean compare my endresult with the version without closures at the end of the article. I think I would prefere the other version. Even if I "beautify" the code by using proper identation and such things, I am not gaining much. Strange, isn't it? Java prevents us from defining usefull library functions because of its syntax bloat. Even though there is neraly no logic left in the method itself and all is done by library functions, I have to write so much code for the little compiler... annoying would I say.

I really hope they decide to add more type inheritance. Of course I know that type inference is no easy task for the compiler builder, but it should be doable. Java has way to go here I guess.

Annotations in Groovy

2006-09-22T17:10:00.000+02:00

Annotations are thought for providing MetaData.

Such MetaData can be used by the compiler or other processing tools by using the source, the bytecode or at runtime. Annotations are not thought to have an semantic effect.... I always have to tell me this... Somehow I really have a problem with that. Anyway, Groovy doesn't have them. And Groovy won't get Annoatations in 1.0. But as soon as 1.0 is out we want to add them. Tools working with annoatations will then most likely work for Groovy too.

The new JUnit is an example of why Groovy should have annotations. It is not sure yet what set of default annotations Groovy will support, but people will be able to write annotations in Groovy. How much of the enhancements though Groovy can be used in annotations is open, we need to look at what the bytecode allows and if we can go around it... and if we want to do that.

And there is Hibernate... avoiding to persist the metaClass field is sometimes very important. We had several problems with serialization of that property, because the MetaClass is not serializeable and so the default serialization might fail. Using bean serialization might make things even worse. And Hibernate...


@Entity
class Customer implements Serializable {

    @Transient
    MetaClass metaClass
    
    @Id
    Long id

    String firstName
    String lastName
    Date birthday

    @Transient
    Integer age

    @Embedded
    private Address homeAddress

    @OneToMany(cascade=CascadeType.ALL)
    @JoinColumn(name="CUSTOMER_ID")
    Set orders
}

I "stole" that example form the Hibernate page and added the transient property for metaClass. Of course you don't have to add the getter and setter methods, as they are already generated by Groovy. Just add buisness code and be lucky ;)

And of course there are more tools using annoatations and I am sure there will be much more of them in the future. I am very curious if other dynamic languages on the JVM will go that way to. But as most scripting languages on the jvm do not generate compatible class objects it might be out of scope for many of them.

Groovy on Speed - a "fast mode" for Groovy

2006-09-11T11:32:00.000+02:00

fast mode? What's that?

The Problem:
Did you ever try to meassure the performance of groovy? It is a quite difficult task. Because of the high integration of Groovy and Java, you are contantly calling Java and Groovy. For example think of some time consuming operations in a Java library. The Groovy part would probably call the library and then return the result. The overhead through Groovy is minimal.

Calling a Groovy method doing some calculations on the other hand might result in bad performance. The solution is quite simple. Identify these parts and write them in Java, by delegating to a Java library or subclass the Groovy class in Java and overwrite the methods in Java.

Of course this is a possibility. But it means you have to use Java for these parts. It means you can't use Groovy syntax for the operations. That sure is no problem for people seeing Groovy as frontend... somewhere... occasionally used, but no essential part. It does indeed become a problem for groovy evangelists.

What exactly is the Problem?
What Goovy makes slower than Java is it's ability to add/intercept methods at runtime and it's slightly different type conversion model. A method invocation in Groovy consists usually of several normal method calls, where the arguments are stored in a array, the classes of the arguments must be retrieved, a key is generated out of them, a hashmap is used to lookup the method and if that fails, then we have to test the available methods for compatible methods, select one of the methods based on the runtime type, create a key for the hasmap and then in the end, do a reflection like call on the method.

Benchmarks usually use the same typs of arguments, so the hasmap kicks in and allows a "fast" method selection. Only counting the method calls, you get around 4 calls before you are able to look in the Hashmap. Then you create the key, do the lookup in the map... given all that I think it is quite a wonder Groovy has such a good performance!

How to make it even faster?
The way to make something faster is either to find an algorithm with a better runtime complexity, or to reduce the constants. If we would not have to create the Object[], we might be able to reduce the costs very much.. I made some meassurments and it seems that invoking a method taking 3 arguments is 3 times slower than a method taking no arguments. That's because we not only have to create the Object[] for the 3 arguments, but we also have three objects which types we need to get, the key becomes longer, and as the key is a String there is a coherence of the length of the string and the number of arguments.

Java does avoid that. Java does select the method at compile time, so there is no Object[], no type extraction and such. Of course the VM checks the types of the parameters, but in case of Groovy the VM does have to do that more than once. The end of the call in Groovy is the same as in Java, but there has much action taken place before. So if we were able to select a method at compile time in groovy, we could avoid that problem completly.

What does it mean?
It means Groovy would be as fast as Java! But that is no general solution. Dynamic typing doesn't allow you to choose the methods at runtime. And the MetaClass facilities won't help here too. so you are changing the semantics of a method call when chanign the typing from dynamic to static.

So what is the Fast Mode then?
The idea is not do dynamic typing every where. In fact there was already a proposal going into that direction. It was even in the codebase, but it was a global switch. I think that is the wrong solution. My solution is to let the programer decide where he wants to have that. For simplicity I will talk about an annotation here, but there was no decision on what to use. Anyway, I would use this annotation for a class, for a method, for a block of code or a single statement. Each so marked part would loose dynamic typing and letting the compiler use static typing to choose the method. This means that the MetaClass is ignored. It does not mean, that we loose any of the addons to the jdk we made. It will still be possible to use "each" or native syntax for BigDecimals, lists and maps. It doesn't mean that you get back checked exceptions or such. I am not sure yet if the compiler should forbid the mixed usage of static typed and dynamic typed arguments, or if it simply should fall back to the normal invocation mode.. Anyway, so instead of asking the MetaClass to tranform the type and make the call the compiler, we will add code to make the type transformation and the method call based on the static types. We loose multimethods then, and we loose the ability to catch unknown methods and properties! So we loose one of the most appealing feature of Groovy. And given the fact, that a builder depends on them we loose them then too. But as you can limit the usage of the fast mode very much it is no problem at all.

Of course this adds much to the complexity of the language.

What about ivokedynamic?
invokedynamic is the great hope of all dynamic languages on the JVM. If it helps us to not to create the Object[] and if it helps us to avoid to create the hasmap key and if it helps us to avoid the type tranformations, then it might be a much better solution than my fast mode. But even if it does all this, it won't be in Java6. It is Java7 at best. And it is not yet defined. The work for this is still in the early stages and while I hope the result will be very good, it is no solution for now.

The Goal
The idea of the "fast mode" is to provide a way to give the maximum performance at certain places. Then you can do calculations in Groovy. I am no fan of using one language for all uasages, but on the other hand I think that such a mode would be very good for Groovy. And it would tell all those benchmark writters out there not to meassure a language with calculations. That is a very stupid thing to do I would say.

The implementation is surely not done in a few minutes and this "fast mode" will not go into Groovy 1.0. But 2.0 or even 1.1 may have that.

Other ideas?
There were other ideas, yes. For exmaple not to simply make the call through the Metaclass, but to let the compiler guess a method and check with the MetaClass if the guess is right. Sadly that doesn't help, because the method selection has still to be done and is the cost intensive factor here. Writing a hotspot engine for Groovy might be a solution. Java is making HotSpot open, so it is theoretically an option. And possible a much better option than anything else. But there are many unclear details.... none has the knowlefge to write something like that, noone knows the API, how is the HotSpot engine added to the normal Java SDK?

We always tried to avoid the need of a special Groovy JVM, even it is jsut the JVM with additional bytecodes. if the HotSpot engine can't be added to any normal JDK then I don't see why this should be any different.

Reality!
I think the space for doing optimizations is big. For example in the "fast mode" we probably don't need to always create a closure. Instead we can inline the closure handling method and the closure itself. But I think only a concrete inplementation of all the possibiilites would be a fair way to tell if something is better. But even if the normal method invocation in Groovy gets a boost fomr a better HotSpot engine, or new bytecode, it only means that the "fast mode" is no longer needed. I would always recommend to use the normal Groovy method invocation and identify the cost intensive parts. Neither will the "fast mode" solve all problems nor will it free you from the usage of profilers.

Remote Closures

2006-09-09T17:58:00.000+02:00

a better RMI?

The discussion about closure support in Java is going on. Well, Groovy does have Closures and I like them much. But I asked myself a question: "What happens when I want to execute a closure on a remote VM?"

I answered myself, that that possibly would be a problem. If we use the normal serialization mechanisms, then we have bad luck, a Closures is not Serializable atm. Well, ok, that could be changed, but does it make sense? Usually a closure is a bit like a annonymous inner class. That means in case of RMI you don't have a stub as RMI would require. So maybe the real question is about how to make Groovy's closures Serializeable?

I had some nasty thoughts about somehow serializing the AST into the class, but then I recognized, that the original source might be good enough for that. And it doesn't bloat the class files which would have a bad effect on classloading time.

So I thought, maybe give them custom read and write methods for serializetion, where we transfer the source as well as the enviroment the closure closes around. This could do the job. But then the MetaClass needs to be serializeable as well.

No, I think if we are able to get the source for the closure, then why not sending this to the remote machine? We would put the source and state information of the closure in a wrapper object and when deserializing we reproduce our closure. We use ObjectStreams that know what to do with the closures and we somehow have to foist our custom streams upon RMI.

Or we don't use RMI at all. If we use the ObjectStreams directly we don't have any problems. I may have time in the future ot take a look at spring and it ability to use different remoting services. But thinking of how RMI is done, I always have the feeling that it is completly surplus. How would I implement Remote Method invocation? I need some kind of cahnnel, transfering the method message, that is name of the method, some kind of id for the object the call is made on and a serialized version of the parameters. On the other end, I need to use the id to lookup the goal object, I need to deserialize the arguments and then just make the call using a MetaClass. It is easy!!

I think I will try an example implementation the next days and keep you informed about the result. The nice thing is, we can use that from Java too, thx to the good integration of Groovy in Java!

Getting a Groovy S.O.D.A

2006-09-08T22:00:00.000+02:00

For all who don't know S.O.D.A., it is a query language for object databases. SODA is used by the successful db4o project - a object oriented database for Java and .NET

Imagine an object as graph. The object itself is a node, every field is a connection to another node, another object. You build up a query by defining an entry point into that graph and setting constraints on the connections to other nodes. I borrow an class from their tutorial, I hope that will be no problem

class Pilot {
  String name
  int points
}

The above defines a class named Pilot with two properties, in Groovy this means a java like class with getters and setters for name and point. Now look at a query:

Query query=db.query();
query.constrain(Pilot.class);
query.descend("name").constrain("Michael Schumacher");

With this I build a query pattern using Pilot as entry point, looking at the connection "name" and constraining that to Objects equals to "Michael Schumacher". So giving this query a database will write out all Object of class Pilot, with the value "Michael Schumacher" stored in "name"

Pretty neat, isn't it? It is typesafe and such... but a bit long... Well Groovy supports operators, why not try something else:

Query query=db.query();
query.constrain(Pilot.class);
query.name == "Michael Schumacher"

we save the descend and the constraint method. But it gets much better when it comes to connecting query parts for logical operations:

Query query=db.query();
query.constrain(Pilot.class);
Constraint constr=query.descend("name")
    .constrain("Michael Schumacher");
query.descend("points")
    .constrain(new Integer(99)).and(constr);

This query asks for all Michael Schumacher with 99 points. I think we can do this much better in Groovy:

Query query=db.query();
query.constrain(Pilot.class);
query.name == "Michael Schumacher" && query.points == 99

hail operators! This is clear and simple, we don't need the temporary variable any longer and saved some lines of code. And of course operations as greater, smaller, equal, not, and, or are all possible.

Is it possible to improve that even more?

Query query=db.query();
query.constrain(Pilot.class);
with (query) {
  name == "Michael Schumacher" && points == 99
}

Maybe a matter of taste if that version is better, but imagine complex queries. I gues you will be lucky not always having to write "query" everywhere.

Another important part of SODA are evaluations. When we use evaluations, then every object our query had as result, is tested against an evaluation instance we provide, and tells the databse to include it in the result or not.

class NameLengthEvaluation implements Evaluation {
  public void evaluate(Candidate candidate) {
    Pilot p = (Pilot) candidate.getObject()
    candidate.include( p.getName().length() < 5 )
  }
}
query="db.query();"

This would get us all Pilots with a name of a length of less than 5. A small helper class will give us very much power for groovy:

class ClosureEvaluation implements Evaluation {
  ClosureEvaluation(c) {this.closure = c}
  def closure
  public void evaluate(Candidate candidate) {
    candidate.include(closure(candidate.object))
  }
}

using this class we can use a closure as evaluation:

Query query=db.query()
query.constrain(Pilot.class)
query.constrain (new ClosureEvaluation() {
  it.name.length()<5
})

overloading the constrain method we can even shorten this:

Query query=db.query()
query.constrain(Pilot.class)
query.constrain { it.name.length() < 5 }

Now it is very short, isn't it? We can also use a variant of the closure with two parameters to allow access to the object container, but given the fact, that we use a closure we can simply reuse already declared variables outside the closure.

Conclusion:
Yes, db4o provides a solution in Java too. I mean a shorter form of SODA queries named native queries. Basically the bytecode of the class is loaded, analyzed and transformed in a set of SODA queries and evaluations. Given these short forms in Groovy I am not sure I "need" native Queries.

Non Local Transfers in Groovy: A 90% Solution

2006-09-01T15:03:00.000+02:00

The discussion about supporting break/continue and return in Closures is a old one.

I was always looking for a 100% solution that just works. Too sad I still haven't found one. But ok, let us look at my new proposal, which is inspired by a blog entry from John Rose about the ongoing discussion of closure support in Java. I think my proposal should cover 99% of all cases.

First let me introduce the term "appended closure". With that I mean closures appended to a method call. An example would be

list.each {println it}

if (atHome) {doHomework()}

Ok, the last could have been a closure, bit is none ;)

My solution now covers only these appended-closures. Closures passed around as variable are not covered. These appended closures are most looking like the normal control structures we have, as for example the while loop.

Now what happens when declaring a while loop? The compiler will make us a label marking the entry point of the loop, a label marking the exit point of the loop, and some gotos for break/continue. in Groovy Closures are inner classes, so they can't jump to a label using the bytecode. So we must transfer that exit state form the closure to the calling point and then do our goto there. There was much discussion about how to do that. We discussed return values, which won't work as we must exit the loop method which may be void. We discussed xceptions, which would work, but we where unsure if we really catch them all at the right places and having to rewrite all closure handling code out there is not nice too. We discussed additional fields, with the problem of using the same closure multiple times.

If you only look at appended closures, then some of the problems go away. We get a defined point where to catch the Exception, look at the fields or whatever. But we need the Exception, because we need to exit the method calling the closure. We need it in the case we want to do a break.

And if we limit break/continue to just this type of closures, we have no real problems. We can even do a labled break/continue and not jump only outside the loop method, but outside the surrounding loop too. So what would it look like?

def list = [1,10,100,10,1]
list.each {
it (it>10) break
println it
}

The intention is to print the values 1 and 10 and then stop processing. My suggestion now is to tranform this code into:

def list = [1,10,100,10,1]
int id = createLabelID()
Closure c = {
  it (it>10) throw new ClosureBreakException(id)
  println it
}
try {
list.each(c)
} catch (ClosureBreakException ce) {
if (ce.id != id) rethrow ce
}

Well, yes, looks like much code. But the compiler does the work for us. What about continue?

def list = [1,10,100,10,1]
list.each {
it (it>10) continue
println it
}

Printing 1,10,10,1. It would be transformed into something like

def list = [1,10,100,10,1]
list.each {
  it (it>10) return
  println it
}

ehm.. yes.. easy ;) All code in the method calling the closure is always executed then. Compared with a for-loop this is the part where the increment happens an the comparision is done. If we want to jump to a different point, like a surroung for loop, we need exceptions too, because we break the inner loop then

outer: while (foo) {
list.each {
      it (it>10) continue outer
      println it
}
}

is basically the same as

while (foo) {
list.each {
       it (it>10) {doContinue = true; break}
       println it
}
if (doContinue) continue
}

But as we use exceptions here, we don't need that "doContinue"

int id = createLabelID()
Closure c = {
  it (it>10) throw new ClosureBreakException(id)
  println it
}
while (foo) {
try {
  list.each(c)
} catch (ClosureBreakException ce) {
  if (ce.id != id) rethrow ce
  continue
}
}

What about combinations of continue and break?
To enable this I suggest the ussage of more than one id

while (foo) {
list.each {
  if (it<0)>10) break outer
}
}

is transformed to

int idBreak = createLabelID()
int idContinue = createLabelID()
Closure c = {
it (it<0)>10) throw new ClosureBreakException(idContinue)
}
outer: while (foo) {
try {
  list.each(c)
} catch (ClosureBreakException ce) {
  if (ce.id == idBreak ) break outer
  if (ce.id == idContinue) continue outer
  rethrow ce
}
}

The implementation is really straight forward.

Speed Issues?
We can tell the people that using break/continue might slow down.

Danger of doing break/continue on the wrong loop?
I think we eliminated that problem with our createID function, which produces a new unique id.

Why is it a 90% only?
What I can't do is:

def c = {if(it>10) break; println it}
list.each(c)

and expect the break to work. That is because we have no defined point where we can catch the exceptions. A normal continue can work.

What is the return value in case of continue?
The same as by break, nothing. People will have to know this. We can't have both, a return value and an exception, if the exception does not the job of transporting the return value. In case of a "collect" I expect to get at last a pratial list. To enable this, we must catch the exception and then store the return value od the loop method in there. The changes to the code I showed above is trivial. The changes to the loop method are trivial too, but must be done. The case of a continue is a bit more difficult. And to say the truth I don't know of a solution here. That is because

... I want to avoid people having to rewrite their closures handling code.
Respecting continue would mean to catch an exception in the closure handling method. I don't think that is nice. If I don't do it in case of a "break" my code still works, but might not return correct data. If I don't do it in case of a "continue" my code doesn't behave right and doesn't return correct data. And while supporting break is not needed in every method, supporting continue would be.

What about "return"?
We could support it the same way we support "break". No real problem. In fact that is the most easy version since we don't have to handle incomplete data.

My Suggestion:
Don't allow "continue" in closures. It might look strange, but it has semantic problems. People can always simulate it by returning from the closure with a special value or such and respecting that value in a loop. But avoid new users getting in trouble here and confronting them with more black magic as we already do, I suggest to replace "continue" with a "closure return". The advantage is, that it returns a value and thus is no problem in aspects of incomplete date. for Java ^ is suggested as additional return, I haven't thought about a name or pattern yet. break/return can be implemented as described above for appended closures. Other closures won't have break/return, the compiler would forbid them there. A bit of a problem is the new return. Becasue using the current return in a closure in current Groovy means to return from the closure if it is in a closure and return from the method if it is not in a closure. When seeing a closure as method this is ok, but we don't want to see them as methods. So "return" would change its semantics compared to older groovy versions.

All in all this suggestion here would allow people to use break and return in closures. It would allow to sue them in appended closures, the most used form of the closure when doing loops. A "closure return", or I should better say "block return" would replace the old continue statement. If people really wish to omit a single step, they have to write their own loop.

The advantages is that I don't have to write additional closure handling code for break/return. Only if I want to avoid the incomplete data in the case I return something. Next advantage is that I don't have to identify the closure somehow in the loop method, I don't care if the current closure caused the problem or something deeper inside. I have a defined point where I can catch the exception and don't have to be afraid that it might interfere with other loop methods.

The other possibility I see is to forbid them all ;)

Groovy and Scala Multithread Actors

2006-08-22T21:07:00.000+02:00

Parallel exection of Methods.

In my last blog entry I used a MetaClass to change the method execution modell. Some days ago a paper surfaced in the german Java newsgroup about parallel execution of methods in Scala. I found it quite interesting, but maybe there is a way to have that for Groovy too?

I think it is possible using a custom MetaClass again. Again we would collect the method calls, but this time we would use multiple threads to execute the methods. Using a pool for that could be a good choice. Anyway, I don't want to go too much into the details this time, but if you look at my last blog entry and exchange the method execution part with something like:

   stack << method 
  while (stack) {
    def call = stack.pop()
    ThreadPool.run(delegate,call)
  } 
}

And then make asynchronous access to the stack to put the frames on, then it would work there too. A special method might be needed to synchronize the method calls again. Something like "metaClass.waitForMethodCalls()". Again we could use a method to switch between parallel and normal execution, just like we have done with recursive and iterative execution before. And of course we have the same disadvantages as the last time. Maybe I will have somewhen a bright idea how to solve the return value problem in a much nicer way than using a global variable. I will let you know it then.

And of course I wouldn't propose such a solution for working on a Cell processor, where we simply use as many Threads for the method calls as cells are available... but maybe, with a bit more research, this could be used to make an efficient execution system without the need to explicitly handle Threads. Yes... well... there is the synchronization problem. Not only to wait for methods to end their calls, but also to synchronize access on variables.

As always when walking on the edge, you should know what you do.

Iterative Method Execution Modell in Groovy?

2006-08-22T18:57:00.000+02:00

I wonder if that is possible... what? uh, sure, you don't know what I mean. Ok, let us see...

def foo(){
 bar()
}

def bar(){
 foo()
}

You may know this example from the tail recursion article a few days ago. But I want to extend that example now and to this:

def foo(){
 bar()
 baz()
}

def bar(){
 foo()
 baz()
}

def baz(){}

So every method makes an recursion step and then calls the empty baz method. Congratulations, we succesfully eliminated a tail recursion. While executing these methods may never end in a endless loop, as we need to keep a trace to remember we have to call the baz methods, the available recursion depth isn't as deep as I wish it would be. The last time we work around the problem using Closures... Maybe this would work today again? But what to put in the closure? The calls? No that's no solution. Maybe a special version, that means "make a recursive call and give me what you intend to execute after". So our foo might transform into

def foo(){
 recursiveCall (this.&bar) {baz()}
}

we would put the Closure on our own Stack, leaving this method and then execute bar first.

def recursiveCall(Closure method, Closure tail){
 stack << tail  
 stack << method
}

And foo would not be called normally, but this way:

def recursiveCallEntry(Closure method) {
 stack << method  
 while (stack) {
   stack.pop().call()  
 }
} 
recursiveCallEntry(this.&foo)

what about tree like structures?

def foo() {
 bar1()
 baz1()
 bar2()
 baz2()
}

could be a tree iteration or something. This would be converted into:

def foo() {
 recursiveCall(this.&bar1){baz1()}
 recursiveCall(this.&bar2){baz2()}
}

And does it still work as it should? After executing these methods we have a stack like this one:

bar2,baz2,bar1,baz1

With bar2 on top. That's bad, bar1 needs to be executed first. So we need to reorder that.. but not in recursiveCall. You may see that recursiveCall, is doing something like a collection step. Collecting all method calls that we need to do. So if we collect all the calls the recursiveCall methods do in a list or something, we can simply revert the list and then put the elements in the correct order on the stack. As our recursiveCallEntry method does have the real control over the custom stack, we can use that method.

def recursiveCallEntry(Closure method) {
 stack << method
 while (stack) {
   frame = []
   stack.pop().call()
   stack.addAll(frame.reverse())  
 }
} 

def recursiveCall(Closure method, Closure tail){   
 frame << method   
 frame << tail
}

As you see we no longer revert the order in recursiveCall, since we do this explicitly with frame.revert in recursiveCallEntry. Using this system we can bypass the normal stack, build our own stack and get a much better depth. But "recursiveCall(this.&bar1){baz1()}" looks really ugly. Oh.. do we need that? why distinguish between the real recursive call and the non-recusrive calls? No, I don't think so.

def foo() {
 recursiveCall this.&bar1
 recursiveCall this.&baz1
 recursiveCall this.&bar2
 recursiveCall this.&baz2
}

That is better, but still not nice. We could use Strings for the method name, but we then loose the object we want to execute the method closure on. Oh.. and parameters? In the end the code would look like

recursiveCall (this, "baz2", null)

to call the baz2 method. what most people probably don't know is that such a line already exists in the bytecode, because each normal method call is done by using a method of ScriptBytecodeAdapter. So if had a flag method or something we could use that to execute our methods and keep the readability. Oh no, again a crazy idea is born.

def foo() {
 switchToIterativeCallSystem()
 bar1()
 baz1()
 bar2()
 baz2()
 switchToRecursiveCallSystem()
}

but wait... do we really have to do that on the ScriptBytecodeAdapter? We can do that with the MetaClass too! We simply need to replace the MetaClass for "this" with our own crazy version...

class IterativeCallerMetaClass extends  DelegatingMetaClass {
 def stack =[]
 def frame
 boolean iterativeCallSystem=false

 public IterativeCallerMetaClass (final MetaClass delegate) {
   super(delegate);
 }

 def invokeMethod(Object object, String methodName, Object[] arguments) {
   if (iterativeCallSystem){
     frame << [object,methodName,arguments]  
   } else {    
     delegate.invokeMethod(object, methodName, arguments);  
   }  
   return null
 } 

 def recursiveCallEntry(Closure method) {   
   stack << method  
   while (stack) {
     frame = []
     def call = stack.pop()
     delegate.invokeMethod(call[0], call[1], call[2]);
     stack.addAll(frame.reverse())   
   }  
 }
}

And the complete calling code would maybe llok like this:

def foo() {
  metaClass.iterativeCallSystem = true
  bar1()
  baz1()
  bar2()
  baz2()
  metaClass.iterativeCallSystem = false
}

this.setMetaClass ( new IterativeCallerMetaClass (this.getMetaClass()) )

Disadvantages? Yes, sure. This completly ignores the return value. And sadly I don't see a way to fizzle it in. I mean a call foo(foo(1)) is really something like "tmp = foo(1); foo(tmp)". That means I need the value of "tmp" while I am still in the collection phase, where it isn't available, since the call isn't done then. Next disadvantage is, that that MetaClass only works on one object. Any foo.toString(), where foo!=this, isn't collected. Which directly leads to the next problem, logic operations are executed before the values are available.

To work around that we can put all that code in Closure and add these closures to the custom MetaClass stack, but that gives again problems with local variables and such. If you have a tree with an unknown number of children and you want to do an depth first search on the children, you would normally do something like that:

def search(node,constraint) {
 if (constraint(node)) return node
 node.children.each{search(it,constraint)}
}

But when you want to use our iterative system...

def search(node,constraint) {
 metaClass.add {
   if (!found && !constraint(node)){
    node.children.each {search(it,constraint)}
  } else {
    found=node
  }
 }
}

That's an ugly level of indirection... I used the global variable found to indicate if we need to search deeper, this avoids the return value. Then the closure will be put on the stack and executed when the data is available, collecting the search calls and later execute each of them. This can only work if all actions other than pure method calls on this are done in closures.

Conclusion:
This let's me think that such an solution should be much more easy in Ruby. But as I don't know enough about Ruby I don't know if there is such a MetaClass system as we have in Groovy. But being able to simply put the method body on the stack instead of a closure would be nice. Anyway, it works. We can also remove the "metaClass.iterativeCallSystem=" lines with the convention that the system is always active. Of course this works with a builder just as fine as with a normal class, you only have to do the job of our MetaClass in the builder then.

You see, many tricks can be done with the MetaClass, even a total different method execution model. Ok, not toally different, as the VM still does the invocation and still keeps a trace, but we successfully shortend that stack trace. Oh, and of course the above works for tail recursion just as well. You can say we included them here already. Of course the solution developed into a completly different direction than my previous post about tail recursion.

And maybe a word about the execution speed. Of course you add an additional level of indirection for method calls, our delegating MetaClass. So instead of doing one mthod call we do much more of them, for example when testing the stack if it is empty, when getting the top frame and more. But if your focus is on avoiding a stack overflow it works just fine.

Tail Recursion in Groovy?

2006-08-16T15:54:00.000+02:00

Is it possible?

Tail recursion can be very useful when traversing big lists or trees using recursive methods. The JVM doesn't support tail recursion - maybe there is a way to simlulate them. But first, what is a tail recursion?

A tail recursion is, well I thik the wikipedia explanation is quite good ;) For me the important thing here is to reduce the Stack pollution and not to get an exception from the JVM, because the trace got too big. So a transformation of one kind or another is needed to avoid that.

I know 3 kinds of techniques to accomlish this:

use the compiler to transform the recursion into a loop
let the vm recognize the recursion and eliminate it
mark tail recursion specially and react different then normal

To number 1 I must say, that we could do this in Groovy - partially. That would only work for self recursive functions and it requires the ability to ask the method execution part if that method call is a call to the current method or not. For Java, the last thing is out of question, there is no such query mechanism at runtime to do that. And there is no need, as all methods are selected at compile time. So Java could have this kind of tail recursion. In Groovy we can't be sure if the method we want to call is really being executed. And we may not be sure that the parameters we give really result in the desired method to be executed. Of course we could implement such a query system in Groovy and then we would have the answer. But that would give us only a partial solution as I mentioned already.

Then number 2. Well as long as the Java people don't decide that Java should have that, there is no way for us to get that done in Groovy. Groovy is not a VM in the VM, it is a runtime which does influence the method calls, yes, but it has no real control over the stack.

So what about number 3? Is there something we could do to get tail recursion? Before I get deep in the details imagine the stack that is created for

def foo(){
 bar()
}
def bar(){
 foo()
}

foo()

We will create a stack element for foo, then, when calling bar, we will create a new stack element for bar. After this one again for foo, and this would go on until the JVM complains about the stack becoming too big. What if we would now not directly cal the method, but return from the current stack frame and then do the call? Using Closures we can simulate that:

def foo(){
 {bar()}
}
def bar(){
 {foo()}
}
def cl = foo()
while (cl!=null) {
 cl = cl()
}

With the result, that the function foo is called and returns with the closure. After this, the closure is called, which does call bar. Then bar returns with the closure and the stack size becomes 0 again. Then the closure is again called. This continues till someone forcefully terminates the program, since we crated an endless loop. Of course it is not very nice to have to do that this way. An autmatism would be nice.

So let us say there are keywords like... trreturn and trcall, well not nice, but it is just for illustration. This signals the runtime to not to simply return after the call was made, but to examine the result and react to this. You need to know, that a normal method call foo() is in fact this: ScriptBytecodeAdapter.invokeMethod("foo"). So instead of using the normal invoke we use the our special invoke method invokeMethodTR.

def invokeMethodTR (String name) {
def c = invokeMethod(name)
while (c instanceof TailCall) {
 c = invokeMethod(c.name)
}
return c
}

and foo would be for the compiler:

def foo(){
 return new TailCall(name:"bar")
}

a normal user would see this:

def foo(){
 trreturn bar()
}

But the initial call to foo must be done using trcall, so the complete program would look like:

def foo(){
 trretrun bar()
}
def bar(){
trreturn foo()
}
trcall foo()

This way to do this has actually two problems:

Calling such a method from Java would give a TailCall isntance.
a void method can't return a TailCall instance

The most easy way to go around that would be not to avoid it. Instead we could use it as marker for the method and remove the trreturn:

TailCall foo(){
 bar()
}
TailCall bar(){
 foo()
}
trcall foo()

The TailCall instance may not only transport the method name and arguments, but also a return value if needed.

Conclusion:
All in all at last one additional keyword is needed to get it working? Maybe using a closure here not even that is needed. We could just say tailcall{foo()} and leave the loop part of the implementation to that method. The compiler is still needed, because someone does have to create the TailCall instances for the return. But the special invokeTR method isn't needed any longer. So any changes to the compiler are doable in no time. Regarding to my question at the start, if it is possible.. well, yes! It's not even very difficult!

Beyond Groovy 1.0: Groovy goes Lisp

2006-08-15T00:25:00.000+02:00

I was always a fan of LISP, but not of the huge amount of (). Regardless the syntax, LISP gets it power through exactly that, the syntax. The late Binding of expressions and the simplicity of the language in syntax and semantic allows to extend the language itself very easily. And that again allows you to write DSLs embedded in your normal program. And how to get something like that to groovy? A list looks in groovy like that:

[1,2,3]

but that doesn't contain a operation. Adding an operation as normal Element would require a closure - I don't think that is very nice. Anyway, if you could remove the "," we would have a much nicer list:

[1 2 3]

Now let us assume that is something like a quoted list in LISP. How to get an operation inside? If we could omit the comma in general, then we could use a normal method call

add 1 2 3

which is the same as

add(1, 2, 3)

and if we need late binding, we can use closures

{add 1 2 3}

now, what is a simple program in LISP? Maybe

(defun add (x y)
(+ x y))
(add 1 2)

and what would it look like in Groovy with the optional comma?

{defun add {x y}
{x+y}}
{add 1 2}

Of course the curly braces are not that nice here, but needed. Anyway, it looks really a bit like the Lisp example, but would it work? Remember, defun would be a method call and "add" a parameter to that method. That means "add" is evaluated before defun is. So a evaluation system is needed here, that allows to handle such cases. So instead of evalutating directly to a value, we simply return a place holder. Let us name that place holder symbol here. So in fact the defun method would get a symbol and two closures. Ok, now let us define that every method call itself is first evaluted using symbols, so

{defun add {x y}
{x+y}}

would return a list with two symbols (defun and add) and two closures. Let us call the function that does this job preeval. In Groovy terms, preeval is a builder or special MetaClass, that intercepts all calls to methods and properties and does return a list of symbols and closures instead. Then a secound stage evaluation process would start and use that list to execute functions. Let us assume "defun" is delegated to some kind of method, then this method might look like:

def defun(Symbol name, Closure parameter, Closure body) {
evaluationSystem. register (
  name,
  preeval(arguments),
  body
)
}

Again preeval would return a list of symbols, this time they wouldn't be mapped to a method, they would be used to define a scope for the body closure. And when, the the body closure asks for x and y, the evalutation system would know, that these two are in fact the arguments. After this method is available, we can define car and cdr using defun:

{defun car {list}
{list[0]}
}
{defun cdr {list}
{list[0..-1]}
}

Of course having a generic name for car and cdr would also allow the "special" functions caaar and cadar and what else. This is enough to build a powerful LISP like system. I would have to look how macros are realized in LISP, but theoretically these basics here should be enough to allow them too. And then we could do code like this:

def result = lispBuilder.eval
{defun groovy { a b}
{a.name.length() + b.name.length()*10}
}
{groovy is lisp}

assert result == 42

Anyway, that is just a crazy idea, but possible. And all this only by making the comma optional. Of course it would work without that too, but {groovy is,lisp} doesn't read as nice as the version without comma. It would also allow us to become more like Smalltalk, but I will show that in another blog entry. I just wanted to show you how it could be made possible to get something with a sligthly feeling on LISP in Groovy. I won't say that the above fragments are really LISP, but I think by developing this idea to its end it would look much like LISP. but it doesn't have to be LISP, it could be also Scheme and then we could compare Kawa and Groovy *lol*

Conclusion:
There are limitations. I mean names can't have all shapes without being put into "" and requiering a qualificator like here: this."strange variable name". And of course you won't be able to stop Groovy from doing early binding for local variables - I am not sure that is really a disadvantage. I wouldn't expect LIPS people to really migrate from LISP to Groovy just because of this blog entry here. But maybe some LISP people out there, which are forced to program on the JVM might want to consider using Groovy in the future ;)

Getting Groovy 1.0 out is currently more important than having crazy ideas about making Groovy into a overcomplicated LISP. But I must say, I really like this idea.

EDIT:
there is a small error in the artice, closures need to be concatenated by } {, there can't be a newline in between. But maybe this is another thing to change?

Show me what you have! Featuring Groovy, Swing and db4o

2006-08-13T17:31:00.000+02:00

Once again I had to write one of these stupid and boring GUIs. You know these things asking you stupid questions and showing irrelevant Data. Of course I do all in Groovy, no need for a IDE provided gui generation tool. So I used the SwingBuilder to make the GUI.

Really, I understand why people are asking for documentation. The 2 examples here are giving good examples, but basically you have to understand that components are registered and each property can be used using the map syntax. Embedding one component in another works using Closures. but there are things the Swing builder currently can't do. For example registering a MouseListener at a component using Closures as the functions executed by the MouseListener. Or a WindowsAdapter with Closures for certain action, like closing the window. So you still have to create a big bunch of useless classes. That's not nice. Ok, they are not that useless, because you can reuse them later.

We (the Groovy Team) had the idea to convert a closure to a class implementing a certain interface with just one method by the MetaClass or using the as-syntax. Just only that this won't help here, a listener normally has more than one method. So for now, I decided to make the helper classes, but maybe we should simply add an method to the Swingbuilder allowing to do such things. Would be possible and not that difficult I guess.

What puzzled me a bit was, that I have to do things like:


scrollpane {
 viewport {
   list()
 }
}

Now I know a viewport is inside the ScrollPane, but that I have to do that explicitly isn't very nice. In fact I would like to have a default property and then be abel to do that:


list (scrollpane:true)

or at last


scrollpane {
 list()
}

Anyway, it doesn't bother me too much.

So I wrote some glue classes like this:


class ClosureWindowClosingAdapter extends WindowAdapter {
   def closure
   void windowClosing(WindowEvent e) {
       closure(e)
   }
}

not much logic there, but it allows me to do things like that:


gui.addWindowListener (
 new ClosureWindowClosingAdapter(closure:{System.exit(0)})
)

After I have done all these little things I got a nice small gui application. Just a few lines long itself, some minor helper classes and yet no data.
Yes, that's the point where db4o comes to play with us. I had many lists basically showing all available instances of one type. And I never understood why I should make a custom model here to keep multiple lists of the same objects, maybe concat these lists so they can notify each other about changes and such.

I decided I want to use a central data holder this time, the objects are extracted from there using query-by-example and, no, the data holder is not a relational database with hibernate in the foreground. I wanted to have something easy, that allows me to change the class structure and doesn't requires as much overhead as hibernate. So I used one of my helper classes and wired some db4o parts in:


class ClosureQueryModel extends AbstractListModel implements ComboBoxModel{
    def cl
     def getElementAt(int i){cl()[i]}
     int getSize(){cl(query).size()}
    def selectedItem
    def addElement(foo){/* ignore */}
    def remove(foo){/* ignore */}
}

combo.setModel(new ClosureQueryModel(cl:closure))

nice isn't it? Again a predefined class would make life more easy here, but well, it isn't that this helper is a big one ;) and yes, I am ignoring removal and additon of elements, since I wanted to make that completely through the database. cl(query)[i] means I need a list or array or something like that as result. Now in db40 4.3 this would have been a problem, since a query on the database does return an instance of ObjectSet, which is no such list or array. But they changed that, and in 5.2, which means I can directly use the result of the query.

For all people here not knowing how such an query looks like:


def objectSet = db.get(Foo)

yes, that's all. Foo is the class of the objects I want to have and the above query allows me to get all instances from the database. So I would create a ClosureQueryModel isntance like that:


new ClosureQueryModel(cl:{db.get(Foo)})

So now my items would be displayed... but... there is no initial data. What is missing is a functionality to add remove and update items. Well, maybe I got here a bit over the edge, because I got this crazy idea of using a mapping to show the editable fields... but on the other hand, any bean editor is allowing comparable to that. Only that I don't use explicit beans and only that I don't use these extra files for beans describing the bean. Anyway, I made a map like this one:


[name:"field",view:"Feld",collection:{it.availableFields}]

to set a property/field named field, with the display name Feld and it is chooseable from a collection, that is returned by a closure {it.availableFields}. so basically I can choose here between predefined values returned by a property or whatver I use in the closure. then I thought that a simple collection is possibly not enough. What about numbers? so I added a "create" to the map, taking a closure that produces the object:


[name:"workload",view:"Auslastung in %",create:{new BigDecimal(it)}]

create and collection would be exclusive then, but I added no test for that.

I thought, ok, the let us have a popup menu, for the crud operations and put all that logic together in a method.


addPopup(
  component:jlist,
  producer:WorkloadEntry.class,
  maping:[
    [name:"workload",view:"Auslastung in %",create:{new BigDecimal(it)}] ,
    [name:"field",view:"Feld",collection:{it.availableFields}]
  ],
  newAction:{db.set(it)},
  updateAction:{db.set(it)},
  deleteAction:{db.delete(it)})
}

producer is the class use to create new instances from, mapping contains the fields/properties set for the new class and how they are displayed, and the *Actions are he database methods for the crud operations. Pretty easy, but the mapping tends to be big for man fields. I should think of something different here.

I don't want to show all of the code of that method here. The method itself is 100 lines long, as much as the rest of the program. A sure sign, that it is too complicated. But well, the basic parts:

I need to display the views:


  args.maping.each { item ->
    label(item.view+":",constraints:gbc1)
      if (item.collection != null) {
        def combo = comboBox(constraints:gbc1)
        combo.actionPerformed = {query.setProperty(item.name, combo.selectedItem)}
        combo.setModel(new ClosureQueryModel(query,item.collection))
      } else if (item.model){
        def combo = comboBox(constraints:gbc1)
        combo.actionPerformed = {query.setProperty(item.name, combo.selectedItem)}
        combo.setModel(new ChoiceModel(item.model))
      } else {
        selection << textField(constraints:gbc1)
      }
   }

saving it would look like:


  args.maping.eachWithIndex { values,index ->
    def sel = getViewComponent(index)
    if (sel instanceof JComboBox){
      obj.setProperty(values.name, sel.selectedItem)
    } else {
      def val = sel.text 
      if(values.create!=null) val = values.create.call(val)
      obj.setProperty(values.name, val)
    }
  }
  [...]
  if (args.newAction!=null) args.newAction.call(obj)
  args.component.model.addElement(obj)
  args.component.model.fireContentsChanged(editWindow, args.component.selectedIndex, args.component.selectedIndex)

You can see the create closure here, called if set and if a JTextField was used to display it. Maybe giving the component instead of the text only would be also a improvement and would allow other components. But this was more of a case study, not a general purpose solution.

Even without the usage of the database, such a component should be thought through. I mean you need something like that quite often.

Conclusion:
Groovy's Swing part currently consists only of the SwingBuilder. That helps a lot creating guis, but when it comes to models and listeners, the SwingBuilder lacks a huge amount of them. Models for spinner and tables are there, but not more. I think, developing Swing with such components can improve your productivity when it comes to make a prototype. You can alaways decide to change the class model and you can always change the gui itself. You could even think about a text, that you enter when a button is pressed and then that text is executed as script. Of course the combination with an plugin such as the eclipse groovy plugin would make things even more nice. But even without such self hosting features, it makes much more fun than plain Java, even if a WYSIWYG editor is involved. If I had more time for Groovy I would no go on and improve the Swing abilities of Groovy big time.. but maybe there is someone else who wants to do that. Groovy is open source and people doing such things are always welcome.

I hope I won't have to say that last part too often. Of course I would like to do all the ideas I have by myself, but I can only spend a few hours every week on Groovy and that is not quite enough to get issues fixed and to add new ideas.

a new blog - oh my, why that again?

2006-08-13T16:42:00.000+02:00

Hi there

I am about to start my new blog here. In the future I want to let you people know about java and groovy related thoughts, problems and stories - maybe other stuff too.