Owner, Delegate and (implicit) this in an Open Block

Many know of course groovy.lang.Closure, they know about delegates and such, but maybe not so many know why these things exist.

What is a Open Block?

That is basically what is at runtime represented as groovy.lang.Closure or short Closure. Please note that I don't use closure, since an open block can be a kind of closure, but is less limited than a closure. Open Blocks are no lambda expressions either, since they can contain 0-n statements. For more syntactical information please go to http://groovy.codehaus.org/Closures.

The captured call

http://groovy.codehaus.org/Builders shows some examples about what builder are, but essentially they are hierarchical structures, with a capturing ability. You don't really need to nest one open block into another to get that of course. But those Builder are exactly the reason why we have owner, delegate and "this".

To explain this in more detail, let us start with a normal Java block
{ foo () }
You notice, this is a simple call to a method named foo, that is supposed to be defined somewhere outside of the code part we look at. For example foo might be defined in the same class, that contains this code block. It is clear, that foo() is equal to this.foo() here. In the case above I call that to have an "implicit this". There are languages that don't have that of course. Smalltalk and JavaScript come to my mind. In Java "this" and the "implicit this" always refer to the enclosing class.

"Implicit this" in Groovy

Groovy now does this a bit different for open blocks. In Groovy the implicit this is like a reference to the capturing mechanism built into the open block, realized by the Groovy MOP and for a builder. Therefore the call will in Groovy be resolved to the owner, the delegate or to "this". Actually, Groovy is the only programming language I know in which "this" and "implicit this" are essentially different. I know of differing type variants for them in other languages, many don't even have an "implicit this"... but if they have it normally means they are aligned.

Coming from wanting to support the Groovy builder structure, it is clear we want some kind of capturing, thus it is clear, that we cannot simply do the call on the class outside. On the other hand, assume you have an XML-Builder, that is supposed to turn all calls into xml tags. How would you distinguish a programmer wanting to produce <foo/> (whatever sense that may have) from wanting to call a method from the class, to for example increase a counter, prepare a state or something similar. Led by this thought we decided to make the "implicit this" different from the explicit one.

People knowing the pre 1.0 history of Groovy a bit, can tell, that in the early days a "this" in such a block referred to the groovy.lang.Closure instance. This was at another point changed into having to have the builder instance passed around and making "this" and "implicit this" equal. Both versions seemed not to be what we wanted. The first version conflicts with the Java style, making that a quite phony structure. The second version is even more phony and it made builders absolutely not a nice experience. The compromise solution was then to have the explicit "this" bypass the MOP of the groovy.lang.Closure part and let it call into the surrounding class directly, while the "implicit this" goes through the groovy.lang.Closure MOP part. Ignoring owner and differing resolve strategies the MOP at this point is simply, look if a delegate is set, and if it is, try calling the method on the delegate. If the call succeeds, we are done, if not, fall back to "this".

Open Blocks interacting with Builders

Having a delegate we can realize many builder already quite easily. But the devil comes with the details. Assume we use our xml builder like this:
xml.outerElement {
  10.times { innerElement () }
}
This is supposed to produce an element named outerElement, containing 10 times an <innerElement/> part. You may ask why this is difficult. There is one thing about builder you have to know, and that is for each element of the hierarchy, that means for each Closure, you have to set a delegate. You can do this only, if you are capturing the method call. But since we capture only calls with "implicit this", the times call will not be captured. It is a qualified call through the usage of the number 10. Still we want to refer to the builder delegate "xml" outside from with the block given in the times call. Since we cannot set the delegate for that one from the builder, we need a different MOP here. And this is the point where "owner" comes into play. The "owner" is the "structure" containing/owning our Closure. That is either a class, or another Closure. So in the example above the Closure in the outerElement call is the owner of the Closure in the times call. We then change our MOP to not simply fall back to "this", instead we fall back to "owner". Then our innerElement() call will at first be resolved to the delegate that times set. But since times did not set one, it will go to the owner, which is the Closure given to the outerElement call. The outerElement call has a delegate set through our xml builder. With that we then create our <innerElement/>, as we want it.

This means all four parts, owner, delegate, this and implicit this, are required elements for the Groovy MOP.

Resolving Strategies

In the part above I stated the delegate will be resolved against first. That is actually not always right. It actually depends on what the builder sets as strategy. And the default in Groovy is to resolve against the owner first. If you think back that at first "this" referred to the Closure itself and not the surrounding class, it should be clear, that this kind of strategy is a left over from back then. Because without it, you would not be able to call any method from the class if you have an "capture them all" builder, like a xml builder usually is. So the reason that this is the default is historic. There have been heated debates about what the default should be and there are multiple ways, all with pros and cons, but this default was the result back then. Later we added a set of strategies you can use... DELEGATE_FIRST to first look at the delegate and then at the owner, DELEGATE_ONLY, to stop after the delegate, OWNER_FIRST the default, OWNER_ONLY to stop resolving the call after looking at the owner and there is TO_SELF as well, which would resolve the call against the Closure itself only. Again I have to add a detail to the MOP here. Even though it is called OWNER_FIRST and DELEGATE_FIRST, the first thing we will do is to try to resolve the call against the Closure itself. That makes the default MOP a 3-step procedure: try to resolve method against Closure instance, owner, delegate. It is similar for DELEGATE_FIRST or the "only"-variants.

At this point you may have also noticed a practical difference between DELEGATE_FIRST and OWNER_FIRST. If you use an all-capturing builder inside of an all-capturing-builder structure and you use owner first, then the method call will be trapped by the outer builder. If you use the delegate first the inner builder will do that instead. It depends on your use cases what is better in your situation.

Static Type Checking

With Groovy 2.0, Groovy now also offers an optional static type checker, that allows static type checking for a subset of Groovy programs. Things like a builder are highly dynamic structures and difficult to check statically. Languages like Kotlin and Scala have a problem here. Sure you can make builders in them, but when it comes to nested builders you have to be more verbose and pass the builder around all the time and something similar to the early stages in Groovy. And I don't want even to mention that for a html builder for example you have to define somewhere methods for every element. Considering xml and its probable infinite amount of elements, you get into trouble here, even if you ignore owner, delegate this and implicit this as well es resolving strategies.

In Groovy++ the "solution" was to make a kind of mixed mode, that simply doesn't fail compilation if a method is not found in the normal static context. In fact that was always one of the conflict points between the Groovy team and Alex. We, that especially includes me, found that ignoring a missing method defies the purpose of a static compilation and that you loose most benefits from this. The only remaining one actually is that everything else is near Java speed. But static type safety is completely lost.

The idea Alex did not come up with was a helper annotation called @DelegatesTo from Peter Niederwieser. The idea is to mark the Closure parameter in the builder method with that annotation to allow it to tell the compiler, what kind of delegate this method will use, maybe including the resolve strategy. We already have a framework for "static type checker plugins", allowing to hook into the type checker and influence how method calls are resolved. we hope with a combination of both we can even solve cases like the xml builder. Of course this is current development and target for Groovy 2.1. We have to see with what Cedric will come up with in the end, but what we discussed so far sounded promising, and finally solves a longstanding problem

Groovy 3

The next major version will be in 2013 Groovy 3.0 and include a new MOP, that is still supposed to get its full shape. Implementation wise the way owner, delegate and (implicit this) are used together with the resolving strategy actually impose quite a problem to an efficient implementation. We don't want to force users using @DelegateTo. That annotation is only a helper for static compilation. The problem stems from the fact, that we have a quite long method resolving process here. we may have to go through a longer chain of Closure objects and have to test each time if a method is present or not. And if the delegate is changed later on, caching becomes almost impossible. This is a problem for the current implementation, for the invokedynamic port and for anything in Groovy 3 as well, as long as the capabilities should stay the same. Probably I will be using a series of SwitchPoint to solve this, I have yet to see, if that really gives me the desired speed. It is not only speed that matters to me here for Groovy 3. One goal in Groovy 3 is to make the MOP more easy and this kind of mechanism is not easy. I hope with the help of the community I will be able to solve this problem as well.