Friday, April 27, 2007

A Consensus Closures JSR Proposal

I had set aside work on the closures prototype for a couple of months to write a JSR proposal that represents a consensus among the folks thinking about this area. You can find it at http://www.javac.info/consensus-closures-jsr.html. One of the things I learned is that unanimous agreement is rarely possible. There are those who feel that nothing should be done to the Java programming language, and such people will not be swayed by simple but powerful additions. Our latest JSR proposal comes as close to achieving consensus as I believe possible. All but one of the authors of the three widely-discussed closures proposals have agreed to support it.

The purpose of the JSR proposal is to define the problems to be solved and circumscribe the permitted solution space. It doesn't mandate a particular solution, though it does offer the Closures for Java specification as an example of a solution to many (but not all) of the problems. This should not be surprising, as that spec was written specifically in an attempt to satisfy the requirements. Still, the spec is a work in progress.

So what is next? I hope we'll have some active discussion at JavaOne about where to go from here.

12 comments:

Ricky Clarkson said...

It looks great at first glance. I'm looking forward to the prototype.

In BGGA, RestrictedClosure changed to RestrictedFunction, and in this JSR, it appears to have changed back to the former.

Sean said...

Is it the strategy then, that the "Closures for Java" will be the basis for an expanded approach that includes features from the other proposals?

Tom Palmer said...

Glad to hear some consensus is going on (and I'm curious to see a new spec following the consensus). Here's hoping we see closures in Java 7. Thanks much.

Karl Peterbauer said...

Good news, really appreciate your work!

Ben said...

Why is your affiliation in the JSR TBD? Surely Google would sponsor your work in this area; why consider doing it as an independent?

past said...

@ben:
I'd speculate that it has something to do with the fact that Google employs both Neal Gafter and Josh Bloch, and the latter apparently is the only one who has not agreed to this proposal.

Bharath said...

Yes. Josh Bloch is conspicuous by his absence.

Skandar said...

Closures and Java Doesn't go together. Java need just a simple solution to handle event programming as Delegates. Closures are more for Dynamic typed languages. And Java is a Mature language doesn't need more features. IF adding more fetures will get bloated and ugly language.

elizarov said...

Is there any chance that it will include checked non-local returns, breaks, and continues? That is, is there any plan to replace *unchecked* UnmatchedNonLocalReturn exception with a checked mechanism that will catch such errors at compile-time (rather than at run-time).

If there any way to access preliminary version of javac with support of closures, then I can volunteer to write a proof-of-concept implementation for checked non-local returns.

Neal Gafter said...

@elizarov: That's what RestrictedFunction is for. We looked at making the runtime exception checked, but it turns out not to work. The problems have to do with a mismatch between the static and dynamic behavior of exceptions.

Patrick Viry said...

Hi,

Let me express a concern about the consensus "closures" proposal (note the quotes around "closures"). I am worried that it mixes two really different things, and that if we don't make it clear right now, we will be responsible for spreading confusion in the heads of generations of programmers.

What is a closure ? Intuitively it's a piece of code (as opposed to a piece of data) that is wrapped in a trunk, together with a parameter passing mechanism : you can carry the trunk away with you or send it somewhere else (typically as a callback function), even across a network. Closures naturally lead to function types, so you get all the nice composition properties common in functional programming. Closures also provide some kind of delayed evaluation, since the code inside the trunk is not run until you explicitely ask for it : this is why closures can be used to implement user-defined control structures (the source of most use cases in the BGGA proposal).

Let us go back to your trunk containing code. You have sent it away to someone else, it may be now in a place that you didn't even now existed. You definitely do not want this code to suddenly modify or resurrect one of your local variables or more generally mess things up in your current context.

On the other hand, if you are using a closure as an implementation for a user-defined control structure, the situation is opposite : the code inside the trunk will never be moved, and you want it to be able to modify local variables or make the current method return.

See how the two notions differ ? Syntactically they look similar, they may even have the same type, but the meaning of everything related to context is different. In the first case, modifying local variables from the context is prohibited (unsurprisingly, Java inner classes have the same restriction), return returns from the closure, this refers to the closure itself. In the second case, modifying local variables from the context is allowed, return returns from the enclosing method, and this refers to the instance from the context. Implementation also is completely different : in one case, inner classes or combinators, in the other case inlining.

In the nice clean world of functional programming, without imperative variables and without non-local exits, this makes no difference, hence the name "closure" in both cases. But we are talking about Java, an imperative language with a lot of context-dependent features, and we cannot make such an approximation. Earlier versions of the BGGA proposal used the names "synchronous" and "asynchronous" to distinguish both cases. I believe these terms borrowed from concurrent programming can only add to the confusion. Let us try to better express what we are talking about.

In the first case, the trunk that you can carry away, we are talking about something quite similar to functions in functional programming (a "closure" in functional programming means a function that refers to local variables from the context -- functional variables or final variables in Java parlance -- and has to carry a local copy of their values). The terms closure or function seem appropriate, although they have recently been much overloaded.

In the second case, we are talking about what is commonly known as macro or as call by name (we are talking here about proper macros, not about the infamous preprocessor macros based on text replacement, which lead to incorrect behaviour). In this case, we are not interested in the functional type associated with a closure : did you notice how all use cases in the BGGA proposal dealing with defining control structures have the same type {=>void} or {=>T} ? We are also not interested in the "trunk" part, since we do not want the code to move. We are only interested in the third feature of closures mentioned above, namely delayed evaluation. The term "macro" is well known and understood, and seems a perfect fit for this purpose. You can have a look at a modern implementation of macros for instance in the Nemerle language at http://nemerle.org/Macros.

Now we have two different names for the two different concepts. We should also have a way to distinguish them syntactically with an eyeblick, ie. something more than just a keyword. We may even wish to give them different types (possibly with some conversion rules) in order to emphasize the fact that are not the same thing.

A last remark : in all the closures proposals I have seen, I deeply miss references to existing languages with similar features. Closures is old stuff (really), and we have to take into account the years of experience accumulated by the community.

Cheers,
Patrick.

Neal Gafter said...

@Patrick: Closures can indeed extend the lifetime of local variables beyond the execution of their local scope. That is a historical fact about closures going back to their roots.

The control aspects of closures appear in languages such as Smalltalk, Ruby, and Scala. Closures used in this (historical) way are strictly more powerful than macros. See, for example, my previous blog post; although subtle, it was intended to make this point.