Sunday, January 28, 2007

A Definition of Closures

There has been some confusion over our proposal to add closures to the Java Programming Language. After all, doesn't Java already have closures in the form of anonymous inner classes? What is the point of adding something to the language that it already has? To some there appears to be a lot in the proposal that has nothing to do with closures, including the control invocation syntax, null as a type, Unreachable, throws type parameters, function interface types, and "nonlocal" returns. In my Javapolis talk I tried to give an explanation for why these features are in the proposal from the practical point of view of what kinds of things would be possible that were not formerly possible. But that begs the question: why do we call it "Closures" for Java? In this blog post I'll try to show how the definition of closures relates to the features of the proposal, and identify which features of the proposal do (and which do not) result from the definition.

Before discussing the definition of closures, it helps to understand the historical context in which the term was introduced.

Lisp was created in the late 1950's by John McCarthy and others at M.I.T. One feature of the language was function-valued expressions, signified by lambda. The name "lambda" was borrowed from a mathematical formalism known as the lambda calculus. Although Lisp was not based on an effort to model that formalism, lambda plays approximately the same role in Lisp as it does in the lambda calculus: lambda is the syntax for a function-valued expression. McCarthy's intent was that Lisp should be designed to be implemented very efficiently, ideally compiled. That desire for efficiency influenced the design of the language.

Lisp used something called dynamic scoping. Logically, in a dynamically scoped language, when a variable reference is evaluated the runtime looks up the call stack until it finds a scope in which a variable of that name is defined. But as a practical matter variable references in a dynamically scoped language can be resolved in constant time simply by maintaining a value cell for each variable name; that value cell caches the variable's current definition. Dynamic scoping is easy to implement in an interpreter or compiler. Some very clever people had found ways to not only take advantage of dynamic scoping, but had developed what would now be thought of as programming patterns that depended deeply on it. But it was soon discovered that dynamic scoping suffered subtle problems, something the Lisp community called the FUNARG problem.

Now we fast-forward to the mid 1970's. On the radio you would hear1 Elton John, Emerson Lake & Palmer, Joni Mitchell, The Captain and Tennille, John Denver, Paul Simon, Paul McCartney and Wings, ABBA, David Bowie, Janis Ian, Aerosmith, Fleetwood Mac, Heart, and Queen. A number of popular Lisp dialects were in use including InterLisp, MacLisp, UCI-Lisp, Stanford Lisp 1.6, and U. Utah's Standard Lisp. All of them were dynamically scoped. It was in this context that Guy Steele and Gerald Jay Sussman developed Scheme, a very simple Lisp dialect.

One thing about Scheme was different2. Scheme was lexically scoped, like the lambda calculus and most mathematical notations, which means that a variable reference binds to the lexically enclosing definition for that name that was active at the time the enclosing lambda form was evaluated. To explain the semantics in terms of the implementation, evaluating a lambda expression was said to produce a closure. This is a function value represented as an object that contains references to the current bindings for all the variables used inside the lambda expression but defined outside it. These are called the free variables. When this closure object, or function, is applied to arguments later, the variable bindings that had been captured in the closure are used to give meaning to the free variables appearing in the code. The term closure describes more than just the abstract language construct, it also describes its implementation.

To many in the Lisp community at the time, it didn't make sense to adopt a Lisp dialect with closures. Not only would it undermine common programming techniques but it would obviously be much less efficient. For a short time these issues were debated, and Guy Steele wrote a series of papers entitled Lambda the Ultimate _____ (where _____ is Imperative, Declarative, GOTO, or Opcode) to help explain the power of lexically scoped lambda (closures). Fast forward only a few years and the debate was largely settled: lexical scoping is Right and dynamic scoping is Wrong and we've all learned our lesson. Since that time the word closure is used to mean lexically scoped anonymous function, but the connotation is that it is possible to get the semantics wrong for any number of reasons, including bugs and concerns about implementation efficiency. It also hints that we should let the language design drive the implementation, not the other way around. Virtually every programming language, whether or not it has something like lambda and anonymous function values, uses lexical rather than dynamic scoping. The basic definition of a closure, however, shows its Lisp roots:

A closure is a function that captures the bindings of free variables in its lexical context.

Around this time, Smalltalk was introduced. Smalltalk is the most pure and simple of the object-oriented languages: everything is an object. Object-oriented languages add a twist to lexical scoping. Rather than binding all names in the lexical scope, free variables appearing in methods are bound in the scope of the object that the method is a member of. In other words, names in a method are bound to members of the "current" object. The current object is accessible by the name "self". Another small but interesting detail is that you can return early from a method in Smalltalk using the syntax "^expression". We'll return (no pun intended) to the significance of this fact later.

Methods aren't the only kind of code abstraction in Smalltalk. There is also an expression form for writing a block expression, which is essentially a lambda. Early dialects had limitations on them, but most modern Smalltalks do not. They are a true analog to Scheme's lambda. Free variables in a Smalltalk block are bound in the enclosing scope, which is typically the scope of some enclosing method. The result of evaluating a block expression is a closure, and like everything else it is an object. In this case the object has a method that you use to invoke the code of the block.

Anonymous functions (closures) were not blindly introduced into Smalltalk just because it seemed like a neat idea, or because they had worked out well in another language. Rather they were integrated fully and carefully into the language. Anonymous functions can properly be integrated into even an existing language, but there is an advantage when adding them early. As Guy Steele's papers demonstrated, they are so powerful that they subsume other language features. If you add them early, you might save yourself the trouble of adding language features that can instead be added as libraries. Smalltalk provides few control constructs directly in the language. Even the conditional "if" is provided as a library method and invoked using blocks.

Two things distinguish blocks in Smalltalk from Scheme's lambda. First, the meaning of "self" within a block refers to whatever meaning it had in the enclosing context. Specifically, it doesn't refer to the closure object itself. Second, the syntax for returning from a method, "^expression", returns from the enclosing method; it doesn't return from the method representing the closure invocation. These two details are a natural consequence of the fact that, while Scheme has only one lexically scoped language construct (variable bindings), Smalltalk has three lexically scoped language constructs: name bindings (like Scheme), the referent of the return syntax, and the meaning of "self". The definition of closures above mentioned only "the bindings of free variables", but that is because the definition was written for the language Scheme, and name (variable) binding is the only lexically scoped construct in Scheme. Common Lisp also has "return" and "goto", and these too are captured lexically in a closure. In order to realize the full power of closures, described in Guy Steele's lambda papers, they must capture all lexically scoped language constructs. Generalizing the definition of closure to cover other languages would require using more language-neutral terminology: instead of "bindings of free variables" we would have something like "lexically scoped semantic language constructs." However, that obscures the origins of the term.

Fast forward more than 25 years, and we're once again listening to some of the same music we listened to in the late 1970's. We are now considering adding closures to Java, a significantly more complex language than either Scheme or Smalltalk. We're not considering them because they seem like a neat idea, or because they worked out well in other languages, or because we're bored. Rather we're considering them: because of the power and flexibility they will add to the programmer's arsenal; because of the improved readability we expect from programs that use closures instead of the existing alternatives; and because of a number of other recently proposed language extensions that will be unnecessary if closures are added. In order to get the full power of closures, they should capture all lexically scoped semantic language constructs. What are the lexically scoped language constructs in Java?

  • The meaning of variable names.
  • The meaning of method names.
  • The meaning of type names.
  • The meaning of this.
  • The meaning of names defined as statement labels.
  • The referent of an unlabelled break statement.
  • The referent of an unlabelled continue statement.
  • The set of checked exceptions declared or caught.
  • The referent of a return statement.
  • The definite assignment state of variables.
  • The definite unassignment state of variables.
  • The reachability state of the code3.

In addition, Java has one other significant difference from either Scheme or Smalltalk: Java is statically typed. That means that each expression has a type at compile-time. So if we add closures, we need to have some appropriate type for a closure. Since a closure is an anonymous function, it is natural to consider adding function types to the language. But this is not a mandate. As you can see by the two variations of our closures proposal (the nominal and the functional versions) we believe it is possible to add closures without adding function types with a limited loss of functionality (higher-order programming becomes impractical). Our proposal for closures addresses every item on this checklist. There are additional features of our proposal (the control invocation syntax and the closure conversion) that don't relate directly to the definition of closures, but which make them integrate very nicely with existing language features. And there are additional features not mentioned in the spec (such as proper tail recursion) that would be helpful to realize the full potential of closures.

What about anonymous inner classes? It turns out that they don't pass muster on any item on this checklist. Let's set aside the fact that local variables from enclosing scopes must be final to be used inside an anonymous class. The problem is that variable names are simply not resolved in the correct scope. They are resolved in the scope of the anonymous class that you're creating, not the enclosing scope. If you're creating an instance of an interface then it's probably not too much of a problem because most interfaces don't have any (constant) variable definitions. But anonymous inner classes fail every other item on this checklist as well, most of them fatally. Most alternative proposals don't actually address any of the items on this list, and so fail to provide the power of closures any more than existing language constructs.

Setting aside all the programming language theory, don't anonymous inner classes provide, in practice, all of the advantages of closures? I believe I've already shown that the answer is no. It is certainly true that for any program you can write using closures, you can write a roughly equivalent program using anonymous inner classes. That's because the Java programming language is Turing-complete. But you will probably find yourself resorting to a significant and awkward refactoring of the code that has nothing to do with the purpose of the code. In fact, you can write a roughly equivalent program using assembly language if you have the stomach for such an effort. On the other hand, true closures increase the power of a language by adding to the kinds of abstractions you can express.

Acknowledgments: my thanks to Gilad Bracha, John Rose, and Guy Steele for filling me in on and fact-checking the terminology and relevant history. Any remaining historical fantasies are my own.

1 This is Guy Steele's impression of the late 1970's music era [personal communication].

2 Scheme was the first lexically scoped Lisp, but certainly not the first lexically scoped programming language. Algol60, for example, was lexically scoped. See also Landin's The Next 700 Programming Languages.

3 Arguably part of the lexical semantics or not, reachability state is valuable to capture in practice.

34 comments:

JodaStephen said...

I have a question :-) If I've understood everything correctly, then an async closure (RestrictedClosure) *will* change the meaning of some of the lexically scoped semantic language constructs (return/continue/break/...). Does this prevent an async closure from being a real closure according to your definitions here? How do other languages handle async closures?

Neal Gafter said...

No, JodaStephen, restricted closures simply won't be allowed to use return, or break/continue to a statement outside the closure.

AlBlue said...

You're defining the term recursively, by saying that an anonymous function is a closure and therefore a clousre is an anonymous function.

In fact, the sets of anonymous functions and lambda closures intersect, but neither one contains the other. It is entirely sensible to have an anonymous function that has no bindings (e.g. increment (lambda x x+1)) is such a case. Also, it's quite possible to have a construct that is a closure -- an instance method implicitly closes over the instance variables -- but it is not an anonymous function.

The problem is that people such as yourself have misunderstood the term 'closure', as in something that makes a term more closed (by binding variables) and mistakenly taken it to be synonymous with 'anonymous function', which it's not. The example in the Lambda Calculus and Lisp are both; but simply taking a definition from Wikipedia and repeating it verbatim does not a definition make.

I asked a number of people from the formal programming community about this and posted my results at http://eclipsezone.com/eclipse/forums/t86911.html for everyone to see. You might also like to discuss the idea on the Lambda the Ultimate weblog (http://lambda-the-ultimate.org/) which is a discussion ground for formal languages programming; it's worth noting that the community here is frustrated by attempts to conflate the semantic concept of closure with the lexical concept of anonymous code functions.

I should point out that I'm not against the idea of Java having such first-order functions -- after all, it makes a lot more sense to have it in as a language feature than having to muck around with reflection all the time -- but the point is that the word 'closure' is wrong for this use. You'll note that pretty much every other langauge -- python, ruby, lisp etc. -- pretty much all use the term 'lambda' for anonymous functions.

Alex.

Neal Gafter said...

Alex: I didn't take my definition from Wikipedia (you may note that the two definitions differ), but rather from the historical context. It was introduced to describe a representation for anonymous functions (or procedures) in a lexically scoped language. The technique was brought into prominence in the context of Scheme. I could not find any definitions or responses to your survey that supports your position that an instance method is a closure over the object it is a member of.

I've been pretty consistent in my use of "closure" as the result of evaluating an anonymous function expression, and not the expression itself. I'm aware of the frustration of the community and I share it.

Finally, I should point out that most of the responses to your survey do support my definition: a closure is still a (trivial) closure even if the number of free variables it closes over is zero.

Patrick said...

Hi Neal--thanks for your continued effort and patience in engaging in this public discussion. I'm much happier with this than with previous changes to Java, which seemed to occur mostly behind closed doors. I have one question--as an interested bystander--what is the relation between the BGGA proposal (and what it would allow for) and what is covered by functors in C++ or the various functor libraries for Java (JGA, for example). I'm asking just out of interest, but it seems there is some overlap--that a library of useful and reusable functors could be implemented using BGGA. Or am I way off here? Thanks, Patrick

Tom Palmer said...

alblue: I'd also say that the current issue is about closures and lexical binding (where anonymous blocks that can be assigned to vars is just the nice/right way to do it). The idea of method references (to avoid reflection) is being discussed separately along with property references/literals. Oh, and Ruby also uses the terms proc and block in addition to lambda for anonymous functions.

Anonymous said...

I'd like see this feature in Java 7.

I think that Eiffel has a good work in this area, they call it Agents you can see from here: http://en.wikipedia.org/wiki/Eiffel_programming_language#Agents

or

http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-367.pdf

Tom Palmer said...

However, on the subject of closure conversion, if instance-attached method references (like "instance#doSomething(String)" or whatever) get into the language, would it be worth supporting closure conversion to explicit interfaces or function interfaces for these? Could be handy for some use cases and maybe keeps things more unified and consistent. Or maybe it would just be overkill and an extra item in a long JLS. I'm not sure at the moment.

Tom Palmer said...

Sorry to post again already, but following my prior comment, maybe "interface literal conversion" is better and more general term than "closure conversion" if it gets applied to method references. (Well, as long as single-abstract-method abstract classes aren't supported, because then "interface" would be a misnomer. But I'll try to avoid further comments for now.)

Tiago Silveira said...

Way to go, Neal! You've done the community a favour. Thank you.

Josh said...

After reading your article it's apparent that closures would be more suited to Java 3 then jdk1.7. Closures would too drastically change the way people write code.

Jochen "blackdrag" Theodorou said...

I think the point of allblue is not so much about the difference between lexical and dynamic scoping. For example if you have "i=j", then j is a free variable. It doesn't matter that this statement alone isn't compilable. In case of "int j=0; i=j" the j is no longer a free variable, it is bound. It is bound by the code. With this in mind it is clear, that any compilable Java program can't have free variables. When looking at an instance method, then it is clear that any access to a field is the usage of a free variable. So if you would execute such a method, by using an instance of the method and applying a class to it (the binding) then it can be seen as closure. But of course only if there is an access to class variables. Of course you can still use an closure as implementation of that and apply a empty binding, but that is the optional part, not the required. And if you keep such a (strange!?) view in mind, then it is clear any block of code not accessing something form outside this block of code can't be a real closure, because it closes over nothing. If you say that it doesn't need to close over something, then everything becomes a closure and the term is useless.
So it has really nothing to do with lexical or dynamic scoping. And in that view inner classes can be seen as closures if they are accessing a local variable or an instance field from the surrounding class for. Of course if the accessed local variable is a primitive or immutable, then the compiler can simply inline it for the inner class. So this one will possibly no closure then. On the other hand, if you think of a reference to the outer this as a way to reference the binding, then the text you wrote about Scheme applies. Same goes if the local variable is a mutable Object (mutable in terms of the compiler does not know if it is immutable), because again a reference to the outer binding is created. So an inner class full fills the requirement of binding at runtime at last in some cases.

I won't say I agree with this view to 100%, I wouldn't even say that I was able to make the point of Alex clear here, maybe not even to 100% correct... but your "check list" is also not very clear to me.

I mean, "this" resolves to the enclosing class, and this is done, even if you apply the lexical view, because each inner class defines a new class. If you think of it as a second way of defining classes, then it is even more clear. Same for shadowing of variables. A local variable can shadow a class variable, so why shouldn't a class variable be able to shadow a local variable or other class variables? If a return is defined to return from the method you are in (lexically), and you are in two methods, then why not return from the inner?

But besides that I must say I don't understand some things in your Blog entry. And that is for example the reachable state of anonymous inner classes. And why that speaks against them being closures at runtime. Same for the (un)assignment state. For example if checking if a variable gets a value before it is read, and if that assignment is in a closure, then I can't say it will get an value for sure. It is the same it is in an if-else and the assignment is only in one of the branches. But maybe you can give some explaining comments on this for the hobbyists in the club of language designers.

I guess you strongly want a closure to be a piece of code, that can be put anywhere, where a block of code is allowed...

Anonymous said...

Neal, realy, what will clusure bring to us that you can not do right now ?

I read a lot and a lot on this subject, but for me this is all like we are bringing more and more functional programming paradigms to java. IE, like static import !

Do we want a object oriented language or do we want a functional one ? That is all the question.

I mean, changes in the language priority is more : introduce runtime genericity (to make genericity usable in genericity, hence in all frameworks), add AOP capabilities (JavaEE 5 has introduced this alreadybut "on their own", let's make it standard).

Java should not follow the language du-jour features but introduce the one that bring real advantages on projects. Runtime Genericity bring tremedous power into frameworks because objects sets can be introspected to perform some housekeeping task (say rendering, editing, ...)... AOP is realy a key aspect in that direction.

fatih said...

Thanks for this excellent blog entry. I am looking forward to the "real" closures to be added in Java 7.

I agree, that statements like "return" must be lexically bound in closures. Otherwise you can not implement methods that behave like control structures.

Keep up the fight!

Henning said...

Excellent article, thanks a lot. We definitely need closures. All the time, we find ourselves struggling with returns from inner classes and checked exception handling, or using hacks like arrays to assign variables out of anonymous inner classes.

The sooner this is added the better, postponing it will only mean more legacy and more trouble later. So please ensure this is added to Java 7!

Henning

Kieron Wilkinson said...

annonymous said: "Do we want a object oriented language or do we want a functional one ? That is all the question."

Personally I would have thought a pure OO language would have to have closures to be called pure OO... For example, Smalltalk is considered to be the "most pure" OO language AFAIK, and what does Smalltalk have? :)

However, Java is not really an object-orientated language - it just has many OO features. I would consider it being made more OO-like by adding closures.

Howard said...

I think that your definition of a closure is a little narrow; in particular the definition that this can only refer to the enclosing object otherwise it isn’t a closure. I don’t think this is the only option for this! The object to which this refers to could be an independent object, e.g. like an inner class. There are many advantages in this alternate definition:

1. You can have multiple methods inside the closure not just one; a redefinition of toString is particularly useful for example when timing an array of things toString can be used to identify what is being timed.

2. When the closure is to be executed at a latter time, i.e. cached and perhaps passed to another thread the independent object is superior since it does not need to pass the outer object if the outer object isn’t referred to.

3. You can have fields in the inner class, you can’t in the closure. The inner class is a type of multiple inheritance; inheriting both its enclosing class and the class/interface it extends and has two this pointers. There are many uses for multiple inheritance because it allows separation of concerns and this is denied in the narrow definition of a closure that you propose.

4. You can inherit partial implementations; i.e. they can be derived from an abstract class, not just an interface.

5. You can see this mismatch between normal OO programming and the closure proposal by the need for closure conversions, i.e. you need a mechanism to convert the closure into a normal object.

On the other side of the coin you will no doubt say that break, continue, and return refer to the enclosing method in your proposal and in an inner class they refer to the method inside the inner class. Having used a number of languages that have closures like you propose I don’t find this a helpful feature, it just creates problems when the closure is executed asynchronously and in any case Java already has a great mechanism for non-local control flow - exceptions.

If you really want some nifty syntax for break, continue, and return out of the outer block make the programmer be explicit, e.g. “name.return value;” could mean that value is returned from an outer method called name. This mechanism is then checked by the compiler, like it were a checked exception, and is part of the closures method’s signature, like a throws clause is. It would be implemented using an exception, but so will your mechanism because you need to generate a runtime exception if the outer method is out of scope when the inner is executed. This alternative is superior because it is checked by the compiler to ensure that if a non-local return is required then any method accepting the closure throws the appropriate exception, if not a compile time error. Similarly for break and continue.

Ola Bini said...

@neal: Very nice writeup. All in all I just have one comment and one question. First, regarding your statement that reads "McCarthy's intent was that Lisp should be designed to be implemented very efficiently, ideally compiled."... Correct me if this is wrong, but in most LISP history articales I've read the story is that McCarthy didn't intend LISP to be a programming language, but a mathematical formalism that could be implemented in itself. It was Steve Russel who realized that if you implemented this LISP-in-LISP-definition in machine code, you would have a Lisp evaluator.
And my question is this; as many already commented, it is possible that adding closures at this point wouldn't have the impact it would've had on a new language. So what kind of changes will you do to the core libraries to support this? Especially the collection classes really would need a major overhaul if we're to benefit from closures.

@Tom Palmer: regarding Ruby's versions of closures, lambda, proc and are different. A block is a code construct. The two Kernel methods lambda and proc take a block and reify it into a Proc instance.

Stefan Schulz said...

This is a great explanation of closure's theory for Java. But I am convinced anyway. Especially, as closures provide means to realize many functionalities as API changes instead of requiring language changes. :)

Something which I may be obsessed with is allowing to hand back control to the closure's caller environment (the place it got "invoke"d). That is, allowing for break, continue and return to exit the closure and returning to the executor. But this is another issue, maybe discussed elsewhere.

Anonymous said...

I googled for versions of the quoted historical research papers, that are accessible without ACM account. Just in case anybody is interested: Funarg problem and 700 languages.

axel said...

I would like to second a point by Howard: Non-local return, break, continue are just not necessary.

The best example is Scheme, the prototypical closures language. Scheme does not have a return statement. It does have call-with-current-continuation, which in terms of usability is roughly equivalent to exceptions in Java, but also consistently copes with escaping closures.

Scheme also has a very general loop construct ("do"), and polymorphic shortcut boolean operators (acting like Smalltalk's ifNotNil:block), which obliviates the need for break and continue. Smalltalk does not need "break", either.

As "ola bini" has pointed out, closures come quite late in Java's life cycle. Therefore some inconsistency might be unavoidable. In case of doubt, would you prefer inconsistency with the OO heritage (inner classes) or with the C heritage (restrained goto) ?

Stefan Schulz said...

Of course, one does not need break, continue or return in a closure. One actually does not need it at all in a language (like one does not need an explicite goto). But it's convenient to use them and the Java folk is used to them. And I do not think, it is unavoidable to have to exclude these statements from being closure-related. Java already provides mechanisms for break and continue to not only relate to the innermost target by using labels, and return could very well be extended similarily.

Howard said...

As Stefan Schulz suggests with regard to non-local jumps: another alternative, to the BGGA suggestion and to my suggestion of "name.return value;", is to style the syntax after a named break or continue. If either Stefan or my proposal was adopted, this would let closures be inner classes (and hence all the advantages outlined in an earlier post).

As Axel seconded and I pointed out non-local jumps are really are in the nice to have category rather than the must have. I can't see that they are a strong argument for one particular closure proposal over another.

feng said...

axel: Scheme does not need return/break/continue because they are equivalent to tail-calls to lexical variables that bind to lambda expressions. Scheme named let loop is a good example (not "do" loop, BTW). I don't know if Scheme invented closure, but it's definitely the first language that gets closure right in terms of definition, semantics and implementation ;-). In this sense, what Neal is trying to accomplish is as close to The Right Closure as anyone wanted in Java.

Howard said...

@Feng,

If you want non-local break, continue, and return then use named ones and stick with the inner class. Not a big deal to me anyway since I hardly ever use them, but if you want them you can have them.

The inner class is the more powerful construct; it has two this pointers, one to the enclosing object and one from the inherited object, i.e. multiple inheritance. Therefore anything you can do with a closure you can do with an inner class, but the reverse is **not** true.

So why not stick with the existing construct? Therefore not confusing everyone with overlapping ideas. And which is the more **powerful** construct anyway?

feng said...

I’d much prefer the current return semantics in the proposal, just for syntactical reason. Using inner class semantics would make code really hard to read because closure syntax is so concise (especially statement block form), you don’t have a syntactic construct like "public R method (T x)" that is explicit and visible like in inner class syntax. All you have is a "{" hanging after a method call. In order to know where a return jumps to, you have to find the right "{" from maze of deeply nested blocks that look almost the same. To me, it is very clear and intuitive that a "return" always jump out of an explicit method definition, be it at top-class level or inner-class level. This is what I remember now, and this is what I’ll need to remember when closure is in the language. When I really want inner-class return semantics, I just write a full-blown inner class. This is what I do now, and this is what I’ll do when closure is available, except I’ll try to refactor and use RestrictedClosure (can we call it ExpressionClosure instead?) most of time.

Nils said...

"The inner class is the more powerful construct; it has two this pointers, one to the enclosing object and one from the inherited object, i.e. multiple inheritance."

Having a reference (or two) is not inheritance.

Michael said...

After programming for 8 years in Common Lisp and then almost 10 in Java:
YESYESYES, just do it!

Peter Lawrey said...

Out of interest, what is the reason for not proposing continue./break with a label.
- Is this because of the added complexity and that this feature is often not used.
- Is it because the exact behaviour is unclear or too difficult to understand in the generalised case.
- Is it that this feature does exist in the languages we are looking to base the closure design on.

I would assume that return, continue, break would be implemented in the generic form as if they were exceptions, but in the optimised form as inlined statements. i.e. If the method using a closure were inlined, it really code perform a return, continue, break just as it does today.

Neal Gafter said...

@Peter: I don't know what you mean by "Out of interest, what is the reason for not proposing continue/break with a label."

First, this blog post is not a proposal, but it does explicitly include the meaning of continue/break labels as part of the lexical scope.

Second, BGGA does indeed capture the meaning of continue/break labels. For example, a labelled break transfers control to the end of the nearest enclosing statement with that label. That's not called out in the BGGA specification because it is already specified in the Java language specification, and we are not proposing to change it.

Anonymous said...

Just to add to the anonymous function part.

In Scheme and Common Lisp functions are objects. Function objects don't have a name. You can't ask a function object for its name. A function is just an object.

A closure is also a function object (with an environment).

A closure can be created in Common Lisp like this:

(defun foo (a)
  (flet ((bar (b) (incf a b)))
    #'bar))

or like this:

(defun foo (a)
  (lambda (b) (incf a b)))

or like this:

(let ((a 42))
  (defun bar (b) (incf a b)))

In the first case FOO returns the local function BAR as a closure.

In the second case FOO returns the anonymous function (a function that is not fbound to any name) as a closure.

In the third case BAR names directly the closure function.

The closure is a function object - so it is possible for the case one and two to set a name to this function. In the third case it's already done.

(setf (symbol-function 'baz) (foo 42))

Now I can call (baz 10) which returns 52 the first time and 62 the second time. Just like (bar 10) - from the third example will do.

We see that naming is a different issue.

So, for Lisp we have the definition you have mentioned:

  A closure is a function that captures the bindings of free variables in its lexical context.

Whether the closure is created from a named function or a lambda expression is of no importance. Whether we find a name for the closure is also of no importance. We can call the closure with a name or without.

sbwoodside said...

You can in fact do everything that proper closures do in Java without significant refactoring ... just some ugly code.

Closures with return values in Java

Neal Gafter said...

@sbwoodside: You've indeed demonstrated that you can do SOME things without significant refactoring, but one example does not prove the rule. See http://www.youtube.com/watch?v=0zVizaCOhME for a discussion of the cases that you cannot handle without significant refactoring.

sbwoodside said...

@Neal Gafter: I actually can't wait for proper syntax for closures in Java. Having watched your talk, I think a lot depends on how much overall control you have of the code base, how complex your API is, etc. For practical purposes, I don't expect to have closures in J2ME for a long, long time, so it's practical to at least observe that some useful closure-ish stuff can be done even without generics.