Sunday, September 17, 2006

Failure of Imagination in Language Design

Failure of imagination - that is, failure to make language constructs orthogonal because you aren't quite sure how people will need to use them together - is one of the most common and severe errors you can make in programming language design. It is an error that is often impossible to correct after the fact. There is an approach to programming language design that you can take to maximize your opportunities to err in this way: consider the interactions of individual language features and decide on a case-by-case basis if the interaction will be allowed or not (or even what the semantics should be) based on whether or not you can find use cases and which semantics best fit those use cases.

This is an approach favored by expert API designers, because the domain of API design is best suited to this approach and these designers expect their expertise to carry from one domain to the other. In an API, the components of the API fit together in particular and well-defined patterns due to the typing and subtyping relationships between the elements; the job of the API designer is to ensure that the patterns you can build with the API elements satisfy a set of use cases.

In a programming language, on the other hand, the elements that are used to assemble programs are ideally orthogonal and independent. Programmers can assemble the elements in arbitrary ways, often unexpected but surprisingly useful ways. The "patterns" movement for example records some of these ideas by creating a common terminology for newly discovered ways of assembling programs from primitives.

To be sure, use cases also play a very important role in language design, but that role is a completely different one than the kind of role that they play in API design. In API design, satisfying the requirements of the use cases is a sufficient condition for completeness. In language design, it is a necessary condition.

Wednesday, September 13, 2006

Nominal Closures for Java (version 0.2)

This post describes a draft proposal for adding support for closures to the Java programming language for the Dolphin (JDK 7) release. It was carefully designed to interoperate with the current idiom of one-method interfaces. The latest version of the proposal and a prototype can be found at http://www.javac.info/.

This revision of the closures proposal eliminates function types, introduces the Unreachable type, and gives related reachability rules. A closure's "return" type is now called its result type. The abbreviated invocation syntax now takes any statement, not just a block, to make nesting work nicely.

Gilad Bracha, Neal Gafter, James Gosling, Peter von der Ahé

Modern programming languages provide a mixture of primitives for composing programs. C#, Javascript, Ruby, Scala, and Smalltalk (to name just a few) have direct language support for delayed-execution blocks of code, called closures. A proposal for closures is working its way through the C++ standards committees as well. Closures provide a natural way to express some kinds of abstraction that are currently quite awkward to express in Java. For programming in the small, closures allow one to abstract an algorithm over a piece of code; that is, they allow one to more easily extract the common parts of two almost-identical pieces of code. For programming in the large, closures support APIs that express an algorithm abstracted over some computational aspect of the algorithm. This proposal outlines a specification of closures for Java without introducing function types.

1. Closure Literals

We introduce a syntactic form for constructing a closure value:

Expression3
Closure
Closure
FormalParameters { BlockStatementsopt Expressionopt }

Example

We can write a closure that adds two to its argument like this:

interface IntFunction {
int invoke(int i);
}
IntFunction plus2 = (int x){ x+2 };

2. The type of a closure

A closure literal has a "closure type" that exists transiently at compile time. It is converted to some object type at compile-time; See Closure Conversion for details. It is a compile-time error if a closure is not subject to a closure conversion.

The type of a closure is inferred from its form as follows:

The argument types of a closure are the types of the declared arguments.

If the body of the closure ends with an expression, the result type of the closure is the type of that expression; otherwise if the closure can complete normally, the result type is void; otherwise the result type is Unreachable.

The set of thrown types of a closure are those checked exception types thrown by the body.

Example

The following illustrates a closure being assigned to a variable of a compatible object type.

interface VoidIntBlock<throws E> {
void invoke(int i) throws E;
}
VoidIntBlock<InterruptedException> sleeper = (int t) { Thread.sleep(t); };

3. synchronized parameters

We allow the qualifier synchronized to be used on formal method arguments. When a closure is passed to such a parameter, the closure conversion (see Closure Conversion) is allowed to close over its context: the closure may use non-final local variables from enclosing scopes, and may use control-transfer statements such as break, continue, and return that transfer control outside of the body of the closure. An overriding method declaration may only add (i.e. may not remove) the synchronized qualfier to formal parameters compared to a method it overrides; an abstract method that overrides one whose parameter is declared synchronized must itself declare the parameter synchronized. If a closure is passed to a parameter that is not declared synchronized, then the only local variables from enclosing scopes it may access are those that are marked final.

Note: this qualifier is necessary to enable API writers to distinguish synchronous from asynchronous receivers of closures. For compatibility with existing APIs, most of which are asynchronous, the default is not allowing closing over the context. The constraint on overriders preserves the substitutability of subtypes: if a closure is allowed when passed to the superclass method, it must be allowed when passed to the subclass method. A consequence of the overriding constraint is that no existing overridable method can be retrofitted to accept synchronous closures without breaking source compatibility. Static methods such as AccessController.doPrivileged can be retrofitted. The synchronized keyword isn't an ideal choice, but it is the best we've found so far.

Names that are in scope where a closure is defined may be referenced within the closure. It is a compile-time error to access a local variable declared outside the closure from within a closure unless the closure is passed to a parameter that is declared synchronized or the variable was declared final.

Note: a local variable that is referenced within a closure is in scope in the body of the closure. Consequently the lifetime of objects referenced by such variables may extend beyond the time that the block containing the local declaration completes.

4. Closure conversion

A closure literal may be assigned to a variable or parameter of any compatible object type by the closure conversion:

There is a closure conversion from every closure of type T to every interface type that has a single method m with signature U such that T is compatible with U. A closure of type T is compatible with a method U iff all of the following hold:

  • Either
    • The result type of T is either the same as the return type of U or
    • There is an assignment conversion from the result type of T to the return type of U; or
    • the result type of T is Unreachable.
  • T and U have the same number of arguments.
  • For each corresponding argument position in the argument list of T and U, both argument types are the same.
  • Every exception type in the throws of T is a subtype of some exception type in the throws of U.

If the closure is not passed to a parameter that was declared synchronized then

  • it is a compile-time error if the closure being converted contains a break, continue or return statement whose execution would result in a transfer of control outside the closure
  • it is a compile-time error if the closure being converted refers to a non-final local variable or parameter declared in an enclosing scope.

The closure conversion to a non-synchronized parameter applies only to closures that obey the same restrictions that apply to local and anonymous classes. The motivation for this is to help catch inadvertent use of non-local control flow in situations where it would be inappropriate. Examples would be when the closure is passed to another thread, or stored in a data structure to be invoked at a later time when the method invocation in which the closure originated no longer exists.

Note: The closure conversion occurs at compile-time, not at runtime. This enables javac to allocate only one object, rather than both a closure and an anonymous class instance.

We are considering an additional qualifier on non-final local variables that allows a closure to access the variable.

Example

We can use the existing Executor framework to run a closure in the background:

 void sayHello(java.util.concurrent.Executor ex) {
ex.execute((){ System.out.println("hello"); });
}

5. Exception type parameters

To support exception transparency, we add a new type of generic formal type argument to receive a set of thrown types. [This deserves a more formal treatment] What follows is an example of how the proposal would be used to write an exception-transparent version of a method that locks and then unlocks a java.util.concurrent.Lock, before and after a user-provided block of code:

interface Block<throws E> {
void invoke() throws E;
}
public static <throws E extends Exception>
void withLock(Lock lock, Block<E> block) throws E {
try {
lock.lock();
block.invoke();
} finally {
lock.unlock();
}
}

This can be used as follows:

withLock(lock, (){
System.out.println("hello");
});

This uses a newly introduced form of generic type parameter. The type parameters E are declared to be used in throws clauses. If the extends clause is omitted, a type parameter declared with the throws keyword is considered to extend Exception (instead of Object, which is the default for other type parameters). This type parameter accepts multiple exception types. For example, if you invoke it with a block that can throw IOException or NumberFormatException the type parameter E would be IOException|NumberFormatException. In those rare cases that you want to use explicit type parameters with multiple thrown types, the keyword throws is required in the invocation, like this:

Locks.<throws IOException|NumberFormatException>withLock(lock) {
whatever();
}

You can think of this kind of type parameter accepting disjunction, "or" types. That is to allow it to match the exception signature of a closure that throws any number of different checked exceptions. If the block throws no exceptions, the type parameter would be the type null.

Type parameters declared this way can be used only in a throws clause or as a type argument to a generic method, interface, or class whose type argument was declared this way too.

6. Non-local control flow

One purpose for closures is to allow a programmer to refactor common code into a shared utility, with the difference between the use sites being abstracted into a closure. The code to be abstracted sometimes contains a break, continue, or return statement. This need not be an obstacle to the transformation. A break or continue statement appearing within a closure may transfer to any matching enclosing statement provided the target of the transfer is in the same innermost ClassBody.

A return statement always returns from the nearest enclosing method or constructor.

If a break statement is executed that would transfer control out of a statement that is no longer executing, or is executing in another thread, the VM throws a new unchecked exception, UnmatchedNonlocalTransfer. Similarly, an UnmatchedNonlocalTransfer is thrown when a continue statement attempts to complete a loop iteration that is not executing in the current thread. Finally, an UnmatchedNonlocalTransfer is thrown when a return statement is executed if the method invocation to which the return statement would transfer control is not on the stack of the current thread.

7. Definite assignment

The body of a closure may not assign to a final variable declared outside the closure.

A closure expression does not affect the DA/DU status of any variables it names.

8. The type Unreachable

We add the non-instantiable type java.lang.Unreachable, corresponding to the standard type-theoretic bottom. Values of this type appear only at points in the program that are formally unreachable. This is necessary to allow transparency for closures that do not return normally. Unreachable is a subtype of every type (even primitive types). No other type is a subtype of Unreachable. It is a compile-time error to convert null to Unreachable. It is an error to cast to the type Unreachable.

Example

The following illustrates a closure being assigned to a variable of the correct type.

interface NullaryFunction<T, throws E> {
T invoke() throws E;
}
NullaryFunction<Unreachable,null> thrower = () { throw new AssertionError(); };

8. Unreachable statements

An expression statement in which the expression is of type Unreachable cannot complete normally.

10. Abbreviated invocation syntax.

A new invocation statement syntax is added to make closures usable for control abstraction:

AbbreviatedInvocationStatement
Primary ( ExpressionList ) Statement
AbbreviatedInvocationStatement
Primary ( Formals ) Statement
AbbreviatedInvocationStatement
Primary ( Formals : ExpressionList ) Statement

This syntax is a shorthand for the following statement:

Primary ( ExpressionList, ( Formals ) { Statement } );

This syntax makes some kinds of closure-receiving APIs more convenient to use to compose statements.

Note: There is some question of the correct order in the rewriting. On the one hand, the closure seems most natural in the last position when not using the abbreviated syntax. On the other hand, that doesn't work well with varargs methods. Which is best remains an open issue.

We could use the shorthand to write our previous example this way

withLock(lock) {
System.out.println("hello");
}

This is not an expression form for a very good reason: it looks like a statement, and we expect it to be used most commonly as a statement for the purpose of writing APIs that abstract patterns of control. If it were an expression form, an invocation like this would require a trailing semicolon after the close curly brace of a controlled block. Forgetting the semicolon would probably be a common source of error. The convenience of this abbreviated syntax is evident for both synchronous (e.g. withLock) and asynchronous (e.g. Executor.execute) use cases. Another example of its use would be a an API that closes a java.io.Closeable after a user-supplied block of code:

class OneArgBlock<T, throws E> {
    void invoke(T t) throws E;
}
<T extends java.io.Closeable, throws E>
void closeAtEnd(OneArgBlock<? super T,E> block, T t) throws E {
    try {
        block.invoke();
    } finally {
        try { t.close(); } catch (IOException ex) {}
    }
}

We could use the shorthand with this API to close a number of streams at the end of a block of code:

closeAtEnd(FileReader in : makeReader()) closeAtEnd(FileWriter out : makeWriter()) {
// code using in and out
}

Acknowledgments

The authors would like to thank the following people whose discussions and insight have helped us craft, refine, and improve this proposal:

Lars Bak, Cedric Beust, Joshua Bloch, Martin Buchholz, Danny Coward, Erik Ernst, Christian Plesner Hansen, Kohsuke Kawaguchi, Doug Lea, "crazy" Bob Lee, Martin Odersky, Tim Peierls, John Rose, Ken Russell, Mads Torgersen, Jan Vitek, and Dave Yost.

Thursday, September 07, 2006

Closures for Java (version 0.1)

This post describes a draft proposal for adding support for closures to the Java programming language for the Dolphin (JDK 7) release. It was carefully designed to interoperate with the current idiom of one-method interfaces. The latest version of the proposal and a prototype can be found at http://www.javac.info/.

This revision of the proposal drops local functions and the special nonlocal return syntax, adds support for generic type parameters that can be used to represent thrown exceptions, and has only one form of closure literal. It also adds an abbreviated invocation statement that supports very convenient control abstraction. Soon I'll present a parallel proposal based on interfaces (that is, without function types) and contrast the two proposals.

Gilad Bracha, Neal Gafter, James Gosling, Peter von der Ahé

Modern programming languages provide a mixture of primitives for composing programs. C#, Javascript, Ruby, Scala, and Smalltalk (to name just a few) have direct language support for function types and inline function-valued expression, called closures. A proposal for closures is working its way through the C++ standards committees as well. Function types provide a natural way to express some kinds of abstraction that are currently quite awkward to express in Java. For programming in the small, closures allow one to abstract an algorithm over a piece of code; that is, they allow one to more easily extract the common parts of two almost-identical pieces of code. For programming in the large, closures support APIs that express an algorithm abstracted over some computational aspect of the algorithm. We propose to add function types and closures to Java. We anticipate that the additional expressiveness of the language will simplify the use of existing APIs and enable new kinds of APIs that are currently too awkward to express using the best current idiom: interfaces and anonymous classes.

1. Function Types

We introduce a new syntactic form:

Type
Type ( TypeListopt ) FunctionThrowsopt
FunctionThrows
throws ThrowsTypeList
ThrowsTypeList
Type
ThrowsTypeList
ThrowsTypeList VBAR Type
VBAR
|

These syntactic forms designate function types. A function type is a kind of reference type. A function type consists of a return type, a list of argument types, and a set of thrown exception types.

Note: the existing syntax for the throws clause in a method declaration uses a comma to separate elements of the ThrowsTypeList. For backward compatibility we continue to allow commas to separate these elements in methods and constructors, but in function types we require the use of the '|' (vertical-bar) character as a separator to resolve a true ambiguity that would arise when a function type is used in a type list.

2. Namespaces and name lookup

The Java programming language currently maintains a strict separation between expression names and method names. These two separate namespaces allow a programmer to use the same name for a variable and for methods. Variables of closure type necessarily blur the distinction between these two namespaces, because they can be used in an expression and they can be invoked.

When searching a scope for a method name, if no methods exist with the given name then variables of the given name that are of function type are considered candidates. If more than one exists (for example, function-typed variable declarations are inherited from separate supertypes), the reference is considered ambiguous.

We allow a function to be invoked from an arbitrary (for example, parenthesized) expression:

Primary
Primary Arguments

3. Closure Literals

We also introduce a syntactic form for constructing a value of function type:

Expression3
Closure
Closure
FormalParameters { BlockStatementsopt Expressionopt }

Example

We can write a function that adds two to its argument like this:

    int(int) plus2 = (int x){ x+2 };

4. The type of a closure

The type of a closure is inferred from its form as follows:

The argument types of a closure are the types of the declared arguments.

If the body of the closure ends with an expression, the return type of the closure is the type of that expression; otherwise if the closure can complete normally, the return type is void; otherwise the return type is null.

The set of thrown types of a closure are those checked exception types thrown by the body.

Example

The following illustrates a closure being assigned to a variable with precisely the type of the closure.

void(int) throws InterruptedException sleeper = (int t) { Thread.sleep(t); };

5. Subtyping

A function type T is a subtype of function type U iff all of the following hold:

  • Either
    • The return type of T is either the same as the return type of U or
    • Both return types are reference types and the return type of T is a subtype of the return type of U, or
    • the return type of U is void.
  • T and U have the same number of declared arguments.
  • For each corresponding argument position in the argument list of T and U, either both argument types are the same or both are reference types and the type of the argument to U is a subtype of the corresponding argument to T.
  • Every exception type in the throws of T is a subtype of some exception type in the throws of U.

6. Exception checking

The invocation of a function throws every checked exception type in the function's type.

7. Reflection

A function type inherits all the non-private methods from Object. The following methods are added to java.lang.Class to support function types:

  public final class java.lang.Class ... {
public boolean isFunction();
public java.lang.reflect.FunctionType functionType();
public Object invokeFunction(Object function, Object ... args)
throws IllegalArgumentException | InvocationTargetException;
}
public interface java.lang.reflect.FunctionType {
public Type returnType();
public Type[] argumentTypes();
public Type[] throwsTypes();
}

Note: unlike java.lang.reflect.Method.invoke, Class.invokeFunction cannot throw IllegalAccessException, because there is no access control to enforce; the function value designates a closure, which does not have access modifiers. Access to function values is controlled at compile-time by their scope, and at runtime by controlling the function value.

8. The type of null

We add support for null and the type of null in function types. We introduce a meaning for the keyword null as a type name; it designates the type of the expression null. A class literal for the type of null is null.class. These are necessary to allow reflection, type inference, and closure literals to work for functions that do not return normally. We also add the non-instantiable class java.lang.Null as a placeholder, and its static member field TYPE as a synonym for null.class.

9. Referencing names from the enclosing scope

Names that are in scope where a closure is defined may be referenced within the closure. We do not propose a rule that requires referenced variables be final, as is currently required for anonymous class instances.

Note: a local variable that is referenced within a closure is in scope in the body of the closure. Consequently the lifetime of objects referenced by such variables may extend beyond the time that the block containing the local declaration completes.

10. Non-local control flow

One purpose for closures is to allow a programmer to refactor common code into a shared utility, with the difference between the use sites being abstracted into a closure. The code to be abstracted sometimes contains a break, continue, or return statement. This need not be an obstacle to the transformation. A break or continue statement appearing within a closure may transfer to any matching enclosing statement provided the target of the transfer is in the same innermost ClassBody.

A return statement always returns from the nearest enclosing method or constructor.

If a break statement is executed that would transfer control out of a statement that is no longer executing, or is executing in another thread, the VM throws a new unchecked exception, UnmatchedNonlocalTransfer. (I suspect we can come up with a better name). Similarly, an UnmatchedNonlocalTransfer is thrown when a continue statement attempts to complete a loop iteration that is not executing in the current thread. Finally, an UnmatchedNonlocalTransfer is thrown when a NamedReturnStatement attempts to return from a function or method invocation that is not pending in the current thread. Likewise, an UnmatchedNonlocalTransfer is thrown when a return statement is executed if the method invocation to which the return statement would transfer control is not on the stack of the current thread.

11. Closure conversion

We propose the following closure conversion, to be applied only in those contexts where boxing currently occurs:

There is a closure conversion from every closure of type T to every interface type that has a single method with signature U such that T is a subtype of the function type corresponding to U.

It is a compile-time error if the closure being converted contains a break, continue or return statement whose execution could result in a transfer of control outside the closure. It is a compile-time error if the closure being converted refers to a non-final local variable or parameter declared in an enclosing scope.

We will want to generalize this rule slightly to allow the conversion when boxing or unboxing of the return type is required, e.g. to allow assigning a closure that returns int to an interface whose method returns Integer or vice versa.

Note: the closure conversion is applied only to closures (i.e. function literals), not to arbitrary expressions of function type. This enables javac to allocate only one object, rather than both a closure and an anonymous class instance. The conversion avoids hidden overhead at runtime. As a practical matter, javac will automatically generate code equivalent to our original client, creating an anonymous class instance in which the body of the lone method corresponds to the body of the closure.

Closure conversion applies only to closures that obey the same restrictions that apply to local and anonymous classes. The motivation for this is to help catch inadvertent use of non-local control flow in situations where it would be inappropriate. Examples would be when the closure is passed to another thread, or stored in a data structure to be invoked at a later time when the method invocation in which the closure originated no longer exists.

Examples

We can use the existing Executor framework to run a closure in the background:

  void sayHello(java.util.concurrent.Executor ex) {
ex.execute((){ System.out.println("hello"); });
}

Here is a definition of an API that locks and then unlocks a java.util.concurrent.Lock, before and after a user-provided block of code:

public static <E extends Exception>
void withLock(Lock lock, void() throws E block) throws E {
try {
lock.lock();
block();
} finally {
lock.unlock();
}
}

This can be used as follows:

withLock(lock, (){
System.out.println("hello");
});

You might find this syntax a bit tedious. Not to worry - see below.

12. Abbreviated invocation syntax.

A new invocation statement syntax is added to make closures usable for control abstraction:

AbbreviatedInvocationStatement
Primary ( ExpressionList ) Block
AbbreviatedInvocationStatement
Primary ( Formals ) Block
AbbreviatedInvocationStatement
Primary ( Formals : ExpressionList ) Block

This syntax is a shorthand for the following statement:

Primary ( ExpressionList, ( Formals ) Block );

Note that this syntax is only a convenience to make some kinds of closure-receiving APIs more convenient to use to compose statements.

We could use the shorthand to write our previous example this way

withLock(lock) {
System.out.println("hello");
}

This is not an expression form for a very good reason: it looks like a statement, and we expect it to be used most commonly as a statement for the purpose of writing APIs that abstract patterns of control. If it were an expression form, an invocation like this would require a trailing semicolon after the close curly brace. Forgetting the semicolon would probably be a common source of error. The convenience of this abbreviated syntax is evident for both synchronous (e.g. withLock) and asynchronous (e.g. Executor.execute) use cases.

13. Definite assignment

The body of a closure may not assign to a final variable declared outside the closure.

A closure expression does not affect the DA/DU status of any variables it names.

14. Exception type parameters

To support exception transparency, we add a new type of generic formal type argument to receive a set of thrown types. [This deserves a more formal treatment] What follows is an example of how the proposal would be used to write an exception-transparent version of the withLock() example we've been using. The syntax is quite tentative at this point (ideas are welcome).

public static <throws E extends Exception>
void withLock(Lock lock, void() throws E block) throws E {
try {
lock.lock();
block();
} finally {
lock.unlock();
}
}

This method uses a newly introduced form of generic type parameter. The type parameter E is declared to be used in throws clauses. If the extends clause is omitted, a type parameter declared with the throws keyword is considered to extend Exception (instead of Object, which is the default for other type parameters). This type parameter accepts multiple exception types. For example, if you invoke it with a block that can throw IOException or NumberFormatException the type parameter E would be IOException|NumberFormatException. In those rare cases that you want to use explicit type parameters, the keyword throws is required in the invocation, like this:

Locks.<throws IOException|NumberFormatException>withLock(lock) {
whatever();
}

You can think of this kind of type parameter accepting disjunction, "or" types. That is to allow it to match the exception signature of a closure that throws any number of different checked exceptions. If the block throws no exceptions, the type parameter would be the type null.

Type parameters declared this way can be used only in a throws clause or as a type argument to a generic method, interface, or class whose type argument was declared this way too.

15. Implementing a Function Type

A programmer may write a class that implements a function type. The implementation is written as a public method with the keyword do in place of the method's identifier. For example

class MyBlock implements void() {
public void do() {
// implementation of the function
}
}

In this way programmers can, for example, create implementations of function types that are serializable.

Acknowledgments

The authors would like to thank the following people whose discussions and insight have helped us craft, refine, and improve this proposal:

Lars Bak, Cedric Beust, Joshua Bloch, Martin Buchholz, Danny Coward, Erik Ernst, Christian Plesner Hansen, Doug Lea, "crazy" Bob Lee, Martin Odersky, Tim Peierls, John Rose, Ken Russell, Mads Torgersen, Jan Vitek, and Dave Yost.

Wednesday, September 06, 2006

The Debate over Closures for Java

We've been continuing work on the closures proposal. I'd like to give you some feedback on what I think about the debate. Tomorrow I'll post an updated proposal, and the next day I'll outline an alternative version of the proposal that avoids function types and contrast it with the current draft; some people think that might be a better approach, though we have some doubts that I'll explain.

Our publication of the draft closures proposal resulted in a great deal of debate and excitement. Even if we end up doing nothing, I think that is very healthy. These kind of things rarely get debated in such public forums. We've been listening to the debate as it rages (on scores of blogs and message boards, and the occasional email), and we're trying to adjust our thinking accordingly. My next post will be an updated version of the proposal, and the following will be an extended discussion of one alternative direction we're considering that addresses some concerns.

There are a few different kinds of feedback we've received, both positive and negative. To those whose feedback has been mostly positive: you've been no help whatsoever. Just kidding. Seriously, thanks for the support. We don't think things are perfect yet, and maybe they will never even be good enough to put anything like closures in the Java programming language. We need to figure out what, if anything, to change to make it better, so your criticism is welcome. If you can quantify precisely how much more fun Java will be to program in, that would help too.

To those with mainly negative or mixed feedback, I'd like to try to classify some of the most common concerns and give general, blanket responses. Please don't take these to mean we're dismissing your concerns. On the contrary, we expect the proposal to evolve, and your concerns are a primary source of information on how the proposal needs to evolve. This classification doesn't cover all the constructive criticism we've received or all the issues we know we need to address, and there is a great deal of overlap between these topics.

Closures don't buy you anything beyond what you can accomplish with inner classes. That just isn't true. See my previous blog post about use cases. Interfaces and inner classes solve the asynchronous use cases just fine, mostly, but they don't touch the synchronous use cases. I don't mean to imply that they can't be extended to cover both sets of use cases, but inner classes in their current form don't do it.

The addition of function types to the language is too much of a price to pay for closures. We have a few reactions to this. First, you're likely to be surprised how much you actually get with closures, and how little it actually costs you in complexity. While it might be possible to get closures without function types - see the alternative proposal for one stab at that approach - what you lose doing things that way is not trivial.

The structural typing of function types doesn't feel right in Java, which normally does its typing via programmer-defined named types. Actually, function types dovetail nicely with interfaces due to the closure conversion. The structural typing you get with closures is much like you would get using generic interfaces and using wildcards heavily in your code, except much simpler.

The proposal introduces a second way to do things that folks already do with interfaces. The concern here is that the language should provide strong guidance on how to do any particular kind of thing, and Java has already chosen to pass snippets of code around as objects that implement interfaces, typically via anonymous classes. Moreover, some existing APIs might look silly doing things the "wrong" way once a new way is introduced. I don't think this concern holds water because virtually all of the existing APIs in the JDK that I'm aware of are best written using interfaces rather than function types. I've heard the debate regarding the JDK notification APIs (for example, in swing) and what to do about them. I think they should be left alone.

Ordinary Java programmers won't be able to understand this stuff. Many folks who say this don't make it clear which parts of the proposal they find most troubling, or perhaps they don't see the proposal as decomposable into troubling and acceptable parts. The Closures for Java draft specification was written to be more like a specification than a tutorial. It is not a suitable thing to read to learn the language construct, any more than reading the Java Language Specification is a suitable way to learn object-oriented programming. Rather you would learn the feature by examples of gradually increasing complexity. I haven't provided that, and I apologize. I've tried to remedy that in my later blog posts but I'm afraid I'm not very good at that kind of writing.

If you believe that closures are inherently a difficult concept for programmers to understand, or some kind of language "trickery", I suspect you probably haven't done much coding in a language that supports them (see Steve Yegge's blog post Language Trickery and EJB, and Curtis Poe's article Sapir-Whorf is not a Klingon). In practice they are pretty straightforward and obvious. Things are complex with anonymous inner classes, and any tutorial would need to explain them as well. I will present an alternative spec in a few days that does away with function types entirely, syntax and all; with that change the spec may seem simpler than the previous one, but I believe as a practical matter it will be harder to use. Also gone are local functions from our proposal entirely. Those who identified function types as the main problem might be satisfied with the version of the proposal lacking function types. But if you have a hard time understanding anonymous inner classes, closures won't make things any easier as we can't remove anything from the language. There also needs to be a prototype implementation that people can actually use to get a feel for what closures can do. I eagerly await an open-source javac on which we can prototype this.

How much more expressive does the language become given the complexity of the additions? The gist of this concern is that adding closures seems to increase the complexity of the language substantially. Is the additional expressiveness worth the additional complexity? The best result would that we end up with an expressive but simple programming langauge. The question is: how much more expressive can you get with minimal changes to the language? We believe that closures adds significantly to the expressiveness of Java in an area that the language is severly lacking. The asynchronous use cases are handled just fine today, but the synchronous use cases like control abstraction and some kinds of functional programming idioms are very difficult.

Are you just doing this to compete with C#/Ruby/Javascript or add the latest experimental/research language feature? Closures have been an idea simmering in the background since the earliest days of Java, and it is the subject of repeated RFEs. Closures predate Java by many years. Closures were in Simula (the original object-oriented language), Lisp, and Smalltalk. They really do make some things easy that are difficult or impossible to express without them. Anonymous inner classes were a half-measure that made some use cases easier, but left some use cases unaddressed. Competition from other languages has shaped the political landscape and the programmer mindset to make closures more acceptable than they were before, so the time is ripe to propose them.

Values of function type can't be made serializable. Agreed, a problem in the previous spec. We intend that you should be allowed to "implement" function types so that you can make them serializable.

I can't stand the syntax. We've heard this before about things that people ended up loving once they started using them. We're open to alternatives. The alternative of just using anonymous class instance creation expressions doesn't work. Closures are not just shorthand for making an anonymous class, because names appearing inside the closure are resolved in the context of the enclosing scope instead of in the context of the anonymous class being created. Perhaps there is a better way of writing a function type. Perhaps we can improve the syntax for writing a closure - but it does require its own distinct syntax in order to get the expressive power. Your suggestions and ideas are welcome.

The synchronous and asynchronous use cases are so different that they deserve completely different syntax. I'm not sure about this argument. First, in the proposal the two kinds of APIs do use completely different syntaxes: one uses function types and the other uses interfaces. A closure means precisely the same thing (if it is allowed) whether used in one kind of API or the other. Any difference in meaning is because of what the API elects to do with the closure (for example, invoke it or not invoke it). Therefore, to understand a use of a closure, you need to consult the documentation or specification of the method it is passed to. In this respect a closure is no different than a value of any other type. I suspect I don't fully understand this concern, and we'll keep listening.

Closures will be implemented using the JVM invokedynamic bytecode; won't that hurt performance? The short answer is: no, not necessarily, and there are alternative implementation strategies that use existing instructions.

Compatibility with the existing language demands an inelegant/kludgy design for closures. Frankly, I don't believe it. I believe that the Java language in JDK7 can have closures AND seem to have been designed with them from the beginning, with virtually all existing APIs using the most appropriate idiom. Some minor syntactic challenges remain: closures would probably better be written with a new keyword (lambda? fun?), but can't due to backward compatibility; a related issue is the difficulty of using "," or "|" in the syntax for disjunction types for type parameters representing thrown exception types. I believe we are likely to converge on a syntax that feels quite natural once people start trying to use it.

A related argument compares the Java programming language to a delicious Oreo® Cookie: you can add syntactic sugar to its outside and still have a nice Oreo cookie, but if you disturb its delicious design center it just won't be the same anymore. Implicit in this argument is a warning (or veiled threat) that hordes of Java programmers will abandon the platform if anything so horrible is done to the language. I think this line of reasoning is overly melodramatic. In fact, I believe the addition of closures will attract a number of programmers who have previously had no choice but to seek alternative languages due to limitations of Java.

Closures are too powerful; programmers should not have the freedom to effectively define their own control constructs, because they will write programs that are too hard to understand. Instead, the argument goes, we should add new statements to the language for each of the important use cases we know about and leave it at that. The gist of this concern is that while the Java programming language today lends itself to easily understood programs, the addition of closures will result in a language in which readers of programs have no firm ground to stand on, and will not be able to tell what a program means. I've only heard this concern raised by a small number of people and our response (thanks to James Gosling) is: by the same logic we shouldn't allow programmers to write their own methods either. If you're reading a program with invocations of unfamiliar methods, you may have to consult the documentation of those methods to understand what the code does; if the code is well-written the name of the method will give you a good hint. This is true whether or not the method receives a closure as an argument. Related to this argument is the concern that the addition of closures will make it possible to write programs that are really hard to understand, but in reality poor programs can be written in any language. This line of reasoning is made by those who consider themselves advocates of the B&D (Bondage and Discipline) school of language design, but I would characterize it as the S&M (Sadism and Masochism) school. The subtext is that we should not trust ordinary programmers with anything so expressive as closures.

Closures should work with classes that have a single abstract method, like java.util.TimerTask, in addition to single-method interfaces. OK, sure, that sounds like a good idea.

There have also been proponents of various other languages like Smalltalk, Ruby, and Javascript noting with smug amusement that the Java folks (or some of them, anyway) finally "get" closures. To them I say "Pfffft!"

Tuesday, September 05, 2006

Control abstraction

All of the examples of synchronous use cases for closures that I've shown so far fall into the general category of control abstraction. The idea is that you have a common pattern appearing in your code, and you want to abstract the common parts while allowing the caller to provide the parts that differ. You can think of these use cases as satisfying a desire to extend the set of statements available in the language. Java is already pretty good at data abstraction, but not so good for control abstraction.

Thanks to a blogger whom I can't identify, I have another simple use case for closures of the synchronous variety (still control abstraction, though). I would love to be able to use this in the implementation of my project, Google Calendar to simplify the code in hundreds of places.

Many places throughout our code we have this pattern appearing:

 
void someOperation() throws SomeException { 
    long t0 = System.nanoTime(); 
    boolean success = false; 
    try { 
        someOperationInternal(); // may fail, throwing SomeException 
        success = true; 
    } finally { 
        long elapsed = System.nanoTime() - t0; 
        logElapsedTime("someOperation", success, elapsed); 
    } 
}

The presence of this code helps us with realtime (i.e. elapsed time) performance measurement, but it clutters up our code and tends to obscure the logic of the application. We would prefer to write a timeOperation method once, thereby enabling us to write the above code as follows:

 
void someOperation() throws SomeException { 
    timeOperation("someOperation", (){ 
        someOperationInternal(); // may fail, throwing SomeException 
    }); 
}

Or, better yet, (presuming the language supports the abbreviated invocation syntax that I'll describe in a future post)

 
void someOperation() throws SomeException { 
    timeOperation("someOperation") { 
        someOperationInternal(); // may fail, throwing SomeException 
    } 
}

The timeOperation method can be written (once) as follows:

 
<E extends Exception> 
void timeOperation(String operationName, void() throws E block) throws E { 
    long t0 = System.nanoTime(); 
    boolean success = false; 
    try { 
        block(); 
        success = true; 
    } finally { 
        long elapsed = System.nanoTime() - t0; 
        logElapsedTime(operationName, success, elapsed); 
    } 
}

This is the best way I've seen to remove this kind of redundant code that is currently repeated hundreds of times throughout our application. This example should help illustrate why closures will help make code much easier to read. Certainly timing support could instead be done with an additional extension to the Java language: a new statement for timing blocks of code. Do you also add a new statement for closing your streams at the end of a block, and one for locking and unlocking your java.util.concurrent locks around a block of code? Once you start down that path, where does it all end? We'd prefer to give ordinary programmers the ability to write these kinds of methods (and perhaps place some of them into the JDK) rather than adding a series of language extensions to address particular use cases.

Besides control abstraction, there are also important synchronous use cases for closures inspired by functional programming idioms with a very different feel, and I'll blog about that separately.