Prexonite Script

Prexonite 1.2.1 – Partial Application and Mercurial

Mercurial

For 1.2.1 I switched my version control system from Subversion over to Mercurial, still hosted as an open source project at assembla.com. Subversion is great but for me there are some benefits in using Mercurial.

Easier than Subversion

I personally find Mercurial easier to use than Subversion, even though there are more concepts involved (local repository versus working copy/pull versus update). The main advantage of Mercurial is that you can clone the main repository, experiment around, create branches, create tags, attempt complicated merges and when it doesn't work out, you can just throw away your local repo and make a new clone (or evacuate the successful commits by cloning locally up the last "good" revision). In Subversion, all the action happens on the central server. If you make a mistake it's logged for all eternity.

Local commits

The second big advantage of Mercurial is being able to commit your changes locally. Sure, every commit should be a working system, but that’s more of an ideal than a realistic scenario. I want to be able to save my work in a state that is at least semi-working before I attempt a risky rewrite. I also don't want to work without revision control wen I'm on the road. Sure there is 3G, but with Mercurial and local commits my laptop battery lasts longer.

Partial Application

One to the big new feature in this and some past releases. Partial application is a new feature of Prexonite Script that allows you to define function values for very simple expressions in a succinct and, hopefully, intuitive way.

f(1, ?) // as a lambda: x => f(1, x)

You can turn almost any simple expression into a partial application by denoting the open arguments as question marks (?). The value of such a partially applied expression will be a function-like value. When the partial application is applied, the questions marks will be substituted with the provided arguments and the expression will be evaluated. Any operand of the expression, that is not a question mark will only be evaluated once at the point where the partial application is created.

Many expressions support partial application:

new List(?), 
x.Append(?), 
?.ToString(), 
System::Console.WriteLine(?),
? is List, 
?~Real

Including all unary and binary operators

1 + ?, 
?*2, 
x & ?, 
y and ?

You can also define multiple open parameters

f(?, 1, ?), 
? == ?

Or map the supplied arguments in arbitrary orders or multiple times

f(?1, 2, ?0, ?0)

Limited to simple Expressions

A key concept of partial application, is that the placeholders can only as direct operands to the operation/function to be partially applied.

f(1+?,g(16*3)+2)

Will not produce a partial application of f, but pass the partial application (1 + ?) to f as an argument. You either have to resort to a full blown lambda expression for this

x => f(1+x,g(16*3)+2)

Or use function composition (also newly introduced)

? + 1 then f(?, g(16*3)+2)

The `then` keyword combines to functions into one:

g then f //means about the same as `x => f(g(x))`

Note how all non-placeholder operands of partial applications can be arbitrarily complex.

Differences between lambda expressions and partial applications

Lambda expressions and partial applications are in most cases not equivalent. The key difference is, that all non-placeholder arguments, the so-called "closed arguments", are evaluated when the partial application object is created, not when the resulting object is called — as is the case with lambda expressions. In the example above, it might therefore be desirable to resort to the more difficult to understand function composition syntax (using `then`) in order to avoid re-computing `g(16*3)+2` over and over again.

Partial applications are sadly not as fast as lambda expressions. Invoking a partial application involves the copying the closed arguments and the freshly supplied actual arguments into an effective arguments array, a step that lambdas don't have to perform. The effect isn't that dramatic, though. Partial application is measurably slower, yes, but within the same range.

The creation of partial applications, on the other hand, is a different story. Whereas for a lambda expression, the runtime just has to capture references to all shared variables, in the case of partial applications, a mapping consisting of entries for both placeholders and the closed arguments has to be computed. The creation of a partial application is an order of magnitude slower than the creation of a closure (the lambda expression function object). So if you care deeply about performance, don't create partial applications in tight inner loops.

Other Changes

Macro references

The illusion that macros can be used just like any function is pretty good, but not perfect. It breaks down miserably, when you try to take the reference of a macro or attempt to create a partial application thereof. For this release, the compiler will detect and reject any such attempts. Macro writer can make use of the `macro\reference` macro to work around this limitation.

For most macros, taking a reference or creating a partial application, doesn't really make sense anyway. But there are some exceptions. I hope that the next release will address these scenarios, and allow macro writers to make their macros compatible with partial application and reference-taking.

Single Quotes

This recent release has also seen the addition of the single quote (') as valid character for identifiers (except at the beginning) and as a separator for number literals.

function bob(cat)
{
  var bob's_cat = cat + "of bob";
  return bob's_cat + " weighs " + 100'000 + "kg";
}

Single quotes in number literals are accepted, but otherwise ignored. They're thrown away right after lexical analysis, so not even macros can find out if there ever were single quotes.

In identifiers, single quotes have no special meaning, they're just characters and you can use them in any way you like. As long as the identifier doesn't start with a single quote, you're fine.

Implementation of operators

Except for the lazy logical ones, all operators in Prexonite are now implemented as commands and no longer part of the various virtual machines. This doesn't just simplify the latter, it also makes partial application of those operators possible. As a user of Prexonite Script, you won't notice the change at all, even if you have assembler code, that uses the operator instructions (these are transparently translated into appropriate command calls).

If you want, you can redefine the built-in operators. Whenever you write `1 + 2` it is compiled as `(+)(1,2)`. If your function `my_plus` is declared with the alias `(+)`, it will be called instead. You probably don't want to do that, but it's possible. (And yes, (+) is a proper identifier. You can even use it in meta data sections)

CIL Extensions

A requirement to implementing partial application efficiently in CIL, but without adding new VM instructions, was the addition of CIL Extensions, a new API that allows commands to gain access to compile-time-constant arguments during CIL code generation. This essentially enables variable arguments instructions. So far, partial application is the only feature that makes use of this.

v.Next

The next planned release is going to be 1.2.2, focusing on generalising the macro system. The most important feature is making the macro system also available to commands, and thus to managed code.

As this will involve a partial rewrite of a substantial portion of the macro system, it could take a while. Hopefully, it'll also support the partial application of macros and a cleaner, more isolated interface for macros.