Prexonite Script

Prexonite 1.2.3 — Best Friends Forever

I was tempted to just skip this silly version number, but here we go: I humbly present to you

(No, I couldn’t resist :3 )

As always, Prexonite release 1.2.3 comes with

• Prexonite v1.2.3 binaries
• Prexonite v1.2.3 source

Sadly linux-compatibility didn’t make the cut, so Prexonite still only builds on windows. The binaries might or might not work with mono.

Stability

In spite of the silly version number, v1.2.3 is a very serious release. A lot of the work of the past couple of months went into making sure that you could depend on what I ship as part of Prexonite. I extended the automated test suite to cover the PSR, while also running existing test cases in multiple configurations.

I don’t really want to call it a long-term-support version, because unless there actually is a need for support (in the form of backported patches), it would be an empty phrase. The idea is more that, starting with the next version, I will be bold and start extending the language in a way that will change how you write Prexonite Script programs.

Breaking changes

This release of Prexonite Script comes with two breaking changes that affect a number of existing programs, both compiled and in script form.

var args fallback was removed

In Prexonite, all local variables are initialized to null, with the notable exception of var args, which holds the list of arguments passed to the function. However, in previous versions of Prexonite, there was a rule that allowed you to have a function parameter called “args” that would not get overwritten with the argument list. Instead the “magic variable” for that particular function would be “\args”. If there was a parameter called “\args”, the magic variable would be called “\\args” and so on.

This behavior is unintuitive to say the least. It also makes writing macros that access the argument list hard to write. So starting with this release, the var args fallback mechanism is no longer part of Prexonite. If you have a parameter called args, tough luck. This change affects the Prexonite execution engines, which means that both compiled programs and scripts are affected.

psr\ast.pxs SI is now implemented as a function

SI from psr\ast.pxs is a very handy shortcut for Prexonite compiler constants and utility functions. SI stands for “SymbolInterpretation”, but has grown to also provide call and return mode constants. In short, it appears very often in macro code. Previously it was implemented as a global variable that was initialized by some code in a build block. While this works perfectly for both the interactive interpreter and just running the script, it doesn’t support serialization (storing the in-memory representation of an application in a *.c.pxs file). While a serialized application can be loaded and executed just fine, it can’t be used to continue compilation, because the build-code that initialized the SI variable at compile-time is absent.

The solution is to implement SI as a function that lazily initializes the global variable. Due to the principle of information hiding underlying Prexonite Script’s function invocation syntax (parentheses are optional if the argument list is empty) this doesn’t pose a problem for existing script code. Existing compiled code doesn’t matter because of static linking. The one area where the change breaks code, is with meta macros (macros that write macro code). Code generated by macro code often refers to SI, because it is simpler to generate than accessing the corresponding static CLR fields (and more performant).

So: If your code refers to ast\gvar(“SI”), you’re in trouble. Now, you could just use ast\func(“SI”), but psr\macro.pxs comes with a better option: ast\getSI gives you a node referring to SI, whatever SI is in your distribution. Just Ctrl+F your code, searching for “SI” (with the quotes) and replace the ast\gvar expression with ast\getSI.

###Improved round-tripping

Up to this point, storing a compiled application into a *.c.pxs file did alter that application in a significant way: In memory, not all functions and global variables are represented by a symbol. Inner functions for instance had no global name that you could refer to. When you store this application, all these “hidden” functions will be represented as top-level functions. When this file is loaded again, the compiler naturally can’t differentiate between functions that were “originally” top-level and “hidden” functions. True, the ParentFunction meta key could be consulted as a heuristic, but there are other examples of functions that have no global symbol (e.g. generated by a macro).

Starting with Prexonite v1.2.3, there is a new meta switch called ‘\sps’, which is short for “Suppress Primary Symbol”. It instructs the compiler, not to create a global symbol for the defined name of the function (the id that immediately follows the function keyword). Instead, all symbols are declared explicitly in the symbols section at the end of the file.

If you previously relied on all functions getting global symbols after round-tripping (storing-then-loading an application), you’ll be in trouble: The storage algorithms now add \sps to all functions (even originally top-level ones) and declare all global symbols explicitly.

Safety and coroutines

Prexonite comes with two execution engines: a bytecode interpreter and a CIL compiler. The former is used by default. You have to explicitly request CIL compilation by issuing the CompileToCil command at runtime (at this time, you cannot compile Prexonite byte code ahead of time). From that point on, the CIL implementations of functions is always the preferred way of executing a function. The interpreter only acts as a fallback mechanism. And there is a good reason for that besides performance: safety.

Exception handling in the bytecode interpreter is really just “best-effort”. After an exception occurs, I cannot guarantee that I will be able execute the corresponding catch or finally block. This is mostly due to the open nature of the interpreter API. It is possible for a stack context to remove itself from the stack, preventing the interpreter from applying the handlers in that stack frame to exceptions. Even worse, from the interpreters view, functions can just stop executing at any time for no apparent reason. For instance when used a coroutines.

While exception handling itself will usually work as advertised, finally-blocks in coroutines are fundamentally broken. They only work when an exception is thrown or when control gets transferred naturally (i.e., reaches the end of the try block). But especially for coroutines there is a very important third scenario: The sequence stops being evaluated at some point (code might just look at the first couple of elements and then abandon the sequence). If your sequence is holding onto an opened file, you’ll be in trouble.

Exceptions are a cool concept but they are also hard to implement correctly (at least in relation to any particular language feature). Coroutine support was easy to add to the interpreter for precisely the same reason it is not safe: The interpreter does not have tight control over the code it executes. The ability to abandon a computation at any point made coroutines almost trivial to implement, while making correct exception handling almost impossible.

For the nearer future this has the following consequences:

• You should use the CIL execution engine whenever possible. Prexonite v1.2.3 still does not compile to CIL by default, however!
• The interpreter will remain part of Prexonite, mostly as a “quick and dirty” way of executing code (build blocks, initialization code, REP-loops)
• The syntactic support for coroutines will remain in Prexonite language. I might one day implement a coroutine transformation targeting CIL.
• The compiler tries to detect and warn about usages of yield inside protected blocks. Yield statements outside protected blocks (i.e., in a naked for-loop) remain perfectly safe.
• You can use the syntax new enumerator(fMoveNext, fCurrent, fDispose) to create native Prexonite sequences (no boxing of CLR objects involved). This can be more performant than coroutines, since the individual functions can be compiled to CIL.

Making the language more friendly

Prexonite Script has never been a simple language, but I have used this release to make some aspects of it a bit less verbose and more programmer-friendly:

Lists allow for trailing comma (‘,’)

Whenever you encounter a comma separated list in Prexonite Script, it is now legal to add an additional comma at the end. This even goes for argument lists, making max(a,b,) a perfectly valid invocation of the max function. Similar features can be found in many of the more pragmatic languages out in the wild. It makes manipulating lists that represent data (often found in embedded domain specific languages) much easier and simplifies Prexonite Script code generators (they no longer have to worry about not adding a comma after last entries).

More sane meta information blocks

In a similar move, meta entries in a meta block need no longer be terminated by a semicolon (‘;’); they are instead separated by semicolons while allowing for trailing semicolons. Existing code (where there always is a trailing semicolon) thus remains valid, while the often found single-entry meta blocks look less silly.

[is test;] vs. [test]

But wait! Where did the ‘is’ keyword go? Well that is now optional too. Essentially whenever you omit the value of a meta entry, it is interpreted as a boolean switch. This also works on the global level, but there the (terminating) semicolon remains mandatory.

Reliability

Up to this point, the scripts found in the PSR (please don’t ask what this acronym stands for) were never really considered a part of Prexonite/Prexonite Script. I just shipped them with every release, made them available to all scripts loaded via Prx.exe. They changed interface frequently and were often quite buggy.

There never was a need to. The PSR isn’t really a standard or runtime library. It’s just a collection of scripts that I found useful for day-to-day hacking in Prexonite Script. Prexonite itself doesn’t depend on the PSR in any way (this makes writing the command line tool a bit awkward at times).

Things have changed a bit with the inclusion of psr\macro.pxs, which is pretty much the only sane way to write macros in Prexonite Script and has de facto attained the status of a proper standard library. As a consequence I have started writing automated tests (of the unit/integration sort) for the most widely used PSR files.

I don’t have an automated testing framework for Prexonite script files yet, but you can use psr\test.pxs today to start implementing your own tests. In the spirit of xUnit it features a meta tag “test” and an “assert” function. Running all tests in an application is accomplished via “run_tests”. You can also specify additional tags that need to be present for a test to be included in the suite. By default, “run_tests” renders its results to standard output via what is called the “basic_ui”. You can provide your own “ui” (a structure that supports certain operations.) For more details, have a look at the implementation of basic_ui. You can of course use psr\struct.pxs to create the structure, I just didn’t want psr\test.pxs to depend on psr\struct.pxs and through that on psr\macro.pxs.

Through some arcane trickery (a bit of T4 and a Prexonite Script file) the automated test written using psr\test.pxs are executed as part of the NUnit test suite for Prexonite itself. One test case (a function) in Prexonite Script is mapped to a set of test cases in the NUnit world.

The way forward

The next release v1.2.4 will not be an ordinary version. After this rather uninteresting iteration, I will be a bit more bold in the next one, as hinted at in the first section. While staying largely backwards compatible, v1.2.4 will include a number of experimental features, that might be disabled by default. It will also break the Prx.exe command line interface. Look for more details in future posts.