Small milestone reach today

Application is now able to:

1. Create Script Engine (SE)
2. SE hooks up to a temporary debug event in Test App
3. Test App asks SE to load LSO-file
4. SE compiles LSO-file into .Net Assembly
5. SE loads .Net Assembly
6. SE Initializes the .Net Assembly and hands over a "LSL BultIn Commands" object (this object is the scripts command-object)
7. Script Manager adds Script to an Object[].Script[] structure
8. By pressing a button in Test App an event is sent to SE
9. SE receives event and queues it to correct ObjectID + ScriptID
10. Separate thread dequeues event and sends it to Script Manager
11. Script Manager executes function
12. llSay() is displayed to debug console

Status on initial SE implementation

With the model described in the two prior postings we will have simple scripts support up and running fairly quickly.

Other developers will be able to extend script support while I continue working on Compiler and ScriptEngine. Both incoming events and outgoing LSL builtin commands needs extending.

We will lack script yield inserting. This is where the preparation of .dll for microthreading will happen.

We will have a simplified version of Event Queue Manager that will use a custom threadpool to execute events and brutally kill any long-running events.

We will havea simplified version of AppDomain handling that basically just loads all scripts into one AppDomain with no code security.

We will have a basic LSL Compiler that can compile the simplest LSL scripts, but that others can extend command support for.

In short, the scripts should be able to llSay() during rez. But can easily be extended by anyone to support the commands we need. The basic structure is in place, and we can build upon that bit by bit.

ScriptEngine - Events

The builtin commands for SE is simple enough. But incoming events could need some more planning.

SE will have a set of events that is fired based on stuff that happens in OpenSim (EventHandler). This EventHandler will translate the event into a LSL-compatible event, including picking up the required parameters.

The EventHandler needs to know what object, and possibly what script, the event is for. Then it will queue this event for the correct script(s) (multiple scripts may subscribe to the same event in an object).

A Event Queue Manager will process the event queue under a separate thread and execute the correct script functions with the correct parameters. By using a separate thread OpenSims event thread is free to return immediately.

What scripts subscribe to what events is determined during script load. Either by dynamically identifying function names, or by having the script hook itself up during load. The functions will be stored by reference to avoid using expensive Reflection-calls for every script event.

One possiblity for extra speed is to dynamically (using CIL) create a call object in same AppDomain as script is located that does some of Event Queue Managers work.

ScriptEngine

The time has come to start implementing script support into OpenSim.

ScriptEngine (SE)
OpenSim will have an interface for SE. It will load SE dynamically using reflection. SE will be in separate .dll Assembly.
During load SE will hook itself up to the necessary events in OpenSim.
This includes: Object rez (look for active scripts), object derez and all events relating to objects (touch, sit, etc).

SE might also during startup need to iterate through existing objects to see what scripts it needs to start. This depends on at what point SE is loaded.

All access to OpenSim from scripts will go through SE.

Compiler
When SE starts a script it will determine script type. If script type is LSO (LSL ByteCode) the script will be compiled into .Net Assembly .dll before load. Compiler is a separate .dll Assembly, but hardcoded into SE.

Loading scripts
* If script is LSO, compile to .Net Assembly
* Modify .Net Assembly to support microthreading (will be done later in project)
* Find available AppDomain for the script (currently only default appdomain)
* Load script into AppDomain
* Create a private instance of BuiltIn commands for the script
* Create object of script handing it the private BuiltIn command instance

Some BuiltIn functions will require information about who is executing them, for example for object permissions or feedback events. Because .Net Assembly can not be trusted (it is uploaded by users) ScriptEngine will create a private instance of BuiltIn commands. This instance will contain and pass on the necessary ID's to OpenSim - as well as work as a translator between LSL-functions and OpenSim API.

Operation
A script instance is made up of ScriptID and ObjectID. This should always result in an unique combination for any script.

ScriptEngine will receive events from OpenSim and forward the events to the required scripts. During this forward it will also enforce microthreading.

Current Status
Class OpenSim.ScriptEngine.DotNetEngine.ScriptEngine has basic framework and is able to load and execute scripts. Only missing event hookups and actual implementation into OpenSim.

Current OPCODE implementation status

This is the current OPCODE implementation status:

OPCODE                  IL_Processor         Implemented

NOOP                    *                    *
POP                     *                    *
POPS                    *                    *
POPL                    *                    *
POPV                    *                    *
POPQ                    *                    *
POPARG                  *                    *
POPIP
POPBP
POPSP               
POPSLR
DUP
DUPS
DUPL
DUPV
DUPQ
STORE                   *                    *
STORES                  *                    *
STOREL                  *                    *
STOREV                  *                    *
STOREQ                  *                    *
STOREG                  *                    *
STOREGS                 *                    *
STOREGL                 *                    *
STOREGV                 *                    *
STOREGQ                 *                    *
LOADP                   *                    *
LOADSP                  *                    *
LOADLP                  *                    *
LOADVP                  *                    *
LOADQP                  *                    *
LOADGP                  *                    *
LOADGSP                 *                    *
LOADGLP                 *                    *
LOADGVP                 *                    *
LOADGQP                 *                    *
PUSH                    *                    *
PUSHS                   *                    *
PUSHL                   *                    *
PUSHV                   *                    *
PUSHQ                   *                    *
PUSHG                   *                    *
PUSHGS                  *                    *
PUSHGL                  *                    *
PUSHGV                  *                    *
PUSHGQ                  *                    *
PUSHIP
PUSHBP
PUSHSP                  *                    *
PUSHARGB
PUSHARGI                *                    *
PUSHARGF                *                    *
PUSHARGS                *                    *
PUSHARGV
PUSHARGQ
PUSHE                   *                    *
PUSHEV
PUSHEQ
PUSHARGE                *
ADD                     *                    *
SUB                     *                    *
MUL                     *                    *
DIV                     *                    *
MOD                     *                    *
EQ                      *                    *
NEQ                     *                    *
LEQ                     *                    *
GEQ                     *                    *
LESS                    *                    *
GREATER                 *                    *
BITAND                  *                    *
BITOR                   *                    *
BITXOR                  *                    *
BOOLAND                 *                    *
BOOLOR                  *                    *
NEG                     *                    *
BITNOT                  *
BOOLNOT                 *
JUMP
JUMPIF
JUMPNIF
STATE
CALL
RETURN                  *                    *
CAST
STACKTOS
STACKTOL
PRINT
CALLLIB
CALLLIB_TWO_BYTE        *                    *
SHL
SHR

Frames, heap and code pointers

LSL ByteCode uses a lot of pointers, both to memory (static frames, local/global, heap) as well as JUMP pointers.

Some of these pointers exist during load, for example argument types, static data, jump positions, etc. While others are allocated and created dynamically during run-time. What they all have in common is that they point to a position in the 16k LSO ByteCode file.

For memory pointers I'm planning to use a dictionary (or possibly an array), where key will be position and data the actual datastructure.

For JUMP positions I will need to scan the code first for JUMP's, then create MSIL labels in the code at these positions. This might require a two-pass run on the ByteCode.

Stack issues

In my first attempt I used the .Net stack and did a direct translation from the LSO ByteCode. I found that there are a couple of problems related to that.

First of all is the obvious problem that I stated earlier, I don't know what datatype is in the stack - but many (most) .Net commands requires me to know that. Wherever LSO can provide datatype for me it is ok, but in many cases LSO is not providing me with that datatype. For example during casting.

Next is the problem that .Net wants the stack served like "classlocation, param1, param2, call to command in classlocation". This means that I need to predict commands before PUSH'ing anything to the stack. Way to complex. I could of course juggle the stack, but not so cool.

So I've made a few changes.

First of all I'm now inheriting the compiled .Net assembly from a baseclass. In this baseclass I've added my own Stack(object), and created my own (generic) PUSH/POP/POPToStack commands. (PUSH to my stack, POP from my stack and POP from my stack to .Net's stack)

Secondly I'm creating most OPCODES as commands in C# (the base class) that is called from the converted LSO ByteCode. This way I have full control of both stack and OPCODE behaviour.

The bonus is that my stack now is in a single object that can be serialized.

LSL scripting support - Progress report

Ok, time for status update again...

Status

Code is currently able to convert from LSL ByteCode to .Net IL, compile it and save it to disk.
I've moved the code from a quick and dirty hack (that compiled the first script) to a bit more organized. It's still a mess, but that can be fixed later. The goal is simply "Version 1: Functional.".

So basically I'm able to convert simple LSL ByteCode-files into .Net .dll. If you load the script (.dll) into Visual Studio you will see the events (from server to script). If you provide the script (.dll) with an object with the righ inheritance the script (.dll) will use that objects LL-functions. Currently only llSay(); is implemented (mapped to Console.WriteLine). This is how the script communicated with the server.

I've added support for a few more OPCODES, so we support basic math and most PUSH/POP functions. Because LSL ByteCode based itself on pure memory addressing and .Net CLR/IL is object oriented a few of the functions will be tricky to implement. For example LSL wants to push a string followed by xx 0x000's to memory, then perform an operation and remove two bytes.

Server to script

For each event in the script I create one function that the server can execute. These functions needs to be late bound during load of .dll. I'm dynamically creating functions for each state+event combination. Because of the naming scheme, and the fact that a script seldonly listen on all events, I've not used any interface for this. The available events can be fetched through a GetEvents() functions that is hardcoded into the assembly when script is converted.
Also the required parameters for each function is created dynamically from the information in the LSO-file.

The naming scheme will be different once I'm done. Maybe something like a switch() inside each event to determine what code chunk should be executed based on what state number we are in.

UInt32 LSLObj.State; // Contains current state number
public void 0_event_state_entry() {} // state_entry event in state number 0
public void 1_event_state_entry() {} // state_entry event in state number 1
public void 0_event_touch_start() {}  // touch_start event instate number 0

Script to server

I've created an interface for all LSL builtin commands, and another class that inherits the interface. All the functions are there, but they are empty and don't have the correct arguments. Anyone can help to add commands by changing this class. This is probably the single remaining area that requires most work. (not sure though)

Since most PUSH/POP and math commands are supported already, we just need to map the builtin function calls to the above mentioned class. This is a simple Switch()-case hardcoding issue.


TODO:
From the top of my head, not in any particular order.

• Support custom functions (currently only LL server events gets compiled)
• Compiling them is easy, but I need to be able to bind them to Call OPCODE somehow. (IL issue)
• Figure out how to reference the LSL builtin commands C# class from IL (IL issue)
• Then bind all ll* functions
• Add support for remaining OPCODES (IL issue)
• This can be a bit tricky since LSL uses memory addressing in stack while .Net IL is object oriented. One possible solution is to implement same memory scheme as LL is using. Either way it's not a big problem, just a bit of work.
• Figure out what OPCODES needs special (custom) commands in order to support interregional transfers and user space multitasking - instead of running CLR/IL OPCODE we run a C# function that does the required work.
• Figure out how one can cast an element in CLR/IL stack without knowing its type, or - keeping track of all types in stack.
• LSL is a bit like "push string, push long, convert to string, concat, push long(channel id), llSay()". Its asking me to cast whatever is on the stack, but CLR/IL requires me to know what type I'm casting from.
• Figure out how to clear CLR/IL stack. CLR/IL requires stack to be empty on return. LSL sometimes leaves items in the stack. For now a Try:Catch is handling the issue.

The full source is here: http://www.konge.net/OpenSim/OpenSim.Region.Scripting_r1.zip.
Please note that this source belongs to OpenSim project (http://opensecondlife.org/wiki/OpenMetaverse) and is licensed thereafter.

Download, run project, choose a LSO-file (for example CloseToDefault.LSO), then process it.

Making some progress

Today I found a "final bug" in my LSL ByteCode parser, and was able to get read seemingly clean ByteCode. I also made a quick hack of merging the CLR RunTime code generator with the LSL ByteCode parser, and compiled my first small function.
It works! :)

Also writing the compiled script to a .dll-file. Necessary as the only way I know of to debug CLR is through the loveable PEVerifier.EXE.

Of course most work remains. States are not supported, dynamic function names not supported, only one OPCODE (PUSHS - push string to stack) and only one BUILTIN command (llSay mapped to Debug.WriteLine) is supported.

Not to mention user space multitasking and moving a running script between regions.

Notice the last two lines of this log...

Debug: Opening filename: LSO\CloseToDefault.lso
Debug: Reading HEADER BLOCK at: 0
Debug: TM - Top of memory (size): 16384
Debug: IP - Instruction Pointer (0=not running): 0
Debug: VN - Version number: 512
Debug: BP - Local Frame Pointer: 16383
Debug: SP - Stack Pointer: 16383
Debug: HR - Heap Register: 233
Debug: HP - Heap Pointer: 240
Debug: CS - Current State: 0
Debug: NS - Next State: 0
Debug: CE - Current Events: 0
Debug: IE - In Event: 0
Debug: ER - Event Register: 0
Debug: FR - Fault Register: 0
Debug: SLR - Sleep Register: 0
Debug: GVR - Global Variable Register: 100
Debug: GFR - Global Function Register: 100
Debug: PR - Parameter Register: 0
Debug: ESR - Energy Supply Register: 0
Debug: SR - State Register: 100
Debug: NCE - 64-bit Current Events: 1
Debug: NIE - 64-bit In Events: 0
Debug: NER - 64-bit Event Register: 5
Debug: Read position when exiting HEADER BLOCK: 100
Debug: Reading STATIC BLOCK at: 100
Debug: Number of Static Blocks read: 0
Debug: No FUNCTION BLOCK found
Debug: Reading STATE BLOCK at: 100
Debug: Reading STATE POINTER BLOCK 1 at: 104
Debug: Pointer: 116
Debug: Total potential EventMask bits: 64
Debug: Reading STATE BLOCK 1 at: 116
Debug: State block Start Pos: 116
Debug: State block Header Size: 5
Debug: State block Header End Pos: 121
Debug: Reading STATE BLOCK 1 HANDLER matching EVENT MASK 0 (state_entry) at: 121
Debug: Reading STATE BLOCK 1 HANDLER EVENT MASK 0 (state_entry) Code Chunk Pointer: 137
Debug: Reading STATE BLOCK 1 HANDLER EVENT MASK 0 (state_entry) Call Frame Size: 0
Debug: Reading STATE BLOCK 1 HANDLER matching EVENT MASK 2 (touch_start) at: 129
Debug: Reading STATE BLOCK 1 HANDLER EVENT MASK 2 (touch_start) Code Chunk Pointer: 181
Debug: Reading STATE BLOCK 1 HANDLER EVENT MASK 2 (touch_start) Call Frame Size: 4
Debug: Reading Event Code Chunk state 0, event state_entry
Debug: CLR:event_state_entry:MethodBuilder methodBuilder = typeBuilder.DefineMethod...
Debug: CLR:event_state_entry:typeBuilder.DefineMethodOverride(methodBuilder...
Debug: CLR:event_state_entry:ILGenerator il = methodBuilder.GetILGenerator();
Debug: Reading Function Code Chunk at: 137
Debug: CodeChunk Header Size: 5
Debug: Function comment:
Debug: Return type: 99
Debug: Reading Code Chunk Argument 1
Debug: Code Chunk Argument 1 return type: 94
Debug: OPCODE: NOOP
Debug: OPCODE: NOOP
Debug: OPCODE: NOOP
Debug: OPCODE: PUSHARGS
Debug: Param1: Hello, Tedd!
Debug: OPCODE: PUSHARGE
Debug: Param1: 0
Debug: OPCODE: PUSHSP
Debug: OPCODE: PUSHARGI
Debug: Param1: 8
Debug: OPCODE: ADD
Debug: Param1: 17
Debug: OPCODE: POPBP
Debug: OPCODE: CALLLIB_TWO_BYTE
Debug: Param1: 23
Debug: OPCODE: RETURN
Debug: Last OPCODE was return command. Code chunk execution complete.
Debug: CLR:event_state_entry:il.BeginCatchBlock(typeof(Exception));
Debug: CLR:event_state_entry:il.Emit(OpCodes.Ldstr...
Debug: CLR:event_state_entry:il.Emit(OpCodes.Call...
Debug: CLR:event_state_entry:il.Emit(OpCodes.Callvirt...
Debug: CLR:event_state_entry:il.Emit(OpCodes.Call...
Debug: Reading Event Code Chunk state 0, event touch_start
Debug: CLR:event_touch_start:MethodBuilder methodBuilder = typeBuilder.DefineMethod...
Debug: CLR:event_touch_start:typeBuilder.DefineMethodOverride(methodBuilder...
Debug: CLR:event_touch_start:ILGenerator il = methodBuilder.GetILGenerator();
Debug: Reading Function Code Chunk at: 181
Debug: CodeChunk Header Size: 5
Debug: Function comment:
Debug: Return type: 99
Debug: Reading Code Chunk Argument 1
Debug: Code Chunk Argument 1 return type: 94
Debug: OPCODE: NOOP
Debug: OPCODE: NOOP
Debug: OPCODE: NOOP
Debug: OPCODE: PUSHARGS
Debug: Param1: Object was touched.
Debug: OPCODE: PUSHARGE
Debug: Param1: 0
Debug: OPCODE: PUSHSP
Debug: OPCODE: PUSHARGI
Debug: Param1: 8
Debug: OPCODE: ADD
Debug: Param1: 17
Debug: OPCODE: POPBP
Debug: OPCODE: CALLLIB_TWO_BYTE
Debug: Param1: 23
Debug: OPCODE: POP
Debug: OPCODE: RETURN
Debug: Last OPCODE was return command. Code chunk execution complete.
Debug: CLR:event_touch_start:il.BeginCatchBlock(typeof(Exception));
Debug: CLR:event_touch_start:il.Emit(OpCodes.Ldstr...
Debug: CLR:event_touch_start:il.Emit(OpCodes.Call...
Debug: CLR:event_touch_start:il.Emit(OpCodes.Callvirt...
Debug: CLR:event_touch_start:il.Emit(OpCodes.Call...
Starting CLR dynamic execution of: event_state_entry
Hello, Tedd!

.Net CLR Runtime

The first thing I did was to see if .Net CLR looked anything like LSL ByteCode at all. As expected it did. It should be possible to convert between the two. But because we need to have scripts cross sims, the standard way of doing things might not suffice.

So, next question. How does one create CLR? Well, there are many ways. The ones I looked at as potential candidates was 1) creating an CLR code file and compiling and 2) creating CLR code run-time.
I chose to go with the latter one.

Here is the important part of my sample Hello World! run-time generated CLR program. The biggest Hello World! program I've ever made.

private void GenerateIL(ILGenerator il, OpenSimAPI.SimWorldAPI WorldAPI)
{
/*
 * TRY
 */
il.BeginExceptionBlock();
// Push "Hello World!" string to stack
il.Emit(OpCodes.Ldstr, "Hello World!");
Push Console.WriteLine command to stack ... Console.WriteLine("Hello World!");
il.Emit(OpCodes.Call, typeof(Console).GetMethod("WriteLine", new Type[] { typeof(string) }));

il.ThrowException(typeof(NotSupportedException));

/*
 * CATCH
 */
il.BeginCatchBlock(typeof(Exception));
// Push "Hello World!" string to stack
il.Emit(OpCodes.Ldstr, "Something went wrong: ");
//call void [mscorlib]System.Console::WriteLine(string)
il.Emit(OpCodes.Call, typeof(Console).GetMethod("Write", new Type[] { typeof(string) }));
//callvirt instance string [mscorlib]System.Exception::get_Message()
il.Emit(OpCodes.Callvirt, typeof(Exception).GetMethod("get_Message"));
//call void [mscorlib]System.Console::WriteLine(string)
il.Emit(OpCodes.Call,
typeof(Console).GetMethod("WriteLine", new Type[] { typeof(string) }));
/*
 * END TRY
 */
il.EndExceptionBlock();

// Push "Return from current method, with return value if present" to stack
il.Emit(OpCodes.Ret);

}

LSO / LSL ByteCode

The task I've taken on me is to create script support for the OpenSim server.

This involves reading the compiled LSL script (LSL ByteCode, machine code for those who haven't heard of ByteCode) and executing it somehow.

The plan is to convert LSL ByteCode into .Net ByteCode, passing it through .Net CLR (MSIL/.Net Assembly Language) and having the JIT (Just In Time) compiler to optimizations.

The result should be blazing fast execution (operating system specific native code) operating within the .Net Virtual Machine.

Security will be through .Net's AppDomain restrictions. Powerfull enough in itself so we don't have to worry about anything security-related.

LSL Builtin function implementation will be done through a C# middle layer, since it is mostly specific to LSL. This will also allow other developers to implement the functions while I continue hacking on the rest of LSL-support.

There are two tricky parts to this project.

1. Running multiple scripts simultaneous
2. Allowing scripts to cross regions while running, thus moving script to another server.

Running multiple scripts simultaneous can not be done by assigning threads to scripts, because the number of scripts will be too high. Therefore user space threading (fibres) has to be implemented. The plan is to do this by inserting breakpoints into the CLR code that allows script execution to jump out and continue execution at a later point.

Solving multitasking in that manner also makes it easier solving the problem of allowing scripts to cross regions while running, since their current execution state can be saved to file and resumed later. The trick is to make sure that stack/heap/all memory is saved in the same manner.

So, let's see how it goes... :)