Static and global variable support has been added. All push/pop functions added.
Simple scripts that use variables, math and internal functions (like llSay) are now supported.
Stack and all internal data are ready for serializing, so running script can be moved between servers - but no fibre/user space multitasking/microthreading has been implemented yet - so move has to happen between events.
Missing support for local variables, custom functions, state, if-statements, jumps, receiving/pulling status on events, etc.
So.. We can start implementation into server! :)
In first version we'll run events using a thread. Maybe adding a timeout to the thread so that it will not run too long per script.
Friday, July 27, 2007
Thursday, July 26, 2007
Current OPCODE implementation status
This is the current OPCODE implementation status:
OPCODE IL_Processor Implemented
NOOP * *
POP * *
POPS * *
POPL * *
POPV * *
POPQ * *
POPARG * *
POPIP
POPBP
POPSP
POPSLR
DUP
DUPS
DUPL
DUPV
DUPQ
STORE * *
STORES * *
STOREL * *
STOREV * *
STOREQ * *
STOREG * *
STOREGS * *
STOREGL * *
STOREGV * *
STOREGQ * *
LOADP * *
LOADSP * *
LOADLP * *
LOADVP * *
LOADQP * *
LOADGP * *
LOADGSP * *
LOADGLP * *
LOADGVP * *
LOADGQP * *
PUSH * *
PUSHS * *
PUSHL * *
PUSHV * *
PUSHQ * *
PUSHG * *
PUSHGS * *
PUSHGL * *
PUSHGV * *
PUSHGQ * *
PUSHIP
PUSHBP
PUSHSP * *
PUSHARGB
PUSHARGI * *
PUSHARGF * *
PUSHARGS * *
PUSHARGV
PUSHARGQ
PUSHE * *
PUSHEV
PUSHEQ
PUSHARGE *
ADD * *
SUB * *
MUL * *
DIV * *
MOD * *
EQ * *
NEQ * *
LEQ * *
GEQ * *
LESS * *
GREATER * *
BITAND * *
BITOR * *
BITXOR * *
BOOLAND * *
BOOLOR * *
NEG * *
BITNOT *
BOOLNOT *
JUMP
JUMPIF
JUMPNIF
STATE
CALL
RETURN * *
CAST
STACKTOS
STACKTOL
CALLLIB
CALLLIB_TWO_BYTE * *
SHL
SHR
Book: CLR via C#, Second Edition
This morning I received the book I ordered from Amazon.com. 1 week before estimated arrival.
The book is named CLR via C# (Second Edition) and is supposedly the bible of understanding CLR.
So 648 pages from now I should be able to utilize my new skills in the LSO project... :)
The book is named CLR via C# (Second Edition) and is supposedly the bible of understanding CLR.
So 648 pages from now I should be able to utilize my new skills in the LSO project... :)
Saturday, July 21, 2007
Looking ahead, fibres
Running thousands of scripts simultaneously on a grid requires some special programming. We can not just issue one thread per script and let it run. The resources required by each OS thread,both in terms of memory and thread switching is too high.
Instead we want to execute a few instructions of a script, then move to the next script. All status about execution location, memory, etc. must be kept within each script. So when the execution thread is moving between scripts it carries no payload from any of the scripts.
An important thing about computer technology is learning about the past. Because old technology is recycled all the time. In this case one could look at the "multitasking" done in Windows 3.11 to pick up a few ideas. Or Windows NT's "fibre" support, a lightweight user space implementation of threads. And that is what it all boils down to, user space threading.
The clue here is to insert code into the script that will save it's position, mark next spot with a "label" so it can jump to that position later, and exit execution. (details will follow once I get this far in the project)
There are two points where I can insert this data. Either during convert, or before load. Before load would be preferred as we then could load any .Net Assembly - great for future .Net scripting support. But the easiest way is during convert. And since this is Version 1 I will follow the easiest road.
Instead we want to execute a few instructions of a script, then move to the next script. All status about execution location, memory, etc. must be kept within each script. So when the execution thread is moving between scripts it carries no payload from any of the scripts.
An important thing about computer technology is learning about the past. Because old technology is recycled all the time. In this case one could look at the "multitasking" done in Windows 3.11 to pick up a few ideas. Or Windows NT's "fibre" support, a lightweight user space implementation of threads. And that is what it all boils down to, user space threading.
The clue here is to insert code into the script that will save it's position, mark next spot with a "label" so it can jump to that position later, and exit execution. (details will follow once I get this far in the project)
There are two points where I can insert this data. Either during convert, or before load. Before load would be preferred as we then could load any .Net Assembly - great for future .Net scripting support. But the easiest way is during convert. And since this is Version 1 I will follow the easiest road.
Frames, heap and code pointers
LSL ByteCode uses a lot of pointers, both to memory (static frames, local/global, heap) as well as JUMP pointers.
Some of these pointers exist during load, for example argument types, static data, jump positions, etc. While others are allocated and created dynamically during run-time. What they all have in common is that they point to a position in the 16k LSO ByteCode file.
For memory pointers I'm planning to use a dictionary (or possibly an array), where key will be position and data the actual datastructure.
For JUMP positions I will need to scan the code first for JUMP's, then create MSIL labels in the code at these positions. This might require a two-pass run on the ByteCode.
Some of these pointers exist during load, for example argument types, static data, jump positions, etc. While others are allocated and created dynamically during run-time. What they all have in common is that they point to a position in the 16k LSO ByteCode file.
For memory pointers I'm planning to use a dictionary (or possibly an array), where key will be position and data the actual datastructure.
For JUMP positions I will need to scan the code first for JUMP's, then create MSIL labels in the code at these positions. This might require a two-pass run on the ByteCode.
Stack issues
In my first attempt I used the .Net stack and did a direct translation from the LSO ByteCode. I found that there are a couple of problems related to that.
First of all is the obvious problem that I stated earlier, I don't know what datatype is in the stack - but many (most) .Net commands requires me to know that. Wherever LSO can provide datatype for me it is ok, but in many cases LSO is not providing me with that datatype. For example during casting.
Next is the problem that .Net wants the stack served like "classlocation, param1, param2, call to command in classlocation". This means that I need to predict commands before PUSH'ing anything to the stack. Way to complex. I could of course juggle the stack, but not so cool.
So I've made a few changes.
First of all I'm now inheriting the compiled .Net assembly from a baseclass. In this baseclass I've added my own Stack(object), and created my own (generic) PUSH/POP/POPToStack commands. (PUSH to my stack, POP from my stack and POP from my stack to .Net's stack)
Secondly I'm creating most OPCODES as commands in C# (the base class) that is called from the converted LSO ByteCode. This way I have full control of both stack and OPCODE behaviour.
The bonus is that my stack now is in a single object that can be serialized.
First of all is the obvious problem that I stated earlier, I don't know what datatype is in the stack - but many (most) .Net commands requires me to know that. Wherever LSO can provide datatype for me it is ok, but in many cases LSO is not providing me with that datatype. For example during casting.
Next is the problem that .Net wants the stack served like "classlocation, param1, param2, call to command in classlocation". This means that I need to predict commands before PUSH'ing anything to the stack. Way to complex. I could of course juggle the stack, but not so cool.
So I've made a few changes.
First of all I'm now inheriting the compiled .Net assembly from a baseclass. In this baseclass I've added my own Stack(object), and created my own (generic) PUSH/POP/POPToStack commands. (PUSH to my stack, POP from my stack and POP from my stack to .Net's stack)
Secondly I'm creating most OPCODES as commands in C# (the base class) that is called from the converted LSO ByteCode. This way I have full control of both stack and OPCODE behaviour.
The bonus is that my stack now is in a single object that can be serialized.
Saturday, July 14, 2007
LSL scripting support - Progress report
Ok, time for status update again...
Status
Code is currently able to convert from LSL ByteCode to .Net IL, compile it and save it to disk.
I've moved the code from a quick and dirty hack (that compiled the first script) to a bit more organized. It's still a mess, but that can be fixed later. The goal is simply "Version 1: Functional.".
So basically I'm able to convert simple LSL ByteCode-files into .Net .dll. If you load the script (.dll) into Visual Studio you will see the events (from server to script). If you provide the script (.dll) with an object with the righ inheritance the script (.dll) will use that objects LL-functions. Currently only llSay(); is implemented (mapped to Console.WriteLine). This is how the script communicated with the server.
I've added support for a few more OPCODES, so we support basic math and most PUSH/POP functions. Because LSL ByteCode based itself on pure memory addressing and .Net CLR/IL is object oriented a few of the functions will be tricky to implement. For example LSL wants to push a string followed by xx 0x000's to memory, then perform an operation and remove two bytes.
Server to script
For each event in the script I create one function that the server can execute. These functions needs to be late bound during load of .dll. I'm dynamically creating functions for each state+event combination. Because of the naming scheme, and the fact that a script seldonly listen on all events, I've not used any interface for this. The available events can be fetched through a GetEvents() functions that is hardcoded into the assembly when script is converted.
Also the required parameters for each function is created dynamically from the information in the LSO-file.
The naming scheme will be different once I'm done. Maybe something like a switch() inside each event to determine what code chunk should be executed based on what state number we are in.
Script to server
I've created an interface for all LSL builtin commands, and another class that inherits the interface. All the functions are there, but they are empty and don't have the correct arguments. Anyone can help to add commands by changing this class. This is probably the single remaining area that requires most work. (not sure though)
Since most PUSH/POP and math commands are supported already, we just need to map the builtin function calls to the above mentioned class. This is a simple Switch()-case hardcoding issue.
TODO:
From the top of my head, not in any particular order.
The full source is here: http://www.konge.net/OpenSim/OpenSim.Region.Scripting_r1.zip.
Please note that this source belongs to OpenSim project (http://opensecondlife.org/wiki/OpenMetaverse) and is licensed thereafter.
Download, run project, choose a LSO-file (for example CloseToDefault.LSO), then process it.
Status
Code is currently able to convert from LSL ByteCode to .Net IL, compile it and save it to disk.
I've moved the code from a quick and dirty hack (that compiled the first script) to a bit more organized. It's still a mess, but that can be fixed later. The goal is simply "Version 1: Functional.".
So basically I'm able to convert simple LSL ByteCode-files into .Net .dll. If you load the script (.dll) into Visual Studio you will see the events (from server to script). If you provide the script (.dll) with an object with the righ inheritance the script (.dll) will use that objects LL-functions. Currently only llSay(); is implemented (mapped to Console.WriteLine). This is how the script communicated with the server.
I've added support for a few more OPCODES, so we support basic math and most PUSH/POP functions. Because LSL ByteCode based itself on pure memory addressing and .Net CLR/IL is object oriented a few of the functions will be tricky to implement. For example LSL wants to push a string followed by xx 0x000's to memory, then perform an operation and remove two bytes.
Server to script
For each event in the script I create one function that the server can execute. These functions needs to be late bound during load of .dll. I'm dynamically creating functions for each state+event combination. Because of the naming scheme, and the fact that a script seldonly listen on all events, I've not used any interface for this. The available events can be fetched through a GetEvents() functions that is hardcoded into the assembly when script is converted.
Also the required parameters for each function is created dynamically from the information in the LSO-file.
The naming scheme will be different once I'm done. Maybe something like a switch() inside each event to determine what code chunk should be executed based on what state number we are in.
UInt32 LSLObj.State; // Contains current state number
public void 0_event_state_entry() {} // state_entry event in state number 0
public void 1_event_state_entry() {} // state_entry event in state number 1
public void 0_event_touch_start() {} // touch_start event instate number 0
Script to server
I've created an interface for all LSL builtin commands, and another class that inherits the interface. All the functions are there, but they are empty and don't have the correct arguments. Anyone can help to add commands by changing this class. This is probably the single remaining area that requires most work. (not sure though)
Since most PUSH/POP and math commands are supported already, we just need to map the builtin function calls to the above mentioned class. This is a simple Switch()-case hardcoding issue.
TODO:
From the top of my head, not in any particular order.
- Support custom functions (currently only LL server events gets compiled)
- Compiling them is easy, but I need to be able to bind them to Call OPCODE somehow. (IL issue)
- Figure out how to reference the LSL builtin commands C# class from IL (IL issue)
- Then bind all ll* functions
- Add support for remaining OPCODES (IL issue)
- This can be a bit tricky since LSL uses memory addressing in stack while .Net IL is object oriented. One possible solution is to implement same memory scheme as LL is using. Either way it's not a big problem, just a bit of work.
- Figure out what OPCODES needs special (custom) commands in order to support interregional transfers and user space multitasking - instead of running CLR/IL OPCODE we run a C# function that does the required work.
- Figure out how one can cast an element in CLR/IL stack without knowing its type, or - keeping track of all types in stack.
- LSL is a bit like "push string, push long, convert to string, concat, push long(channel id), llSay()". Its asking me to cast whatever is on the stack, but CLR/IL requires me to know what type I'm casting from.
- Figure out how to clear CLR/IL stack. CLR/IL requires stack to be empty on return. LSL sometimes leaves items in the stack. For now a Try:Catch is handling the issue.
The full source is here: http://www.konge.net/OpenSim/OpenSim.Region.Scripting_r1.zip.
Please note that this source belongs to OpenSim project (http://opensecondlife.org/wiki/OpenMetaverse) and is licensed thereafter.
Download, run project, choose a LSO-file (for example CloseToDefault.LSO), then process it.
Thursday, July 12, 2007
Making some progress
Today I found a "final bug" in my LSL ByteCode parser, and was able to get read seemingly clean ByteCode. I also made a quick hack of merging the CLR RunTime code generator with the LSL ByteCode parser, and compiled my first small function.
It works! :)
Also writing the compiled script to a .dll-file. Necessary as the only way I know of to debug CLR is through the loveable PEVerifier.EXE.
Of course most work remains. States are not supported, dynamic function names not supported, only one OPCODE (PUSHS - push string to stack) and only one BUILTIN command (llSay mapped to Debug.WriteLine) is supported.
Not to mention user space multitasking and moving a running script between regions.
Notice the last two lines of this log...
It works! :)
Also writing the compiled script to a .dll-file. Necessary as the only way I know of to debug CLR is through the loveable PEVerifier.EXE.
Of course most work remains. States are not supported, dynamic function names not supported, only one OPCODE (PUSHS - push string to stack) and only one BUILTIN command (llSay mapped to Debug.WriteLine) is supported.
Not to mention user space multitasking and moving a running script between regions.
Notice the last two lines of this log...
Debug: Opening filename: LSO\CloseToDefault.lso
Debug: Reading HEADER BLOCK at: 0
Debug: TM - Top of memory (size): 16384
Debug: IP - Instruction Pointer (0=not running): 0
Debug: VN - Version number: 512
Debug: BP - Local Frame Pointer: 16383
Debug: SP - Stack Pointer: 16383
Debug: HR - Heap Register: 233
Debug: HP - Heap Pointer: 240
Debug: CS - Current State: 0
Debug: NS - Next State: 0
Debug: CE - Current Events: 0
Debug: IE - In Event: 0
Debug: ER - Event Register: 0
Debug: FR - Fault Register: 0
Debug: SLR - Sleep Register: 0
Debug: GVR - Global Variable Register: 100
Debug: GFR - Global Function Register: 100
Debug: PR - Parameter Register: 0
Debug: ESR - Energy Supply Register: 0
Debug: SR - State Register: 100
Debug: NCE - 64-bit Current Events: 1
Debug: NIE - 64-bit In Events: 0
Debug: NER - 64-bit Event Register: 5
Debug: Read position when exiting HEADER BLOCK: 100
Debug: Reading STATIC BLOCK at: 100
Debug: Number of Static Blocks read: 0
Debug: No FUNCTION BLOCK found
Debug: Reading STATE BLOCK at: 100
Debug: Reading STATE POINTER BLOCK 1 at: 104
Debug: Pointer: 116
Debug: Total potential EventMask bits: 64
Debug: Reading STATE BLOCK 1 at: 116
Debug: State block Start Pos: 116
Debug: State block Header Size: 5
Debug: State block Header End Pos: 121
Debug: Reading STATE BLOCK 1 HANDLER matching EVENT MASK 0 (state_entry) at: 121
Debug: Reading STATE BLOCK 1 HANDLER EVENT MASK 0 (state_entry) Code Chunk Pointer: 137
Debug: Reading STATE BLOCK 1 HANDLER EVENT MASK 0 (state_entry) Call Frame Size: 0
Debug: Reading STATE BLOCK 1 HANDLER matching EVENT MASK 2 (touch_start) at: 129
Debug: Reading STATE BLOCK 1 HANDLER EVENT MASK 2 (touch_start) Code Chunk Pointer: 181
Debug: Reading STATE BLOCK 1 HANDLER EVENT MASK 2 (touch_start) Call Frame Size: 4
Debug: Reading Event Code Chunk state 0, event state_entry
Debug: CLR:event_state_entry:MethodBuilder methodBuilder = typeBuilder.DefineMethod...
Debug: CLR:event_state_entry:typeBuilder.DefineMethodOverride(methodBuilder...
Debug: CLR:event_state_entry:ILGenerator il = methodBuilder.GetILGenerator();
Debug: Reading Function Code Chunk at: 137
Debug: CodeChunk Header Size: 5
Debug: Function comment:
Debug: Return type: 99
Debug: Reading Code Chunk Argument 1
Debug: Code Chunk Argument 1 return type: 94
Debug: OPCODE: NOOP
Debug: OPCODE: NOOP
Debug: OPCODE: NOOP
Debug: OPCODE: PUSHARGS
Debug: Param1: Hello, Tedd!
Debug: OPCODE: PUSHARGE
Debug: Param1: 0
Debug: OPCODE: PUSHSP
Debug: OPCODE: PUSHARGI
Debug: Param1: 8
Debug: OPCODE: ADD
Debug: Param1: 17
Debug: OPCODE: POPBP
Debug: OPCODE: CALLLIB_TWO_BYTE
Debug: Param1: 23
Debug: OPCODE: RETURN
Debug: Last OPCODE was return command. Code chunk execution complete.
Debug: CLR:event_state_entry:il.BeginCatchBlock(typeof(Exception));
Debug: CLR:event_state_entry:il.Emit(OpCodes.Ldstr...
Debug: CLR:event_state_entry:il.Emit(OpCodes.Call...
Debug: CLR:event_state_entry:il.Emit(OpCodes.Callvirt...
Debug: CLR:event_state_entry:il.Emit(OpCodes.Call...
Debug: Reading Event Code Chunk state 0, event touch_start
Debug: CLR:event_touch_start:MethodBuilder methodBuilder = typeBuilder.DefineMethod...
Debug: CLR:event_touch_start:typeBuilder.DefineMethodOverride(methodBuilder...
Debug: CLR:event_touch_start:ILGenerator il = methodBuilder.GetILGenerator();
Debug: Reading Function Code Chunk at: 181
Debug: CodeChunk Header Size: 5
Debug: Function comment:
Debug: Return type: 99
Debug: Reading Code Chunk Argument 1
Debug: Code Chunk Argument 1 return type: 94
Debug: OPCODE: NOOP
Debug: OPCODE: NOOP
Debug: OPCODE: NOOP
Debug: OPCODE: PUSHARGS
Debug: Param1: Object was touched.
Debug: OPCODE: PUSHARGE
Debug: Param1: 0
Debug: OPCODE: PUSHSP
Debug: OPCODE: PUSHARGI
Debug: Param1: 8
Debug: OPCODE: ADD
Debug: Param1: 17
Debug: OPCODE: POPBP
Debug: OPCODE: CALLLIB_TWO_BYTE
Debug: Param1: 23
Debug: OPCODE: POP
Debug: OPCODE: RETURN
Debug: Last OPCODE was return command. Code chunk execution complete.
Debug: CLR:event_touch_start:il.BeginCatchBlock(typeof(Exception));
Debug: CLR:event_touch_start:il.Emit(OpCodes.Ldstr...
Debug: CLR:event_touch_start:il.Emit(OpCodes.Call...
Debug: CLR:event_touch_start:il.Emit(OpCodes.Callvirt...
Debug: CLR:event_touch_start:il.Emit(OpCodes.Call...
Starting CLR dynamic execution of: event_state_entry
Hello, Tedd!
.Net CLR Runtime
The first thing I did was to see if .Net CLR looked anything like LSL ByteCode at all. As expected it did. It should be possible to convert between the two. But because we need to have scripts cross sims, the standard way of doing things might not suffice.
So, next question. How does one create CLR? Well, there are many ways. The ones I looked at as potential candidates was 1) creating an CLR code file and compiling and 2) creating CLR code run-time.
I chose to go with the latter one.
Here is the important part of my sample Hello World! run-time generated CLR program. The biggest Hello World! program I've ever made.
So, next question. How does one create CLR? Well, there are many ways. The ones I looked at as potential candidates was 1) creating an CLR code file and compiling and 2) creating CLR code run-time.
I chose to go with the latter one.
Here is the important part of my sample Hello World! run-time generated CLR program. The biggest Hello World! program I've ever made.
private void GenerateIL(ILGenerator il, OpenSimAPI.SimWorldAPI WorldAPI)
{
/*
* TRY
*/
il.BeginExceptionBlock();
// Push "Hello World!" string to stack
il.Emit(OpCodes.Ldstr, "Hello World!");
Push Console.WriteLine command to stack ... Console.WriteLine("Hello World!");
il.Emit(OpCodes.Call, typeof(Console).GetMethod("WriteLine", new Type[] { typeof(string) }));
il.ThrowException(typeof(NotSupportedException));
/*
* CATCH
*/
il.BeginCatchBlock(typeof(Exception));
// Push "Hello World!" string to stack
il.Emit(OpCodes.Ldstr, "Something went wrong: ");
//call void [mscorlib]System.Console::WriteLine(string)
il.Emit(OpCodes.Call, typeof(Console).GetMethod("Write", new Type[] { typeof(string) }));
//callvirt instance string [mscorlib]System.Exception::get_Message()
il.Emit(OpCodes.Callvirt, typeof(Exception).GetMethod("get_Message"));
//call void [mscorlib]System.Console::WriteLine(string)
il.Emit(OpCodes.Call,
typeof(Console).GetMethod("WriteLine", new Type[] { typeof(string) }));
/*
* END TRY
*/
il.EndExceptionBlock();
// Push "Return from current method, with return value if present" to stack
il.Emit(OpCodes.Ret);
}
LSO / LSL ByteCode
The task I've taken on me is to create script support for the OpenSim server.
This involves reading the compiled LSL script (LSL ByteCode, machine code for those who haven't heard of ByteCode) and executing it somehow.
The plan is to convert LSL ByteCode into .Net ByteCode, passing it through .Net CLR (MSIL/.Net Assembly Language) and having the JIT (Just In Time) compiler to optimizations.The result should be blazing fast execution (operating system specific native code) operating within the .Net Virtual Machine.
Security will be through .Net's AppDomain restrictions. Powerfull enough in itself so we don't have to worry about anything security-related.
LSL Builtin function implementation will be done through a C# middle layer, since it is mostly specific to LSL. This will also allow other developers to implement the functions while I continue hacking on the rest of LSL-support.
There are two tricky parts to this project.
- Running multiple scripts simultaneous
- Allowing scripts to cross regions while running, thus moving script to another server.
Running multiple scripts simultaneous can not be done by assigning threads to scripts, because the number of scripts will be too high. Therefore user space threading (fibres) has to be implemented. The plan is to do this by inserting breakpoints into the CLR code that allows script execution to jump out and continue execution at a later point.
Solving multitasking in that manner also makes it easier solving the problem of allowing scripts to cross regions while running, since their current execution state can be saved to file and resumed later. The trick is to make sure that stack/heap/all memory is saved in the same manner.
So, let's see how it goes... :)
Subscribe to:
Posts (Atom)