Setting Up A Sandbox Shard

How to create a SANDBOX SHARD

by Phoenix Rising version 0.01 pre Alpha - July 2011

Shorah Explorers

The aim of this document is to detail as closely as possible the exact steps to create your own Sandbox Shard using GoW's CWE Client and Zrax's Dirtsand Server specifically for writers and age builders who want to test MOULa compatible ages. As a basis for this document we will use Zrax's README at https://github.com/H-uru/dirtsand but with a lot more detailed editing based on first hand experience setting up the Sandbox Shard. We have permission from Zrax to use his README in this wiki but with a much expanded and detailed version which we have added. We have chosen to use GoW's CWE Client since it has been heavily updated so that it may be compiled easily and with a much more recent version of DirectX SDK and other tweaks such as reduction of lag by Tsar Hoikas. This simply means that it has better support and hopefully better gameplay on your fancy new computer!

IMPORTANT NOTE: This is a work in progress and gets highly technical, and as such is not recommended for newbies. Some changes may happen along the way. If you want to add to this wiki, please be nice! We want this to appeal to everyone young and old and have a really cool wiki to show for it. If you have helpful suggestions feel free to add them, but kindly refrain from deleting any sections. A lot of hard work has gone into documenting this procedure for the sake of all the Uru fans out there. At the end of the day all this effort is due to teamwork, since noone can do such a mammoth task alone!

HARDWARE PREREQUISITES: The aim of this wiki is to teach a writer/builder/developer to test CWE compatible ages. For that at least two computers are needed (one of which must have an able graphics card).

As a benchmark we will be building the sandbox based on:

CLIENT MACHINE: A Dell Inspiron running Windows 7 64 bit, with a 1Gb Ati Graphics card, and
SERVER MACHINE: An Asus Eeepc mini laptop running Ubuntu 10.10 (Maverick).

The specs are not crucial, except for the graphics card on the Windows Machine and the Flavour of Linux on the SERVER. We have chosen Ubuntu 10.10 because although we still have to configure stuff, we dont like configuration nightmares!

A simple test to see if your Windows CLIENT can handle all of CWE's graphics requirements is to ramp up all your Graphics settings to maximum when you run the launcher for TPOTS and see if your computer runs smoothly or has screen jitter/stutter. If your machine can handle these high settings you will be less restricted in your age building later on.

NETWORK CONNECTION: This wiki will attempt to document one of the two approaches to network connectivity.

SANDBOX MODE - SERVER and CLIENT on a local hub using DHCP or fixed IP. (No internet)
SHARD MODE - SERVER on dynamic DNS or fixed IP, and CLIENT accessing through the internet.

SHARD MODE will be covered in another wiki at a later date.

Both configurations have their merits. The first provides the "offline" sandbox mode for creating your own ages and testing the content, even though you will still be connecting via a network of some kind. The second makes your shard accessible to other players over the internet. This you will use when you are ready to go live!

TERMS and ABBREVIATIONS:

Cyan has come a long way since the release of Myst. Their original 2D "point and click" game eventually evolved into a fully three dimensional gameplay with unrestricted movement and engaging avatar animations. Although Myst was the forefather of MOUL, in our wiki we wont be covering the MYST engine or how to set it up. It is too outdated. If you really want to play Myst you can use your original Myst CD content, Drizzle and TPOTS.

We need to draw a line as to what versions of Cyan's engine we use, and what purpose each version serves. Below is a list of the commonly accepted Versions.

Only one version will be used to setup our SANDBOX SHARD. The CWE Client using the MOULa content!

Choru - Cyan's Closed Beta.
Ubiru - Ubisoft's Closed Beta.
ABM - Ages Beyond Myst - The vanilla single player version of Uru.
Prologue - The original MMO version of Uru. (Essentially an Open Beta)
UruUpdate12 - Update package to fix bug with ATI Graphics cards.
DNI - To D'ni Expansion Pack - an add-on extension that adds some of the Prologue Ages along with a recap of the Prologue story.
TPOTS - The Path of the Shell Expansion Pack - adds even more content to ABM.
CC - Complete Chronicles - a compilation of the first 4 packages as a single distribution.
UU - Untìl Uru - A no longer officially supported online version of ABM.
D'mala - The Cyan Worlds Until Uru Shard.
MOUL - Myst Online: Uru Live - The Third Online Incarnation of Uru
MOULagain / MOULa - Myst Online: Uru Live Again - Resurrected version of MOUL

and finally the Open Source Client that has been given to the community by Cyan Worlds, namely:

CWE - CyanWorlds.com Engine - the recently released Open Source client based on MOUL - Note: not a full source release.

If you want to start building ages, we do not recommend following this wiki beyond the following few lines. There is a fine line between a creative mind and a technical mind (and very few people are both), and if you are interested in creating ages, this wiki will not help you. There are many tutorials on the Guild of Writers to help you build your first age.

ASSUMPTIONS TO FOLLOW THIS WIKI

You will be compiling the CWE client on a WinXp 32 Bit machine.
You know how to use Windows command line
You know how to use a Linux command line
You know how to compile C++ software on Windows
You know how to compile C++ software on Linux
You are fluent using both systems and understand the subtleties of each, such as: case sensitivity, backslash versus forward slash, and what a terminal is.
You know how to use scalc or excel

If you fail to do any of the above, it's time for you to go back to the drawing board and do some study and practice before moving forward. You can find some useful tutorials below at:

http://guildofwriters.com/

Ok, let's begin:

As this wiki unfolds you will notice that we will have to do work on both the SERVER and the CLIENT at any one time, so be prepared for it before it happens. We will refer to the client as Windows, and the server as Ubuntu for ease of reference.

Github and the branches

There are 3 main projects that we will pull from github.

plasma
dirtsand
moul-scripts

The 4th one is the set of tools that you can build on your linux box. We prefer to use the Plasma_Win32 Tools, so compiling the tools for linux won't be covered here.

Now it is important to wrap your brain around the idea of a branch, and which one to choose on github. The important thing to remember is that you will want the branches that made for the CWE/dirtsand engine, so to keep you focussed on what you looking for, here is a quick list; (Note, we won't be covering how to build a Cyan compatible engine, since Cyans MOUL still runs on Python 2.3 and CWE worls on Python 2.7)

In github look for the branches tab and select each for the following projects;

plasma - master branch
dirtsand - master branch
moul-scripts - python27 branch

An easy way to get the code is to click the "Downloads" button to the far right of "branches" tab.

On Windows - Let's compile CWE

Our starting point is to build the CWE client based on the "master" branch on github since it is a relatively simple process and will help you get your feet wet.

To build the client you can follow the GoW tutorial and video clip below:

There is no need to rewrite such a detailed and well written tutorial. The guys who created this deserve a big thank you from all the uru fans out there.

Once you are done, you will have 3 files in the

.\Plasma\build\Sources\Plasma\Apps\

directory, which you will use later.

.\plClient\debug\plClient.exe
.\plUruLauncher\debug\plUruLauncher.exe
.\plPythonPack\Debug\plPythonPack.exe

How you choose to layout your directory structure for CWE development is up to you, but if you want to keep a semblance of order, we suggest the following structure that you will want to create on your Windows hard drive to avoid a panic and tedious searches for things.

On Windows make a folder called "CWE Sandbox" (or whatever blows your hair back)

Inside this folder create the following folders:

"CWE Repository"  ( this is where you store source code, dependancies, etc. )
"CWE Tools"       ( This is where you save plasmashop, prp_explorer, prptool, plPlasma, etc. )
"INI Files"       ( a directory to store the dirtsand.ini, server.ini and zrax.ini files, and your modified versions of those files. )
"Installers"      ( a place to store your original MOUL installation files, etc )
"plEngine"        ( A copy of the compiled executables plClient.exe, plUruLauncher.exe and plPythonPack.exe )
"Tutorials"       ( Your favourite tutorials to use as a reference )

A few extra folders that are very useful are:

.\CWE Sandbox\CWE Repository\CWE Dependancy\ ( save all your Compiler dependencies that you downloaded here
 in executable or source code form... cmake, git, DxSDK, openal, devlibs, PhysX, VS2008-SP1, etc )
.\CWE Sandbox\CWE Repository\Plasma\ ( the folder where you download and save the CWE source code to )

This structure is not set in stone. If you have your own structure simply move on.

We also recommend you create a "Games" folder outside of the "CWE Sandbox" folder. You will have a few versions of one game, because sometimes things can go wrong and you need to start with a fresh copy, or one version when patched only works with a specific shard.

Just remember that our focus is using the CWE client with Dirtsand server! All other shards do not apply to this wiki.

A good way to manage all these versions is to install ABM-TPOTS and MOULa to your Default Windows Drive and to NEVER USE IT (Just make sure you unplug your network cable or block your firewall on completion of the MOULa installation, because we dont want it to automatically install Cyans updates when it finishes installing). Then when we need it we COPY the version we want to the GAMES folder, then rename it to something unique, as below...

.\Games\
.\Games\Cyan Worlds\Uru - Ages Beyond Myst - Deep Island        ( patched ABM-TPOTS version )
.\Games\Cyan Worlds\Uru - Ages Beyond Myst - TPOTS              ( default ABM-TPOTS expansion/CC version )
.\Games\Cyan Worlds\Uru Live                                    ( MOULa version )
.\Games\Cyan Worlds\Uru Live - Dirt Sand                        ( MOULa version with Zraxs server.ini file )
.\Games\Cyan Worlds\Uru Live - Sandbox                          ( your MOULa Sandbox with plEngine files )

Get used to the fact that when you copy an ABM-TPOTS or MOUL install you may have to go and make a cup of coffee because you are looking at around 2,5 to 4,5 gigs per copy, and that is not a fast operation even on a dual or quad core computer.

It is good to install the ABM-TPOTS version so that if any problems come up with MOUL/CWE version, you can run ABM-TPOTS to see if your hardware is still okay.

Now, time to delve into the SERVER. The next portion is Zrax's heavily edited README for his Dirtsand server. It is assumed that you have Ubuntu 10.10 installed, and that you wrote down the superuser password somewhere safe! Phoenix says: "Yip, I had to reinstall Ubuntu once because I lost the root accounts password, so yes, it can happen to you too!" It is also assumed that you can get to the Ubuntu desktop.

On Ubuntu - DirtSand Howto

:::::::::::::::::::::::::::SERVER SECTION - Compliments of Zrax - heavily edited by Phoenix

Originally: DIRTSAND - The D'ni In Real-Time Server And Network Dæmon by Zrax

Introduction

DIRTSAND is a full featured MOULa-compatible server platform for POSIX-compliant operating systems, written in C++ and released under the GPL version 3+. Currently, it has only been tested on Linux, but in theory it should work on other Unixes as well. There are, however, currently no plans for Windows development or support.

We are going to attempt to use a moderately spec'd eeepc laptop for the SANDBOX SHARD since we only expect 1 or 2 users to be logged in at one time, for the sake of "offline" testing.

Specs of EeePC: 1 GiB ram, 1.5 GHz CPU

For ease of implementation we will be assuming that the server will be running on Ubuntu 10.10 (Maverick) Desktop version. This has proven to be the less painful way to delve into SERVER setup.

Building the code

Prerequisites or dependancies are libraries or programs that another program needs to run or to compile properly. Without them you will get many errors. Do not proceed with this section of the wiki if you have not satified the list below.

A quick way to find out what is installed on your Ubuntu box is to click "Applications" and launch "Ubuntu Software Center". Type in a part of the name of the prerequisite and the Software Center will show you what is installed on your Ubuntu machine.

Prerequisites:

GCC 4.4+ (might work with other C++0x compliant compilers, but untested)
Postgres (libpq) -- the actual database server can reside anywhere
OpenSSL
libreadline
zlib
git (to get the sources)

The default install of Ubuntu does not install all the items above. To install the packages that are lacking, open a terminal window, type the commands below (one at a time) and follow the prompts before moving on to the next item in the list. This should satisfy the list of prerequisites above.

For those of you with Ubuntu 10.10, this is the easy part:

$ sudo apt-get install postgresql postgresql-contrib-8.4 libpq-dev apache2 cmake cmake-qt-gui g++ libreadline-dev

1) Now, create a user and path for the DIRTSAND server to run from.

$ sudo useradd dirtsand -d /opt/dirtsand -m -p <password> -s /bin/bash

Remember to drop the brackets <> when you create a password and watch your case (UPPERCASE or lowercase)!

CAUTION: If you simply copy and paste Linux commands from this wiki, sometimes the terminal will execute the command automatically without you pressing enter. For safety, copy the command to a text editor, change the command as needed, then copy to the terminal window. Otherwise you may have to uninstall everything and start again!

2) Once done, check out a copy of the "dirtsand" source code: (Ubuntu defaults to your HOME folder when you run the terminal, so your destination folder 'dirtsand' will be inside your HOME folder. Double check that you choose the right branch on github before you run the following command...

$ git clone git://github.com/H-uru/dirtsand.git dirtsand

NOTE: On github the branch you want to checkout is named "master". By default the command above will checkout this branch.

this will take a while, so maybe its time for a cup of coffee.

$ cd dirtsand

3) Compile it for your system...

$ mkdir build && cd build
$ cmake -DCMAKE_INSTALL_PREFIX=/opt/dirtsand ..
$ make && sudo make install
$ cd ..

If you run into any errors about finding libraries or headers, make sure you have the *development* versions of all of the required libraries, and that they are in your path. You can also use the "cmake-gui" to help 'cmake' locate the missing paths and files.

$ cmake-gui

You have it already when you ran apt-get earlier on.

Setting up a server - WITH a superuser account

You will need a working PostgreSQL server which DIRTSAND can use to store its data. Although you don't need superuser access to the postgres server, you will need to have a database which you can add schemas, tables, etc into.

SIDE NOTE: Setting up postgres functionality without a superuser account is currently beyond the scope of this document... If you need to do this, keep in mind that you may also have to edit the database initialization scripts to work against an existing user and/or database.

Moving on, for the default installation, the provided scripts will create a dirtsand database and set its ownership to a 'dirtsand' database user, which can directly map the system 'dirtsand' user created in the "Building the code" instructions above. For better security, it is recommended to use a password (as shown in the steps below), which can be configured in the server settings as described in the "configure dirtsand" step.

1. Set up the postgres user

For those new to Linux, remember to drop the brackets <> around the password.

$ sudo -u postgres psql -d template1
template1=# CREATE USER dirtsand WITH PASSWORD '<password>';
template1=# CREATE DATABASE dirtsand WITH TEMPLATE = template0 ENCODING = 'UTF8';
template1=# ALTER DATABASE dirtsand OWNER TO dirtsand;
template1=# \q

2. Install the UUID functionality

This may be provided by your OS distribution. In Ubuntu, simply install the postgresql-contrib-8.4 package (Check the apt-get list at the start of the SERVER SECTION) to provide the necessary libraries and installation scripts. If your distribution does not provide a contrib or uuid-ossp bundle, you can get it and build it yourself from the sources provided at: http://www.ossp.org/pkg/lib/uuid/

Once you have the ossp, you can add it to the dirtsand database by running the command:

$ sudo -u postgres psql -d dirtsand < /usr/share/postgresql/8.4/contrib/uuid-ossp.sql

3. Set up the dirtsand database

$ sudo -u postgres psql -d dirtsand < db/dbinit.sql
$ sudo -u postgres psql -d dirtsand < db/functions.sql

NOTE: If you see the message 'ERROR: language "plpgsql" does not exist' while importing functions.sql, you will also need to create the plpgsql language in the database:

$ sudo -u postgres psql -d dirtsand
dirtsand=# CREATE LANGUAGE plpgsql;
dirtsand=# \q

After this, you should re-import the functions.sql file to properly create the necessary dirtsand functions by running...

$ sudo -u postgres psql -d dirtsand < db/functions.sql

If there were no other errors, your database should be ready for DIRTSAND

4. Configure dirtsand

A sample dirtsand.ini has been provided in the root of the dirtsand sources. You will need to copy this to your install directory and then edit the fields you need for the server. Specifically, you will need to adjust the server addresses and the RC4 keys. If you have dirtsand installed to somewhere other than /opt/dirtsand, you will also need to point the configuration to the right paths too.

$ sudo cp dirtsand.sample.ini /opt/dirtsand/dirtsand.ini
$ sudo chown dirtsand /opt/dirtsand/dirtsand.ini
$ su - dirtsand
$ <your-favorite-editor> dirtsand.ini

If you use gedit as <your-favorite-editor> you may find it won't run in the dirtsand account, so you will need to use sudo.

To generate the RC4 keys, you can simply run the keygen command from within the dirtsand interactive console:

$ bin/dirtsand
ds-902> keygen new       (This one will take a little while)
ds-902> quit

Any errors that dirtsand spits out about config files and postgres passwords can be ignored, since you haven't provided a configuration file yet.

You should now have a bunch of keys output on your terminal.

In your terminal, go to Edit, Select All, and Copy. Then paste the output to a file and save it as "Dirtsand.keys".

NOTE: You will need to copy this file to your Windows box later as well as the "dirtsand.ini" file.

The first block (labeled Server keys) in "Dirtsand.keys" is the set you should paste into your "dirtsand.ini" in the directory /opt/dirtsand/. Replace the dummy lines (with the '...' values) with the Server section in your "Dirtsand.keys" file.

FOR LATER on Windows: The second set of keys will be placed directly in the client's "server.ini" file.

NOTE: This requires either the H-uru fork of CWE or PlasmaClient.

Caution -- the vanilla MOUL client cannot load keys from a file, and you will have to enter the keys as byte arrays directly into the executable. Later we will explain how to add the Client keys on the Windows box.

Now before we move on, you need to download the moulscripts from github to Ubuntu. The least painful way to do this is to browse to;

https://github.com/H-uru/moul-scripts

...change the branch to "python27" and click "Downloads" to the far right of the "branches" tab.

Choose "*.tar.gz" and save it to your "HOME" folder on your ubuntu box.

NOTE: For those who like the git command line, you know what to do.

When you've saved the "Python27" branch of moulscripts, you will have to unpack it. You will see 3 folders in "Python27" once you have extracted the archive:

./dirtsand/dat
./dirtsand/Python
./dirtsand/SDL

One of these needs to be renamed to fall in line with the naming convention in the dirtsand.ini file.

Rename "./Python27/dat" to "./Python27/ages" (This is to save you a couple of head scratches later on) like so:

$ mv dat ages

The files inside these folders are all the age, Python and SDL information dirtsand needs for the ages in UN-encrypted format.

Leave them unencrypted for now becuase you will see later you need two copies of these files in unencrypted and encrypted format! More on that later...

CAUTIONARY NOTE: Since Plasma, dirtsand, moul-scripts and lihPlasma are in HEAVY alpha, you might find things don't compile or throw out lots of errors. First check that you are compiling on a 32 bit WinXp system. Win7 64 bit is a little rebellious when it comes to compatibility.

On Windows - Authdata, Manifests and Data

5) Now, move over to your Windows box.

Firstly, you should copy the Dirtsand.keys and dirtsand.ini file over to your Windows box now, and place them in ".\CWE Sandbox\INI Files\" for easy reference.

You need to create some server-side data such as manifests, encrypted python.pak and the content (data) of MOUL, but make special note that although this is created on Windows, you will copy the newly created files over to your UBUNTU box later!

SIDE NOTE: If you want to do this on Ubuntu, feel free to clone the source code for libhsPlasma tools at https://github.com/H-uru/libhsplasma We wont be covering how to do that here, since the Windows version of libhsPlasma comes with some nicely pre-compiled binaries which you can download here...

http://libhsplasma.googlecode.com/files/Plasma_Win32_r801.zip

EXTRA SIDE NOTE: The Win32 version of libhsPlasma is used even on 64 bit machines to avoid any potential errors creeping in.

For this you will be using plPythonPack.exe that was compiled earlier on and copied to your "plEngine" folder and PlasmaCrypt.exe in your "Plasma_Win32" (libhsplasma) download.

CAUTION: Phoenix says; I originally compiled the client on Win7 64bit. There were some major errors I could not solve. As soon as I compiled a fresh build on my WinXP 32 bit system, everything fell into place.

Okay lets get going... What we found is this; dirtsand needs two versions of the "ages" and "SDL" files. One version is UN-encrypted and the other is encrypted, but each goes in a specific place.

Note: When we refer to encrypted Path, we only mean that the files inside the folders are encrypted, NOT THE FOLDER ITSELF!

Unencrypted Paths: <For dirtsands first launch>

/opt/dirtsand/ages/ <holds un-encrypted file versions of moulscripts>
/opt/dirtsand/Python/ <holds un-encrypted file versions of moulscripts>
/opt/dirtsand/SDL/ <holds un-encrypted file versions of moulscripts>

Encrypted Paths:

/opt/dirtsand/authdata/ages/ <holds encrypted file versions of moulscripts>
/opt/dirtsand/authdata/Python/ <holds encrypted file versions of moulscripts>
/opt/dirtsand/authdata/SDL/ <holds encrypted file versions of moulscripts>

dirtsand.ini searches for these locations by default:

File.Root = /opt/dirtsand/data
Auth.Root = /opt/dirtsand/authdata
Sdl.Path = /opt/dirtsand/SDL
Age.Path = /opt/dirtsand/ages

Now the one conflict here is that 'dat' and 'data' can be confusing

'dat' must be renamed to 'ages' to align to 'Age.Path' in dirtsand.ini
'data' is the copy of MOULa content in your "dat" folder of your fresh MOUL install off your windows box.

The most important server-side data for the client to access are the "authdata" server-provided files -- specifically, the ENcrypted "ages" and "SDL" and "python.pak" files.

SIDE NOTE: External Plasma clients require these files in order to function, so you will need to provide them unless you are planning on making your server only work with Internal client builds.

The difference between an Internal and an External build is... ( still to add )

To create the files, do the following:

NOTE: If you want to setup a PATH variable and shortcut this process, feel free to do so if you know how.

in firefox browse to https://github.com/H-uru/moul-scripts/tree/python27
to the far right of the branch menu click "Downloads"
Choose .zip format
Once downloaded create a folder called "moulscripts" in "\CWE Sandbox\plEngine\" and unzip the new zip file into it <NOTE: the zip file is a few folders deep. Just copy the "Python27" folder to "moulscripts" folder>
Make a copy of the "Python27" folder, and then do 2 things; rename the copied folder to "authdata" AND move it INTO the "Python27" folder. Now be careful with the next two steps!!
copy plPythonPack.exe, msvcr100d.dll and python27_d.dll into the "authdata" folder.
Launch "cmd" and browse to the "moulscripts\Python27\authdata" folder ( - if you dont know how to navigate in a Windows Command Prompt, you're not ready for this! )
in the terminal windows type...

plPythonPack.exe Python

...and press enter. The command should return without errors. If there any errors you may have used a 64 bit system to compile "plPythonPack.exe".

Your windows folder structure should look like this;

.\CWE Sandbox\plEngine\moulscripts\Python27\ages
.\CWE Sandbox\plEngine\moulscripts\Python27\Python
.\CWE Sandbox\plEngine\moulscripts\Python27\SDL
.\CWE Sandbox\plEngine\moulscripts\Python27\authdata\ages
.\CWE Sandbox\plEngine\moulscripts\Python27\authdata\Python
.\CWE Sandbox\plEngine\moulscripts\Python27\authdata\SDL

After you run step 7 you will notice 1 new file in the "moulscripts\Python27\authdata\Python" folder. Make sure you dont have any errors in the output.

.\moulscripts\Python27\authdata\Python\python.pak

Although we have just packed the Python folder into "python.pak" it is still not encrypted.

To encrypt this file set up libhPlasma tools on Windows: (to encrypt python.pak)

 NOTE: For the Linux fans you can compile your own at https://github.com/H-uru/libhsplasma

 NOTE: For simplicity we use the Plasma_Win32_r801.zip file off of http://libhsplasma.googlecode.com/files /Plasma_Win32_r801.zip

...and then set the PATH variable on windows to point to this folder. You dont want to use the Win64 version of libhPlasma, just in case you get any strange conflicts along the way.

Once you have the PATH variable updated, you have to reboot for the changes to take effect.

Now, the files that needs to be encrypted is the python.pak file you created earlier and all the "SDL" and "ages" files in the ".\CWE Sandbox\plEngine\moulscripts\Python27\authdata\" folder.

Navigate to the .\CWE Sandbox\plEngine\moulscripts\Python27\authdata\Python\ folder and type (drop the inverted brackets in the command below):

prompt:> PlasmaCrypt droid -key {paste the key.droid value from ".\CWE Sandbox\INI Files\dirtsand.ini" here} python.pak

It should look like;

prompt:> PlasmaCrypt droid -key 86756......283746 python.pak

This 'key.droid' must be 32 digits long.

Next you need to encrypt the contents of the "ages" and "SDL" folders.

This can be quote a task so to make it easier, it is best to create two spreadsheets and add your droid key.

Your command should look like:

PlasmaCrypt droid -key < key here > Ahnonay.age

You can build a list of files using the "dir -b > sdl-enc.csv" and "dir -b > ages-enc.csv" command piped to a file, for easy editing

Add droid key to spreadsheet
save as "csv" with space seperator
edit with scite or notepad, and remove all spaces and inverted commas
save the files with a "bat" extension into their respective folders in ".\CWE Sandbox\plEngine\moulscripts\Python27\authdata\" ie: "ages" and "SDL" folders.
use the terminal and browse to each folder and execute the batch file.
If you did it right, when you open any one of the files in scite, it will look like gibberish with the starting line, "NotTheDroids"

Now you are ready to create the manifest files.

MANIFEST FILES - Creating the manifest on Windows (will be copied to Ubuntu later):

In order to let the client know what files are available, you will also need to provide the following files in the "authdata" folder:

python_pak.list
SDL_sdl.list
AGES_ages.list

...which are simple manifest files of the format:

 Path\filename.ext,size-in-bytes

So, inside the python_pak.list file, you might have:

 Python\python.pak,123456

And inside the SDL_sdl.list file you would have;

 SDL\ahnonay.sdl,27311
 SDL\ahnonaycathedral.sdl,3524
 SDL\etc, etc

Note the use of a backslash here, since this path is provided directly to the client (which assumes a Windows path). Comments (starting with the '#' character) and whitespace will be ignored when parsing this file.

To create these files you will need the command prompt, a text editor and a spreadsheet program.

Open a text editor and save a blank file as Python_pak.list

In the command prompt, navigate to the "moulscripts\Python27\authdata\Python\" folder and type:

prompt:> dir

The entry for python.pak should have a file size in bytes alongside it to the left.

Type out the file size in bytes into the text editor (without spaces):

  Python\python.pak,***

becomes (the value to the right is only an example)

  Python\python.pak,510544

save the file as "Python_pak.list", and you are done with your first manifest list.

Next... change to the SDL folder...

prompt:> dir SDL > sdl.txt

This will create a file called sdl.txt in the root of "moulscripts\Python27\authdata\SDL" with all the file sizes in it. Now it is easy to load this into a spreadsheet program and do a clean up. We take it you know how to do that.

Then once in the spreadsheet, reorder the columns so that the filename comes first, and the file size second. You will need to add a column in front of all the columns with the SDL folder name and backslash "SDL\".

In scalc or excel a typical row would look like: (after all your fixing!)

Now, save it as csv (with comma as the separator), and open it in a text editor, and check it by hand to make sure it is right. (You will have to remove the first comma after the backslash when looking at the csv file contents - find and replace '\,' with '\')

Once you are happy with it, change the name of the file to SDL_sdl.list

That is manifest file number 2 done!

Now do the same for "ages". The result should be AGES_ages.list.

NOTE: Remember, don't confuse "data" with "dat" in moulscripts.

 "dat" should rightly be renamed to "ages"

 "data" is a renamed copy of the "dat" folder in your fresh MOUL install that dirtsand will use to "serve" data (levels) to the client.

Now, assuming you have prepared a fresh MOUL installation to your Games folder, browse to it and copy the "dat" folder to new folder called "data" and place this folder (as big as it is) in your "Python27" folder in "moulscripts".

So now, in ".\moulscripts\Python27\" you should have:

.\moulscripts\Python27\ages\
.\moulscripts\Python27\data\
.\moulscripts\Python27\SDL\
.\moulscripts\Python27\Python\
.\moulscripts\Python27\authdata\ages\
.\moulscripts\Python27\authdata\SDL\
.\moulscripts\Python27\authdata\Python\
.\moulscripts\Python27\authdata\Python\python.pak
.\moulscripts\Python27\authdata\AGES_ages.list
.\moulscripts\Python27\authdata\SDL_sdl.list
.\moulscripts\Python27\authdata\python_pak.list

Now before moving on, you will need to encrypt the age and fni files in the data folder;

This is done by using PlasmaCrypt with the droid key. Easy enough if you use a spreadsheet to help you build a batch file. If you do not encrypt the age and fni files, the Client will crash!

PlasmaCrypt droid -key 26353523........846464353 Ahnonay.age
PlasmaCrypt droid -key 26353523........846464353 Ahnonay.fni
etc... for all age and fni files!

Now copy the entire moulscripts folder to the HOME folder on your Ubuntu box.

CAUTION: If your python_pak.list is starts with a capital "P", you will get an error when you try to login later. Remember that Linux is CASE SENSITIVE! See error below!

secure file preloader failed

On Ubuntu - Bringing in Authdata, Manifests and Data into Dirtsand

6) "Authdata" files

For dirtsand to provide the game files to the client, we need to do four things:

1) set up unecrypted versions of the ages, python and SDL folders in the root of dirtsand, and
2) set up a directory for "authdata" for the encrypted ages, python and SDL folders in the dirtsand folder.
3) bring in the data with encrypted age and fni files (and process it into a .mfs and compressed version for serving via network)

Here are some of the spreadsheets and scripts created for this purpose; Use at your own risk and double check everything!

http://www.falsebaypost.co.za/cwe-sandbox/Sandbox Scripts.zip

Step 1: Dirtsand "root" folder

The reason for the first step above is that when dirtsand is launched for the first time, dirtsand needs to read the un-encrypted "ages", "Python" and "SDL" files to configure properly, so unencrypted versions are placed in the root of dirtsand...

/opt/dirtsand/

In this folder you will have unencrypted copies of "ages", "Python" and "SDL" so that dirtsand can configure properly when run for the first time.

/opt/dirtsand/ages/<unecrypted age and fni files>
/opt/dirtsand/SDL/<unecrypted SDL files>
/opt/dirtsand/Python/<unecrypted Python files>
and
/opt/dirtsand/data/<converted data files to send to client>

Step 2: Dirtsand "authdata" folder

Now to before we move on, remember we will be creating the "authdata" folder in...

 /opt/dirtsand/)

and then putting the "ages", "Python" and "SDL" folders and their ENCRYPTED files in the respective subdirectories within the "authdata" folder.

The folders:

 /opt/dirtsand/authdata/ages
 /opt/dirtsand/authdata/Python
 /opt/dirtsand/authdata/SDL

The files:

 /opt/dirtsand/authdata/<manifest>.list
 /opt/dirtsand/authdata/ages/**1.age
 /opt/dirtsand/authdata/ages/**1.fni
 /opt/dirtsand/authdata/ages/**n.age
 /opt/dirtsand/authdata/ages/**n.fni
 /opt/dirtsand/authdata/Python/python.pak
 /opt/dirtsand/authdata/SDL/**1.sdl
 /opt/dirtsand/authdata/SDL/**2.sdl
 /opt/dirtsand/authdata/SDL/**n.sdl

STILL TO CONFIRM: Only after you run dirtsand for the first time you may delete these 3 folders in the root of "/opt/dirtsand/" leaving the original "ages" and "SDL" folders in "authdata" untouched.

Remember that the manifest *.list files are not encrypted and go in your "authdata" folder! Do not put them in the "ages", "Python" or "SDL" folders.

Now remembering the structure above, plug in your memory stick in Ubuntu and in Nautilus (or a terminal) copy the moulscripts folder to your home directory.

Now before you can add the files to dirtsand you must change its ownership otherwise you'll be back at this step later! ; )

Open a terminal (it opens in your HOME directory by default) and type:

$ sudo chown -R dirtsand moulscripts

The -R flag will make sure "chown" changes all files even inside folders.

Now, we can go to our terminal window, and type:

$ su - dirtsand

If you type "ls" you will see that you are in the root of the "dirtsand" folder

$ cp -Rf /home/<username>/moulscripts/Python27/ages .
$ cp -Rf /home/<username>/moulscripts/Python27/data .
$ cp -Rf /home/<username>/moulscripts/Python27/Python .
$ cp -Rf /home/<username>/moulscripts/Python27/SDL .
$ cp -Rf /home/<username>/moulscripts/Python27/authdata/*.* opt/dirtsand/authdata/
$ cp -Rf /home/<username>/moulscripts/Python27/authdata/ages opt/dirtsand/authdata/
$ cp -Rf /home/<username>/moulscripts/Python27/authdata/Python opt/dirtsand/authdata/
$ cp -Rf /home/<username>/moulscripts/Python27/authdata/SDL opt/dirtsand/authdata/

3) Data Server

DIRTSAND includes a full data server with its services. You will need to set up a data/ directory and the necessary manifests. The data folder sits in root of dirtsand...

/opt/dirtsand/data/

The data manifests sit inside this folder...

/opt/dirtsand/data/<manifest>.mfs

NOTE: At a later date you may choose to create the ThinExternal.mfs and External.mfs files, which describe the files necessary for the patcher and client (respectively) to run. Don't worry about this now.

You will need to create age manifests of the form "<agename>.mfs" for each age and prp file (excluding the .fni files), which will be requested by the client when it attempts to link to an age.

The format of the manifest files is as follows:

remote_filename.ext,local_filename.ext.gz,decompressed_hash,compressed_hash,decompressed_size,compressed_size,flags

There is a helper script provided in bin/dsData.sh which will gzip a file and generate a manifest line with the correct hashes and sizes for you automatically. Take care that the remote path expects to use a Windows path/filename, so it should use backslashes instead of forward ones, whereas the local filename should use Unix slashes.

SIDE NOTE: Alternatively, you can specify a remote file server address in your dirtsand.ini, which will allow the files to be fetched from an external file server. This won't be covered in this wiki.

The best way to start this process of converting the data folder to the correct format is to run the command;

$ ls -b /opt/dirtsand/data >> data.list

This will create a file with a listing of the filenames in the data/ folder.

Now load it into scalc and create a layout for each file as follows;

/opt/dirtsand/bin/dsData.sh AhnonayCathedral.age >> AhnonayCathedral.mfs
/opt/dirtsand/bin/dsData.sh AhnonayCathedral_District_BuiltIn.prp >> AhnonayCathedral_District_BuiltIn.mfs
/opt/dirtsand/bin/dsData.sh AhnonayCathedral_District_Textures.prp >> AhnonayCathedral_District_Textures.mfs

Do this for EVERY age and prp file.

When it is done you should have a ".gz" and a ".mfs" version for each file. You wont see the original files anymore because they have been packed into the ".gz" archive.

Now comes the tricky part... In each .mfs file you need to append the tag "dat\" at the beginning of the line. If you don't your Client will become corrupt when you login. You can do this by hand... and take a week to do it, or build a spreasdsheet to help you create a script, which means an hour or so tops.

The best way to this is to use a spreadsheet as a basis to create a shell script.

Start by creating a small file called "stub" in the data folder. Inside it put the line "dat\" and save it.

The spreadsheet layout should be as follows; (Make sure that a "column" is where a "space" is) and...

...DON'T FORGET THE SEMI COLON AT THE END - You will get weird things happening if you do...

#! bin/bash
paste stub AhnonayCathedral.mfs | sed 's/\t//' >> AhnonayCathedral.mfs1;
paste stub AhnonayCathedral_District_BuiltIn.mfs | sed 's/\t//' >> AhnonayCathedral_District_BuiltIn.mfs1;
paste stub AhnonayCathedral_District_LinkRoom.mfs | sed 's/\t//' >> AhnonayCathedral_District_LinkRoom.mfs1;
paste stub AhnonayCathedral_District_Textures.mfs | sed 's/\t//' >> AhnonayCathedral_District_Textures.mfs1;

What this will do is join the line "dat\" to the line in each ".mfs" file and save it as a new ".mfs1" file.

Then to get the filename corrected, once you check that the ".mfs1" file contents is correct by using your trusty text editor, delete all the ".mfs" files (or backup them up somewhere else) and run the next command;

#! bin/bash
mv AhnonayCathedral_District_BuiltIn.mfs1 AhnonayCathedral_District_BuiltIn.mfs;
mv AhnonayCathedral_District_LinkRoom.mfs1 AhnonayCathedral_District_LinkRoom.mfs;
mv AhnonayCathedral_District_Textures.mfs1 AhnonayCathedral_District_Textures.mfs;

Now you should have a ton of ".gz" and ".mfs" files in the data folder in dirtsand.

On Windows - Building a new MOUL install

8) Setting up your Windows Client:

///Note from Phoenix: Added a step by step for a fresh windows MOUL client installation

If you want to setup your Windows client from scratch with your compiled versions plClient and plUruLauncher and your new server.ini file, follow these steps:

Install MOULInstaller.exe (There is a later version MoulInstaller887.exe)

Watch out when the installer finishes... it tries to connect to Cyan, so make sure you got your network cable unplugged temporarily. You dont want their updates for your shard!

Then into the root of your fresh Sandbox installation do this:

Backup your UruLauncher.exe
Backup your UruExplorer.exe
Copy your compiled plClient.exe from .\CWE Sandbox\plEngine\ to the root of your Sandbox install and rename it to UruExplorer.exe
Copy your compiled plUruLauncher.exe .\CWE Sandbox\plEngine\ to the root of your Sandbox install and rename it to UruLauncher.exe
Copy wrap_oal.dll
Copy NxCharacter.dll
Copy NxCooking.dll
Copy NxExtensions.dll
Copy OpenAL32.dll
Copy python27.dll
Copy python27_d.dll
Copy resource.dat
Edit tos.txt with the GNU GPL license

To setup the server.ini file...

Download Zrax's ini file at http://moul.zrax.net/zraxnet.ini and save it to .\CWE Sandbox\INI Files\ on your Windows box.
Get a copy of Plasma_Win32_r801.zip and unzip it to .\CWE Sandbox\CWE Tools\Plasma_Win32\
Copy the zraxnet.ini file into Plasma_Win32
Rename zraxnet.ini to mynet.ini
Open a Windows Command Prompt and change your directory to .\CWE Sandbox\CWE Tools\Plasma_Win32\
Run the following command:

Prompt:> PlasmaCrypt decrypt mynet.ini

Once the command is finished, open the file with a text editor (Use one that keeps format in place - Like Scite or Scintilla, just easier than notepad)

Open your Dirtsand.keys file and paste your 6 Client auth keys that were generated from dirtsand over the 6 Client auth keys in the mynet.ini file. If you open the mynet.ini file on your windows box and see mumbo jumbo, you did not decrypt your file correctly, so try the decrypt command again.

Now the last entries in the mynet.ini file are:

   Server.Gate.Host "xxx.xxx.xx.xxx" (IP address assigned to Ubuntu -

       - to find this go to your ubuntu terminal and type ifconfig, and use the address after "inet addr:")

   Server.Auth.Host "xxx.xxx.xx.xxx" (same as above)

   Server.DispName "The journey continues..." (whatever catch phrase kicks your hair back)

   Server.Status "<same as above>/shardstatus.php" (contains a simple PHP print command,

       - eg: 'hello world' or 'server is up' - Needs explanation on setting up simple Apache -   Anyone?

   Server.Signup "<same as above>/newacct.php" (no idea how to create the account creation file...

       - and how it ties back to the dirtsand command "addacct <username> <password>" ) - Anyone?

Once done you will need to encrypt the mynet.ini file with the xtea parameter in PlasmaCrypt. Just run the command:

Prompt:> PlasmaCrypt xtea mynet.ini

The command returns a success if all went well, copy mynet.ini file to...

       .\CWE Sandbox\INI Files\

and to

       .\Games\Cyan Worlds\Uru Live - Sandbox\

The reason you make two copies is that if you muck up your URU Live - Sandbox installation, you have a backup copy of your mynet.ini file in a safe place.

Now rename mynet.ini in your ".\Games\Cyan Worlds\Uru Live - Sandbox\" directory to server.ini and... badda-bing! you have your own custom server.ini file ready to link to your sandbox shard.

On Ubuntu - Running the Sandbox Server, pgAdmin3 and Wireshark

9) Run the server:

Assuming everything else went smoothly, you should now be able to start your server and connect to it!

To start the server... You will use this command every time you want to start dirtsand after a reboot.

$ bin/dirtsand dirtsand.ini

You'll have to create an account first, which can be done from the console:

$ ds-902> addacct <username> <password>

NOTE: The first time you run this can be quite odd since no vault is yet created. It seems to error out. When it happens just press enter and you should get to the dirtsand prompt.

SETTING UP pgAdmin3:

To check if you have created a user successfully, you will need to download pgadmin3...

$ sudo apt-get install pgadmin3

This will help you access postgres in a friendly gui environment. Setting up pgadmin3 is quite simple once it is installed. Click on "Applications" , "Programming" , "pgAdmin3"

The gui will launch. Once open click on the icon thats looks like an electrical plug to create a new server connection.

The fields should be exactly as below unless you changed them when you set up postgres a number of steps back.

  Name: dirtsand
  Host: 127.0.0.1
  Port: 5432 - leave this as it is
  SSL: blank - (Not important right now... keep it simple)
  Maintenance DB: postgres
  Username: dirtsand
  Password: <whatever you chose for your password>

Leave the rest of the window as it is. You will notice the windows has no "OK" button. Just after you entered your password, please "Enter" and the connection will be made. A dialog will hop up about "Saving passwords", read it and click "OK".

If it fails, you entered something wrong, so try again by clicking on the plug icon. You have succeeded to login to the database when you see in the left hand pane.

"Server Groups", below that "Servers", and below that "dirtsand"

To check the new player you added, click on "dirtsand", then the + sign to the left of it to expand it. Then "Databases" , "dirtsand" , "Schemas" , "auth" , "Tables" , "Accounts"

Now, look for the little blue icon in the toolbar that looks like a spreadsheet... and click it.

You should be looking at your first account that you created with the command;

 ds-902> addacct <username> <password>

If you dont see the exact username you created, you will have to backtrack your steps in this wiki, and try to figure out where you went wrong.

If all looks well, save your connection... "File" , "Save Definition" , then give it a cool name like "Sandbox Shard" and click "Save" otherwise when you next launch pgAdmin3 you will have to manually reconnect again.

SETTING UP Wireshark:

This tool is very useful to check if your client is connecting to your Ubuntu shard. To install it run...

$ sudo apt-get install wireshark

It is best to do a reboot, and when back in Ubuntu DON'T run it from the "Applications" menu...

Open a terminal, and type...

$ sudo wireshark

NOTE: If you run it from the App menu you won't find your ethernet controller.

Now, once wireshark has launched you will get a warning. Acknowledge it, and click "OK". Please don't do anything rash while using wireshark. Just follow these instructions.

Once you click "OK" you should be taken to the wireshark main interface. Look under Capture on the main page, and identify your ethernet controller. Generally it is called "eth0" but you may have some other configuration.

If you click on "eth0" wireshark will start to capture all traffic over that interface immediately. This is a good sign that wireshark and your ethernet controller are both working!

Stop the live capture by clicking on the icon with the little red cross.

Click on "Capture" in the top menu, and click "Options"

Now, be careful... the only field you want to fill out is the one to the right of "Capture Filter".

You will need to jump to your Windows box quickly and in a command terminal run

  prompt:> ipconfig

Look through the list and look for the IPv4 address for your computer... Something like this;

Ethernet adapter Local Area Connection:

 Connection-specific DNS Suffix  . :
 Link-local IPv6 Address . . . . . : ****::****:****:****:***%**
 IPv4 Address. . . . . . . . . . . : 124.122.23.11  << Your IP address will be here!
 Subnet Mask . . . . . . . . . . . : 255.255.255.0
 Default Gateway . . . . . . . . . : 235.211.11.1

Now back on Ubuntu fill in that field just mentioned, type;

host 124.122.23.11 (replace with whatever your IP is - the word "host" is crucial!)

Do not change any settings beside this, and simply click "START", then "Continue without Saving"

What this will do is ignore all other internet traffic except traffic from your Windows client. This makes reading wireshark much easier than having to sift through a ton of traffic!

The screen should be blank, so first we need to test the connection. First, in a new terminal on Ubuntu, run...

$ /sbin/ifconfig

Look for "inet addr" the IP address under "eth0" of your Sandbox and write it down. Take this to your Windows box.

On your Windows box, your command prompt should still be open after you ran ipconfig. Now run a ping command...

prompt:> ping ***.***.**.** (type out the full IP address)

Now brace yourself, hehe! As you hit "enter" keep a beady eye on your Ubuntu wireshark screen. You should get about 10 lines of connection reports. This is great... your client is "seeing" your Server!

Now in wireshark on Ubuntu click "Capture" , "Restart" to clear the screen.

Here is where you now hold your breath, because you are about to see if your recently compiled plClient.exe (and all the work you have done) can establish a connection with your shard.

On Windows - Logging into the Sandbox Server

10) Run the Client: On Windows...

Launch UruExplorer (Make sure you run it from the ".\Games\Cyan Worlds\Uru Live - Sandbox\" directory and not from the desktop or startmenu), and enter the username and password you created in the step above on your Ubuntu box.

As you hit "Login" watch the wireshark interface on Ubuntu. It should start to show activity... this is very good!

/// Note from Phoenix : Dont know if the shard needs:

updated Cyan content or stock installer content (this will change the manifest files and their sizes)
does it need the 4 plasma dll's from Plasma 2.6.4 SDK?
what is this resources.dat in MOUL-OS?
Should /LocalData parameter be added to a plClient shortcut instead of using plClient?
Must the unencrypted Python and age files be added to the "CWE Sandbox" folder?

By now your SANDBOX should be ready to rock!

On Ubuntu - How to start again!...

11) How to Uninstall dirtsand:

If you dont have a working Sandbox Shard by now, don't despair! Even the veterans make mistakes. Usually it is those mistakes that are priceless, because we learn something out of it. Next time you try can only be better.

///Note from Phoenix: Needs dirtsand uninstall-howto if things go horribly wrong, and we want to reinstall dirtsand afresh.

The topics that will be covered...

How to delete the database
How to uninstall dirtsand
How to remove the dirtsand user
What else to clean up
Starting again!

Happy debugging!

Additional Information

Git/download links: