Important! Please read the announcement at http://myst.dustbird.net/uru-account.htm
Also! Please read the retirement announcement at http://myst.dustbird.net/uru-retirement.htm

Setting Up A Sandbox Shard with Dataserver

From UamWiki
Revision as of 07:05, 10 August 2011 by Phoenix (talk | contribs) (typo)

Contents

How to create a SANDBOX SHARD with Dataserver

For Non-Programmers

License: GNU GPL v3


You may copy and distribute this document as long as the name of the original author, version number and date stays in place.


by Phoenix Rising

Version 0.1

August 2011



Shorah Explorers

The aim of this wiki is to teach a writer/builder/developer how to build a Sandbox Shard to test CWE compatible ages.

This document will detail as closely as possible the exact steps to create your own Sandbox Shard using GoW's CWE Client and Zrax's Dirtsand Server. This wiki page is specifically for writers and age builders who want to test MOULa compatible ages with Dataserving capability.

We have permission from Zrax to use his README in this wiki but with a much expanded and detailed version due to some undocumented issues that have been clarified. We have chosen to use GoW's CWE Client since it has been heavily updated so that it may be compiled easily and with a much more recent version of DirectX SDK and other tweaks such as reduction of lag. This simply means that it has better support and hopefully better gameplay on your fancy new computer!


Difference between the Sandbox Shard with Dataserver and Tiny Sandbox Shard

The major difference between the two is that the Sandbox Shard is a fully blown dataserver which provides a way to keep unsynchronised clients up to date using the same data, while the Tiny Sandbox Shard keeps all its data within the Client.

Each has their own advantages and disadvantages;

The Tiny Sandbox Shard - Internal Client Setup

is easy to setup, easy to maintain and can help you test CWE compatible ages quickly. However you need to make sure that each client has exactly the same content, and if you test ages on one machine, you need to manually copy that content to the other machine as well.

The Sandbox Shard with Dataserver - External Client Setup

is useful for when you are planning (but not yet ready) to release your Shard to the public, and want to keep all the test clients that are connected to your Shard in sync. It is quite a beast to setup and has confused a few pro's, so dont try this until you have attempted the Tiny Sandbox Shard.

IMPORTANT NOTE

This work is almost complete and gets highly technical, and as such is not recommended for newbies. Minor changes may happen along the way so if you want to add to this wiki, please get permission from the author first! We want this to appeal to every Non-Programmer young and old and have a really cool wiki to show for it. If you have helpful suggestions feel free to drop the author a line. A lot of hard work has gone into documenting this procedure for the sake of all the Uru fans out there. At the end of the day all this effort is due to teamwork, since no-one can do such a mammoth task alone!

HARDWARE PREREQUISITES

We will be using at least two computers (one of which must have an able graphics card).

As a benchmark we will be building the Sandbox based on:

  • CLIENT MACHINE: A Dell Inspiron running Windows 7 64 bit, with a 1Gb Ati Graphics card, and
  • SERVER MACHINE: An Asus Eeepc mini laptop running Ubuntu 10.10 (Maverick).

The specs are not crucial, except for the graphics card on the Windows Machine and the Flavour of Linux on the SERVER. We have chosen Ubuntu 10.10 because although we still have to configure stuff, we dont like configuration nightmares!

A simple test to see if your Windows CLIENT can handle all of CWE's graphics requirements is to ramp up all your Graphics settings to maximum when you run the launcher for TPOTS and see if your computer runs smoothly or has screen jitter/stutter. If your machine can handle these high settings you will be less restricted in your age building later on.


NETWORK CONNECTION

This wiki will attempt to document one of the two approaches to network connectivity.


  1. SANDBOX MODE - SERVER and CLIENT on a local hub using DHCP or fixed IP.
  2. SHARD MODE - SERVER on dynamic DNS or fixed IP, and CLIENT accessing through the internet.


SHARD MODE will be covered in another wiki at a later date.

Both configurations have their merits. The first provides the "offline" sandbox mode for creating your own ages and testing the content, even though you will still be connecting via a network of some kind. The second makes your shard accessible to other players over the internet. This you will use when you are ready to go live!

Assumptions and Caveats to follow this Wiki

  • You will be compiling the CWE client on a WinXp 32 Bit machine.
  • You know how to use Windows command line
  • You know how to use a Linux command line
  • You know how to compile C++ software on Windows
  • You know how to compile C++ software on Linux
  • You are fluent using both systems and understand the subtleties of each, such as: case sensitivity, backslash versus forward slash, and what a terminal is.
  • You know how to use scalc or excel
  • You know what a batch file is
  • You know what a script is


If you fail to do any of the above, we really suggest that you do not attempt this.


The major caveat is that will have have to create some spreadsheets that are used as a template to create batch and script files. As tedious as it may seem this is due to a few issues;


  • The official plasma tools does not support recursion.
  • Your newly compiled plPythonPack does not support recursion.
  • If you had to encrypt one file at a time you would be here for two weeks. Just take a look at how many files are in the 'SDL', 'age' and 'dat' folders!
  • The scripts give you a way to update bulk content packages aimed at the content YOU want on YOUR shard.
  • Once the scripts are created, you wont have to create them again, just remember to save them somewhere safe.


Ok, let's begin:


As this wiki unfolds you will notice that we will have to do work on both the SERVER and the CLIENT at any one time, so be prepared for it before it happens. We will refer to the client as Windows, and the server as Ubuntu for ease of reference.

Github and the branches

There are 3 main projects that we will pull from github.

  1. https://github.com/H-uru/Plasma
  2. https://github.com/H-uru/dirtsand
  3. https://github.com/H-uru/moul-scripts/tree/python27

An easy way to get the code for each is to click the "Downloads" button to the far right of "branches" tab.

You will also need to download the windows version of plasma tools to help you encrypt the various files later.

(Note, we won't be covering how to build a Cyan compatible engine, since Cyans MOUL still runs on Python 2.3 and CWE runs on Python 2.7)


On Windows - Let's compile CWE

Our starting point is to build the CWE client based on the "master" branch on github since it is a relatively simple process and will help you get your feet wet.


To build the client you can follow the GoW tutorial and video clip below:



An Undocumented item in the above tut

Just remember that you want to compile a 'Release' build. The default tut above compiles a 'debug' build. When you run a debug build it throws out errors with numerous dailog boxes, so it is preferable to build the 'Release' build for uninterrupted gameplay.

The two build and their dependancies; (You can find these dll's in the 'cwe-prefix' folder)

Release version needs - python27.dll Debug version needs - python27_d.dll and ms****.dll's


In Visual Studio simply go to the 'Build' menu, 'Configuration Manager', 'Active Solution Configuration' and select 'Release' and save the project.


Once you have compiled, you will have 3 files in the 

.\Plasma\build\Sources\Plasma\Apps\

directory, which you will use later. 

.\plClient\release\plClient.exe
.\plUruLauncher\release\plUruLauncher.exe
.\plPythonPack\release\plPythonPack.exe

Now, time to delve into the SERVER. The next portion is Zrax's heavily edited README for his Dirtsand server. It is assumed that you have Ubuntu 10.10 installed, and that you wrote down the superuser password somewhere safe!

On Ubuntu - DirtSand Howto

The original README is here...

For ease of implementation we will be assuming that the server will be running on Ubuntu 10.10 (Maverick) Desktop version. This has proven to be the less painful way to delve into SERVER setup.

CAUTION

If you simply copy and paste Linux commands from this wiki, sometimes the terminal will execute the command automatically without you pressing enter. For safety, copy the command to a text editor, change the command as needed, then copy to the terminal window. Otherwise you may have to uninstall everything and start again!

Building the code

Prerequisites or dependancies are libraries or programs that another program needs to run or to compile properly. Without them you will get many errors. Do not proceed with this section of the wiki if you have not satisfied the list below.

Prerequisites:

  • GCC 4.4+ (might work with other C++0x compliant compilers, but untested)
  • Postgres (libpq) -- the actual database server can reside anywhere
  • OpenSSL
  • libreadline
  • zlib


The default install of Ubuntu does not install all the items above. To install the packages that are lacking, open a terminal window, type the command below and follow the prompts before moving on to the next item in the list. This should satisfy the list of prerequisites above.

You may find that "libpq" will not install. This only happens on a fresh install of Ubuntu 10.10. Simply update the essential updates for Ubuntu, reboot, and try again.


For those of you with Ubuntu 10.10, this is the easy part: 

$ sudo apt-get install postgresql postgresql-contrib-8.4 libpq-dev apache2 cmake cmake-qt-gui g++ libreadline-dev


1) Now, create a user and path for the DIRTSAND server to run from. 

$ sudo useradd dirtsand -d /opt/dirtsand -m -p <password> -s /bin/bash


Remember to drop the brackets <> when you create a password and watch your case (UPPERCASE or lowercase)!

2) Once done, unzip your copy of the "dirtsand" source code in your HOME directory. 

$ cd dirtsand


3) Compile it for your system... 

$ mkdir build && cd build
$ cmake -DCMAKE_INSTALL_PREFIX=/opt/dirtsand ..
$ make && sudo make install
$ cd ..

If you run into any errors about finding libraries or headers, make sure you have the *development* versions of all of the required libraries, and that they are in your path. If you really get stuck, you can also use the "cmake-gui" to help 'cmake' locate the missing paths and files. 

$ cmake-gui

You have it already when you ran apt-get command earlier on.

Setting up a server - WITH a superuser account

You will need a working PostgreSQL server which DIRTSAND can use to store its data. Although you don't need superuser access to the postgres server, you will need to have a database which you can add schemas, tables, etc into.

For the default installation, the commands below will create a 'dirtsand' database and set its ownership to a 'dirtsand' database user, which can directly map the system 'dirtsand' user created in the "Building the code" instructions above. For better security, it is recommended to use a password (as shown in the steps below), which can be configured in the server settings as described in the "configure dirtsand" step.


1. Set up the postgres user

Now run the following commands and remember to drop the brackets <> around the password. 

$ sudo -u postgres psql -d template1
template1=# CREATE USER dirtsand WITH PASSWORD '<password>';
template1=# CREATE DATABASE dirtsand WITH TEMPLATE = template0 ENCODING = 'UTF8';
template1=# ALTER DATABASE dirtsand OWNER TO dirtsand;
template1=# \q

2. Install the UUID functionality

This may be provided by your OS distribution. In Ubuntu, simply install the postgresql-contrib-8.4 package (Check the apt-get list at the start of the SERVER SECTION) to provide the necessary libraries and installation scripts. If your distribution does not provide a contrib or uuid-ossp bundle, you can get it and build it yourself from the sources provided at: http://www.ossp.org/pkg/lib/uuid/

Once you have the ossp, you can add it to the dirtsand database by running the command: 

$ sudo -u postgres psql -d dirtsand < /usr/share/postgresql/8.4/contrib/uuid-ossp.sql

3. Set up the dirtsand database

$ sudo -u postgres psql -d dirtsand < db/dbinit.sql
$ sudo -u postgres psql -d dirtsand < db/functions.sql

NOTE: If you see the message 'ERROR: language "plpgsql" does not exist' while importing functions.sql, you will need to create the plpgsql language in the database: 


$ sudo -u postgres psql -d dirtsand
dirtsand=# CREATE LANGUAGE plpgsql;
dirtsand=# \q


After this, you should re-import the functions.sql file to properly create the necessary dirtsand functions by running... 

$ sudo -u postgres psql -d dirtsand < db/functions.sql

If there were no other errors, your database should be ready for DIRTSAND

4. Configure dirtsand

A sample dirtsand.ini has been provided in the root of the dirtsand sources. You will need to copy this to your install directory and then edit the fields you need for the server. Specifically, you will need to adjust the server addresses and the RC4 keys. If you have dirtsand installed to somewhere other than /opt/dirtsand, you will also need to point the configuration to the right paths too. 

$ sudo cp dirtsand.sample.ini /opt/dirtsand/dirtsand.ini
$ sudo chown dirtsand /opt/dirtsand/dirtsand.ini
$ su - dirtsand
$ <your-favorite-editor> dirtsand.ini

If you use gedit as <your-favorite-editor> you may find it won't run in the dirtsand account, so you will need to exit the dirtsand user, use sudo then log back in to the 'dirtsand' user.

To generate the RC4 keys, you can simply run the keygen command from within the dirtsand interactive console: 

$ bin/dirtsand
ds-902> keygen new       (This one will take a little while)
ds-902> quit

Any errors that dirtsand spits out about config files and postgres passwords can be ignored, since you haven't provided a configuration file yet.

You should now have a bunch of keys output on your terminal.

In your terminal, go to Edit, Select All, and Copy. Then paste the output to a file and save it as "Dirtsand.keys".

NOTE: You will need to copy this file to your Windows box later as well as the "dirtsand.ini" file.

The first block (labeled Server keys) in "Dirtsand.keys" is the set you should paste into your "dirtsand.ini" in the directory /opt/dirtsand/. Replace the dummy lines (with the '...' values) with the Server section in your "Dirtsand.keys" file.

FOR LATER on Windows: The second set of keys will be placed directly in the client's "server.ini" file.


NOTE: This requires either the H-uru fork of CWE or PlasmaClient.

Caution -- the vanilla MOUL client cannot load keys from a file, and you will have to enter the keys as byte arrays directly into the executable. This will not be covered here.

Later we will explain how to add the Client keys on the Windows box.

Now before we move on, you need to unzip the 'moulscripts' Python27 branch to your HOME folder.

When you have unzipped the "Python27" branch of moulscripts, you will see 3 folders in "Python27" folder: 

./dirtsand/dat
./dirtsand/Python
./dirtsand/SDL

One of these needs to be renamed to fall in line with the naming convention in the dirtsand.ini file.

Rename "./Python27/dat" to "./Python27/ages" (This is to save you a couple of head scratches later on) like so: 

$ mv dat ages

The files inside these folders are all the age, Python and SDL information dirtsand needs for the ages in UN-encrypted format.

Leave them unencrypted for now because you will see later you need two copies of these files in unencrypted and encrypted format! More on that later...

CAUTIONARY NOTE

Since Plasma, dirtsand and moul-scripts are in HEAVY alpha, you might find things don't compile or throw out lots of errors. First check that you are compiling on a 32 bit WinXp system. Win7 64 bit is a little rebellious when it comes to 32 bit compatibility.

On Windows - Data, Authdata and Manifests

Create a Fresh MOUL Installation

First, install MOULInstaller887.exe with your internet connection unplugged.

Watch out when the installer finishes... it tries to connect to Cyan, so make sure you got your network cable unplugged temporarily. You don't want their updates for your shard! The reason is, Cyans content is based on python 2.3, and your shard only runs python 2.7.


NOTE

If you let the Installer update with Cyans latest content, you will still get the same result.

Rename your new installation to a name that suits you. You will be referring to it a little later.

Now, you should copy the Dirtsand.keys and dirtsand.ini file over to your Windows box now, and place them in their own folder for easy reference.

For dataserving to work you will need to create some server-side data such as manifests, encrypted 'ages', 'SDL' and python.pak and the content (data) of MOUL, but make special note that although this is created on Windows, you will copy the newly created files over to your UBUNTU box later!

The two tools you will use is plPythonPack.exe that was compiled earlier on and PlasmaCrypt.exe in your "Plasma_Win32" (libhsplasma) download.

For dataserving, the most important server-side data for the client to access are the "authdata" server-provided files -- specifically, the ENcrypted "ages" and "SDL" and "python.pak" files.


You will need to do a few things;

  1. Copy the Dirtsand.keys and dirtsand.ini file over to your Windows box now, and place them in their own folder for easy reference.
  2. Download libhsplasma from http://libhsplasma.googlecode.com/files/Plasma_Win32_r848.zip and unzip it to a safe place.
  3. Setup a PATH variable to your unzipped Plasmatools (libhsplasma) folder and reboot your WinXP box.
  4. unzip a fresh copy of moulscripts from https://github.com/H-uru/moul-scripts/tree/python27 and unzip it.
  5. Make a copy of the "Python27" folder, and then do 2 things; rename the copied folder to "authdata" AND move it INTO the "Python27" folder.
  6. copy plPythonPack.exe and python27.dll into the unzipped 'Python27' folder.
  7. Launch "cmd" and browse to the "\Python27\" folder
  8. in a windows terminal type...
plPythonPack.exe Python

...and press enter. The command should return without errors. If there any errors you may have used a 64 bit system to compile "plPythonPack.exe".

Your windows folder structure should look like this; 

\Python27\ages
\Python27\Python
\Python27\SDL
\Python27\authdata\ages
\Python27\authdata\Python
\Python27\authdata\SDL

You should notice 1 new file in the "\Python27\authdata\Python" folder named python.pak. 

\Python27\authdata\Python\python.pak


Although we have just packed the Python folder into "python.pak" it is still not encrypted.

To encrypt this file navigate to the '\Python27\authdata\Python\' folder and type (drop the inverted brackets in the command below): 

prompt:> PlasmaCrypt droid -key {paste the key.droid value from ".\CWE Sandbox\INI Files\dirtsand.ini" here} python.pak


It should look like; 

prompt:> PlasmaCrypt droid -key 86756......283746 python.pak

This 'key.droid' must be 32 digits long from your dirtsand.ini file.

Next you need to encrypt the contents of the "ages" and "SDL" folders. Look at the download 'Sandbox Scripts' for examples. I dont guarantee they work for your setup, so use at your own risk. 

http://www.falsebaypost.co.za/cwe-sandbox/Sandbox Scripts.zip

NOTE: Unfortunately there is no recursive support in plasmacrypt, so either do each file in each folder one by one (and take a week to do it!) or use the spreadsheet technique below or use the "Sandbox Scripts". The technique below is a little time consuming (about an hour per spreadsheet), but you will only have to do it once.

NOTE - For Admins

If you intend to manage your own Shard, knowing how to update your content is crucial.

We are going to create a spreadsheet with file listings for each 'ages' and 'SDL' folder that will be used to create the batch files.

Once you save the spreadsheet as a normal file, your final list of commands in each batch file should look like:

Example of ages-enc.bat 

PlasmaCrypt droid -key <key here> Ahnonay.age
etc
etc
etc

Example of sdl-enc.bat 

PlasmaCrypt droid -key <key here> Ahnonay.sdl
etc
etc
etc


If you want to do this yourself you can build a list of files using; 

dir -b > sdl-enc.csv
and
dir -b > ages-enc.csv

If you do create it yourself, this command will pipe its output to a file, so that you can edit it easily in scalc or excel easily.

Remember to:

  1. Add YOUR droid key to spreadsheet
  2. save as "csv" with space seperator
  3. edit with scite or notepad, and remove all spaces and inverted commas
  4. save the files with a "bat" extension into their respective folders
  5. check the format against the example above, ages-enc.bat and sdl-enc.bat.
  6. use the cmd terminal and browse to each folder and execute the batch file.
  7. If you did it right, when you open any one of the files in scite, it will look like gibberish with the starting line, "NotTheDroids"


Now you are ready to create the manifest files.


MANIFEST FILES

Creating the manifest on Windows (will be copied to Ubuntu later):

The server tells the client which files are available. For this you will also need to provide the following files in the "authdata" folder in unecrypted format (these are called manifests). If you have a better way to avoid using spreadsheets, just do it ; ) 

python_pak.list
SDL_sdl.list
AGES_ages.list

The format must be: 

  Path\filename.ext,size-in-bytes


So, inside the python_pak.list file, you will have one line; 


  Python\python.pak,123456


Note the use of a backslash here, since this path is provided by the server directly to the client (which assumes a Windows path).

To create these files you will need the command prompt, a text editor and a spreadsheet program.

Open a text editor and save a blank file as Python_pak.list

In the command prompt, navigate to the "\Python27\authdata\Python\" folder and type: 

prompt:> dir

The entry for python.pak should have a file size in bytes alongside it to the left of the filename.

Type out the file size in bytes into the text editor (without spaces): 

   Python\python.pak,*****


becomes (the filesize value after the comma is only an example) 

   Python\python.pak,510544


save the file as "Python_pak.list", and you are done with your first manifest list.

Next... navigate to the SDL folder... 

prompt:> cd ..
prompt:> cd SDL
prompt:> dir SDL > sdl.txt

This will create a file called sdl.txt in the root of "\Python27\authdata\SDL\" with all the file sizes in it. Now it is easy to load 'sdl.txt' into a spreadsheet program and do a clean up.


In scalc or excel a typical row would look like: (after all your fixing!)

Manifest.jpg

Now, save it and open it in a text editor, and check it by hand to make sure it is right. It should look like; 


  SDL\ahnonay.sdl,27311
  SDL\ahnonaycathedral.sdl,3524
  SDL\etc,3453
  SDL\etc,476585
  SDL\etc,345
  SDL\etc,23

You can use Sandbox Scripts as an example. Once you are happy with it, change the name of the file to SDL_sdl.list

That is manifest file number 2 done!

Now do the same for "ages". The result should be AGES_ages.list. Similar to below; 

  ages\ahnonay.sdl,27311
  ages\ahnonaycathedral.sdl,3524
  ages\etc,3453
  ages\etc,476585
  ages\etc,345
  ages\etc,23


NOTE - confusion over 'dat' and 'data' naming convention

Remember, don't confuse "data" with "dat" in moulscripts. This is the undocumented part in the dirtsand README that has been clarified.

  • "dat" should rightly be renamed to "ages" to fall in line with the naming convention within dirtsand.ini
  • "data" is a renamed copy of the "dat" folder in your fresh MOUL install that dirtsand will use to "serve" data (levels) to the client.

Now, assuming you have prepared a fresh MOUL installation into your Games folder, browse to it and copy the "dat" content inside your fresh MOUL folder to a new folder called "data" and place this folder (as big as it is) in your "Python27" folder.

So now, in "\Python27\" you should have: 


\Python27\ages\
\Python27\data\
\Python27\SDL\
\Python27\authdata\ages\
\Python27\authdata\SDL\
\Python27\authdata\Python\
\Python27\authdata\Python\python.pak
\Python27\authdata\AGES_ages.list
\Python27\authdata\SDL_sdl.list
\Python27\authdata\python_pak.list

Now before moving on, you will need to COPY the encrypted contents of the '\authdata\ages\' folder into the 'data' folder... The reason is that the original content from MOUL is encrypted for Cyans droid key and not your Sandbox Shard, so you need to copy and overwrite all the *.age and *.fni files in '\Python27\data\' with your newly encrypted copies from '\Python27\authdata\ages'.


Now... copy the entire 'Python27' folder to the HOME folder on your Ubuntu box.

On Ubuntu - Bringing in Authdata, Manifests and Data into Dirtsand

OVERVIEW

For dirtsand to provide the game files to the client, we need to do four things:


  • 1) set up unecrypted versions of the 'ages', 'python' and 'SDL' folders in the root of dirtsand
  • 2) set up a directory for "authdata" for the encrypted 'ages', 'python' and 'SDL' folders in the dirtsand folder.
  • 3) bring in the BIG 'data' folder with encrypted .age, .fni and .prp files and process those files with dsData.sh
  • 4) place the unencrypted manifests in the 'authdata' folder.


Here are some of the spreadsheets and scripts created for this purpose; Use at your own risk and double check everything! 


http://www.falsebaypost.co.za/cwe-sandbox/Sandbox Scripts.zip


Dirtsand "root" folder - Explanation

The reason for the first step above is that when dirtsand is launched for the first time, dirtsand needs to read the un-encrypted "ages" and "SDL" files to configure properly, so unencrypted versions are placed in the root of dirtsand... 


/opt/dirtsand/


In this folder you will have unencrypted copies of "ages" and "SDL" so that dirtsand can configure properly when run. 


/opt/dirtsand/ages/
/opt/dirtsand/SDL/

You will also copy the big 'data' folder to the dirtsand folder; <converted data files to send to client> 

/opt/dirtsand/data/

Dirtsand "authdata" folder - Explanation

Now to before we move on, remember we will be creating the "authdata" folder to point to... 


 /opt/dirtsand/authdata)


and then putting the "ages", "Python" and "SDL" folders and their ENCRYPTED files in their respective subdirectories.


The folders: 


 /opt/dirtsand/authdata/ages
 /opt/dirtsand/authdata/Python
 /opt/dirtsand/authdata/SDL


And the manifests will then go here; 


 /opt/dirtsand/authdata/<manifest>.list
 /opt/dirtsand/authdata/ages/*.age
 /opt/dirtsand/authdata/ages/*.fni
 /opt/dirtsand/authdata/Python/python.pak
 /opt/dirtsand/authdata/SDL/*.sdl

NOTE: Remember that the manifest *.list files are NOT encrypted and go directly in your "authdata" folder.


CHANGE OWNERSHIP

Now remembering the summary above, plug in your memory stick into Ubuntu, and then in Nautilus (or a terminal) copy the Python27 folder to your HOME directory.

Now before you can add the files to dirtsand you must change their ownership otherwise you'll be back at this step later! ; )


Open a terminal (it opens in your HOME directory by default) and type: 


$ sudo chown -R dirtsand Python27


The -R flag will make sure "chown" changes the ownership of all files... even inside folders.

Now, we can go to our terminal window, and type: 


$ su - dirtsand


If you type "ls" you will see that you are in the root of the "dirtsand" folder 


/opt/dirtsand/

Now using a terminal you can copy the files over; 

$ cp -R /home/<username>/Python27/*.* /opt/dirtsand/.

SETUP DATA

To use the data server with its services you will need to run the 'dsData.sh' script on the content of the data sitting in; 


/opt/dirtsand/data/


NOTE - No recursive support

There is no recursive functionity in dsData.sh, so you can do this one file at a time or follow the steps below and use the spreadsheet method or refer to the Sandbox Scripts.

Extra NOTE - Patcher files

At a later date you may choose to create the ThinExternal.mfs and External.mfs files, which describe the files necessary for the patcher and client (respectively) to run. Don't worry about this now.


You will need to create 'age', and 'prp' manifests of the form "<agename>.mfs" which will be requested by the client when it attempts to link to an age.

NOTE: Dont process the 'fni' files with dsData.sh. Just leave them as they are.


The format of the manifest files is as follows: 


remote_filename.ext,local_filename.ext.gz,decompressed_hash,compressed_hash,decompressed_size,compressed_size,flags


The helper script provided in bin/dsData.sh will gzip a file and generate a manifest line with the correct hashes and sizes for you automatically. Take care that the remote path expects to use a Windows path/filename, so it should use backslashes instead of forward ones, whereas the local filename should use Unix slashes.

The best way to start this process of converting the data folder to the correct format is to run the command; 


$ ls -b /opt/dirtsand/data >> data.list


This will create a file with a listing of the filenames from the 'data' folder.

Now load it into 'scalc' or other spreadsheet program and create a row for each file as follows; 


/opt/dirtsand/bin/dsData.sh AhnonayCathedral.age >> AhnonayCathedral.mfs;
/opt/dirtsand/bin/dsData.sh AhnonayCathedral_District_BuiltIn.prp >> AhnonayCathedral_District_BuiltIn.mfs;
/opt/dirtsand/bin/dsData.sh AhnonayCathedral_District_Textures.prp >> AhnonayCathedral_District_Textures.mfs;
etc;
etc;
etc;

Do this for EVERY age and prp file. Be careful to INCLUDE the semi-colon at the end of each line.

Save this file as with a .sh extension and open it with a text editor. Then remove all the fluff that does not look like the format description above.

Now save it and run it from inside the 'data' folder.

When it is done you should have a ton of ".gz" and a ".mfs" version for each file. You wont see the original files anymore because they have been packed into the ".gz" archive.


Now comes the tricky part... In each .mfs file you need to append the tag "dat\" at the beginning of the line. This corresponds to the 'dat' folder in your MOUL install on windows. If this is not included, the client will write the updated data into the root of your MOUL.

To append 'dat\', you can either do this by hand... (and take a week to do it), or build a spreadsheet to help you create a script, which means an hour or so tops out of your time.


Start by creating a small file called "stub" in the 'data' folder. Edit it and put the line "dat\" without the inverted commas and save it.


The spreadsheet layout should be as follows; (Don't forget the semi colon at the end of each line.) 


#! bin/bash
paste stub AhnonayCathedral.mfs | sed 's/\t//' >> AhnonayCathedral.mfs1;
paste stub AhnonayCathedral_District_BuiltIn.mfs | sed 's/\t//' >> AhnonayCathedral_District_BuiltIn.mfs1;
paste stub AhnonayCathedral_District_LinkRoom.mfs | sed 's/\t//' >> AhnonayCathedral_District_LinkRoom.mfs1;
paste stub AhnonayCathedral_District_Textures.mfs | sed 's/\t//' >> AhnonayCathedral_District_Textures.mfs1;
etc;
etc;
etc;

Now save it with a '.sh' extension and open it in a text editor to clean it up so it looks like above. Do not delete the single inverted comma's, eg: '

What this will do is join the line "dat\" to the line inside each ".mfs" file and save it as a new ".mfs1" file.

Then to get the filename corrected, once you check that the ".mfs1" file contents is correct by using your trusty text editor, delete all the ".mfs" files (or backup them up somewhere else) and run the next command based on another spreadsheet; 


#! bin/bash
mv AhnonayCathedral_District_BuiltIn.mfs1 AhnonayCathedral_District_BuiltIn.mfs;
mv AhnonayCathedral_District_LinkRoom.mfs1 AhnonayCathedral_District_LinkRoom.mfs;
mv AhnonayCathedral_District_Textures.mfs1 AhnonayCathedral_District_Textures.mfs;
etc;
etc;
etc;


Now you should have a ton of ".gz" and ".mfs" files in the data folder in dirtsand.

Remember that if you set up another shard, you can simply reuse your new scripts that you built from spreadsheets, so this big task is now finally done and you wont need to play with spreadsheets beyond this point.

On Windows - Setup your Sandbox Client

To setup your Sandbox Client you will need your compiled version of plClient and a new server.ini file.


In the root of your fresh Sandbox installation that you created earlier on do this:


  • Delete UruLauncher.exe
  • Delete UruExplorer.exe
  • Copy your compiled plClient.exe to the root of your Tiny Sandbox Client
  • Copy wrap_oal.dll and OpenAL32.dll from a defacto Cyan MOUL installation
  • Copy NxCharacter.dll, NxCooking.dll, NxExtensions.dll and PhysX_Loader.dll from C:\Program Files (x86)\AGEIA Technologies\AGEIA PhysX SDK\v2.6.4\Bin\win32
  • Copy python27.dll from C:\cwe-prefix\dll to the root of your client
  • Copy resource.dat from http://www.guildofwriters.com/tools/resource.dat to the root of your client
  • Edit tos.txt with the GNU GPL license
  • If you dont have the correct version of PhysX download and install it from http://www.nvidia.com/object/physx-9.10.0513-driver.html


Then to setup the server.ini file...


  • Copy the Zrax's ini file at http://moul.zrax.net/zraxnet.ini and save it to your Sandbox.
  • Rename zraxnet.ini to mynet.ini
  • Open a Windows Command Prompt, change to that directory and run the following command:


Prompt:> PlasmaCrypt decrypt mynet.ini


Once the command is finished, open the file with a text editor (Use one that keeps format in place - Like Scite or Scintilla, just easier than notepad)


Open your Dirtsand.keys file and paste your 6 Client auth keys that were generated from dirtsand over the 6 Client auth keys in the mynet.ini file. If you open the mynet.ini file on your windows box and see mumbo jumbo, you did not decrypt your file correctly, so try the decrypt command again.

Now the last entries in the mynet.ini file are: 


   Server.Gate.Host "xxx.xxx.xx.xxx" (IP address assigned to Ubuntu -

       - to find this go to your ubuntu terminal and type ifconfig, and use the address after "inet addr:")

   Server.Auth.Host "xxx.xxx.xx.xxx" (same as above)

   Server.DispName "The journey continues..." (whatever catch phrase kicks your hair back)

   Server.Status "<same as above>/shardstatus.php" (contains a simple PHP print command,

       - eg: 'hello world' or 'server is up' - Needs explanation on setting up simple Apache -   Anyone?

   Server.Signup "<same as above>/newacct.php" (no idea how to create the account creation file...

       - and how it ties back to the dirtsand command "addacct <username> <password>" ) - Anyone?


Once done you will need to encrypt the mynet.ini file with the 'xtea' parameter in PlasmaCrypt. Just run the command: 


Prompt:> PlasmaCrypt xtea mynet.ini


The command returns a success if all went well, copy mynet.ini to your Sandbox install on windows and rename it to 'server.ini'

On Ubuntu - Running the Sandbox Server

9) Run the server:

Assuming everything else went smoothly, you should now be able to start your server and connect to it!

To start the server... You will use this command every time you want to start dirtsand after a reboot. 


$ bin/dirtsand dirtsand.ini


You'll have to create an account first, which can be done from the console: 


$ ds-902> addacct <username> <password>


NOTE: The first time you run this can be quite odd since no vault is yet created. It seems to error out. When it happens just press enter and you should get to the dirtsand prompt.

If you need to verify that an account has been made, or you want to check that your server is listening for your windows client, you can follow the pgadmin and wireshark steps at the end of this wiki page.

On Windows - Logging into the Sandbox Server

10) Run the Client: On Windows...

Launch plClient and enter the username and password you created in the step above.

After the license screen CWE should load and you will be taken to menu to create an avatar. You know the rest.

Happy Exploring!

Additional Information

Git/download links:


SETTING UP pgAdmin3

To check if you have created a user successfully, you will need to download pgadmin3... 

$ sudo apt-get install pgadmin3


This will help you access postgres in a friendly gui environment. Setting up pgadmin3 is quite simple once it is installed. Click on "Applications" , "Programming" , "pgAdmin3"

The gui will launch. Once open click on the icon thats looks like an electrical plug to create a new server connection.

The fields should be exactly as below unless you changed them when you set up postgres a number of steps back. 

  Name: dirtsand
  Host: 127.0.0.1
  Port: 5432 - (leave this as it is)
  SSL: blank - (Not important right now... keep it simple)
  Maintenance DB: postgres
  Username: dirtsand
  Password: <whatever you chose for your password>


Leave the rest of the window as it is. You will notice the windows has no "OK" button. Just after you entered your password, please "Enter" and the connection will be made. A dialog will pop up about "Saving passwords", read it and click "OK".


If it fails, you entered something wrong, so try again by clicking on the plug icon.


You know you have succeeded to login to the database when in the left hand pane you see;


"Server Groups", below that "Servers", and below that "dirtsand"


To check the new player you added, click on "dirtsand", then the + sign to the left of it to expand it. Then "Databases" , "dirtsand" , "Schemas" , "auth" , "Tables" , "Accounts"

Now, look for the little blue icon in the toolbar that looks like a spreadsheet... and click it.

You should be looking at your first account that you created with the command; 

 ds-902> addacct <username> <password>


If you dont see the exact username you created, you will have to backtrack your steps in this wiki, and try to figure out where you went wrong.

If all looks well, save your connection... "File" , "Save Definition" , then give it a cool name like "Sandbox Shard" and click "Save" otherwise when you next launch pgAdmin3 you will have to manually reconnect again.


SETTING UP Wireshark

This tool is very useful to check if your client is connecting to your Ubuntu shard. To install it run... 

$ sudo apt-get install wireshark</source>

It is best to do a reboot, and when back in Ubuntu DON'T run it from the "Applications" menu...

Open a terminal, and type...

<pre>
$ sudo wireshark

NOTE: If you run it from the App menu you won't find your ethernet controller.

Now, once wireshark has launched you will get a warning. Acknowledge it, and click "OK". Please don't do anything rash while using wireshark. Just follow these instructions.

Once you click "OK" you should be taken to the wireshark main interface. Look under Capture on the main page, and identify your ethernet controller. Generally it is called "eth0" but you may have some other configuration.

If you click on "eth0" wireshark will start to capture all traffic over that interface immediately. This is a good sign that wireshark and your ethernet controller are both working!

Stop the live capture by clicking on the icon with the little red cross.

Click on "Capture" in the top menu, and click "Options"

Now, be careful... the only field you want to fill out is the one to the right of "Capture Filter".

You will need to jump to your Windows box quickly and in a command terminal run 

  prompt:> ipconfig


Look through the list and look for the IPv4 address for your computer... Something like this;

Ethernet adapter Local Area Connection: 

 Connection-specific DNS Suffix  . :
 Link-local IPv6 Address . . . . . : ****::****:****:****:***%**
 IPv4 Address. . . . . . . . . . . : 124.122.23.11  << Your IP address will be here!
 Subnet Mask . . . . . . . . . . . : 255.255.255.0
 Default Gateway . . . . . . . . . : 235.211.11.1


Now back on Ubuntu fill in that field just mentioned, type; 

host 124.122.23.11 (replace with whatever your IP is - the word "host" is crucial!)

Do not change any settings beside this, and simply click "START", then "Continue without Saving"

What this will do is ignore all other internet traffic except traffic from your Windows client. This makes reading wireshark much easier than having to sift through a ton of traffic!

The screen should be blank, so first we need to test the connection. First, in a new terminal on Ubuntu, run... 

$ /sbin/ifconfig

Look for "inet addr" the IP address under "eth0" of your Sandbox and write it down. Take this to your Windows box.

On your Windows box, your command prompt should still be open after you ran ipconfig. Now run a ping command... 

prompt:> ping ***.***.**.** (type out the full IP address)

Now brace yourself, hehe! As you hit "enter" keep a beady eye on your Ubuntu wireshark screen. You should get about 10 lines of connection reports. This is great... your client is "seeing" your Server!

Now in wireshark on Ubuntu click "Capture" , "Restart" to clear the screen.

Here is where you now hold your breath, because you are about to see if your recently compiled plClient.exe (and all the work you have done) can establish a connection with your shard.

A Little History

Cyan has come a long way since the release of Myst. Their original 2D "point and click" game eventually evolved into a fully three dimensional gameplay with unrestricted movement and engaging avatar animations. Although Myst was the forefather of MOUL, in our wiki we wont be covering the MYST engine or how to set it up. It is too outdated. If you really want to play Myst you can use your original Myst CD content, Drizzle and TPOTS.

We need to draw a line as to what versions of Cyan's engine we use, and what purpose each version serves. Below is a list of the commonly accepted Versions.

Only one version will be used to setup our SANDBOX SHARD. The CWE Client using the MOULa content!


  • Choru - Cyan's Closed Beta.
  • Ubiru - Ubisoft's Closed Beta.
  • ABM - Ages Beyond Myst - The vanilla single player version of Uru.
  • Prologue - The original MMO version of Uru. (Essentially an Open Beta)
  • UruUpdate12 - Update package to fix bug with ATI Graphics cards.
  • DNI - To D'ni Expansion Pack - an add-on extension that adds some of the Prologue Ages along with a recap of the Prologue story.
  • TPOTS - The Path of the Shell Expansion Pack - adds even more content to ABM.
  • CC - Complete Chronicles - a compilation of the first 4 packages as a single distribution.
  • UU - Untìl Uru - A no longer officially supported online version of ABM.
  • D'mala - The Cyan Worlds Until Uru Shard.
  • MOUL - Myst Online: Uru Live - The Third Online Incarnation of Uru
  • MOULagain / MOULa - Myst Online: Uru Live Again - Resurrected version of MOUL

and finally the Open Source Client that has been given to the community by Cyan Worlds, namely:

  • CWE - CyanWorlds.com Engine - the recently released Open Source client based on MOUL - Note: not a full source release.


If you want to start building ages, we do not recommend following this wiki beyond the following few lines. There are many tutorials on the Guild of Writers to help you build your first age.

Retrieved from "http://myst.dustbird.net/w/index.php?title=Setting_Up_A_Sandbox_Shard_with_Dataserver&oldid=2060"