Elenesski Object Database  EODB and NEODB
The Memento Pattern Database for Unity
21 Local - Usage Guide

The following Usage Guide exists in order to help you get up to speed with the the Elenesski Object Database.

Opening and Saving the Database

Everything in the system is focused around a database. Creating a database is easy, you simply say:

EODB myDatabase = new EODB("C:\Documents\Data");

EODB will then prepare the environment and allow now load or save objects to the database. The database keeps an in memory copy of the database, and when you want to persis the data out you say:

myDatabase.CommitToFileSystem();

This command writes the database copy back to the hard drive. If you want to make a new copy of the data while saving, you can say:

myDatabase.CommitToFileSystem("C:\Documents\NewData");

Cached and Non-Cached databases

By default, when you open a database with:

EODB myDatabase = new EODB("C:\Documents\Data");

The entire database is loaded into memory. This is also known as a cached database. While this strategy is okay for most cases, large databases can cause issues on mobile platforms, therefore you can open the database non-cached:

EODB myDatabase = new EODB("C:\Documents\Data",null,false);

This creates a duplicate of the "Data.db" file called "Data.tp", and object changes are written in/out of this temporary file.

When you .CommitToFileSystem, the platform reads the .tp file and writes a new Data.db file with all of its indexes. Note, the ".tp" file is not deleted after a save, allowing you to save multiple copies of a database.

Database Format

The database is stored as a single file. You can index your data to make it easier to search later. Each index requires a separate file. The name of the index is the extension added to the file root. For example, if the path is "C:\Documents\Data" and you create an index called "total", you will find two files in "C:\Documents" called:

Data.db    - The database containing all your data.
Data.total - Your custom index called "total".

If you open the database, non-cached, see above, there is a third file too, called:

Data.tp    - The cached database containing working copies of your data.

Defining Indexes

Relational database indexes create a dictionary of unique non-null values found in a column of a table. While EODB indexes can do that, the feature is more powerful:

  • You can derive any value you want and store it in the index and have it associated with your object via its OID.
  • The value you store in the index does not have to match the value stored in your object.
  • And by extension, the type of index doesn't have to match any columns in your class.

To begin, you need to declare which indexes are available in your database. This is done when you open the database:

EODB myDatabase = new EODB("C:\Documents\Data",new Dictionary<string, Type> {
            {"cdname",typeof(string)}
        });

This declares a custom indexes called "cdname". What it means is that the index called "cdname" will can contain "string" values. Each string value is now associated to your object, via it's OID, or Object Identifier.

When opening the database, the entire index is loaded into memory, regardless of whether the database is cached or not. While there is no limit as to the number of indexes, indexes can chew up a lot of memory.

Only supported types (see EODBTypeIO.Initialize()) can be used for an Index. Note: MemoryStream is a supported type, however, we recommend against using it as a type of index because it can result in a lot of used up memory.

More on indexes later on.

27 x 2 Types Supported

To read/write object data, what you write must resolve to a known type. It's generally easy to extend the types provided and it's also possible to write type converters, where the internal object format is different than what is in the database (it's converted as it's written to the database, and converted back when it's read from the database):

Out of the box, these types are supported, including Arrays for all of these types

  • Primitive Types
    • bool (unsigned 1 byte integer with 0 for false and 1 for true)
    • byte (unsigned 1 byte integer)
    • short (signed 2 byte integer)
    • int (signed 4 byte integer)
    • long (signed 8 byte integer)
    • float (signed 4 byte floating point)
    • string (up to 2Gb in size, which is impractical since the database limit is 2Gb)
  • Structures and Classes
    • Vector2 (2 floats representing X,Y)
    • Vector3 (3 floats representing X,Y,Z)
    • Rect (2 Vector3's representing position and size)
    • RectTransform (several Vector2 and Vector3 representing position, size, anchor and pivot)
    • Quaternion (4 floats representing X,Y,Z,W)
    • Color (4 floats representing R,G,B,A)
    • DateTime (1 8-byte long, representing DateTime.tick)
  • Large Objects
    • Texture2D (With transparency support)
    • AudioClip (all of the samples within the audio clip)
    • Mesh (vertices, triangles and UVs arrays)
  • Miscellaneous
    • MemoryStream (any large object such as images; for example, if you have an Image can save it to a stream, you can use a MemoryStream to save it.)
      • Note, while it is possible to store large objects like Images in the database, it is strongly recommended that you store the images in the file system instead and store the filename of the file instead. It will result in better I/O.

Supported types are declared in the EODBTypeIO.Initialize() method. Each supported type is declared by calling this method from Initialize():

private static void Add<T>(Func<Stream, object> aRead, Action<Stream, object> aWrite) {}

For example, an Integer is declared this way in Initialize():

Add<int>(ReadInt, WriteInt);

And defined this way:

#region --- INTEGER ---

internal static object ReadInt(Stream aStream) {
    byte[] BYTES = Read(4, aStream);
    return BitConverter.ToInt32(BYTES, 0);
}

internal static void WriteInt(Stream aStream, object aInt) {
    Write((int) aInt, BitConverter.GetBytes, aStream);
}

#endregion

Type Extensions

Extending the types is relatively easy to do. You simply add the type you want to read/write to the Initialize method and then write the appropriate low-level methods to read/write the type value to/from the stream. For example 3 lines of code to read/write Quaternions:

#region --- QUATERNION ---

public static object ReadQuaternion(Stream aStream) {
    return Structure.Read<float, float, float, float, Quaternion>(aStream, (X, Y, Z, W) => new Quaternion(X, Y, Z, W));
}

public static void WriteQuaternion(Stream aStream, object aQuaternion) {
    Quaternion QUATERNION = (Quaternion) aQuaternion;
    Structure.Write(aStream,QUATERNION.x,QUATERNION.y,QUATERNION.z,QUATERNION.w);
}

#endregion

Type Conversions

In some cases it may be necessary to write a type converter which converts aspects of your object into oe format that you persist into the database and then back again. String is often used in this case. For example, you want to create data as a JSON string, then parse it as it comes back to load into the database..

Type conversions occur in a class's Definition() method, and this is described in more detail in the section "Class Definitions ::: Structure". In a nutshell, when you describe your structure, you provide a function that retrieves the data to be stored (YOUR DATA -> database string) and an action that accepts the stored data that you transform into your object (database string -> YOUR DATA).

Understanding Static and Dynamic Data

Each object within EODB has a unique identifier called an OID (Object IDentifier). There are many situations where the data you are storing is for a singleton. A single object representing some aspect of your game, such as, audio settings. In these cases, you need to hard-code the object identifier, and when you do this, this is referred to as Static Data. While you can assign any non-zero OID, by convention, negative value OIDs are considered static data and positive OIDs are considered dynamic.

It's recommended that you create a const for all static data objects:

public const int cAudioSettings = -10;
public const int cVideoSettings = -11;
public const int cMusicSettings = -12;
public const int cGameSettings = -13;

When the data is dynamic, you will need an index to find the objects, since you cannot predict what the OID will be that was assigned to an object.

Versions

Associated with each object is a Version. Generally you don't need to worry about Versions, but they exist to handle object variations. Since EODB doesn't declare tables, columns or rows, each stored object is a collection of attributes gleamed from the object and persisted in the database.

One immediate problem arises from this strategy; how do you load objects that were created before you made changes to your game? For example, game version 1 created an object with 3 fields and game version 2 created 4 fields. How do you differentiate the original version of the data with the newer version of the data? By storing an evaluating an explicit version identifier.

The second way you might use versions is to make your definition files easier. Since objects can take many different forms, it might be easier to say this form is version 1, this other form is version 2, and so on.

The version number is part of the IEODBClass interface, and is assigned when you load an object or prior to saving it, such as:

    public void DBSave() {
        this.Version = 2;
        new EODBDescriptor(this);
    }

The Object Header

Every object in the database has a header. The header contains 4 attributes:

  • OID is the unique identifier for the object. (4-bytes)
  • Version is the unique version id for the object. (1-byte)
  • Size is the number of bytes this object needs to be stored in the database. (4-bytes)
  • Class Name is a string identifying the class of the object. (4+N bytes)

After these attributes are written out, the order you "Define" your class and the number of bytes you need for the object defines how much information is stored in the database for your object. This means that if you had a string with 25 characters in it, then changed it to 250 characters, then your object becomes 250 characters long (plus 4 bytes for the string length) and the space needed for the header. The database automatically handles where the object is stored, allowing it to grow and shrink over time without wasting space by objects that change size over time.

Depending on your needs, you may need to know an object's version to load it properly. You will most likely use the class name in in combination with a Factory when you have indexes that have OIDs from many different objects. 99% of the time you won't need Version or Class Name most of the time. They exist so that there is a standard way to deal with objects.

You can retrieve the object header using your database:

// returns null if -10 is not a known OID in the database.
EODBClassHeader myHeader = myDatabase.GetHeader( -10 );   // where -10 is the OID

Definition Classes

Any class in your solution can be used to read/write from a database. All you have to do is implement the IEODBClass interface:

public interface IEODBClass {
    EODB Database { get; }
    int OID { get; set; }
    byte Version { get; set; }
    string ClassName();
    void Definition(EODBDescriptor aDescriptor);
    void DBSave();
}
  • Database is the database where this object was loaded from or where it will be saved to. In almost all cases, the Database is statically defined and you simply write a facade to reference it:
      public EODB Database { get { return GlobalStatic.EODBDatabase; } }
    
    Otherwise, you will need to assign Database in your object's constructor or if you are implementing this with MonoBehaviour with your script's Awake() logic, and have some way to set it by declaring the Database with a setter:
      public EODB Database { get; set; }
    
  • OID is the unique identifier for the class.
  • Version is the version of the object. This is used to handle variations in objects that are from databases created after they have been updated.
  • ClassName() is a method that gives a hard coded class name. It doesn't have the letter for letter name of the object. It can be an abbreviation. That might be an abbreviation in order save space since it is stored for each object. In most cases, the Class Name is used when extracting from an index that has objects from more than one class and you need a Factory Method to figure out which constructor to use for a given OID.
  • Definition() is a method that define the order and, by implication, the structure of how an object is written to the database and loaded from the database. The next section talks about this method in more detail.
  • DBSave() declares a method that can save the object.

Class Definitions

When writing your object to the database, EODB stores binary data without any header information. For example writing two floats (4 bytes each) will result in 8 bytes stored in the database. If you subsequently read those 8 bytes as a long instead of 2 floats, you will get completely erroneous data. To ensure that you are reading the attributes in the same order you wrote them, there is a single method in a class implementing IEODBClass, called Definition(). This method defines the order that fields are identified and is called once for a save and a second time for a load.

The Defintion() method accepts a EODBDescriptor, and you in turn call one of it's four methods:

  • Structure() is called to read/write attribute data such as primitive types, structures and simple classes. For structures and classes, these must resolve down to one of the known types defined in EODBTypeIO. This class is described further below section.
  • Index() is called to write out attribute data or derived data to one or more indexes. The only requirement is that the data you sent to the index is the same type of data the index expects. If it is different, it throws an exception.
  • List<T>() is called when you have a attribute data stored within an IEnumerable<T> such as a IList<T>, List<T>, T[], etc.
  • KeyPair<T>() is called when you have attribute data stored within an IEnumerable<KeyValuePair<TKey,TValue>> such as SortedList<TKey,TValue> or Dictionary<TKey,TValue>.

Structure

Structure is used to define attribute level data and what it defines must be a known type, as defined by the "Type Support" section above. Using the Structure is generally fairly easy, it consists of a GETTER function that returns data and a SETTER action that accepts some data. The type of data has to match:

public void Structure<T>(Func<T> aGetter, Action<T> aSetter) { .. }

It's convenient to use LAMBDA and an anonymous delegate for the getter and setter:

aDescriptor.Structure(()=>CharacterPosition,W=>CharacterPosition = W);

This calls structure, specifying the value to write out, and how to get the data back into the object during a read.

This call implies that you can only read and write from local variables. But this is not the case. Any attribute value is completely acceptable, such as:

aDescriptor.Structure( ()=>gameObject.transform.position , A=>gameObject.transform.position = A);
aDescriptor.Structure( ()=>gameObject.transform.rotation , A=>gameObject.transform.rotation = A);
aDescriptor.Structure( ()=>gameObject.GetComponent<MeshRenderer>().materials[0].color 
                     ,  A=>gameObject.GetComponent<MeshRenderer>().materials[0].color = A);

The only requirement for what is acceptable are the types that you declare in EODBTypeIO.Initialize().

While I am using LAMBDA here, you can put in your own methods, to do more complicated conversions:

aDescriptor.Structure(GetData,SetData);

Index

Index is used to define attribute level data that can be included into a index. As mentioned earlier, you can index any kind of data that is a known type and that type doesn't have to be the same as the data it's referencing. Ultimately you are using an index to search for data, therefore, whatever format simplifies you subsequent searching is acceptable:

public void Index<T>(string aIndexIdentifier, Func<T> aData) { ... }

Any value you want can be placed into the index, and whatever value you specify becomes associated with your OID. For example, this is valid:

aDescriptor.Index("name", ()=>Last + ", " +First);

This places the concatenation of "Last, First" into the index "name" is associated with whatever your object's OID is.

In some cases, you may want to explicitly remove an object from the index. You do so with either of these two calls:

aDescriptor.Index("name" , ()=>null);
aDescriptor.Index("name" , null);

Collections

This method allows you to read and write ICollection<T>, enabling saving and loading of:

  • List<T>
  • HashSet<T>
  • Stack<T>
  • Queue<T>
  • Dictionary<TKEY,TVALUE>
  • SortedList<TKEY,TVALUE>
  • Others

See the Tutorials for videos on using Collections with List<> and Dictionary<>

Note: See Arrays below for saving Array[] objects.

Collections work on individual elements of the supplied subject and function similarly to IEODBClass.Definition, but with extra parameters. During a save, your aDefintion Action<> is called multiple times for each instance in the Subject. During a load, you add values to aSubject.

public void Collection<T>(Action<EODBDescriptor, ICollection<T>, T> aDefinition, 
                          ICollection<T> aSubject) {

During a read a default value is passed to aSubject, but is ignored. Instead, you describe the attributes you need to read for one T and exit. aDefinition is called each time it encounters an entry in the file. You don't need to worry about how many you get, just your logic needs to handle the loading of several individual ones.

Handling Array[] of a Class

For every main type that EODB supports, it also supports an Array equivalent of that type.

You need to use Array<T>() only when saving and loading arrays when working with object instances. Working with primitive types and structures (i.e. Vector3) is much easier, because you can use .Structure().

You need to use Array<T>() to save and load arrays of class instances, however, to set the context as to where to add the object, you need an additional method to identify how to add a new T. This extra method provides an additional feature, it allows you to construct an object that contains the context in which the Add takes place. This allows you to handle multi-level hierarchies. Note: This add method also exists on an alternate form of the Collection() method.

public void Definition(EODBDescriptor aDescriptor) {
    aDescriptor.Structure(()=>Name,A=>Name=A);
    aDescriptor.Structure(()=>Position,A=>Position=A);

    if ( aDescriptor.isLoading )
        GamerFoods = new Food[0];

    aDescriptor.Array(FoodDefinition,GamerFoods,AddFood);
}

private void FoodDefinition(EODBDescriptor aDescriptor, Food[] aGamerFoods, Food aFood, Action<Food> aAddFood) {

    string THE_NAME = "";
    int THE_COUNT = 0;

    aDescriptor.Structure(()=>aFood.InventoryName,NAME=>THE_NAME = NAME);
    aDescriptor.Structure(()=>aFood.InventoryCount,COUNT=>THE_COUNT = COUNT);

    if (aDescriptor.isLoading) {
        aAddFood(new Food(THE_NAME, THE_COUNT));
    }
}

public void AddFood(Food aFood) {
    Array.Resize(ref GamerFoods, GamerFoods.Length + 1);
    GamerFoods[GamerFoods.Length - 1] = aFood;
}