Elenesski Object Database  EODB and NEODB
The Memento Pattern Database for Unity
31 Network - Usage Guide

NEODB Is still under development (Jan 2016), at the moment we are developing a sample game to show how to use NEODB to create a game without a dedicated server.

NEODB Features

These are some of the significant features of NEODB:

  • Uses a "push pull" architecture that is easy to implement for APP developers. No RPC calls.
  • Uses Unity's WWWForm mechanism for data exchange which supports up to 16Mb messages.
    • By default, and by our recommendation, you should keep your data light weight and not create objects needing more than 64K. (This can be changed, of course.)
  • Simple data architecture for storing data on an Internet server.
  • Greatly simplified object declarations using by inheriting from NEODBClass.
  • Handle complex but sequenced asynchronous calls with NEODBChain.
    • NEODBChain supports hierarchical chains so that chains can contain sub-chains.
  • Supports paged and limited search returns, returning only a limited number of rows starting from a particular point.
  • Supports returns of the indexed data only. See "31 Network - Understanding Latency" (examples) for an explanation and example of how effective indexed data searching can be.
  • Supports a variety of index find methods so that only one round-trip between the APP and server is required.
    • Supports the ability pull multiple rows from the remote database with a single round-trip.
    • Implements most of the EODB feature set, but extends EODB to make network efficient searching possible with 12 adaptive search algorithms (offering 12 combinations each).
  • Specifically coded to prevent SQL injection attacks so that you can get search criteria from the user and not compromise your database security.
  • Implements a PHP server which can be implemented with Linux or Windows (IIS 8) servers.
    • It is strongly recommended that you implement your PHP Server on a host that supports SSL and HTTPS, because NEODB offers no protection against "man-in-the-middle" attacks.
  • Uses mySQL which is the most widely available open source database.
  • Supports multiple open network databases at a time.
    • Supports multiple connections against the same database at the same time.
    • NEODB uses EODB which provides you with both a networked and a local database solution that uses nearly the same calling infrastructure.
  • NEODB was written to work in shared hosting environments, but it will also work in enterprise-grade server farm environments like AWS or Azure where there are multiple threads updating the database.
  • Source provided, so that you can see how things work and make changes if you want.

133 Different Ways to Search and Retreive Objects

NEODB offers 133 different ways to find then process your data.

  • Search by class name
  • 11 core ways to search an index, which each method providing 12 combinations each of:
    • Equal, Not Equal, Less Than, Less Than or Equal, Greater Than, Greater Than or Equal, Between, Not Between, Like (wildcard), Not Like, All
    • 2 different retrieval techniques (return the original objects or return just the indexed data)
      • When pulling index only data you provide your own action to receive the indexed data and process it.
    • 3 different sorting options (none, ascending, descending)
    • 2 ways to limit how much data you get:
      • unlimited rows
      • paged data (the start and limited N rows)
        • For example, with paging, it's possible to get rows 51 to 60 in a result set with a single call. This can dramatically improve your app's network efficiency by reducing how much data you get with a call.

Data Push Pull

NEODB implements a "data push pull" architecture. This means that each APP (game or application) PUSHes the data they want to the SERVER (the PHP Server/database), and other APPs then PULLs the data down into their application. This architecture has it's strengths and weaknesses:

STRENGTHS

  • Usually does not require special firewall settings for APPs to access the internet. It is only exchanging data, there are no remote procedure calls between APPs or SERVERs, so no complex firewall rules.
  • It's the easiest network architecture to code because the APP is only exchanging data.
  • There are no complicated data needs, NEODB can support almost any format the way EODB can.
  • NEODB can implement exclusive locking at a session level. APPs can lock objects to prevents others from overwriting your data.
  • NEODB supports 133 different search methods which all support paged data with a single round trip (assuming you code it that way).
  • Does not use CCU (Concurrent Users) because your game process runs on the client not on a remote server.
    • CCU-enabled servers are better architecture for real-time games.
    • NEODB is a better architecture for near real-time games or when when you want to store information about a player's successes in a largely single-player game type. For example, top scores, external storage or sequenced games where only periodic updating is required.
      • NEODB can handle near real-time transactions with updates at least 2 seconds or more. WARNING: Having too much I/O out of your ISP's data center may result in extra fees. Therefore it is strongly recommended that you carefully study how much I/O your game will need and make sure you don't get caught with a bill for extra I/O if your game becomes popular.

WEAKNESSES

  • More network and database I/O is required because you need to push and save, then load data to pull. This may impact your server costs if you use too much bandwidth than your ISP is expecting you to use.
  • Care must be taken writing data to the database, because it is very easy to overwrite somebody else's changes if you don't implement good data concurrency.
    • NEODB does offer a permanent exclusive locking solution that prevents updates, deletes or clear locking commands against objects that don't have the right Lock ID.

Security

NEODB does not provide security between the app and the server, and instead relies on it's users to provide it's own transport security; usually SSL on a server to support HTTPS. See the page "30 Network Architecture" in the main menu for additional information.

NEODB does provide application level security with a method to lock rows. Once a lock is issued, the only way to update, delete or unlock the row is to provide the correct lock ID. NEODB allows an application developer to come up with their own scheme for identifying LOCK IDs.

General Security Advisory

NEODB does not encrypt the data it sends from the app to the server. It's is highly recommended that you use a server that supports SSL so that you can make calls to your server using HTTPS. Without SSL/HTTPS, any hacker betweeen you and the server can see the information you are sending. See the section "30 Network - Architecture", in the main menu, for additional information.

NEODB does NOT provide any protection for Impersonation. In fact, every app is distributed to 3rd parties is subject to impersonation. The best defence against this kind of attack is to not store private, important, confidential, sensitive or secret information in your database or anything else that would make your environment a target for hackers and remember to back up your environment frequently.

Exclusive Locks

NEODB provides the ability to implement exclusive locks. A "lock id" can be added to any record in the system, and remains in place until the record is "unlocked". If a Lock is specified on a record, then only updates, deletes, or lock clears on an object work ONLY if the LOCK ID you specify matches the lock id on the object.

In most cases, the way to implement locks is to tie it to a username/password object, and use the OID of the username as the LOCK ID. This way when a username is created only that username can be updated by the user that created it, and everybody is locked out from changes to it.

NEODB Server Data Format

Data for NEODB is stored on the remote server in a mySQL database and the server itself does not maintain any session states. The main reason for this is to manage the amount of memory your remote server needs to process requests and to allow you to reimplement your NEODB on a server farm where multiple database threads don't have to be kept in sync. This allows you to scale NEODB on such platforms as AWS or Azure without have to worry about which thread your requests get sent to on the farm.

The Database consists of these 2 key tables, plus one table per index that you define (see below).

NEODB_OBJECT_DATA contains all of your objects. It consists of two columns:

  • OID - The unique identifier for your object. This is indexed to enable indexed database loads.
  • LOCK_ID - When non-zero, identifies the ID that has an exclusive lock on this data. LOCKs can be exclusively added and removed but do not expire.
  • CLASS - This is the name of the class for your object. (Note: Class is included in DATA, so this is a duplication, however, for searching it simplifies the way data is extracted from the database.)
  • DATA - This is the data you send to NEODB. By default, this will be a base64 encoded memory stream that is created by EODB by the APP. By default, the maximum number of characters is 65,536, but it is encoded in base 64, so the practical limit is about 49000 characters. It's recommended that you avoid large objects as this will dramatically increase your network and database I/O, and is a quick way to exceed your ISP's resource allotments.
  • UPDATE_DTTM - This is the date/time in which the data was last updated. It allows you to run clean up old data, such as, deleting user accounts that have not been active for a certain time. Time is measured in seconds since Jan 1, 1970.

NEODB_NEW_OID contains a single row which is the last OID created. It's incremented by the count passed to the ServerGetNewOID() method; default is 1. You must assign an OID to your object before calling your first SAVE otherwise you'll get an exception. You can still use negative numbers to indicate static data but for dynamic data the OID must be assigned before you save the data for the first time. Note: PHP issues a FOR UPDATE clause on the select allowing high performance database servers to have multiple concurrent threads requesting OIDs. FOR UPDATE prevents the same OID from being reissued when two threads request OIDs at the same time.

  • OID - The largest OID ever assigned in the database. Once used it's never reused. Default to 0 on database creation.

NEODB Exceptions

NEODB uses exceptions to communicate to your APP about the failure of your NEODB request:

// Thrown when NEODB does support a feature of EODB in NEODB.  Usually there is a Server or Local equivalent of it
NEODBNotSupportedException

// General exception thrown when the request fails.
NEODBGeneralException

// Thrown when the request server request was too old.
NEODBSessionExpired

// Thrown when the request sequence is different than expected.  This could be an internal error or a hijacking attempt.
NEODBSequenceError

// Thrown when lock setting or clearing doesn't work
NEODBLockError

NEODB Configuration

// Standard connection string details for the mySQL database
$NEODB_HOST     = "127.0.0.1:3306";     // HOST IP and PORT.  Can be a DNS name too.
$NEODB_DBNAME   = "database_name";      // NAME of the database within the mySQL Instance
$NEODB_USER     = "database_user";      // USER with read/write access to the database
$NEODB_PASSWORD = "!!Password!!";       // PASSWORD of the user

// Indexes; see documentation ... consists of a name and a column definition
$NEODB_INDEXES = array(
    array ("NAME","varchar(25)"),
    array ("LEADER","int")
);

Multiple Database Connections

Normally you can only make one server call at a time. This is why we created NEODBChain. There are situations where more than one database call has to be made at the same time. To solve this problem, you simply use multiple NEODB object instances and connect them to the same PHP server.

Asynchronous vs. Synchronous

In order to be successful with NEODB, you need to wrap your head around the asynchronous nature of the calls. Failing to do this will result in code that does not work the way you expect it to. The biggest difference is that when you call the server asynchronously, control returns to you immediately and the actual server response will be seconds later through success or failure method calls.

In the local database version EODB, when you make a call to the database to perform some task, that call goes away, does some work, and returns with an answer. Your code will wait until the database has finished.

However, when using the network version of NEODB, when you make a call to the database to perform some task, that call returns right away. While NEODB is contacting the database, your application will need to perform some sort of task to let the user know a request is being processed. This could be a spinning busy icon or perhaps you can do work on the canvas while it waits for an update to take place.

In the background, NEODB calls the server, and a few moments to 1 minute later, it gets a reply. The reply is either successful or a failure. We hope that every call will be a success, however, networks can be finicky and fail for reasons having nothing to do with us. As a result, your APP will need to deal with several types of failure for every call it makes. Because any one of these possible reasons could occur, and they can occur at any time:

  • The server isn't configured properly.
  • The server isn't available due to some ISP problem.
  • The server isn't available due to some Internet problem.
  • The session the APP was using timed out.
  • The session the APP was using had a sequencing error.
  • The request you made didn't return the result you were expecting.

As a result, you must be prepared to handle all of these possible errors in your code.

Success and Failure Callbacks

Fortunately, NEODB makes it easy to handle asynchronous calls with callbacks. Every NEODB method that involves the server has at least two method references, one for when a successful call completes and the other for failure. Either of the two is called when the SERVER finally responds with information.

_TheNetwork.ServerGetToken(GotToken,NetworkFailed);

private void GotToken() {
    // Called when we successfully got a token from the database
}

private void NetworkFailed(Exception aException) {
    // Called when our call failed.  aException contains information about the failure.
    // We have decided to not intercept the exception, and just throw it.
    throw aException;
}

When working with classes that implement IEODBClass, you will need to construct a EODBDescriptor with additional parameters to identify the success method and failure method. Failure to do this will cause an exception to be thrown, expecting the Sucess and Failure methods to be identified. The only difference between the two are the constructors, which defines a

new NEODBDescriptor(aOID, this, false, SuccessfulLoad, FailedLoad);

private void SuccessfulLoad() {...} 

private void FailedLoad(Exception aException) {...}

Dynamic Indexes

Dynamic Indexes operate the same as regular indexes with three exceptions:

  • Dynamic Indexes can only be string based indexes.
  • Dynamic Indexes must be named starting with "DYN_" (including underscore).
  • When deleting a dynamic index, all the data that was created for the index is deleted as well.

Dynamic Indexes are used for situations where the data is temporary, such as events. For example, in the Online Poker Example, the host creates events to communicate with players for a given hand. When the hand concludes, those events are deleted and new set of events are created for the next hand. This allows players to enter the game, or crash and reload, without the host having to manage the sync'ing of new players.

Intelligent Indexes

NEODB offers the following FIND methods for finding data on an index:

  • ServerFindAll, ServerFindEquals, ServerFindNotEquals, ServerFindLessThan, ServerFindLessThanOrEqual, ServerFindGreaterThan, ServerFindGreaterThanOrEqual, ServerFindBetween, ServerFindNotBetween, ServerFindLike and ServerFindNotLike

While these find capabilities are extensive and do cover several a lot of searching needs, if you find you need to search, filter, then search again, resulting in more than one trip to the server, you will probably need to create an Intelligent Index.

Creating Indexes with Intelligent Design

Intelligent Indexes serve a specific purpose; to enable a single trip to the server to find information for your game. Basically you want to create values in an index that would allow you to get exactly the objects you need for your game by using a "ServerFindEquals" call.

The way it works is:

  • Determine the condition that you would need to execute later on in your game.
  • Devise a string value that can be calculated when you save your object, then store that value in a new index.
  • If an object does not match the string value, don't store it.
    • If don't want to store an object in an index, simply pass a "null" to the index.

Using this strategy, you can now use the index to get only the objects that meet your search criteria without having to "select then filter".

  • Select some objects from NEODB.
  • Locally scan the retrieved objects for items of interest.

The problem with a post selection filter is the time it takes to find and transport all the objects your initial selection takes. Assuming each object is 1K, If you have to pull out 150 objects to display 3, that means 147 objects have to be selected and transported to the app, only to be thrown away. Clearly if you could select only the 3 you needed,

Architectural Best Practice - Façades and Decoupling

This is an important architectural strategy when implementing NEODB. Many junior developers make a simple mistake when developing that causes a lot of work/pain later, that could have been avoided. They choose a technology like NEODB and then begin to integrate the technology directly into their code. This approach is a bad idea, and this section is to discuss why this is a bad idea and offer about a better approach.

This approach should be adopted for all libraries you buy and plan on using!

Coupling

While NEODB is a great environment for storing your objects in a remote database, it has some limitations. It's biggest limitation is that pushing and pulling data from a database is time consuming and potentially expensive if you have a lot of people doing it, and it may become apparent that a different networking solution is required, such as Unity's Networking package.

By directly interfacing your code into NEODB, means that if you make the decision to replace NEODB with something else, every place you wrote code that used NEODB will now need to be rewritten and replaced. This can cause a huge maintenance headache as there may be dozens or hundreds of places where this integration took place. The maintenance is created because you "coupled" your code to the library. For more information on coupling, see Wikipedia Coupling (link).

A better strategy is to decouple the NEODB from your code using a Façade.

Façades

Façade is a GoF pattern that creates an interface between two systems, or in the case, between NEODB and your code. For more detailed information see Wikipedia Facade (link).

The façade, implemented as a new class, allows you to create methods that describe the behaviour of what you want NEODB to do, then create methods that implement this behaviour. While it may see a bit of overkill to use a new class to interface to NEODB, but it will save you hours of work later if you need to replace you did something with NEODB or ultimately have to replace NEODB (or EODB) later with something else.

It can even save you time as you develop, when you want to use the same behaviour in more than one place in your code.

It's a better strategy too when attempting to revise objects, fixing bugs, etc. In a job many years ago, the developers did none of this and simple changes which should have taken 10-20 minutes took days because there were hundreds of places where the same code was implemented, that had to be changed and retested.

Essentially what it allows you to do is see, in a single place, all the things you wanted your networked database to do, and replace the code only in one place. It's still maintenance, but you've contained the maintenance to a single class which is means you know your code has been upgraded to use the new environment and it's easier to test the changes.