For example, in a customer management database sales and returns are recorded. For some analytics it is necessary to compute the total sales for a given customer. Calculated properties are used to add a “total_sales” property to each customer, adding up all sales, and subtracting from it all returns by that customer. After the “total_sales” property is established, for each customer the total sales is known.

Calculated Property Sharing

In a situation where several analysts are working in an enterprise, each of them might need the “total_sales” property for the specific analysis that needs to be done.

The worst approach would be if each analyst creates his own definition of “total_sales”. The reason that this a sub-optimal solution is that some might subtract the returns from the “total_sales”, others might not. While “total_sales” here is kept simple, in a real situation it might be a lot more complex, easily leading to disagreement between the analysts of how to properly define it.

An approach that is a lot better is to define the calculated property “total_sales” once, and share this definition with all analysts. In this case, all follow the same definition and all will have the correctly computed value for each of the customers.

Phased Analytics

Even if the definition of a calculated property is shared, each analyst has to apply it. This takes computational effort. Furthermore, analysts might apply it at different times against the base document set. In this case, each analyst might see a different value of “total_sales” for the same customer, depending on how much buying and returning activity is taking place.

While all analysts share the correct definition, they do not share necessarily the same base document set. In order to overcome this problem, it is possible to share a common state of the analytics data. In a first phase those calculated properties are executed that are important for more than one analyst. Once this happened, this augmented document set is then shared with the analysts in a second phase.

In this case, all analysts see the same set of customers and all customers have the “total_sales” property added to them. Now the analysts work with the same document set where the common calculated properties are present already.

While the example has two phases, more phases are possible, of course. In each phase a set of calculated properties is computed and the result is made available.

Data Staging

For efficiency reasons it might be necessary to materialize the calculated properties once they are computed in a given phase. The materialization writes the collections of documents (including the calculated properties) into a staging database. Analytica provides the command “saveas()” for this. Each phase has a corresponding staging database.

While from an IT perspective this reduces the repeated computation of calculated properties, from an analytics perspective it looks as if the “right” base documents are available. Analysts now access data as if they are base document sets.

This approach provides IT with a mechanism to prepare data for analysis, while for analysts the complexity of data preparation is reduced and this frees them up to the more involved analytics.

Analytica provides several functions that can be of help. One function is order(); this function establishes an order amongst a set of documents (documents are processed in natural order otherwise). A second function is row(); this function determines the index of a document. A third function is index(); this function supports index-based access to document.

Example Document Set

Sensor data store measurements of sensors. A possible sensor is the temperature at a location. The following document set shows a few sensor measurements. Assume that these are stored in a database with name ‘sensor’, and in a collection with name ‘data’:

{"sensor":1, "temp":50}
{"sensor":1, "temp":55}
{"sensor":1, "temp":65}
{"sensor":1, "temp":90}

Example Analytics Rules

The goal of the analysis is, to have each sensor data document have its value and the difference to the next value. The last value’s difference should be ‘null’.

First, an index property is added to each document:

set(sensor.data.index, 
    row())

Then, the temperature of the next document in order is added:

set(sensor.data.nexttemp, 
    propertyvalue("temp", index(index + 1, sensor.data)))

Finally, the difference is computed for those documents where ‘nexttemp’ has a value:

set(sensor.data.tempdiff, 
    xlif(nexttemp = "n/a", null, temp - nexttemp))

Result

After the calculated properties are established, the result document set looks like the following:

{"sensor":1,"temp":50,"index":1,"nexttemp":55,"tempdiff":-5}
{"sensor":1,"temp":55,"index":2,"nexttemp":65,"tempdiff":-10}
{"sensor":1,"temp":65,"index":3,"nexttemp":90,"tempdiff":-25}
{"sensor":1,"temp":90,"index":4,"nexttemp":"n/a","tempdiff":null}

Summary

Analytica supports the computation of values based on an ordered document set. The example has shown how to accomplish computing the regular values as well as a corner case. While the example has shown a basic approach, more complex scenarios can be built on this foundation.

• omit the property
• set the property value to “null”

How can documents be transformed so that they have actually a value? The following shows Analytica’s approach for those cases where this approach is required.

Test Document Set

The following dialog establishes the test document set:

>mongo
MongoDB shell version: 2.2.0
connecting to: test
> use nullValues
switched to db nullValues
> db.nv.save({"a": null})
> db.nv.save({"a":1})
> db.nv.save({"c":2})
> db.nv.find()
{ "_id" : ObjectId("51096c45c905109d93d892d4"), "a" : null }
{ "_id" : ObjectId("51096c4bc905109d93d892d5"), "a" : 1 }
{ "_id" : ObjectId("51096d2ac905109d93d892d6"), "c" : 2 }
>

The collection “nv” has three documents. Two containing a property “a”, whereby one of those has given “a” the value of “null”.

Transformed Schema

The goal is to add a calculated property “b” that has the value of “a” if a is not null, otherwise zero (0). The schema transformation therefore adds a property “b” that represents the transformed schema.

The calculated property “b” is defined as follows (Analytica Shell syntax):

-->set nullValues.nv.b = xlif(a=null, 0, a)

This expression adds a property “b” to every document that has either the value zero (0) or the original value of “a”.

But what abot the case where the document does not have “a” at all? Analytica interprets the absence of a property as “null”, so the documents missing the property altogether are actually included.

The documents look now like this in the Analytica server (Analytica Shell syntax) :

-->get nullValues.nv
{
    "nv" : [
        {
            "_id" : "51096c45c905109d93d892d4",
            "a" : null,
            "b" : 0
        },
        {
            "_id" : "51096c4bc905109d93d892d5",
            "a" : 1,
            "b" : 1
        },
        {
            "_id" : "51096d2ac905109d93d892d6",
            "c" : 2,
            "b" : 0
        }]
}
-->

Benefit of Schema Transformation

Effectively, the above approach makes the schema more uniform across documents. This is the classical space – time trade-off.

The additional property requires storage space, but makes any analytics expressions easier since those do not have to distinguish if a property has a value or is “null”. Otherwise, all analytics expressions would have to include the if-then-else clause. The above approach has the if-then-else once in the calculated property, and from then on forward all analytics expressions can assume a valid value.

“Traditional Analytics” follows the query – result approach. For example, a query is formulated determining an aggregation over a subset of the base data to be analyzed, and the system returns the requested aggregation as a query result.

The client program that issues the query receives the result and will further process it. If the client needs several aggregations, then the client has to issue several queries and will receive several results in return.

Other clients, that might need the same aggregations or a subset of those, also have to issue queries and process the returning results.

What’s Wrong With That?

What’s wrong with clients running queries to obtain results? From a naive viewpoint, there is nothing wrong with that as this is what clients do. However, from an analytics viewpoint there are several aspects wrong with that approach:

• Result Consistency. Clients running the same queries at different times might obtain different results. Unless a snapshot semantics is implemented that ensures that all clients see the same state of the data set, inconsistency is unavoidable.
• Conceptual Consistency. If several clients needs the same aggregations, they might implement those in their own queries. It might be that all these queries are actually semantically equivalent, but they might not be. If as part of a query a concept (like revenue) has to be computed, chances are that different clients implement that concept of ‘revenue’ differently depending on their knowledge of this concept.
• Query Reuse. In traditional systems it is not possible to structure queries into reusable queries or reusable parts of queries. If the same query has to be reused, it is done by code duplication leading to change management problems.
• Query Management and Supervision. Query systems do not have an inventory of the queries being executed and consequently it is impossible to manage and to supervise the query definition change and execution.

In short, the organization following this approach relies on the clients (aka software engineers or data scientists) to do it right. The system itself has no means to support a consistent query definition, management and supervision.

Analytica’s Approach

Analytica is addressing all the issues within its approach and implementation in a structured and conceptually clear way.

Result Consistency

Analytica ensures result consistency by its analytics snapshot semantics. The base data set that is being analyzed is taken as a ‘snapshot’ and all analytics is executed against this snapshot. So no matter how many clients are accessing the Analytica server, they all see the same data set in the same state.

The snapshot can be reset in the sense that a new snapshot is taken. From that point on forward all client requests operate on the new snapshot.

Conceptual Consistency

In Analytica the concept of a query does not exist. Analytica’s approach is to augment and to extend the conceptual model the base data set presents with additional concepts. This mechanism is called ‘calculated properties’. These calculated properties are added to the base data set as if they are part of it from the beginning.

For example, an aggregation is a new concept in the base data set. Clients that need the aggregation will access this additional property. Therefore, all clients use the same definition of that aggregation as well as the same value.

Calculated properties ensure conceptual consistency.

Query Reuse

Calculated properties are defined by a computation. This computation says how its value is computed and after the computation the value is stored. Whenever a client accesses a calculated property either the value is returned if it exists already, or it is being computed the first time on the snapshot.

Reuse is implemented by the ability of a computation of one calculated property being able to refer to another calculated property. Therefore, concepts that are not in the base data set can be added one-by-one and reused in order to derive to those properties that finally make up the analytics result.

Query Management and Supervision

Calculated properties are artifacts supported by Analytica. As such they are explicitly defined at the interface of Analytica. Internally they are explicitly represented and managed.

As a consequence, it is possible to list all calculated properties. This allows the analytics team and its engineers and data scientists to examine the existing definitions at any time.

Since all calculated properties are explicit, their dependencies are determined by Analytica and they are executed in that dependency order. So clients do not have to manage the definitions and their correct execution order themselves.

Change management is applied automatically. When a calculated property is changed, Analytica will determine if any reuse dependencies are broken and will message this to the engineer or data scientist.

Summary

Based on the novel approach to analytics by Analytica, all of the issues and problems around traditional query – result approaches have been resolved and addressed in a consistent manner.

Emphasis is given to conceptual and data consistency so that organizations using Analytica can derive to a better and consistent analytics of their base data sets.

Analytics operations access the transactional data set to retrieve the required data. Once the data is required, the analytics operations take place. A slightly different mode is that the analytics operations are taking place interleaved with the retrieval of the transactional data.

Analytics Result Inconsistency

Analytics operations might take longer than the time of transactional data update and there might be several analytics operations necessary to execute for a desired report. In principle, if nothing special is done, each of the analytics operations see a different state of the transactional data set. This in turn means that they can be inconsistent wrt. each other.

As example, if one analytics operation counts the number of on-line users (users logged in), and another analytics operations adds the total time the currently on-line user spent on-line, then these two numbers will be inconsistent in a highly active web site as the second analytics operation might add times from users that the first one did not see.

Analytica Snaphots

Analytica addresses inconsistency explicitly by implementing a snapshot semantics from the analytics viewpoint. Analytica fetches the required transactional data before starting any analytics operation. This means that all analytics operations are taking place on a fixed and non-changing transactional data set.

This means that no matter how many analytics operations are taking place, all share the same version or status of the transactional data set and so the results are consistent with each other (given that the analytics rules are correct).

Snapshot Refresh

With a snapshot semantics the problem of data staleness appears. This is addressed by being able to tell Analytica to refresh the transactional data set and to re-run the analytics operations on the newly acquired transactional data set. (The REST API for that is called executeallsetstatements()).

Life Cycle

With the ability to snapshot the transactional data set and refresh it at any time, customers now can implement their required analytics lifecycle. For example,

• A customer can refresh his analysis every 2 hours
• A customer can refresh his analysis every morning at 6am for all employees starting their workday to see fresh, and also the same data
• A customer can provide an environment where end users can refresh their own data, but not more than once an hour

These are only a few examples how he ability to snapshot and refresh provides a customizable analytics experience while preserving data consistency.

Cool and Interesting Technology

From a distributed computing viewpoint, REST is cool technology due to the programming language and platform independence, paired with the availability on most platforms. While this is not specifically relevant for Analytica per se, it is important from an architecture and infrastructure perspective.

Integration into Existing Infrastructure

REST, because of its availability, it a good mechanism for integrating analytics functionality into any existing infrastructure. Analytica does not force a separate technology stack to be created, but is designed to be integrated into an existing infrastructure. While Analytica provides end user clients, they are not the only avenue to access analytics functionality. REST is equally powerful and so an explicit decision can be made to integrate Analytica into existing infrastructure.

Programming Language Independence

Most current programming languages support REST invocations (over the HTTP protocol). Plus, the data structures are implemented in JSON, which is an ASCII representation. Based on this, Analytica can be used from any programming language and does not force developers to learn a new programming language at all when using Analytica.

Multiple Consistent User Interfaces

The REST API provides not only programming language independence, but also end user client independence. Analytica itself provides several clients (Excel, Shell, iPhone App). Since all these end user clients are based on the REST API, their functionality is consistent across these user interfaces. Furthermore, customers of Analytica can build their own custom user interface clients without running the risk of inconsistencies or missing out on functionality.

Existing User Interface Integration

Many companies have already different forms of dash boards and analytics interfaces. With Analytica in the mix to support the NoSQL database native access, the companies can integrate Analytica functionality into their existing user interface infrastructure making the analytics functionality seamless to their end users. REST is a great way to accomplish this integration.

Single Data Representation Paradigm

Analytica operates natively on the JSON representation of document-oriented databases. In addition, the result of analytics rules is itself represented in JSON. The REST API returns all results as JSON documents (without exception). This means that Analytica provides a consistent data representation paradigm and the continued processing of analytics results in JSON is guaranteed.

Development Assurance and Efficiency

Last, but not least, a REST API makes the interactions and the data transfer (both invocation and result) explicit. This supports the development process greatly as during development and debugging it is possible to explicitly look at the communication patterns and details.

From the perspective of analysis, however, only a subset of the documents of a collection might be needed and others are irrelevant. For example, when analyzing user accounts, only named user accounts might be interesting, but not anonymous user accounts.

Operating on all documents causes operational inefficiency if only a subset of the documents is relevant. From an analytics perspective, only relevant documents should be part of the analyzed data set. Document selection is required to increase efficiency. Not only that, if only the relevant documents are available, the analytics rules will become easier as they do not have to exclude the non-relevant ones.

Document Selection

Removing documents not needed for analysis is called document selection. Document selection means that only those documents are made available to analytics that are relevant for analysis. Those that are not relevant are not made available. Instead of using the documents of the base collections, selected documents are accessed.

An initial approach in Analytica is a two-step approach:

• For each base collection, create a calculated collection that only contains the relevant documents.
• Define the analysis on the calculated collections.

While in principle this approach is valid, it is inefficient as Analytica has to create and to maintain calculated collections to achieve the document selection.

A more suitable approach is Virtual Collections.

Virtual Collections

Virtual collections is a concept introduced by Analytica. Virtual collections are defined within Analytica and are available as if there were defined directly in the database (hence the name ‘virtual’ collection). However, virtual collections are not expressed as calculated properties; instead, they are expressed as native database queries.

For example, the following virtual collection selects named users from a collection containing user accounts (Analytica shell syntax):

setvc namedUsers AccountManagementDB uam "{NamedUser: true}"

The command ‘setvc’ has four parameters

1. Virtual collection name. The first parameter is the name of the virtual collection being created. In the above example this is ‘namedUsers’. Once this definition is added into the system, a collection called ‘namedUsers’ is available as if it were coming directly from the database.
2. Database name. The second parameter is the name of the database that contains base collections. And this is also the database that will contain the virtual collection. This is not a true containment as the virtual collection will not be stored back into the database, however, from a naming convention, the virtual collections belongs to that database.
3. Base collection name. The third parameter is the base collection from which the documents of the virtual collection are selected.
4. Native query. The last parameter is the native database query (using the query language of the database). This query is performed in order to select the documents for the virtual collection. In the above example, only those documents are selected where the property ‘NamedUser’ has the value true.

It is possible to define as many virtual collections as needed in Analytica; there is no restriction.

Operational Efficiency and Analysis Simplification

Virtual collections provide two major benefits:

• Operational efficiency. Operational efficiency is increased as only those documents are made visible to Analytica that are truly needed. The documents that are not needed will not be processed by Analytica at all.
• Analysis simplification. Since the documents are already selected by means of the virtual collection, the selection does not have to be made within the analytics expressions. This makes the expressions a lot simpler and easier to understand.

In summary, the benefits of virtual collections are significant and if a meaningful selection is possible, virtual collections are a perfect approach and methodology.

Special Operational Collections

In order to support users in accomplishing this, Analytica makes key operational data sets available as special collections within each database that it connects to. There are 3 special collections for each MongoDB database:

• __serverstats__
• __dbstats__
• __collstats__

‘__serverstats__’ exposes the status of the MongoDB server, ‘__dbstats__’ exposes the database status and ‘__collstats__’ exposes the status of the collections of the database.

The content of the collections is the result of the respective MongoDB commands that are exposed on the server, database and collection level. MongoDB does not make these data available as collections itself, however, in order to be able to use the data inside Analytica for analytics, Analytica makes them available. Analytica accesses the data sets programmatically and exposes them as the special collections.

Operational Collection Content

In the following an example for each of the three operational collections is given. The system is connected to a database ‘GD’ and the contents of the 3 collections is shown for ‘GD’.

Serverstats

The following is the content of ‘__serverstats__’:

-->get GD.__serverstats__
{
    "__serverstats__" : [
        {
            "host" : "THEFORCE",
            "version" : "2.2.0",
            "process" : "mongod",
            "pid" : 1612,
            "uptime" : 385506,
            "uptimeMillis" : 385505975,
            "uptimeEstimate" : 203152,
            "localTime" : "12/17/2012 12:00:00 AM",
            "locks" : [
                {
                    "dot" : [
                        {
                            "timeLockedMicros" : [
                                {
                                    "R" : 5972047,
                                    "W" : 73724293
                                }],
                            "timeAcquiringMicros" : [
                                {
                                    "R" : 15774116,
                                    "W" : 2729949
                                }]
                        }],
                    "GD" : [
                        {
                            "timeLockedMicros" : [
                                {
                                    "r" : 929843,
                                    "w" : 0
                                }],
                            "timeAcquiringMicros" : [
                                {
                                    "r" : 8610,
                                    "w" : 0
                                }]
                        }]
                }],
            "globalLock" : [
                {
                    "totalTime" : 385505975000,
                    "lockTime" : 73724293,
                    "currentQueue" : [
                        {
                            "total" : 0,
                            "readers" : 0,
                            "writers" : 0
                        }],
                    "activeClients" : [
                        {
                            "total" : 0,
                            "readers" : 0,
                            "writers" : 0
                        }]
                }],
            "mem" : [
                {
                    "bits" : 64,
                    "resident" : 44,
                    "virtual" : 14202,
                    "supported" : true,
                    "mapped" : 7023,
                    "mappedWithJournal" : 14046
                }],
            "connections" : [
                {
                    "current" : 1,
                    "available" : 19999
                }],
            "extra_info" : [
                {
                    "note" : "fields vary by platform",
                    "page_faults" : 665747,
                    "usagePageFileMB" : 215,
                    "totalPageFileMB" : 32720,
                    "availPageFileMB" : 26461,
                    "ramMB" : 16361
                }],
            "indexCounters" : [
                {
                    "note" : "not supported on this platform"
                }],
            "backgroundFlushing" : [
                {
                    "flushes" : 3414,
                    "total_ms" : 120661,
                    "average_ms" : 35.3429994141769,
                    "last_ms" : 10,
                    "last_finished" : "12/17/2012 12:00:00 AM"
                }],
            "cursors" : [
                {
                    "totalOpen" : 0,
                    "clientCursors_size" : 0,
                    "timedOut" : 0
                }],
            "network" : [
                {
                    "bytesIn" : 1903842,
                    "bytesOut" : 49941988,
                    "numRequests" : 22259
                }],
            "opcounters" : [
                {
                    "insert" : 1943,
                    "query" : 95048,
                    "update" : 17,
                    "delete" : 0,
                    "getmore" : 1702,
                    "command" : 9097
                }],
            "asserts" : [
                {
                    "regular" : 0,
                    "warning" : 3,
                    "msg" : 0,
                    "user" : 12,
                    "rollovers" : 0
                }],
            "writeBacksQueued" : false,
            "dur" : [
                {
                    "commits" : 30,
                    "journaledMB" : 0,
                    "writeToDataFilesMB" : 0,
                    "compression" : 0,
                    "commitsInWriteLock" : 0,
                    "earlyCommits" : 0,
                    "timeMs" : [
                        {
                            "dt" : 3060,
                            "prepLogBuffer" : 0,
                            "writeToJournal" : 0,
                            "writeToDataFiles" : 0,
                            "remapPrivateView" : 0
                        }]
                }],
            "recordStats" : [
                {
                    "accessesNotInMemory" : 2243,
                    "pageFaultExceptionsThrown" : 194,
                    "GD" : [
                        {
                            "accessesNotInMemory" : 15,
                            "pageFaultExceptionsThrown" : 14
                        }]
                }],
            "ok" : 1
        }]
}
-->

There is one property called “dot”. This property has been renamed by Analytica. Originally, the property name is “.”; however, MongoDB does not allow dots being part of property names; in order to make this available and allow users to define analytics on it, the “.” was renamed to “dot”.

Dbstats

The content of ‘__dbstats__’ is:

-->get GD.__dbstats__
{
    "__dbstats__" : [
        {
            "db" : "GD",
            "collections" : 3,
            "objects" : 10,
            "avgObjSize" : 624.4,
            "dataSize" : 6244,
            "storageSize" : 36864,
            "numExtents" : 3,
            "indexes" : 1,
            "indexSize" : 8176,
            "fileSize" : 67108864,
            "nsSizeMB" : 16,
            "ok" : 1
        }]
}
-->

Collstats

The content of ‘__collstats__’ is:

-->get GD.__collstats__
{
    "__collstats__" : [
        {
            "ns" : "GD.People",
            "count" : 6,
            "size" : 6084,
            "avgObjSize" : 1014,
            "storageSize" : 28672,
            "numExtents" : 1,
            "nindexes" : 1,
            "lastExtentSize" : 28672,
            "paddingFactor" : 1,
            "systemFlags" : 1,
            "userFlags" : 0,
            "totalIndexSize" : 8176,
            "indexSizes" : [
                {
                    "_id_" : 8176
                }],
            "ok" : 1
        },
        {
            "ns" : "GD.system.indexes",
            "count" : 1,
            "size" : 64,
            "avgObjSize" : 64,
            "storageSize" : 4096,
            "numExtents" : 1,
            "nindexes" : 0,
            "lastExtentSize" : 4096,
            "paddingFactor" : 1,
            "systemFlags" : 0,
            "userFlags" : 0,
            "totalIndexSize" : 0,
            "indexSizes" : [],
            "ok" : 1
        }]
}
-->

Operational Analytics

As it turns out, each of the 3 operational collections has one document each. And because Analytica implements the customary snapshot semantics in context of analytics, the content of the 3 operational collections will not change unless an explicit refresh is done by the user.

In context of operational analytics of the MongDB server and databases, it is of course desirable to have a dynamic or time-series-based analysis available. In order to accomplish this, the various status data need to be collected regularly in order to be able to determine trending or aggregation of the operational data.

Analytica is going to provide a monitoring application that collects the operational data in regular intervals as specified by the user. Based on the collected operational data, time-series based operational analytics of the MongoDB server and databases is possible. Stay tuned for a cookbook entry on this specific topic soon.

Documents that do not have to comply to a global document schema are being regarded as a plus in terms of application system programming, however, in context of analysis the fact that each document might have its own schema can be quite problematic when computing aggregates.

Example of a Problem introduced by different Schemas

Let’s look at two documents (stored in a collection called ‘het’ for heterogeneous):

{ "_id" : ObjectId("50c65a7cb4dbabc20a115510"), 
  "x" : "d1", 
  "v" : 5 }
{ "_id" : ObjectId("50c65a8cb4dbabc20a115511"), 
  "y" : "d2" }

For these two documents, let’s implement count(v) and avg(v). In MongoDB there is only one aggregation query necessary to compute both values:

db.het.aggregate(
    { $group : {
        _id : 1,
        noDocs: { $sum : 1},
        total: { $sum : "$v" },
        average: { $avg : "$v" }
    }}
);

The result (copied from the shell) is:

{"result" : [
        {
                "_id" : 1,
                "noDocs" : 2,
                "total" : 5,
                "average" : 5
        }
],
"ok" : 1}

This example clearly shows results that would been regarded as ‘wrong’ in a relational world, but what about a schema-less situation? Is the average of 5 correct or false?

At this point we could regard the average being correct in a schema-less world; and we’d be done at this point. However, if it is necessary to change the way the average is computed to take missing properties in account, then different alternative paths can be taken:

• Functions that assume schema-less documents
• Build explicit schema-aware expressions
• General ‘Uniformification’ of documents
• Analytica’s Uniformification approach

The alternatives are discussed next.

Functions for schema-less Documents

One path is to change the formula of how averages are computed. This means that average is not computed based on the number of the values, but the number of the values is given to an average() function separately from the number of documents. So instead of average(<set of values>) it is average(<set of values>,<number of documents>).

This approach basically means that all analytics functions have to be redefined based on the assumption that each document can potentially have its own schema.

Explicit Schema-aware Expressions

Instead of rewriting all functions, it is possible to build the appropriate expressions directly. For example, the average can be computed as the sum of the values divided by the number of documents. Instead of asking MongoDB to compute the average, the application computes the average itself by using the ‘counting’ result of MongoDB: 5 / 2 = 2.5

In this alternative, the author of the expression has to be aware of the fact that the documents are not following the same schema and the author has to be aware of all variations that can exist in order to write the proper expressions.

General ‘Uniformification’ Approach

An alternative approach to changing the functions is to make all documents follow the same schema. This then suggests a two step approach

• Make the schema of all documents in a collection uniform
• Perform analytics on the assumption of uniform document schemas

While this introduces a nice homogeneous schema approach, in principle every time a document is inserted (or updated or deleted) possibly the uniformification has to happen again to take any additional changes into consideration.

Analytica’s Uniformification Approach

Analytica does an implicit uniformification that covers many cases and might be sufficient in general. Analytica adds implicitly missing properties. This means that if Analytica knows about a property that is part of one document, but not of another, it implicitly adds one with value ‘null’. The result of this approach is for the above example (using Analytica’s shell):

-->executexl average(uniform.het.v)
{
 "average" : 2.5
}
-->

What is going here is quite involved. The average() function adds up all values of ‘v’. It uses a sum() function that knows that ‘null’ does not contribute to the result of sum(). However, the average() function itself counts all documents, including those where ‘v’ has the value ‘null’. So it arrives to a value of 2.5 without requiring any explicit transformation of the documents.

While this approach covers many cases, it might not solve all issues around heterogeneity. For example, if the value a missing property should be assumed to be the empty string instead of ‘null’.

Discussion

While NoSQL databases that provide schema-less documents are an interesting and productive approach from an application system development perspective, from an analytics perspective they can prove being a challenge, as this blog outlined.

However, Analytica and its understanding of the document structure supports many heterogeneous situations ‘out-of-the-box’ reducing the explicit transformation requirements significantly.

Calculated properties are a major concept of Analytica since they support storing computed values into documents. In terms of analytics, calculated properties are used to compute and to store values that are relevant for reporting, e.g., aggregations, subsets of documents, and so on.

Document-Local Virtual Properties

A first example was introduced in this blog: http://www.analytica.com/2012/11/30/nosql-native-reporting-on-json-data-analytica-and-mongodb/. The example shows how to compute the total score of all games a player played – and that for each known player.

It is called ‘document-local’ since each document representing a player was extended by ‘totalscore’ that has a different value for each player.

Document-Global Virtual Properties

Sometimes it is necessary to compute values across documents, but not for each document. For example, an interesting use case is the maximum score any player has achieved. How would that be computed? And, more importantly, where would that be stored as a property so that its value can be retrieved?

Analytica introduces the notion of virtual properties that can be added at the database level, across all documents in all collections. The following dialog on the shell shows the approach:

-->connect localhost 27017 --- --- gms MongoDB
Connected data sources:
 DataSource: gms (Type: MongoDB, Host: localhost, Port: 27017)

The first command connects to a database called ‘gms’. This database has a collection called ‘players’. Each document in this collection has a sub-collection ‘games’, each of which has a sub-collection of ‘sessions’. Each session has the accomplished ‘score’.

-->set gms.maxScore = max(players.games.sessions.score)
 Success.

The second command is a ‘set’ statement. This set statement creates a virtual property called ‘maxScore’ and the value is maximum score across all players, their games and sessions. The value is then stored in ‘maxScore’.

-->get gms.maxScore
 {
 "maxScore" : 712500
 }
-->

In order to retrieve the value of the virtual property ‘maxScore’, the ‘get’ command is issued asking for the ‘maxScore’ value stored at the database level. The result is 712500.

This example has shown that it is not only possible to compute values across documents in collections, but also store them on a document-global level.

Now let’s add a virtual property ‘minScore’ also:

-->set gms.minScore = min(players.games.sessions.score)
Success.
-->get gms.minScore
{
    "minScore" : 100
}
-->

At this point, there are two virtual properties defined at the database level.

Database-Global Virtual Properties

What if values have to be computed across databases? Analytica has a solution for this case also. The root concept is called ‘Universe’. All databases are inside a Universe and therefore the Universe is the place to store virtual properties that are global across databases.

The following shell dialog shows how virtual properties are defined on the level of the Universe. Since we only have one connected databases in this example, we use its values.

-->set UNIVERSE.minPlusMax = sum(gms.minScore,gms.maxScore)
Success.
-->get UNIVERSE.minPlusMax
{
    "minPlusMax" : 712600
}
-->

Now the Universe has a virtual property that contains the sum of the two values that are on the level of the database ‘gms’.

Summary

It is possible in Analytica to store virtual properties not only inside the document (and its sub-collections), but also across collections on the database level as well as across databases on the Universe level. This supports a natural and easy way to scope virtual properties and their values as needed.