Work with external data

Work with external data

Learn about the different concepts required to work with external data.
For the actual API examples, see JSONStore examples.

Pull

Many systems use the term pull to refer to getting data from an external source.
There are three important pieces:

External Data Source

This source can be a database, a REST or SOAP API, or many others. The only requirement is that it must be accessible from either the MobileFirst Server or directly from the client application. Ideally, you want this source to return data in JSON format.

Transport Layer

This source is how you get data from the external source into your internal source, a JSONStore collection inside the store. One alternative is a MobileFirst adapter.

Internal Data Source API

This source is the JSONStore APIs we can use to add JSON data to a collection.

We can populate the internal store with data that is read from a file, an input field, or hardcoded data in a variable. It does not have to come exclusively from an external source that requires network communication.

Example pull scenario

All of the following code examples are written in pseudocode that looks similar to JavaScript.
Use MobileFirst adapters for the Transport Layer. Some of the advantages of using MobileFirst adapters are XML to JSON, security, filtering, and decoupling of server-side code and client-side code.
External Data Source: Backend REST endpoint
Imagine we have a REST endpoint that read data from a database and returns it as an array of JSON objects.
app.get('/people', function (req, res) {
  var people = database.getAll('people');
  res.json(people);
});
The data that is returned can look like the following example:
[{id: 0, name: 'carlos', ssn: '111-22-3333'}, {id: 1, name: 'mike', ssn: '111-44-3333'}, {id: 2, name: 'dgonz' ssn: '111-55-3333')]
Transport Layer: MobileFirst adapter
Imagine created an adapter called people and you defined a procedure called getPeople. The procedure calls the REST endpoint and returns the array of JSON objects to the client. You might want to do more work here, for example, return only a subset of the data to the client.
function getPeople () {
  var input = {
    method : 'get', path : '/people'
  return WL.Server.invokeHttp(input);
}
On the client, we can use the WL.Client.invokeProcedure API to get the data. Additionally, you might want to pass some parameters from the client to the MobileFirst adapter. One example is a date with the last time that the client got new data from the external source through the MobileFirst adapter.
WL.Client.invokeProcedure({
  adapter : 'people',   procedure : 'getPeople'.
  parameters : []
})
.then(function (responseFromAdapter) {
  // ...
});
You might want to take advantage of the compressResponse, timeout, and other parameters that can be passed to the invokeProcedure API.
Alternatively, we can skip the MobileFirst adapter and use something like jQuery.ajax to directly contact the REST endpoint with the data to store.
$.ajax({
  type: 'GET',   url: 'http://example.org/people', })
.then(function (responseFromEndpoint) {
  // ...
});
Internal Data Source API: JSONStore
After we have the response from the backend, we can work with that data using JSONStore.
JSONStore provides a way to track local changes. It enables some APIs to mark documents as dirty. The API records the last operation that was performed on the document, and when the document was marked as dirty. Then use this information to implement features like data synchronization.
The change API takes the data and some options:

replaceCriteria

These search fields are part of the input data. They are used to locate documents already inside a collection. For example, if you select:
['id', 'ssn']

as the replace criteria, pass the following array as the input data:
[{id: 1, ssn: '111-22-3333', name: 'Carlos'}]

and the people collection already contains the following document:
{_id: 1,json: {id: 1, ssn: '111-22-3333', name: 'Carlitos'}}

The change operation locates a document that matches exactly the following query:
{id: 1, ssn: '111-22-3333'}

Then the change operation performs a replacement with the input data and the collection contains:
{_id: 1, json: {id:1, ssn: '111-22-3333', name: 'Carlos'}}

The name was changed from Carlitos to Carlos. If more than one document matches the replace criteria, then all documents that match are replaced with the respective input data.

addNew

When no documents match the replace criteria, the change API looks at the value of this flag. If the flag is set to true, the change API creates a new document and adds it to the store. Otherwise, no further action is taken.

markDirty

Determines whether the change API marks documents that are replaced or added as dirty.

An array of data is returned from the MobileFirst adapter:
.then(function (responseFromAdapter) {
  var accessor = WL.JSONStore.get('people');
  var data = responseFromAdapter.invocationResult.array;
  var changeOptions = {
    replaceCriteria : ['id', 'ssn'], addNew : true, markDirty : false
  return accessor.change(data, changeOptions);
})
.then(function() {
  // ...
})
We can use other APIs to track changes to the local documents stored. Always get an accessor to the collection that you perform operations on.
var accessor = WL.JSONStore.get('people')

Then, we can add data (array of JSON objects) and decide whether you want it to be marked dirty or not. Typically, to set the markDirty flag to false when you get changes from the external source. Then, set the flag to true when we add data locally.
accessor.add(data, {markDirty: true})

We can also replace a document, and opt to mark the document with the replacements as dirty or not.
accessor.replace(doc, {markDirty: true})

Similarly, we can remove a document, and opt to mark the removal as dirty or not. Documents that are removed and marked dirty do not show up when we use the find API. However, they are still inside the collection until we use the markClean API, which physically removes the documents from the collection. If the document is not marked as dirty, it is physically removed from the collection.
accessor.remove(doc, {markDirty: true})
Push

Many systems use the term push to refer to sending data to an external source.
There are three important pieces:

Internal Data Source API

This source is the JSONStore API that returns documents with local-only changes (dirty).

Transport Layer

This source is how to contact the external data source to send the changes.

External Data Source

This source is typically a database, REST or SOAP endpoint, among others, that receives the updates that the client made to the data.

Example push scenario

All of the following code examples are written in pseudocode that looks similar to JavaScript.
Use MobileFirst adapters for the Transport Layer. Some of the advantages of using MobileFirst adapters are XML to JSON, security, filtering, and decoupling of server-side code and client-side code.
Internal Data Source API: JSONStore
After we have an accessor to the collection, we can call the getAllDirty API to get all documents that are marked as dirty. These documents have local-only changes to send to the external data source through a transport layer.
var accessor = WL.JSONStore.get('people');
accessor.getAllDirty()
.then(function (dirtyDocs) {
  // ...
});
The dirtyDocs argument looks like the following example:
[{_id: 1, json: {id: 1, ssn: '111-22-3333', name: 'Carlos'}, _operation: 'add', _dirty: '1395774961,12902'}]

The fields are:

_id

Internal field that JSONStore uses. Every document is assigned a unique one.

json

The data that was stored.

_operation

The last operation that was performed on the document. Possible values are add, store, replace, and remove.

_dirty

A time stamp stored as a number to represent when the document was marked dirty.
Transport Layer: MobileFirst adapter
Choose to send dirty documents to a MobileFirst adapter. Assume we have a people adapter defined with an updatePeople procedure.
.then(function (dirtyDocs) {
  return WL.Client.invokeProcedure({
    adapter : 'people', procedure : 'updatePeople', parameters : [ dirtyDocs ]
  });
})
.then(function (responseFromAdapter) {
  // ...
})
You might want to take advantage of the compressResponse, timeout, and other parameters that can be passed to the invokeProcedure API. On the MobileFirst Server, the adapter has the updatePeople procedure, which might look like the following example:
function updatePeople (dirtyDocs) {
  var input = {
    method : 'post', path : '/people', body: {
      contentType : 'application/json',   content : JSON.stringify(dirtyDocs)
    }
  return WL.Server.invokeHttp(input);
}
Instead of relaying the output from the getAllDirty API on the client, you might have to update the payload to match a format that is expected by the backend. You might have to split the replacements, removals, and inclusions into separate backend API calls.
Alternatively, we can iterate over the dirtyDocs array and check the _operation field. Then, send replacements to one procedure, removals to another procedure, and inclusions to another procedure. The previous example sends all dirty documents in bulk to the MobileFirst adapter.
var len = dirtyDocs.length;
var arrayOfPromises = [];
while (len--) {
  var currentDirtyDoc = dirtyDocs[len];
  switch (currentDirtyDoc._operation) {
    case 'add':
    case 'store':
      arrayOfPromises.push(WL.Client.invokeProcedure({
        adapter : 'people',     procedure : 'addPerson',     parameters : [ currentDirtyDoc ]
      }));
    case 'replace':
    case 'refresh':
      arrayOfPromises.push(WL.Client.invokeProcedure({
        adapter : 'people',     procedure : 'replacePerson',     parameters : [ currentDirtyDoc ]
      }));
    case 'remove':
    case 'erase':
      arrayOfPromises.push(WL.Client.invokeProcedure({
        adapter : 'people',     procedure : 'removePerson',     parameters : [ currentDirtyDoc ]
      }));
}
$.when.apply(this, arrayOfPromises)
.then(function () {
  var len = arguments.length;
  while (len--) {
    // Look at the responses in arguments[len]
});
Alternatively, we can skip the MobileFirst adapter and contact the REST endpoint directly.
.then(function (dirtyDocs) {
  return $.ajax({
    type: 'POST', url: 'http://example.org/updatePeople', data: dirtyDocs
  });
})
.then(function (responseFromEndpoint) {
  // ...
});
External Data Source: Backend REST endpoint
The backend accepts or rejects changes, and then relays a response back to the client. After the client looks at the response, it can pass documents that were updated to the markClean API.
.then(function (responseFromAdapter) {
  if (responseFromAdapter is successful) {
    WL.JSONStore.get('people').markClean(dirtyDocs);
})
.then(function () {
  // ...
})
After documents are marked as clean, they do not show up in the output from the getAllDirty API.
Parent topic: JSONStore advanced topics