Implement client-based JPA preload utility

Program guide > Programming for JPA integration > Client-based JPA preload utility overview

Implement client-based JPA preload utility

You can implement preloading and reloading of data in your application with a JPA utility.
JPA allows you many different ways of interacting with a backend data-store.

Procedure
Use the StateManager interface

Use the setObjectGridState method of the StateManager interface to set the ObjectGrid state to one of the following values: OFFLINE, ONLINE, QUIESCE or PRELOAD. The StateManager interface prevents other clients from accessing the ObjectGrid when it is not yet online.
For example, set the ObjectGrid state to PRELOAD before you load the maps with data. After the data load is complete, set the ObjectGrid state back to ONLINE. See Setting the availability of an ObjectGrid for more information.

When you are preloading different maps in one ObjectGrid, set the ObjectGrid state to PRELOAD one time and set the value back to ONLINE after all maps finish data loading. This coordination can be done by the ClientLoadCallback interface. Set the ObjectGrid state to PRELOAD after the first preStart notification from the ObjectGrid instance, and set it back to ONLINE after the last postFinish notification.

If preload maps from different Java™ virtual machines, you have to coordinate between multiple Java virtual machines. Set the ObjectGrid state to PRELOAD one time before the first map is being preloaded in any of the JVMs, and set the value back to ONLINE after all maps finish data loading in all the JVMs.
Client-based preload example

To pre-load the data, first clear the map to be preloaded. In the case of an entity map, if any relation is configured as cascade-remove, clear the related maps.

Run the query to JPA for the entities in a batch. The batch size is 1000.

For each batch, build a key list and value list for each partition.

For each partition, call the data grid agent to insert or update the data on the server side directly if it is an eXtreme Scale client. If the grid is a local instance, directly insert or update the data in the ObjectGrid maps.

The following sample code snippet shows a simple client loading. In this example, the CUSTOMER map is configured as an entity map. The Customer entity class, which is configured in the ObjectGrid entity metadata descriptor XML file, has a one-to-many relation with Order entities. The Customer entity has the CascadeType.ALL option enabled on the relation to the Order entity. Before the ClientLoader.load method is called, the ObjectGrid state is set to PRELOAD.
// Get the StateManager 
StateManager stateMgr = StateManagerFactory.getStateManager();

// Set ObjectGrid state to PRELOAD before calling ClientLoader.loader
stateMgr.setObjectGridState(AvailabilityState.PRELOAD, objectGrid);

ClientLoader c = ClientLoaderFactory.getClientLoader();

// Load the data
c.load(objectGrid, "CUSTOMER", "custPU", null,
    null, null, null, true, null);
        
// Set ObjectGrid state back to ONLINE
stateMgr.setObjectGridState(AvailabilityState.ONLINE, objectGrid);
Reload a map

Reload a map is the same as preloading a map except that the isPreload argument is set to false in the ClientLoader.load method. In the reload method, the flow of data is...

Run the provided query on the ObjectGrid map and invalidate all the results. In the case of an entity map, if any relation is configured with the CascadeType.INVALIDATE option, the related entities are also invalidated from their maps.

Run the provided query to the JPA to query the JPA entities in batch. The batch size is 1000.

For each batch, build a key list and value list for each partition.

For each partition, call the data grid agent to insert or update the data on the server side directly if it is an eXtreme Scale client. If the grid is a local eXtreme Scale configuration, directly insert or update the data in the ObjectGrid maps.

The following sample shows how to reload maps. Compared to the preload sample in the previous section, the main difference is that a loadSql and parameters are provided. This sample only reloads the Customer data with id between 1000 and 2000.
// Get the StateManager 
StateManager stateMgr = StateManagerFactory.getStateManager();

// Set ObjectGrid state to PRELOAD before calling ClientLoader.loader
stateMgr.setObjectGridState(AvailabilityState.PRELOAD, objectGrid);

ClientLoader c = ClientLoaderFactory.getClientLoader();

// Load the data
String loadSql = "select c from CUSTOMER c 
    where c.custId >= :startCustId and c.custId
< :endCustId ";
Map<String, Long> params = new HashMap<String, Long>();
params.put("startCustId", 1000L);
params.put("endCustId", 2000L);

c.load(objectGrid, "CUSTOMER", "customerPU", null, null, 
    loadSql, params, false, null);

// Set ObjectGrid state back to ONLINE
stateMgr.setObjectGridState(AvailabilityState.ONLINE, objectGrid);
Remember: This query string observes both JPA query syntax and eXtreme Scale entity query syntax. This query string is important because it runs twice, once to ObjectGrid to invalidate the matched ObjectGrid entities and then to the JPA to load the matched JPA entities.
Call a client loader
Use the preload method in the Loader interface...
void preloadMap(Session session, BackingMap backingMap) throws LoaderException;
This method signals the loader to preload the data into the map. A loader implementation can use a client loader to preload the data to all its partitions. For example, the JPA loader uses the client loader to preload data into the map. See JPA loaders overview for more information.
An example of how to preload the map using the client loader in the preloadMap method follows. The example first checks whether the current partition number is the same as the preload partition. If the partition number is not the same as the preload partition, no action occurs. If the partition numbers match, the client loader is called to load data into the maps. It is important to call the client loader in one and only one partition.
<codeblock>ObjectGrid og = session.getObjectGrid();
int partitionId = backingMap.getPartitionId();
int numPartitions = backingMap.getPartitionManager().getNumOfPartitions();

// Only call client loader data in one partition
if (partitionId == preloadPartition) {

    ClientLoader loader = ClientLoaderFactory.getClientLoader();

    // Call the client loader to load the data
    try {

        loader.load(og, backingMap.getName(), txCallback.getPersistenceUnitName(), 
                    null, entityClass, null, null, true, null);
    } catch (ObjectGridException e) {
        LoaderException le = new LoaderException("Exception caught in ObjectMap " + 
                    ogName + "." + mapName);
        le.initCause(e);
        throw le;
    }
}</codeblock>
Remember: Configure the backingMap attribute "preloadMode" to true, so the preload method is executed asynchronously. Otherwise, the preload method will block the ObjectGrid instance from being activated.
Manual client load

The ClientLoader.load method provides a client load function that satisfies most scenarios. However, to load data without the ClientLoader.load method, you can implement the own preload.
A template of a manual client load follows:
// Get the StateManager 
StateManager stateMgr = StateManagerFactory.getStateManager();

// Set ObjectGrid state to PRELOAD before calling ClientLoader.loader
stateMgr.setObjectGridState(AvailabilityState.PRELOAD, objectGrid);


// Load the data
... 

// Set ObjectGrid state back to ONLINE
stateMgr.setObjectGridState(AvailabilityState.ONLINE, objectGrid);
If you are loading data from the client, consider using DataGrid as described in the following section.
Preloading data with DataGrid

When loading from the client side, using a DataGrid agent could increase performance. By using a DataGrid agent, all the data reads and writes occur in the server process. You can also design the application to make sure DataGrid agents to multiple partitions run in parallel to further boost performance.
To implement data preload with a DataGrid agent, see the following example.
After you create the data preload implementation, you can create a generic Loader to complete the following tasks:

Query the data from database in batches.

Build a key list and value list for each partition.

For each partition, call the agentMgr.callReduceAgent(agent, aKey) method to run the agent in the server in a thread. By running in a thread, you can run agents concurrently on multiple partitions.

The following snippet of code is an example of how to load using a DataGrid agent:
import java.io.Externalizable;
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectOutput;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;

import com.ibm.websphere.objectgrid.NoActiveTransactionException;
import com.ibm.websphere.objectgrid.ObjectGridException;
import com.ibm.websphere.objectgrid.ObjectGridRuntimeException;
import com.ibm.websphere.objectgrid.ObjectMap;
import com.ibm.websphere.objectgrid.Session;
import com.ibm.websphere.objectgrid.TransactionException;
import com.ibm.websphere.objectgrid.datagrid.ReduceGridAgent;
import com.ibm.websphere.objectgrid.em.EntityManager;

public class InsertAgent implements ReduceGridAgent, Externalizable {

    private static final long serialVersionUID = 6568906743945108310L;

    private List keys = null;

    private List vals = null;

    protected boolean isEntityMap;

    public InsertAgent() {
    }

    public InsertAgent(boolean entityMap) {
        isEntityMap = entityMap;
    }

    public Object reduce(Session sess, ObjectMap map) {
        throw new UnsupportedOperationException(
            "ReduceGridAgent.reduce(Session, ObjectMap)");
    }

    public Object reduce(Session sess, ObjectMap map, Collection arg2) {
        Session s = null;
        try {
            s = sess.getObjectGrid().getSession();
            ObjectMap m = s.getMap(map.getName());
            s.beginNoWriteThrough();
            Object ret = process(s, m);
            s.commit();
            return ret;
        } catch (ObjectGridRuntimeException e) {
            if (s.isTransactionActive()) {
                try {
                    s.rollback();
                } catch (TransactionException e1) {
                } catch (NoActiveTransactionException e1) {
                }
            }
            throw e;
        } catch (Throwable t) {
            if (s.isTransactionActive()) {
                try {
                    s.rollback();
                } catch (TransactionException e1) {
                } catch (NoActiveTransactionException e1) {
                }
            }
            throw new ObjectGridRuntimeException(t);
        }

    }

    public Object process(Session s, ObjectMap m) {
        try {

            if (!isEntityMap) {
                // In the POJO case, it is very straightforward, 
                // we can just put everything in the
                // map using insert
                insert(m);
            } else {
                // 2. Entity case.
                // In the Entity case, we can persist the entities
                EntityManager em = s.getEntityManager();
                persistEntities(em);

            }
            return Boolean.TRUE;
        } catch (ObjectGridRuntimeException e) {
            throw e;
        } catch (ObjectGridException e) {
            throw new ObjectGridRuntimeException(e);
        } catch (Throwable t) {
            throw new ObjectGridRuntimeException(t);
        }

    }

    /**
     * Basically this is fresh load.
     * @param s
     * @param m
     * @throws ObjectGridException
     */
    protected void insert(ObjectMap m) throws ObjectGridException {

        int size = keys.size();

        for (int i = 0; i
< size; i++) {
            m.insert(keys.get(i), vals.get(i));
        }

    }


    protected void persistEntities(EntityManager em) {
        Iterator<Object> iter = vals.iterator();

        while (iter.hasNext()) {
            Object value = iter.next();
            em.persist(value);
        }
    }

    public Object reduceResults(Collection arg0) {
        return arg0;
    }

    public void readExternal(ObjectInput in) 
      throws IOException, ClassNotFoundException {
        int v = in.readByte();
        isEntityMap = in.readBoolean();
        vals = readList(in);
        if (!isEntityMap) {
            keys = readList(in);
        }
    }

    public void writeExternal(ObjectOutput out) throws IOException {
        out.write(1);
        out.writeBoolean(isEntityMap);

        writeList(out, vals);
        if (!isEntityMap) {
            writeList(out, keys);
        }

    }

    public void setData(List ks, List vs) {
        vals = vs;
        if (!isEntityMap) {
            keys = ks;
        }
    }

    /**
     * @return Returns the isEntityMap.
     */
    public boolean isEntityMap() {
        return isEntityMap;
    }

    static public void writeList(ObjectOutput oo, Collection l) 
      throws IOException {
        int size = l == null ? -1 : l.size();
        oo.writeInt(size);
        if (size > 0) {
            Iterator iter = l.iterator();
            while (iter.hasNext()) {
                Object o = iter.next();
                oo.writeObject(o);
            }
        }
    }

    public static List readList(ObjectInput oi) 
      throws IOException, ClassNotFoundException {
        int size = oi.readInt();
        if (size == -1) {
            return null;
        }


        ArrayList list = new ArrayList(size);
        for (int i = 0; i
< size; ++i) {
            Object o = oi.readObject();
            list.add(o);
        }
        return list;
    }
}
Parameters for load method with ClientLoader
The load method for ClientLoader allows you various configuring for the requirements.

Parent topic:
Client-based JPA preload utility overview

+
Search Tips | Advanced Search