Client-based JPA preload utility programming

WebSphere eXtreme Scale Programming Guide > System APIs and plug-ins > Use eXtreme Scale with JPA > Client-based JPA preload utility overview

Client-based JPA preload utility programming

The client-based Java™ Persistence API (JPA) preload utility loads data into eXtreme Scale backing maps using a client connection to the ObjectGrid. You can implement preloading and reloading of data in the application.

Use the StateManager interface

Use the setObjectGridState method of the StateManager interface to set the ObjectGrid state to one of the following values: OFFLINE, ONLINE, QUIESCE or PRELOAD. The StateManager interface prevents other clients from accessing the ObjectGrid when it is not yet online.
For example, set the ObjectGrid state to PRELOAD before you load the maps with data. After the data load is complete, set the ObjectGrid state back to ONLINE. See Setting the availability of an ObjectGrid for more information.
When you are preloading different maps in one ObjectGrid, set the ObjectGrid state to PRELOAD one time and set the value back to ONLINE after all maps finish data loading. This coordination can be done by the ClientLoadCallback interface. Set the ObjectGrid state to PRELOAD after the first preStart notification from the ObjectGrid instance, and set it back to ONLINE after the last postFinish notification. If you need to coordinate between multiple Java virtual machines, the application must handle the coordination.

Client-based preload example

The flow of data pre-loading follows:

Clear the map to be preloaded. In the case of an entity map, if any relation is configured as cascade-remove, the related maps are also cleared.
Run the query to JPA for the entities in a batch. The batch size is 1000.
For each batch, build a key list and value list for each partition.
For each partition, call the data grid agent to insert or update the data on the server side directly if it is an eXtreme Scale client. If the grid is a local instance, directly insert or update the data in the ObjectGrid maps.

The following sample code snippet shows a simple client loading.
// Get the StateManager 
StateManager stateMgr = StateManagerFactory.getStateManager();

// Set ObjectGrid state to PRELOAD before calling ClientLoader.loader
stateMgr.setObjectGridState(AvailabilityState.PRELOAD, objectGrid);

ClientLoader c = ClientLoaderFactory.getClientLoader();

// Load the data
c.load(objectGrid, "CUSTOMER", "custPU", null,
    null, null, null, true, null);
        
// Set ObjectGrid state back to ONLINE
stateMgr.setObjectGridState(AvailabilityState.ONLINE, objectGrid);
In this example, the CUSTOMER map is configured as an entity map. The Customer entity class, which is configured in the ObjectGrid entity metadata descriptor XML file, has a one-to-many relation with Order entities. The Customer entity has the CascadeType.ALL option enabled on the relation to the Order entity.
Before the ClientLoader.load method is called, the ObjectGrid state is set to PRELOAD.
The parameters used in the ClientLoader.load method follow:

objectGrid : The ObjectGrid instance. It is a client-side ObjectGrid instance.
"CUSTOMER" : The map to be loaded. Since the Customer has a cascade-all relation with Order entities, the Order entities will be loaded too.
"custPU" : The JPA persistence unit name for the Customer and Order entities.
null : The persistenceProps map is null, which means the default persistence properties configured in the persistence.xml will be used.
null : The entityClass is configured as null. It will be set to the entity class configured in the ObjectGrid entity meta-data descriptor XML for the map "CUSTOMER", in this case, Customer.class.
null : The loadSql is null, which means the default "select o from CUSTOMER o" will be used to query the JPA entities.
null : The query parameter map is null.
true : This indicates the data loading mode is preload. Therefore, clear operations will be called to both the CUSTOMER and ORDER maps to clear all the data before loading due to the cascade-remove relation between them.
null : The ClientLoaderCallback is null.

For more information about the required parameters, see ClientLoader API.

Reload example

Reload a map is the same as preloading a map except that the isPreload argument is set to false in the ClientLoader.load method.
In the reload mode, the flow of data loading is as follows:

Run the provided query on the ObjectGrid map and invalidate all the results. In the case of an entity map, if any relation is configured with the CascadeType.INVALIDATE option, the related entities are also invalidated from their maps.
Run the provided query to the JPA to query the JPA entities in batch. The batch size is 1000.
For each batch, build a key list and value list for each partition.
For each partition, call the data grid agent to insert or update the data on the server side directly if it is an eXtreme Scale client. If the grid is a local eXtreme Scale configuration, directly insert or update the data in the ObjectGrid maps.

A reload sample follows:
// Get the StateManager 
StateManager stateMgr = StateManagerFactory.getStateManager();

// Set ObjectGrid state to PRELOAD before calling ClientLoader.loader
stateMgr.setObjectGridState(AvailabilityState.PRELOAD, objectGrid);

ClientLoader c = ClientLoaderFactory.getClientLoader();

// Load the data
String loadSql = "select c from CUSTOMER c 
    where c.custId >= :startCustId and c.custId
< :endCustId ";
Map<String, Long> params = new HashMap<String, Long>();
params.put("startCustId", 1000L);
params.put("endCustId", 2000L);

c.load(objectGrid, "CUSTOMER", "customerPU", null, null, 
    loadSql, params, false, null);

// Set ObjectGrid state back to ONLINE
stateMgr.setObjectGridState(AvailabilityState.ONLINE, objectGrid);
Compared to the preload sample, the main difference is that a loadSql and parameters are provided. This sample only reloads the Customer data with id between 1000 and 2000.
Notice this query string observes both JPA query syntax and eXtreme Scale entity query syntax. This query string is important because it runs twice, once to ObjectGrid to invalidate the matched ObjectGrid entities and then to the JPA to load the matched JPA entities.

Call a client loader in a Loader implementation

In the Loader interface, there is a preload method:
void preloadMap(Session session, BackingMap backingMap) throws LoaderException;
This method signals the loader to preload the data into the map. A loader implementation can use a client loader to preload the data to all its partitions. For example, the JPA loader uses the client loader to preload data into the map. See JPA loaders overview for more information.
An example of how to preload the map using the client loader in the preloadMap method follows. The example first checks whether the current partition number is the same as the preload partition. If the partition number is not the same as the preload partition, no action occurs. If the partition numbers match, the client loader is called to load data into the maps. It is important to call the client loader in one and only one partition.
ObjectGrid og = session.getObjectGrid();
int partitionId = backingMap.getPartitionId();
int numPartitions = backingMap.getPartitionManager().getNumOfPartitions();

// Only call client loader data in one partition
if (partitionId == preloadPartition) {

    ClientLoader loader = ClientLoaderFactory.getClientLoader();

    // Call the client loader to load the data
    try {

        loader.load(og, backingMap.getName(), txCallback.getPersistenceUnitName(), 
                    null, entityClass, null, null, true, null);
    } catch (ObjectGridException e) {
        LoaderException le = new LoaderException("Exception caught in ObjectMap " + 
                    ogName + "." + mapName);
        le.initCause(e);
        throw le;
    }
}
Manual client load

The ClientLoader.load method provides a client load function that satisfies most scenarios. However, if to load data without the ClientLoader.load method, you can implement the own preload.
A template of a manual client load follows:
// Get the StateManager 
StateManager stateMgr = StateManagerFactory.getStateManager();

// Set ObjectGrid state to PRELOAD before calling ClientLoader.loader
stateMgr.setObjectGridState(AvailabilityState.PRELOAD, objectGrid);

// Load the data
... 

// Set ObjectGrid state back to ONLINE
stateMgr.setObjectGridState(AvailabilityState.ONLINE, objectGrid);
If you are loading the data from the client side, loading the data using a DataGrid agent could increase performance. By using a DataGrid agent, all the data reads and writes occur in the server process. You can also design the application to make sure DataGrid agents to multiple partitions run in parallel to further boost performance.
To implement data preload with a DataGrid agent, see the following example.
After you create the data preload implementation, you can create a generic Loader to complete the following tasks:

Query the data from database in batches.
Build a key list and value list for each partition.
For each partition, call the agentMgr.callReduceAgent(agent, aKey) method to run the agent in the server in a thread. By running in a thread, you can run agents concurrently on multiple partitions.

Example: Data preload with a DataGrid agent
If you are loading the data from the client side, loading the data using a DataGrid agent could increase performance. By using a DataGrid agent, all the data reads and writes occur in the server process. You can also design the application to make sure DataGrid agents to multiple partitions run in parallel to further boost performance.
An example of how to load the data with a DataGrid agent follows:
import java.io.Externalizable;
import java.io.IOException;
import java.io.ObjectInput;
import java.io.ObjectOutput;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;

import com.ibm.websphere.objectgrid.NoActiveTransactionException;
import com.ibm.websphere.objectgrid.ObjectGridException;
import com.ibm.websphere.objectgrid.ObjectGridRuntimeException;
import com.ibm.websphere.objectgrid.ObjectMap;
import com.ibm.websphere.objectgrid.Session;
import com.ibm.websphere.objectgrid.TransactionException;
import com.ibm.websphere.objectgrid.datagrid.ReduceGridAgent;
import com.ibm.websphere.objectgrid.em.EntityManager;

public class InsertAgent implements ReduceGridAgent, Externalizable {

    private static final long serialVersionUID = 6568906743945108310L;

    private List keys = null;

    private List vals = null;

    protected boolean isEntityMap;

    public InsertAgent() {
    }

    public InsertAgent(boolean entityMap) {
        isEntityMap = entityMap;
    }

    public Object reduce(Session sess, ObjectMap map) {
        throw new UnsupportedOperationException(
            "ReduceGridAgent.reduce(Session, ObjectMap)");
    }

    public Object reduce(Session sess, ObjectMap map, Collection arg2) {
        Session s = null;
        try {
            s = sess.getObjectGrid().getSession();
            ObjectMap m = s.getMap(map.getName());
            s.beginNoWriteThrough();
            Object ret = process(s, m);
            s.commit();
            return ret;
        } catch (ObjectGridRuntimeException e) {
            if (s.isTransactionActive()) {
                try {
                    s.rollback();
                } catch (TransactionException e1) {
                } catch (NoActiveTransactionException e1) {
                }
            }
            throw e;
        } catch (Throwable t) {
            if (s.isTransactionActive()) {
                try {
                    s.rollback();
                } catch (TransactionException e1) {
                } catch (NoActiveTransactionException e1) {
                }
            }
            throw new ObjectGridRuntimeException(t);
        }

    }

    public Object process(Session s, ObjectMap m) {
        try {

            if (!isEntityMap) {
                // In the POJO case, it is very straightforward, 
                // we can just put everything in the
                // map using insert
                insert(m);
            } else {
                // 2. Entity case.
                // In the Entity case, we can persist the entities
                EntityManager em = s.getEntityManager();
                persistEntities(em);

            }
            return Boolean.TRUE;
        } catch (ObjectGridRuntimeException e) {
            throw e;
        } catch (ObjectGridException e) {
            throw new ObjectGridRuntimeException(e);
        } catch (Throwable t) {
            throw new ObjectGridRuntimeException(t);
        }

    }

    /**
     * Basically this is fresh load.
     * @param s
     * @param m
     * @throws ObjectGridException
     */
    protected void insert(ObjectMap m) throws ObjectGridException {

        int size = keys.size();

        for (int i = 0; i
< size; i++) {
            m.insert(keys.get(i), vals.get(i));
        }

    }


    protected void persistEntities(EntityManager em) {
        Iterator<Object> iter = vals.iterator();

        while (iter.hasNext()) {
            Object value = iter.next();
            em.persist(value);
        }
    }

    public Object reduceResults(Collection arg0) {
        return arg0;
    }

    public void readExternal(ObjectInput in) 
      throws IOException, ClassNotFoundException {
        int v = in.readByte();
        isEntityMap = in.readBoolean();
        vals = readList(in);
        if (!isEntityMap) {
            keys = readList(in);
        }
    }

    public void writeExternal(ObjectOutput out) throws IOException {
        out.write(1);
        out.writeBoolean(isEntityMap);

        writeList(out, vals);
        if (!isEntityMap) {
            writeList(out, keys);
        }

    }

    public void setData(List ks, List vs) {
        vals = vs;
        if (!isEntityMap) {
            keys = ks;
        }
    }

    /**
     * @return Returns the isEntityMap.
     */
    public boolean isEntityMap() {
        return isEntityMap;
    }

    static public void writeList(ObjectOutput oo, Collection l) 
      throws IOException {
        int size = l == null ? -1 : l.size();
        oo.writeInt(size);
        if (size > 0) {
            Iterator iter = l.iterator();
            while (iter.hasNext()) {
                Object o = iter.next();
                oo.writeObject(o);
            }
        }
    }

    public static List readList(ObjectInput oi) 
      throws IOException, ClassNotFoundException {
        int size = oi.readInt();
        if (size == -1) {
            return null;
        }


        ArrayList list = new ArrayList(size);
        for (int i = 0; i
< size; ++i) {
            Object o = oi.readObject();
            list.add(o);
        }
        return list;
    }
}
Parent topic
Client-based JPA preload utility overview

Related concepts

Client-based JPA preload utility overview

Related tasks

Troubleshoot loaders
Related reference

JPA loader programming considerations

+
Search Tips | Advanced Search