WebSphere XD Dynamic Operations

 

+

Search Tips   |   Advanced Search

 

Overview

WebSphere XD uses virtualization of WebSphere environments and a goals-directed infrastructure to deliver dynamic operations

With WebSphere XD, rather than deploying applications directly to a server, applications are deployed to a subset of servers in a resource pool. WebSphere XD's virtualized infrastructure is predicated on two new constructs:

  • node groups (resource pools)
  • dynamic clusters

Static environment have dedicated servers for each application, and servers are sized based on application requirements during peak load. Production environments must be prepared for the load during this, possibly, short period, which means that during non-peak hours servers are under-utilized.


Figure 3-1 shows three static clusters that are used for three different applications. Two static clusters spread each over three nodes while the third one includes two nodes. The peak usage time of the three applications is at different time intervals. Application 1 has most load at 8 AM. The utilization is at 100% and the three available nodes cannot handle the workload. Some requests have to wait or the users might even get a service unavailable error message. At the same time, the other five nodes hosting applications 2 and 3 are under-utilized. However, in a static cluster they cannot be used for supporting application 1 peak load. At 12 PM and 5 PM the same thing happens for applications 2 and 3: some nodes are completely loaded while other nodes still have capacity available.

To guarantee a good quality of service in this example, one possibility would be to add additional nodes. However, doing so would increase the cost and would not solve the underlying problem. In this case the problem is that we cannot share our resources and thus optimize utilization.

It is normal practice to run servers at less than 50% average utilization to account for peak load times and also to provide some overcapacity for failover events.

 

Planning and configuring node groups

With WebSphere XD we can take better advantage of system resources because it allows us to share resources dynamically. This is achieved by adding all our nodes into a resource pool called node group. Resources in a node group are then dynamically assigned to applications as needed. Here's an example:

All nodes that were used in the static environment are now added into a node group. Within the node group all resources are dynamically allocated to the various applications based on predefined service policies and the current workload.

In this example, the workload at 8 AM for application 1 is increasing, as a result this application is also started on another node. So now there are four nodes available to handle the requests. At 12 PM the workload of application 1 has declined, but application 2 is at its peak. Now application 2 is spread over five nodes while application 1 only runs on two nodes and application 3 on only one node. At 5 PM the requests for application 3 have reached their maximum. Therefore it is started on more nodes while the resources allocated to other applications are decreasing.

All incoming requests are forwarded to the nodes and there is no delay in service.

The three applications in the example do not fully utilize the system resources in the node group, which makes it possible to either add more applications to this node group or to reduce the overall number of nodes and to run at a higher average utilization.

A node group is a set of systems (nodes) that are capable of hosting one or more applications. There can be more than one node group within one WebSphere XD cell. An application is placed into a node group and is optimized based on service policies and service goals.

Applications can run on any node in the node group. Having more dynamic clusters and thus applications in a node group generally allows for better system utilization than having multiple node groups and less applications mapped to each node group.

In any case, the utilization depends on workload intensity. So we may achieve good utilization in a large node group that hosts one application with high workload, or in a small node group with a big number of applications that each receive only a small workload.

Best practice is that all our nodes belong to one big node group. System utilization is optimized when the application placement controller has all nodes in just one node group.

However, this approach is resource intensive because all applications are deployed on each node and all licences have to be purchased for the whole node group. So depending on our environment it might be better to balance the number of applications and the number of nodes that are part of a node group.

A new feature in IBM WebSphere XD V6.0 is that now a single node can be a member of more than one node group. This is called overlapping node groups.

It is also possible to include an IBM WebSphere XD V5.1 node into an IBM WebSphere XD V6.0 cell, but we cannot create a dynamic cluster on this node.

 

Creating a node group

Before creating a node group all nodes must be added into the cell. A node can be added either by using the addNode command or by using the Administrative Console.

To create a node group, log on to the Administrative Console:

  1. Select System administration | Node groups

    Click New

  2. Enter the node group name and description and click Apply.

  3. Add nodes to the node group. Click...

    Node group members | Add

    ...and select the nodes to be added.

  4. Save and synchronize the new configuration.

To add nodes using the command line:

$WASHOME\bin\wsadmin.sh -profile NodeGroupProcs.jacl 
wsadmin>addNodegroupMember nodegroupName nodeName  
wsadmin>removeNodegroupMember  nodegroupName nodeName 

After we create a node group, we can create dynamic clusters.

 

Deleting a node group

  1. Uninstall all applications deployed to the dynamic clusters on that node group.

  2. Remove members from node group

  3. Delete node group...

    System administration | Node groups | node group | Delete

    To remove using scripts...

    $WASHOME\bin\wsadmin.sh -profile NodeGroupProcs.jacl 
    wsadmin>removeNodegroupMember  nodegroupName nodeName 
    

We do not need to delete the node group's dynamic clusters before deleting a node group, they are removed automatically when the node group is deleted.

 

Setting a node into maintenance mode

Each application is deployed on each node within the node group and can be started everywhere. Therefore it is difficult to perform system maintenance, for example rebooting a server, applying operating system updates, etc. - because we never know when an application is started on a node.

The solution to this problem is to set a node into maintenance mode so that the application placement controller excludes this node from all automatic application placement.

To set a node into maintenance mode, open the Administrative Console and go to...

System administration | Nodes | nodename

...which we want to perform maintenance for and click Set Maintenance. Note that doing this does not stop processing on that node. If we want to stop all processes either click Set Maintenance, Immediate Stop or select the node and click Stop.

We can recognize that a node is in maintenance mode by the small icon in the Maintenance column of the nodes overview page.

To reset the maintenance mode, select System administration | Nodes. Select the node to which we want to restore to service and click Unset Maintenance.

After resetting the maintenance mode of a node, the Node Agent is not automatically started. We can start it by using the startNode command in the profile_name/bin directory on that system.

 

Planning and configuring dynamic clusters

A dynamic cluster is limited to a single node group and cannot spread across node group borders. A dynamic cluster is similar to a static cluster from WebSphere Network Deployment (ND) with several key differences...


Static cluster Dynamic cluster
Definition A static cluster is a group of application servers in a WebSphere ND environment that participate in workload management. In a dynamic cluster we define service goals and policies for our application. WebSphere XD assigns nodes and resources to our application according to these policies.
Cluster management We define the application servers that are in the cluster and then start or stop the application servers in the cluster. A dynamic cluster can start and stop instances of application servers as needed.
Cluster templates When we define a static cluster, we can select an application server template that all the application server instances we create are based on. There is no possibility to change all application servers in a cluster at once as it is not possible to change an existing server template. Therefore we need to change each application server in the cluster individually. When we define a dynamic cluster, we can select an application server template for the application server instances. If we make changes to the template within the dynamic cluster later, the changes are propagated to all application server instances in the cluster.
Application server weights We explicitly assign a weight value to each application server instance. The Dynamic Workload Manager (DWLM) is enabled by default and assigns weights to the application server instances. These weights change dynamically with the system load.
Applicability If we use static clusters in a WebSphere XD cell, the behavior is identical to static clusters in WebSphere ND. We can use dynamic clusters in a WebSphere XD environment only.

 

Create a server template

Server templates are created based on existing application servers. Therefore we first need to create a new application server (or use an existing one) that has all desired configuration settings prior to creating the server template. After creating the server template we can delete the original application server.

To create an application server template based on an existing server:

  1. Select Servers | Application servers.

  2. Click the Templates... button at the top of the server list.

  3. Click New.

  4. Select a server from the list to build the template from and click OK.

  5. Enter a name and description for the template and click OK.

  6. Save and synchronize our configuration.
The new template is added to the list of templates and available for selection the next time we create an application server or cluster (static or dynamic).

Attention: After we have created a dynamic cluster based on a server template, we can make changes to the template within the dynamic cluster under...

Servers | Dynamic Clusters | cluster_name | Server template

Changes made to the dynamic clusters' server template are propagated to all application servers within this cluster.

It is however not possible to change the originating server template itself under...

Servers | Application servers| Templates

We can only delete it and create a new one based on an application server that contains all configuration settings we need.

 

Creating a dynamic cluster

Creating a dynamic cluster is similar to creating a static cluster, but includes some dynamic cluster specific configuration settings:

  1. In the Administrative Console go to...

    Servers | Dynamic Clusters | New

    Enter the following information...

    Dynamic Cluster name For example RedbookCluster10.
    Map to node group For example RedbookNodeGroup1.
    Prefer local enabled Check if we want to optimize Enterprise Java Bean client request routing. Prefer local enabled insures that out of process calls (from one application server to another or an application client to an application server) are first attempted on the local node, and requests are only directed to a remote node if the EJB is not running locally.

    The best practice is to deploy EJB (Web) clients in the same EAR as the EJB within a single application server, this eliminates the need for out of process calls which will degrade performance.

    Default application server template
    Minimum number of cluster instances We can select to have one or more application servers started at all times or to stop all application servers in times of inactivity. The latter is called lazy application start.
    Maximum number of cluster instances
    Allow more than one instance to start on the same node Check if we want to allow more than one application server instance to be started on the same node.

  2. After entering all necessary information click Next and Finish.

  3. Save the configuration.

It is also possible to create or delete dynamic clusters using the command line.

$WASHOME\bin\wsadmin.sh -profile createDynamicCluster.jacl nodegroupName dynamicClusterName

$WASHOME\bin\wsadmin.sh -profile deleteDynamicCluster.jacl nodegroupName dynamicClusterName

 

Setting the operating mode

After creating the dynamic cluster, we have to choose the operating mode. WebSphere XD supports three different types of operating modes:

  • Automatic
  • Supervised
  • Manual

 

Automatic mode

In automatic mode the application placement controller automatically starts and stops application servers as needed when the workload changes.

 

Supervised operating mode

The supervised operation mode is identical to the automatic mode except that all actions have to be approved by an administrator. Tasks needing approval are displayed in the Administrative Console under...

Runtime Operations | Task Management| Runtime Tasks

All runtime tasks have an approval timeout. When the timeout is reached and the administrator did not accept the task, then the task is automatically declined. As a result, for an application with a high availability or continuous availability requirement, we would need to have an administrator to handle runtime tasks at all times.

Therefore, unless we have around the clock staffing, the supervised mode is only feasible for high availability or continuous availability applications during the introduction phase of WebSphere XD. After application rollout and operation validation has finished successfully, the cluster should be set to automatic mode.

 

Manual operating mode

The manual mode does not support automated application placement or runtime task suggestions. However, dynamic workload management is supported. When using the manual mode we basically combine a dynamic cluster with a static cluster: all automated processes are disabled but we take advantage of dynamic workload management.

 

Changing a dynamic cluster

Once we have created a dynamic cluster we should only edit the dynamic cluster settings and not the single application server because individual node settings will be overwritten by cluster settings.

The best practice is to create a single application server first and tune it for our application. When we have found the best settings for our application, create a template from this application server, then create a dynamic cluster from the template. We can then delete the original application server.

 

Deleting a dynamic cluster

Before deleting a dynamic cluster, we must first uninstall any application deployed to that cluster. Then go to...

Servers | Dynamic Clusters

Before a dynamic cluster can be deleted, it must be set to manual mode. When it is in manual mode, select the dynamic cluster and click the Delete button. Save and synchronize our changes.

 

Migrating a static cluster to a dynamic cluster

It is possible to migrate a static cluster to a dynamic cluster. To do so, follow these steps:

  1. >Create a node group.
  2. >Create a dynamic cluster based on a server from our static cluster.
  3. >Remap the application to the new cluster.

To remap an application, in the Administrative Console go to...

Applications | Enterprise Applications | appname | Map modules to server

Here we can change the application to cluster mapping.

 

Lazy application start

The lazy application start feature in IBM WebSphere XD V6.0 optimizes server resource allocation during times of inactivity. The smallest size for a dynamic cluster is zero, implying that an application can be configured for execution but not running in any application server instances. When a request for that application is received by the On Demand Router (ODR), an application server for that application is started automatically on any node in the node group. When a request arrives for one of the hibernating application clusters, it is queued in the ODR until the dynamic cluster is active.

Depending on the startup time of this application and the queue timeout of the ODR it is possible that the first few requests eventually receive an "HTTP Error 503 - server unavailable" and But by the time when these clients retry the request, the application server should be started. There is a special Web page for this error which can be individually set up with an automatic redirect after a few seconds.

A typical environment where the lazy application start feature is beneficial is an environment in which the ratio of the number of dynamic clusters to the number of servers is high, and where many dynamic clusters are not accessed for long periods of time. In such an environment, it is beneficial to hibernate idle dynamic clusters temporarily (stopping all server instances), thereby releasing valuable resources to be used by active dynamic clusters.

Lazy application start can only be configured for the whole dynamic cluster.

To enable lazy application start, go to...

Servers | Dynamic Clusters | dynamic_cluster_name

On the upcoming panel check the Stop all instances during periods of inactivity radio button.

The second setting for lazy application start is Time to wait before stopping instances. By default it is set to 60 minutes. But this value really depends on our application and user behavior. We can look at the server activity logs if we cannot ask our users directly for information. In these logs, we can discern if and how long servers are idle during the day and when an application has its peak time.

It is also possible to configure lazy application start during creation of the dynamic cluster.

 

Vertical stacking

Allows us to have more than one application server instance in a dynamic cluster on the same node. The benefit of this capability is better hardware utilization if a systems' CPU and memory is not fully used with a single application server on a node.

To determine how many application server instances can run in parallel on a node, take a look at the resources needed when only one instance is active. To observe the resources needed by an application, we can observe the CPU Utilization and Free Memory (MB) in the Administrative Console under...

Runtime Operations | Runtime Topology

...or use operating system-specific system monitoring tools.

We recommend that we determine the stacking number for a dynamic cluster with no other application servers running on that node. First, start one instance. While monitoring the effective throughput, increase the workload intensity. When throughput saturates, start one more instance. When adding a new instance does not improve the throughput, the stacking number is 1. If adding a new instance improves the throughput, we can conclude that the application has some internal bottleneck that prevents it from effectively using the entire box within a single application server. Thus we can continue to increase workload and (possibly) the number of active instances until no improvement in throughout can be achieved.

Repeat this approach individually for every application that can possibly run on that node. This way, we can decide for each cluster (and thus application) whether it should be using stacking or not.

When determining the stacking number for a dynamic cluster, we do not have to consider any other dynamic clusters, because we only want to learn how many instances of this dynamic cluster are needed to fully utilize the system. So when we have multiple clusters/applications, each one of them would be able to fully utilize the system if no other application is running concurrently (for example, because these applications only run at certain times of the day, week, or month). The stacking number will become the maximum number of instances that are allowed to execute on a node for this dynamic cluster, but at any time, a smaller number may be started, depending on the current workload. At run time the application placement controller, ODR, and DWLM will work together to make sure that the node is not overloaded and all applications meet their policies.

We configure vertical stacking for the whole dynamic cluster. To enable this feature, select Servers | Dynamic Clusters in the Administrative Console and click the desired cluster. Select Allow more than one instance to start on the same node. Under Number of instances enter the number of instances we want to be started in parallel on each node.

The vertical stacking number must be determined for each node. When we have heterogeneous nodes we might have different stacking numbers for each node. We can define different vertical stacking numbers per node within the dynamic cluster by using a custom property named numVerticalInstances.nodename(for example numVerticalInstances.node1).

We can enter this custom property in the Administrative Console under...

Servers | Dynamic Clusters | clustername | Custom Properties

It is also possible to configure vertical stacking during creation of the dynamic cluster.

 

See also

  1. Optimize resource usage and reduce costs, Part 1: Strengthen an enterprise intranet using WebSphere XD
  2. IBM eServer Literature Web site
  3. Taurus: A Taxonomy of the Actual Utilization of Real UNIX and Windows Servers
  4. Sagittarius: Statistics and Analysis of the Demographics of UNIX and Windows Servers
  5. Sirius: Statististical Interpretations of a Real Inventory of UNIX Servers
  6. Scorpion Update: An Evolving Method of Analyzing and Optimizing the Corporate IT Server Infrastructure