Script-based LPAR event handling

Management of LPARS can be automated by using the SSH remote command execution functionality of the HMC. With this functionality you can create automated processes which are managed and executed from a central system through SSH remote execution.

You can select a server to perform central operations like the NIM or CSM Server.

Single and simple remote command execution

You can retrieve information or change the LPAR configuration using the HMC command line interface.

For example, to obtain a list of all LPARS and their profile properties for managed system p5+-9133-55A-SN10D1FAG...

ssh wasp5l@riogrande "lssyscfg -r lpar -m p5+-9133-55A-SN10D1FAG"

Example 3-2 Output of HMC command lssyscfg for Managed System

#ssh wasp5l@riogrande "lssyscfg -r lpar -m p5+-9133-55A-SN10D1FAG"
wasp5l@riogrande's password:
name=wasp5l_vio, lpar_id=1, lpar_env=vioserver, state=Running, resource_config=1, os_version=0.0.0.0.0.0, logical_serial_num=10D1FAG1, default_profile=wasp5l_vio_limited, curr_profile=wasp5l_vio_limited, work_group_id=none, shared_proc_pool_util_auth=0, power_ctrl_lpar_ids=none, boot_mode=norm, lpar_keylock=norm, auto_start=1, redundant_err_path_reporting=0
name=de_lpar, lpar_id=2, lpar_env=aixlinux, state=Running, resource_config=1, os_version=0.0.0.0.0.0, logical_serial_num=10D1FAG2, default_profile=loaner_lpar, curr_profile=loaner_lpar, work_group_id=none, shared_proc_pool_util_auth=0, power_ctrl_lpar_ids=none, boot_mode=norm, lpar_keylock=norm, auto_start=0, redundant_err_path_reporting=0
name=br_lpar, lpar_id=3, lpar_env=aixlinux, state=Running, resource_config=1, os_version=0.0.0.0.0.0, logical_serial_num=10D1FAG3, default_profile=br_lpar, curr_profile=br_lpar, work_group_id=none, shared_proc_pool_util_auth=0, power_ctrl_lpar_ids=none, boot_mode=norm, lpar_keylock=norm, auto_start=0, redundant_err_path_reporting=0
name=uk_lpar, lpar_id=4, lpar_env=aixlinux, state=Running, resource_config=1, os_version=0.0.0.0.0.0, logical_serial_num=10D1FAG4, default_profile=uk_lpar, curr_profile=uk_lpar, work_group_id=none, shared_proc_pool_util_auth=0, power_ctrl_lpar_ids=none, boot_mode=norm, lpar_keylock=norm, auto_start=0, redundant_err_path_reporting=0
name=us_lpar, lpar_id=5, lpar_env=aixlinux, state=Running, resource_config=1, os_version=0.0.0.0.0.0, logical_serial_num=10D1FAG5, default_profile=us_lpar, curr_profile=us_lpar, work_group_id=none, shared_proc_pool_util_auth=0, power_ctrl_lpar_ids=none, boot_mode=norm, lpar_keylock=norm, auto_start=0, redundant_err_path_reporting=0
name=pt_lpar, lpar_id=6, lpar_env=aixlinux, state=Running, resource_config=1, os_version=0.0.0.0.0.0, logical_serial_num=10D1FAG6, default_profile=pt_lpar, curr_profile=pt_lpar, work_group_id=none, shared_proc_pool_util_auth=0, power_ctrl_lpar_ids=none, boot_mode=norm, lpar_keylock=norm, auto_start=0, redundant_err_path_reporting=0
name=pl_lpar, lpar_id=7, lpar_env=aixlinux, state=Running, resource_config=1, os_version=0.0.0.0.0.0, logical_serial_num=10D1FAG7, default_profile=pl_lpar, curr_profile=pl_lpar, work_group_id=none, shared_proc_pool_util_auth=0, power_ctrl_lpar_ids=none, boot_mode=norm, lpar_keylock=norm, auto_start=0, redundant_err_path_reporting=0
#

To limit output to partition name (name) and operating environment (lpar_env)...

[0:root@MADRID:]/home/root # ssh wasp5l@riogrande
wasp5l@riogrande's password:
Last login: Mon Oct 2 15:51:11 2006 from madrid.itsc.austin.ibm.com
wasp5l@riogrande:~> lssyscfg -r lpar -m p5+-9133-55A-SN10D1FAG -F name,lpar_env
wasp5l_vio,vioserver
de_lpar,aixlinux
br_lpar,aixlinux
uk_lpar,aixlinux
us_lpar,aixlinux
pt_lpar,aixlinux
pl_lpar,aixlinux

In both of these examples, you would need to manually enter the password for user wasp5l. The next step would be to enable remote execution without a password prompt. If you want to automate this procedure, perform an SSH key exchange from your management console onto the HMC.

See: Setting up secure script execution between SSH clients and the HMC.

Complex remote command execution

A more complex remote command execution can be a script to run on AIX that sends commands via ssh into the HMC to collect information about the HMC, the systems managed by the HMC, and the LPARS in those systems.

For a detailed explanation of how to have this script run without prompting for the password of the HMC user, see the Techdocs Library document passAIX ssh client to pSeries HMC

Automation of partition start up and shut down sequence

To remotely start all logical partitions with another system profile as the default assigned one in order to check changed settings before activating them by default. If the new system profile, Virtual_IO_Client_Profile, exists, you could use the following command:

chsysstate -m p5+-9133-55A-SN10D1FAG -o on -r sysprof -n Virtual_IO_Client_Profile

As another example, suppose you want to stop and start an LPAR in order to activate a partition profile for testing purposes. This testing would take place after business hours, and without manual intervention. If the temporary partition profile de_limited exists, the following commands would be executed by a scheduled process.

Use the chsysstate command to stop the partition by issuing the shutdown -F command to shut down an AIX or Virtual I/O Server partition.

chsysstate -m p5+-9133-55A-SN10D1FAG -r lpar -o osshutdown -n de_lpar --immed

Issue the command lssyscfg to get the actual state of the partition.

lssyscfg -r lpar -m p5+-9133-55A-SN10D1FAG --filter ""lpar_names=de_lpar"" -F state

If the command returns Not Activated for the state of the partition, start the partition.

chsysstate -m p5+-9133-55A-SN10D1FAG -r lpar -o on --id 2 -f de_limited

The same result can be achieved by a single command while using the --restart option together with the -o osshutdown option:

chsysstate -m p5+-9133-55A-SN10D1FAG -r lpar -o osshutdown --restart -n de_lpar --immed

Applications and the drmgr command

By default, an application running in a DLPAR does not receive any information about changes in the resources; instead, the application has to explicitly register itself to receive such resource change notifications. When a DR event occurs, the applications are made aware of resource changes if they have registered a DR script. The DR script can be any type of shell script, Perl script, or even an AIX 5L executable.

The DR manager notifies the registered DR scripts three times after a DLPAR operation request is initiated using either the graphical user interface or command line interface on the HMC.

During the check phase (before doing anything, an application can deny or grant the DR event from proceeding at this time)
During the pre-phase (just before altering a resource)
During the post-phase (after altering a resource)

The pre-phase and the post-phase run if the outcome of the check phase was a grant.

The DR manager calls each DR script with the name of the phase as the first argument. For example, checkacquire indicates the check phase for the resource add, preacquire indicates the pre-phase for the resource add, and postacquire indicates the post-phase of the resource add. Similarly, checkrelease, prerelease, and postrelease are the first arguments for the resource remove. The resource type is identified (CPU or memory) using the second argument.

Application scripts are invoked for both add and remove operations. When removing resources, scripts are provided to resolve conditions imposed by the application that prevent the resource from being removed. The presence of particular processor bindings and the lack of pinnable memory might cause a remove request to fail. A set of commands, as listed in Table 3-3, is provided to identify these situations so that scripts can be written to resolve them.

Table 3-3 DLPAR processes resource dependencies

ps bindprocessor kill ipcs lsrest lsclass chclass

Command	Purpose
Display bindprocessor attachments and block system call status at the process level.
Display online processors and make new attachments.
Send signals to processes.
Display pinned shared memory segments at the process level.
Display processor sets.
Display Workload Manager (WLM) classes, which might include processor sets.
Change WLM class definitions.

The drmgr command must be used to manage DLPAR scripts. The following functions are provided by the drmgr command:

Listing the registered DLPAR scripts and showing their information.
>Registering or uninstalling DLPAR scripts in the DLPAR script database.
>Changing the script install directory path. The default directory is /usr/lib/dr/scripts/all.

You can use the command drmgr -l to check all installed DLPAR scripts on the Partition. Example 3-4 illustrates checking all partitions by using the dsh command executed on the CSM Management Server.

Example 3-4 drmgr -l output for all nodes

#dsh -n pt,pl,uk,br "drmgr -l"

pt.itsc.austin.ibm.com: DR Install Root Directory: /usr/lib/dr/scripts pt.itsc.austin.ibm.com: Syslog ID: DRMGR pl.itsc.austin.ibm.com: DR Install Root Directory: /usr/lib/dr/scripts pl.itsc.austin.ibm.com: Syslog ID: DRMGR uk.itsc.austin.ibm.com: DR Install Root Directory: /usr/lib/dr/scripts uk.itsc.austin.ibm.com: Syslog ID: DRMGR br.itsc.austin.ibm.com: DR Install Root Directory: /usr/lib/dr/scripts br.itsc.austin.ibm.com: Syslog ID: DRMGR

For more detailed information about this topic, refer to IBM Redbooks publication Partitioning Implementations for IBM eServer p5 Servers, SG24-7039

Additionally, sample DLPAR scripts are provided in the directory /usr/samples/dr/scripts, which is included in the AIX fileset bos.adt.samples; refer to 4.3, AIX base operating system samples.