Unable to run package on node

Symptoms

When you try to bring up the package in service guard, the package wont coming up with below errors:

[[email protected] ~]# cmrunpkg <packagename>
Running package <packagename> on node node2
The package script for <packagename> failed with no restart. <packagename> should not be restarted
Unable to run package <packagename> on node node2
Check the syslog and pkg log files for more detailed information
cmrunpkg: Unable to start some package or package instances.

Its same also when we try to bring up the package on the other node.

Cause

When we look at to the logs file locate inĀ /usr/local/cmcluster/run/log/<packagename>.log, below errors found:

Sep 20 00:09:03 – Node “node2”: Exporting filesystem on /opt/apps
exportfs: internal: no supported addresses in nfs_client
exportfs: <ip_address>:/opt/apps: No such file or directory

exportfs: internal: no supported addresses in nfs_client
exportfs: <ip_address>:/opt/apps: No such file or directory

exportfs: internal: no supported addresses in nfs_client
exportfs: <ip_address>:/opt/apps: No such file or directory

exportfs: internal: no supported addresses in nfs_client
exportfs: <ip_address>:/opt/apps: No such file or directory

exportfs: internal: no supported addresses in nfs_client
exportfs: <ip_address>:/opt/apps: No such file or directory
ERROR: Function export_fs
ERROR: Failed to export -o rw @nfs1:/opt/apps
Sep 20 00:09:04 – Node “node2”: Unexporting filesystem on @nfs1:/opt/apps

## Failed to start package <packagename>, rollback steps #####
Sep 19 23:44:20 [email protected] tkit_module.sh[32107]: Install directory operation mode selected.
WARNING: Stoping rmtab synchronization proces: /usr/local/cmcluster/conf/<packagename>/sync_rmtab.PID does not exist
Sep 19 23:44:20 – Node “node2”: Unexporting filesystem on @nfs1:/opt/apps
exportfs: Could not find ‘@nfs1:/opt/apps’ to unexport.
ERROR: Function un_export_fs
ERROR: Failed to unexport @nfs1:/opt/apps

Sep 20 00:09:05 [email protected] master_control_script.sh[31933]: ###### Failed to start package for <packagename> ######

Check the status of services of nfs.

[[email protected] ]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
[[email protected]]#

The reason why the cluster packages wont start up is because the service of nfs is stopped and those need to be running up.

 

Resolutions

We may start the nfs services;

[[email protected]]# /etc/init.d/nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS mountd: rpc.mountd: svc_tli_create: could not open connection for udp6
rpc.mountd: svc_tli_create: could not open connection for tcp6
rpc.mountd: svc_tli_create: could not open connection for udp6
rpc.mountd: svc_tli_create: could not open connection for tcp6
rpc.mountd: svc_tli_create: could not open connection for udp6
rpc.mountd: svc_tli_create: could not open connection for tcp6
[ OK ]
Starting NFS daemon: rpc.nfsd: address family inet6 not supported by protocol TCP
[ OK ]
Starting RPC idmapd: [ OK ]

Verify the nfs service;
[[email protected]]# /etc/init.d/nfs status
rpc.svcgssd is stopped
rpc.mountd (pid 17790) is running…
nfsd (pid 17810 17809 17808 17807 17806 17805 17804 17803) is running…
rpc.rquotad (pid 17773) is running…

Then, the package can be run;
[[email protected]]# cmrunpkg <packagename>
Running package <packagename> on node node2
Successfully started package <packagename> on node node2
cmrunpkg: All specified packages are running
[[email protected]]#

Lastly, verify the status of packages in the cluster;

[[email protected] ~]# cmviewcl

CLUSTER STATUS
<clustername> up

SITE_NAME Site1_pri

NODE STATUS STATE
node1 up running

SITE_NAME Site2_sec

NODE STATUS STATE
node2 up running

PACKAGE STATUS STATE AUTO_RUN NODE
<packagename> up running disabled node2

##################################################