Updating IBM MQ Storage Size in CP4I

If you have already deployed your IBM MQ operator in IBM Cloud Pak for Integration (CP4I) but want to increase the total amount of storage available for Persistent Volume Claims of the three log types (if using PVC) then the correct method is to first edit the PV / PVC and then update the MQ Deployment YAML and apply it.

The three log types that can rely on a persistent volume claim are:

• Queue Manager – Holds data normally under /var/mqm. Will contain all persisted data and recovery logs, if no other volumes are specified.
• PersistedData – PersistentVolume details for MQ persisted data, including configuration, queues and messages. Required when using multi-instance Queue Manager
• RecoveryLogs - Persistent volume details for MQ recovery logs.

These must use persistent claims if you are running multi-instance. Here is an example of the Queue Manager specification that is deployed on CP4I. (This is a small snippet of the overall yaml)

spec:

queueManager:

name: QM01

storage:

queueManager:

type: persistent-claim

defaultDeleteClaim: false

persistedData:

enabled: true

size: 500M

class: rook-cephfs

type: persistent-claim

recoveryLogs:

enabled: true

size: 500M

class: rook-cephfs

type: persistent-claim

Example Recovery Log Storage Size Update

The following steps give an example set of steps of updating the recovery log which can be applied too any of the three MQ log types listed above.

1. Login to the OpenShift console as an Administrator
2. Navigate to Storage -> Persistent Volume Claim
3. In the search bar type "${queueManagerName}-ibm-mq-recovery-logs"
   e.g. qm01-ibm-mq-recovery-logs
4. Select the burger bar menu and select "Expand PVC"
5. Type the required value i.e. 700 MiB and select "Expand"
6. To validate the change: Navigate to the Workloads -> Pods
7. Select the 'Ready' qm01 pod which will be either qm01-ibm-mq-0 OR qm01-ibm-mq-1
8. Switch to the 'Terminal' view and run `df -h | grep mqm-logs` which will show the available storage has increased
   ${ip-address-and-port}:/volumes/csi/csi-vol-550e9a08-f915-11eb-9348-0a580afe140b/a1d2871a-aa2c-46cb-a422-19eca1ef0b36  700M   48M  852M   6% /mnt/mqm-log
   
9. It's important to ensure you update the deployment YAML (configuration as code) in order to have a consistent record of your queue manager configuration should you build a new environment etc. from the same config.
   
   Go to your deployment YAML for the Queue Manager which we have stored locally. 
10. Update the recovery log size to 700M and save the file. e.g.
    
    recoveryLogs:

    enabled: true

    size: 7000M

    class: rook-cephfs

    type: persistent-claim
11. On a terminal where you are already logged into the OpenShift cluster run the following command:
    `oc apply -f ${yamlfilename}.yaml` e.g. `oc apply -f my-qm.yaml`
    
    This should result in "queuemanager.mq.ibm.com/qm01 configured"

Note: updating the Queue Manager YAML before updating the persistent volume will lead to the Queue Manager showing a failed status in the MQ operator view, although the container will continue to run.

IBM MQ – AMQ3817E – DRBD Errors

 Recently when rebuilding an MQ Estate, the previous deployment was torn down and a new installation and configuration was deployed to the same physical Virtual Machines. However, when it came to creating the new Queue Manager there was an error thrown.

 

The reason for the rebuild was to upgrade the version to IBM MQv9.2 which includes some changes to the way RDQM pacemaker and drbd kernel modules are installed.

 

Command:

crtmqm -rr p -rt s -rl thisIP -ri otherIP -rn otherHostname -rp port -lla -fs 20 -lp 6 -ls 3 qmname

 

Error:

AMQ3817E: Replicated data subsystem call '/usr/sbin/drbdadm new-resourceqmname 0 --auto-promote=no' failed with return code '20'.Command 'drbdsetup new-resource qmname 0 --auto-promote=no' did not terminate within 5 seconds

AMQ3812E: Failed to create replicated data queue manager configuration.

 

To debug I did the following:

1.     Checked the MQ Error logs

2.     Check there were no outstanding processes that could be blocking the crtmqm command

3.     Check the correct drbd kernel module was installed on the server. (modinfo drbd)

 

There were no errors in the MQ Logs, no processes running for drbd and the correct kernel module was installed. Sufficiently stumped I ran lsmod drbd | grep drbd on the server and compared it to another environment.

 

There was no drbd_transport_tcp module on the output of the environment with the issues only the drbd module. When there are no RDQM Queue Managers running lsmod drbd | grep drbd hould return empty, which seems to suggest the drbd module was hanging. 

 

To resolve this there are two possible actions:

a.     Reboot the machine

b.     Remove the remaining ‘hanging’ drbd module using rmmod drbd

 

After completion, run lsmod drbd | grep drbd and the result should be empty. Try to build your Queue Manager again and it should create successfully.

A Refreshing problem to have...

So to begin, I'll set the scene.
I was creating a script to test an IIB set up i had/have. I was setting the HTTP port and the SOAP port to use the global listener rather than embedded listeners and also testing a IBM MQ message flow.

After creating the script, I deployed a bar file of three flows; one for a simple request and respond on MQ, one that sends a request to the HTTP port, and a final flow that sends a request for a SOAP wsdl.

Now the bar file deployed and the http request worked fine. Well done me. But upon testing the SOAP request I kept getting the error "The resource does not exist".

The test i was deploying was as such
       "curl -i http://localhost:myPort/mySoapTest?wsdl"
So i tested;

• The spelling of the command
• The flow inside the bar file
• The HTTP command for SOAP
• That the bar file wasn't corrupt
• That the execution group was running 
• The integration server was deployed
• The same command but at the default port of "7080"

VOILA. It worked. Which suggested the SOAP connector wasn't listening globally (to the broker).

To check i ran;

   "mqsireportroperties myBroker -e IVP -o ExecutionGroup -n soapNodesUseEmbeddedListener"

which returned "false" showing the soap node should be using the global port and not 7080.
To check this I ran the following command.

  "mqsireportproperties myBroker -b httplistener -o HTTPListener -a"

The results showed that HTTP Connector had two connectors. 

One of these connectors suggested that the HTTP port was set to the global port, the other was a SOAP connector and suggested it was still using its default port.

So it would appear, that the two are out of sync and in order to solve this problem, all i needed was a simple "refresh" to make sure all the connectors are up-to-date.

So my advice is this.

refresh MQ Queue Managers and IIB brokers after listener and port changes.*

*maybe after other things too... I'm yet to learn these.