Updating IBM MQ Storage Size in CP4I

If you have already deployed your IBM MQ operator in IBM Cloud Pak for Integration (CP4I) but want to increase the total amount of storage available for Persistent Volume Claims of the three log types (if using PVC) then the correct method is to first edit the PV / PVC and then update the MQ Deployment YAML and apply it.

The three log types that can rely on a persistent volume claim are:

• Queue Manager – Holds data normally under /var/mqm. Will contain all persisted data and recovery logs, if no other volumes are specified.
• PersistedData – PersistentVolume details for MQ persisted data, including configuration, queues and messages. Required when using multi-instance Queue Manager
• RecoveryLogs - Persistent volume details for MQ recovery logs.

These must use persistent claims if you are running multi-instance. Here is an example of the Queue Manager specification that is deployed on CP4I. (This is a small snippet of the overall yaml)

spec:

queueManager:

name: QM01

storage:

queueManager:

type: persistent-claim

defaultDeleteClaim: false

persistedData:

enabled: true

size: 500M

class: rook-cephfs

type: persistent-claim

recoveryLogs:

enabled: true

size: 500M

class: rook-cephfs

type: persistent-claim

Example Recovery Log Storage Size Update

The following steps give an example set of steps of updating the recovery log which can be applied too any of the three MQ log types listed above.

1. Login to the OpenShift console as an Administrator
2. Navigate to Storage -> Persistent Volume Claim
3. In the search bar type "${queueManagerName}-ibm-mq-recovery-logs"
   e.g. qm01-ibm-mq-recovery-logs
4. Select the burger bar menu and select "Expand PVC"
5. Type the required value i.e. 700 MiB and select "Expand"
6. To validate the change: Navigate to the Workloads -> Pods
7. Select the 'Ready' qm01 pod which will be either qm01-ibm-mq-0 OR qm01-ibm-mq-1
8. Switch to the 'Terminal' view and run `df -h | grep mqm-logs` which will show the available storage has increased
   ${ip-address-and-port}:/volumes/csi/csi-vol-550e9a08-f915-11eb-9348-0a580afe140b/a1d2871a-aa2c-46cb-a422-19eca1ef0b36  700M   48M  852M   6% /mnt/mqm-log
   
9. It's important to ensure you update the deployment YAML (configuration as code) in order to have a consistent record of your queue manager configuration should you build a new environment etc. from the same config.
   
   Go to your deployment YAML for the Queue Manager which we have stored locally. 
10. Update the recovery log size to 700M and save the file. e.g.
    
    recoveryLogs:

    enabled: true

    size: 7000M

    class: rook-cephfs

    type: persistent-claim
11. On a terminal where you are already logged into the OpenShift cluster run the following command:
    `oc apply -f ${yamlfilename}.yaml` e.g. `oc apply -f my-qm.yaml`
    
    This should result in "queuemanager.mq.ibm.com/qm01 configured"

Note: updating the Queue Manager YAML before updating the persistent volume will lead to the Queue Manager showing a failed status in the MQ operator view, although the container will continue to run.

IBM MQ – AMQ3817E – DRBD Errors

 Recently when rebuilding an MQ Estate, the previous deployment was torn down and a new installation and configuration was deployed to the same physical Virtual Machines. However, when it came to creating the new Queue Manager there was an error thrown.

 

The reason for the rebuild was to upgrade the version to IBM MQv9.2 which includes some changes to the way RDQM pacemaker and drbd kernel modules are installed.

 

Command:

crtmqm -rr p -rt s -rl thisIP -ri otherIP -rn otherHostname -rp port -lla -fs 20 -lp 6 -ls 3 qmname

 

Error:

AMQ3817E: Replicated data subsystem call '/usr/sbin/drbdadm new-resourceqmname 0 --auto-promote=no' failed with return code '20'.Command 'drbdsetup new-resource qmname 0 --auto-promote=no' did not terminate within 5 seconds

AMQ3812E: Failed to create replicated data queue manager configuration.

 

To debug I did the following:

1.     Checked the MQ Error logs

2.     Check there were no outstanding processes that could be blocking the crtmqm command

3.     Check the correct drbd kernel module was installed on the server. (modinfo drbd)

 

There were no errors in the MQ Logs, no processes running for drbd and the correct kernel module was installed. Sufficiently stumped I ran lsmod drbd | grep drbd on the server and compared it to another environment.

 

There was no drbd_transport_tcp module on the output of the environment with the issues only the drbd module. When there are no RDQM Queue Managers running lsmod drbd | grep drbd hould return empty, which seems to suggest the drbd module was hanging. 

 

To resolve this there are two possible actions:

a.     Reboot the machine

b.     Remove the remaining ‘hanging’ drbd module using rmmod drbd

 

After completion, run lsmod drbd | grep drbd and the result should be empty. Try to build your Queue Manager again and it should create successfully.

Validating SSL configuration for IBM MQ using mqcertck

In the past in order to perform SSL checking we would have been required to verify manually that the key repository was in the correct location with the correct permissions. We would also have needed to verify the correct CHECKLABL parameter was being used.

As of IBM MQ version 8.0.0.4 (Not on the appliance) we can instead use the mqcertck command. This does the above testing from a previous MQ support pack. The command requires a queue manager is defined but also optionally allows the channel, key repository, port, a specific user or checklabl parameter to be defined and checked.

For full details see the knowledge centre;
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_8.0.0/com.ibm.mq.ref.adm.doc/q120895_.htm

I began to use this command for my Pure Application MQ and IIB testing to coincide with legacy verification tests used after the success of a deploy.. The first usage gave me the following output based on scripts that had been ported to Pure from legacy stand alone scripts.

  [mqm@XXXXXXXXXX root]$ mqcertck MQXXX01

  5724-H72 (C) Copyright IBM Corp. 1994, 2015.

  +----------------------------------------------------------

  | IBM MQ TLS Configuration Test tool                       

  +----------------------------------------------------------

  | mqcertck has ended. See above for any problems found. 

  | If there are problems then resolve these and run this   

  | tool again.                                             

  |                                                         

  +----------------------------------------------------------

This appears to suggest that there were no SSL/TLS configuration issues, thought there isn't a "successful completion" message. Being a firm believer that nobody is perfect, I wanted to test that all was well in the deploy configuration and that the above outcome is expected.

First I altered the queue manager CHECKLABL to a generic 'my labels' and got the following output when the command was rerun.

  echo "ALTER QMGR CERTLABL( 'mylabels' )" | runmqsc MQXXX01

  echo "REFRESH SECURITY TYPE(SSL)" | runmqsc MQXXX01

  [mqm@XXXXXXXXXX root]$ mqcertck MQXXX01

  5724-H72 (C) Copyright IBM Corp. 1994, 2015.

  +----------------------------------------------------------

  | IBM MQ TLS Configuration Test tool                       

  +----------------------------------------------------------

  | Problem identified:                                      

  |  No certificate could be found for the queue manager     

  |  MQXBK01 

  |                                                          

  | Advice:                                                  

  |  Queue managers will use a certificate with the label    

  |  set in the Queue Manager's CERTLABL attribute.          

  |  There is no certificate with the label mylabels 

  |  in the key repository being used by the queue manager   

  |  (/var/mqm/qmgrs/MQXXX01/ssl/MQXXX01.kdb) 

  |                                                          

  | Possible resolution:                                     

  |  A valid certificate with the label mylabels 

  |  needs to be added to the key repository.                

  |                                                          

  | Exceptions:                                              

  |  SSL channels being used to connect to another queue     

  |  manager do not need a personal certificate if the remote

  |  end of the channel at that queue manager has SSLCAUTH   

  |  (SSL client authentication) set to OPTIONAL.            

  |  This is true for channels which initiate the connection 

  |  e.g. sender or requester channels                       

  +----------------------------------------------------------

  | mqcertck has ended. See above for any problems found. 

  | If there are problems then resolve these and run this   

  | tool again.                                             

  |                                                         

  +---------------------------------------------------------- 

A certificate with the stated label - as expected - couldn't be found. So I reverted back to the original labels.

  echo "alter qmgr certlabl( ‘Original Label’ )" | runmqsc MQXXX01

  echo "REFRESH SECURITY TYPE(SSL)" | runmqsc MQXXX01

  [mqm@XXXXXXXXXX root]$ mqcertck MQXXX01

  5724-H72 (C) Copyright IBM Corp. 1994, 2015.

  +----------------------------------------------------------

  | IBM MQ TLS Configuration Test tool                       

  +----------------------------------------------------------

  | mqcertck has ended. See above for any problems found. 

  | If there are problems then resolve these and run this   

  | tool again.                                             

  |                                                         

  +----------------------------------------------------------

I then wanted to test if the command couldn't find the correct .kdb file so I went to the relevant directory and "mv"ed the .kdb to another name in the same location.

  cd /var/mqm/qmgrs/MQXXX01/ssl

  mv MQXXX01.kdb MQAAA01.kdb

  [mqm@XXXXXXXXXX root]$ mqcertck MQXXX01

  5724-H72 (C) Copyright IBM Corp. 1994, 2015.

  +----------------------------------------------------------

  | IBM MQ TLS Configuration Test tool                       

  +----------------------------------------------------------

  | Problem identified:                                      

  |  No key repository could be found for the queue manager  

  |  MQXBK01 

  |                                                          

  | Advice:                                                  

  |  Queue managers use the SSLKEYR attribute to identify the

  |  location of the SSL key repository to use.              

  |  No key repository file could be found at the location   

  |  specified in the queue manager's SSLKEYR attribute      

  |  (/var/mqm/qmgrs/MQXXX01/ssl/MQXXX01.kdb) 

  |                                                          

  | Possible resolution:                                     

  |  Alter the queue manager's SSLKEYR attribute to point at 

  |  the correct key repository and/or create a key          

  |  repository at the specified location.                   

  +----------------------------------------------------------

  | mqcertck has ended. See above for any problems found. 

  | If there are problems then resolve these and run this   

  | tool again.                                             

  |                                                          

  +----------------------------------------------------------

The output for this failure clearly states that the key repository as defined on the queue manager could not be found. When I moved the file back and ran the command again. The command again confirmed the SSL setup was correct.

mv MQAAA01.kdb MQXXX01.kdb

  [mqm@XXXXXXXXXX root]$ mqcertck MQXXX01

  5724-H72 (C) Copyright IBM Corp. 1994, 2015.

  +----------------------------------------------------------

  | IBM MQ TLS Configuration Test tool                       

  +----------------------------------------------------------

  | mqcertck has ended. See above for any problems found. 

  | If there are problems then resolve these and run this   

  | tool again.                                             

  |                                                         

  +----------------------------------------------------------

To conclude, when it comes to mqcertck: no news is good news.

IBM MQ Exits - Good To Knows

I recently co-created a security and message exit. The main code was written in C for a linux machine. The exit was later needed for a windows machine. Here are some of the things I learnt when migrating from a linux exit to windows and reminders to myself for the future.

- Windows exits are compiled into dlls.

 - Whilst unix machines will allow you to compile C using gcc with declarations made at will throughout the code. It is integral that all these declaration are made at the start of each function for the windows platform.

- Another key thing learnt was that whilst in many examples you may see exporting functions in windows in the following form "__declspec( dllimport )" or by declaring "C extern". Being new to security exits on windows I hadn't realised that the external functions needed to be declared in a definition file or a .def in the following format.

"EXPORTS

ExitName"

- Exits names are case sensitive.

- Whilst Security Exits are needed both on the sender and receiver channels, message exits are only required on one of the channels.

It is highly recommended that anyone creating Exits have a knowledge of both MQ and the operating system/coding language being used and takes the time to understand the implications of the code being used and the security issues that may arise.

There is a useful resource on the IBM Knowledge centre for compiling exits which can be found here:
http://www.ibm.com/support/knowledgecenter/SSFKSJ_7.5.0/com.ibm.mq.dev.doc/q028160_.htm

Creating a custom add on for IBM Pure Application (PureApp)

I wanted to define the uid and gid for the mqm user and mqm group in a pure application pattern. On deploy, if the mqm user doesn’t already exist the IBM MQ Advanced component will create the user for you with a set uid and gid.

In PureApp you can deploy scripts but these are only initiated after the components have been primarily installed which automatically sets the uid and gid. To set your own uid and gid you need to modify the user and groups id’s after the component is installed with the following commands in a script;

if [[ ${user} == "mqm" ]]; then

   echo "making directory /var/mqm owned by mqm"

   chown -R mqm:mqm /var/mqm ;

   find /var/mqm -gid 1414 -exec chgrp mqm {} \;

   find /opt/mqm -gid 1414 -exec chgrp mqm {} \;

fi

However, this would not be considered good practise. Instead you should use an add on which is implemented on initiation of the operating system node. There is an included add on called “default add user”

As you can see from the picture this allows you to define a user and a password. The problem is that the uid and the gid are set randomly by the operating system and for the user mqm the home directory would be /home/mqm but should be /var/mqm for the installation of MQ.

Instead I decided to make a custom add on by going to “catalog -> Add-ons” in pure and finding the current “Default add on” and downloading it from the parameter “add on package file”. This unpackaged zip has three files; an executable file which holds the commands to create the user, the cbscript json file which holds the properties that can be seen on the PureApp interface and a file called extended attributes which just has some extended attributes, you do not need to touch this file.

I altered the file names in the cbscript.json so that the PureApp system sees the new add on as a different add on and doesn’t try to rewrite the old, the important part here is to change "name": and/or "version":. See the image below for an example.

I then added parameters for the user to input; mqm_uid, mqm_gid, mqm_userhome, mqm_group and altered username to mqm_username. These parameter names must not collide with environmental pre-existing parameters, for example the parameter “uid” would cause a failure because the parameter “uid” is already on the operating system environment.

Then I edited the adduser executable file (and renamed it addmqmuser) so that it created a user based on the parameters given.
 

After saving the changes that had been made, I compressed the three files into a singular zip and called it addmqmuser.zip which can then be imported directly into the pure app system. Go to “catalog -> Add-ons” and select “create new” which asks for a zip file. Select import and voila the add on is ready to be used. You should be able to see it and use it on your pattern.

                                     
You can see if this has deployed on “pattern -> virtual system instances” finding the correct deploy and using the “+” next to the node name in the virtual machine perspective. It will be included in the “script packages” section and can be verified with a green tick.

CAUTION: These custom add ons should be made with care and are difficult to troubleshoot when they fail. You will want to test them on a “safe” environment (non-impact) before using them anywhere live.

By following these simple instructions you can now create an add on of your own which can set users and groups on the pattern interface with the parameters you need. 

A Refreshing problem to have...

So to begin, I'll set the scene.
I was creating a script to test an IIB set up i had/have. I was setting the HTTP port and the SOAP port to use the global listener rather than embedded listeners and also testing a IBM MQ message flow.

After creating the script, I deployed a bar file of three flows; one for a simple request and respond on MQ, one that sends a request to the HTTP port, and a final flow that sends a request for a SOAP wsdl.

Now the bar file deployed and the http request worked fine. Well done me. But upon testing the SOAP request I kept getting the error "The resource does not exist".

The test i was deploying was as such
       "curl -i http://localhost:myPort/mySoapTest?wsdl"
So i tested;

• The spelling of the command
• The flow inside the bar file
• The HTTP command for SOAP
• That the bar file wasn't corrupt
• That the execution group was running 
• The integration server was deployed
• The same command but at the default port of "7080"

VOILA. It worked. Which suggested the SOAP connector wasn't listening globally (to the broker).

To check i ran;

   "mqsireportroperties myBroker -e IVP -o ExecutionGroup -n soapNodesUseEmbeddedListener"

which returned "false" showing the soap node should be using the global port and not 7080.
To check this I ran the following command.

  "mqsireportproperties myBroker -b httplistener -o HTTPListener -a"

The results showed that HTTP Connector had two connectors. 

One of these connectors suggested that the HTTP port was set to the global port, the other was a SOAP connector and suggested it was still using its default port.

So it would appear, that the two are out of sync and in order to solve this problem, all i needed was a simple "refresh" to make sure all the connectors are up-to-date.

So my advice is this.

refresh MQ Queue Managers and IIB brokers after listener and port changes.*

*maybe after other things too... I'm yet to learn these.