Tuesday 7 September 2021

Microsoft Azure PowerShell Storage Creation Error - 403

I was recently completing a Microsoft Azure lab and was starting a new PowerShell window. Within the PowerShell setup page you need to set a storage account and file share in order to use the PowerShell. The specific (advanced) settings you need to select are:

  • Subscription
  • Cloud Shell Region
  • Resource Group
  • Storage Account
  • File Share
I already had a pre-defined subscription and resource group and the region appeared to be pre-selected based on my location (Europe). The next step is to create a new storage account and file share but pressing 'Create' gave a 403 error and the below error message.

{"error":{"code":"RequestDisallowedByPolicy","target":"the123cloud","message":"Resource 'the123cloud' was disallowed by policy. Policy identifiers: '[{\"policyAssignment\":{\"name\":\"Allowed locations\",\"id\":\"/providers/Microsoft.Management/managementGroups/temp/providers/Microsoft.Authorization/policyAssignments/1111111111111\"},\"policyDefinition\":{\"name\":\"Allowed locations\",\"id\":\"/providers/Microsoft.Authorization/policyDefinitions/22222222222222\"}},{\"policyAssignment\":{\"name\":\"Allowed locations - Exclude IoTCentral\...


The keywords in the error were "RequestDisallowedByPolicy" and "Allowed locations" which essentially means the Cloud Shell Region selected was disallowed either because it wasn't selected within the subscription policy OR was intentionally excluded.

  1. Select a valid region in the 'Cloud Shell Region' which is allowed
  2. Ask the administrator to allow the location in the subscriptions policies

Monday 9 August 2021

Updating IBM MQ Storage Size in CP4I

If you have already deployed your IBM MQ operator in IBM Cloud Pak for Integration (CP4I) but want to increase the total amount of storage available for Persistent Volume Claims of the three log types (if using PVC) then the correct method is to first edit the PV / PVC and then update the MQ Deployment YAML and apply it.

The three log types that can rely on a persistent volume claim are:

  • Queue Manager – Holds data normally under /var/mqm. Will contain all persisted data and recovery logs, if no other volumes are specified.
  • PersistedData – PersistentVolume details for MQ persisted data, including configuration, queues and messages. Required when using multi-instance Queue Manager
  • RecoveryLogs - Persistent volume details for MQ recovery logs.

These must use persistent claims if you are running multi-instance. Here is an example of the Queue Manager specification that is deployed on CP4I. (This is a small snippet of the overall yaml)

spec:
queueManager:
name: QM01
storage:
queueManager:
type: persistent-claim
defaultDeleteClaim: false
persistedData:
enabled: true
size: 500M
class: rook-cephfs
type: persistent-claim
recoveryLogs:
enabled: true
size: 500M
class: rook-cephfs
type: persistent-claim

Example Recovery Log Storage Size Update

The following steps give an example set of steps of updating the recovery log which can be applied too any of the three MQ log types listed above.

  1. Login to the OpenShift console as an Administrator
  2. Navigate to Storage -> Persistent Volume Claim
  3. In the search bar type "${queueManagerName}-ibm-mq-recovery-logs"
    e.g. qm01-ibm-mq-recovery-logs
  4. Select the burger bar menu and select "Expand PVC"
  5. Type the required value i.e. 700 MiB and select "Expand"
  6. To validate the change: Navigate to the Workloads -> Pods
  7. Select the 'Ready' qm01 pod which will be either qm01-ibm-mq-0 OR qm01-ibm-mq-1
  8. Switch to the 'Terminal' view and run `df -h | grep mqm-logs` which will show the available storage has increased
    ${ip-address-and-port}:/volumes/csi/csi-vol-550e9a08-f915-11eb-9348-0a580afe140b/a1d2871a-aa2c-46cb-a422-19eca1ef0b36  700M   48M  852M   6% /mnt/mqm-log
  9. It's important to ensure you update the deployment YAML (configuration as code) in order to have a consistent record of your queue manager configuration should you build a new environment etc. from the same config.

    Go to your deployment YAML for the Queue Manager which we have stored locally. 
  10. Update the recovery log size to 700M and save the file. e.g.
    recoveryLogs:
    enabled: true
    size: 7000M
    class: rook-cephfs
    type: persistent-claim
  11. On a terminal where you are already logged into the OpenShift cluster run the following command:
    `oc apply -f ${yamlfilename}.yaml` e.g. `oc apply -f my-qm.yaml`

    This should result in "queuemanager.mq.ibm.com/qm01 configured"

Note: updating the Queue Manager YAML before updating the persistent volume will lead to the Queue Manager showing a failed status in the MQ operator view, although the container will continue to run.

Thursday 29 July 2021

TLS Profiles in IBM API Connect v5 - Archive Post

This blog has been posted to retain an article written in September 2018 in my personal blog. Whilst it covers an older version of the product which is out of support in April 2022 it still might provide a useful reference for TLS Profiles for API Connect

Date: 2018/09/12 

by Aiden Gallagher & Peter Reeves (our podcast)

Introduction 

In API Connect - as in any system - transmission of data across a network can be protected and secured from tampering or theft using by using x509 key pairs in a key store and trust store. Messages can be secured for a number of reasons and in different ways, this provides:

  • Confidentiality - Encryption of messages to ensure they are not read by unintended parties.
  • Integrity - Signing messages to ensure they haven’t been altered in transit.
  • Non-Repudiation - Ensuring messages are sent by a specific author or program. 

In API Connect, a TLS Profile is created. This references: 

a) certificates to be presented as part of authentication, and 

b) trust stores to validate other applications’ certificates.

The TLS Profile has the settings for the TLS version and whether mutual authentication should be used.

Please see - TLS Profiles in APIC: Cloud Manager Console and API Manager at the bottom of this article for images of the TLS Profiles within API Connect.

This article will cover configuring TLS profiles on APIC - both for inbound and outbound traffic. The reader is expected to be comfortable with basic APIC topologies, the basic infrastructure of DataPower, and a basic understanding of x509 certificates. For introductions to these topics, see the below links:

https://www.ibm.com/support/knowledgecenter/en/SSMNED_5.0.0/com.ibm.apic.cmc.doc/ssl.html 
http://www.redbooks.ibm.com/redbooks/pdfs/sg247620.pdf

Authentication

When two applications wish to communicate with each other through an API Connect architecture, they can specify three different levels of authentication; 

No Authentication

Requires no Client-ID headers to be presented

One-way authentication 

Expects the sender to provide certificates that the receiver trusts. 

Mutual authentication. 

Requires both applications to present certificates that the other trusts.

 

No Authentication

Figure 1 - No Authentication

In Figure 1, an example is given of a request and response where no authentication is required or provided. 

  1. APP1 makes a request to APP2 across a network. 
  2. APP2 accepts the request and sends a response.

In this implementation, any application that has a network connection to APP2 is able to make a request. Providing they have the correct credentials, and a well-formed request, a response will be given. Messages sent without being signed are susceptible to being stolen or read during transit, and malicious requests may be made to APP2 such as a denial of service attack.

One-way Authentication using TLS Profiles

Figure 2 - One-way Authentication

In Figure 2, an example is given of a request and response where the receiving application expects the sender to provide a certificate which it trusts.

  1. APP1 makes a request to APP2. 
  2. APP2 establishes whether it trusts messages received from APP1.
    1. APP2 expects a certificate to be provided. APP1 has provided a certificate.
    2. APP2 validates whether the certificate is signed by a trusted Client Authority (CA).
  3. APP2 accepts the request and sends a response.

This level of authentication allows a server or receiving application to verify that an application making a request is trusted, and prohibits unauthorised applications gaining information it shouldn’t.

 For the client, this allows the request message to be protected through signing.  

Mutual Authentication 

Mutual Authentication works by a client and a server performing a handshake. This uses Secure Sockets Layer (SSL) and Transport Layer Security (TLS) Protocols which have a specific handshake procedure to describe how the two applications can authenticate each other for secure message sending and receiving.

Below is an overview of how the handshake works based on the section 7.4 of the TLS RFC: https://tools.ietf.org/html/rfc5246#section-7.4

Figure 3 - TLS Handshake

There are eleven steps for completing a TLS Handshake as shown in Figure 3 - TLS Handshake:

  1. Initially the server needs to make a ‘hello’ request to the client in order initiate the trust sequence. The two applications at this point can exchange security enhancement capabilities and reset configurations such as the hash and encryption algorithms. The server is entitled to make the request at any time and can repeat the process if it needs to but needs to know about the client before it can make this request.
  2. Typically, the server won’t know about the client until a connection is made, so for most first-time connections the first step will be for the client to send a ‘hello’ to the server as an introduction of itself to the server. After this first connection the client will respond with a ‘hello’ whenever the server makes a hello request.
  3. The server will respond to the client ‘hello’ with its own ‘hello’ with the purpose of choosing an algorithm that both the server and the client accept.
  4. The server will also send a certificate immediately following the server ‘hello’, except for anonymous negotiations. This provides the certificate chain of the server. The same structure of certificate will need to be used by the client later. 
  5. The server will also send a server key exchange if the certificate is not sent or does not provide enough data to complete the handshake. It should also only be sent providing the method is legal - RSA, DH_DSS, DH_RSA methods, for example, are not legal. 
  6. Providing that the negotiation is not anonymous, the server will request a client certificate. Anonymous servers requesting client certificates will fail. 
  7. The server tells the client it has finished sending all the information that it can or will send. At which point the client will validate that the server certificates are valid.
  8. If a client certificate had been requested - in step 6 - the client will provide a certificate chain. If there are no available certificates, the client will send a blank certificate which the server can choose to accept or reject for continuing the handshake process. 
  9. The client must then send their client key exchange, which provides the clients key information. This will either be an RSA-encrypted secret or a set of applicable parameters to approve a pre-master key with the server.
  10. At this point the client send a verification message for the associated client certificate.
  11. The server and the client now exchange cipher specifications and send a finished message. The two must confirm the messages received are the same. This then establishes a trusted connection which allows application data to be shared between the two applications.

Mutual Authentication TLS using TLS Profiles

Figure 4 - Mutual Authentication using TLS Profiles

In Figure 4 - Mutual Authentication using TLS Profiles, an example is given of mutual authentication when using TLS Profiles. 

  1. APP1 makes a request to APP2. 
  2. While invoking the TLS handshake, APP1 presents its certificates from the associated TLS Profile. The presented certificate is validated against APP2s Certificate Authorities (CA) trusted certificate chain stored in the TLS Profiles trust store.
  3. APP2 also presents its certificates to APP1 which completes the TLS handshake by also validating against the APP1 Certificate Authorities (CA) trusted certificate chain stored in the TLS Profiles trust store.
  4. Once the TLS Handshake is completed a response is given to any further requests whilst all the objects in both TLS Profiles remain valid.

A trust store may have one or all of the following three objects in its trust store certificate; 

  • Root CA - A root CA is a self-signed certificate that issues trusted certificates for use to applications and services. By having the Root CA in the trust store, implicit trust of any and all certificates signed by the Root CA is given. This leaves an application open to malicious use by any entity with access to a Root CA signed certificate.
  • Intermediate CA - To protect the Root CA keys, an intermediate CA can be issued and used in a trust store. This provides a link to the Root without exposing it. Using this CA gives a subset of applications access to the Root CA, reducing the risk of malicious connections to an application.
  • Common Name (CN) Certificate - Trusting a CN certificate means that traffic has to originate from a DNS domain that matches the CN. This gives a much more restricted trusted access to an application.

Each trust store object gives different layers of protection to the application which should be considered when creating the trust store certificate. 

In API Connect, the three authentication types need to be considered for all of the following situations;

  • When an application is calling an API deployed on the API Gateway Domain.
  • When an API on the DataPower Domain is calling another API locally.
  • When an API is calling out from the DataPower Domain to another application (externally hosted microservices, application, API, database etc.). 

TLS Profiles in API Connect

High Level Overview

  • Each API Connect DataPower Service creates a DataPower Domain on the servers included on the service (v5): https://www.ibm.com/support/knowledgecenter/en/SSMNED_5.0.0/com.ibm.apic.cmc.doc/config_gateway.html
  • A DataPower Service (Domain) must define a TLS Profile.
    • If a TLS Profile is not defined by the user, the default TLS Profile will be used.
  • Inbound traffic is managed by the TLS Profile that is referenced by the DataPower Services.
  • Outbound traffic is managed by TLS Profiles that are;
    • Defined, managed and made Public in the Cloud Manager Console - which makes them available to all Organisations and therefore all APIs. Caution: This will mean any Provider Organisation in the API Hub can use the TLS Profile and thus may have certificates to allow access to systems they should not have access to.
    • Defined and managed within an Organisation on API Manager.
  • APIs must include TLS Profiles in their swagger definitions/yaml. Please see Example API Yaml with TLS Profile at the bottom of the article.
  • TLS Profiles are stored on DataPower Servers within the Domain
  • TLS Profiles are expected in a p12 format
  • To be able to change a TLS Profile for a DataPower Service in the CMC, all servers must first be deleted from the Service.

To create a TLS Profile, follow the below knowledge centre instructions:
https://www.ibm.com/support/knowledgecenter/en/SSMNED_5.0.0/com.ibm.apic.cmc.doc/ssl.html 
https://www.ibm.com/support/knowledgecenter/en/SSFS6T/com.ibm.apic.apionprem.doc/task_apionprem_ssl.html 

API Connect Traffic Flows using TLS Profiles

In this section, some flows will show how TLS Profiles can be used within API Connect to secure traffic flowing through it and its associated objects. In each scenario, we will begin from a blank APIC topology featuring a single API Gateway with no APIs or products published, a single Management Server which entails both the APIM and CMC. Three applications are shown; a database, MicroService and a secondary Gateway, these represent any and all downstream applications that could be used.

Note: In the diagrams, Domains will be shown in the CMC where they are logically described and managed in APIC by an APIC administrator. In reality, the configuration is pushed to the API Gateway where the traffic flows through, not through the CMC or the Management Server. 

Default Settings

This scenario describes how API Connect protects traffic flow if used without any changes being made post-install.  

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. An API is published that calls a downstream service, which in turn connects to the system of record and responds with the required data.

Figure 5 - Default Settings

  1. API1 is published to the Dev Domain with no TLS Profile in the definition.
  2. APP1 calls API1 using one-way authentication by accepting the gateway’s self-signed certificate.
  3. API1 invokes an API on the Internal Gateway which does not enforce mutual authentication TLS for inbound traffic, thus the APIs request is processed without any TLS certificate on its part.
  4.  A response to the request is received from the system of record and passed back to APP1 through both the internal and APIC API Gateway. 

API Defined TLS Profile

This scenario describes how API Connect presents certificates to downstream applications by setting a TLS Profile in the API definition.  

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. An API is published that calls a downstream service which enforces one-way authentication. The downstream service then connects to the system of record and responds with the required data.

Figure 6 - API Defined TLS Profile

  1. API1 is published to the domain with a TLS Profile in the definition
  2. APP1 calls API1 using one-way authentication by accepting the gateway’s self-signed certificate.
  3. API1 calls the downstream service API presenting certificates in TLS Profile. The downstream service and API1 complete one-way authentication with the presented certificate validated against the downstream services trust store.
  4. A response to the request is received from the system of record and passed back to APP1 through both the internal and APIC API Gateway.

Domain Level Mutual Authentication

This scenario describes how API Connect completes mutual authentication for inbound traffic and presents certificates to downstream applications.  

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. An API is published that calls a downstream service which enforces one-way authentication. The downstream service then connects to the system of record and responds with the required data.

Figure 7 - Domain Level Mutual Authentication

  1. API1 is published to the domain with a TLS Profile in the definition. The TLS Profile is created in API Manager and does not enforce Mutual Authentication.
  2. APP1 calls API1 but must first complete a TLS Handshake.
  3. Mutual Authentication is successfully completed.
  4. API1 calls the downstream service API presenting certificates in TLS Profile 4. The downstream service and API1 complete one-way authentication with the presented certificate validated against the downstream service’s trust store.
  5. A response to the request is received from the system of record and passed back to APP1 through both the internal and APIC API Gateway.

Outbound Mutual Authentication

This scenario describes how API Connect completes Mutual Authentication TLS Handshake for downstream traffic.  

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. An API is published that calls a downstream service which enforces mutual authentication. The downstream service then connects to the system of record and responds with the required data.

Figure 8 - Outbound Mutual Authentication

  1. API1 is published to the domain with a TLS Profile in the definition. This TLS profile enforces mutual authentication on outbound connections to the microservice.
  2. APP1 calls API1 using one-way authentication by accepting the gateway’s self-signed certificate.
  3. API1 then invokes an API on a downstream microservice but must first complete a TLS Handshake.
  4. Mutual Authentication TLS handshake is completed successfully.
  5. A response to the request is received from the system of record and passed back to APP1 through the microservice and the APIC API Gateway. 

Total Mutual Authentication

This scenario describes how API Connect completes Mutual Authentication for both inbound and outbound traffic.

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. All communication in and out of the API Gateway must complete mutual authentication. 

Figure 9 - Total Mutual Authentication

  1. API1 is published to the domain with a TLS Profile in the definition.
  2. APP1 calls API1 but must first complete a TLS Handshake.
  3. Mutual Authentication is successfully completed.
  4. API1 then invokes an API on a downstream microservice but must first complete a TLS Handshake.
  5. Mutual Authentication TLS handshake is completed successfully.
  6. A response to the request is received from the system of record and passed back to APP1 through the microservice and the APIC API Gateway.

APIs calling same domain APIs

This scenario describes how API Connect completes Mutual Authentication with API calls made to other APIs on the same API Gateway.

 Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. All communication in and out of the API Gateway must complete mutual authentication. API1 uses OAuth, which is defined in API2, and uses mutual authentication due to the configuration settings on the API Gateway.

 

Figure 10 - APIs calling same domain APIs

  1. Both APIs are published to the domain with TLS Profiles in the definitions (API1 and the DataPower Service use a public TLS Profile which enforces MATLS).
  2. APP1 calls API1 but must first complete a TLS Handshake.
  3. Mutual Authentication is successfully completed.
  4. API1 then invokes API2 which resides on the same domain. It completes Mutual Authentication with the DataPower Domain – like an external application calling the domain as the Domain expects mutual authentication from all incoming traffic indiscriminately.
  5. API2 then invokes an API on a downstream microservice but must first complete a TLS Handshake.
  6. Mutual Authentication TLS handshake is completed successfully.
  7. A response to the request is received from the system of record and passed back to APP1 through the microservice and the APIC API Gateway.

Summary

Securing APIs and the traffic that flows through an API Connect architecture is an important task that needs considering early on in the design phases of an architecture. Enabling TLS and mutual authentication is always going to be harder in an existing infrastructure that is already serving traffic, so it is important to understand the flows and considerations as early as possible.

This article should give a good understanding of how API Connect traffic will flow and how to secure messages passing through the system in a secure way.


TLS Profiles in APIC

Cloud Manager Console

Figure 11 - TLS Profiles in the Cloud Manager Console 

API Manager

Figure 12 - TLS Profiles in API Manager

Figure 13 - TLS Profile Trust Store and Present Certificate

Example API Yaml with TLS Profile

Figure 14 - API Specification with TLS Profile


Tuesday 23 March 2021

If Software Development Has Peaked, why do Projects still fail?

by Aiden Gallagher & Peter Reeves (our podcast)

Introduction

In modern application and integration development it is easier than ever to configure, build, and deploy services and applications. The growing ease of use and maturity of infrastructure provisioning and software configuration systems are ensuring that prototypes, development, and production deployments can happen as quickly as possible.

Docker, Cloud, Kubernetes, Serverless deployment, autoscaling, and improvements to server virtualisation initiation all mean that a new program can be setup in days rather than weeks, that code can be pushed to production in hours rather than days and in many cases that the automation of many of these tasks means testing, operations, and security checks are already incorporated into the process. 

In many senses it feels like the development and deployment of software has peaked. But then, projects continue to fall behind schedule, they continue to cost more money than anticipated and they continue to contain bugs at the simplest of user flows. So, if software development has peaked, why do projects still fail?

Table of Contents

Article 1: Non-Software Considerations

  1. Partial Decision Makers
  2. Requirement Gathering is slow
  3. T Shaped Experts
  4. Simple does not Equal Cheap
  5. Cultural and Task Disparity Across Organizations
  6. Governance Perception
  7. Who ‘owns’ this work?
  8. Communication is Key

Article 2: Technical Considerations

  1. Over Engineering of Automation
  2. Infrastructure still exists
  3. Upgrade Acceleration
  4. Debugging takes time

Conclusion

Projects have greatly improved the time in which they can develop and release features and products, greater flexibility is possible through better working methods and product improvements and the skill required to develop, build, deploy and test code is decreasing as complexity is abstracted from users.


But projects continue to fail for a number of reasons including poor communication, an over reliance on expertise abstraction, poor cooperation with other teams and a lack of contextualisation of how projects and teams fit into wider business goals.


These issues can become overwhelming when not understood and exacerbated when ignored. But by having a knowledge of what might lead to problems in a project and knowing the warning signs teams can work to resolve the issues before or as they start to occur.