TLS Profiles in IBM API Connect v5 - Archive Post

This blog has been posted to retain an article written in September 2018 in my personal blog. Whilst it covers an older version of the product which is out of support in April 2022 it still might provide a useful reference for TLS Profiles for API Connect

Date: 2018/09/12 

by Aiden Gallagher & Peter Reeves (our podcast)

Introduction 

In API Connect - as in any system - transmission of data across a network can be protected and secured from tampering or theft using by using x509 key pairs in a key store and trust store. Messages can be secured for a number of reasons and in different ways, this provides:

• Confidentiality - Encryption of messages to ensure they are not read by unintended parties.
• Integrity - Signing messages to ensure they haven’t been altered in transit.
• Non-Repudiation - Ensuring messages are sent by a specific author or program. 

In API Connect, a TLS Profile is created. This references: 

a) certificates to be presented as part of authentication, and 

b) trust stores to validate other applications’ certificates.

The TLS Profile has the settings for the TLS version and whether mutual authentication should be used.

Please see - TLS Profiles in APIC: Cloud Manager Console and API Manager at the bottom of this article for images of the TLS Profiles within API Connect.

This article will cover configuring TLS profiles on APIC - both for inbound and outbound traffic. The reader is expected to be comfortable with basic APIC topologies, the basic infrastructure of DataPower, and a basic understanding of x509 certificates. For introductions to these topics, see the below links:

https://www.ibm.com/support/knowledgecenter/en/SSMNED_5.0.0/com.ibm.apic.cmc.doc/ssl.html 
http://www.redbooks.ibm.com/redbooks/pdfs/sg247620.pdf

Authentication

When two applications wish to communicate with each other through an API Connect architecture, they can specify three different levels of authentication; 

No Authentication	Requires no Client-ID headers to be presented
One-way authentication	Expects the sender to provide certificates that the receiver trusts.
Mutual authentication.	Requires both applications to present certificates that the other trusts.

 

No Authentication

Figure 1 - No Authentication

In Figure 1, an example is given of a request and response where no authentication is required or provided. 

1. APP1 makes a request to APP2 across a network. 
2. APP2 accepts the request and sends a response.

In this implementation, any application that has a network connection to APP2 is able to make a request. Providing they have the correct credentials, and a well-formed request, a response will be given. Messages sent without being signed are susceptible to being stolen or read during transit, and malicious requests may be made to APP2 such as a denial of service attack.

One-way Authentication using TLS Profiles

Figure 2 - One-way Authentication

In Figure 2, an example is given of a request and response where the receiving application expects the sender to provide a certificate which it trusts.

1. APP1 makes a request to APP2. 
2. APP2 establishes whether it trusts messages received from APP1.
1. APP2 expects a certificate to be provided. APP1 has provided a certificate.
2. APP2 validates whether the certificate is signed by a trusted Client Authority (CA).
3. APP2 accepts the request and sends a response.

This level of authentication allows a server or receiving application to verify that an application making a request is trusted, and prohibits unauthorised applications gaining information it shouldn’t.

 For the client, this allows the request message to be protected through signing.  

Mutual Authentication 

Mutual Authentication works by a client and a server performing a handshake. This uses Secure Sockets Layer (SSL) and Transport Layer Security (TLS) Protocols which have a specific handshake procedure to describe how the two applications can authenticate each other for secure message sending and receiving.

Below is an overview of how the handshake works based on the section 7.4 of the TLS RFC: https://tools.ietf.org/html/rfc5246#section-7.4

Figure 3 - TLS Handshake

There are eleven steps for completing a TLS Handshake as shown in Figure 3 - TLS Handshake:

1. Initially the server needs to make a ‘hello’ request to the client in order initiate the trust sequence. The two applications at this point can exchange security enhancement capabilities and reset configurations such as the hash and encryption algorithms. The server is entitled to make the request at any time and can repeat the process if it needs to but needs to know about the client before it can make this request.
2. Typically, the server won’t know about the client until a connection is made, so for most first-time connections the first step will be for the client to send a ‘hello’ to the server as an introduction of itself to the server. After this first connection the client will respond with a ‘hello’ whenever the server makes a hello request.
3. The server will respond to the client ‘hello’ with its own ‘hello’ with the purpose of choosing an algorithm that both the server and the client accept.
4. The server will also send a certificate immediately following the server ‘hello’, except for anonymous negotiations. This provides the certificate chain of the server. The same structure of certificate will need to be used by the client later. 
5. The server will also send a server key exchange if the certificate is not sent or does not provide enough data to complete the handshake. It should also only be sent providing the method is legal - RSA, DH_DSS, DH_RSA methods, for example, are not legal. 
6. Providing that the negotiation is not anonymous, the server will request a client certificate. Anonymous servers requesting client certificates will fail. 
7. The server tells the client it has finished sending all the information that it can or will send. At which point the client will validate that the server certificates are valid.
8. If a client certificate had been requested - in step 6 - the client will provide a certificate chain. If there are no available certificates, the client will send a blank certificate which the server can choose to accept or reject for continuing the handshake process. 
9. The client must then send their client key exchange, which provides the clients key information. This will either be an RSA-encrypted secret or a set of applicable parameters to approve a pre-master key with the server.
10. At this point the client send a verification message for the associated client certificate.
11. The server and the client now exchange cipher specifications and send a finished message. The two must confirm the messages received are the same. This then establishes a trusted connection which allows application data to be shared between the two applications.

Mutual Authentication TLS using TLS Profiles

Figure 4 - Mutual Authentication using TLS Profiles

In Figure 4 - Mutual Authentication using TLS Profiles, an example is given of mutual authentication when using TLS Profiles. 

1. APP1 makes a request to APP2. 
2. While invoking the TLS handshake, APP1 presents its certificates from the associated TLS Profile. The presented certificate is validated against APP2s Certificate Authorities (CA) trusted certificate chain stored in the TLS Profiles trust store.
3. APP2 also presents its certificates to APP1 which completes the TLS handshake by also validating against the APP1 Certificate Authorities (CA) trusted certificate chain stored in the TLS Profiles trust store.
4. Once the TLS Handshake is completed a response is given to any further requests whilst all the objects in both TLS Profiles remain valid.

A trust store may have one or all of the following three objects in its trust store certificate; 

• Root CA - A root CA is a self-signed certificate that issues trusted certificates for use to applications and services. By having the Root CA in the trust store, implicit trust of any and all certificates signed by the Root CA is given. This leaves an application open to malicious use by any entity with access to a Root CA signed certificate.
• Intermediate CA - To protect the Root CA keys, an intermediate CA can be issued and used in a trust store. This provides a link to the Root without exposing it. Using this CA gives a subset of applications access to the Root CA, reducing the risk of malicious connections to an application.
• Common Name (CN) Certificate - Trusting a CN certificate means that traffic has to originate from a DNS domain that matches the CN. This gives a much more restricted trusted access to an application.

Each trust store object gives different layers of protection to the application which should be considered when creating the trust store certificate. 

In API Connect, the three authentication types need to be considered for all of the following situations;

• When an application is calling an API deployed on the API Gateway Domain.
• When an API on the DataPower Domain is calling another API locally.
• When an API is calling out from the DataPower Domain to another application (externally hosted microservices, application, API, database etc.). 

TLS Profiles in API Connect

High Level Overview

• Each API Connect DataPower Service creates a DataPower Domain on the servers included on the service (v5): https://www.ibm.com/support/knowledgecenter/en/SSMNED_5.0.0/com.ibm.apic.cmc.doc/config_gateway.html
• A DataPower Service (Domain) must define a TLS Profile.
• If a TLS Profile is not defined by the user, the default TLS Profile will be used.
• Inbound traffic is managed by the TLS Profile that is referenced by the DataPower Services.
• Outbound traffic is managed by TLS Profiles that are;
• Defined, managed and made Public in the Cloud Manager Console - which makes them available to all Organisations and therefore all APIs. Caution: This will mean any Provider Organisation in the API Hub can use the TLS Profile and thus may have certificates to allow access to systems they should not have access to.
• Defined and managed within an Organisation on API Manager.
• APIs must include TLS Profiles in their swagger definitions/yaml. Please see Example API Yaml with TLS Profile at the bottom of the article.
• TLS Profiles are stored on DataPower Servers within the Domain
• TLS Profiles are expected in a p12 format
• To be able to change a TLS Profile for a DataPower Service in the CMC, all servers must first be deleted from the Service.

To create a TLS Profile, follow the below knowledge centre instructions:
https://www.ibm.com/support/knowledgecenter/en/SSMNED_5.0.0/com.ibm.apic.cmc.doc/ssl.html 
https://www.ibm.com/support/knowledgecenter/en/SSFS6T/com.ibm.apic.apionprem.doc/task_apionprem_ssl.html 

API Connect Traffic Flows using TLS Profiles

In this section, some flows will show how TLS Profiles can be used within API Connect to secure traffic flowing through it and its associated objects. In each scenario, we will begin from a blank APIC topology featuring a single API Gateway with no APIs or products published, a single Management Server which entails both the APIM and CMC. Three applications are shown; a database, MicroService and a secondary Gateway, these represent any and all downstream applications that could be used.

Note: In the diagrams, Domains will be shown in the CMC where they are logically described and managed in APIC by an APIC administrator. In reality, the configuration is pushed to the API Gateway where the traffic flows through, not through the CMC or the Management Server. 

Default Settings

This scenario describes how API Connect protects traffic flow if used without any changes being made post-install.  

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. An API is published that calls a downstream service, which in turn connects to the system of record and responds with the required data.

Figure 5 - Default Settings

1. API1 is published to the Dev Domain with no TLS Profile in the definition.
2. APP1 calls API1 using one-way authentication by accepting the gateway’s self-signed certificate.
3. API1 invokes an API on the Internal Gateway which does not enforce mutual authentication TLS for inbound traffic, thus the APIs request is processed without any TLS certificate on its part.
4.  A response to the request is received from the system of record and passed back to APP1 through both the internal and APIC API Gateway. 

API Defined TLS Profile

This scenario describes how API Connect presents certificates to downstream applications by setting a TLS Profile in the API definition.  

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. An API is published that calls a downstream service which enforces one-way authentication. The downstream service then connects to the system of record and responds with the required data.

Figure 6 - API Defined TLS Profile

1. API1 is published to the domain with a TLS Profile in the definition
2. APP1 calls API1 using one-way authentication by accepting the gateway’s self-signed certificate.
3. API1 calls the downstream service API presenting certificates in TLS Profile. The downstream service and API1 complete one-way authentication with the presented certificate validated against the downstream service’s trust store.
4. A response to the request is received from the system of record and passed back to APP1 through both the internal and APIC API Gateway.

Domain Level Mutual Authentication

This scenario describes how API Connect completes mutual authentication for inbound traffic and presents certificates to downstream applications.  

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. An API is published that calls a downstream service which enforces one-way authentication. The downstream service then connects to the system of record and responds with the required data.

Figure 7 - Domain Level Mutual Authentication

1. API1 is published to the domain with a TLS Profile in the definition. The TLS Profile is created in API Manager and does not enforce Mutual Authentication.
2. APP1 calls API1 but must first complete a TLS Handshake.
3. Mutual Authentication is successfully completed.
4. API1 calls the downstream service API presenting certificates in TLS Profile 4. The downstream service and API1 complete one-way authentication with the presented certificate validated against the downstream service’s trust store.
5. A response to the request is received from the system of record and passed back to APP1 through both the internal and APIC API Gateway.

Outbound Mutual Authentication

This scenario describes how API Connect completes Mutual Authentication TLS Handshake for downstream traffic.  

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. An API is published that calls a downstream service which enforces mutual authentication. The downstream service then connects to the system of record and responds with the required data.

Figure 8 - Outbound Mutual Authentication

1. API1 is published to the domain with a TLS Profile in the definition. This TLS profile enforces mutual authentication on outbound connections to the microservice.
2. APP1 calls API1 using one-way authentication by accepting the gateway’s self-signed certificate.
3. API1 then invokes an API on a downstream microservice but must first complete a TLS Handshake.
4. Mutual Authentication TLS handshake is completed successfully.
5. A response to the request is received from the system of record and passed back to APP1 through the microservice and the APIC API Gateway. 

Total Mutual Authentication

This scenario describes how API Connect completes Mutual Authentication for both inbound and outbound traffic.

Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. All communication in and out of the API Gateway must complete mutual authentication. 

Figure 9 - Total Mutual Authentication

1. API1 is published to the domain with a TLS Profile in the definition.
2. APP1 calls API1 but must first complete a TLS Handshake.
3. Mutual Authentication is successfully completed.
4. API1 then invokes an API on a downstream microservice but must first complete a TLS Handshake.
5. Mutual Authentication TLS handshake is completed successfully.
6. A response to the request is received from the system of record and passed back to APP1 through the microservice and the APIC API Gateway.

APIs calling same domain APIs

This scenario describes how API Connect completes Mutual Authentication with API calls made to other APIs on the same API Gateway.

 Use Case: A consumer Application (APP1) wishes to get information from a record owned by the provider company. All communication in and out of the API Gateway must complete mutual authentication. API1 uses OAuth, which is defined in API2, and uses mutual authentication due to the configuration settings on the API Gateway.

 

Figure 10 - APIs calling same domain APIs

1. Both APIs are published to the domain with TLS Profiles in the definitions (API1 and the DataPower Service use a public TLS Profile which enforces MATLS).
2. APP1 calls API1 but must first complete a TLS Handshake.
3. Mutual Authentication is successfully completed.
4. API1 then invokes API2 which resides on the same domain. It completes Mutual Authentication with the DataPower Domain – like an external application calling the domain as the Domain expects mutual authentication from all incoming traffic indiscriminately.
5. API2 then invokes an API on a downstream microservice but must first complete a TLS Handshake.
6. Mutual Authentication TLS handshake is completed successfully.
7. A response to the request is received from the system of record and passed back to APP1 through the microservice and the APIC API Gateway.

Summary

Securing APIs and the traffic that flows through an API Connect architecture is an important task that needs considering early on in the design phases of an architecture. Enabling TLS and mutual authentication is always going to be harder in an existing infrastructure that is already serving traffic, so it is important to understand the flows and considerations as early as possible.

This article should give a good understanding of how API Connect traffic will flow and how to secure messages passing through the system in a secure way.

TLS Profiles in APIC

Cloud Manager Console

Figure 11 - TLS Profiles in the Cloud Manager Console 

API Manager

Figure 12 - TLS Profiles in API Manager

Figure 13 - TLS Profile Trust Store and Present Certificate

Example API Yaml with TLS Profile

Figure 14 - API Specification with TLS Profile

If Software Development Has Peaked, why do Projects still fail?

by Aiden Gallagher & Peter Reeves (our podcast)

Introduction

In modern application and integration development it is easier than ever to configure, build, and deploy services and applications. The growing ease of use and maturity of infrastructure provisioning and software configuration systems are ensuring that prototypes, development, and production deployments can happen as quickly as possible.

Docker, Cloud, Kubernetes, Serverless deployment, autoscaling, and improvements to server virtualisation initiation all mean that a new program can be setup in days rather than weeks, that code can be pushed to production in hours rather than days and in many cases that the automation of many of these tasks means testing, operations, and security checks are already incorporated into the process. 

In many senses it feels like the development and deployment of software has peaked. But then, projects continue to fall behind schedule, they continue to cost more money than anticipated and they continue to contain bugs at the simplest of user flows. So, if software development has peaked, why do projects still fail?

Table of Contents

Article 1: Non-Software Considerations

1. Partial Decision Makers
2. Requirement Gathering is slow
3. T Shaped Experts
4. Simple does not Equal Cheap
5. Cultural and Task Disparity Across Organizations
6. Governance Perception
7. Who ‘owns’ this work?
8. Communication is Key

Article 2: Technical Considerations

1. Over Engineering of Automation
2. Infrastructure still exists
3. Upgrade Acceleration
4. Debugging takes time

Conclusion

Projects have greatly improved the time in which they can develop and release features and products, greater flexibility is possible through better working methods and product improvements and the skill required to develop, build, deploy and test code is decreasing as complexity is abstracted from users.

But projects continue to fail for a number of reasons including poor communication, an over reliance on expertise abstraction, poor cooperation with other teams and a lack of contextualisation of how projects and teams fit into wider business goals.

These issues can become overwhelming when not understood and exacerbated when ignored. But by having a knowledge of what might lead to problems in a project and knowing the warning signs teams can work to resolve the issues before or as they start to occur. 

If Software Development Has Peaked, why do Projects still fail? Part 2

by Aiden Gallagher & Peter Reeves (our podcast)

Technical Considerations

There are technical specific considerations that cause projects to fail too. Such as software and hardware build and deploy processes. Sometimes it’s simple errors that haven’t been accounted for, other times it might be bad technical implementation estimates or a high knowledge gap to fill. But other issues are less obvious:

 1. Over Engineering of Automation

Automation is the golden standard of modern, agile development. The more consistent and quicker something can be taken to production the better. This might be catching issues earlier to reduce overall cost to test in each environment or having an automated rollback strategy that means issues can quickly be reverted or new features rolled out to a small sample size.

But getting overly hung up on automation can lead to setbacks. Forcing automation without a real use case can eat up valuable project time. For example, automating a yearly update that takes 5 minutes to complete is hardly worth the effort.

Additionally, where automation is too complex there can be little gain and lots of pain. Take automating a test that requires 5 or 6 different applications that regularly change, or where there is no control of the connecting service e.g. scraping a webpage. The overall effort to complete is more hassle than the test itself.

There are also other areas which still cannot be automated, like user acceptance testing for flow ease of use and ‘obviousness’. Or user test cases that require negative testing – all the permutations of which cannot be known, else they would be covered in the initial testing. 

Remediation:

• Complete an analysis of savings vs cost to complete
• Consider the complexity of the automation and what might be missed e.g. UAT we might be able to pass all technical flows but not visual tests. If it doesn’t feel intuitive or right, then people won’t want to use the application/site.
• Be comfortable replacing automation with manual tests if the automation is taking too much time or fails regularly- this time should be factored back into the savings/cost analysis

Warning Signs:

• Automation tasks overrunning
• Lots of failures in the automation themselves that require team time to fix

2. Infrastructure still exists

Projects continue to fail because somewhere infrastructure still exists. The problem may be obfuscated away from the project team by being on a Software as a Service (SaaS) offering or controlled by a separate team, but failures and considerations of the infrastructure are still highly relevant to the project.

There’s also the hosting of the systems, moving to a cloud based off-premises network might mean severe limitations to the boundaries of the project. For example, an organisational project that wants to start working with a UK government entity might not be eligible if all the infrastructure lives in European datacentre. This can be resolved but there will a cost associated from switching providers whether it’s just one app or many as security, connectivity and skilling up are required.

As with all externalising of responsibilities to allow teams to focus on specific solutions - including with internal virtualization teams - there is a loss of control of system version, upgrade strategy and when outages will or might take place. If a project needs the latest version of Linux to create a new feature or expand an existing feature, the project success timelines might be pushed back whilst awaiting a convenient time for the external team to action.

There is also the case where an off-project infrastructure ownership can inhibit flexibility. A couple of examples might be, the ability to store a file such as a private key locally,  the ability to integrate with certain software such as an Active Directory Group. 

Remediation:

• Make applications that are easily portable to other infrastructure providers
• Understand Infrastructure limitations and requirements during design phases. This is especially important when doing iterative designs, so the constraints and boundaries are well understood going forward
• Have alerts for infrastructure changes or updates setup that the team can evaluate any effects as needed

Warning Signs:

• No mention of infrastructure in the design
• Failure due to changes at the infrastructure level

3. Upgrade Acceleration

Many modern applications pride themselves on their ability to push out new updates very quickly with daily or even weekly changes to production code. This works fine until the project is tied to the same upgrade cycle as a connecting application. Take Kubernetes for example, which used to provide its version support for 9 months; up to 12 months at the time of writing. 

This means there will definitely be an upgrade required in the next 12 months, of which, the latest version could be completely different to the one being used. Different mechanisms for assigning storage, managing deployments etc. are hard to predict and do not offer the stability that might be required for some projects.

Given an example where a customer feedback form is hosted in a container on Kubernetes, after a year the support expires, but it took 2 months to productionise, 3 months to rollout to all customers after the new feature is proven to work and be of value. A normal project might want to put the feedback form into a legacy status and just feed and water it to update logos etc. However, 4 months in the team needs to divert resources to upgrade the Kubernetes version to a supported version.

A perfectly working app might ordinarily have been left for a long time, but short support timelines mean imposed additional work where the upgrade could have serious impacts on the feature/application. This increases the cost of the project and reduces the quantifiable monetary benefits.

This also extends to any dependencies which are subject to lots of change, and any integrating applications with short release cycles. This also makes it hard for application teams to support as the inner workings of the system e.g. Kubernetes changes too fast to stay up to date.

The final part of this problem is the impact it has on workers who are expected to become experts on the systems, this is more difficult when it is under constant change and when combined with a requirement to have a much broader knowledge base. 

Remediation:

• Include upgrade cadence and complexity into solution design and selection
• Create automation tools for upgrades that are maintainable and manageable by long term support teams
• Select long term support options where possible

Warning Signs:

• Multiple upgrade requirements over a short period of time e.g. a couple of development cycles
• Lots of failures when performing upgrades 

4. Debugging takes time

With increased deployment velocity achieved in lean and agile projects, we are able to deploy quicker into production. But when bugs occur, they can be difficult to debug and understand. This might take time and may require that the production system is reserved for failure understanding only. If this time hasn’t been accounted for the whole project timescale can be knocked off balance.

In projects where debugging issues and fixing errors haven’t been accounted for, the onus to address defects can be pushed onto developers and testers who are forced to rush workload. This leads to further failures and an exacerbation of new, avoidable bugs getting into Production.

This can be alleviated by code ownership in production by the developing team. Project teams will be more likely to speak up and stop bad behaviours if they own the code in production, but this relies on the understanding at a management level.

Another issue can be caused when implementing ‘hot fixes’ based around assumptions of failure. This usually occurs because of a lack of time to debug and understand a problem. It also relies on an assumption that automated tests work and are up to date which requires trust. The less time a developer has to debug and get a fix that works, the less trust there will be, the more issues that will arise and the more likely for a project to fail. 

Remediation:

• Ensure issues are well understood before implementing fixes by testing issue hypothesis rigorously and keeping logs for long term analysis if required.
• Plan in time for defect management either a dedicated resource or dedicated time for the wider team

Warning Signs:

• Bug fixes are getting into production but are ending up back with developers as issues continue
• Unable to meet current cycle plan because of handling defects which suggests inadequate planning for defect
• Fixes without adequate problem determination

If Software Development Has Peaked, why do Projects still fail? Part 1

by Aiden Gallagher & Peter Reeves (our podcast)

Non-Software Considerations

There are lots of reasons a project might fail, but not all of them are software related, nor are they the sole dependencies of the technical members of a team. The following concerns are based on observation from projects that have failed before.

1. Partial Decision Makers

The first stage in any project is to get an understanding of the business goals. This isn’t necessarily requirement gathering right away but simply ensuring that the value of the project and the expected return on investment are understood. It also means understanding what, if not done, will cause those goals or the project itself to become uneconomical.

In one example, a car sales company has an implementation team with two tasks that they need to complete

• A new application to allow users to ‘test drive’ the car in an interactive application. The new app will increase sales by 30% to about £300,000 per year.  
• Updating how user order changes are sent to manufacturers from a manual email to automatic real-time event sending. This will save 5 workdays a month at around £10,400 savings per year. 

If the team has limited resources but not key information such as knowledge of the business goals or financial implications. For example, when deciding which task to prioritise, if the technical SMEs on the ground do not know that each day where the new app is not functional the business is losing money. 

However, the team regularly ‘feel’ the impact of having to send order updates via email which is time consuming and error prone. They think this is a relatively simple implementation that will save someone more time that could be allocated to other projects such as the new app.

The team will pick the emotionally preferable task as opposed to what matches business and financial goals. What seems like a sound decision based on technical facts has a secondary impact on the business.

 Remediation:

• Put deliverables into context for the implementation team including why it’s important for the business and the impact it will have e.g. This new feature will bring $X value
• Ensure improvement tasks are quantified e.g. adding a pipeline to add a new security key will save us 3 days per quarter and will take 10 days to complete
• Ensure task planning take goal importance into consideration and make them real and relatable e.g. ‘Get NewProject into Production’ is a technical goal, but the business context might be NewProject needs to have 100 users by the next Sprint to be profitable. The quicker it’s in production the more likely the ability to meet the goal which should drive planning.

Warning Signs:

• During planning meetings or informal discussions the team seem confused about the approach or question the sensibility of the plan. If there is disconnect it suggests the business goals related to each task isn’t well understood

2. Requirement Gathering is slow

Once a new task has been assigned to a team, it takes time to get all the requirements. However, in more agile projects, getting code into Production quickly and securely is a key benefit as is being able to pivot the strategy based on updated requirements. 

For example, a new feature lets a user set their address when they sign up for services, but feedback shows the user needs to be able to update the address; the requirements are that the update should be instantaneously pushed to the database.

The implementation team need to know what requirements they have on the database and vice versa but getting access to the specialists can be difficult as they are already committed to other project deliverables. 

Testing requires both teams to provide resources, access control is little understood by the new development team and as development begins changes might be needed which do not work with the system being integrated with. 

In the database example, the database team confirm that an update can be made to the database, the application team allow unlimited daily updates so if a mistake is made the new address can be resubmitted. However, the database processes data in batches with a first in, last out process which means the first - wrong address - is updated to the records.

Small gaps in product functionality, product knowledge and an inability to get the time of the relevant teams or specific members of the team exacerbate this problem, especially when applications are integrating with lots of other systems.   

Remediation:

• Define the limits of integrated systems so the design can be iteratively updated but within the confines of what is possible e.g. the database can accept any inbound data format but has a hard limit of 1000 characters and 1MB.
• For features, products and projects requiring external integration with another team, line of business or organisation then a dedicated resource should be requested to ensure iterative design can be reviewed at the same cadence as changes
• For new or unfamiliar features and products, the team; designers, developers and architects should be given sufficient time to prepare and gain education/experience. Where relevant and possible, the team should look to ‘loan’ relevant resources from within the organization
• Lock-in requirements for development cycles and allow new and updated requirements at the next design and development iteration. If this isn’t possible the current development cycles should be pushed back to a design cycle. This reduces changes further in the route-to-live workflow.

Warning Signs:

• Trivial issues such as size limits or unsuitable data formatting are being found in the development or testing phases. This suggests that design and architect team members are not familiar or up to date with requirements
• Team raises that there is little knowledge of a certain area
• The team are handling lots of new items with lots of unknowns e.g. lots of new versions of products are coming into play at the same time. 

3. T Shaped Experts

It is becoming easier and easier to begin working with a product with only a little deep knowledge of it, its features, or the platform it is deployed to. A subject matter expert who once specialised in a single product might now be expected to handle a wider spectrum of associated tasks including security, testing and integration with other tools. There is also a requirement to understand and be able to deploy in Kubernetes, apply relevant checks and measures through into production and provide support in production whilst developing new features.

Not all SMEs are able to quickly build up knowledge of lots of different products and have the ability to design and implement them to the same standard that a specialised SME could do, this might mean that good practices (learnt over time) are not incorporated causing later problems in the project. In theory with quickly moving teams, the same thing could be replicated multiple times over a product, project or applications life as the same ‘lesson’ is learnt by new people.

To accommodate this, small teams with a common ‘wide’ knowledge base and individual deep technical skills work together to form a robust skill base. E.g. an integration SME, an API SME, a security SME, a product support SME collectively complete the tasks on a team. But finding the right people can be hard.

Remediation:

• Provide sufficient time and resources for skilling up in new technologies
• Where possible, allow SMEs to be placed on programs and projects that need them e.g. pooling a certain skill type (cryptography) rather than expecting everyone to have a deep knowledge of it
• Communicate with the team to find interesting roles for each team member, especially when learning new skills. For example, by allowing team members to self-select tasks during planning. 
• Allow for longer development and delivery timelines when team members are learning or unfamiliar with a concept or feature
• Pair team members with more skilled members assisting less skilled members on their deep skill subject 

Warning Signs:

• The team is not performing as well as previously e.g. delivering at the same cadence
• Lots of small issues are arising that have been found when other team member performed similar tasks. This suggests a new team member is learning something the team has skills to perform, the new team member should be assisted by experience.
• Members of the team are showing fatigue or high levels of stress. They may even have told a team mate they are struggling.

4. Simple does not Equal Cheap

There is an expectation that when something sounds simple it should be easy and therefore cheap. Take for example, buying coffee from an online coffee delivery service. There are three functions; selecting your order, paying for your order, planning a delivery route to ensure coffee remains hot and arrives in accordance with Service Level Agreements (SLAs).

Selecting and processing of an order seems the simplest. However, simple in itself does not mean the design itself is simple; the stock list needs to be up to date, the online application needs to synchronise with the in-store systems, the online system needs a queueing system to handle demand throughput, the order has now been paid for but needs to remain on the system until delivered meaning the system and its messages needs to be highly available.

The feature itself whilst seemingly ‘simple’ has lots of other considerations at the technical level which are not always immediately obvious. This either means things are initially missed meaning lots of retrofitting after launch, or lots of prework to ensure the system functions efficiently and as expected, all which all has a cost. 

 Remediation:

• Ensure that all project members including management and delivery leads have a good understanding of the effort requirements relating to different pieces of work
•  Trust technical leads on the considerations, requirements and need to understand certain tasks
• Perform rigorous planning sessions to ensure all the work for a task is understood
• Avoid planning with preconceptions of ‘simple’ – lots of simple tasks can take a long time

Warning Signs:

• Lots of unplanned work is being scheduled
• Tasks are taking longer than originally planned for
• Technical timings for tasks are being driven by business requirements e.g. Feature needs to be in by next week, but with current resources the estimate is two weeks’ worth of work 

5. Cultural and Task Disparity Across Organizations

At an organisational level, all goals are generally aligned at the highest level; make a profit, open new stores, increase sales, improve costs etc. but on a team by team and project by project level the prioritisation of these goals varies.

As with “Partial Decision Makers”, an infrastructure provisioning team charged with decreasing costs by 10% may not see a new feature project as high a priority as server utilisation work. This might be because there isn’t a wider understanding of the organisations primary goals and each projects advancement of those goals, or at times a differing opinion of what is going to provide the best results.

When this involves separate organizations and the two beginning to collaborate to integrate systems, a problem arises where both organizations don’t have the business context of the teams that they are working with. If there is more value for one team than the other the collaboration and drive to succeed don’t marry well and can lead to delays or work not completed efficiently or effectively. 

A separate concern is when the culture within teams, whether cross-organization or internal, are very different. This could be work hours, lead times to complete work, processes that are seen as too quick (fragile or insecure) or too process ridden (wasteful or costly) and even the way communication takes place. Think of an Agile or Lean team versus a more traditional waterfall implementation, where on one side changes should be possible quickly with new evidence, on the other careful planning ensures budget is well understood and teams do not get overloaded. 

Remediation:

• Agree methods of working ahead of project initiations such as: how often people come into the office, general availability hours (10am – 3pm), communication techniques etc.
• Complete an all-hands to allow each team to describe what they are working on and more important ‘why’. What is the business benefit and how will it impact others and customers? Try not to make this “We completed this task ahead of time” but instead “We completed this task ahead of time which means we saved on costs and customers should not get X benefit.”
• Do a review of past projects to show what tangible benefits came from them in the long term e.g. over a year, two years etc.

Warning Signs:

• Required resources are not available when needed
• Two teams/projects have been given goals that do not have aligned solution or implementation, but one is reliant on the other
• A non-dedicated resource is provided. Usually this is because a resource is pooled or responds only to tickets. This can mean issues aren’t resolved quickly because there isn’t the capacity to do so.

6. Governance Perception

Governance is fast becoming embedded into more modern deployment strategies. The introduction of ‘production like’ as early as possible has always been a goal, but often the red tape has now been cut up to allow for faster route-to-live. In some cases the impact of not following these processes can be felt for years and in more extreme cases the lifespan of the project.

Site reliability engineers are more often being embedded into project teams becoming more flexible as they understand first-hand the development requirements and changes. This means processes are embedded and refined to ensure maximum compliance and forward planning for production support whilst removing pointless processes that genuinely block progress or agility.

However, some of the processes are still insisted upon – rightly – because of the experience of previous deployments which are common no matter what development style has been used to generate the features, how flexible your systems are or how many approvals are needed. 

When these are continued to be seen as ‘blockers’, ‘delays’ or ‘pointless’ and common-sense governance practices are ignored, then a program becomes more likely, and maybe even inevitably likely to fail.

 Remediation:

• Promote the retrospective sharing amongst teams of governance pitfalls that have been met and solutions that have overcome them
• Review governance processes regularly to validate the precaution cost against the risk cost. If the precaution is more expensive can be it be scaled back to a cheaper alternative. 
•  Invite site reliability engineers, security engineers and production support to review the processes in place. Are they more or less restrictive? Have better systems been put in place elsewhere?

Warning Signs:

• Release of code is taking more time than the design and development
• Work in progress is getting ‘stuck’ at governance steps with high ‘time in activity’ e.g. An approval process taking several days.
• Valid governance processes are causing discontent amongst the team. It might be that the risk cost is not understood or that the precautions are overly restrictive.

7. Who ‘owns’ this work?

Scope creep has and probably always will be a major issue in the planning and implementation of a project (see Requirement gathering is slow). This doesn’t change with Agile which, although flexible, is better when focused on specific aims, new tasks inserted two days into a new sprint can cause havoc and mean a rapid realignment of other expectations.

When new tasks are combined with collaboration with many teams, the who and when element of the task can make for a constrained relationship. This might mean other work suffers but any hostility from the stress of change can damage cooperation in the future and the general progress of a project.

This usually can be resolved by planning for the ‘unknown’ and fostering positive relationships. This also requires a level of reasonableness, in that an acceptance that new work will likely mean a push out on timeline of current work or an increased cost to complete the new work to the same timelines.

In lean enterprise this can be alleviated if the same team manages from development through to Production and if changes are contained with less close coupling. 

Remediation:

• Be open to new requirements but ensure there is a process for getting them added to the backlog of work to be properly measured and dependencies managed with the relevant teams
• Reduce dependencies on other teams as much as possible. Teams with greater internal control - either because they own or have the ability to develop in their own isolated space - can accommodate new requirements easier. However, the team still need to know how to do the work
• Allow headroom for unforeseen tasks in the planning. This might mean having some resources every cycle available to down tools and help with defects, new requirements etc.

Warning Signs:

• New project announced
• Change of organizational direction
• News of competitors, regulatory changes or new promotions. 

8. Communication is Key

In films, books, tv etc. as you watch the plot unravel the common theme is usually – all of the drama was avoidable if the characters would communicate better or at all. The same happens to be true for projects.

A typical example might be where two teams have individual timelines and deadlines for completion, they may report to different line management and their goals might be different. If something changes in one of the plans e.g. deadlines, then there needs to be communication to ensure that that can be accommodated by the relevant people and that the prioritisation of any changes to both schedules makes the most sense for the whole organization.

Communication is also integral within a team, where team members are saying they sincerely cannot achieve deadlines AND remain secure and compliant either the ramifications need to be understood and accepted or the timelines need to be extended.

This is the hardest of the non-software issues to ‘fix’ because there is a fine line between communicating and talking. Communication shouldn’t be a distraction to complete work; however a lack of communication can quickly break down a project.

Trust is also important to any communicating team(s). A project manager needs to trust when something cannot be done either in time or securely, a developer needs to trust that a prioritisation of a task has business success in mind. If a team risks anger, embarrassment or even the breakdown of a vendor/partner relationship they will not feel comfortable communicating a true representation of the world.

Remediation:

• Create an environment where feedback and concerns can be raised and are positively acted on
• Ensure where plans are changed and altered there is a good communication channel with other dependent teams. Where possible ensure there is a good understanding of how other teams work through playbacks, demos and in some cases secondments

Warning Signs:

• Communication between teams might be forthright e.g. a ‘tone’ in an email
• Team members raise issues to others in the team and not to management
• There is a misunderstanding of how other teams operate e.g. why things take ‘so long’ or why there is a delay or why work can’t be put into the other teams workflow straight away