If Software Development Has Peaked, why do Projects still fail? Part 2

by Aiden Gallagher & Peter Reeves (our podcast)

Technical Considerations

There are technical specific considerations that cause projects to fail too. Such as software and hardware build and deploy processes. Sometimes it’s simple errors that haven’t been accounted for, other times it might be bad technical implementation estimates or a high knowledge gap to fill. But other issues are less obvious:

 1. Over Engineering of Automation

Automation is the golden standard of modern, agile development. The more consistent and quicker something can be taken to production the better. This might be catching issues earlier to reduce overall cost to test in each environment or having an automated rollback strategy that means issues can quickly be reverted or new features rolled out to a small sample size.

But getting overly hung up on automation can lead to setbacks. Forcing automation without a real use case can eat up valuable project time. For example, automating a yearly update that takes 5 minutes to complete is hardly worth the effort.

Additionally, where automation is too complex there can be little gain and lots of pain. Take automating a test that requires 5 or 6 different applications that regularly change, or where there is no control of the connecting service e.g. scraping a webpage. The overall effort to complete is more hassle than the test itself.

There are also other areas which still cannot be automated, like user acceptance testing for flow ease of use and ‘obviousness’. Or user test cases that require negative testing – all the permutations of which cannot be known, else they would be covered in the initial testing. 

Remediation:

• Complete an analysis of savings vs cost to complete
• Consider the complexity of the automation and what might be missed e.g. UAT we might be able to pass all technical flows but not visual tests. If it doesn’t feel intuitive or right, then people won’t want to use the application/site.
• Be comfortable replacing automation with manual tests if the automation is taking too much time or fails regularly- this time should be factored back into the savings/cost analysis

Warning Signs:

• Automation tasks overrunning
• Lots of failures in the automation themselves that require team time to fix

2. Infrastructure still exists

Projects continue to fail because somewhere infrastructure still exists. The problem may be obfuscated away from the project team by being on a Software as a Service (SaaS) offering or controlled by a separate team, but failures and considerations of the infrastructure are still highly relevant to the project.

There’s also the hosting of the systems, moving to a cloud based off-premises network might mean severe limitations to the boundaries of the project. For example, an organisational project that wants to start working with a UK government entity might not be eligible if all the infrastructure lives in European datacentre. This can be resolved but there will a cost associated from switching providers whether it’s just one app or many as security, connectivity and skilling up are required.

As with all externalising of responsibilities to allow teams to focus on specific solutions - including with internal virtualization teams - there is a loss of control of system version, upgrade strategy and when outages will or might take place. If a project needs the latest version of Linux to create a new feature or expand an existing feature, the project success timelines might be pushed back whilst awaiting a convenient time for the external team to action.

There is also the case where an off-project infrastructure ownership can inhibit flexibility. A couple of examples might be, the ability to store a file such as a private key locally,  the ability to integrate with certain software such as an Active Directory Group. 

Remediation:

• Make applications that are easily portable to other infrastructure providers
• Understand Infrastructure limitations and requirements during design phases. This is especially important when doing iterative designs, so the constraints and boundaries are well understood going forward
• Have alerts for infrastructure changes or updates setup that the team can evaluate any effects as needed

Warning Signs:

• No mention of infrastructure in the design
• Failure due to changes at the infrastructure level

3. Upgrade Acceleration

Many modern applications pride themselves on their ability to push out new updates very quickly with daily or even weekly changes to production code. This works fine until the project is tied to the same upgrade cycle as a connecting application. Take Kubernetes for example, which used to provide its version support for 9 months; up to 12 months at the time of writing. 

This means there will definitely be an upgrade required in the next 12 months, of which, the latest version could be completely different to the one being used. Different mechanisms for assigning storage, managing deployments etc. are hard to predict and do not offer the stability that might be required for some projects.

Given an example where a customer feedback form is hosted in a container on Kubernetes, after a year the support expires, but it took 2 months to productionise, 3 months to rollout to all customers after the new feature is proven to work and be of value. A normal project might want to put the feedback form into a legacy status and just feed and water it to update logos etc. However, 4 months in the team needs to divert resources to upgrade the Kubernetes version to a supported version.

A perfectly working app might ordinarily have been left for a long time, but short support timelines mean imposed additional work where the upgrade could have serious impacts on the feature/application. This increases the cost of the project and reduces the quantifiable monetary benefits.

This also extends to any dependencies which are subject to lots of change, and any integrating applications with short release cycles. This also makes it hard for application teams to support as the inner workings of the system e.g. Kubernetes changes too fast to stay up to date.

The final part of this problem is the impact it has on workers who are expected to become experts on the systems, this is more difficult when it is under constant change and when combined with a requirement to have a much broader knowledge base. 

Remediation:

• Include upgrade cadence and complexity into solution design and selection
• Create automation tools for upgrades that are maintainable and manageable by long term support teams
• Select long term support options where possible

Warning Signs:

• Multiple upgrade requirements over a short period of time e.g. a couple of development cycles
• Lots of failures when performing upgrades 

4. Debugging takes time

With increased deployment velocity achieved in lean and agile projects, we are able to deploy quicker into production. But when bugs occur, they can be difficult to debug and understand. This might take time and may require that the production system is reserved for failure understanding only. If this time hasn’t been accounted for the whole project timescale can be knocked off balance.

In projects where debugging issues and fixing errors haven’t been accounted for, the onus to address defects can be pushed onto developers and testers who are forced to rush workload. This leads to further failures and an exacerbation of new, avoidable bugs getting into Production.

This can be alleviated by code ownership in production by the developing team. Project teams will be more likely to speak up and stop bad behaviours if they own the code in production, but this relies on the understanding at a management level.

Another issue can be caused when implementing ‘hot fixes’ based around assumptions of failure. This usually occurs because of a lack of time to debug and understand a problem. It also relies on an assumption that automated tests work and are up to date which requires trust. The less time a developer has to debug and get a fix that works, the less trust there will be, the more issues that will arise and the more likely for a project to fail. 

Remediation:

• Ensure issues are well understood before implementing fixes by testing issue hypothesis rigorously and keeping logs for long term analysis if required.
• Plan in time for defect management either a dedicated resource or dedicated time for the wider team

Warning Signs:

• Bug fixes are getting into production but are ending up back with developers as issues continue
• Unable to meet current cycle plan because of handling defects which suggests inadequate planning for defect
• Fixes without adequate problem determination

If Software Development Has Peaked, why do Projects still fail? Part 1

by Aiden Gallagher & Peter Reeves (our podcast)

Non-Software Considerations

There are lots of reasons a project might fail, but not all of them are software related, nor are they the sole dependencies of the technical members of a team. The following concerns are based on observation from projects that have failed before.

1. Partial Decision Makers

The first stage in any project is to get an understanding of the business goals. This isn’t necessarily requirement gathering right away but simply ensuring that the value of the project and the expected return on investment are understood. It also means understanding what, if not done, will cause those goals or the project itself to become uneconomical.

In one example, a car sales company has an implementation team with two tasks that they need to complete

• A new application to allow users to ‘test drive’ the car in an interactive application. The new app will increase sales by 30% to about £300,000 per year.  
• Updating how user order changes are sent to manufacturers from a manual email to automatic real-time event sending. This will save 5 workdays a month at around £10,400 savings per year. 

If the team has limited resources but not key information such as knowledge of the business goals or financial implications. For example, when deciding which task to prioritise, if the technical SMEs on the ground do not know that each day where the new app is not functional the business is losing money. 

However, the team regularly ‘feel’ the impact of having to send order updates via email which is time consuming and error prone. They think this is a relatively simple implementation that will save someone more time that could be allocated to other projects such as the new app.

The team will pick the emotionally preferable task as opposed to what matches business and financial goals. What seems like a sound decision based on technical facts has a secondary impact on the business.

 Remediation:

• Put deliverables into context for the implementation team including why it’s important for the business and the impact it will have e.g. This new feature will bring $X value
• Ensure improvement tasks are quantified e.g. adding a pipeline to add a new security key will save us 3 days per quarter and will take 10 days to complete
• Ensure task planning take goal importance into consideration and make them real and relatable e.g. ‘Get NewProject into Production’ is a technical goal, but the business context might be NewProject needs to have 100 users by the next Sprint to be profitable. The quicker it’s in production the more likely the ability to meet the goal which should drive planning.

Warning Signs:

• During planning meetings or informal discussions the team seem confused about the approach or question the sensibility of the plan. If there is disconnect it suggests the business goals related to each task isn’t well understood

2. Requirement Gathering is slow

Once a new task has been assigned to a team, it takes time to get all the requirements. However, in more agile projects, getting code into Production quickly and securely is a key benefit as is being able to pivot the strategy based on updated requirements. 

For example, a new feature lets a user set their address when they sign up for services, but feedback shows the user needs to be able to update the address; the requirements are that the update should be instantaneously pushed to the database.

The implementation team need to know what requirements they have on the database and vice versa but getting access to the specialists can be difficult as they are already committed to other project deliverables. 

Testing requires both teams to provide resources, access control is little understood by the new development team and as development begins changes might be needed which do not work with the system being integrated with. 

In the database example, the database team confirm that an update can be made to the database, the application team allow unlimited daily updates so if a mistake is made the new address can be resubmitted. However, the database processes data in batches with a first in, last out process which means the first - wrong address - is updated to the records.

Small gaps in product functionality, product knowledge and an inability to get the time of the relevant teams or specific members of the team exacerbate this problem, especially when applications are integrating with lots of other systems.   

Remediation:

• Define the limits of integrated systems so the design can be iteratively updated but within the confines of what is possible e.g. the database can accept any inbound data format but has a hard limit of 1000 characters and 1MB.
• For features, products and projects requiring external integration with another team, line of business or organisation then a dedicated resource should be requested to ensure iterative design can be reviewed at the same cadence as changes
• For new or unfamiliar features and products, the team; designers, developers and architects should be given sufficient time to prepare and gain education/experience. Where relevant and possible, the team should look to ‘loan’ relevant resources from within the organization
• Lock-in requirements for development cycles and allow new and updated requirements at the next design and development iteration. If this isn’t possible the current development cycles should be pushed back to a design cycle. This reduces changes further in the route-to-live workflow.

Warning Signs:

• Trivial issues such as size limits or unsuitable data formatting are being found in the development or testing phases. This suggests that design and architect team members are not familiar or up to date with requirements
• Team raises that there is little knowledge of a certain area
• The team are handling lots of new items with lots of unknowns e.g. lots of new versions of products are coming into play at the same time. 

3. T Shaped Experts

It is becoming easier and easier to begin working with a product with only a little deep knowledge of it, its features, or the platform it is deployed to. A subject matter expert who once specialised in a single product might now be expected to handle a wider spectrum of associated tasks including security, testing and integration with other tools. There is also a requirement to understand and be able to deploy in Kubernetes, apply relevant checks and measures through into production and provide support in production whilst developing new features.

Not all SMEs are able to quickly build up knowledge of lots of different products and have the ability to design and implement them to the same standard that a specialised SME could do, this might mean that good practices (learnt over time) are not incorporated causing later problems in the project. In theory with quickly moving teams, the same thing could be replicated multiple times over a product, project or applications life as the same ‘lesson’ is learnt by new people.

To accommodate this, small teams with a common ‘wide’ knowledge base and individual deep technical skills work together to form a robust skill base. E.g. an integration SME, an API SME, a security SME, a product support SME collectively complete the tasks on a team. But finding the right people can be hard.

Remediation:

• Provide sufficient time and resources for skilling up in new technologies
• Where possible, allow SMEs to be placed on programs and projects that need them e.g. pooling a certain skill type (cryptography) rather than expecting everyone to have a deep knowledge of it
• Communicate with the team to find interesting roles for each team member, especially when learning new skills. For example, by allowing team members to self-select tasks during planning. 
• Allow for longer development and delivery timelines when team members are learning or unfamiliar with a concept or feature
• Pair team members with more skilled members assisting less skilled members on their deep skill subject 

Warning Signs:

• The team is not performing as well as previously e.g. delivering at the same cadence
• Lots of small issues are arising that have been found when other team member performed similar tasks. This suggests a new team member is learning something the team has skills to perform, the new team member should be assisted by experience.
• Members of the team are showing fatigue or high levels of stress. They may even have told a team mate they are struggling.

4. Simple does not Equal Cheap

There is an expectation that when something sounds simple it should be easy and therefore cheap. Take for example, buying coffee from an online coffee delivery service. There are three functions; selecting your order, paying for your order, planning a delivery route to ensure coffee remains hot and arrives in accordance with Service Level Agreements (SLAs).

Selecting and processing of an order seems the simplest. However, simple in itself does not mean the design itself is simple; the stock list needs to be up to date, the online application needs to synchronise with the in-store systems, the online system needs a queueing system to handle demand throughput, the order has now been paid for but needs to remain on the system until delivered meaning the system and its messages needs to be highly available.

The feature itself whilst seemingly ‘simple’ has lots of other considerations at the technical level which are not always immediately obvious. This either means things are initially missed meaning lots of retrofitting after launch, or lots of prework to ensure the system functions efficiently and as expected, all which all has a cost. 

 Remediation:

• Ensure that all project members including management and delivery leads have a good understanding of the effort requirements relating to different pieces of work
•  Trust technical leads on the considerations, requirements and need to understand certain tasks
• Perform rigorous planning sessions to ensure all the work for a task is understood
• Avoid planning with preconceptions of ‘simple’ – lots of simple tasks can take a long time

Warning Signs:

• Lots of unplanned work is being scheduled
• Tasks are taking longer than originally planned for
• Technical timings for tasks are being driven by business requirements e.g. Feature needs to be in by next week, but with current resources the estimate is two weeks’ worth of work 

5. Cultural and Task Disparity Across Organizations

At an organisational level, all goals are generally aligned at the highest level; make a profit, open new stores, increase sales, improve costs etc. but on a team by team and project by project level the prioritisation of these goals varies.

As with “Partial Decision Makers”, an infrastructure provisioning team charged with decreasing costs by 10% may not see a new feature project as high a priority as server utilisation work. This might be because there isn’t a wider understanding of the organisations primary goals and each projects advancement of those goals, or at times a differing opinion of what is going to provide the best results.

When this involves separate organizations and the two beginning to collaborate to integrate systems, a problem arises where both organizations don’t have the business context of the teams that they are working with. If there is more value for one team than the other the collaboration and drive to succeed don’t marry well and can lead to delays or work not completed efficiently or effectively. 

A separate concern is when the culture within teams, whether cross-organization or internal, are very different. This could be work hours, lead times to complete work, processes that are seen as too quick (fragile or insecure) or too process ridden (wasteful or costly) and even the way communication takes place. Think of an Agile or Lean team versus a more traditional waterfall implementation, where on one side changes should be possible quickly with new evidence, on the other careful planning ensures budget is well understood and teams do not get overloaded. 

Remediation:

• Agree methods of working ahead of project initiations such as: how often people come into the office, general availability hours (10am – 3pm), communication techniques etc.
• Complete an all-hands to allow each team to describe what they are working on and more important ‘why’. What is the business benefit and how will it impact others and customers? Try not to make this “We completed this task ahead of time” but instead “We completed this task ahead of time which means we saved on costs and customers should not get X benefit.”
• Do a review of past projects to show what tangible benefits came from them in the long term e.g. over a year, two years etc.

Warning Signs:

• Required resources are not available when needed
• Two teams/projects have been given goals that do not have aligned solution or implementation, but one is reliant on the other
• A non-dedicated resource is provided. Usually this is because a resource is pooled or responds only to tickets. This can mean issues aren’t resolved quickly because there isn’t the capacity to do so.

6. Governance Perception

Governance is fast becoming embedded into more modern deployment strategies. The introduction of ‘production like’ as early as possible has always been a goal, but often the red tape has now been cut up to allow for faster route-to-live. In some cases the impact of not following these processes can be felt for years and in more extreme cases the lifespan of the project.

Site reliability engineers are more often being embedded into project teams becoming more flexible as they understand first-hand the development requirements and changes. This means processes are embedded and refined to ensure maximum compliance and forward planning for production support whilst removing pointless processes that genuinely block progress or agility.

However, some of the processes are still insisted upon – rightly – because of the experience of previous deployments which are common no matter what development style has been used to generate the features, how flexible your systems are or how many approvals are needed. 

When these are continued to be seen as ‘blockers’, ‘delays’ or ‘pointless’ and common-sense governance practices are ignored, then a program becomes more likely, and maybe even inevitably likely to fail.

 Remediation:

• Promote the retrospective sharing amongst teams of governance pitfalls that have been met and solutions that have overcome them
• Review governance processes regularly to validate the precaution cost against the risk cost. If the precaution is more expensive can be it be scaled back to a cheaper alternative. 
•  Invite site reliability engineers, security engineers and production support to review the processes in place. Are they more or less restrictive? Have better systems been put in place elsewhere?

Warning Signs:

• Release of code is taking more time than the design and development
• Work in progress is getting ‘stuck’ at governance steps with high ‘time in activity’ e.g. An approval process taking several days.
• Valid governance processes are causing discontent amongst the team. It might be that the risk cost is not understood or that the precautions are overly restrictive.

7. Who ‘owns’ this work?

Scope creep has and probably always will be a major issue in the planning and implementation of a project (see Requirement gathering is slow). This doesn’t change with Agile which, although flexible, is better when focused on specific aims, new tasks inserted two days into a new sprint can cause havoc and mean a rapid realignment of other expectations.

When new tasks are combined with collaboration with many teams, the who and when element of the task can make for a constrained relationship. This might mean other work suffers but any hostility from the stress of change can damage cooperation in the future and the general progress of a project.

This usually can be resolved by planning for the ‘unknown’ and fostering positive relationships. This also requires a level of reasonableness, in that an acceptance that new work will likely mean a push out on timeline of current work or an increased cost to complete the new work to the same timelines.

In lean enterprise this can be alleviated if the same team manages from development through to Production and if changes are contained with less close coupling. 

Remediation:

• Be open to new requirements but ensure there is a process for getting them added to the backlog of work to be properly measured and dependencies managed with the relevant teams
• Reduce dependencies on other teams as much as possible. Teams with greater internal control - either because they own or have the ability to develop in their own isolated space - can accommodate new requirements easier. However, the team still need to know how to do the work
• Allow headroom for unforeseen tasks in the planning. This might mean having some resources every cycle available to down tools and help with defects, new requirements etc.

Warning Signs:

• New project announced
• Change of organizational direction
• News of competitors, regulatory changes or new promotions. 

8. Communication is Key

In films, books, tv etc. as you watch the plot unravel the common theme is usually – all of the drama was avoidable if the characters would communicate better or at all. The same happens to be true for projects.

A typical example might be where two teams have individual timelines and deadlines for completion, they may report to different line management and their goals might be different. If something changes in one of the plans e.g. deadlines, then there needs to be communication to ensure that that can be accommodated by the relevant people and that the prioritisation of any changes to both schedules makes the most sense for the whole organization.

Communication is also integral within a team, where team members are saying they sincerely cannot achieve deadlines AND remain secure and compliant either the ramifications need to be understood and accepted or the timelines need to be extended.

This is the hardest of the non-software issues to ‘fix’ because there is a fine line between communicating and talking. Communication shouldn’t be a distraction to complete work; however a lack of communication can quickly break down a project.

Trust is also important to any communicating team(s). A project manager needs to trust when something cannot be done either in time or securely, a developer needs to trust that a prioritisation of a task has business success in mind. If a team risks anger, embarrassment or even the breakdown of a vendor/partner relationship they will not feel comfortable communicating a true representation of the world.

Remediation:

• Create an environment where feedback and concerns can be raised and are positively acted on
• Ensure where plans are changed and altered there is a good communication channel with other dependent teams. Where possible ensure there is a good understanding of how other teams work through playbacks, demos and in some cases secondments

Warning Signs:

• Communication between teams might be forthright e.g. a ‘tone’ in an email
• Team members raise issues to others in the team and not to management
• There is a misunderstanding of how other teams operate e.g. why things take ‘so long’ or why there is a delay or why work can’t be put into the other teams workflow straight away

Improving My Garden Soil

Introduction

Having recently moved to a new house I was hoping to use a bit of garden next to the garage to grow some vegetables, bringing back the days when we had an allotment. The area had previously had three small trees planted and lots of long coarse grass and a little bit of rhubarb.

Slowly clearing the grass and the rotting tree wood, the ground was hard to dig typical to a clay type soil and over winter the ground retained surface water which led to ice forming. This post is a summary of my investigation into our garden soil quality, what makes up soil and how to improve the drainage.

Skip the ‘what is soil’ section if you just want to see how I fixed the drainage

What is Soil?

Soil is a collection of organic matter, minerals, gas, liquids and organisms which support plant growth, stores water and is a home for organisms such as worms. 

Soil is made up of ‘horizons’ which are layers of soil:

Horizon	Characteristics
O - Organic (Humus)	Typically made up of leaves and dead wood
A - Surface (Top soil)	Minerals mixed with organic matter
B - Subsoil	Mineral rich that have been filtered from other levels (leached)
C - Substratum	Earth’s surface deposit that makes up soil
R - Bedrock	A selection of rock including granite, basalt, quartzite, limestone or sandstone

Horizon O is the ‘humus’ layer which is non-living, finely divided matter made from the decomposition of plant and animals. As it continues to decompose it becomes usable by plants for growth. Humus broadly comes in three formations:

·       Mull which is normally in hot and humid grasslands or densely wooded areas. Has lots of worms, insects and is high in alkaline

·       Mor which is acidic doesn’t have many insects and organisms to break down the matter so a compact layer forms below the surface. These are usually in cold and high-altitude regions and have lots of fungi

·       Moder is between mor and mull extremes. There is a greater abundance of insect and their digestion but the humus and decomposition don’t mix well showing a clear separation

Horizons O and A are what makes up most of the soil used in gardening and comes in a variety of types and they each have specific properties:

Soil Type	Properties
Clay	A heavy soil type which high levels of nutrients
Silt	A light and moisture retentive soil which is good for fertility
Sandy	A light, warm and dry soil that is low in nutrients and often acidic
Loam	A mixture of clay, silt and sandy soil
Peat	A soil that contains a high amount of organic matter which retains lots of moisture.
Chalk	Ranges from a light to heavy soil which is high in alkaline because of calcium carbonate and lime in the structure.

What affects Soil Drainage?

Soils drain at different rates depending on their composition. Clay is thicker so doesn’t drain as quickly and is also impacted by changing temperatures, contracting in the heat and expanding in the cold so a more clay-like soil drains slower than other types.

When soil becomes compacted its drainage becomes worst, with fewer air pockets there is less space for water to drain into leading to water sitting on the surface or near to the surface.

Both clay-like soils and compacted soils can be exacerbated if the lay of the land doesn’t encourage run off. In fields you might see drainage through a field or on the outskirts, in some gardens you might see a French drain but, in most gardens, water run off isn’t as well managed due to flat gardens or small ditches throughout. This leads to water pooling on the surface. 

If soil sits on a slope, rainwater can cause the soil to slowly ‘creep’ downwards. On larger slopes like hills or coasts this might lead to landslides, in the garden it mostly leads to soil degradation.

From a more scientific basis, permeability – ability for water to pass through – of the soil affects the rate of drainage. Sand and gravel see faster water volume reduction than clay. When water sits between soil particles, the pressure – pore pressure – dictates water drainage ‘seepage’. The increased pressure slows down the seepage which is why the change in water volume slows over time.

My Garden Soil

Figure 1 - Before

When we first moved in, there was lots of plastic, clutter and long grass growing in a space next to the garage where we are going to grow some veg and use as a seating area. The first step was to tidy the area up.

Figure 2 - Post Tidy Up

After tidying there were some clear patches in grass, rubble throughout and more importantly three sets of old trees that were rotting in the trunk at ground level which suggests sustained rot from pooling water at the base. To combat this I dug out the stumps.

Figure 3 - Logs Removed

Whilst digging the soil was clumpy, very hard to dig and much wetter than the rainfall warranted. The grass was still patchy and the ground generally uneven.

Figure 4 - After the Log Removal

After a particularly bad downpour and even a little bit of snow, there was a lot of water retention which became clearer after everything was cleared

Figure 5 - Water Retention

Testing Soil Type

Before making any changes to the soil it’s important to check what type of soil is in the garden. This will help to understand how to fix the problems and what might need adding to the soil for the best vegetable growth.

It’s possible to map soil content with the type e.g. clay, sand, silt, loam or somewhere in the middle e.g. sandy loam, silty loam or clay loam. To complete the test:

1.    Get a big jar

2.    Fill with soil from the garden

3.     Remove any big stones and break up the mud

4.     Add a spoon or two of washing up liquid and add water almost to top

5.     Shake thoroughly and wait a week for the soil too settle

6.    The soil will settle into three distinct layers, sand at the bottom, silt in the middle and clay at the top. Above the last distinct line will be some water which might still be a little muddy. Use a ruler to measure the height of each layer and mark the total height. 

7.    Calculate the length of each and the percentage of the jar that is sand, silt and clay. 

Percentage = (length / total length) * 100 

e.g. Silt Layer Percentage = (1.93/7.78) * 100 = 24.8

 

See the below example for the measurements.

Figure 6 - Soil Test Percentages

8.     Add the percentage of the sand and clay into the website below: https://www.nrcs.usda.gov/wps/portal/nrcs/detail/soils/survey/?cid=nrcs142p2_054167

The output given will be a triangle with a plotted point showing the soil type

Soil Test Result

Having completed the test, the results showed that the soil in the garden is a ‘Sandy Loam’ which is a long way off being a ‘Clay Loam’ which is what was expected.

Figure 7 - Soil Test Results

This was a surprise as the soil behaves like clay by being hard to dig and doesn’t drain well. What is even more surprising was that it would need a lot of clay to have moved into the sandy clay loam. 

The soil test results suggest that drainage should have been better, but the garden did meet the elements in the section "What affects Soil Drainage". There was a lot of compaction so not enough air in the soil this in turn means the water loaded heavily on the soil and drainage is impacted by a lack of permeability

Improving Soil Drainage

Having discovered there was a problem with the drainage of the soil, I started to research the best method for improving the soil drainage. 

To improve drainage you might:  

• Add organic matter (food waster, compost, manure)
• Create a ditch or run-away for water to follow
• Reduce compaction and ‘aerate’ the soil 
• Plant water hungry ferns and bushes to reduce water in the soil

The first task was to dig over the area that was retaining the most water. At the same time the soil was aerated with a fork, plastic and weeds were removed and compost dug into the soil at about 5 – 10cms throughout.

Figure 8 - Dig over and Aerate

It would have been possible to just dig the specific sub sections that were showing water retention however if just sections were dug over and compost added there would have been spots of good drainage amongst a wider garden of compact soil. 

This can lead to water pooling in the holes that have been dug which would exacerbate rather than solve the problem as water would just pool at the bottom of the hole. This can be seen at the edges of the improved soil as the soil gradient, and the non-compacted soil meets the compacted soil.

Figure 9 - Water Retention on the Edges

There were also spots where the soil level sunk, and water gathered at the bottom of the dug over area. This is most likely because the compost and the existing soil were not adequately mixed which means the water passed through the compost and sat on top of the unmixed soil at the bottom.

To fix this, more soil was dug over, more organic matter added to the soil (vegetable peels, potatoes and bread mixed into the soil) and some more compost dug in.

Figure 10 - More Soil Treatment

Finally, raised beds were added as it would reduce the compaction on the vegetable growing spaces and ensure the soil has the correct ingredients for growing and level of the soil which drops slightly at the back edge towards the fence so the water has some run off.

Figure 11 - Completed Dig Over & Raised Bed

The result is that the soil no longer has water pooling, the water retention in the soil has reduced, there is a natural water gutter just in front of the fence and plenty of space to grow some vegetables.

Figure 12 - Final Garden Picture

Conclusion

In this blog post we have looked at the garden before any changes and issues that came with it. This led to some research into what soil is, why it was behaving the way it was and how to test the soil ready for improvement. 

With the soil results there were improvements made to the back soil ready to plant some vegetables. The rest of the soil will need some changes, but that’s something to worry about next year.

Building IBM ACE Bars using Azure DevOps and a Windows Build Server

For the last few months I have been looking at the most appropriate way to build ACE artefacts. This ranges from what makes good practice for ensuring consistency, what allows the best governance procedures to be followed and what is the best way to isolate the build and deploy of the artefacts?

One of the more common pipeline tools that is being used is Azure DevOps and so I've been playing with the mqsicreatebar, mqsiapplybaroverrides and mqsideploy commands in IBM ACEv11. The following are steps I wanted to achieve to get deployable units in ACE:

• Build a Base BAR file
• Apply environment specific overrides to the Base BAR to create an environment BAR file

To complete the build process we need to be able to run ACE commands. To do this we could use an external Server but when using a Windows machine there are certain ways these commands might be run. 

The following are some tips on how I completed the steps using Azure DevOps and Windows in my demo building scenario.

Method of Execution

Having decided to produce a script to be run on the Windows machine the first decision was whether to run as a batch script, bash script or PowerShell. 

• PowerShell is the Azure Cloud preferred method and comes with the PowerShell pipeline
• Batch files are out of the box, well understood by most and so likely to be easily supported
• Bash scripts are cross platform, easily portable and possibly better understood than batch scripts

In the end I chose bash because of the portability and ability to move across Cloud and Operating Systems with more ease and increases the number of people who can edit and support the scripts in future.

Running ACE Commands on Bash on Windows

Bash is not currently native to Windows machines and as such you have to run a bash environment on the Windows system. This will usually be Git Bash so to run in the Build pipeline in Azure using the command line task:

“C:\Program files\Git\bin\bash.exe” myscript.sh

 In order to run ACE commands, the ACE profile needs to be setup using the mqsiprofile command. In the Build pipeline in Azure using the command line task, run the following:

“Drive:\InstallPath\server\bin\mqsiprofile” && “C:\Program files\Git\bin\bash.exe” myscript.sh

Check the format of each ACE command 

It's important to check the format of each of the commands in ACE as some of the command are already executable in shell e.g. cmd format and do not need the file path specificying in the bash script, others are in BAT format so need the full path specifying. One example of this and specific to our needs is the mqsiapplybaroverrides command which needs the .bat extension name when being called in the bash script and also needs the full path to the file.

Passing Azure Variables to the Script

Azure variables need to be passed to the script via the command line task. Azure uses the following format to reference existing variables $(myVariable). Variables can be passed in using the Variables tab e.g. appName (to specify which to build) or predefined Azure variables e.g. System.ArtifactsDirectory

Example buildACE.sh $(appName) $(System.ArtifactsDirectory)

#!/bin/bash

appName=$1

buildAddress=$2

mqsicreatebar -data ${appName} -b ${buildAddress}/${appName}_Base.bar -a ${appName} -deployAsSource

drive:/installPath/server/bin/mqsiapplybaroverrides.bat -b ${buildAddress}/${appName}_Base.bar -p ${appName}/dev-overrides.properties -o ${buildAddress}/${appName}_DEV.bar -r

The following example will take the application being built and the build output location (Artifact) as parameters which are used in the bash script. 

As you can see the mqsicreatebar can be ran without specifying a path because we have the mqsiprofile set, but the mqsiapplybaroverrides needs thee full path to the batch file defining.

The output from the above script would be a base bar and a dev bar for the application name supplied in the Azure Build. Further scripting can be implemented to produce BARs for more environments.

IBM MQ – AMQ3817E – DRBD Errors

 Recently when rebuilding an MQ Estate, the previous deployment was torn down and a new installation and configuration was deployed to the same physical Virtual Machines. However, when it came to creating the new Queue Manager there was an error thrown.

 

The reason for the rebuild was to upgrade the version to IBM MQv9.2 which includes some changes to the way RDQM pacemaker and drbd kernel modules are installed.

 

Command:

crtmqm -rr p -rt s -rl thisIP -ri otherIP -rn otherHostname -rp port -lla -fs 20 -lp 6 -ls 3 qmname

 

Error:

AMQ3817E: Replicated data subsystem call '/usr/sbin/drbdadm new-resourceqmname 0 --auto-promote=no' failed with return code '20'.Command 'drbdsetup new-resource qmname 0 --auto-promote=no' did not terminate within 5 seconds

AMQ3812E: Failed to create replicated data queue manager configuration.

 

To debug I did the following:

1.     Checked the MQ Error logs

2.     Check there were no outstanding processes that could be blocking the crtmqm command

3.     Check the correct drbd kernel module was installed on the server. (modinfo drbd)

 

There were no errors in the MQ Logs, no processes running for drbd and the correct kernel module was installed. Sufficiently stumped I ran lsmod drbd | grep drbd on the server and compared it to another environment.

 

There was no drbd_transport_tcp module on the output of the environment with the issues only the drbd module. When there are no RDQM Queue Managers running lsmod drbd | grep drbd hould return empty, which seems to suggest the drbd module was hanging. 

 

To resolve this there are two possible actions:

a.     Reboot the machine

b.     Remove the remaining ‘hanging’ drbd module using rmmod drbd

 

After completion, run lsmod drbd | grep drbd and the result should be empty. Try to build your Queue Manager again and it should create successfully.

Untangling Governance Definitions - Patterns, Blueprints, Templates, Standards and Reference Architecture

By Aiden Gallagher & Peter Reeves

*Disclaimer: These are our own definitions and might not map to your specific organisation. And, of course, these are our own opinions, and not necessarily those of our employer

Introduction

For the last few weeks, Peter and I have been talking about Integration Patterns, blueprints and architectures to try and boil down a common definition which we can use when talking about these concepts.

We found that halfway through a conversation we would start merging elements and classifying the same integration item as different things.

To remedy this, we decided to write down the definitions as a reference point that we could refer back to. In this blog post you will be able to see these definitions, and when we talk about Patterns, Blueprints, Templates, Standards and Reference Architectures you will know what we mean too.

Why do we care?

Before we share the definitions, it’s important to know why we care about the words themselves. We want to be sure that when we are discussing concepts that we are all referring to the same things and in the same way, related to how artefacts fit into the wider organizational picture and the correct governance procedures when designing systems.

Definitions

Below you will find our definitions and with them an example of how these are applied outside of the computing world. In this example, we will apply them to a housing project and “dwellings”.

Pattern

Definition – A pattern is a refined repeatable concept which is a solution to a common problem.

Example – Given the common problem of a human needing a place to live; have a roof over our heads, running water, sanitation, heat and the ability to cook, a pattern standardises the way we build a ‘dwelling’ that acts as a solution to these problems.

Blueprint

Definition – A blueprint is a detailed design for how to implement a pattern using a certain technology.

Example - A blueprint of the dwelling pattern could be a terraced house which provides dwellings against a limited land space. It could also be a bungalow which provides accessible dwellings for those who cannot use an upstairs.

Templates

Definition – A template is an implementation of a blueprint with further details on the specifications for items of that type. It might be followed exactly, or it might be altered to fit the need of the circumstance.

Example – A terraced house in Darlington was previously built with certain specification. The designer wants to provide more garden space as such they can tweak the template of a terraced house during design. But the template largely remains the same.

Standards

Definition – Standards are an approach, design or method that is applied across all interconnected systems and applications to ensure consistency of approach, and this benefits support, testing, and development.

Example – The plug sockets in all houses are the same size, as that meets the requirements for the majority of household appliances. This means all devices are interchangeable, which in turn means less customisation during build and hence is cheaper to build and more convenient for the end users.

Reference Architectures

Definition – A reference Architecture is a set of documents or information that describe a topology or system that can be used for common and related patterns. The information, if followed, will provide commonality across deployed templates and blueprints.

Example - A large plot of land is bought for building development requiring sewage, electricity, water, gas, and road network. A reference architecture describes how this infrastructure can be setup to accommodate different home styles (patterns) the instructions for building each home (blueprint), and potentially any customisations to the blueprint to fit other requirements (templates, and templated usage). Throughout the whole build process the same style of roofs, doors and windows are purchased (standards).

Integration Example

In the integration world we might apply the above definitions to the following use case.

A hybrid-cloud deployment strategy has been documented which shows how availability, disaster recovery, networking, and deployments of servers, and various business functions might be achieved in a reference architecture for that system.

The cloud deployment has a number of topologies (or building blocks for topologies). These are used as necessary to build integration patterns (e.g. request-reply which needs to solve some high-level problems like: access to a database, transactionality, reliability etc.)

Having selected the reference architecture which will support specific patterns, a blueprints to implement desired patterns using those cloud offerings can be written by the organisation – this could form a high-level design of how the new system will look like, this is further refined and customised in the low-level design. 

Looking at a Hybrid Cloud implementation, there are some orchestrated deployments packaged as part of a cloud offerings to solve specific business needs, which is in itself like an off-the-shelf blueprint combined.

To build the system, a series of templates are used for each cloud product with necessary minor tweaks that meet small nuances of the pattern and blueprint requirements. Throughout all these processes the organization’s standards are used to ensure naming conventions are the same, disaster recovery can be performed by any of the support teams, that port mappings follow the same style for easier communication with the network teams, and any other core business procedures are followed.

The end result is a system that was built quickly, in a reusable, extensible, and familiar way and that is now supportable by the wider organization.

Benefits

By using Patterns, Blueprints, Templates, Standards and Reference Architectures there are obvious tangible benefits:

Standardisation

·      Less complexity when implementing solutions as everything is known.

·      Cost reduction of multiple teams solving the same problems repeatedly (why reinvent the wheel?)

Reusability

·      Once the patterns, blueprints etc. have been shaped and refined, it becomes easier to reuse them than to start the building process from scratch- saving time and energy exponentially.

·      They are easy to use.

·      They are easy to deploy.

·      They are easy to support.

Familiarity

·      There is less project-specific knowledge required when people switch roles as the core patterns (and blueprints) are the same.

·      Individuals are able to get the 'gist' of what is happening in a new project- even for customised docs, build etc- because everything looks and feels the same and was built with the same principles, even if the low-level implementation is slightly different.

·      This familiarity makes it easier and quicker to fix, debug and test new projects.

·      This leads to a reduction in dependencies on individual resources.

Future Proofing for the whole Organisation

·      Improving the pattern by one team improves for all.

·      The whole organisation is carried into modern practices together because of the usage of patterns, blueprints etc.

Limitations

There are some limitations to the usage of Patterns, Blueprints, Templates, Standards and Reference Architectures which tie closely together:

Complexity

·      Sometimes complexity is required, maybe because something has never been done before, or there are new nuances such as security regulations that make a solution difficult to implement using existing patterns.

·      Trying to fit the patterns into these use cases might cause more harm than good e.g. over-engineering, barrier to release time, tendency towards “You aren’t going to need it” (YAGNI) behaviours and a bloated system.

Blocker to change

·      It can be hard to change a pattern because it’s a known entity which people are familiar with. The pattern then becomes a tool to block improvements.

·      It can be hard to get change because the benefits of a new system might not be understood by the wider organisation who must accept these changes.

·      Patterns become tied to governance process. This can undermine the governance process which can be deemed as ‘red tape’.

Stifling Innovation

·      Working only to the pattern can stifle the creativity which comes naturally to a lot of people in an organisation. This means that the pattern and its prescriptive, restrictive methods can lead to a lack of innovation and improvements.

·      It can be hard to get investment for new ideas because there are no any proven benefits, which is perpetuated by a lack of ability to improve existing patterns.

For all these limitations, it is possible with good organisation practices to be able to work around these requirements. This might be by encouraging minimum viable products (MVPs), allowing colleagues to use new tools in their day to day life, and having governance process that look to solve organisational problems and ongoing reviews of current business processes.

Conclusion

In this blog post, we have given our definition of Patterns, Blueprints, Templates, Standards and Reference Architectures, along with a real-world example and a computing example alongside an exploration of the benefits and limitations of using them.

Have any thoughts? Do you vehemently disagree with our definitions? Have a better computing example?  Get in touch to tell us what you think