This is a personal blog, postings on this site are my own and don’t necessarily represent IBM’s positions, strategies or opinions.
I blog about some of my day to day work findings in integration, programming and project competitions. I also blog about personal projects such as painting, the allotment, robot building, node modules or the history of IT.
1. Pick your two teams and click select teams. Hint: If you don't pick any teams or only pick one, just click "select teams" and it'll randomly pick two teams.
This dynamically generates 11 selection boxes with each countries squad displayed in each box, ready to pick your team. You will also need to select your formation or it won't work!
2. Pick your players. Not sure who to pick? Just click random select and it will pick a formation. This will fill in each position with either a Goalkeeper, a Defender, a Midfielder or a Forward depending on the formation and the players preferred position.
Hint: Not all player of a certain type will be put in their best position i.e. an RB in a CB position!
3. Hit simulate match. After you hit "Simulate Match" you'll see a screen with all the players selected and the positions they were put in. At which point you can either do a "Quick Match" which simulates the match with updates to the statistics or you can "Watch Match" which will give a graphical view.
Technology has undergone a period of growth and expansion since the 1970s. With this growth has come a wide diversity in languages, platforms and protocols used to develop and execute functionality on systems and to connect the systems.
Integration technologies were and still are the industry’s answer to this connectivity demand and companies have invested heavily in developing and marketing products that supply the capability to integrate systems, whether in house or through dedicated commercial integration software products.
But do commercial integration products still have a place in the modern industry, or are home grown integrations — using custom, bespoke software and Open-source implementations — a more flexible and efficient way for organisations to integrate their disparate systems?
In this article we will discuss why greenfield software development might be better suited to home-built software rather than using commercial integration software products, but why, as they become more established and stable, there is a necessity to use an out-of-the-box commercial software solution to address ongoing business needs.
The Evolution of Integration
Business systems and machines began as silos, completing their independent functions in isolation. Integration is the process of connecting one or more of these systems together, but as system integration was adopted, the communication was found to be unreliable and slow, and it could be difficult to interface between them.
Different integration software techniques were developed to solve these problems including messaging, file transfer, remote procedure calls and shared databases. Companies that had previously been dedicated to their primary business goals slowly began to adopt IT and integration systems, which expanded and grew to consume larger amounts of the business’ resources.
This led to the creation of specialised commercial integration software created and maintained by companies like IBM to provide simpler and easier integrations for companies, to allow them to focus on their core aims.
Organisations that have evolved with very-early IT (1960s-1990s), tend to have large IT estates and have a large amount of commercial integration products embedded within then. They have gained knowledge of how to safely manage, maintain and build systems that meet diverse business, regulatory and consumer requirements, through long standing processes and governance procedures that ensure integrity. Many are recognised for this ingrained culture and slow response to market demand.
How is the integration landscape changing?
The IT world has been changing. Computing has become part of everyday life and the prominence of mobile devices mean companies are at the beck and call of their customers, day and night; as they wish to use their applications, interact on social media, make queries and raise issues.
For example, Daisy is shopping on Etsy at 2am and wants to make a payment through their bank. If there is an issue, Daisy uses the live chat support where a chatbot tries to help before passing Daisy on to a person. The payment needs some form of authentication which needs to be setup and used to complete the transfer right away. Anything less will be a pain and Daisy might be tempted to switch to another bank.
The systems involved in meeting Daisy’s requirements, require backend integration with Daisy’s account and payment data which might be spread across disparate systems. The resolution to Daisy’s problem may also require the calling of multiple different services to complete a business process with responses integrated between the systems.
To deliver for people like Daisy and maintain a competitive advantage, companies have the need to be able to rapidly build solutions that access, synchronize and combine data from disparate systems across the organization in order to react to new innovation. This is done through integration applications and software which in turn, must be highly available, secure, able to keep up with market needs, able to innovate at speed and where possible, able to reduce costs.
The types of changes are shown in Figure 1 — Changing IT Landscape Factors (2021) but can be summarised as such:
· Lightweight runtimes through container technologies are helping to deploy smaller, more flexible applications that can be owned by teams and deployed more easily.
· API Led Approach is making it easier and simpler to access data from other systems such as databases. This can be used to provide additional value or can be monetised for organisation gain.
· Cloud Native application deployments built for the cloud which are more portable (saving costs), scalable to meet changing demands and utilising implied resiliency of cloud deployment. (http://ibm.biz/cloudnativedefined)
· Advanced Tooling to automate all aspects of the application, system and integration life. Create, build, deploy, monitor, alert and debug using automated tooling, improving security, access control and time to market in the process.
Figure 1 — Changing IT Landscape Factors (2021)
These changes have allowed existing companies to provide better services, but many struggle to fully capitalise on the benefits because of their ways of working. Greenfield companies are building new applications, software and features from scratch based on these principles, and are not encumbered by existing processes, governance, maintenance of legacy systems. They also do not have an enforced requirement to utilize existing commercial integration solutions already embedded in the organisation.
Greenfield Software Solutions
Greenfield companies have emerged to disrupt existing markets, in almost every industry, from the way we socialise (Facebook, Twitter, Instagram, LinkedIn), the way we bank (Monzo, Revolut), the way we shop (Amazon, Etsy), travel (Skyscanner, Uber), order our food (just eat, Deliveroo), manage our finances (PensionBee, Moneybox, eToro, Freetrade) and much more.
When building these new companies there is a decision whether to use commercial integration software already in the market, Open-source technology that can be adapted by the company or their partners, or integrating systems using custom software. Whilst commercial Integration software solutions have been widely utilised and relied upon across industries, they are constantly compared to custom, in-house (or Open-source), code-only integration solutions.
One criticism of commercial integration software solutions is the lack of flexibility. For example, a business may want to take advantage of a new feature such as Open API v3, but commercial products may take a while to catch-up with the specification. In-house code could be updated and the feature prioritised in a much simpler and quicker process that goes straight from developer to the live system.
Commercial integration products do provide a low-code approach, but could be seen as a “black-box” in terms of their functionality, which can prove important when it comes to debugging. Byars (2021) argues that code-based solutions are easier to “diff” for changes [2]. However, if the volume and complexity of code is scaled out to match the breadth of functionality that commercial integration products provide, this debugging process could become equally or more challenging.
Businesses may also be hesitant to implement commercial integration products to avoid vendor lock-in. The counter balance to this is the reduced effort for businesses by utilising existing, commercial integration products which are backed by documentation and a wide support structure of product experts built up over many years.
Greenfield companies often use Open-source because it provides them with the ability to quickly adapt software to meet their needs. This assumes they have the skills and time to build software as well as running their businesses, and that they can keep those skills once the initial project is complete. For implementing leading edge capabilities, this may be the only way to create the functionality sufficiently quickly, because making adaptations to commercial integration products requires communication, waiting for changes to be discussed, prioritised, architected, coded, tested and eventually released. This is before the Greenfield company runs its own testing that these new features meet its requirements.
This lack of control when using commercial integration products is the focus when high-impact issues affect systems. The Akamai CDN outage (July 2021) and Log4J library vulnerability (December 2021) are prime examples, companies felt unable to deal with the issues due to a reliance on external entities’ support and communication. In-house software can be worked on internally with additional resources, temporary workarounds and fixes in the hands of internal teams.
When to keep on developing
Teams building in-house integration software and developing on top of Open-source can continue to do so providing they are capable of maintaining and managing the code base long term. This means ensuring there is large team of good developers with the knowledge and ability to support all the tools and code used within the systems, and that talent can be retained going forward which can often mean ensuring the challenges are exciting and the company a good place to work.
If internal development teams are more agile and able to deliver good quality fixes to bugs and issues, faster than the commercial integration software company, it lends the team a competitive advantage especially if competitors are using commercial integration products that are taking a while to catch up with the latest trends.
In-house software allows direct and rapid communication with partners, consumers and communities who can request new features directly (e.g., through forums) and the support teams can provide more direct support to those requests and in some cases communicate live updates.
The key reasons to continue internal development is to meet unique use cases and rapidly changing requirements, to ensure software growth can meet the pace of company growth and identify where new features and experiments are still needed to provide a good quality product that can compete on the market.
Using commercial integration products
Commercial integration products have been honed and enhanced over many years with a well-established knowledge base, progressive evolution, and subject matter experts who specialise in integrating systems and working through industry wide problems. The primary reasons why these might be used over In-house software include:
· Saving time and cost by not re-solutioning common use cases that have been solved before
· Established support models from consultancy firms or contractors to build on the tools using previous experience to accelerate time to market
· Reduce complexity of integrating with others
Companies that use commercial integration products do not need to start from scratch, from day one they can integrate point A to point B, with mandatory support mechanisms in place for issues found, a breadth of experience for common solutions and no necessity to understand deep level integration concepts such as protocol implementation. The more unique the functions, features and skills that are required to integrate, the less cost-effective commercial integration products become, because of the commercial consequences of supporting edge case scenarios.
Growth is a key element of success for any company, especially greenfield, but at some point, there is a need to adopt more professionalised and established methods of management, with a renewed focus on existing customer retention. As products and companies become more established, government and regulatory requirements might be imposed to protect consumers. For example, General Data Protection Regulations (GDPR) implementation cost US Fortune 500 and UK FTSE 350 companies a combined $8.9 billion and was an additional cost of unplanned business and continued maintenance [3]. Commercial integration products provide this for a range of customers reducing the individual cost of supporting long term regulatory costs.
For commercial integration product vendors, creating and maintaining software products is their core business. For other businesses, software creation is a distraction from their core business. This is especially true when supporting business as usual systems and applications.
By Contrast, Open-source integration software has a support model that relies on community good will. There is no liability to fix problems, and security breaches can be easier to exploit due to the openness of the source code. There are obvious benefits of participating in Open-source communities to work on and improve key industry features as a collective. However, Open-source can become outdated, the community can move to other projects and active members in an organisation can move on to other challenges. This increases dependencies on the remaining active members and at times a need to pivot entirely to another tool which can be costly.
Commercial integration software companies have a business driver to produce integration products that work over a large proportion of the industry, over a long period of time. Development teams have expansive experience of integrating systems and have evolved with the market over product lifetimes which for some, are now reaching three decades of life. These products have the experience of past failure, resolving issues, enhancements to meet changing needs, they are also liable for the support mechanisms they have in place to resolve issues as they arise and patch security bugs as quickly as possible.
Established products are incentivized to evolve with changing IT landscapes to retain business and often work with customers to ensure continued positive relationships. When the company grows, so too does the integration product servicing it.
If a greenfield company begins to hit turbulent times, it is difficult to justify the cost of managing and maintaining In-house integration solutions. Focus needs to be on customers, sales/retention and becoming profitable. The cost of creating and maintaining integration software can increase if there are inexperienced staff, if supporting existing products becomes untenable because it has grown too big or because employees have moved elsewhere, then costs can balloon, and projects stagnate.
Whilst there are advantages to greenfield software solutions in terms of control of resolving security vulnerabilities, commercial integration software products have had many years to consider and enhance the security hardening on the product. Utilisation of these pre-built Integration products reduces the duration of effort required for security planning on a brand-new product, the same can be said for networking, high availability etc.
It is important to consider the cost of integrating with other organisations and business. When producing code internally, companies have an element of control. But this can become an unwieldy when integrating externally and is exacerbated the more integration points that are involved. There may also be greater support and documentation available for integrating an out-of-the-box solution with an external product, with pre-built connectors and accelerators to integrate with well-known systems. For example, no-one today would consider writing an EDI integration solution as that is so well established.
Choosing an integration model
Commercial integration software and products provide evident benefits when integrating using well-known systems through established mechanisms. They provide a strong support through experienced integration developers and specialists providing companies the ability to focus on their primary aims rather than the integration of internal and external systems.
However, building in house integration software has its own clear advantages. It provides flexibility and easy adaptation to meet edge case scenarios and innovate ahead of the industry and competitors.
This is not a binary decision; it is possible to switch models although there is some initial outlay cost for the transition. Applications using commercial integration software can expand to start using custom code for specific purposes to meet rapidly changing requirements, before adopting commercial integration products once the new features are incorporated.
At the growth stage, challenges are exciting with a pool of interested and experienced developers. As a project becomes more stable with a requirement to support customers and business, the key driver is maintaining the system and its integrity which might be better met by the support model of buying commercial integration products.
Should a business driver or disruptive feature impact a ‘stable’ element of companies’ applications, then it may be moving back into the ‘growth’ phase requiring greater flexibility that can be provided by In-house integration solutions.
A real-life example is at LinkedIn, which was a first of a kind professional networking and career development social network. It started as a home-grown monolith but as growth escalated there was a need to better cache and store data in streams in a way that wasn’t readily available in commercial integration products.
Where existing products did meet requirements, such as databases, these commercial products were utilised. But the need to rapidly meet new requirements drove In-house software solutions. Over time, as user numbers stabilised many of these tools were release as Open-source including Kafka, which today stands as a commercial integration product utilised by other organisations looking to achieve the same functionality [1].
Conclusion
Some Greenfield companies have fast overtaken their legacy competitors or are closing in on the market providing features and innovation faster than established companies could manage. To achieve this, greenfield companies have custom built integration software and/or built on top of Open-source allowing them to rapidly change course, experiment with new features and be flexible in how they do business as systems and processes evolve.
Over time, these custom-built systems become more fixed and stable to support a large customer base with an expectation of stability. To meet these requirements, the support mechanism of commercial integration products is attractive. As are the out-of-the-box solutions that allow accelerated integration with other systems, drawing on the experience of the commercial integration products resource pool for business-as-usual requirements like upgrades, security and basic integration methods and techniques.
By using commercial integration software whose focus is on integrating and who have mastered the methodology over decades to meet diverse industry needs, these companies can focus on their primary business aims i.e., travel, finances etc. and not information technology and its integration.
When needing to be flexible and rapidly grow, building custom code and applications can allow a company to disrupt the market, expand its client base and become more profitable. As the code base stabilises and peak market share has been gained, the ability to continue to service those customers and businesses, requires an integration product that is supportable.
For many, they will transition (at some point) from an internally managed code base to an imported product produced and supported by a commercial integration software.
Special thanks to Kim Clarkfor his review and advice
Whether writing an article, blog, code, creating a video or any other document, it's important to get other people to review what you have created. This ensures the quality is good, the content reads well and that the message being portrayed is correctly expressed.
Looking for a Reviewer
First, you need a reviewer. Reviewers come in all types and forms, but generally you want someone who can read and comment on the technical content of the document, someone to assess the readability alone and either could judge the relatability. Does it read well for all the intended audience(s), does it make sense and does it resonate with other people's needs? Basically, is it worth reading?
Once you have found a reviewer, have a brief chat to discuss the document:
Explain who is the intended audience and the context of the writing
Check if the language is suitable for non native speakers
Ensure they know they can be open and honest, it's pointless asking someone who will just say "looks great".
Post Review
Once the reviewer has completed their review and sent it back to you, if they have done their job well there is likely to be a number of comments. Questions, suggested changes and direct edits, but a sea of changes can be difficult to process and accept.
A lot of times after a review, I tend to go through the 5 stages of grief:
Denial - There's no way they understood the point of the article, there's nothing that needs changing. It's perfect! come on!
Anger - Well, I'm not making THAT change. This comment doesn't even make sense...
Bargaining - Okay, well, actually maybe I didn't write that bit very well. Okay, it's possible that I have rambled a bit too much.
Depression - Why do I write anything at all? I should hang up the pen or let someone else write this instead. Time for a sprinkle of imposter syndrome!
Acceptance - That change I made thanks to that comment is pretty good. Actually, this reads quite well now. This is so much better.
I dramatise for effect but it certainly is a whirlwind of emotions when you get comments, especially at first or with a new reviewer. Over time it gets easier and when you have a couple of reviewers you trust its easier to jump straight to bargaining.
Make sure to thank your reviewer for their time (they didn't have to read it, and they didn't have to put a lot of time / energy into it either, they could have skim read).
Coping with review comments
There are a number of ways to cope and handle the review comments, in order to bypass at least 'denial' and 'anger' stages and possibly after a while the 'depression' stage too.
Here are my 6 top tips for coping with review comments:
Time to process - take some time, have a cup of tea/coffee, sleep on it and come back when you are happy / not too stressed
Assume the reviewer is acting in good faith - even when they are not
Try it out - Make a copy, try it out, does it look or feel better. You won't know if its better with the changes until you make them. The worst that happens is you back out your changes and revert to the original but generally there are always improvements to make
Has the reviewer misunderstood the audience, purpose or context? If so, is it obvious in the text? find out where the misunderstanding could have happened and either ask the reviewer to reread or add the context into the document
Have a call and talk it through - go through the comments, discuss what changes could be made, clarify the context, explain your thought process. This can accelerate future iterations and help get published quicker.
Don't take it personally!
Conclusion
Document reviews are very important, they make our work better by ensuring the words we write have an impact. Whether improving readability, making it more accessible, helping get direct to the point and increasing how many people actually consume the content.
You can listen to our latest podcast on New Year's Resolutions here
Introduction
Each year I make a couple of New Year’s Resolutions, including a reading and running goal. To stay on track, I know I can quickly calculate how many kilometres I should have run so far this year or how many pages I should have read.
Rather than work this out manually every day, I thought I would create an Apple shortcut to give me an update. I hadn’t made one before, so thought I would combine learning something new with helping keep to my goals, so here’s how I did it.
Figure 1 - My Goal Shortcuts
The following results are from my first day playing with Apple Shortcuts. Thank you for your patience with the poorly named variables and probable sloppy steps to achieve the end results. Feel free to comment where I can save time or add efficiency.
Reading Goal
My goal is to read 8000 pages in a year, so I need to read 8000/365 pages per day (~22 pages a day). Times this by the number of days passed this year and you have the daily target. Easy enough in January, bit harder as you move through the months.
The following is what I set up in the Shortcut:
“Set a variable” as the Current Date called ‘Today’
“Get the time between” the last day of last year (31/12/2021) and the variable ‘Today’
“Set a variable” called ‘NumOfDay’ to the result of “Get the time between”
Do a “Calculation” of 8000 / 365
“Set a variable” called ‘PagesPerDay’ to the result of the “Calculation” function
Do a “Calculation” of ‘PagesPerDay’ times ‘NumOfDay’ i.e. on 8th January – 8 * 22 = 176
“Set a variable” called ‘PagesSoFar’ to the result of the ‘PagesPerDay’ times ‘NumOfDay’ calculation
Round the ‘PagesSoFar’ to One Place
Speak the Rounded Number
Below are the images to make it a bit clearer.
Figure 2 - How many pages should I have read shortcut
The title of the Shortcut was “How Much Should I Have Read” which is what you say to Siri to get the output. Like today 9th January responds with 198.
Running Goal
I can repeat the same steps for my running goal of 600km. Changing the variable names and the 8000 to 600.
Figure 3 – How many kilometres should I have run shortcut
So, I know how many kilometres I should have run, but how can I track how many I have actually run using Apple Shortcuts? Well, there doesn’t seem to be an out of the box Strava, Nike+ or Fitbit shortcut to get run totals or add runs.
Tracking Runs
There is an extensive set of Apple Notes shortcuts, so I set up a Note called “Running” where I can add keep a tally of runs. From that note, I was able to create two additional shortcuts, “Add Run to Total” and “What Distance have I Run”.
Figure 4 - Running Note
Add Run to Total
To add new runs to the Note, the Notes “Append” shortcut was used with the dictated text to allow a number to be said. Start off by saying “Hey Siri, Add run to Total”, Siri asks for input, you say how many kilometres you did on your run and it will add it to the end of the note.
The following is what I set up in the Shortcut:
Add “Dictate Text” function
Use the “Get numbers from” function from the dictated text
“Set a variable” called ‘RunNum’ to the Calculator Numbers
Use the “FindNote” function looking for a note with “Running” in the name
Use the “Append” function to add ‘Numbers’ to the returned note.
Figure 5 - Add Run to Total Shortcut
What distance have I done?
As I’m adding the runs, there needs to be an easy way of totalling them and then asking Siri to tell me the total without me having to add it all up. This is fairly simple if we iterate over the runs in the Note and add them to a variable before getting Siri to speak the variable back to us.
The following is what I set up in the Shortcut:
Use the “FindNote” function looking for a note with “Running” in the name
Split the returned Note by New Lines (one for each run)
Use the “Calculate” function to instantiate a variable soon with 0 + 0
“Set a variable” called ‘TotalRuns’ which is a result of the previous calculation
Start a loop of each line of the note by using the “Repeat” function with each item in the split text
Discount the title line “Running” using the “If” function with a check that the repeated item isn’t “Running”
Inside the if add the current lines run amount ‘Repeat Item’ to the running total variable ‘TotalRuns’
Use “Set a variable” to set the already created ‘TotalRuns’ to the calculated value
End the IF
End the Repeat
Set some text using the “Text” variable to be “km this year”
Use the “Speak” function to output the ‘TotalRuns’ variable
Use the “Speak” function to output the ‘Text’
This will output the total run distance this year from the Running Note with “km this year” after i.e., 12 km this year
Figure 6 - What Distance Have I Done shortcut
So, What Next?
I can add runs to the Note, I can get the total distance I have ran and the target number of kilometres for the number of days passed. Next might be to get how many kilometres over or under the target I am.
Other shortcuts that might come in useful:
Automatically add items to my “AnyList” shopping list
