Original post is on medium.

Authors: Sachin Tanna and Aiden Gallagher

Introduction

Technology has undergone a period of growth and expansion since the 1970s. With this growth has come a wide diversity in languages, platforms and protocols used to develop and execute functionality on systems and to connect the systems.

Integration technologies were and still are the industry’s answer to this connectivity demand and companies have invested heavily in developing and marketing products that supply the capability to integrate systems, whether in house or through dedicated commercial integration software products.

But do commercial integration products still have a place in the modern industry, or are home grown integrations — using custom, bespoke software and Open-source implementations — a more flexible and efficient way for organisations to integrate their disparate systems?

In this article we will discuss why greenfield software development might be better suited to home-built software rather than using commercial integration software products, but why, as they become more established and stable, there is a necessity to use an out-of-the-box commercial software solution to address ongoing business needs.

The Evolution of Integration

Business systems and machines began as silos, completing their independent functions in isolation. Integration is the process of connecting one or more of these systems together, but as system integration was adopted, the communication was found to be unreliable and slow, and it could be difficult to interface between them.

Different integration software techniques were developed to solve these problems including messaging, file transfer, remote procedure calls and shared databases. Companies that had previously been dedicated to their primary business goals slowly began to adopt IT and integration systems, which expanded and grew to consume larger amounts of the business’ resources.

This led to the creation of specialised commercial integration software created and maintained by companies like IBM to provide simpler and easier integrations for companies, to allow them to focus on their core aims.

Organisations that have evolved with very-early IT (1960s-1990s), tend to have large IT estates and have a large amount of commercial integration products embedded within then. They have gained knowledge of how to safely manage, maintain and build systems that meet diverse business, regulatory and consumer requirements, through long standing processes and governance procedures that ensure integrity. Many are recognised for this ingrained culture and slow response to market demand.

How is the integration landscape changing?

The IT world has been changing. Computing has become part of everyday life and the prominence of mobile devices mean companies are at the beck and call of their customers, day and night; as they wish to use their applications, interact on social media, make queries and raise issues.

For example, Daisy is shopping on Etsy at 2am and wants to make a payment through their bank. If there is an issue, Daisy uses the live chat support where a chatbot tries to help before passing Daisy on to a person. The payment needs some form of authentication which needs to be setup and used to complete the transfer right away. Anything less will be a pain and Daisy might be tempted to switch to another bank.

The systems involved in meeting Daisy’s requirements, require backend integration with Daisy’s account and payment data which might be spread across disparate systems. The resolution to Daisy’s problem may also require the calling of multiple different services to complete a business process with responses integrated between the systems.

To deliver for people like Daisy and maintain a competitive advantage, companies have the need to be able to rapidly build solutions that access, synchronize and combine data from disparate systems across the organization in order to react to new innovation. This is done through integration applications and software which in turn, must be highly available, secure, able to keep up with market needs, able to innovate at speed and where possible, able to reduce costs.

The types of changes are shown in Figure 1 — Changing IT Landscape Factors (2021) but can be summarised as such:

· Lightweight runtimes through container technologies are helping to deploy smaller, more flexible applications that can be owned by teams and deployed more easily.

· API Led Approach is making it easier and simpler to access data from other systems such as databases. This can be used to provide additional value or can be monetised for organisation gain.

· Cloud Native application deployments built for the cloud which are more portable (saving costs), scalable to meet changing demands and utilising implied resiliency of cloud deployment. (http://ibm.biz/cloudnativedefined)

· Advanced Tooling to automate all aspects of the application, system and integration life. Create, build, deploy, monitor, alert and debug using automated tooling, improving security, access control and time to market in the process.

Figure 1 — Changing IT Landscape Factors (2021)

These changes have allowed existing companies to provide better services, but many struggle to fully capitalise on the benefits because of their ways of working. Greenfield companies are building new applications, software and features from scratch based on these principles, and are not encumbered by existing processes, governance, maintenance of legacy systems. They also do not have an enforced requirement to utilize existing commercial integration solutions already embedded in the organisation.

Greenfield Software Solutions

Greenfield companies have emerged to disrupt existing markets, in almost every industry, from the way we socialise (Facebook, Twitter, Instagram, LinkedIn), the way we bank (Monzo, Revolut), the way we shop (Amazon, Etsy), travel (Skyscanner, Uber), order our food (just eat, Deliveroo), manage our finances (PensionBee, Moneybox, eToro, Freetrade) and much more.

When building these new companies there is a decision whether to use commercial integration software already in the market, Open-source technology that can be adapted by the company or their partners, or integrating systems using custom software. Whilst commercial Integration software solutions have been widely utilised and relied upon across industries, they are constantly compared to custom, in-house (or Open-source), code-only integration solutions.

One criticism of commercial integration software solutions is the lack of flexibility. For example, a business may want to take advantage of a new feature such as Open API v3, but commercial products may take a while to catch-up with the specification. In-house code could be updated and the feature prioritised in a much simpler and quicker process that goes straight from developer to the live system.

Commercial integration products do provide a low-code approach, but could be seen as a “black-box” in terms of their functionality, which can prove important when it comes to debugging. Byars (2021) argues that code-based solutions are easier to “diff” for changes [2]. However, if the volume and complexity of code is scaled out to match the breadth of functionality that commercial integration products provide, this debugging process could become equally or more challenging.

Businesses may also be hesitant to implement commercial integration products to avoid vendor lock-in. The counter balance to this is the reduced effort for businesses by utilising existing, commercial integration products which are backed by documentation and a wide support structure of product experts built up over many years.

Greenfield companies often use Open-source because it provides them with the ability to quickly adapt software to meet their needs. This assumes they have the skills and time to build software as well as running their businesses, and that they can keep those skills once the initial project is complete. For implementing leading edge capabilities, this may be the only way to create the functionality sufficiently quickly, because making adaptations to commercial integration products requires communication, waiting for changes to be discussed, prioritised, architected, coded, tested and eventually released. This is before the Greenfield company runs its own testing that these new features meet its requirements.

This lack of control when using commercial integration products is the focus when high-impact issues affect systems. The Akamai CDN outage (July 2021) and Log4J library vulnerability (December 2021) are prime examples, companies felt unable to deal with the issues due to a reliance on external entities’ support and communication. In-house software can be worked on internally with additional resources, temporary workarounds and fixes in the hands of internal teams.

When to keep on developing

Teams building in-house integration software and developing on top of Open-source can continue to do so providing they are capable of maintaining and managing the code base long term. This means ensuring there is large team of good developers with the knowledge and ability to support all the tools and code used within the systems, and that talent can be retained going forward which can often mean ensuring the challenges are exciting and the company a good place to work.

If internal development teams are more agile and able to deliver good quality fixes to bugs and issues, faster than the commercial integration software company, it lends the team a competitive advantage especially if competitors are using commercial integration products that are taking a while to catch up with the latest trends.

In-house software allows direct and rapid communication with partners, consumers and communities who can request new features directly (e.g., through forums) and the support teams can provide more direct support to those requests and in some cases communicate live updates.

The key reasons to continue internal development is to meet unique use cases and rapidly changing requirements, to ensure software growth can meet the pace of company growth and identify where new features and experiments are still needed to provide a good quality product that can compete on the market.

Using commercial integration products

Commercial integration products have been honed and enhanced over many years with a well-established knowledge base, progressive evolution, and subject matter experts who specialise in integrating systems and working through industry wide problems. The primary reasons why these might be used over In-house software include:

· Saving time and cost by not re-solutioning common use cases that have been solved before

· Established support models from consultancy firms or contractors to build on the tools using previous experience to accelerate time to market

· Reduce complexity of integrating with others

Companies that use commercial integration products do not need to start from scratch, from day one they can integrate point A to point B, with mandatory support mechanisms in place for issues found, a breadth of experience for common solutions and no necessity to understand deep level integration concepts such as protocol implementation. The more unique the functions, features and skills that are required to integrate, the less cost-effective commercial integration products become, because of the commercial consequences of supporting edge case scenarios.

Growth is a key element of success for any company, especially greenfield, but at some point, there is a need to adopt more professionalised and established methods of management, with a renewed focus on existing customer retention. As products and companies become more established, government and regulatory requirements might be imposed to protect consumers. For example, General Data Protection Regulations (GDPR) implementation cost US Fortune 500 and UK FTSE 350 companies a combined $8.9 billion and was an additional cost of unplanned business and continued maintenance [3]. Commercial integration products provide this for a range of customers reducing the individual cost of supporting long term regulatory costs.

For commercial integration product vendors, creating and maintaining software products is their core business. For other businesses, software creation is a distraction from their core business. This is especially true when supporting business as usual systems and applications.

By Contrast, Open-source integration software has a support model that relies on community good will. There is no liability to fix problems, and security breaches can be easier to exploit due to the openness of the source code. There are obvious benefits of participating in Open-source communities to work on and improve key industry features as a collective. However, Open-source can become outdated, the community can move to other projects and active members in an organisation can move on to other challenges. This increases dependencies on the remaining active members and at times a need to pivot entirely to another tool which can be costly.

Commercial integration software companies have a business driver to produce integration products that work over a large proportion of the industry, over a long period of time. Development teams have expansive experience of integrating systems and have evolved with the market over product lifetimes which for some, are now reaching three decades of life. These products have the experience of past failure, resolving issues, enhancements to meet changing needs, they are also liable for the support mechanisms they have in place to resolve issues as they arise and patch security bugs as quickly as possible.

Established products are incentivized to evolve with changing IT landscapes to retain business and often work with customers to ensure continued positive relationships. When the company grows, so too does the integration product servicing it.

If a greenfield company begins to hit turbulent times, it is difficult to justify the cost of managing and maintaining In-house integration solutions. Focus needs to be on customers, sales/retention and becoming profitable. The cost of creating and maintaining integration software can increase if there are inexperienced staff, if supporting existing products becomes untenable because it has grown too big or because employees have moved elsewhere, then costs can balloon, and projects stagnate.

Whilst there are advantages to greenfield software solutions in terms of control of resolving security vulnerabilities, commercial integration software products have had many years to consider and enhance the security hardening on the product. Utilisation of these pre-built Integration products reduces the duration of effort required for security planning on a brand-new product, the same can be said for networking, high availability etc.

It is important to consider the cost of integrating with other organisations and business. When producing code internally, companies have an element of control. But this can become an unwieldy when integrating externally and is exacerbated the more integration points that are involved. There may also be greater support and documentation available for integrating an out-of-the-box solution with an external product, with pre-built connectors and accelerators to integrate with well-known systems. For example, no-one today would consider writing an EDI integration solution as that is so well established.

Choosing an integration model

Commercial integration software and products provide evident benefits when integrating using well-known systems through established mechanisms. They provide a strong support through experienced integration developers and specialists providing companies the ability to focus on their primary aims rather than the integration of internal and external systems.

However, building in house integration software has its own clear advantages. It provides flexibility and easy adaptation to meet edge case scenarios and innovate ahead of the industry and competitors.

This is not a binary decision; it is possible to switch models although there is some initial outlay cost for the transition. Applications using commercial integration software can expand to start using custom code for specific purposes to meet rapidly changing requirements, before adopting commercial integration products once the new features are incorporated.

At the growth stage, challenges are exciting with a pool of interested and experienced developers. As a project becomes more stable with a requirement to support customers and business, the key driver is maintaining the system and its integrity which might be better met by the support model of buying commercial integration products.

Should a business driver or disruptive feature impact a ‘stable’ element of companies’ applications, then it may be moving back into the ‘growth’ phase requiring greater flexibility that can be provided by In-house integration solutions.

A real-life example is at LinkedIn, which was a first of a kind professional networking and career development social network. It started as a home-grown monolith but as growth escalated there was a need to better cache and store data in streams in a way that wasn’t readily available in commercial integration products.

Where existing products did meet requirements, such as databases, these commercial products were utilised. But the need to rapidly meet new requirements drove In-house software solutions. Over time, as user numbers stabilised many of these tools were release as Open-source including Kafka, which today stands as a commercial integration product utilised by other organisations looking to achieve the same functionality [1].

Conclusion

Some Greenfield companies have fast overtaken their legacy competitors or are closing in on the market providing features and innovation faster than established companies could manage. To achieve this, greenfield companies have custom built integration software and/or built on top of Open-source allowing them to rapidly change course, experiment with new features and be flexible in how they do business as systems and processes evolve.

Over time, these custom-built systems become more fixed and stable to support a large customer base with an expectation of stability. To meet these requirements, the support mechanism of commercial integration products is attractive. As are the out-of-the-box solutions that allow accelerated integration with other systems, drawing on the experience of the commercial integration products resource pool for business-as-usual requirements like upgrades, security and basic integration methods and techniques.

By using commercial integration software whose focus is on integrating and who have mastered the methodology over decades to meet diverse industry needs, these companies can focus on their primary business aims i.e., travel, finances etc. and not information technology and its integration.

When needing to be flexible and rapidly grow, building custom code and applications can allow a company to disrupt the market, expand its client base and become more profitable. As the code base stabilises and peak market share has been gained, the ability to continue to service those customers and businesses, requires an integration product that is supportable.

For many, they will transition (at some point) from an internally managed code base to an imported product produced and supported by a commercial integration software.

Special thanks to Kim Clark for his review and advice