The State of State - how state fits into modern applications

By Aiden Gallagher and Andy Garratt - Special thanks to Callum Jackson for his review and thoughts.

Introduction

Modern applications are built to be light-weight, easy to deploy and expendable - especially when combined with containerised deployments. Because of this, a train of thought suggests that applications should therefore be stateless.

"My Application is Stateless."

Whilst an application can be stateless, this does not help in scenarios where applications simply cannot be stateless such as an application processing booking which includes add-ons, or payments being processed with a card holders bank. 

Even if state is secured primarily in some data store using some of the patterns outlined later in the article, it is still essential for processes like transformation, security and movement to be performed which are not best suited in the data store layer.

In this article we will discuss why state is important, how stateless processes are different to stateless applications, why stateful applications can be integral to a sophisticated and reliable system and why applications without any state have limited value.

Table of Contents

INTRODUCTION

12 FACTOR APPLICATIONS

SO, WHAT IS STATE?

CONSIDERATIONS OF STATE

STATE PATTERNS

Availability Patterns

Last Man Standing

Primary and replicas

Sharding

Storage Patterns

Data Lakes

Caching

Contention and locking

Transient State Patterns

Cargo Pattern

Event Sourcing

STATE PATTERNS IN CONTAINERS

Third Party Delegation (Volumes)

Replica Sets

Stateful Sets

MODERN STATEFUL APPLICATIONS

CONCLUSION

12 Factor Applications

One of the primary reference points for modern applications is the 12-Factor methodology, which gives advice on good practice when creating software as a service or web application.

A guiding principle of the 12-Factor (modern) Application is that applications should be stateless and that there is no sharing between processes.

"Twelve-factor processes are stateless and share-nothing. Any data that needs to persist must be stored in a stateful backing service, typically a database." - https://12factor.net/processes

The driver behind this decision is the ability to have self-contained processes that can be deleted, rebuilt or upgraded easily and without risk to user journeys; but state is important and in many cases it cannot and should not be avoided.

Even in the case of containers, state can be managed if thought about and cared about. The application is able to gain greater control and independence by not relying on an external service where a large amount of trust must be held by the application developer in regards to the reliability and security of the datastore.

So, what is State?

"The particular condition that someone or something is in at a specific time." - Oxford Dictionary

There are three types of state that are used as part of the management, creation and usage of applications; whether to handle the configuration of an application, the replication of applications for availability or the messages that pass through the application itself. These are;

·      Long Term State - State that is held for a long period of time, usually in a database but occasionally in long term memory. This state is designed to be accessible over longer periods that might span generation of applications, software versions and even underlying products themselves.

Examples: Tax Returns, Grades received at university, Birth Death and Marriage Records, purchase records over time etc.

Additionally, there is transient state in applications - live information that is subject to change whilst part of a transaction. The information is important to the complete process that is being fulfilled and without it, the transaction will fail, will be repeated or will no longer be subject to audit.

·      Persisted Transient State - Whilst completing an application process, each point that the process reaches in the journey needs to be recorded for ‘in-flight recovery’ of transactions being made.

Examples: Payment failing - did it fail before or after the payment was made?, Application failed whilst adding new stock - Is the stock displaying the right amount or does it still need increasing?

·      Non-Persisted Transient State - During an application process the current progress is not needed. Often this is because a failure will not result in any harmful effects to the system the application interacts with.

Examples: An API that retrieves all existing supermarkets in a town, reloading a webpage with static data

State comes into play for a variety of reasons whether to pass relevant user information such as their permissions and account details, or to ensure that transactions are persisted, and their actions only committed a fixed number of times. State is also what is used to describe an application’s make up; its sizing’s, its connections, how it is expected to handle traffic by the wider system and how many instances are required - without this state applications cannot be deployed.

State can be difficult because it requires the tracking of a piece of data and any changes to that piece of data, it requires thoughts to what should happen to the state should a failure occur either in retrieving relevant state, the application flow, the application or the runtime hardware of the transaction. It is this difficulty that leads to the avoidance of its usage in the first place. 

As we see in the 12 Factor Application methodology rather than deal with all state as a necessary and needed part of an application, long term state and persisted transient state is instead passed to other services whilst non-persisted transient state is described as permittable through a single-transaction cache. However, the cache data is not assured and is never assumed to be available for any future transactions. 

Methodologies - like 12 Factor applications - that infer that state should be reduced and eliminated where possible misunderstand the importance of state within an application, and the limited value these applications provide.

There are methods for handling state that do not require a blanket ban on its usage which will be explored later in the article.

Considerations of State

There are a number of factors that derives state of an application and its acceptance into an application. It is important to understand how that state should be handled both in the application and at the storage layer. Below are some factors to be considered;

·      State 'Life' - What is the requirement for the state to be kept? What are the ramifications of losing the state? If a user had to resubmit the transaction what would the impact be? Can we lose the data safely? How should data be stored and how can it be accessed if the application was to be lost i.e. will the database lock

·      Order and Sequence - Does the State rely on the order and sequence of event being adhered to? Will calls be required to be synchronous or asynchronous? How does this fit into a general wider modern application strategy?

·      Governance – Who is permitted to use the State? How can the state be accessed? Can data be altered in transit? 

In Databases, state is managed by applying ACID properties:

·      Atomicity - Determination of whether a transaction happens or does not happen

·      Consistency - Use of standards to enforce consistency across the application

·      Isolation - Controlling of concurrency to give the application complete independence where other parts of the same application cannot see or interfere with the transaction being completed

·      Durability - When a transaction is completed the onus is on the application that the transaction has been passed to handle that state even if the system is to shut down.

Similar techniques can be applied to applications to aid in achieving transient stateful applications.

State Patterns

There are several patterns which are currently used by existing applications to ensure that state is handled correctly;

Availability Patterns

Last Man Standing

This pattern looks to dynamically assign voting power based on the number of remaining servers in a cluster to allow a set of weighted applications and servers to reduce to a single entity in the event of issues arising in the cluster. This looks to allow applications and state to continue to function as the number of entities in the cluster are reduced.

Advantages:

• Allows all servers in a system to be utilised for availability by readjusting a quorum
• Reduces chance of outages by allowing a single server to survive on its own

Disadvantages: 

• Can lead to contention as to which is the current source of truth
• Adds complexity to the architecture of the system
• Requires that less than half the quorums 'votes' fail in an outage so that quorum can be rebalanced

Primary and replicas

By having a primary source of data that either pushes its state or has its state copied, then the Primary can be used as the single point of truth for all applications. The use of replicas means that there is a backup of data should anything happen to the primary whereby a replica might become the primary source of data.

Advantages: 

• Little to no downtime in the event of an outage at the data source. 
• Updating of replicas can be performed on changed files only making the procedure relatively quick

Disadvantages: 

• Requires a lot of additional infrastructure
• Has to contend with availability requirements
• Can be affected by disassociation between Primary and secondary in the event of a network outage

Sharding

This method separates data into independent, self-contained data blocks known as 'portions' which can be moved across a cluster. This is also sometimes referred to as partitioning and in a database might be completed either at a vertical or horizontal level i.e. columns or rows respectively. 

Note: Many benefits listed below also make Sharding a ‘storage’ pattern too

Advantages: 

• The spreading of data allows for greater requests to the database and faster processing in horizontal scaling
• Searching for data can be quicker by looking for data across shards before honing into a individual shard rather than searching a whole database you search smaller parts of it to find the relevant data. i.e. searching for 'Fred' in Folder 'F' is much simpler than searching for the same name in Folder 'Names'.
• Mitigates against outages by making the risk on a subset of the database not the whole thing, each partitions availability can be handled in isolation.

Disadvantages: 

• Greater risk of loss of data or data corruption if not correctly thought out and implemented
• Hard to revert back to a single database after Sharding has occurred
• Partitioned can become 'unbalanced' meaning the risk is pushed into a single partition reducing the benefits. i.e. partitions based on home address would have a significantly more data for United States addresses than Monaco.

Storage Patterns

Data Lakes

This method stores all data being used in a common resource in its raw format. This data can then be selected and used as and when each system requires the information.

Advantages: 

• Data can be obtained at any time, in any order by the connected application giving greater flexibility
• Stores different data types in a single data store
• Can be queried more diversely making it accessible to different application types.

Disadvantages: 

• Data can end up sat unused in a giant expensive system – sometimes known as a data swamp
• Storing all data in a single location can create risk as a single point of failure – even if highly available
• Querying can take time which may lead to latency in transactions

Caching

This method stores data in a temporary storage area. This is usually relatively small data based on previous computation or the result of a previous request which is retrieved by the application or database from the cache on the next request. There are many forms of caching, below are some generic local caching advantages and disadvantages.

Advantages: 

• Allows for reduced latency as data is already stored locally to the application or database meaning no waiting for further processing or a response
• Throughput can be improved as a result of speed enhancements 

Disadvantages: 

• Large Caches can cause memory usage on the application processing the requests which might slow the application processing speed
• Inconsistencies may be found in the cache as the true state changes

Contention and locking 

This method, most commonly found in database update operations looks to lock either an entire table, a single row or a single cell in order to prevent contention between two versions of a table updated by two entities at a given time. In applications the same principles can be utilised to ensure that data and resources cannot be accessed whilst being used or altered. 

Advantages: 

• Ensures that conflicting data cannot be saved at the same time
• Ensures that the most up to date data is being accessed by all connecting applications

Disadvantages: 

• Applications might have to 'wait' to update the data source
• Applications waiting for access to the data source might fail, timeout or become indefinitely blocked
• Applications that have locked the data source will be tightly coupled with issues of the data source i.e. if it runs out of space and needs manual intervention to proceed

Transient State Patterns

Cargo Pattern

This method is used for ‘picking up’ and ‘dropping off’ state as part of a transaction and determines if state is carried to the application at a specific time i.e. during an event, or if the application actively retrieves the data it requires at a determined point in the transaction.

Advantages: 

• State is only retained and collected in the relevant locations – all state not carried everywhere
• State can be stored by the relevant parties providing greater security of data
• Based on the States status at each point of the transaction, it can be determined how far through a transaction a request has gone (for repudiation etc.)

Disadvantages: 

• Data has to be stored between processes causing latency and overhead
• Messages and state might need to be cleaned by a secondary process

Event Sourcing

This method stores information in an event object. At a basic level this pattern looks to ensure that changes are made in the order in which they are sequenced which allows us to understand how data has changed over time, not just how it currently exists.

Advantages: 

• State can be traced across time and therefore a great picture of state changes over time can be established and losses of data greater understood
• Application state can be destroyed and rebuilt by rerunning an event sequence
• Greater control over how mistaken changes are rebuilt i.e. a change to the sequence before replay or a parallel implementation with correct change to understand divergence/affect with the new change

Disadvantages: 

• Larger data storing required
• Event objects themselves still need retaining and their availability secured

These patterns are not new and can be used in most modern applications and are the basis of many software’s both in containerised infrastructures and on traditional static deployments. But how do these patterns apply and how can they be used, safely, to allow applications to manage and use state.

State patterns in containers

When considering state in the context of containerised application there are three main patterns that can be used to manage and ensure that state is correctly handled and retained to prevent data loss. It is important to note that containers and their configuration are stateful concerns of containerised systems and must be treated as a stateful artefact which is a dependency of the application.

Third Party Delegation (Volumes)

State is never retained on the container or its application. Instead all Stateful data is passed off to a third party which can then be accessed via the application when the state is again needed. One example might be accessing a public certificate from a public endpoint via a REST API, or the configuration of a container is stored in a source repository.

Advantages:

• Applications are expendable; can be killed or might fail without too much worry
• State can be handled by none containerised application, modules etc
• Configuration of containers and application data can be restored with relative ease

Disadvantages:

• Availability of the Third Party needs to be assured
• Messages lost in transit need to be completed again which adds complexity to integration flows

Replica Sets

Replicates the configuration of a container template, ensuring that a set number of replica containers exist. This is used to provide a load balancing style of high availability.

Advantages:

• Availability of the container is assured
• Can access persisted data on a third party without impacting the end user

Disadvantages:

• Messages lost in transit need to be completed again which adds complexity to integration flows
• Reliance on the underlying container management system
• May be dependent on being able to retrieve updates of the configuration templates from their storage system

Stateful Sets

An application that has state retained within the container would need to be rebuilt with the same state if it was to be lost, additionally the order and uniqueness might be relevant to the redeployment decision of the container by the container management system. This would also need a volume; however, this might be locked to a container that has specific characteristics.

Advantages:

• Applications are expendable; can be killed or might fail without too much worry
• Not reliant on third party availability and connectivity
• Messages not lost in transit as they are stored and retrieved from a volume

Disadvantages:

• Requires stability in naming and sourcing of the container i.e. DNS
• Requires the volume to be persisted
• There are some complexities that might require manual intervention during container deployment

Modern Stateful applications

In modern applications replication of both the container and the state therein needs to be considered in the context of applications that can be terminated, scaled up/down or stopped both unexpectedly and on demand. 

The container management system handles a lot of this contention through the use of container stateful patterns which might include a mixture of stateful sets, replica sets and the storage of state on third party applications.

But state is not expendable and will continue to be important for a multitude of requirements such as the processing of payments - which should never be completed twice or that need to be audited. Container systems should still consider state patterns as dependence on external or third parties simply moves stateful requirements elsewhere. 

Even where state is offloaded the availability and connection to the third-party storage has to be hard wired into the applications functionality as contention, dissociation and other data issues might occur during transit or whilst making a connection.

By using a mixture of existing state patterns alongside container management system stateful functionality it is possible to create stateful applications that fit into the modern application architecture.

Conclusion

In the above article, we have shown what state is, how existing applications deal with state and how state can be effectively handled and managed by modern applications in containers.  The article looks at pattern usage in these environments and describes how stateful application might be scaled.

A Year In Books - 2019

In 2019 I set myself a challenge to read 26 books but managed to sneak in a last book between Christmas and New Year to take me to 27.  In this post I'll give my overall rating for each book and a little description as to why.

Books of 2019 - a few missing because they were borrowed or returned

1. Raymond E Feist - Silverthorn (1985) [8.5/10]

Great follow up to 'The Magician' and what felt particularly good is that it was a completely different storyline to the previous book. Classic adventure book about following a quest but with multiple storylines interwoven following the characters from the previous book. Very easy read.

2. Neil Oliver - A History of Ancient Britain (2011) [9.2/10]

This book is fairly large for saying that it ends in the Roman period of Britain. This is less a look into the political landscape of Britain and more how the ancient peoples lived and died. What struck me was how early Brits began to claim land through ancestral rights and that the only buildings might have only been graves in early Britain. There's loads of really useful and interesting information!

3. Markus Zusak - The Book Thief (2007) [9/10]

I watched the film several years ago on a plane which was fairly ok, but the book was really well written and shows a normal family who didn't support the nazis but had to appear to in order to fit in. As an avid book reader it was very heartwarming to read.

4. Bernard Cornwell - The Sword Song (2007) [7/10]

Having read the previous books in the series, this was pretty repetitive hence the 7. Nothing wrong with the book at all, written as well as all his other books.

5. Bernard Cornwell - The Burning Land (2009) [8/10]

Next in the series after the above. Much better than the previous story wise. Uthred doing Uthred stuff and helping form England. I also started listening to the history of England podcast so some of the names and words started lining up.
https://thehistoryofengland.co.uk

6. Bernard Cornwell - Heretic (2004) [7/10]

The third and final book in the Grail series to conclude whether Thomas of Hookton gets the grail and avenges his father. Nice way to end the series, a little predictable but that might be because I didn't have a break between books, so three Bernard Cornwell's in a row. 

7. Robert C. Martin - Clean Code: A Handbook of Agile Software Craftmanship (2008) [6.5/10]

A long read which was I have reviewed in my blog before. Some great tips to take forward in my day to day work. This is definitely not a 'fun' read and some of the tips seem somewhat obvious, but putting the thought process of writing clean code into words was extremely useful.

8. The Freedom Writers with Erin Gruwell - The Freedom Writers Diary (1999) [2.8/10]

I don't think there was anything wrong with the book, I enjoyed the film but for whatever reason I couldn't really get into the style of writing. There were lots of diary entries talking about the same events and it just wasn't my cup of tea unfortunately.

9. Christina Dalcher - Vox (2018) [8.4/10]

A modern take on The Handmaidens Tale it reminded me of a book I read along time ago, but can no longer find (it was a modern take on 'animal farm'). I like it when old books with a very pertinent message are rethought and this one is fantastic. In a world where women are limited to a certain number of words it obviously grates because its ridiculous, but makes you think that if things are not as they should be, in work, at home etc. we are all responsible to stand up and help each other. very well written!

10. Neil Gaiman - The graveyard book (2001) [9.5/10]

Best book of the year for me. It was the first Neil Gaiman book I had read and it was so well written and magical.

11. Neil Gaiman - American Gods (2004) [9/10]

Very different to The Graveyard Book. Read this whilst on holiday in the USA and there are some disturbing parts, but it's a very innovative story about the gods brought to America by the people who remember them. Great read although thought the ending fell a little flat - but that might just be because my holiday was over.

12. The Secret Barrister - The Secret Barrister: Stories of the law and How It’s broken (2018) [9/10]

Before reading this book I hadn't really given the law, court, the judiciary service and life after a rightful or wrongful accusation. This is a little bit of a heavy read for most, not exactly a page turner but gives a really important insight into how the law works in the UK, what is affecting the system and how it could affect you, the law abiding reader and thus why you and I and everyone should be invested in keeping it impartial and fair. 100% recommend as a read.

13. Pat Baker - The Silence of the Girls (2018) [7.4/10]

Looking at the battle of Troy from the point of view of the women who might have lived through the times was a great premise. The view is normally about the fighting, heroism and adventure these epic battles brought and how warrior men were affected (The Odyssey for example) so it was refreshing to see the world from a different point of view.

14. J. K. Rowling - Harry Potter and the Philosophers Stone (1997) [8/10]

Classic. I had read it before and fancied a nice, easy read which it duly delivered on. You can't deny the magic in the pages especially when every time you read it it takes you back to being a carefree child.

15. Terry Pratchett & Neil Gaiman - Good Omens (1990) [7/10]

Had slightly higher expectations of this book, partially because of its already humungous following and the TV series but it was signifcantly not as good - in my opinion - as any Terry Pratchett or Neil Gaiman book I had read before. It was funny throughout and an easy read.

16. Adrian Tchaikovsky - Children of Time (2015) [9.2/10]

I'm sure other books have been written with similar plots before but this really was a great read. Humans escaping an uninhabitable earth must fight for a place on a terraformed world already inhabited. This has evolution, space travel, human politics and even had me giving sympathy to spiders - and I'm a massive arachnophobe. 

17. Anthony Burgess - A Clockwork Orange (1962) [7/10]

This takes a while to get past the writing style which is purposely obstructive. The first quarter of the book is gruesome and not particularly nice at all, however, it does make you think about what makes people violent, and whether the ability to have rage, anger and the choice to be good or bad is an essential part of being human.

18. Martin Amis - Time's Arrow (1991) [6.5/10]

Written by flowing backwards through time the book is a great concept and reads easy enough, but the plot doesn't really go anywhere. lots of little funny parts so, worth a read.

19. Khaled Hosseini - The Kite Runner (2003) [7.2/10]

A book I had intended to read for several years but never got round to, it shows the senseless destruction caused by war and ideology, it shows how wealth, power and achievements can be diminished in an instance and the small bubbles we inhabit within our own communities. 

20. Henry Williamson - Tarka the Otter (1927) [6.5/10]

I hadn't realised how much otter hunting had occurred and, how recently the practice has continued to. The book shows from the otters point of view as we follow his life, his childhood and how it fights to survive.

21. Michel Faber - Under the Skin (2000) [7.4/10]

This was unusual, and post reflection makes me wonder what the book was trying to say. However, I struggled to put it down the I was reading it and finished it in a few days.

22. Neil Gaiman - Stardust (1999) [9/10]

I love the film and have watched it several times. Its amazing how true the film was to the book; which was just as funny, exciting and brilliant as the film had been.

23. Stephen King - Misery (1987) [9.4/10]

Another book that had been on my read list for a while. Stephen King is phenomenal at making you feel like you are living the story, the fear of the author and the experience they are having on the page. He puts so much thought into the plot, the characters and how events play out and this shines through as you read because it becomes so believable.

24. Roger Hutchinson - Calum's Road (2006) [7.2/10]

We took a trip to Scotland this August and one of the books I found from looking for "books based in Scotland" was Calum's road. Calum himself was a no nonsense man who tried to make his community lived in and hang onto his ancestral way of life by building a road. It was interesting to see how rural life has been affected by centralised government, education and a move away from the local.

25. H.G.Wells - War of the Worlds (1898) [7.5/10]

Although this was written donkeys ears ago, it still read as incredibly realistic and plausible. An astonishingly easy read. The film with Tom Cruise did it a great justice and the book I read came with lots of interesing information on H.G. Wells

26. Stella Gibbons - Cold Comfort Farm (1932) [7.3/10]

This book is hilariously written and was a great one to complete my 26 book challenge on. A little mystery, michevious and buckets of dry laughs which are the essential ingredients for a good read.

27. Christina Henry - Lost Boy (2017) [8.8/10]

A novel spin on the classic story of Peter Pan showing the boy who never grows up in a much darker light then you ever might have thought. This is almost a thriller and follows one of the lost boys called Jamie in his adventures on the island where they all live.

A History of Application and Service Integration

For the last few years, I have been writing the knowledge I have been acquiring into a document, looking at how the products I work with in integration came to be, what led to their prominence and the problems they were and continue to try to solve.

This slowly got bigger and bigger as I incorporated the knowledge and discussions I had with my colleague Andy Garratt. The document looked at EAI, ESB, FTP, TCP/IP, SOA, Middleware, Hub and Spoke, ACID, AIA, Microservices, REST, SOAP, XML and many many more integration topics to boot. There was so much in there, that we felt we needed to share the knowledge with the wider world to help set the context of integration to those new to the industry, and maybe even those who are already in the industry.

As of this week, we now have published "A History of Application and Service Integration" which is available to download and read on iBooks: https://books.apple.com/gb/book/a-history-of-application-and-service-integration/id1489103440

Below is the accompanying video that describes A brief History of Integration in under 3 minutes.

Alternatively, you can download the pdf directly from here.

Bringing the Chaos Monkey to heel

Introduction

Systems, microservices, services and applications can go wrong. Error messages in the middle of the night, important messages getting stuck on the system, memory and disk space overload, network glitches and any other number of unknown conditions that might befall these systems. 

To gain confidence in these systems being able to continue working - despite issues - is a focal consideration in any architecture. The ability for a system to be able to service consumers despite issues in one or several locations is known as its availability whilst the ability to restore working order after issues is the systems resilience. Through availability and resilience, a system can be created that is capable of dealing with any issues that befalls it.

Confidence is gained by performing stringent tests on a systems hardware, software and connectivity points. Sometimes these tests are spread across multiple environments with specific purposes i.e. integration test or user test. In other methods, testing is performed on a live system itself to gain the confidence that in these adverse scenarios the system can handle or recover itself.

Controlled destruction of parts of a live system – to prove its availability and resilience - is known as chaos engineering which has manifested in the container world through the Netflix model of the Chaos Monkey.

In this article we will look at what Chaos Monkey is, how it benefits systems, when it is appropriate to use chaos monkey, considerations when using chaos monkey and considerations for gaining confidence in systems generally.

What is Chaos Monkey?

In 2010, Netflix published an article referring to transition to the Cloud. One of the lessons learned was to fail constantly in a controlled manner in order to ensure that the system would continue working in the event of a failure. Thus, was born the application ‘Chaos Monkey’ whose job was to randomly kill instances and services.

A further article in 2011 described Chaos Monkey in greater detail. An actively chaotic application would randomly disable, kill and remove instances to prove the system could handle the failure and gracefully bring itself to a healthy status.  

The chaos was controlled by ensuring it only happened during the working day, with a team of specialists available to handle any fallout and delivery any fixes that might be required. This raises an understanding of how the system behaves in events that are not ‘expected’ and builds a wealth of knowledge by the teams handling the various systems of both cause and effect.

The article went on to describe a whole “Simian Army” of chaos which would independently test: 

·      Latency

·      Cross Regional Dependencies 

·      Clean-up of old systems

·      Best Practices Conformance

·      Health Checks

https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c

https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116

Chaotic State

What is not addressed in either of the articles is the ability to handle State. State being messages and configuration that need to be retained. State is important for a number of different reasons, these might be security, assurance of service delivery or even regulatory i.e. an audit.

There are three main types of State;

·      Long Term State - State that is held for a long period of time, usually in a database but occasionally in long term memory.

·      Persisted Transient State - tracking and recording a process through each of its parts so that it can be resolved in the event of a failure i.e. if a payment fails before completion

·      Non-Persisted Transient State – data that is required, but is not needed to be tracked i.e. stock check

The state might be a message, the configuration of an application or the configuration of a deployment management system itself. Chaos Monkey doesn’t mention how to handle these concerns, how the chaos is mitigated or the restriction you may need to place on the ‘chaos’. 

Consider in a container management system such as Kubernetes – how would you handle the deletion of a Helm chart from the system, would the helm chart be checked and pulled from another system? What about the reference and connection to the repository that holds all the state of the system?

Consider a system that handles a payment agreement between two systems – a salespoint and a bank. If the transaction is accepted by the bank just before the chaos monkey application kills the process, the sales point will not have confirmation of the payment and thus the customer making a purchase on the sales point will be charged but will not get their goods. 

This speaks more about managing state in cloud and container-based systems but also means that the chaos monkey application needs to be aware of state too and considered during the application design.

Chaos in Production

Chaos Monkey works well in systems like Netflix primarily because the loss of service during the working day to a live system is manageable from both a support and customer relations point of view. If, as a user, it isn’t possible to stream a video for a period of time the consumer might be annoyed by the inconvenience but little more. 

In a system where a vital service is being provided, the loss of service even for seconds may have adverse effects. Defence Agencies, Health Care, Bank Processing, Aviation and other high-risk systems it would be difficult to justify the risk of a system being down, even for testing. 

In high-risk systems the loss of service can have real life-changing effects to consumers. If, for example, a train line system was to drop a red-light signal the result could be injury or even death. To intentionally break the live system through a form of testing seems irresponsible when the costs of these systems failing are so high. 

To alleviate this risk, some of these systems will perform testing and upgrading in disaster recovery systems which resemble exactly the system in Production. This is not always possible though, as some systems require the disaster recovery system to be aware and available of production at all times which can lead to contention of which of the two is the active production system.

In the previous section, the discussion of state raised the issue around a transaction being killed mid process. In test environments this would only have an effect on test accounts that are being used, but in a live production system, real customers will face the effects of state failure.

A final thought is that the purposeful destruction of something that is live and working seems against what many infrastructure and solution teams try to build - resilient systems that are architected to run and survive. In the Netflix article the analogy of getting a flat tyre is used, the chaos monkey being that you give yourself a flat tyre on the driveway on a Sunday morning. If chaos monkey was truly chaotic, the analogy would really involve blowing the tyre out whilst driving along the motorway.

This seems dangerously unsafe for any system, indeed puncturing a tyre on the driveway is a significantly less random form of chaos. It is more like a test environment in any case and is a very drastic action to take. In the flat tyre analogy, the driver could get the same confidence by testing each of the component parts independently – inflating the tyre, changing the tyre, ensuring you have the correct tools etc. 

In all readings, chaos monkey comes across more as controlled production testing. Even without chaos monkey, systems should complete disaster recovery once a year to perform the most drastic of tests. As such, chaos monkey is not truly chaos and since it is testing should be treated as such.

Adequate Planning & Testing

If chaos monkey can be dealt with as testing, there are some reasons why it will not be described as ‘chaos’. The application performing the killing of services and instances can only do what it has been programmed to do and as such its remit is known and could be planned for without the actual destruction.

Whilst unknown consequences may occur from chaos monkey this can - and in most cases should – be performed early in the test cycle in lower environments. This also raises the importance of having a production like environment before production which is a replica in every way.

Since chaos monkey is testing, there are the typical positive and negative considerations that comes with it. The cost, value and risk. Testing on the scale of chaos monkey requires willing to perform testing in production by those who own and support it, the budget to spend time finding issues over completing new functionality.

Real Chaos

In the real world, chaos is just reality in play. There is no way to plan for the unthinkable, unplannable and unconsidered. Problems will always occur and by its nature chaos is chaotic. Unmanageable, unplannable, uncontrollable – in a way that cannot ever be represented through test.

If chaos cannot be controlled, then it should be instead be prepared for. Chaos monkey allows for this preparation by testing the fall backs systems have in place. There are several thought processes that can be put in place;

·      High Availability – what happens if each component part breaks? Can it fail over? Can I resume service quickly and easily?

·      Disaster Recover – what happens if an entire active system fails? Can it fail over? How long does it take to return to service and how much time should be set aside to get the system back to its full availability? (Recovery Time Objective and Recovery Point Objective)

·      State – how is state handled in the event of each failure? Is the system able to recover a transaction mid process? 

·      Risk – what risk is acceptable? How is each risk mitigated?

·      Testing – Is there a safe place to test all eventualities outside of Production? Can Production be replicated sincerely?

Conclusion

Whilst chaos monkey is not chaotic it has great worth as a testing facility. It can be used to gain confidence in a system, understand the consequences of actions and mitigate risk. Chaos monkey activities are relatively controlled and managed and as such constitutes an advanced cloud-based testing strategy.

In contrast to the teachings of chaos monkey, testing might not always be possible – or safe – to perform in Production but should be instead performed in a test environment. At least one of these test environments should look exactly like production in every way, where each component of each fail-safe strategy is tested.

Beard Trimmer Salvaging

For my 18th Birthday (almost 9 years ago!) I got a Phillips QT4013/23 Series 3000 Beard Trimmer from my Auntie and Uncle, it has served me effortlessly since. I probably had no beard to trim until I was 20 which means it has worked hard these last seven years.

Recently the head snapped and unfortunately there were no replacements heads which meant it was to be consigned to the scrap heap. Instead, I thought I would see what I can salvage from within.

On a side note - its very very frustrating that you cannot just buy parts and end up buying something completely new - incredibly wasteful.

Disassembled

The razor head itself came off quite easily but as you can see from the top all it would be good for now is a hair comb. The razor blade clicks out, which evidently means that *it* can be replaced - it's made up of a spring and two white prongs which are moved from left to right by the rest of the razor.

The rest of the workings were trapped inside the three plastic cases on the left. The dial in the centre of the far left plastic pieces, pushes a motor higher or lower. The motor causes the prongs on the razor  to move from left and right causing the cutting motion.

The motor is powered by two batteries on a motherboard with a switch at the back which can be turned on or off by the right most plastic cover. The additional benefit of the battery setup is that it can be recharged by applying the charger to the bottom which will be useful in future projects - as I can make a solar charger charge it up.

Not a lot that can be saved here except the battery, metal pins and the motor. The rest will be consigned to the scrap head :'(

Football Simulation Engine v3

2 Years ago I published my first node package module (npm) in npmjs to allow football match results to be simulated. Yesterday I uploaded the latest version FootballSimulationEngine-v3.0.1 which can be installed by running:
          npm install footballsimulationengine

What's New?

startPOS and relativePOS have changed

The biggest - and breaking - change was to update the arrays used to track where players currently are on the pitch (startPOS) and where players are moving towards (relativePOS). These were not very descriptive and resulted in several GitHub issues so they have been changed to better reflect what they do. "startPOS" is now "currentPOS", "relativePOS" is now "intentPOS".

"currentPOS": [340,0],

"fitness": 100,

"injured": false,

"originPOS": [340,0],

"intentPOS": [340,0],

Tests tests tests

I have been reading more and more about the importance of tests and clean code, in fact its been prominent on this blog:

• https://aidensgallyvanting.blogspot.com/2019/03/observations-of-test-driven-development.html 
• https://aidensgallyvanting.blogspot.com/2019/03/enforcing-test-driven-development-into.html
• https://aidensgallyvanting.blogspot.com/2019/05/practising-clean-code-in-nodejs.html

As described, I have added lots and lots of tests into the code making it more readable, easier to make code changes, has led to variables and functions being named more accurately. The result is 143 passing (199ms) tests and roughly 55% test coverage.

55.88% Statements 1401/2507

44.32% Branches 655/1478

82.13% Functions 170/207

53.14% Lines 1227/2309

Penalty Position - Corner and Freekicks

Players no longer form an arbitary collection of bodies at a set location for freekicks and corners, instead they now gather in the penalty box which is a defined dimension on the pitch. This makes corner taking more realistic and improves goal opportunities from corners.

Identifiers

Previously, the current ball holder and the team they were playing were matched against their names for certain logic. i.e. can the player kick the ball? First, we need to know if that player has the ball. Now each match, player and team are given an identification number during initiation. 

{

"matchID": "78883930303030001",

"kickOffTeam": {

"teamID": "78883930303030002",

"name": "ThisTeam",

"rating": 88,

"players": [

{

"playerID": "78883930303030100",

 

Red Cards

Players who get red cards will no longer be part of the playing action. This is shown by current position being set to ['NP', 'NP'] which means 'Not Playing'. If the current position is set the player will not be targeted for passing and will not make any movements.

Skill Rating Utilization

Player skills are better utilised including to make shots on or off target depending on their skill against a random number generation i.e. players with shooting skill of 40 will be less likely to shoot on target than players with shooting skill of 70. This is also applied for saving shots that are in the box with the skills of the goalkeeper determining if the ball is 'saved' or not when in reach. 

Future iterations will take this further for more sophistication. If you have any thoughts for how to do this be sure to raise an issue in GitHub.

Player Stats Improvements

Player statistics from during the game have been greatly improved with statistics for fouls, passes, shots and tackles. This will improve reporting of player information. 

If there are others that can/should be added, raise an issue in GitHub.

'goals': 0,

'shots': {

'total': 0,

'on': 0,

'off': 0

},

'cards': {

'yellow': 0,

'red': 0

},

'passes': {

'total': 0,

'on': 0,

'off': 0

},

'tackles': {

'total': 0,

'on': 0,

'off': 0,

'fouls': 0

}

Conclusion

There are still some improvements to be made, to player movement (which still isn't quite there), passing statistics, intelligence of decisions being made and whole lotta tests to be added. I'll be blogging soon about how I've used this latest version to simulate the 2019/2020 season to showcase how the engine can be used to simulate a season for a football manager game.

Let me know what you think, either in the comments, or by raising an issue on Github or emailing me on aiden.g@live.co.uk! Happy footballing!