How do you cope with variability in scalable distributed systems?

Therefore, while designing a distributed system, which of the concepts of "Consistency", "Availability" and "Partition Tolerance" should be chosen by being aware of the CAP theorem, the possible consequences of this abandonment should be shared with stakeholders. In this way, for example, we do not fall into a dilemma, such as having to include a component that does not have "Partition Tolerance" in the system, and then having to add this feature to the whole system, or we do not make promises to stakeholders that we cannot keep.

Aside from embracing failure, there are several other key practices that can help ensure the reliable and efficient performance of scalable distributed systems: Automation: Automate deployment processes, scaling procedures, and recovery mechanisms. This reduces the chances of human error and allows the system to respond quickly and consistently to changing demands. Data Replication and Redundancy: Replicating data across multiple nodes and data centers can improve fault tolerance and reduce data loss. Load Balancing: Distributing incoming requests evenly across the available resources can prevent overloading specific nodes and ensure that the workload is efficiently distributed, thereby enhancing system performance and reliability.

"Ways in which things go right are special cases of the ways in which things go wrong." ~ John Allspaw 1) Identify the potential system failures, include all network disruptions as a list. like Database failure, cache failure, or dependent microservice failure. 2) Stack rank based on customer impact. 3) Solve the top failures. Implement panic mode which can serve basic functionalities for customer and gracefully shutdown lower impact services to recover faster.

Simplicity is the hardest position to achieve in software. It is balance between delivering for current problems and the ability to tackle futuristic problems. Ability to see what can be the next potential problem and what not helps to filter the futuristic problems. - Understanding the existing problems in depth. - Aligning with organization goal in projects. - Having the conversation with your leader who has the bird's view of projects. can help you to understand more on the futuristic problems and stack rank them. Software design adhere to rule not to introduce solutions for every futuristic problems but the next important ones. Keep what is absolutely needed, eliminate all others.

Uncoordinated systems, wherein work items can be handled independently, scale the best. Note that coordination is not a binary feature but a spectrum.

Latency directly improves the customer satisfaction. Also, it increases the throughput of the system in major cases. Other than external improvements like redis caching, load balancing, request coleasing, some of places you can look for latency improvements inside the application are 1) Avoiding redundant outbound calls and caching them in variables/thread bindings. 2) Optimising database queries. Queryable fields in p0 cases should have been indexed/point-in queries. 3) Removing the costly frameworks/abstraction which is not needed. 4) Fine tuning the framework's configurations like database connectors, http connectors, object mappers etc.

When we talk about scalability in today's tech landscape, automation and orchestration emerge as these game-changing powerhouses. I mean, think about it. We're dealing with this uncertainty and variability, and what do we need? We need tools that can adapt on the fly. That's where automation steps in. It's like having this digital assistant that takes care of all those mundane, repetitive tasks that can bog us down. And not only does it make our lives easier, but it also cuts down on errors and costs. It's a win-win. But wait, there's more! Orchestration, my friends. This is where we take things to the next level. We're not just automating individual tasks; we're orchestrating the entire symphony of our system components.

Agile leadership and scalability play crucial roles in successful adoption of automation and orchestration tools. Teams must adapt quickly to emerging trends and changing landscapes. Agility: Scalability enables an organization to adapt/automate and maintain agility as it grows. It’s not about adding more teams but ensuring effective coordination and alignment across teams. Lean Thinking: Embrace lean principles—minimizing waste, maximizing value, and optimizing processes. Cross-Functional Collaboration: Encourage collaboration across functions and teams. Facilitate communication, knowledge sharing, and alignment. Find The Right Partner: Commit to tech development, allocate software/resources, and focus on value (vs. repetitive tasks).

Observability is the key factor and the continuous process in scalable systems. Performance disruptions has to be alerted instantly and handled gracefully. In addition to monitoring key SLOs, keeping the monitoring pipeline healthy should be the top priorities. Weekly SLI review, Incident Management processes like post-mortems help us to apply the continuous improvements in any scalable systems. Business Planning should include the above process in all systems and consider it as separate management activities rather consider it as a part of tech debt/works.

Look before you leap US Patent 8670971 A method is provided for evaluating workload consolidation on a computer located in a datacenter. The method comprises inflating a balloon workload on a first computer that simulates a consolidation workload of a workload originating on the first computer and a workload originating on a second computer. The method further comprises evaluating the quality of service on the first computer's workload during the inflating and transferring the workload originating on either the first or the second computer to the other of the first or second computer if the evaluating the quality of service remains above a threshold.

Also design to be secure. A well designed scaleable system lends itself to supporting security concepts like Separation of Concerns and Defence in Depth. Avoid making your architecture too 'flat', it makes it easier for a hacker to find a weakness and move through the system.