Your team is divided on the server crash cause. How do you navigate conflicting opinions to find the truth?

Review server logs to identify errors, warnings, or any abnormal behavior before, during, and after the outage, as well as any recent changes to the system, such as software updates, configuration changes, or new deployments that could have affected performance of the server.

Antes de ler 800 mil linhas de logs pra minimizar o impacto, precisa abrir gerenciador de tarefas pra ver se o consumo de algum recurso está no máximo travando tudo, além de analisar o comportamento do servidor, qual erro está apresentando? O que parou de funcionar nele?

Please keep in mind that time is critical when these events impact critical systems. I recommend implementing a band-aid first to restore functionality. This will give your team the ‘runway’ to spend the proper time analyzing the root cause.

My first step in the analysis would be to trust the system logs. Once I understand the underlying problem, I would involve team members for further analysis, keeping in mind that the entire team is responsible for the failure.

There is only one truth and that is server logs not people’s opinions, and the sooner the IT industry sits up and eradicates that sort of behaviour the better for the rest of us who know how to find out the truth to any situation

Review the logs and metrics to find common patterns or anomalies when the crash happened. We can correlate this information with different theories suggested by the team. We can use third party debugging tools or monitoring apps to find the root cause. We need to document every steps of analysis and finding.

Once the data is collected, I lead my team in evaluating each theory presented. We consider the likelihood of each scenario based on the evidence, maintaining an impartial stance to ensure all plausible causes are thoroughly examined. This methodical approach helps us avoid prematurely dismissing viable theories.

Well user human error not intentionally most times are a starting point sign. Check that as a starting point. I have found in my experience and analysing the error is a start. Do not assume then move onto cause of crash all

We then select the most likely theories and devise tests to validate them. For instance, if a memory leak is suspected, we simulate similar conditions to monitor memory usage. This iterative testing process helps us either confirm or rule out potential causes, guiding us closer to the actual issue.

All hypotheses must be support for a lot of data & metadata. Get together the maximum that you can, try to simulate the same event, learn the maximum what you can from it. Get all learn lessons documented, spread the new info for all to avoid future problems.

Testing hypothesis: Try to reproduce the issue as much as you can. Sometimes, it is not possible to recreate the exact problem due to tools not being available, such as hardware applications or an exact environment. However, creativity and innovation to work around and recreate the issue as closely as possible is a key benefit, not only when testing your hypothesis, but also in finding the solution. Another important aspect of this practice is that when configuring the environment to reproduce the problem, you may encounter and get to know a lot of facts related to it that you were not aware of.

Following the evaluation of theories, choose the most plausible ones and design tests to validate them. For instance, if a theory suggests a memory leak caused the crash, monitor memory usage under similar conditions to test this hypothesis. Conduct tests in a controlled environment to confirm or rule out potential causes. Remember, hypothesis testing is iterative; if initial tests are inconclusive, refine your approach and continue testing until you reach a definitive conclusion.

Putting theories to the test. Transform theories into testable hypotheses. Develop a rigorous testing plan, creating controlled environments that mimic the conditions of the crash. Conduct systematic experiments, meticulously documenting each step and result. Be prepared for unexpected outcomes, they often hold the key to deeper understanding. Analyze your test data with a critical eye, using it to validate or refute each hypothesis.

Une très bonne communication garantie un bon suivi et une traçabilité sur le problème. Le référencement d'une documentation associée sur un SharePoint où via un outils ITSM tel que glpi ou Itop garanti une bonne base de connaissances pour une prise en main plus aisée

Clear communication is vital. I present our findings to the team using visuals like charts or graphs to ensure everyone understands the evidence and reasoning behind our conclusions. Open dialogue helps address any remaining doubts and fosters a collaborative approach to problem-solving.

Keeping everyone informed.Craft a clear, concise report that outlines your investigation process, presenting data and theories objectively. Highlight the most probable causes based on your evidence, but don't shy away from addressing uncertainties or conflicting data. Present your findings to the team, encouraging further discussion and insights. Remember, effective communication is key to building consensus and moving forward united.

From the very start of this process, it has to be communicated to the team that this exercise is to prevent the same issue happening again. Be clear that it is not a witch-hunt on who made the error or caused it. But a team effort to further strengthen the server, or the system.

With a consensus on the cause, we implement the necessary solutions, such as patching software or adjusting system configurations. Documenting these steps and communicating them to all relevant parties helps prevent future issues and enhances system reliability.

After the cause is identified , implement the solution to prevent future occurrences. After implementing the solution monitor the system for signs of same issues. Retrospective review will be a good idea after implementing the solution to reduce occurrence of similar issues.

From theory to practice. With the root cause identified, develop a comprehensive plan of action. Prioritize immediate fixes while also considering long-term improvements to prevent future occurrences. Assign clear responsibilities and deadlines, ensuring accountability. Rigorously test all solutions before deployment, and maintain heightened vigilance post implementation to catch any unforeseen issues.

When it comes to the root cause of a problem we always need to back our answers with evidence. This evidence needs to be presented with the answer. The cause of the crash was due to this and here are the logs to support it. So when there are conflicting truths, the evidence can only back one of the truths if any at all.

Don't fix it ASAP. Assuming you have a decent redundancy this is an opportunity to develop better monitoring for the future and possibly even automatic recovery.

Although you are trying to build or maintain a resilient system, keep in mind you are also building a resilient team. If a team member's work or actions, according to your data of course, seems to cause most of the issues- consider a retraining program for the person or let them go- they might flourish elsewhere.

I have a philosophy, in IT, which may be of use here. I’ve said it and written it several different ways, but the premise is: “All information is data, but to make sound decisions we need the facts.” I usually follow this up with some sort of request for facts, like: “show me facts” or something like that. Facts should be simple and should speak for themselves. Facts should not be burdened with emotion or justification. In this type of event, help your team focus on fact collecting, leaving all else aside; especially emotion.

Logs, logs.. need evidence of the crash and look at what was happening on the system (cpu, disk, I/o, hardware failures…etc) when the crash happened. Look for past similar crashes. Make sure all servers and services have complete monitoring in place. Centralized logging makes triage much simpler.