Ethan Banks

Look for inexpensive solutions to network downtime issues. Often, there's just no budget to buy a new shiny object. So, be resourceful. Look at how what you have can be reconfigured to be more resilient. You don't have to have the latest and greatest gear to build a resilient network. With some thoughtfulness and a bit of ingenuity, you can often find ways to improve network uptime with some cheap or free design changes.

Learning from the experience of network downtime is a team exercise. IT management should foster blamelessness. What exactly happened should be clearly understood. What changed? (A configuration update was made. A routing peer was lost. A link went down. A firewall crashed. Whatever. Something changed.) That change started what chain of events that resulted in network downtime. And what, exactly, do we mean by "network downtime" in this incident? Was a whole data center down? Or was the Internet inaccessible for a while? What could have been done to prevent this aka how does this never happen again? Having a blameless discussion with all team members is educational, fosters camaraderie, and grows the group's confidence and skills.

Verifying that the network is functioning as expected after a fix is key. However, testing network behavior during failure is also key. A good network design eliminates single points of failure, budget-permitting. Introducing failures during scheduled test periods helps prove the network can automatically recover when a failure happens. Routing protocols should quickly converge on a new path when the primary path is lost. Switching domains should settle on a new spanning-tree topology when the root bridge is lost. Redundant network appliances, such as firewall active/standby pairs, should carry on when one member of the pair falls over. Redundant links should have enough carrying capacity for all traffic if one link is lost. Etc.

"Frequent" network downtime is unusual, in that the network is the foundation on which all services depend. In the face of truly frequent downtime, say multiple times per month of network outages, isolating the issue might be more complex than it sounds. The likelihood is that there is more than one issue, and that the "issue" isn't a device malfunctioning or other failure. It's that the network itself requires a redesign. Therefore, a chronically failing network needs expert eyes to evaluate the network and recommend fundamental design improvements to improve uptime and survive failures. That redesign will likely be disruptive to operations, and will therefore be implemented in phases. Unless you're lucky and a quick fix can be found.

Network automation is a tricky discipline. Don't get ahead of yourself. To automate, you must first understand the network process you're automating very well. Consider that automation isn't really about writing a playbook or script that helps a single task be completed more quickly. While automation will start there, the ultimate destination is a collaborative system that everyone in the organization will use together, replacing individual, one-device-at-a-time CLI work. The super-cool, one-off Python script you write for yourself today will be the tool that's part of a system everyone will use tomorrow.

For the average network running both switching & routing, consider the following... - Switchport optimization. Assign the switchport mode (trunk/tagged, access, etc.). Use port security to prevent unauthorized access if you're in a regulated environment. Shutdown ports feeding to unused wall jacks in common areas. - Spanning-tree optimization. Control root bridges. Protect your STP topology with guards that prevent a misconfigured switch from changing it. - Routing protocol optimization. Use security to control neighbors. Keep configurations as simple as possible. For example, don't use multi-area OSPF without a specific design requirement. - In all cases, monitor devices logs for unusual messages related to switching & routing.