The zebra printer that brought down the network
5h agoOne day there was a bunch of switches in the manufacturing plant that kept going up and down at random times in the morning. There had been some networking changes such as wifi upgrades, network attached battery backups and camera installations that i though could of been a culprit to the increased network traffic on the older switches. After doing a ton of investigation around the traffic on the switch ports I was able to identify through a wireshark session that a zebra printer in the office was sending out massive amounts of broadcast traffic across the network not only crashing switches but also slowing down end users computers.
We had sporadic network outages across the manufacturing plant and office over a 4 hour period while we were troubleshooting. This caused delays in manufacturing. When this happens it typically caused problems in lead times for our customers as well.
Some lessons learned throughout this process that we improved on was setting storm control for broadcast traffic on our switches to limit the amount of traffic that could be sent over a single port but also we started to put into place specific vlans for various device types to help contain network traffic a little better. With this zebra printer being a little older we also started to update a lot of our fleet to newer equipment to prevent some of the issues older equipment caused with troubleshooting on the network