Just time to have a quick rant about the work machines, on Tuesday night while out at dinner I got a call from Marco telling me that the site appeared to be down, a quick talk with him (this was actually a quick talk, I spoke too quickly and Mike had to take over) and worked out that it was one of our web application nodes being locked due to an APC bug (it's on my todo list to track down).
All was fine as freddie managed to graceful Apache on the machine and I continued eating.
Before we got home I got another call telling me the servers were completely gone now, couldn't ping our main load balancing machine but the web application and database servers were all up. Got someone to open a ticket with our hosting company and went to finish eating.
Got home, logged in to our main database server and then went around checking to see if I could identify what was up, by this point our hosts were planning to reboot the server because that always fixes everything. Took me about 30 seconds to work out our external NIC was down on the load balancer, ethtool showed me everything was up just that there was no active link.
Responded in ticket asking them not to touch the machine and look at the cause to see if a retainer clip had came loose or if someone had tripped over a cable. One hour later we were told the port on the switch had been deactivated by a security mechanism, apparently some form of DoS.
Found out this morning that we'd apparently just had more than 15, 000 packets a second go through the port and this triggered it, no DoS. Probably release traffic but the fact it took them 2 hours to work this out.
In case anyone is wondering who the hosting company is its The Planet, the servers may be cheap but sometimes I think their support can be troublesome.
Hehe... I got two words for you, reboot technician.
It tooks us roughly two days to figure out why a server at ev1/theplanet was constantly rebooted. Apparently a "reboot technician" - whatever that is - issued them on the wrong server.
I should add that the issue at hand was not so much about figuring it out, but more about getting them to admit it. :-)
Posted by: till | February 14, 2008 at 12:23 PM
Only arouse human pursuit of beauty, she can get beautiful desire itself.Do you think so?
Posted by: Ugg Cardy | November 01, 2010 at 09:21 AM
Thanks for your this informaiton,i'm learn more in that!
Posted by: Buy GHD | November 01, 2010 at 09:26 AM
The decoding is completely case insensitive and ignores all punctuation apart from exclamation marks.
Posted by: Pearl Pendant | December 25, 2010 at 04:59 AM
It tooks us roughly two days to figure out why a server at ev1/theplanet was constantly rebooted. Apparently a "reboot technician" - whatever that is - issued them on the wrong server.
2945abc45 1231
Posted by: ipad accessories | December 31, 2010 at 05:49 AM
Many thanks for the article. I will have a link back to this information from our fresh blog.
http://www.cheaphatcaps.com
http://www.cheaphatcaps.com
http://www.cheaphatcaps.com
Posted by: cheap Monster Energy Hats | July 14, 2011 at 08:03 AM