Customer War Stories
iGLASS augments and enhances IT teams and their companies, helping to improve network uptime through proactive monitoring their networks, servers, applications and websites.
But you don’t have to take our word for it. Here are some real-world examples, or “Customer War Stories”, straight from the trenches.
Backhoe Breaks Bank’s Backbone. Bummer.
A financial services customer leased fiber links from a regional carrier to network their offices together. One night, their connectivity was interrupted and caused their regular nightly backup to fail.
As part of a predefined run-book, and rather than wake the customer at 3:00 AM, our NOC engineers opened a ticket with the fiber carrier. The carrier verified the outage and sent a crew to find the break. A big yellow backhoe was in the area and, yep, that’s exactly where they found the fiber cut.
There had been a water main break and a utility crew had accidentally cut their fiber while digging. Once service was restored, iGLASS restarted the backup and updated the trouble-ticket. When the customer arrived to work the next morning, refreshed from a good night’s rest, they had a full accounting of what had transpired and how it was handled. Always look for the backhoe!
Logging in the New Year
In the wee hours of the morning, a few days after the world welcomed the new year, one of our customer’s key web servers started running out of space on it’s primary hard drive.
A batch job ran nightly to delete old log files but had been accidentally hardcoded for the prior year. Since it was now the new year, the job didn’t run and the log files started filling up the disk. iGLASS was monitoring the disks, detected the problem and our NOC staff jumped into action. With a predefined run-book as their guide, the NOC logged into the server and manually cleared out the old files. They followed up with a call to the the customer’s IT staff during regular business hours, the problem was identified and ultimately fixed. Our quick action avoided a potentially costly outage. If the hard drive had filled up, the server would have crashed, directly affecting revenue.
Always Test Your Generators
One of our cable system operators had planned ahead and installed a UPS and backup generator at each of its key hub sites. During a recent power outage, one of their backup generators failed to start, putting their systems at risk.
But because iGLASS proactively monitors each UPS and generator 24x7, we immediately alerted the customer and they had a local technician start the generator manually before the UPS batteries ran out of power. iGLASS saved the company from lost data, lost revenue and the massive headaches that accompany a total outage. Our customer followed up by implementing a weekly generator test.
Late one evening in June, several Ethernet interfaces on a law firm’s core router shut down. Our NOC called the customer’s on-call IT engineer, who then drove out to their datacenter to investigate.
It turns out a piece of paper had been sucked against an air intake vent on the chassis, causing a card to shut down due to high temperature. The shutdown saved the card and iGLASS enabled the customer to minimize the downtime. Once the card cooled down, service was restored, we verified the links were back up (and the offending paper was introduced to the shredder!)
Copper Caper Causes Customer to Call Cops
A cable operator in Indianapolis had a remote hub site housed inside a strip mall. iGLASS monitored a spike in ambient temperature from within the hub and immediately notified the customer’s on-call technicians.
At first they thought it was a false alarm since they hadn’t heard from their security monitoring firm (who was supposed to monitor environmental alarms too) but when the temp hit 109F, they dispatched someone to check it out.
Upon arrival at the site the tech discovered some brazen thieves had harvested all of the copper off of the primary and backup HVAC units causing a complete cooling failure. He opened the doors and turned on some fans, avoiding a potentially catastrophic equipment failure and customer outage. Police were notified but the suspects are still at large.
Don’t Forget about Set-tops!
A cable customer in Alabama had a digital video outage but none of their other tools raised the alarm.
Well, none of their other tools monitored set-top boxes! They used to think monitoring their DOCSIS plant was sufficient. It wasn’t until their DMOD locked up and stopped issuing IP addresses that they truly realized the value of monitoring their triple-play environment.
Don’t forget the set-tops!
A customer in Mississippi had a complete power outage at a remote hub site. iGLASS NOC engineers let them know within minutes and they sent a technician to investigate.
Upon arriving at the site, the tech was surprised to find the crispy remains of a 7-ft python inside one of their circuit breaker cabinets. No doubt the python was also surprised, albeit briefly, when it made contact with two terminals which would have been better left alone.
The critter was removed, breakers reset and power restored with minimal disruption as (thankfully) the outage occurred in the middle of the night. A missing conduit knock-out on the breaker cabinet was also closed up to avoid a repeat performance.
Traps can be a Good Thing!
A customer in Rochester, N.Y., is one of the nation’s largest metro-area fiber optics broadband providers to ISPs throughout the northeast.
While iGLASS was monitoring their core router for availability, bandwidth by VLAN, CPU and other key metrics, the status of internal power supplies usually isn’t monitored proactively. Because iGLASS will receive and process SNMP traps and Syslog however, we had them covered.
One Thursday in November Power Supply B died on the router and we alerted the customer within minutes. They didn’t have a spare, so they ordered one and had it shipped overnight. They swapped it out on Friday and then on Friday evening Power Supply A died! It turned out they had a power issue coming into the cabinet (it was running Hot!) and a quick visit by an electrician squared them away. If iGLASS hadn’t alerted them to the power issue however, the outage could have affected literally thousands of customers. A great example of why monitoring SNMP Traps (and Syslog) is so important!
VOD-Addicted Middle-Schooler Busted!
A cable MSO in Austin, Texas, came to us with a problem. A single, working lady received her cable bill and it listed over $100 worth of VOD purchases. The problem was… she never watched VOD.
Because iGLASS monitors VOD utilization down to specific set-tops, we were able to pinpoint the dates and times each feature was viewed – weekdays between 2:45 and 3:45 PM. The operator “staked out” the lady’s condo accordingly and made a startling discovery: A middle-schooler lived next door and got out of school at 2:30. He walked home and (because he was forbidden from ordering VOD features at home) grabbed his remote control, walked to the lady’s patio, pulled up a chair and ordered movies on her TV through her glass patio door!
Her TV was compatible with his, and so no one was the wiser; at least until iGLASS ratted him out! The kid’s parents paid the cable bill and the lady implemented Parental Controls on her cable box, just in case. Revenue assured; kid busted!
50-Count … Uh, Make That 50-“Shot” Fiber Outage
A customer in Memphis, TN had a 50-count fiber running through a remote part of town.
Shortly after midnight on the new morning of January 1st, several fiber links went down simultaneously. That was unusual in that normally a fiber cut affects all of the fibers and not just some of them. iGLASS NOC personnel notified the customer immediately of course, and technicians were dispatched.
It appears that as part of New Years’ celebrations, some local revelers decided it would be fun to shoot off their guns – lots of them. One of the rounds pierced the armored jacket of the fiber and severed just those fibers in the middle of the bundle! Just look at the photo to see what the techs found at the site of the outage.