Dirty power

Posted by: Laura

Dirty power - 01/02/2006 23:40

Has anyone experienced having a location that has dirty power problems and what solution have you come up with? I have a store that in the last 4 months has gone through 50 hard drives. It was on APC ups's, since this started they have had a new grounded dedicated circuit installed and are on a Trip-Litte line conditioning ups but are still going down . The computer system is an old P2 running SCO UNIX with SCSI drives.
Posted by: tfabris

Re: Dirty power - 01/02/2006 23:57

At that point, I'd suspect something other than the power coming from the wall.

For example, you replaced hard drives, you replaced UPSes, but what about the computer itself and its power supply/motherboard?
Posted by: Laura

Re: Dirty power - 02/02/2006 01:02

The computer has been swapped out at least 15 times. The power company is putting up something to monitor the incoming power tomorrow and also a APC line-conditioner is replacing the Trip-Litte unit. Today the overhead lights in the store went off and then back on down the rows. The computer crashed 10 minutes later, this has happened before when I have been on site.

Did I jinx the board? It was down for me for a while?
Posted by: gbeer

Re: Dirty power - 02/02/2006 01:56

Quote:

Did I jinx the board? It was down for me for a while?


Nope, I saw it down too. Also some post icons on the main page seem to be fubar.
Posted by: Ezekiel

Re: Dirty power - 02/02/2006 13:04

While this solution may be a bit extreme (a couple of thousand dollars IIRC - call them), you might have some luck with a switched power supply such as this one from Interpower.

It takes the incoming AC voltage, uses a switching device to create high-voltage DC and then re-creates an AC waveform via PWM. We use them for creating 230V/50Hz power off of our 115V/60Hz lines for AC motor testing.

In your case you'd set the input and output to 115V/60Hz. We used this device to help with a spate of dirty power (which made our motor speeds vary) with good results.

If it didn't solve the problem you could probably return it.

-Zeke
Posted by: lastdan

Re: Dirty power - 02/02/2006 15:32

I had a problem (long ago) and our ups would freak out from time to time. I stuck a volt meter in the wall socket and set it to record. the results were troubling.
I called the power company and they came out and put a recording meter on our circuit. a few days later they came back and showed me a paper graph of just how 'dirty' the power was.
I think they replaced a transformer down the block or something, it fixed the problem in any case.

so yes, I believe it does happen.
Posted by: Laura

Re: Dirty power - 21/02/2006 23:22

To update things, DP&L said the power coming into the store is clean and the contracted electricians said the power in the store is clean also. They did see a flutter on one sign wave that they said wasn't enough to damage any equipment but a drive failed at that time. The new APC we had the battery went dead when I performed a self test on it. It was replaced today. I also replaced the laser printer in the office today as a precaution after being told that it could possibly have a resistance problem with the print head.

All I know is that the store remains a nightmare. I have 27 stores to take care of and spend 70% to 80% of my time at the one store. Yesterday we were talking about having a priest come to exorcise the place. I'm trying to get someone in to check the power management system since some of the times drives have been lost some of the lights go off and then back on in sequence. I wish that I had more authority to tell someone what to do.
Posted by: lectric

Re: Dirty power - 22/02/2006 18:59

Ummm, what kind of wiring is it? How much stuff do you have plugged into that circuit? Are there any motors or heaters on the same leg of the circuit? Is the equipment overly hot? Do the older monitors tend to have a brown spot on the top of them?
Posted by: Laura

Re: Dirty power - 25/02/2006 13:04

They are now checking into hiring an electrical consultant to come try and find the problem. The electricians that they have had in so far have found nothing. The store has crashed almost every day this week and twice yesterday. The problem is getting worse and worse. I have now been relieved of the other projects going on and have other techs taking care of my other stores so I can be free to pretty much live at the problem store.

I have asked to borrow the company jet to fly me and the store manager to an undisclosed location for a week or two of paid vacation when this is all over The upper management people got a kick out of that.
Posted by: mlord

Re: Dirty power - 25/02/2006 16:38

It really sounds like something "dirty" is plugged into the conditioned side of the power -- something like an electric motor or other noisy high-current device (eg. air-conditioner). Each time it comes on, the whole circuit takes a glitch, and something new gets fried.

Hopefully the consultant will figure things out. I would start by tracing all connections to see what exactly is sharing the conditioned power. I'd also probably split the power into two separately conditioned segments, and then see which half has troubles. Then split that half, and so on..

Cheers
Posted by: Laura

Re: Dirty power - 25/02/2006 22:37

Supposedly all of the computer equipment is on it's on dedicated grounded circuit which was replaced last week with no help. I did notice that the Hughes satellite receiver would blink 0 3 every once in a while which means that it thinks that it has lost signal and wants to dial out, but that lasts just a matter of 30 seconds. It will be replaced this coming week since it shouldn't be doing that, we did disconnect the modem from it. It is connected to the switch and could be sending something through the network cable. I tried a Trip-lite network cable line suppressor on the Unix server but then the help desk lost connection and couldn't ping it though I could still connect on-line to them so we took that off.

It's beyond me or anyone else trying to work on the problem so far to figure it out. The electricians were supposed to do load tests on the motors on the different equipment but never did and we don't know why. The work orders go through four different channels and who knows what gets said to whom. The home office kept saying it was a hardware failure on the Unix server and or drives until the IBM Windows 2000 server started reporting power supply loses once in a while and lately showed a 9 second loss on the network card the same time the Unix server crashed.

It's all just an unending nightmare to me. I now have two of our equipment depots supplying drives to me because one can't keep up with my demand. And I have a special team on our help desk working on it and they are putting in a tremendous amount of time also. I want to wake up soon and have it all go away.
Posted by: Phoenix42

Re: Dirty power - 26/02/2006 12:41

Quote:
Supposedly all of the computer equipment is on it's on dedicated grounded circuit which was replaced last week with no help.


Supposedly our server room was on a dedicated circuit as well. Until one night some one hit a light in the parking lot and took out the elavotors and our servers. Just because some one says that it is does not make it so. Some electrician that followed later may have taken a shortcut.

Good luck, Laura.
Posted by: Laura

Re: Dirty power - 02/03/2006 02:12

Store stayed up for 4 whole days before it came crashing down again today. I had a conference call from 4 people in our store planning department, which is very out of protocol since they should have talked to my supervisor, and one of the questions they asked me was I a certified low voltage electrician. I felt like asking them what planet were they from to think that I would be especially working as a field tech for the wages I get paid. I told them no and they said they felt they should have one come check things out and I agreed, 5 months ago they should have. They were surprised to here that the problems started in October, they thought it had been going on since August and I felt like telling them thanks for waiting so long to get their heads out of their asses to do something. My department is trying to bill them for all of the equipment gone through and all of my hours at that store. What a fricking mess that doesn't seem like it's going to end before we burn the store down so they have to rebuild it and redo all of the wiring.

PS: I told my bosses today that when we get back from the spa we are going to need months of therapy for post-traumatic stress syndrome.
Posted by: Laura

Re: Dirty power - 03/03/2006 00:32

The low voltage specialist is coming tomorrow morning and our main equipment depot in PA is now supplying the computers and drives, pre-tested, instead of coming from Michigan. I'm praying to the unix gods tonight that they find the problem.
Posted by: Laura

Re: Dirty power - 04/03/2006 14:49

The network specialist came yesterday with his high tech tester and found nothing wrong after a few hours and said he felt it was electrical. Of course while moving the computers on the floor around, the power cord was accidentally pulled from the unix tower and that was all she wrote for those drives. Upon coming up it threw SCSI errors and panic K-trap errors and after 5 more hours the computer and main and backup drives were replaced. Also he kicked off some registers and that took hours to fix. It was not a fun day and I'm sure they'll be sending others to check more things next week which means more crap for me to fix.
Posted by: Laura

Re: Dirty power - 18/03/2006 10:34

The problems still continue though the frequency has slowed down somewhat. The computer was up a whole week and a half before it crashed and the automated script filling machine went down too. The regional vice president visited the store Thursday and I got to tell him that I didn't think enough was being done and it was dragging on forever. When the district manager explained to him how much sales the store has lost and how many pharmacy customers have transferred to other stores, he listened. A generator has now been ordered for the store to run the office and pharmacy equipment for 30 days, if the store remains stable then the local power company will be held accountable, and if it continues to crash, the whole store will be rewired. I also have a technician from the company's main equipment depot in PA coming Tuesday to spend 4 days in the store to give me a break and see if he can find something that has been missed.
Posted by: wfaulk

Re: Dirty power - 18/03/2006 15:49

"Well, here's your problem. Your store's set to 'evil'."
Posted by: n2toh

Re: Dirty power - 18/03/2006 17:44

Quote:
While this solution may be a bit extreme (a couple of thousand dollars IIRC - call them), you might have some luck with a switched power supply such as this one from Interpower.

It takes the incoming AC voltage, uses a switching device to create high-voltage DC and then re-creates an AC waveform via PWM. We use them for creating 230V/50Hz power off of our 115V/60Hz lines for AC motor testing.

In your case you'd set the input and output to 115V/60Hz. We used this device to help with a spate of dirty power (which made our motor speeds vary) with good results.

If it didn't solve the problem you could probably return it.

-Zeke


The unit you described is usually called an On Line UPS, and are the best option for locations with bad power.

Most APC units I've dealt with are stand by units, they let the power from the utility pass thou them until a problem occurs then they react to the problem. they are good for blackouts but not much else. I'd stay away from the APC Matrix 5000 series I've seen more than one with less then 100% load catch fire for no reason.

When you had the power quality checked did they use a True RMS meter, and a spectrum analyser to the measure your harmonic current load?
Posted by: Laura

Re: Dirty power - 19/03/2006 11:38

Ah, I knew the answer was going to be something simple
Posted by: Laura

Re: Dirty power - 10/05/2006 08:38

I thought I would update this and say that the problem was fixed by the PA tech. The problem went away when he forced the Store Planning department to take the heater off of the power management system that they had said they had did two months before when they actually hadn't. Why the Unix tower and drives were the main thing affected, I'm not sure. I guess it was more susceptible to electrical interference than anything else. Yesterday I was thanked by the top dog in the Field Technical group for my hard work and dedication at that store. Life has finally gone back to more normal working hours and the constant stress and worry went away.
Posted by: lectric

Re: Dirty power - 10/05/2006 20:25

Wow.... Yeah, having massive heating elements on a circuit with electronics is a really bad idea. The servers were likely experiencing undervoltages constantly.

Glad it's fixed, and glad for the kudos!