Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 1 of 2 1 2 >
Topic Options
#225478 - 30/06/2004 16:30 /proc/empeg_ir - Missing interrupts.
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
I'm debugging an issue on one of Paul's players where occassionally it throws a wobbly and starts doing erratic things. One failure mode is repeatedly switching between Player and Auxillary every 2 seconds. Another is fast-forwarding through tracks. Both modes are as if the player has received a button down code (IR_RIO_SOURCE_PRESSED or IR_RIGHT_BUTTON_PRESSED) without receiving the corresponding button up.

Paul has indicated that this player has suffered from the problem for a long time. In looking at the problem I've discovered the following in /proc/empeg_ir

Quote:
Valid sequences: 0
Repeated sequences: 0
Unfulfilled repeats: 76
Malformed sequences: 3
Spurious transitions: 412
Missed interrupts: 94808
Timings buffer hwm: 64



This is after about 20-21 hours of uptime (Stupidly forgot to check that before rebooting). Missed interrupts are occuring at an average rate of about 75 per minute, and I think that this is the root cause of the problem. Note that the missed interrupts are not regular like clockwork, the actual rate does vary - one minute may have 78 misses, the next 60, and the next 85.

My player which doesn't suffer from this problem had the following after 10 minutes;

Quote:
Valid sequences: 0
Repeated sequences: 0
Unfulfilled repeats: 0
Malformed sequences: 0
Spurious transitions: 0
Missed interrupts: 0
Timings buffer hwm: 12


Does anyone know what could be causing a missed IR interrupt? I suspect that this is a hardware fault, hopefully a cold solder joint that could be touched up, but I'm not sure where. Any pointers?
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225479 - 30/06/2004 17:05 Re: /proc/empeg_ir - Missing interrupts. [Re: genixia]
pgrzelak
carpal tunnel

Registered: 15/08/2000
Posts: 4859
Loc: New Jersey, USA
Hmmm!!! I never tried debugging that! Very cool!

I know there were some other folks here who had alternating source (aux / player) problems or other random IR signals. This might be related.
_________________________
Paul Grzelak
200GB with 48MB RAM, Illuminated Buttons and Digital Outputs

Top
#225480 - 30/06/2004 20:40 Re: /proc/empeg_ir - Missing interrupts. [Re: pgrzelak]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Ok progress.

This is related to the Button LED hack. With the LEDs turned off, the only missed interrupts are a few that occur during the portion of the boot cycle that they are lit. As soon as the LEDs are turned on interrupts start going missing. I'm running some more tests to try and work out whether the quantity that go missing is related to the illumination level or not. First indication is that it is not.

Now as far as I can see there is one very obvious interaction that could be at play here. The PWM current flow through the LED circuitry could be at a frequency that interferes with the IR reception.
Another possibility is that the oscillating current drawn by the knob button is causing a fluctuation in the IR receiver's supply voltage and that the IR supply isn't dealing well with it.

Now this doesn't happen on my player that also has button LEDs, so I'm still wondering what the difference is.
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225481 - 30/06/2004 22:26 Re: /proc/empeg_ir - Missing interrupts. [Re: genixia]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Code:
Valid sequences:      52

Repeated sequences: 504
Unfulfilled repeats: 5
Malformed sequences: 880
Spurious transitions: 202
Missed interrupts: 14510
Timings buffer hwm: 58

That's off my empeg which I use at home. It's been powered for the last day or so and I've not had any problems with IR at all. [edit]I don't have button LEDs at all as well[/edit]


Edited by tman (30/06/2004 22:28)

Top
#225482 - 01/07/2004 14:34 Re: /proc/empeg_ir - Missing interrupts. [Re: tman]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Shucks. I really didn't want to read that!

Anyway, your missed interrupt rate is significantly lower than Pauls - 14000 vs 90000 in a day, so I'm still not going to rule it out. I've seen only a handful of symptoms, not 90000, do if this was the cause then 14000 might not have been enough to make it occur in your player.

Of course, now I have to wonder why your player without button LEDs has this missing interrupt issue when there seems to be a link on Pauls player, and my player doesn't suffer at all.
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225483 - 01/07/2004 14:55 Re: /proc/empeg_ir - Missing interrupts. [Re: genixia]
pgrzelak
carpal tunnel

Registered: 15/08/2000
Posts: 4859
Loc: New Jersey, USA
Could it be the age or the IR receiver growing faulty with time?
_________________________
Paul Grzelak
200GB with 48MB RAM, Illuminated Buttons and Digital Outputs

Top
#225484 - 01/07/2004 15:02 Re: /proc/empeg_ir - Missing interrupts. [Re: genixia]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Code:
Valid sequences:      52
Repeated sequences: 544
Unfulfilled repeats: 5
Malformed sequences: 963
Spurious transitions: 324
Missed interrupts: 24497
Timings buffer hwm: 62

18 hours or so later and it's now this so I seem to get about 14000 missed interrupts a day. I've no idea what is causing this however as that particular empeg is pretty much stock apart from the new single Toshiba 60G HD inside.

Top
#225485 - 01/07/2004 19:55 Re: /proc/empeg_ir - Missing interrupts. [Re: tman]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Ok, during a loopy period, the rate of missed interrupts is nearly 500/minute.


Edited by genixia (01/07/2004 19:59)
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225486 - 02/07/2004 00:05 Re: /proc/empeg_ir - Missing interrupts. [Re: genixia]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Hmm. It turns out, having just repeated some experiments, that maybe the LEDs really don't have anything to do with it. Even with the LEDs off now I am seeing missed interrupts. I cannot explain why the earlier experiment gave the opposite result. Was that just a fluke?

I've also re-confirmed that my player doesn't ever drop interrupts. The differences between Paul's players and mine are that his have 160GB vs 60GB (non-related, surely?), and that his is running v2 final whereas I'm running v3a7. Both have the same hijack version (v394). Oh, mine has 64MB RAM, his 48MB.

Could it be a software issue? I don't think so. Trevor, what are you running?
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225487 - 02/07/2004 08:53 Re: /proc/empeg_ir - Missing interrupts. [Re: genixia]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
v3a7, Hijack v383, 16MB RAM, single Toshiba 60GB HD and one of Patrick's tuner kits. This is all very strange.

I've had a poke around in the kernel source and everywhere which increments that specific counter has "/* Shouldn't happen */" next to it. I don't know if this means that it's not ever possible in a valid IR sequence or if it's not ever possible in the code. It's detected a couple valid sequences even though I've not used the empeg remote in the last day.... Hmm...

Top
#225488 - 02/07/2004 13:24 Re: /proc/empeg_ir - Missing interrupts. [Re: tman]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Quote:
I've had a poke around in the kernel source and everywhere which increments that specific counter has "/* Shouldn't happen */" next to it.

Yeah, I'd noticed that too.

I think that we can rule out hijack or the player software being the root cause. Maybe kernel tweaking could help deal with the problem better, or possibly even eliminate it totally, but I don't think that this is a software problem per se.
It's also unlikely IMO to be something that can only be prevented by installing 64MB of RAM. (!) And the drive configuration doesn't look like a culprit either. I'm fairly certain that this is something specific to the display hardware.

I've just thought of another test that we really should run, Paul has two other identical players in his possession that we have not checked yet. Paul, you there? Can you fire up the other units for a few hours and then dump the contents of /proc/empeg_ir into this thread?
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225489 - 02/07/2004 13:26 Re: /proc/empeg_ir - Missing interrupts. [Re: genixia]
pgrzelak
carpal tunnel

Registered: 15/08/2000
Posts: 4859
Loc: New Jersey, USA
Soitenly. I have a player (Hermit) at the office, so I will just let it random play for a while.

By the way, initial state before playing...

empeg:/empeg/bin# cat /proc/empeg_ir
Valid sequences: 0
Repeated sequences: 0
Unfulfilled repeats: 0
Malformed sequences: 0
Spurious transitions: 0
Missed interrupts: 0
Timings buffer hwm: 31

Edit: Two hours later, all of the values are exactly the same. But I have not used the IR or buttons, really...


Edited by pgrzelak (02/07/2004 15:27)
_________________________
Paul Grzelak
200GB with 48MB RAM, Illuminated Buttons and Digital Outputs

Top
#225490 - 02/07/2004 17:38 Re: /proc/empeg_ir - Missing interrupts. [Re: pgrzelak]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Quote:
Edit: Two hours later, all of the values are exactly the same. But I have not used the IR or buttons, really...


That's fine - the numbers from Traveller are without using IR or buttons too, just boot and let it run.

I think that clinches the hardware argument. I wonder if Mike, Hugo or Mark have any insights into this.
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225491 - 03/07/2004 22:13 Good News, Bad News. [Re: genixia]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Paul,

The good news is that your player is not currently suffering from this problem.
The bad news is that I want my display board back.

The problem definitely moves with the display board. I suspect that I'm not going to get any answers soon, and that I'm going to end up shipping Traveller back as-is, with the advice that swapping the display with one of your spares is a good idea.
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225492 - 04/07/2004 12:40 Re: Good News, Bad News. [Re: genixia]
pgrzelak
carpal tunnel

Registered: 15/08/2000
Posts: 4859
Loc: New Jersey, USA
No worries. If it is localized to the display board, that really limits the amount of items it could be. I suspect that it is strongly related to the IR receiver itself.

I can always transplant the display board into a lesser used player or spare (Archive), along with a note describing the problem.

Thanks for tracking and identifying this! At least I know I wasn't going crazy!!!
_________________________
Paul Grzelak
200GB with 48MB RAM, Illuminated Buttons and Digital Outputs

Top
#225493 - 04/07/2004 13:43 Even better news. [Re: pgrzelak]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Quote:
Valid sequences: 0
Repeated sequences: 0
Unfulfilled repeats: 0
Malformed sequences: 0
Spurious transitions: 0
Missed interrupts: 4
Timings buffer hwm: 12



After 11 hours. Those 4 happened during boot up so I'm not in the least worried about them. I gave up being cerebral and went to "look for the bad solder joint" technique. This is after retouching any joints that looked remotely like being candidates for being dry or under-soldered, and the IR receiver, Ironically I forgot to do the IR receiver the first time around, put the board back in and was surprised that the problem was still there. Interestingly though, at this stage turning the LEDs off killed the problem - I don't know whether that was another 'fluke' boot or whether there was an intermittent joint on the main display board that was exacerbating the problem. Anyway, retouching the IR receiver cured it.

No swapping required.
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225494 - 04/07/2004 13:58 Re: Even better news. [Re: genixia]
pgrzelak
carpal tunnel

Registered: 15/08/2000
Posts: 4859
Loc: New Jersey, USA
WOW!!! Excellent!!! Great detective work!

This might be a good FAQ topic. I know I was not the only person having this trouble. But it was so intermittent (most of my listening in the car is short bursts with lots of button presses, so the problem did not have much of a chance to manifest itself), I did not try any debugging.
_________________________
Paul Grzelak
200GB with 48MB RAM, Illuminated Buttons and Digital Outputs

Top
#225495 - 04/07/2004 14:16 Re: Even better news. [Re: pgrzelak]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31563
Loc: Seattle, WA
So refresh my memory. What exactly were the symptoms leading up to the desire to go hunting in proc/empeg_ir?
_________________________
Tony Fabris

Top
#225496 - 04/07/2004 14:56 Re: Even better news. [Re: tfabris]
genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411
Occassional random loopiness occurring every few hours, as if the player has received a button code. Possible incarnations include (in order of observed frequency);

Rapid switching between Player and Aux as if someone is stabbing the 'Source' button every 2 seconds.
Continuous Fast Forwarding through tracks as if the right button is being held down.
Player shutting itself down as though the top button were held down.

In all cases, a valid button press, knob twiddle or IR code breaks the cycle and restores proper behaviour. There may be other similar symptoms that I didn't see.

If this problem is suspected then the test is to check /proc/empeg_ir, both immediately after boot and then after the player has run for a couple of hours.This can be done;

from a shell prompt by typing "cat /proc/empeg_ir". or
through hijack's web interface by visiting "http://empegs_ip_address/proc/empeg_ir".

Within that output we are interested in the "Missed Interrupts" figure. It should be considered normal if a few (<20) are seen at bootup. After a couple of hours, numbers in the thousands are cause for concern. A player reporting ~9000 is known to suffer from the loopiness. A player reporting ~3000 has not been witnessed suffering, although it's possible that it does, only less frequently. A good player has a negligible or zero increase from the value reported after boot.
_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top
#225497 - 04/07/2004 16:38 Re: Even better news. [Re: genixia]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31563
Loc: Seattle, WA
Done.
_________________________
Tony Fabris

Top
#225498 - 03/11/2005 14:03 New control problem [Re: genixia]
maczrool
pooh-bah

Registered: 13/01/2002
Posts: 1649
Loc: Louisiana, USA
We've got an MK2 player here which doesn't take inputs from the remote or via buttons or encoder except on very rare occasion when a lot of the buttons are mashed at once and repeatedly, an occasional command will squeek through, usually repeated over and over until the unit is shutdown. We don't see huge numbers of missed interrupts although there are some. All the connections to the PIC from the pushbuttons and encoder test good as does the cable connection to the board. We've tried resoldering the IR receiver but that didn't change anything.

Anyone else seen this and know how to correct it?
Stu
_________________________
If you want it to break, buy Sony!

Top
#225499 - 03/11/2005 14:17 Re: New control problem [Re: maczrool]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
I'm sure you've done this, but have you swapped the display board to see if it's a problem on that board or not?
_________________________
Bitt Faulk

Top
#225500 - 03/11/2005 14:20 Re: New control problem [Re: wfaulk]
maczrool
pooh-bah

Registered: 13/01/2002
Posts: 1649
Loc: Louisiana, USA
Oh yes, sorry forgot to mention that. Yes swapping displays cures the problem.

Thanks though,
Stu
_________________________
If you want it to break, buy Sony!

Top
#225501 - 03/11/2005 14:24 Re: New control problem [Re: maczrool]
pgrzelak
carpal tunnel

Registered: 15/08/2000
Posts: 4859
Loc: New Jersey, USA
I assume you swapped the display board with a known good one at this point, and identified that the problem is on the display board. Check the connections around the buttons - maybe something is broken there, and is flooding the input.
_________________________
Paul Grzelak
200GB with 48MB RAM, Illuminated Buttons and Digital Outputs

Top
#225502 - 03/11/2005 15:28 Re: New control problem [Re: pgrzelak]
maczrool
pooh-bah

Registered: 13/01/2002
Posts: 1649
Loc: Louisiana, USA
Yep, we had checked all that out too.

Thanks,
Stu
_________________________
If you want it to break, buy Sony!

Top
#225503 - 03/11/2005 18:13 Re: New control problem [Re: maczrool]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31563
Loc: Seattle, WA
Bad PIC? Is the button input controlled by a PIC, and is that PIC on the display board?
_________________________
Tony Fabris

Top
#225504 - 03/11/2005 19:35 Re: New control problem [Re: tfabris]
maczrool
pooh-bah

Registered: 13/01/2002
Posts: 1649
Loc: Louisiana, USA
Could be a bad PIC except that remote commands don't work either and I don't believe these interface with the PIC. This PIC (I believe) converts button presses into IR data, but it is sent over a different line to the main board than the IR receiver. Yes this PIC is on the display board.

Thanks,
Stu
_________________________
If you want it to break, buy Sony!

Top
#225505 - 04/11/2005 10:34 Re: New control problem [Re: maczrool]
StigOE
addict

Registered: 27/10/2002
Posts: 568
From Patrick's post here button presses are sent down the same line as remote commands, if I understood it correctly.
Quote:
Button presses are actually encoded as IR signals, and sent down the CIR RX line


Stig

Top
#225506 - 04/11/2005 18:42 Re: New control problem [Re: maczrool]
schofiel
carpal tunnel

Registered: 25/06/1999
Posts: 2993
Loc: Wareham, Dorset, UK
This is probably the button PIC resetting continuously. Probably means either the PIC or the reset circuit has gone bad. Seen it a few times, fixing one now.
_________________________
One of the few remaining Mk1 owners... #00015

Top
#225507 - 10/11/2005 16:53 Re: New control problem [Re: schofiel]
maczrool
pooh-bah

Registered: 13/01/2002
Posts: 1649
Loc: Louisiana, USA
Yeah I see now the IR receiver and buttons/encoder go to the same PIC. We had checked the reset pin previously, and it is just a resistor to 5 volts I believe.

Update:
We found the problem. It was a bad solder joint at pin 2 of the PIC near the buttons. Although it showed continuity to the IRRX line, the signal was not reaching the PIN reliably. After a quick touch up of the solder joint, control seems 100% again. Apparently the slight pressure of the probe was enough to establish continuity were none was without the probe. Tony, it would be great if you could add this to the FAQ.

Thanks,
Stu


Edited by maczrool (10/11/2005 17:14)
_________________________
If you want it to break, buy Sony!

Top
Page 1 of 2 1 2 >