I thought I'd start a new thread on this one, since it seems to be important and it's currently buried at the bottom of another thread.

I now take back everything I said about doubting it was a software problem. I am now convinced, as others were before me, that it's a software problem. Jfranke, sorry I accused you of having stuck buttons.

As it stands right now, thanks to a post in that other thread from Corby, I can reproduce the bug intermittently. If I figure out how to reproduct it reliably, I'll officially send it to [email protected] (if needed). Until then, let's discuss it here and see what other folks can do to make it happen.

Now, understand that the steps that I take to reproduce the bug are not things that someone would do in the real world. But we have reason to believe that certain real-world circumstances might also induce the same bug. The goal here is to figure out how to reproduce it "in the laboratory", so to speak, so that the Empeg folks can locate the section in the code that causes the bug. If they can kill it in the "laboratory" situation, then it might fix the bug in the real world.

The bug is a runaway loop where the volume continues to move up or down after the button is released. The interesting thing is, when the volume hits rock-bottom, the runaway loop eats so many CPU cycles that the visualizations start to slow down their frame rate.

Here's how I reproduce it:

1) Disconnect the Empeg from your amplification to prevent speaker damage.

2) Hold opposite commands on the remote and the front panel. For example, the volume down button on the remote and the volume up on the front panel.

3) Get the into a pattern where the two controls are "fighting" for the volume, and you see the volume bar blinking up and down as the two commands jockey for control. You may have to start the process more than once to get this to happen.

4) After that pattern has run several cycles, release both buttons simultaneously. I do not know how long the pattern needs to run before the bug is induced.

5) Most of the time, the volume just stops right there. But intermittently, it "runs away" from you. This is the bug. Funny thing is, when the bug is induced, the new beta-9b "0db stop" only temporarily pauses the volume at 0db. Eventually, it runs away past 0db, too! How about that?

Like I said, no one would ever do that in the real world. But that's just how to induce it deliberately. There are likely other more realistic (although rare) situations that might cause it to happen.

Okay, everyone, get those buttons a-pushin.



-- Tony Fabris -- Empeg #144 --
Caution: Do not look into laser with remaining good eye.

Hey,

Good theory regarding the use of remote/buttons at the same time, but I'm positive that I was only using the front buttons when I reproduced this problem...
-mark

...proud to have one of the first Mark I units

I'm positive that I was only using the front buttons when I reproduced this problem...

Oh, of course. I just happened to be able to reproduce it when I did both IR and front panel buttons. It's possible that the usage of both controls at the same time had nothing to do with it. When I reproduced it, it was very intermittent. In ten minutes of trying, I only got it to happen twice.

I'm just trying to help figure out how to repeat the problem so that we can A) prove it's real, and B) help the Empeg folks track down the problem.

If, as Hugo said, the buttons trigger a piece of firmware that sends virtual IR commands to the mainboard, then there's a possibility that the new rotary control would do the same thing. If so, there's a possibility that the same bug might occur on the Mark 2's. If that's the case, then we should try our best to help find this bug before they start shipping Mark 2's in quantity. It would be a shame for more speakers to get blown because of it.



-- Tony Fabris -- Empeg #144 --
Caution: Do not look into laser with remaining good eye.

I was on the way home from work, and had the volume 'run away' into overdrive. It happened right after I had reached over to hit the right hand F/P button to advance to the next track. As I pushed the button, there was a wrinkle in the road so it felt to me like I had just pushed it for an 'ever so slightly' longer period of time than normal, but there was the sensation of having just pushed the button twice, more rapidly than could normally be done. The volume indicator popped up and was showing a run away toward overdrive (though I was no longer touching anything on the empeg, and the remote was sitting in the passenger seat). I'm certain that Rob, Mike, et. al. have taken into account switch bounce, but I am left wondering how well the Empeg deals with a rapid succession of switch closures. I don't know how the F/P interface is dealt with (interupts?), but a reasonable theory would be that if one switch closure 'came in' and was being dealt with, and before the first closure handler completed, a second switch closure was detected and interupted the first instance of the handler and clobbers a value in the process. (I'm making the broad sweeping assumption that something got clobbered.) Maybe? Since this isn't something that happens on a regular basis; it certainly took me by surprise, and I immediately was trying to figure out what had happened. Hopefully, this might lend some 'clue' to narrow this one down. Oh, I almost forgot to mention this; I've got developer-9c installed in the Empeg. Hmmm ... one more thing, I've noticed that you can, using the remote, get a fairly quick jump in the volume. This one I've chalked up to 'double clicking' the remote's 'volume increase' button, but it can be very subtle; there are times when it has 'jumped' from say, -20dB to -18dB on what (dubiously) felt like a single button press. Carefully, pressing the volume increase button while at -20dB increases the volume to -19.5dB. Anyway, I hope this is useful feedback.

This is very interesting. They said they had worked on the volume controls in 9c, but here you are with a report of runaway volume in 9c.

Has anyone else experienced runaway volume problems since 9c?

Note that the "jump from 20 to 18" is probably intentional (a fairly recent addition). If you hit the button in rapid succession, it assumes you want to turn it up faster. I happen to like this new feature.


-- Tony Fabris -- Empeg #144 --
Caution: Do not look into laser with remaining good eye.

The interface is serial, but it sends state changes and so won't get confused by a too-quick double-click. If a second press comes in while it's sending the last button-up, it'll be ignored until the next loop, where it will transmit the new valid state (button down, or whatever) to ensure that the main board's state is in sync wit the front board.

Hugo

Hmmm, since the 'jump from -20 to -18' is intentional, then it would seem reasonable to conclude that the runaway volume that I experienced can be attributed to 'invalid input'. You mentioned that the empeg interprets a rapid series of button presses to mean 'I want the volume to increase rapidly'. Since it did seem like I hit the button twice (more rapidly than could otherwise be done), maybe it just figured, 'Ok, this is what you asked for'. Granted, the volume setting should not run away, and I can't help but think that the circumstances under which this occured were just extremely unusual. I'll fiddle around a bit and see if I can get the volume to run away again (not that i'll put a great deal of effort into it).

So where does this leave us? If you look at my original bug report on the overdrive, it was also connected to pressing the "next track" button on the main unit.

This is obviously not an easily repeatable bug, but it is one that is potentially quite dangerous.

Corby
SN#320, 6-Gig Blue

We're looking at the code that interprets the signals from the main board in the kernel - it's possible this is missing a press/release if there is IR data coming in at the same time.

Hugo

We're looking at the code that interprets the signals from the main board in the kernel - it's possible this is missing a press/release if there is IR data coming in at the same time.

Cool!

Although, keep in mind that many who reported the bug "in the wild" say that it happened with face-button presses only. I did happen to reproduce it twice by combining face-button presses with IR presses, but my results were inconclusive.


-- Tony Fabris -- Empeg #144 --
Caution: Do not look into laser with remaining good eye.

Although, keep in mind that many who reported the bug "in the wild" say that it happened with face-button presses only. I did happen to reproduce it twice by combining face-button presses with IR presses, but my results were inconclusive.

IR input doesn't have to come from a remote control. The sun is rather good at generating it too.

--
Mike Crowe
I may not be speaking on behalf of empeg above :-)

One further note; the volume 'run away' that I experienced, happened at night. Hence, the sun should not have been a factor.

Well, for what it's worth, the volume ran away from me again. I had just pressed the button to advance to the next track (front panel), and it advanced to the next track, and then started advancing the volume; stepping up at the seemingly appropriate intervals. Although, I must admit that it did choose a rather appropriate time to do it :) a killer tune happened to be the next one in the shuffle. I noticed nothing out of the ordinary (other than the volume increasing without me involved); though, now that I think about it, the last time it happened I had it in 'shuffle' mode as well. Maybe that's a common factor? In any event, great design work!, the Empeg sure knows when to kick up the volume and jam for a while! :)

Yepp i had the same at night time too, although it did not occur to me with 9c, but that is just to the fact that i didnt use it till then, because i was away :)
It is sure to happen to me again too ...

Hey ... Find that one please :)

ANY NEWS ON THAT ?? This on still sucks big time, and I wonder if EMPEG even realized whats going on. This one needs to get fixed now. It kills things.

Cheers, J.

Rob, Mac, etc.

Have you managed to make any progress on this bug? My girlfriend hit this twice on the weekend (within an hour of each other) very nearly causing her to crash on the motorway.

Have you managed to re-produce this in a lab setting at all? Did Tony's (et al)information help?

She is now refusing to use the empeg in the car at all unless someone else is in the car with her.

Perhaps she's just unlucky....

Cheers,

Paul.

Paul Haigh, 6GB, Blue
Reg: 4120 - Serial 00254

Not wanting to be rude here, but if you have that many problems with pressing (eg) power off (press & hold the top button) if there is a problem, don't operate any of the controls whilst the car is moving.

The software, as has been said, is still in beta. If you suffer from this a lot, I suggest using the remote - or ideally ensure that the input gain of your amp is set so that 0db is as loud as you will ever want it (which is how it should be set), meaning that a trip to 0db isn't a major inconvenience.

The trouble is, we can't replicate the problem here. Well, we can if we press all the buttons continuously and repeatedly (eg: about 400-500 press/release cycles maybe in a few seconds to get it to recurr). At this point we have an IR log of tens thousands of entries - hard to find the problem in - assuming that the problem is caused by the kernel missing a key up message from the frontboard. We are still looking for the problem, but the current suspicion rests with something in the system causing bad IRQ latency and so the IR stream can't be recognised by the state machine.

Hugo

Hugo,

Thanks for the prompt response - do you guys never sleep?

I didn't mean my post to be aggressive at all, apologies if it sounded that way, I had just got off the phone with my (irate) girlfriend .

To be honest, she's not a experienced user of the empeg, and when she did pluck up the courage (she's a bit of a technophobe ) It 'all went wrong' (her words, not mine!).

The problem is that, my MP3's are not normalised, so you find that you often change volume 'on the fly', which was what had caused it to go bannanas. The fact that it happened twice (in the same trip) just makes my girlfriend think that she's jinxed....

I really was just interested in whether you had managed to narrow down the cause at all or if other things had overtaken it in priority.

The last thing I want is to sound like I'm having a pop at you guys, I still think that the empeg is the best thing since sliced bread and the level of support is *still* fantastic.

Thanks again for the update.

Perhaps there is something else that we can do to help?

Cheers,

Paul.

Paul Haigh, 6GB, Blue
Reg: 4120 - Serial 00254

There's not really a huge amount that can be done - we have to isolate a way to reproduce the fault and then work out why it's failing. The Mk2, as a safety measure triggered by these discussions, sends the key up code for a released button twice in sucession to ensure that the main CPU won't miss it. We can also ask it for a button state report, as the Mk2 has simple signalling to the display board for setting the display brightness level.

The fault is quite possibly related to the "EEEEEEK!" messages, which seem to point to strange things happening in the onboard-timer department of the StrongARM - if you get an EEEEEEK! then the IR data will most likely be pants. I'll go over the code again and see if there's anything else I can do - it's strange though, as the code is very simple and almost everyone who works here has looked over it by now :)

Sorry if I came across as a bit annoyed, I'd been up painting my gf's living room until 1am this morning (which was great fun!) and had to drive the empeg a-class back to Cambridge with Rob's empeg in it, which has rather too much lionel ritchie on it for my taste. In fact, no lionel ritchie is my taste :)

My empeg isn't currently suitable for in-car use, as it's a bastard lovechild of a MkII main board and a MkI casing...

Hugo

...with Rob's empeg in it, which has rather too much lionel ritchie on it for my taste.

Eeeeew. Now there's a side of Rob we didn't want to know about. I assume you took a shower after getting back to Cambridge, right? I mean, here I am on the other side of the Atlantic, and I feel like I need a shower now. (shudder.)

Tony Fabris
Empeg #144

Not wanting to be rude here, but if you have that many problems with pressing (eg) power off (press & hold the top button) if there is a problem, don't operate any of the controls whilst the car is moving.

Well, when the unit gets in this state, it ignores all input from either the console or the remote. But I have certainly mentally prepared myself to simply rip the unit out if it ever happens again.

The problem, of course, is for people who haven't experienced this. It is an extremely surprising and unsettling event, and potentially quite dangerous. I run my Empeg in a convertible, so the maximum volume I would allow for with the top down on a highway (i.e. the ideal 0db setting) is a setting that is unbearable with the top up.

I'm not saying anything you guys don't already know, but it will be important to find some solution for the Mark I owners as well.

Corby
6-Gig Blue, #320

> when the unit gets in this state, it ignores all input from either the console or the remote.

That's news to me.. I've experienced this bug in the car several times, and pressing another button always cancels the runaway action. The problem is that the "key up" code is getting lost somewhere, so simply pressing the same button again will fix it.

Rob

"and had to drive the empeg a-class back to Cambridge with Rob's empeg in it, which has rather too much lionel ritchie on it for my taste. In fact, no lionel ritchie is my taste :)"

*cough* That's not my empeg, that's the press review empeg. I've never bought a Lionel Richie CD in my life! (I didn't put all that Steps on there either).

Rob

It's all lies.

I dunno, I've seen the Empeg ignore a press-and-hold before. Short presses might work to stop the runaway, but I'll bet that it'll ignore a single press-and-hold on the volume-down button during a runaway. Anyone care to try and test that assumption?

When the volume goes crazy like that, everyone's first instinct is to press and hold that volume-down button as hard as they can. I know it's my first instinct. Since the user interface is already designed around the "hold=volume, press=track" metaphor (and in fact penalizes you sometimes when you forget that metaphor), you should expect the user to behave that way in that situation.

Tony Fabris
Empeg #144

It's all lies.

Sure. We believe you. No, really.

Tony Fabris
Empeg #144

Hugo, John and myself have spent quite a bit of today investigating this and we think we've made some progress. The interim upgrades section of the website has a new kernel upgrade that contains the fruits of our work. We'd be interested in any test reports.

See http://www2.empeg.com/upgrades/test/interim/.

--
Mike Crowe
I may not be speaking on behalf of empeg above :-)

Hugo, John and myself have spent quite a bit of today investigating this and we think we've made some progress. The interim upgrades section of the website has a new kernel upgrade that contains the fruits of our work.

You guys are the best. You've consistently demonstrated a genuine desire to make the Empeg the best it can be. Thanks for working so hard on these issues.

I know how hard it is to fix an intermittent and hard-to-reproduce software bug. This one was very hard to reproduce, so please be patient while us users try out the fix file. It might be a little while before we get back to you with test results.

Now, if we could only cure Rob of that Lionel Ritchie thing...

Tony Fabris
Empeg #144

> Now, if we could only cure Rob of that Lionel Ritchie thing

I knew this would stick :-) It wasn't my album! Blame [email protected] - he bought it!

Rob

We can't really expect the user to behave in any way at all - this is a bug, and it's not expected to be quite so long term that we document it in the user guide :-)

I pointed out the simple corrective procedure of pressing and releasing the button a second time for the convenience of anyone who experiences the problem. The real solution is to fix the bug, and we're hoping very much that the developers did just that earlier today.

Rob

I knew this would stick :-) It wasn't my album! Blame [email protected] - he bought it!

I was just trying to get your goat, Rob. My goal was to get you to yell at Hugo for bringing it up. Did it work?

So don't worry- I was just... pushing your buttons. BADA-BING!
(Thank you, ladies and gentlemen, I'm here all week...)

Tony Fabris
Empeg #144

It did work, yes :)

Sorry Rob, I didn't know it wasn't your music. There was also a lot of Bon Jovi on there, which was frightening...

Hugo
ps: Carter the Unstoppable Sex Machine (USM) did a great piss-take of Lionel Ritchie's song style and the way his videos are just consisted of lots of good looking women - called "Bring on the girls". Fruitbat (the ex-guitarist - the band split a few years back unfortunately) maintains an amazingly good website which will most likely have the lyrics on it :)

Wahoo, another person whoose heard of Carter the USM, i guess they were a little more popular over that side of the globe. Yeah that was a great song, if only my Carter CDs weren't amoung the 120 that got stolen recently. Still i got a $2000 account setup at my local CD store at the insurance companies expense out of it :)

I just experienced this bug for the first time on the weekend. I was scrolling through the playlist menu using the right console button, suddenly it ran away scrolling through them all by itself. A quick press on the left button stopped it.

Now, if we could only cure Rob of that Lionel Ritchie thing...

Unfortunately I think this would require changes to the upgrade file format.

--
Mike Crowe
I may not be speaking on behalf of empeg above :-)

I just experienced this bug for the first time on the weekend. I was scrolling through the playlist menu using the right console button, suddenly it ran away scrolling through them all by itself. A quick press on the left button stopped it.

This is very important information. It's the first time I've seen it reported outside the volume control.

Hugo, Mike... were you aware that it could occur in that situation? The work you did on the interim fix... was it at the button-press level or at the volume control level?

Tony Fabris
Empeg #144

The interim fix is at kernel level, nothing to do with any specific function or menu on the empeg: it addresses the issue of lost keypresses (or releases).

Hugo

OK, I installed the kernelpatch with the result of no change. The volume still 'hangs', actually there seems to be no change in anything, it's happening like once or twice a day.

Cheers, J.

This isn't normal behaviour at all. Nobody else has (yet) experienced the stuck button bug under Beta 10, and even before then it was a very rare bug that might happen once or twice a month.

Are you sure the buttons aren't physically sticking? This can happen if the panel screws are over tightened.

Rob

I'm sorry, I screwed up big time, I am still running 9c! I wasn't aware about 10 beein released (guys, why don't you send out notifications ? Or did you and I got off you mailinglist ?)

I will install 10 tonight! Thanks a lot, J.

We plan to release 10A in a week or so (which is almost identical to 10, but with a couple of tweaks that somehow got missed out) and at that time we'll send out an email to all clients advising them to upgrade.

If anyone is having issues with 10 (not new feature suggestions, just problems with the ones that are in there) then now is the time to inform [email protected] before 10A is finalised.

Rob

I'm sorry to say I experienced my first lost key-up today, on beta-10 (consumer). I was navigating the playlist menu when I pressed the "up" key. They key-up event was missed which resulted in the unit backing out of all of the menus and powering off.

This is the first lost key event I have had, the buttons do not appear to be physically sticking.

(I'll send this to the bugs email too)

__
Unit serial number 47 (was 330 in the queue)...

Cool. Looks like the volume-bug is gone in 10! Good work! And I really enjoy the new features a lot, thanks again, J.