UpgradeSplitter

I was just using Sven's UpgradeSplitter to check out some stuff in the upgrade files, and it appears that beta13 (perhaps amongst others) breaks it. There's a new chunk, 0x04, that appears that UpgradeSplitter doesn't know about.

First off, it doesn't handle it very gracefully. It just immediately bombs out. Removing the ``return 1'' line in the default: case solved that.

Second, I was wondering if anyone could help me verify that it is what I think it is. It appears to be an indicator of upon which models the upgrade can be applied. The ``car2'' upgrade's chunk contains ``7\n9\n'', which seems to match the /proc/empeg_id codes for the Mk2 and Mk2a, which seems just right. But the ``car1'' upgrade's chunk contains ``1\n'', which makes intuitive sense, but doesn't match the /proc/empeg_id ID for the Mk1 of 4.

Does anyone have any more insight into this?

I think your guess is probably correct, but only the empeg people
can answer definitvely.

There's also an interesting new type 0x05, CHUNK_UNTARDRIVE0.
I keep meaning to try it out but haven't got around to doing so yet.

I hadn't seen Sven's upgrade splitter when I wrote mine, which
knows how to handle both these types, and, indeed, any other type.
See http://empeg.org.uk/up.html

It appears to be an indicator of upon which models the upgrade can be applied.

Yup.

But the ``car1'' upgrade's chunk contains ``1\n'', which makes intuitive sense, but doesn't match the /proc/empeg_id ID for the Mk1 of 4.

In order to cover various prototypes and halfway-house development units, some of which didn't have empeg_id's at all, "1" means "none, or anything less than 7".

As you can see for yourselves in lib/protocol/upgrader.cpp, CHUNK_UNTARDRIVE0 was never implemented.

Peter

I didn't even think to check the emptool source. Duh.

Thanks for the info.

Hi.

In theory, the UpgradeSplitter on my homepage should be able to handle type 0x04 chunks, at least I did update my sources and I seem to remember putting it up on my page. Also, I think it also handles type 0x05 chunks.

The type 0x04 chunk is the HWREV(ision) chunk, enumerating the hardware revisions the named update file may be applied to (one HWrev per line). Why the MkI updates only state "1", and not 1 and 4 is beyond my knowledge however.

cu,
sven

Hi.

The data part of a type 0x05 chunk really is a tar archive which gets extracted to /drive0. The only thing about it that I didn't check yet is wether /drive0 is formated or erased before this happens, but I _think_ it isn't.

EDIT: To clarify this, as it is easily mistaken: I know that it is _intended_ behaviour to untar that chunks data part to /drive0, but (like Peter said) was never implemented. I did think however, that it was implemented in the 2.0 betas somehow.

Also, I want to note that it was a conscious decision to only allow known chunk types in upgrade files. This way I wanted to prevent the user from unintentionally creating an upgrade file in an unsupported (by empeg) format.

I also tried to sum as much information in the "index" file as possible. Therefore I don't read single lined texts from external files but from the index itself. I loosened that approach for the 0x04 and 0x05 chunk types though, because I didn't have the time to do it right.
That being said, I would choose a different approach today, allowing the user to override the default behaviour and thus allowing for unknown chunk types.

cu,
sven

The sources on your web page didn't seem to reflect that change.

Hi.

Oh well, I will have the following weekend almost completely free and to myself. I planned on cleaning up that source and creating a preliminary version of a custom upgrade file (with boot from /dev/hda2, with quite a bit of extra software preinstalled). These tasks just got an increased priority on my todo list.

cu,
sven

Rob has explicitly stated that SonicBlue are the only ones allowed to distribute empeg software. Which means that no one else can distribute a .upgrade file, as it contains their copyrighted software.

The best thing we can do is write a program that will take a SonicBlue-distributed .upgrade file, pop it open, insert new software, then stitch it back up.

However, if you want to make a new .upgrade file for your own benefit, go ahead.

Bitt,

you are only partly right. The .upgrade file I am talking about won't touch the originals software partition and won't include it either. It will just be applied on top of a "classic" upgrade file. Also my hda2 image won't include the customized init executable that is included with the empeg's original .upgrade files.

In addition to this, I plan to provide a shell script / batch file along with it that takes

1. a patched hijack kernel,
   
2. an original .upgrade file and
   
3. my upgrade file,

unpacks the upgrade files and merges their pieces with the hijack kernel and creates a new upgrade file. As I won't be distributing any software that is copyrighted by empeg/sonicblue, they obviously can't object to it.

cu,
sven

Wow. This is EXACTLY what I think we're all hoping for... If you get that working as you've explained it, it could be really beneficial for a lot of the non-Linux types around here. It'll save me from having to answer as many emails asking "how do I install emptriv?" too.

Hi.

I know, I know, and worst thing is I promissed to create that beast months ago, if not years. kimbotha once said he would take that project over, but he never released anything either.
I plan to integrate the original or modified versions of:

• telnetd
• preinit
• rsync or rsyncd
• find
• HiJack (obviously I need a modified kernel anyway to boot from /dev/hda2)
  

Any other ideas? I don't want to put too much on the base system, as I plan to integrate some sort of package management which installs new software on the music partition (after the package itself is uploaded by the user using ftp).

cu,
sven

Well... A way way back I installed a whole slew of standard UNIX utils from a debian tarball that had been posted on this BBS... I think those basics (things like tee, join, cut, gawk, etc) should be there in addition to what you've mentioned. Basically all the important UNIX stuff that the standard developer image leaves out.

I agree, everything else should be set up as an add-on package. Your strategy sounds good.

Here's my concern though... Your strategy seems to be to use an alternate root partition on /dev/hda2 and chrooting to it. If this is done, the standard root partition (/dev/hda5) is wasted, right?

I always thought the best solution would be to install a standard linux setup on the regular root partition and use /dev/hda2 for the user apps (packages) that people want to install. This would keep the apps OFF the music partition, where they tend to cause problems with synchronization. I don't think there's more than 16mb of things we really need on the root partition, is there? Everything else would seem to be package-able, and reside on /dev/hda2. At least in my mind..

The way I would imagine it working, when I go to upgrade, I would take the empeg-supplied .upgrade file, along with a 3rd-party upgrade file with the user stuff, and feed them into some kind program which combines them and sends them over to the Empeg. The root partition from the empeg-supplied upgrade file would be ignored, and instead we'd use the stuff that's in the 3rd-party upgrade file. We'd just use the stuff under the /empeg directory from the official upgrade file. User packages (which could be selected during this mythical upgrade-building process) would all go to /dev/hda2.

This would make sure we're not wasting /dev/hda5, and as I said, avoid problems with apps running on the music drive and causing fsck's and syncs to fail.

Would this work?

Have you managed to get stuff to install on hda2? Is there some trick
to it? When I try to upload an upgrade I've made containing an ext2
image as CHUNK_PUMPHDA2, it goes funny. The player just re-starts part
of the way through the upgrade sequence and then gets understandably
confused as it sees the hda2 image data as serial commands.

If I use CHUNK_PUMPHDA5 instead, it works.

Hi.

Well... A way way back I installed a whole slew of standard UNIX utils from a debian tarball that had been posted on this BBS... I think those basics (things like tee, join, cut, gawk, etc) should be there in addition to what you've mentioned. Basically all the important UNIX stuff that the standard developer image leaves out.


Agreed.

Here's my concern though... Your strategy seems to be to use an alternate root partition on /dev/hda2 and chrooting to it. If this is done, the standard root partition (/dev/hda5) is wasted, right?

No. My approach is different from that, for copyright and technical reasons.

1. For copyright reasons, I can't integrate the original software into my partition images. Therefore, my partition image needs to be modified before upload, copying the original binaries in, or it needs to access the original software on the empeg from a different (the original) software partition.
2. I can't create a software that integrates the original binaries into my image for a simple reason: Those who will benefit the most from my upgrade file are those that are _not_ Unix guys but want additional software on their empegs. And those people most certainly use Windows, which is not capable of writing to an ext2 image.
   

I always thought the best solution would be to install a standard linux setup on the regular root partition and use /dev/hda2 for the user apps (packages) that people want to install. This would keep the apps OFF the music partition, where they tend to cause problems with synchronization. I don't think there's more than 16mb of things we really need on the root partition, is there? Everything else would seem to be package-able, and reside on /dev/hda2. At least in my mind..

Well, like said above, I need to use two partitions, one for my root image and one for the original software. and I don't see how installing additional software on the music partition could affect synchronization (apart from using up space). As long as it isn't installed in the "fids" subdirectory of the music partition.

Installing the packages on the music partition has the additional benefit of not being affected by any update to the empeg software or my image.

This would make sure we're not wasting /dev/hda5, and as I said, avoid problems with apps running on the music drive and causing fsck's and syncs to fail.

I currently have telnetd running from the music partition, and no sync or fsck failed yet. Are there any verified incidents where that happened?

Ciao,
Sven

Hi.

I didn't have much time to check if/how good it works. But there would be a way around that problem if needed. I would really like to hear about this from the empeg people.

Regards,
Sven

I can't create a software that integrates the original binaries into my image for a simple reason: Those who will benefit the most from my upgrade file are those that are _not_ Unix guys but want additional software on their empegs. And those people most certainly use Windows, which is not capable of writing to an ext2 image.

Hm there has been some talk of getting e2tools compiled for win32 to do just this... I think this would be beneficial because the stuff from /empeg/ on the empeg-distributed upgrade could be merged with your stuff from your upgrade file (/bin, /usr, etc) and put onto /dev/hda5 (the default root partition.) This leaves /dev/hda2 free for user apps. And it'd be legit. If e2tools was working on win32, wouldn't it be theoretically possible to combine the files onto one partition?

I currently have telnetd running from the music partition, and no sync or fsck failed yet. Are there any verified incidents where that happened?

Yeah, I think so...

http://empeg.comms.net/php/showthreaded.php?Cat=&Board=empeg_tech&Number=96153

http://empeg.comms.net/php/showflat.php?Cat=&Board=empeg_tech&Number=101269

I currently have telnetd running from the music partition, and no sync or fsck failed yet. Are there any verified incidents where that happened?

Yeah, I think so...

The first thread you mentioned isn't really related, but the second one got me on track:
Running an application from the music partition is not the problem, but the application in question must not have a file from the music partition open (which my telnetd hasn't) and its current working directory must not be on the music partition (my telnetd is running with cwd=/sbin IIRC). emptriv on the other hand probably keeps its trivia files open (which I think are stored with the binary) and
it is probably running with cwd=<path of emptriv>.
So what I need to implement is a way to install the software on one of the possible program partitions (hda2 or hda5) automatically if needed.

The only problem to decide on is wether I want to install all packages on one of the program partitions or to install only those that might interfere with sync/fsck. If I did the later, the package author would be required to indicate if the package might interfere with them.
It is probably easiest to just install all packages on the software partition(s).

Regarding the e2fs tools on win32: If they really where available and functioning reliably, that would certainly be interesting.

cu,
sven

The emptriv questions are stored in a "questions" subdirectory. It does run with cwd=<path to emptriv> but that's probably just my sloppy coding.

It is probably easiest to just install all packages on the software partition(s).

Right, which is why I was hoping that hda5 could be used for the "standard" stuff (the /empeg stuff plus /bin, /usr/bin, etc) and hda2 could hold all the user apps. But I guess you're right in that it doesn't make much sense to have it only available for Linux, which, without e2fstools on Win32, would be the case. I still think it's the ideal solution though.

But I guess you're right in that it doesn't make much sense to have it only available for Linux, which, without e2fstools on Win32, would be the case. I still think it's the ideal solution though.

yes, it would be ideal. It would also be a way to circumvent the (unverified) problem that PUMP_HDA2 does not work as expected.

cu,
sven

Ah, false alarm. I had forgotton to gzip the new filing system image.
Once I did that, it all worked perfectly!

I used only my upc utility and upgclient with filesystem images I built myself---I needed
nothing from empeg's upgrade.

I created a mini-debian installation by picking apart the packages from
http://www.uk.debian.org/debian/dists/potato/main/binary-arm/
but it's really hard to get a useful subset in 16Mb. Even 32Mb isn't
enough for the basic C development tools. So I ended up using both hda5
for / and hda2 for /usr.

I can do hda5 and hda2 together or separately. You don't need to do a
separate CHUNK_PUMPHDA for each partition---indeed, you don't seem to
need this at all.

That was a false alarm---I hadn't compressed my disk image.

Feel free to integrate any/all of the shell script I wrote for setting up /dev/hda2 as a /programs0 mount

Apologies for bumping this thread back up with a silly question -

To create my own upgrade file containing the apps I currently have installed (basically to make a reinstall much easier if I mess anything up) am I correct in thinking that as long as my apps are all on hda5 I just need to replace the pumphda5 compressed image in the official empeg released upgrade file with a gzipped copy of my hda5?

And if that is the case, does that also take into account the hijack kernel? Or is it just in flash?

The .upgrade files usually have a replacement kernel within them. If you are making your own .upgrade file, then you could stick a Hijack kernel into it in place of the vanilla one.

When using my upgrader utility to upgrade a player over ethernet, the tool ignores the replacement kernel in the .upgrade file.

Cheers

the tool ignores the replacement kernel in the .upgrade file.

can you change the upgrader utility so that you can override this with a switch, for those times when you want vanilla kernel for troubleshooting? will it cause problems to overwrite hijack while using one of hijack's functions? just stinks how i have to revert back to the slow serial way when i do need a stock kernel.

upgrader dumps the kernel from the .upgrade file into the current working directory, so you can install it separately if you really want to blow away Hijack.

-ml

No need for a slow serial connection to install a non-Hijack kernel: just FTP it to /proc/empeg_kernel on the player, and then reboot.

Of course, after discarding Hijack, you're stuck with slow serial for the next kernel installed..

-ml

For just flashing kernels I tried Upgrader a few times but had cygwin1.dll issues every time so I gave up. If I can get that sorted I will definitely use it, as ftp-ing to /dev/flash_kernel only works intermittently.

What I really want to do with upgradesplitter/builder is use it as a backup of my system and make a build which includes the hijacked kernel, emphatic, tts, telnetd and anything else which I have installed after the last upgrade.

So if I'm reading correctly, all I need to be able to do is create the disk image which goes into pumphda5?

How do I do this? Anyone able to help me? I know it should be simple, but I can't figure it out, and I don't see any info anywhere.

(oh, and in trying to get the latest hijack kernel, it appears the sourceforge hijack site is down just now)

Huh? What cygwin1.dll issues?

Also why does FTP the kernel to /dev/flash_kernel (/proc/kernel_flash) only work sometimes? I do it all the time and it works perfectly for me.

To create the disk image you need a Linux machine or something that understands Ext2.

The error is a line not found in cygwin1.dll, so it is entirely possible that I have messed up cygwin along the way somewhere.

The flash-kernel thing is strange. To get the ftp client to send it, I first need to telnet to the empeg and rw. Then send the file. Then ro. Then sometimes when I reboot I have the new kernel, sometimes not. It used to work 90% of the time, now about 20%.

Most of my boxes are unix or linux, and I can easily hook one up to talk to the empeg, but I still don't understand the format of the .disk file which is gziped into upg_pumphda5. What do I use to create that file?

Copy cygwin1.dll into the same directory as the upgrader. That should eliminate any problems with it not being able to find the DLL.

As about the flash_kernel thing. Try using /proc/kernel_flash instead.

To create the image you need to mount a file using the loopback device with ext2 format.

Cheers! I'll give all 3 of those a try this evening...

The flash-kernel thing is strange. To get the ftp client to send it, I first need to telnet to the empeg and rw. Then send the file. Then ro. Then sometimes when I reboot I have the new kernel, sometimes not. It used to work 90% of the time, now about 20%.

Time for you to find a real FTP client, not to bash/trash Hijack's FTP services!

There is ZERO (absolutely NO) need to set the drive(s) RW just to upload a new kernel to the Empeg.

-ml

That's what I thought, originally. I use leechftp, which is generally pretty good.

Wasn't in any way getting at any of hijack's functionality - I am a confirmed next-hijack-version addict - I normally assume I have just configured something incorrectly or corrupted something

I know most of the problems on my network are down to Microsoft - some fixable, some just MS. All my *nix boxen just work. They way they always have.

>All my *nix boxen just work. They way they always have.

Mine would too, if I'd just leave them alone

Actually, my internet server is *still* running a (mostly) 1998 distro. I keep making motions to update it to the 2000's, but just never seem to want to disturb it since it works almost perfectly all the time.

Cheers

Tried cygwin1.dll in the same directory, and I get the same error when I try to run upgrader (same error with v6, v4 and an earlier one):

The procedure entry point__getreent could not be located in the dynamic link library cygwin1.dll

Unfortunately I have no idea how to troubleshoot an error like that on a Windoze box.

Any ideas?

get the latest cygwin? also, you're gonna need ncftput. even after that, you wouldn't be able to run it from command prompt, because a couple of the ncftput's commands require /dev/null. easiest way is to run it from cygwin's bash shell.

Hmmm - I reckon I might just use the Linux upgrader from linux then, as I can't find anywhere where I can get the full cygwin without connecting to the internet during setup (ie I just want to download the entire thing onto one ofthe machines outside my network and burn it onto CD to install inside my perimeter)

Anyhow, it might all be irrelevant as I can't figure out how to make a copy of my empegs current hda5 to compress into the upgrade file anyway. I have gone through all the disk image and loopback howtos I can find but all seem to require a loopback device to exist on the empeg.

Has anyone managed this, and how did you do it?

Hi Mark,

After trying various things, I eventually gave up and recreated the flash_kernel device from scratch, now it all works perfectly.

So I guess I corrupted it somewhere along the line.

Cheers

How the heck do you corrupt the device file?!? Anyway... You could just use /proc/kernel_flash instead. It's part of the proc filesystem so it's automagically created.

That is a good question, and one I have no answer to

I tried to use /proc/kernel_flash and it doesn't work for me. I can write the new kernel to it, if I first delete the old one and occasionally it will upgrade. Sometimes it won't. It is too unreliable for me to use. The timestamp on the file shows it has the right one, but I have no idea why it doesn't use it.

Anyway, now I have rebuilt the /dev/flash_kernel it works every time, which is nice.

Delete the old one?

I've even used the useless commandline FTP client that comes with Windows and it works... I just do bin, hash, put zImage /proc/kernel_flash, quit

i thought it was /proc/empeg_kernel. maybe there are more than one aliases.

Umm. Yeah I was do it from memroy and getting confused with the one he'd created in /dev/

Yup - delete the old one. Very strange - that's why I've been puzzled, as I have had to telnet in to do it, and I'm sure that's just not right. And I couldn't remember whether it was empeg_kernel, kernel_flash or flash_kernel either