backup (again)

As I was hearing a lot of requests from people for a backup-utility, and they didn't have the ability or knowledge to use any of the available programs(great work, ynot & synergy), I have spent some time creating a windows backup-utility for the empeg. It still needs displayserver running on the empeg.
It works like this:
You create a CSV file with emplode. The CSV file contains all the installed MP3's and playlists.
Start empegbackup.
Give the location of the CSV-file.
Give the directory where the data should be stored.
Give the URL of displayserver running on your empeg.
Click start.

Here's a screenshot:


You can get it here

Installation Instructions :
Just unzip and run.

Frank van Gestel

Frank, let me be the first to say it:

YOU DA MAN!

The corresponding entry in the FAQ has been updated.

___________
Tony Fabris

Speechless....

Does this save as one big file or as individual mp3's? If it is one big file, how do you get the info back on the empeg?

32GB (JUST UPGRADED!) Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

Multiple files are saved. every MP3 or playlist on the empeg is stored as one datafile and one "tag"-file. The datafile is transferred from the empeg with the filename as stored on the empeg. tag-files are recreated using the data in the CSV-file. I haven't tried an actual restore yet, but I didn't claim it to be a restore program, did I

Frank van Gestel

Wow! I really appreciate your hard work on this.

Thanks!



80000078

You never cease to amaze. Nice work! Now it just needs a way to work without displayserver. Seems like everytime I install it it ends up messing up my playlists. Maybe you could help me figure this one out?

________
Donato
MkII/Blue/40GB/080000565

a couple things.

source to the empeg backup?

I was considering spending the time to get amanda network backup to work with the empeg.. and maybe write up some scripts to backup the playlist partition with dd.

hrm....


12gig red mk2 -- 080000125

source to the empeg backup?
Source is nothing special; about 100 lines of code. I used Delphi 4, which has standard internet components. I downloaded a CSV-component from http://www.jansfreeware.com
The program justs iterates the CSV-file and downloads every file found using the HTTP-component. you can download a specific file by looking up it's FID in the CSV file and send a HTTP-request to displayserver : http://empeg/ttt.mp3?file=FID. The requested file file will be streamed back.
When 2.0 is finished, it will probably have a way to restore too, so the app is going to change. This was just a fast way to let people backup their empeg. I liked the stats about the network.
I like stats...

write up some scripts to backup the playlist partition
Have you tried ynot's perl-script? Maybe you could combine it with the CSV to do selective backups.
I like perl...

BTW: Does anyone by chance have a small standalone perl-executable for empeg? I've tried crosscompiling without result.

Frank van Gestel

Backing up without displayserver would be nice.. if only there were some way to do it, some hook in emptool for example, if only..

I found precompiled Perl in the archive you pointed to in this your post . It is compiled, but not statically linked. I unpacked archive debian-image_990822_tar.gz somewhere in /drive0, moved perl into path, and tried running it. As it complained about missing .so libs, I symlinked one by one into /lib (or was it /usr/lib?) - I could not remember how to list dynamic link libraries a program requires :-).

Dragi "Bonzi" Raos
Zagreb, Croatia
Q#5196, MkII#80000376, 18GB green

I could not remember how to list dynamic link libraries a program requires

ldd

Frank van Gestel

Why can't I refere to another post any more? Copy and past an URL should not be that difficult, even for me

Anyway, tha pachage where I found Perl is http://fvgestel.dyndns.org/empeg/debian/debian-image_990822_tar.gz . Wait! It is on your site! Why are you trying to crosscompile it when you already have it? Or did I missunderstand something?

Dragi "Bonzi" Raos
Zagreb, Croatia
Q#5196, MkII#80000376, 18GB green

I have experimented with the perl that was in that debian file, but it was too large; I was looking for a stripped out version for basic scripting. I can also remember doing a try compiling on the empeg, but I think it needed floating point. Maybe a perl4 binary will do.

Frank van Gestel

Why can't I refere to another post any more? Copy and past an URL should not be that difficult, even for me

Normally, it does work. Somehow, the link you posted lost the question mark following the PHP script name...

...showthreaded.php?Cat=...

Put the question mark back in, and it works.

Are you using a non-English browser? Maybe the question mark is a different character in your localization.

___________
Tony Fabris

is it just me or is Rob eluding to the point that there is already something in the emptool source to accomplish a backup....


|| loren.cox ||

is it just me or is Rob eluding to the point that there is already something in the emptool source to accomplish a backup....

...or is he just teasing us?

Well, they *did* promise playlists (but not tunes) backup a while ago...

Dragi "Bonzi" Raos
Zagreb, Croatia
Q#5196, MkII#80000376, 18GB green

I always figured that if you looked closely enough at the emptool sources, you could figure out how to pull/push any files you wanted. I'm surprised no one's done it yet.

As I was researching the "Reboot" function of the logo editor, I seem to recall having seen stuff like that. I never researched it any further, though.

Perhaps it's one of those things that isn't explicitly stated in the source code, but you can figure it out if you map out all of the protocols.

___________
Tony Fabris

I think so, when I first looked at the source, I figured someone was going to come up with a backup utility in a few weeks because I noticed the code in protocolclient.cpp had ReadFidToMemory and WriteFidFromMemory. I'm a very novice C and C++ reader, so I may be wrong here, but it appears that ReadFidToMemory is used by emptool to read the playlist database, the tags file, etc.

Assuming ReadFidToMemory can read any file on the empeg to memory, then that memory could be writen to a local file. So, it would seem like the framework to copy fids back from the player is already there, the source just needs to be abused a bit.


Am I close Rob? I would have experimented way back when, but I never found the time to relearn the little C++ I know.

I think you're right. I've never tried to compile the emptool for win32 as I don't have visual C++. I only looked at the code to see how a USB-connection was established and programmed that in Delphi. In this post is someone who managed to get emptool compiled for win32 with ethernet support. The distribution contains some .lib files, which I will try to import in Delphi, though I think VC++ .lib files aren't the same as Borland .lib files. Maybe someone with access to VC++ could create a DLL?


Frank van Gestel

I don't really know much about programming or backups, it's all beyond me, I'd better just leave you guys to it eh?

Rob

I still like the "cows_have_spoons" command...

Paul G.
SN# 090000587 (40GB Green)

I noticed the 4GB bug seems to be present still... or else I was having some other problem. But, by editing the outputted CSV file, I was able to continue where the back up stalled. Very easy to use!



32GB (JUST UPGRADED!) Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

So, was the 4GB "bug" traced to displayserver, windows, linux or anything else? BTW: it is very easy to resume when it konks out after 4GB by editing the CSV file...

32GB (JUST UPGRADED!) Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

The original bug was caused by myself, by being so stupid to forget that files have sizelimits, but hey, it was though out and implemented in about 30 minutes. The backup-utility however shouldn't have this limit, as it is transferring and writing multiple small files to disk. I know of some problems with displayserver and playlists, but files should download correctly. Maybe a FAT/FAT32 limit on the number of directory entries?
BTW have you got the player running while backing up? When you stop the player app, you will notice an increase in speed of about 100%, at least that's my experience(250kbs/500kbs).
I haven't downloaded that amount myself; I stopped at 2 Gb.
I will try to make a full backup soon, and let you know...

Frank van Gestel

Full backup didn't work for me - the speed started getting into the negatives :) and eventually stopped.
-mark

...proud to have owned an Empeg since 00287

Yep, speed got into negatives after about 1000 files (around 4gb or so)...

32GB (JUST UPGRADED!) Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

The cause of this is a memory leak in displayserver. 4GB is about the limit it will transfer before linux kills the process. Just reboot the player and keep going, that may work.

Sean

80000078

That is what I have been doing. Reboot the player, edit the CSV file and resume.

32GB (JUST UPGRADED!) Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

Well I tried the full backup and indeed it stopped after 3.xx Gb. I edited the CSV file to contain only multiple copies of playlists (which are very small) and it seemed the limit isn't related to size, but the number of downloads. Looked through the code and yes, the serve_file routine exits at the end, before closing the socket file descriptor. Due to the nature of the app, displayserver keeps allocating new filedescriptors till the limit is reached and dies...
I will make a version 1.2 where this will be fixed...

Frank van Gestel

Frank, you have to set up some Paypal account or something so you can get paid for this!

32GB (JUST UPGRADED!) Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

Thanks Frank! I was more than happy to wait for displayserver 2 for this to get fixed but this is great!

Sean

#78

Well, how about I give him a free "thank you" ice-cream at the Amersfoort users meet in a couple of weeks (you are gonna be there, aren't you Frank?)

One of the few remaining Mk1 owners... #00015

Just tried this on my empeg, and got an error "List index out of bounds (8)" when clicking the Start button.

It's running on a Win2000 Pro station, csv from my 20 GB Mark II, running Displayserver fine.

Any ideas?

did you edit the CSV at all? Mine hung after I altered the CSV file in Excel, so I had to edit it in Notepad...

32GB (JUST UPGRADED!) Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

did you edit the CSV at all?

Nope. Just did a clean dump from emplode. Give me a program with a CSV import feature, and I will break it :-) (/me remembers everyone loving the empeg remote control via serial that came out a while back, and it hated me :-)

I get "List index out of bounds (15)" _every other_ time I click 'Start'. The other time, nothing happens. Any ideas?

Since one of my two 80GB disks in my (unbacked up) Win2000 RAID-0 stripe set just failed, I'd like to get most of my MP3's back at least (though losing near to 160GB of media files is depressing)

I thought the whole point of RAID was so that you didn't get failures like that?

(Don't worry, I know what you mean. I've had my own share of bad experiences with stripe sets.)

___________
Tony Fabris

I thought the whole point of RAID was so that you didn't get failures like that?

(Don't worry, I know what you mean. I've had my own share of bad experiences with stripe sets.)

I try to avoid striping-only RAID sets. OK, you get large continuous logical volumes at expense of (slightly worse than) linear decrease in reliability (measured as probability of your filesystem staying with you (as opposed to bidding you farewell) at any given time interval). Mirroring is waste of space, Level 5 and similar incur performance penalty.... As Doug says, tanstaafl!

I don't even use RAID arrays for database storage. I prefer to hand-allocate volumes on continouos regions of physical drives (taking into account that seek times vary across the disk), manually assigning tablespaces, logspaces etc to them, and let DBMS manage mirroring (of course, where I can). More often than not I find after a while several heavily (and mostly concurrrently) used tables allocated into the same physical chunk of storage, competing for read head's attention. Ah, one never learns ones application's secrets completely.

I think that something like Veritas volume manager (with not only drive spanning, striping, mirroring, but hotspot detection and online reconfiguration) should be as standard part of every OS as text editor .

Now, was it OT or what?

Cheers!


Dragi "Bonzi" Raos
Zagreb, Croatia
Q#5196, MkII#80000376, 18GB green

In my experience, RAID 0 = BAD. I have built exactly two RAID 0 systems, and have experienced 100% failure in that arena. I had one Cheetah 10,000 RPM hard drive decide it was hungry and eat itself, and also one WD 2GB that found better things to do than serve my data (one of which was being thrown off the roof...)

Oh yeah, and back to topic... VERY nice Frank!

-Trevor

-----
Mk 2, Green 12GB 080000349

What does your CSV file look like?

Here is how the first line should look like :

"fid","type","length","title","artist","comment","genre","source","year","bitrate","codec","duration","offset","samplerate","tracknr","options","refs","contents"

the order of fields isn't important. I only use the field which has the columnheader "fid" as the same index, which is probably always the first, so it is important that you keep the first line when deleting some lines.

Frank van Gestel

Hmm, using 1.03 I get:

"fid","type","length","title","artist","bitrate","codec","duration","offset","samplerate","tracknr","genre","source","year","comment","refs","contents"

which is in a different order. But as you say that shouldn't make a difference. I haven't edited the CSV file at all - it is "as is" from export from Emplode. I'd love to get what looks like a tremendous utility working. Would upgrading to Displayserver 2.0 (alpha) make any difference?

Re: RAID-0 - I used it before with happy experiences, but as they say "the past is no guide to future performance". I was aware of the risks going in so I only have myself to blame. Worse thing is that I'm sure that the data could be recovered (by replacing the stepper motor, most likely) but the data (or rather the time to rip CDs and DVDs) isn't worth the $1000+ cost to get it done. Oh well...

I will try compiling a (network only) version of the emptool library some day in the near future. I will tell you if it worked out well.
However, I already dug in emptools sources for a while, and it seems as if that ReadFidToMemory is just the function we are looking for. PLUS it seems to provide a way to backup any player configuration files (well those of the standard player at least), including identity, general and network setup. I really wonder why noone of those mighty programmers around here didn't go and try this out month ago. I hope I get it to work, but even if not, I think I will be able to create a empbackup/emprestore program set for linux at least.
I will take a deeper look at the emptool sources during the weekend, and hope to be able to tell more by monday.

So what is the currect status on this tool? Is the 2GB limit still present? Also, Frank, do you still have a PayPal account set up for this?

32GB Mk. II in a WRX
Detroit, MI USA
www.PfeifferBeer.com

Hi.

I didn't get the emptool library source to compile under windows yet, at least not in the way I wanted it to compile. There are some facts about it available at my homepage http://www-new.incase.de/framed/empeg/, namely in the programming section. If someone wrote a tool to get a specific FID as a file from the player (and I think someone did this already, just don't have the time to look for it right now), and - even more importantly - upload a file to a specific FID on the player (which is yet to be done AFAIK), it would be fairly easy to write a small C wrapper to those tools that downloaded the complete music database (MP3s included) from the player to a directory on your disc or to upload the whole directory back to the player.

cu,
sven

proud MkII owner (12GB blue/green/smoked, was #080000113 is #090001010)

I would glady pay someone to develope an easy to use tool like this. I don't have the know-how to do it, and feel bad for crying about the lack of a back up utility. But as my collection nears 6000 songs, this would be nice.

32GB Mk. II in a WRX
Detroit, MI USA
www.PfeifferBeer.com

Once 2.0 beta is out, i think Frank will have a solution.

Sean

I would glady pay someone to develope an easy to use tool like this.

Uhhh, yeah. What he said.

At the risk of repeating myself, all I want is a Windows Icon (or a menu selection in Emplode) I can click on that will ask me what directory path I want to save to, and then the program would write a single big file that is unreadable, unplayable, uneditable -- just a big tarball that can only do one thing: restore an empeg to exactly the state it was in when the backup was made. It could even refuse to restore to anything other than the empeg that created it.

From my position of invincible ignorance, it seems like this would be an easy program to write. But then, my programming skills are pretty heavily taxed just to write a program in Q-Basic that displays "Hello, World" on the screen.

tanstaafl.

"There Ain't No Such Thing As A Free Lunch"

just a big tarball that can only do one thing: restore an empeg to exactly the state it was in when the backup was made

It would have to be a series of small tarballs, each one not to exceed 2gb in size.

That's probably the reason Frank's program didn't work properly after 2gb: The Windows 95 file system has a 2gb limit on single files. That's upped to 4gb on Win98 if you use Fat32.

___________
Tony Fabris

Hi.

That's probably the reason Frank's program didn't work properly after 2gb: The Windows 95 file system has a 2gb limit on single files. That's upped to 4gb on Win98 if you use Fat32.

While this is true (Win98/FAT32 can handle 4GB files), the windows explorer is (and many other apps are) not able to handle files of more than 2GB so you are strongly advised to only create files up to that size.
One way or the other: It is not possible to back up a complete empeg collection to a single file, at least not under normal Windows or even Linux file systems. A tar to a tape would be possible though, at least under Linux.

cu,
sven

proud MkII owner (12GB blue/green/smoked, was #080000113 is #090001010)

I understand the copyright concerns of wanting the single big file approach, but I have to admit that I want the actual mp3's back. I did most of the uploading with my notebook (and some with my PC). My notebook doeesn't have 32GB of free space, so most of the songs have been erased. And considering that some of the files are on another machine (that only has 12GB total drve capacity), I was not able to archive all my hours and hours and hours of encoding. I want to go buy a 80GB drive, and use it to copy my mp3's to my pc.

32GB Mk. II in a WRX
Detroit, MI USA
www.PfeifferBeer.com

I say, if you have MkII, go ftp route.

Dragi "Bonzi" Raos
Zagreb, Croatia
Q#5196, MkII#80000376, 18GB green

One way or the other: It is not possible to back up a complete empeg collection to a single file, at least not under normal Windows or even Linux file systems. A tar to a tape would be possible though, at least under Linux.

Or to a raw partition, which is what I'm currently thinking of doing.

normal Windows or even Linux file systems

Linux 2.4 ext2 and Windows 2000 NTFS have large file support (my 2000 machine had a 3Gb IMAP logfile that I'd forgotten I had told Outlook Express to make).

Peter

Has anyone tried building rsync for the empeg?

--
Toby Speight - 040103385

Hi.

Has anyone tried building rsync for the empeg?

I just did a build of rsync 2.4.6 for the empeg, which is available from my site (http://www.incase.de/framed/empeg). Have a look at the files section. I did not build any installation files neither did I test it. I only did the build.

cu,
sven

proud MkII owner (12GB blue/green/smoked, was #080000113 is #090001010)

"RSYNC is a program for copying files between networked computers. It is "special" because if the same file already exists, it copies only the differences between the files, therefore saving network traffic. Thus, you can make a mirror copy of a hard drive, and keep the two copies identical over a modem link."

This sounds really cool, maybe the functionality of this could be built into amold's utility somehow.

Sean

For those of you who are still looking for a backup utility that can suck all the mp3 files off of an empeg, someone else has written it.

See the post here: http://empeg.comms.net/php/showthreaded.php?Cat=&Board=empeg_general&Number=38723

Amold wrote a nice utility for Windows that works with USB or IP and simple sucks every mp3 down to a directory....

Thought I would point it out since it wasn't mentioned in this thread recently.

=-Clarke

______________________________________
Queue 351, Now Mk II #60000022 18gb, Blue

Thanks smu; I've downloaded and will test later.

BTW, some of your links are broken in /framed/empeg/empeg.shtml you link to a bunch of *.html that should be *.shtml (for example, files.shtml...).

--
Toby Speight - 040103385

For the linux users:
empeg> tar cf - /drive0 | nc 123.123.123.123 12345

123.123.123.123> nc -l -p 12345|tar xf -

Works quite well.

Now if only somebody could give me the nc sources so I can build a statically linked arm version ....

>if only somebody could give me the nc sources
>so I can build a statically linked arm version

http://rtr.ca/empeg/nc-1.10-4.src.rpm (includes arm patches).

-ml

Netcat sources?
That one is easy: Do an ftpsearch (for example with ftpsearch.lycos.com) for nc110.tgz Or tell me your email, the sources are only about 75k. Oh well, I just attached them to this post.
I was able to build a binary for the empeg some time ago (but can't find it now), but for some reason, I can't get it to compile and work under (and for) mandrake 8.0.
Update: I was able to compile it for both Mandrake 8.0 and the empeg, but I had to comment out a "#define HAVE_BIND" statement for both. Seems to work now.

cu,
sven