Perl Script for Empeg Backups

"Yet Another Thread About Backups???"

Well... I just added capacity to my Empeg, and with 1.1 coming, I'm planning a massive re-organization of my MP3's. It's much easier to work with the files themselves via tags rather than move things around in Emplode. So I wanted a way to get my MP3's all back to my hard drive. I may be re-ripping some or all of my collection in the near future as well, depending on WMA support. Now that I have the space, I might as well use it!

Anyway, fvgestel's Displayserver 1.1 is a splendid program that adds a HTTP server, and makes this process very easy. However, I wasn't able to find ANY "Web Whacker" type programs to suck down the entire Empeg structure. Most of them wouldn't follow the playlist links properly, and regardless of configuration, couldn't save the MP3's with the proper filenames, or in the proper folders. The backup facility in displayserver is nice, but it creates one massive file that may be too difficult to work with. The other solution was a filename linker script that was written some time ago, but that doesn't work well with single MP3's, because it would try to create a structure based on the Album in the ID3 tag, and would therefore not replicate the structure properly.

Frank is working on a new server that may have other backup capabilities, but in the mean time, I have cooked up a Perl script that uses libwww-perl to suck down an enitre Empeg's music. It opens up the MP3list page, and recursively grabs your MP3's, and places them in the right folders. Right now I've only tested it in Windows, but theoretically all you should need to do is change some of the character substitutions for your platform.

Known problems:

1. If you have duplicate playlists, or songs in multiple playlists, it's just going to dutifully copy them down twice. In preparation for a backup, you may want to remove duplicate playlists to avoid this.
2. Frank's displayserver doesn't tack on track numbers, and in many cases, you don't want it to. If you have album playlists, this program will download them in the order they were added to Empeg. If you followed the directions in Tfabris' FAQ on importing files, this won't be a problem, because you can just sort by date to get them in track order. You DID do that, didn't you? (Track numbers may also be in your tags, and a program like MP3ext can help you out.
3. Does not address world hunger, but that's targeted for a future release.

I wrote this in like an hour, so it's not perfect, and doesn't have any command line options yet, you have to hard code your own paths. Anyway I'm going to attach it here, anyone who has Perl for Windows and libwww-perl, and wants to backup their Empeg database, should find this useful. I think I can turn it into an EXE file and remove the dependency on Perl, but I'll wait until I hear about someone being interested first. :)



-Tony
MkII #554

Definitely interested in it, and am currently hacking on it for UNIX... Well... hacking on it when I have the time. I was trying to figure out the best way to do this, and you've saved me the trouble of having to do the logic.

No problem, that's why I posted it. It seemed like a textbook recursive programming project, and I hadn't done much with recursive stuff in Perl. I will say that this sucker's a memory hog right now, but it works, which is all we want Perl scripts to do. Writing this in C would have taken me a lot longer. This should work on UNIX perl, but I haven't had a chance to test it yet.

-Tony
MkII #554

No problem, that's why I posted it. It seemed like a textbook recursive programming project, and I hadn't done much with recursive stuff in Perl. I will say that this sucker's a memory hog right now, but it works, which is all we want Perl scripts to do. Writing this in C would have taken me a lot longer. This should work on UNIX perl, but I haven't had a chance to test it yet.



Well, it does work on linux/bsd, after a few minor edits (path markers). I'll probably add in some command lining and make it a bit more chatty, probably also optionable....

I'm not seeing it burn much memory as of yet on my test box, but I've also got a fair amount of memory to shove at it (1G).

I'll attach my edits here once I get it more generic with command lines.

In reply to:

---

Well, it does work on linux/bsd, after a few minor edits (path markers). I'll probably add in some command lining and make it a bit more chatty, probably also optionable....

---

Yeah all the stuff I didn't care about when I wrote it. :) Good idea, though, for those who don't want to deal with the hardcoded stuff. Once this is mature enough, I'll turn it into an EXE for Windows users who don't have perl installed.

With Frank's displayserver and this script (and, of course, Perl) backing up files will no longer be a chore. Except that doing these backups makes me wish the Empeg had 100BaseT!

-Tony
MkII #554

. Except that doing these backups makes me wish the Empeg had 100BaseT!



I'm wishing for gigabit, myself... transferring files between servers on my backbone is SOOOOO much easier... When I fired your script up, I wasn't sure it was working at all due to the 10baseT...

Patience is a virtue... Unfortunately, it's not one that I possess. I see what you mean about the Memory usage... I haven't gone OVER 64M yet, but it's wanted to come close...

If I'm reading this right, these scripts just use displayserver to grab the MP3 files, right?

This is nice, but what we really need is a way to back up the empeg's database information. I don't have any problem putting the MP3 files back onto the empeg if I lose a hard disk. What I have a problem doing is reorganizing my playlists afterwards, re-creating all of the linked stuff for my custom playlists that I had before. In the case of 1.1, this includes Wendy Filters, too.

___________
Tony Fabris

Yah, this is just a quick and dirty to grab the MP3's. I tried no fewer than six Web leeching programs and none of them did the job, so I made my own.

Backing up configuration data seems like it would involve consolidating info from /empeg/var and also reading the data off the flash. I'll look into this. The perl script was just meant to fill my immediate need to get my MP3's back for sorting and re-tagging purposes, and I thought it would be useful for anyone in a similar predicament. I agree the next logical step is to get a backup/restore config facility, and that sounds like a fun project.

-Tony
MkII #554

Not being a programmer myself, I can't explain how relieved I am to know that you guys are working on this! Everytime I read of someones hard drive failing, or empeg being stolen I just cringe at the thought of losing all my mp3s! I use two machines to load the mp3's to the empeg so there isn't one machine that has all the songs on it. Also, I started to run out of disc space and had to remove many albums from my desktop machine. I am planning on getting a huge drive JUST for empeg backups, but there hasn't really be a good way to do them (over 2gig) until now!

12gig Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

Also, I started to run out of disc space and had to remove many albums from my desktop machine.

So if you're so worried about losing the MP3s, why aren't you making backups before removing them from your desktop machine? CDRs and tapes are cheap.

Also, remember that if these really are your files (you ripped them yourself instead of leeching them from Napster), then you've still got the original CDs and at the very worst could simply re-rip them.

I've had two hard disk problems since I've owned my empegs, and both times, the actual MP3 files were the least of my worries. The only hard parts were re-doing the EQ presets to my liking, and re-organizing my custom playlists.

___________
Tony Fabris

Backing up configuration data seems like it would involve consolidating info from /empeg/var and also reading the data off the flash.

Some of it is in var (the config.ini), but most of the relevant data is either in FIDs or stored on the scratch partition. In my thread about "partition help" FVGestel described a method to save off the scratch partition.

I'm sure that if I had enough forethought, I'd be able to save off the playlist data simply by exporting the CSV file at regular intervals. I wouldn't have an "automatic restore", but at least I'd be able to see what songs were in the custom playlists and it'd be fairly easy for me to reproduce them. Of course, I never think to save off a CSV when I'm synching, so it's kind of my fault.

You know, a lot of the functionality needed for backing up your playlists could be implemented in the open-source versions of emptool...

My dream would be for Emplode to just "cache" the data from the last couple of successful synchs and allow you to force that stuff back onto the empeg in a pinch, perhaps prompting you for the locations of the MP3 files first. That way, I'd never have to consciously back something up. And I'm not talking about breaking the paradigm of Emplode being a dumb terminal. I just mean have emplode remember the playlist organization just in case of a tragedy. It's already in the PC's memory after emplode synchs sucessfully.

___________
Tony Fabris

Otay...

Minor hacks.. Include:

Unix centric now... Windows not allowed. Actually, the reverse the / \ relationships again, and your good to go.
Converts / in Playlists-or-Song titles to -. This one kicked me a bit.
Commandline options. First argument is the empeg address, second argument is the directory you want it to go to.
Added a little verbosity.

My dream would be for Emplode to just "cache" the data from the last couple of successful synchs and allow you to force that stuff back onto the empeg in a pinch,

MY dream would be to have a utility in Emplode that would simply mirror the entire contents of the empeg, including database, to my local hard drive. The resulting output file would not have to be editable, playable, changeable, displayable or otherwise usable in any way -- just so long as I could mirror it back to the empeg to do a complete restore. I think that a file of this nature might be exempt from empeg's justifiable concerns about enabling piracy. The program could even be made so that it would only restore to the empeg that wrote it (same serial #).

If you wanted to take it a step further, make it a "smart" mirror that would skip over any files unchanged since the last backup. The first backup you did would take a long time, but subsequent backups would only add new additions to the backup file, and over-write modified files (i.e., the database files). And a "really smart" mirror process might do a full synchronization -- add new files, delete old files, update changed files -- but this might be overkill.

I would pay money for such a program, either as an enhancement to Emplode or a standalone utility, and I suspect there are hundreds of other people out there on the bbs who would do likewise.

Since my programming skills do not encompass much more than writing a "Hello, World" screen display in Q-Basic (and I'd have to find my Q-Basic book to even remember how to do that!), I leave the implementation of this "dream" to my betters. (Are you listening, Frank?)

tanstaafl.

"There Ain't No Such Thing As A Free Lunch"

Hmmm.... Well, That actually is pretty doable.

Rsync over the ethernet and dd should take care of the problem in pretty short order. You could pull a seperate file for each partition. This would make it pretty version dependant, however.

Hmmm. I'm going to have to take a quick look at this.

Rsync over the ethernet and dd should take care of the problem in pretty short order.

Yes, but that would involve installing stuff on the empeg, and would require a developer image. The things Doug and I would like to see should be built into emplode and should not require anything more than an regular install of the player software.

(But your solution would still be cool for those of us that don't mind working at the bash prompt.)

___________
Tony Fabris

Given that it's not likely that Empeg is going to put this functionality into the player software at this point, I don't see much choice besides this approach. I'm certainly not going to suggest it to them... I want 1.1 first!!!!!!!

Certainly it would be better to have this done by the Emplode software, and in reality, it really shouldn't be that big of deal for Empeg to do this. Enough so, that I'm really pretty curious as to why they haven't already. Certainly I understand the b*llsh*t about copying Mp3's, but the EQ settings, dimmer, etc. and playlists (maybe) should be easily backed up. The playlist is a maybe simply because it would be pretty easy for that to be out of sync.

My approach is certainly a brute force attack, as I'd basically grab images of the partitions, run them through a quick compressor, and pull them off via the RSYNC server protocol. Total upload is the rsync binary, and a config file. Maybe a cut down inetd to launch displayserver on demand at port 80, and the rsync server when needed.

It's not that big of a deal to do, but it's pretty wasteful to grab a few settings. Frankly, I'd rather pull the mp3's via the perl script than in this manner, but it's certainly doable this way, assuming you have the space. The rest of the stuff on the system should pull off pretty small.

This script (both my version and synergy's) are slightly broken. Specifically, URL's aren't being URI-escaped, so it's going to miss the occasional file. The # character causes serious problems, and on Windows, it wasn't substituting the colon character properly, which broke things.

Anyway here's a more recent version, with the same warranty (none) as before. I fixed several things, and it looks like I'm getting all my files okay. Uses URI::Escape. Synergy, you might want to add the same things to yours, I have a feeling your version also chokes on the pound sign, which has to specifically be substituted (uri_escape doesn't do it for some reason.)


-Tony
MkII #554

I agree with tanstaafl's idea of an ideal utility.

I don't know about you, but with a dial-up connection, filling up a 12gig player with files from Napster would take about an eternity or so. I own all of the CD's that are on my empeg. But, considering it takes me about 2 hours to rip a CD to my hard drive, I would much rather have a simple backup utility to do a backup rather than re-do all my ripping.

12gig Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

Yup.... it broke on the #... I didn't notice because only a few of my songs have that character. The / was in a playlist, which F*cked things everywhere.

Attached is a quick merger/fix for Unix. I'll probably Zip both scripts up and put them out on a website simply so a user can quick grab both, and use whichever he needs. Unless you want it... It's your baby, not mine... I just slapped it a little.

Was this script ever put in an .exe form for us windows guys? (Sorry, don't even really know what Pearl is...)

32GB (JUST UPGRADED!) Mk. II BLUE
Detroit, MI USA
www.PfeifferBeer.com

I don't think Ynot_ ever did...

Basically, all you should have to do is install perl ( I THINK it's on the resource kit), and then run from a command line :

c:>perl script.pl

That should take care of it. Oh... Plus you'd have to get the CPAN modules listed in the script as well, as I'm sure MickeySoft didn't put them there.

Hell, just format, install debian, and be done with it.

I haven't gotten around to it yet. The perl resource kit has a perl2exe thing. I'll hook that up when I have a free minute at home.

-Tony
MkII #554

Ynot_...

I'm sure you have thought about this, but you will need to make sure that you pull in the command line options part from my mods (or your own), as once that sucker is compiled, it's going to be difficult to change the settings...

Third time is a charm...

Unix version only, Correctly takes care of the # problem. Inside the script there is a debug flag to set... much status results when set, defaults off.

Command line options:

getmp3.pl Empeg_IP Directory_to_download_to

I *THINK* this is really good this time.

Unix users should only have to run the file, no hacking inside needed...

SHOULD.



Oh yeah.... If you store playlists in a Playlist labeled "Playlists", this script will happily ignore them. This works for me so I don't get duplicates. It might not work for you.

Yah I have already done that (command line opts), just haven't posted it here yet. Yeah editing those hardcoded paths might be a little tough.

Plan B is bundle the EXE with a hex editor. :)

-Tony
MkII #554

IANAPP (I am not a Perl Programmer), but wouldn't it be easier to use an ini file or something?

_~= Dearing =~_
"WAY too happy about having #99."

IANAPP (I am not a Perl Programmer)

I always wondered what that stood for - Radio 1 are forever going on about it.

Peter

And I always thought it was 'Hiya Nappa' - a greeting to the family members.

Paul.

Paul Haigh, Reg. 4120
(mk1) 6GB, Blue, 00254
(mk2) 12GB, Red, 00357

Ynot,

when i execute your I do not know anything aboutt perl except that I need it to use your script. I would love to get some advice. When I execute your perl script with the ip address of my empeg substituted in I get the following response from perl:
Bareword found where operator expected at getmp3s.pl line 25, near "$server = "h
ttp"
(Might be a runaway multi-line "" string starting on line 24)
(Missing operator before http?)
Bareword found where operator expected at getmp3s.pl line 25, near "//empeg"
(Missing operator before empeg?)
Bareword found where operator expected at getmp3s.pl line 26, near "$urlroot = "
http"
(Might be a runaway multi-line "" string starting on line 25)
(Missing operator before http?)
Bareword found where operator expected at getmp3s.pl line 26, near "//empeg"
(Missing operator before empeg?)
String found where operator expected at getmp3s.pl line 43, near "$dest =~ s/[?*
""
(Might be a runaway multi-line "" string starting on line 26)
(Missing semicolon on previous line?)
syntax error at getmp3s.pl line 25, near "$server = "http"
Unmatched right square bracket at getmp3s.pl line 43, at end of line
Unmatched right curly bracket at getmp3s.pl line 96, at end of line
syntax error at getmp3s.pl line 96, near "}"
Execution of getmp3s.pl aborted due to compilation errors.


Any idea how to correct this?
Thanks,

BG

Bag... Couldn't reply via email, but anyways..


You need to escape your localroot path;

you have

$localroot = "f:\";


Needs to be

$localroot = "f:\\";

The two slashes are not a mistake.

Yeah, what he said... :) You know, fvgestel's backup utility probably does a better job than my Perl script does these days. I seriously don't think my script will work with Displayserver 2.0...

-Tony
MkII #554

Not out of the box it won't....

But it will.....

Should just be a matter of creating a view on displayserver 2 that looks like the old one.

I'm going to look into it shortly.

It could be even in a more flexable way than wuth 1.1
You could read in the js-array and create perl-arrays from them :

foreach (grep /fidarray/ @linesP) {

   eval '\$'.$_;

}

then you could choose how to name your files by specifying index numbers for attributes

Frank van Gestel

Basic Question:
what does this perl script back up exactly? Namely, are playlits, dbase, eq backed up?
Does it RESTORE, also?

Thank you!

Just MP3's. Restore would be done by re-emploding the MP3's. No restore program per se. It was just a quick hack to grab MP3's from Displayserver automatically.