Known format for "database"?

Is the format of the database file explained somewhere? I'm working on ways to speed up syncs. I just don't want to spend time figuring it out if people already know. Obviously, I see song names and such, but is there a pattern to all the non-printable hex codes in there? I haven't seen one after looking for only a few seconds, but I'm sure I can find one with time. I looked on riocar.org, but I only saw this type of info for the FIDs. Thanks!

I believe that the empeg guys have claimed that it's documented in the source code to the Linux command line utilities.

Ah, ok. I'll check it out. Thanks!

I believe you mean "documented" -- in quotes.

True. Or maybe I meant documented by the source code.

If I'm not totally stupid - or overworked - the command line utilities just _tell_ the player to rebuild the database.
Request::BuildAndSaveDatabase
just send a packet to the player
and the player - AFAIK - is closed source

"Injecting" a new tune onto the player and into the playlists via ftp (hijack) is quite easy (and reliable !!), but I don't gain anything if I have to start jemplode afterwards to rebuild the database.

The format of the database itself doesn't look that complicated,
so please share everything you find out!

With a little help from Roger and smu, I have figured out the formats of the database, playlists and tag files.

The next upcoming version of mp3tofid will build these files on your PC, considerably faster than on the player (at least on my PC).

Source will be included. Stay tuned.

Pim

Well, thanks for saving me the trouble! I was going to add it to your program once I figured out what was going on. Are you able to preserve dynamic database info, like play count?

Great.

That'll save me some trouble reverse engineering ;-)

Can't wait to get my hands on it...

Any ETA ?!

Yeah, but they know how to parse it, though. Work backwards from there. Although, as Pim says, his next version of mp3tofid should support it.

In reply to:

---

Are you able to preserve dynamic database info, like play count?

---

No, unfortunately not. AFAIK, the playcount is stored in the dynamic partition, and only written back to the tag files and the database upon a database rebuild.

There's no way to do this if the database is being built off the player.

Pim

In reply to:

---

Any ETA ?!

---

Should be there within two weeks. Basically, everything is working. I just need to do more database and playlist consistency checking. Previously, emplode would do that while rebuilding the database, but now I have to do it all, as emplode is no longer necessary in the sync process.

Pim

It's not supposed to be in the tag files at all. As long as you leave the FIDs alone, the play counts should stay correct.

Any word on this? A format for playlists would be super right about now.

Greg

I've just announce mp3tofid 2.00 in the programming forum.


But perhaps it's easier to explain the database files
in English than in C code.

The database files are probably also much easier to
write than to parse.

The playlists file is simply a concatenation
of all playlist files in FID order. This is easily
seen as the first part of playlists is exactly the
same as the root playlist (/drive*/fids/100).

The tags file and the database file are
tightly related. tags is a text file containing all tag
names in all tag files (/drive*/fids/*1). Order does
not seem to matter, though I guess "type" needs to
be on the first line. The tags file needs to be padded
with newlines until it is 256 lines long.

database is a binary file, containing all tag values
for all FID's in FID order. All nonexisting FID's
until the FID with the highest FID number need to be
included, as an empty record, closed by 0xFF.

Nonempty records consist of one byte being an index into
the tags file, one byte stating the length of the tag
value and then the tag value itself. The record is
terminated with a 0xFF character, just like the
nonexisting FID's.

FID 0 is special and needs to be in the database,
even though it does not exist in the fids dirs.
FID 0 has just one tag: "type=illegal"

This all means tag values may not be longer than 255
characters, and may not contain newlines. This
currently is not enforced by mp3tofid.

Pim

Tags and Database I know have working with success, thanks to Frank's very handy efforts in education.

The playlists file seems to more more obscure than it should be.

I would expect a playlist fid in the playlist file would be followed by all the track fids and sub-playlist fids immediately afterwards. But this doesn't seem to be the case, at least for me.

Last time I looked at it, a playlist and it's contents would be separated by a long list of other unrelated fids.

Any help in grasping the ability to identify all the fids associated with a given playlist would be a super plus.

Greg

Well, the playlists is probably much harder to parse than to create. Like I said, it's a concatenation of all playlist tags in FID order.

So to find subplaylists, you will need to calculate the offset in the playlists file for your subplaylist.

To do that, you wil have to look up all the lengths of all playlists in the database file and build an offset table from there.

That's why it's very important that the length tag for playlists is correct.

Pim

Thanks.

Your code saved me some time.

I wrote this small util (attached) to check if I understood the format.
Nothing fancy - or clean, but easy to understand.

Question:
Who decides when if there should be a play_count tag in the database or not ?
I have a few for some songs, but they are wrong.

for the database, if you're not using emplode, is it possible to recreate it using different field lengths? i ask because before, i asked about the ability to lengthen the comment field from 256 to something like 1024 for the purpose of having a whole tracklist of a one mp3 album. [mostly mixed compilations].

No. The length limitation is because of the single byte used to express it.

If it's just a concatenation of playlist tags in FID order, then where do we get the offset that tells you how far away our subplaylist is? A playlist entry doesn't have an offset tag to tell you where it is from it's parent, unfortunately. As for looking at the lengths of the playlists in the database file to figure out the offset, this doesn't seem to work for a number of cases ... the following is an example.

I've got a playlist called Webb Wilder (17c1, length 52). It's original contents have long been deleted, as of now it only contains one subplaylist a really long ways away: Acres of Suede (13c01, length 48). Other than my personal knowledge that these two are parent and child, I can find no other relationship in any of the database files. There's no offset tag for the playlists, and the length of the parent list isn't even close to the real offset between it and the child.

Greg

It's not a catenation of playlist FIDs. It's a catenation of the contents of the playlist FID files. It lists the children of each playlist.

Ok ... I think I'm getting the picture now. So to get this working you have to:

1. Note the order of the playlist entries in the database file, and their length tags.
2. If you want to find what fids are children of a given playlist (X), you sum up all the lengths of playlists (1) to (X) to make an offset (Y).
3. You open up the playlists file, go to offset (Y), and read in (length of playlist X) fids.

I guess the main assumption I'm making is that fid lists in the playlist file appear in the same order as their parent fids in the database file. Does this pseudo-code sound right?

Greg

This appears to work. Maybe we'll have a RioPlay showing the real playlists from the empeg soon.

Greg

Yes.

In reply to:

---

Question:
Who decides when if there should be a play_count tag in the database or not ?
I have a few for some songs, but they are wrong.

---

I think playlist counts are stored in the dynamic partition, on a secret location, using FID numbers as offset. This means if you delete a tune, a next tune might be stored with the same FID number, and thus inherit the playcount of your old tune.

Pim

The player database does not allow tag values longer than 255 chars, because there's only one byte to indicate the tag value length.

But there are some options. Currently, mp3tofid does not include the comment field into the database to save on size. So in theory, its length in the tag file should not matter. But if you ever would recreate the database using the player and/or emplode, bad things could happen.

And you also have the option to include custom tags. mp3tofid uses two custom tags: "loadfrom" to remember the location of the tune on the host PC, and "frames", an MP3 frame count. Both of these tags will not go into the database built by mp3tofid. You could create your own custom tags, and split the data like you suggested.

If you keep the tag values short enough, the player should happily stuff them into the database shoud it decide to rebuild it. But bear in mind that as the database size increases, free memory on the player decreases.

Pim

will it be possible to express the comment field as two bytes in later versions of emplode? would it break anything?

In reply to:

---

If you keep the tag values short enough, the player should happily stuff them into the database shoud it decide to rebuild it. But bear in mind that as the database size increases, free memory on the player decreases.

---

yah, tony and i had a discussion about this 1st time around.. and my point was that one big tag is better than N amount of little tags re-iterating some stuff (album name, year, artist, genre, etc).

its not like i'm going to store 65535 bytes of data in there or something. just annoying when i want to know what song is playing and my comment field cuts off.

In theory, this is very possible. But it would change the database format which needs to be interpreted by the player. So the player would need to be able to distinguish old-style and new-style databases.

I don't think it's very likely this change will ever happen.

Pim