ReadFID ?

anyone try to get ReadFID to work ? if send a transferrequestpacket, I don't seem to get transferresponse, but just an ack packet.

Here's an exe that will download files off your empeg. For now, you have to figure out the FID using the CSV export from emplode.
I'm still working on refining it, so be prepared for strange behaviour. However, if you use Win9x, I'd greatly appreciate your giving this a spin.

And please don't ask me for source yet either. I'll eventually post it, but i have some big plans for the tool first.

One thing to watch out for: Sometimes it seems like the empeg won't respond for a long time. Just leaving it alone seems to work. Or, you can power-cycle (unplug the power, actually) the unit and restart the download.

Heh, I knew there was a way to do it. Wish I would have had a chance to research it myself more carefully. I could see this function in the source code, I just never bothered to try to implement an emptool subset.

NICE WORK !!!!

It works for me, although I get an error message "unable to write file 87" if I enter a random fid. If I actually look up the fids and enter a valid one, that seems to work.

So the next step is to retrieve a file list, followed by selecting from the list, right? And of course a Windows GUI. That's do-able, right?

Again, nice work!

___________
Tony Fabris

And please don't ask me for source yet either. I'll eventually post it, but i have some big plans for the tool first.

Sorry, but if you use our lib/protocol code you don't have a choice. The code is released under GPL and any public binary MUST be accompanied with source code.

In any case, releasing the source in the early stage of a project is a good way to get feedback from people who REALLY know the empeg protocol inside out.

Rob

The code is released under GPL and any public binary MUST be accompanied with source code.

Jeez, glass houses, Rob, glass houses. Anyone remember this little bruhaha?



___________
Tony Fabris

The only reason I did not want to post code is that I don't like half-baked versions floating around.
However, since you want to get legal on me, I will post the current source later today (pacific time) on my ftp site at ftp.blarg.net:users/amol/empeg.

At a rough guess, it was precisely because of that 'limited nuclear engagement'...

Geoff
---- -------
Got one of the first Mark 2 empegs...

Yeah, very well. Now we have our revenge

I'm not just being awkward. If anyone is using lib/protocol it is a good idea for our developers to see what they're doing. This could save someone a LOT of hassle if the unsupported functionality were to change, or if there were problems with the way something was being done.

If the author has a problem with this he can negotiate a special licence with the copyright holder (us).

Rob

Have you considered registering the project on Sourceforge and taking advantage of some decent revision control?

Rob

Very nice work!
It looks as if you also tried to get USB interface working.
MKI owners would love to see this.

John

I did try to get USB working, but the $%^# thing returns CRC errors from readfile. Since the USB code for windows is not fully included in emptool, I didn't have much to help me troubleshoot.

maybe in the future.

I think it would really be great to see a full fledged windows port of emptool, including support for serial, USB and ethernet. That would be a good starting point for a whole lot of nice tools.
Seeing the source of emplode would be even better of course, but I wouldn't even dream of that happening.

cu,
sven

proud MkII owner (12GB blue/green/smoked, #080000113)

Since the USB code for windows is not fully included in emptool, I didn't have much to help me troubleshoot.

That is what has always stopped me from starting on a backup tool, as I have a Mk1 I couldn't get very far without the missing USB code.

(well I could use serial I guess, but that wouldn't be much fun)
__
Unit serial number 47 (was 330 in the queue)...

since downloading from the empeg, is an officialy ignored aspect, what are the chances of getting a bug fix ?

The thing is that the empeg is apparently setting TCP_NODELAY on the connection socket. This is fine if all you're doing is small data transfers. But it degrades performance horribly for large file transfers.

Think 100 system calls instead of 1 to receive 1 packet of 16k.

I meant 10 calls to get 16k. not 100 !
(multiply that by the number of 16k packets for a 4-5 MB mp3, and you can see it adds up.)

The thing is that the empeg is apparently setting TCP_NODELAY on the connection socket. This is fine if all you're doing is small data transfers. But it degrades performance horribly for large file transfers.

Think 100 system calls instead of 1 to receive 1 packet of 16k.

The only effect of TCP_NODELAY on large file transfers, is that the final packet of the eleven needed to transfer 16K of data, is emitted immediately rather than hanging around in case there's a subsequent packet that can be coalesced with it.

The player always writes the response to a ReadFID command using a single write() call for both packet header and data; it always reads from the data socket into a 64Kbyte buffer. I don't know what 100 system calls you might be thinking of.

Peter

well, the PUSH bit is set on each and every one of the MTUs coming in. Which means (at least for windows clients), that each of those must be delivered to the application as they come in.
so, a 16k packet = ~12 1460 byte datagrams.

or 12 reads vs. 1 if all the 16 were buffered (if the PUSH bit was set only on the last packet, not each one).

( the 100 was a typo).

To further clarify. I know the difference between nagling and the PUSH bit. however, I don't think linux (or any stack) allows you to set the push bit from the app.
I *think* I read somewhere that some stacks will set the PUSH bit on every outgoing datagram if TCP_NODELAY is set.

If that's not true, I will have to think harder on this one then..

I *think* I read somewhere that some stacks will set the PUSH bit on every outgoing datagram if TCP_NODELAY is set.

Not the Linux one. And besides, PUSH is to do with whether to return from a blocking read after the first packet, or to wait for further packets; no client stack in the world will return *part* of the data from its input buffer just because one packet had the PUSH flag set. PUSH is not a TCP record boundary; there is no such thing.

If you are only receiving one packet per read call on Windows, it's because you're calling read more often than the wire rate of the packets.

Peter

you're completely and utterly wrong.

1. And besides, PUSH is to do with whether to return from a blocking read after the first packet, or to wait for further packets;
No, it indicates whether a receiving TCP should deliver the data to the application (at the same time ACKing to the remote side), or wait for the 200ms delayed ACK interval, in the hope of receiving more data. It has ABSOLUTELY nothing to do with blocking or non-blocking reads.

no client stack in the world will return *part* of the data from its input buffer just because one packet had the PUSH flag set. PUSH is not a TCP record boundary; there is no such thing.
If the PSH flag is set, it WILL be delivered to the application.

It does not matter what the receiving stack's buffer size is. I know it doesn't on Windows, and I suspect at least on BSD-derived implementations it's the same.

Have you actually tried your theory out ? Try receiving a file from the empeg in 16k chunks instead of 1024 like the current emptool seems to.


If you are only receiving one packet per read call on Windows, it's because you're calling read more often than the wire rate of the packets.
That's wrong again. you're telling me that the wire rate is at the most 579k/sec on a 10MB ?

I don't have a linux machine to investigate this behaviour myself. but I will try openbsd and see what it does.

In any case, if you're sure it's not caused by TCP_NODELAY then there's not much you can do. So I guess i'll have to live with it.

you're telling me that the wire rate is at the most 579k/sec on a 10MB?

Er, yes.

Peter

it's closer to 1 mbytes/sec, actually.

I do winsock programming for a living.

anyway, like I said if it's not TCP_NODELAY, there's nothing to be done.

thanks for responding. and my apologies if I was being rude.

The wire speed using the protocol is ~600kb/sec. Using a raw TCP pipe is quicker (1mb/sec, even on the empeg). The protocol is a big overhead, but it's easier for us to run the same protocol over everything, reliable (eg, tcp) or not.

Hugo

ok, that makes sense.

is it a request-response protocol ? I tried pipelining multiple requests to the empeg, but that didn't help.
I'm trying to understand where your overhead is. The checksum seems to be one bottleneck, but I don't know if it accounts for the whole delay.

Is there a better solution in 2.0 ?

Is there a better solution in 2.0?

There's a better solution for storing data to the empeg (you can expect 700-800K/sec; empeg's IO architecture can't sustain saturation Ethernet speed in through eth0 and out to disk simultaneously -- remember empeg has no PCI).

No similar optimisation has been applied to fetching data from the empeg.

Peter

There's a better solution for storing data to the empeg

Ah, here we are, here's a much more complete post on the subject.

Peter

A couple more questions:

1. can the empeg accept connections from two TCP clients ? i.e., is it even work trying to sync over two connections instead of 1 ?

2. On the same note, can it transfer independently over USB and TCP ? could I be reading a fid on USB and another on TCP at the same time ?

thanks

is it even worth trying to sync

No, and later versions have better protection against this. As I remember, there's only one command parser running which takes input from whateever channel is available.

Even if you could, you wouldn't get more speed; the limitation is generally I/O, and ethernet is the most efficient out of all the empeg's I/O (16-bit databus, 6k of buffer) - compare this to the USB chip with 8-bit databus, slower cycle times and 128 bytes of buffer.

Hugo

Hey Amol.

Are there any news on your download tool? I can't test it currently because my empeg is about 200km away from me. Does it work over USB, serial and Ethernet, or just one or two of those? Will you release the source? (If you used the emptool sources, you would have to do so.)

cu,
sven

proud MkII owner (12GB blue/green/smoked, was #080000113 is #090001010)

I have to release the source only if I release the tool :-)

actually, I was trying to get it working over USB before dumping the app+source on my ftp site.
it seems to work now. i'm just writing a crude backup utility that will walk the database and dump everything that's not a playlist.
should be tonight or tomorrow.

meanwhile, if there's a specific aspect of the emptool source you need to understand feel free to ask away.

i'm just writing a crude backup utility that will walk the database and dump everything that's not a playlist.

But on the contrary, playlists (and tag files) are the most interesting things to back up (provided we have a way to upload them back to the player)! So, my suggestions for additional empegmirror features (not very complicated, I believe) would be:

1) Along with a tune file copy the corresponding tag file. Include FID in the filenames of both. (Ehm, use id3lib to patch tune file's ID3 tags according to tag data ? Well, perhaps too much; this can easily be done with a stand-alone utility.)
2) Copy playlists as well.
3) Allow for backing up just a part of the player's content (so that users without large enough desktop storage still can back up piecemeal, say to CD-R). The program could have two optional parameters, FID to start from and amount of storage available; it would print the last FID copied when finished.
4) Add copying in another direction.
5) Back up, as a special file, the part of flash containing EQ presets and similar setup data. I have no idea whether we know enough to do that, though.

This way we could back up and subsequently restore our empegs almost completely.

I think this is a start of a very nice utility . I don't plan to use it at the moment since ftp does all I need, but not all users mess with their empeg software, add development tools etc. Besides, there are MkI players without ethernet...

Dragi "Bonzi" Raos
Zagreb, Croatia
Q#5196, MkII#80000376, 18GB green

a proper backup utility is of course the ultimate goal. I'm still in the design stage for that.