If the bindle identifier string (typically "com.softwarecompany.appname") matches and the version number match, the app is listed as "installed." You have to trash the existing app, which may be a demo version, to be able to use the buy button in the store.

I really wish Apple had invested a little more time and effort into setting this up to allow migration from stand-alone app versions. I suppose however that that would also open up a desire to migrate back in the other direction which could cause some headaches.

I also wish there was more reporting available. As it stands, developers are losing a lot of what has previously made the self-distribution model so powerful from a market/marketing insight perspective. You'll never know who's buying your software for example.