Lots of hype today over a new tool someone released that digs into a database iOS 4 devices create. I fired it up this morning, after decrypting my iPhone backup, and it did pretty much show the areas I've been in since iOS 4 came out. This included a nice streak across the US when I drove from Austin to Southern California late last year.

I've been looking into it deeper today, and figured some of the info might be of interest to others here. First up is a response Apple sent to the House of Representatives last summer. Page 6 talks about the data collection, basically Apple switched from using Google for cell triangulation and Skyhook for WiFi to their own database in iOS 3.2 and above.

People looking at the sqllite database directly have noticed the coordinates seem to be the locations of WiFi base stations or cell phone towers and not the location of the phone at the time. Some attendees of WWDC 2010 have commented that Apple talked a bit about this file, and the purpose is to cache location information for power savings reasons. The idea is that the phone has to do far fewer calls over the network to get this data, especially for people who use location services in the same areas frequently.

Not really sure what I think about this yet, beyond curiosity to see what the data reveals if I map it out with more precision. An update to iOS may start wiping this data from time to time depending on how severe the outcry is.

It's not really a privacy concern any more than the information you enter into your own phone. The issue here is that there isn't a setting to control the behavior or to manually wipe the data. I suspect/hope we might see something like that in the future. Something else Apple might do is eliminate any association of the data with dates.

The data on mine definitely shows very rough areas I've been "around" but it's very far from exhaustive. It also displays a start date earlier than my iPhone purchase. The points on the map are a lot more localized than the areas I've actually been to with the phone.

Couple it with a Cellebrite-equipped cop pulling you over and you might be more concerned about privacy.

http://www.myfoxdetroit.com/dpp/news/sou...k-20110420-wpms

If I get arrested in Michigan, my first call will be to someone who has my iTunes password so they can remotely wipe my phone. Thanks for the heads up.

From what I've been able to dig into, turning off Location Services entirely would disable this, but I'm not sure if it clears the existing file or not. May have to play around with that and see what happens. The downside of course is that all location based services do get shut down with that switch, so not even the basic Google Maps would function.

My current iOS devices are set to wipe the device if the unlock passcode is not entered properly after a certain number of attempts. This is mostly to protect work data that is on the devices. Unfortunately I'd likely not have time to blindly trigger this, as the phone goes into a lockdown for a minute after 5 failed attempts, and only a few more failed after that would initiate the wipe. It is really a shame that data privacy in the US is so weak these days. Even though I remained within the US, the Border Patrol had the right to search my laptop when they stopped me during my drive from Austin to SoCal. Do I have anything incriminating on the laptop? Not that I'm aware of, but I also don't want other people just randomly searching it. Nor would I want a cop in Michigan scanning my smartphone with a Cellebrite.

So what stops "them" from dissembling it and going for a lower level readout?

Much clearer post from someone who has already done a field trial to see what was tracked.

http://www.willclarke.net/?p=247

Definitely only recording where cell towers and wifi spots are, and not the exact location of the phone. Still potentially bad that it shows what areas a person has visited, but only if the phone is taken and plugged into a computer or Cellebrite type device.

I'm not sure how feasible it is to dismantle an iPhone or iPad and read data directly off the flash chip. My current understanding is that with the 3GS and above, all data on that flash chip is always encrypted, but I don't know how secure the key is.

It's currently easier with the iPhone to just jailbreak the device to get around the protection, even with the latest 4.3.2 release. For now, the iPad 2 has yet to be broken in that way, so I put a little more trust in it.

Security wise with my company, a compromised phone would at most get someone access to my e-mail. Anything worth accessing is protected in other ways that can be locked down quickly. E-mail could also be wiped, leaving someone with just the cached past few days worth of messages on the device. I do also have both active with the Find my iPhone service, allowing a wipe command to be sent over the air.

With recent devices I believe the first thing iOS does is delete the key that the rest of the data on the device is encrypted with.

...assuming that your phone can be reached over the air, no? Surely all they have to do is put your phone in a Faraday cage or similar?

Or put it in Airplane mode, turn off the 3G radio and make sure it's not on their WiFi network, turn off the FInd my iPhone feature (if it's not password protected), or turn off the phone.

The market for faraday cage resale isn't as big as one would think.

I don't have a particular paranoia about cops or other government agents searching my things, it's mostly an annoyance about the processes they are allowed to do (such as potentially scanning the entire contents of a smartphone during a routine traffic stop). If I were ever in a situation where they were deeply searching my things to the point of dismantling them or defeating the first layer of security, I probably have bigger issues to worry about at that point.

Most of my worries are more about what happens if i lose a device, or if one is stolen. There are plenty of ways to stop the wipe command from arriving, but I'm not as concerned about the device if they have cut off all network access. It's mostly a concern about what anyone could access directly using the device, requiring it to be online and also receptive to the wipe command. Today, that concern is mostly related to any work related things on the device. Down the road with payment options and other uses of the phone, it will mostly be a desire to prevent fraudulent charges. I already use my phone today to pay for things at Starbucks. I could easily see it replacing my full credit card in the next few years.

Senator Al Franken* wants to know what the deal is.

I don't think this sort of thing surprises any of us when it's discovered, and that makes me kind of sad. I expect phones to have these sorts of features, but it's very "Apple" of Apple to bake this tracking "feature" in with no prominent mention of it or explanation of its purpose and no way to turn it off.

I'm interested to hear back about what they're doing with the info.

* I still get a kick out of that.

I think it's best described as a "caching feature" rather than "tracking feature." It doesn't actually track you, not only because it doesn't mark your position, but also because none of that information is transmitted anywhere.

Also, it was apparently openly discussed last fall. This week it made a bigger splash in the media because of the app that visualizes the data.

If it was only a "caching feature", there would be little need to retain all of the data indefinitely, as is apparently done.

It could very well simply be something that wasn't implemented - a purge feature. I'll bet money right now that it's coming though. Obviously because the issue has been brought to the mainstream.

None of this is an excuse for Apple not having been more forthright about this, nor for not including a method to clear/purge the data automatically/manually or turning the feature off entirely.

I believe the consensus at this time is that the data helps the device to save battery. Somehow.

My data is nowhere near as dense as some of the examples I've seen others post. No Oakville, barely any Mississauga, no Toronto, no Montreal, no Tremblant and nothing South of Burlington - all places I've spent significant amount of time connected to 3G and passing by wifi hotspots. And honestly, I've not spent any significant amount of time at the 407 and 400 where you see that cluster in Vaughan. At most, I can remember driving through along the highway in fact.

Yeah, I linked to an Ars story about Franken, just didn't spell it out, and I do share some amusement in him being a senator now. Will be interesting to see the response this time, since the one provided to Markey last summer was a handy quick summary into what Apple was doing location wise, and much easier to parse then the full EULA.

Someone posted this on reddit


The EULA does point out on page one how to disable anything location wise, and I'd assume it includes the population of the local cache. Will have to test myself if it is wiped if the switch is turned off, or just frozen at that point. Noone seems to be clear on that.


The current iOS 3.2 implementation (it shipped on the iPad 1 first) was known about for a while, and the previous implementation has also been known about. More on it can be found here, written by Alex Levinson, one of the people who first started investigating it.

And the same thing has been found on Android phones, with one difference in how the data is retained. Android will only cache 50 cell phone sites and 200 wifi spots, and when it hits the limit, the oldest entries are removed to make room for the newest.

https://github.com/packetlss/android-locdump

Interestingly, the source code that manages this moved from being part of the open side of Android over to the closed side at some point.

Thinking about this more, an easy solution for the tracking fears would be to replace this cache the phone creates for AGPS with pieces of the database directly from Apple when syncing/updating the phone. It could do a quick location update, and seed the database with information in a 100-200 mile radius. Not sure how large the DB would be for that, but the size could be adjusted to still maintain quick performance. Maybe even let the user pick a general area, for frequent travelers.

Not sure if Apple would be willing to do this though, since the master lists for WiFi access points are quite valuable. Skyhook's sole purpose for existing is to generate these lists, and sell access to them.

Amusingly people will still happily tweet and post to facebook.

Yes, it's a bit daft of Apple to allow that level of caching, but real world impact... not so much.

Seems the echo chamber continues to hype this one, to the point where two users have decided to file a lawsuit.

The Wall Street Journal also is reporting that turning off Location Services still causes the iPhone to collect the location info for cell towers and WiFi spots. I however found the opposite in my testing with a freshly wiped 4.3.2 iPhone. In my test, I did have an initial cache created, but it never added any data even when I opened Maps with location off. If I turned location back on, the cache updated. I did the test by driving the same route south of my house twice, the first time with location disabled. I did a sync to a laptop not normally used, wiping the backups each time to force iTunes to do a full backup each sync. I chose to head south due to the initial cell data in the database showing cell towers up to 25 miles north of my home, but nothing south.

I'm going to file a bug with Apple since turning off location services doesn't clear the cache and see what they say. It does seem to halt the collection of new data though, so anyone specifically worried about this can switch off location services and encrypt their existing backups.

I didn't bother to dig deeper to see if any of this was sent to Apple, mostly since their EULA and response to congress last summer already confirmed they collect other location data on an anonymous basis if people use location services.

Apple formally responds with a FAQ, and a software update will be made available later to do 3 things:

1. reduces the size of the crowd-sourced Wi-Fi hotspot and cell tower database cached on the iPhone,
2. ceases backing up this cache, and
3. deletes this cache entirely when Location Services is turned off.

After personally looking into the files and testing it this weekend, most of what Apple says seems to be plausible, and heres why I believe so:

I think it was an oversight that the cache wasn't being purged, possibly due to last minute iOS 4 changes. Apple has been rushing things lately on the iOS side, and a stressed out engineer could have missed a change he was tasked to make. iOS 3 contained a different cache file called h-cells.plist. iOS 4 (rather 3.2, the iPad only release) changed it to the consolidated.db, and added WiFi. These changes occurred as Apple began using their own WiFi service instead of Skyhook for WiFi and Google for cell tower info. I'm not 100% sure if the previous h-cells.plist was being purged on a regular basis, but from what I've seen, it wasn't as extensive as consolidated.db.

The file being backed up is likely a proper bug/mistake. H-cells.plist wasn't ever backed up, and was stored in /Library/caches/locationd for the root user. The new consolidated.db file is also stored in the same folder path. So where does the bug come in? iOS 4 added a persistent settings file to allow location to be enabled or disabled on an app by app basis. This is controlled by the initial user prompt when using an app, or via a control panel of per app toggles in the system Settings app. iOS 3 and prior would prompt multiple times to use location and lacked a central settings panel. This setting file also gets stored in /Library/caches/locationd. I'd bet that the person who implemented it added the full /Library/caches/locationd folder to the backup include list, instead of his one file. The proper way would have been to put the settings file in the normal place, /Library/Preferences. Definite code/implementation review failure here.

Number three is tied in with Apple claiming it's a bug that turning off Location Services doesn't always disable the cache. Deleting the file will ensure this isn't an issue. In my private testing, turning off Location Services did stop updating the file, but others are reporting it didn't stop for them.

At this point, Apple will still be called to task by the U.S. government as well as several other governments. In the end, I expect they will be required to have some sort of opt-in/opt-out switch on whether any of this data, anonymized or not, is sent back to the mothership. I expect similar requirements for other phone vendors.

The zillion dollar question for me is whether this will ultimately be an opt-in system or an opt-out system. My guess is there will be a united front from the vendors to push for opt-out, and that will probably win in the U.S., but not necessarily other countries.

According to Apple, none of this information is sent from the iPhone TO Apple, and in fact it's downloaded FROM Apple to the iPhone. Which does make sense, because I saw plenty of small dots on my own map, that likely represented WiFi base stations, that my iPhone was never in range of.

This would very much suggest that some location data is sent, in some form, to Apple:

"5. Can Apple locate me based on my geo-tagged Wi-Fi hotspot and cell tower data?
No. This data is sent to Apple in an anonymous and encrypted form. Apple cannot identify the source of this data."

Which would make sense if they are building up their own wifi geo-location database.

and also:

"These calculations are performed live on the iPhone using a crowd-sourced database of Wi-Fi hotspot and cell tower data that is generated by tens of millions of iPhones sending the geo-tagged locations of nearby Wi-Fi hotspots and cell towers in an anonymous and encrypted form to Apple"

I suppose the information does have to go both ways, but the cached data shown on the maps is a slice (albeit one tailored to you) out of Apple's DB.

I'm sure Apple's DB can safely toss out any data even remotely associated with identifying a particular phone. In the end they have a huge map made up of geo-locations for hotspots and cell towers which they're storing, and not having to pay Skyhook for.

I don't have a problem with that. It's the same story for Skyhook and Google. Though I wouldn't be surprised to find out that Google's data includes some identifying bits. Of large companies, I don't trust any less than Google at this point.

I see this as fair, and the process will help educate people a bit more about the issues modern phones raise. And I do think companies, including Apple, need to ensure proper reviews occur with any systems associated with handling location data. This is very similar to the privacy outcry over Buzz, with lots of misinformation, some real information, and good long term changes to ensure a slip-up doesn't occur again.

The iPhone does have an opt-out switch already, (the Location Services setting on the front page of Settings) and I could see this flipping to opt-in at the system level, similar to how all apps face an opt-in by default.

It makes no sense at all -- or at best is a misleading half-truth. The information they're talking about ("all the cell towers in Toronto") is sent from Apple to Iphone. Why is that particular slice of the global database sent? Because the Iphone has sent to Apple a request saying, "send me all the cell towers near <this specific location>". Apple still gets told your whereabouts.

Peter

Exactly. For the life of me I can't understand why so many companies don't get that it's better to lay all your cards on the table when this stuff happens than to hem and haw and obfuscate. If El Steve-o had just come out right away and said "we're collecting stuff, it's not very detailed, we'll put in a mechanism to turn it off without losing any location-aware functionality," this could have blown over before senators started getting involved. See also, Sony on the PSN outage, TEPCO on the scale of the containment problems at Fukushima, etc.

I guess the nature of cover-ups is that we don't hear so much about the successful ones, but it seems to me the risk of letting these questions linger does more harm than any good that comes from hiding the truth.