To the Moon. Microsoft and Open Data.
It has been an interesting week, and Microsoft deserves the credit. As an open-source developer, Microsoft and me have throughout the last 10 years gone together like oil and water, rarely getting along, but sometimes making an interesting salad dressing.
The latest instalment begins in 2008, when I attended Return to the Moon, an event hosted by BCIT ( http://www.bcit.ca/returntothemoon ) which introduced me to the concept of lunar mining. Yes, you read that right, lunar mining. As crazy as it sounds, there are some well-funded and exceptionally bright individuals advocating mining the moon for HE3, some have even visited the moon and have a story to tell. There are videos of the presentations at the previous link and you can be prepared to learn some really cool things about nuclear fusion from those talks.
Being a GeoGeek from a mining family and knowing what it takes to pull off mining here on terra firma, I naturally asked about the GIS implications and the answers weren't encouraging. If you're not a research scientist working with NASA, your mapping abilities are pretty much limited to simple visualization of the lunar surface. To my knowledge there are no GIS-based systems in place for ore-body classification, site feasibility study, tools which have decades of research here on earth but that wont work out of the box on the Moon... If you know anything about mining, turns out, getting equipment to the moon might be the easy part. Without the right GIS tools, lunar mining would end up looking more like an episode of Gold Rush Alaska in space. I'm picturing astronauts bouncing around looking for glory holes and pay streaks -- hilarious, but not exactly a path towards true energy sustainability on earth.
So where does Microsoft fit in? March 24, 2009 rolls around (interestingly today is their two year anniversary) and NASA and Microsoft Research announce they've solved the planetary mapping problem and release www.worldwidetelescope.org in an "Open Goverment"/P3P type partnership. The fine print is that the data for Microsoft's 'telescope' is in a format never used before -- its got some serious polar benefits and I'm not knocking the technology, but they used an entirely new projection called TOAST to represent the data, and none of the standard mining GIS tools can even think about working in TOAST. They need LCC or Mercator type data, or pretty much anything that works in a standard spatialreference.org compatible projection. So I begin the process of requesting the data figuring if I have the raw data, somehow I can get it converted to what we need to use, and start setting up some real mining tools for the moon.
To make a long story short, here's where 2 years of frustration sets in. My attempts to contact NASA for the data just got me a pointer to world-wide-telescope, my attempts to contact MS Research got me a pointer to their SDK and their Terms of Service. Don't like it, talk to NASA. Round in circles I go, and my prospects for convertible data are looking slimmer and slimmer. So I simmer the issue, and go on about my other 50 public interest projects.
Enter OpenGovWest. Microsoft sends out their Open Data folks and a fellow by the twitter name of @Nik_G. So I send him a tweet -- saying something to the effect of 'this ms cloud stuff sounds great... but... where's this wwt data'. He replies, but again a pointer to the SDK and it doesn't go anywhere. Fast forward a year, and I'm following Nik on twitter still, he's espousing the benefits of the MS cloud for open data, and I'm now getting totally frustrated.
From my perspective, MS has got NASA to release this data, through them, and its a closed ecosystem. In my view the data has been captured, and my prospects for lunar mining tools are looking about as likely as me ever visiting the place. So I call em out on it. If you were following my twitter two days ago, well, it wasn't pretty and I'm not proud of it. Mea Culpa. That said, Nik_G while obviously pissed at me, decided rather than fighting or trying to be defensive that he'd get to the bottom of why this data isn't really, truly, open. Some footwork with the MS directory and he puts me in touch with MS Research and the folks that actually worked on the project. Now we're getting somewhere -- almost.
Initial response is again a pointer to the SDK and to the URL structure for TOAST tiles. Whoop-ie. I'm faced again with the prospect of trying to scrape TOAST from the SDK and convert it in bulk to mercator. Ya. Right. Even if this were a valid way to transfer geographic data, its going to take months of scraping and its definitely against the SDK ToS.
It turns out that MS contends it doesn't posses the data themselves, rather that their TOAST projection is offered from a server in NASA's datacenter called wwt.nasa.gov. So now the pointers going back to NASA. I've run up that flagpole before, its a pointer to WWT, and rinse repeat. Stuck in the cycle again.
But wait. I'm not. Thanks to @Nik_G, he suggest that MS Research might be willing to share their NASA contacts who actually know what's going on with this data. Maybe this time I wont just get a pointer back to WWT. So I send an email, expecting to get laughed at, and sent back to talk to NASA's front door. But, and here's where the "MS is Evil" branding falls apart -- not only do I not get laughed at, but they share their contact and provide some actually useful information about their arrangement with NASA.
I immediately fire off an email to the NASA contact provided and viola, I'm now on the path to finding real, usable data. Instantly I'm linked to actually compatible WMS servers, http://onmoon.jpl.nasa.gov/ and as a bigger win, I'm told the same process that created the TOAST representation for the high resolution data can also be used to create "basically any combination of projection and image formats supported by proj4 and GDAL." ... score!
The process at this point is ongoing, I need to get back to NASA and chase the data down, but its now at least encouraging and at the beginning of this process I promised Nik that I would blog my experience. And there you have it, mea culpa, Microsoft's plans for open data no longer seem conspiratorial or concerning to me. That said, the take away of course is that MS has great people like Nik working for them but that they also have terrible accessibility -- it took me 2 years, building frustration, criticizing the organization, and annoying a well-intentioned employee to finally break through the response network loop. That's definitely the wrong way to go about this workflow, but I understand how it can happen.
The positive note is there are folks like Nik that seem to genuinely want to advance open data, and if MS can continue to build a strong community presence, it might just be able to repair its reputation as it relates to competition with open source. But then again, registering the Firefox keyword to promote IE9 doesn't help.
Submitted for $0.02.