How to Display Big Data On A Google Map?

How to display 300+ GPX cycling routes over a Google Maps base layer

  • GPX Visualization on Google Maps: I have 300+ GPX files of bike rides that I'd like to be able to link to somehow over a Google Maps base layer, visually approximating http://i.imgur.com/hkeSPwK.png. This is related to my http://ask.metafilter.com/242816/Endgame-tools-for-a-cyclist-trying-to-bike-down-every-street-in-a-city about cycling every road in a town; I need to visualize what roads I have/haven't ridden, and link that progress to a blog. Would like it to be as low-weight as possible; the 300 (uncompressed) GPX files amount to 100+mb currently (route+track+waypoints) and is only getting larger -- vanilla GMaps doesn't play nice with that many files. Much more information inside... [If there's a smarter way to approach any of this, I'm all ears. I have very little idea what it is I'm doing here, and if you can correct any wrong assumptions/bad concepts on my part you'd be doing me a huge kindness; so fire away.] So there used to be a site ('http://www.jonathanokeeffe.com/strava/multi-ride-mapper/') that worked with the Strava API that would display all of a user's rides. It was made by a guy named Jonathan O'Keefe, and it would produce a map that http://i.imgur.com/hkeSPwK.png (while retaining the full GMaps functionality). It was low-weight and loaded quickly, and pulled the data off of the Strava API. Here's the http://pastebin.com/WmyKbNst for the site in question But Strava changed (and then re-changed) its API. Now I can't link to a map of my rides to someone who isn't a Strava member and signed in. This sucks, and I'd like to recreate that functionality. SO, I have the GPX files (and, also, the Garmin FIT files, if it makes a difference). They're heavy; in total the 300 rides in GPX are around 100mb. They're hosted on my server; here's a link to the http://www.bigmapblog.com/misc/strava/GPXs/00_GPXs_2013-11-01.zip (5mb, ZIP file) if it's of any use. I'm a babe in the woods, and I'm not sure what I need to do to do this; but what I think I'd like done approximates:Pointing google maps to a sub-directory on my website where all the GPXs live (eg .../strava/gpxs/)Not sure what happens this step; possibly magicRide overlays of, say, 2px in an opaque color are overlayed on a Google Map base layer (visually approximating http://i.imgur.com/hkeSPwK.png)Does this require use of the Google Maps API? What's the most hand-holdingest way to walk me through setting such a thing up? I've researched this for several hours now and actually feel dumber about this than when I started. Other stray questions (I'm sure I'll have more):How do I (and: 'do I need to') reduce the size of the (100+mb, total) GPX rides, and what data can be stripped while preserving the visualization? How is this best done?I know I don't need 'heart rate' or 'cadence' info, but I'm not sure if I can strip one or more of the following: route, track, waypoints. Not sure which would be most useful to strip, size-wise, or indeed how this would even be done.How would I create a 0.4km "halo" of anonymity around my houseAgain, J. O'Keefe's (amazing) Multiple Ride Mapper used to do this great. Through no fault of his, the Strava API neutered the functionality -- I can't link to non-strava members. His site gets/got the data from the Strava API. Looking to cut Strava out of the loop here, and use my data how I'd like. I'm very sure I don't know enough about what I'm asking to have included all the pertinent information, and I welcome the chance to clarify anything I've left unclear. Many, many thanks for your time and your expertise.

  • Answer:

    I'm so glad you asked this question, I had wondered about doing this and you were my push to do it :) I wrote up how I did it using http://www.gpsbabel.org/ and leaflet, both mentioned above. You can see my notes and a leaflet map http://dl.dropboxusercontent.com/u/428425/map/map.html. I didn't use Google Maps. Leaflet uses OpenStreetMap. I only included three runs in my map. I did load all 144 runs I did in 2013 into one GPX file. It took several seconds to load on my beefy computer. I think GPSBabel and maybe the GPX plugin for leaflet have the option to simplify map files, but that's beyond what I had time to figure out. I also think you could put each activity onto a layer so you could pick and choose what to show. But I didn't really have time to show that either.

jjjjjjjijjjjjjj at Ask.Metafilter.Com Visit the source

Was this solution helpful to you?

Other answers

To your 0.4km "halo" of anonymity around your house, you need http://www.gpsbabel.org/. As per https://wiki.openstreetmap.org/wiki/Using_filters_with_GPSBabel#Exclude_points_within_given_radius_.28ie._privacy_issue.29 instructions, you could do (assuming you live at 38.61,-90.25): gpsbabel -i gpx -f infile.gpx -x radius,distance=400m,lat=38.61,lon=-90.25,nosort,exclude -o gpx -F outfile.gpx You might wish to add a few random exclusion points near your house. Your tracks have enough waypoints that it would be easy to back-calculate where you live. Leaving only tracks with time and elevation, I can get the files down to ⅓ of the original size. If you wanted smaller, there are ways of filtering and smoothing tracks to reduce points in gpsbabel, but these might lose some of the detail you needed to capture every turn in every road. For web display, if tmcw shows up in this thread: what he said, by way of pre-emptive seconding ☺

scruss

Experiment 1: Google Maps 1. Go to Strava activity, export as GPX 2. http://www.gpsvisualizer.com/map_input?form=googleearth via gpsvisualizer.com 4. Right-click on the output kmz file "Copy Link Location". 5. Open Google Maps. Paste the kmz file link in google maps search box. Presumably the GPSVisualizer link won't live forever, so you need to move your KMZ files to some hosted server eventually. I can't find a way to display multiple kmz files on google maps, but a kmz file can contain multiple tracks. See https://maps.google.com/maps?q=http://kml.lover.googlepages.com/leTourDeFrance2005.kmz. Here are my kmz output files (these links probably won't stay working long): http://www.gpsvisualizer.com/display/20131202124837-21774-map.kmz http://www.gpsvisualizer.com/display/20131202130515-23128-map.kmz View them in google maps (probably won't stay working long): - https://maps.google.com/maps?q=http://www.gpsvisualizer.com/display/20131202124837-21774-map.kmz - https://maps.google.com/maps?q=http://www.gpsvisualizer.com/display/20131202130515-23128-map.kmz Experiment 2: Google Earth 1. Download google earth 2. Translate GPX to KMZ as above 3. Save the output kmz files to your computer. 3. Load one or more kmz files into Google earth This option is nice if you want to have all your data off-line. Google earth will display multiple kmz files (I tried it with my two files above). There are probably many ways to translate from GPX to KMZ. I used GPSVisualizer.com because it was convenient. Could probably download a utility that will do them in bulk if you want...

sarah_pdx

And if you're at all OK with using command line tools: https://github.com/mapbox/togeojson will convert the gpxes to geojson. then https://github.com/mapbox/geojsonio-cli will upload it to geojson.io http://geojson.io/#id=gist:anonymous/9feca8555bf77659e408&map=14/38.5870/-90.3299 It will look something like this: npm install -g togeojson geojsonio && togeojson 12_09_2012.gpx | geojsonio (You need to have something called node.js installed for either of those to work.)

kjell

Does the author of the multi-map thing know it's broken? Seems like something that he could fix on his end. Strava itself has a "heat map" thing that miiight be suitable. Just takes a second, try it out on their site.

RustyBrooks

> Does the author of the multi-map thing know it's broken? The problem is is that it's not really 'broken' (at least in any fixible way); Strava API v2 and v3 has removed public access to that data. The author knows about the changes (and has worked several times to restore what functionality he can), but it's taken away, now, and the access to the data required to make it work like it did previously is gone. A heel-turn quasi-shady move on Strava's part, to be sure; but they've thrown their lots in with denying their users access to their created data. The backlash was strong, but died down quickly. It ain't coming back.> Strava itself has a "heat map" thing that miiight be suitable.Yeah, I was real excited for a second when they rolled that out about 3 months ago, but it's a total flop. It http://i.imgur.com/TM2Rg3L.png, and looks worse once you drill down to whatever http://i.imgur.com/5iSZ7xk.png they've capped it at. I can't tell much from that about which roads I've been on. In summary, I guess it does passably well on the "heat" part, and terribly bad on the "map" part.

jjjjjjjijjjjjjj

I understand the changes to the API. But this guy doesn't have to approach it as "a member of the public". He could have an account for his app and log in with it. I've done that kind of thing in lots of different web sites that added similar restrictions. Sorry the heat map sucks. I'd never really checked it out. I took one shot at making something with the google maps API, got kinda frustrated by it, and quit. Wish I knew an app that would work for you but I can't really think of one.

RustyBrooks

> But this guy doesn't have to approach it as "a member of the public". He could have an account for his app and log in with it. I'm trying to display my data to the public, though. I have the GPX data and want to display it over a (GMaps or OpenStreetmap) basemap while placing no restrictions on who can see it. I appreciate the suggestions and the effort, though; and I sympathize with your frustrations at using the gmaps API... it was a fish-out-of-water thing for sure. I feel a bit better knowing that it wasn't just me, so thanks for that.

jjjjjjjijjjjjjj

I would pose this question to the guy who built VeloViewer.com because I remember him writing some stuff on his blog about possibly building a heatmap visualization. I've kicked some emails around with him and he is very cool and enthusiastic about this stuff.

dgran

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.