In 2018, I worked on (in collaboration with the talented Rae Valters (@raevalters) and ultimately abandoned a project called Oinki, which would have been a vegan iPhone and android app, focused on finding the coolest vegan stuff in London. It was structured as a map app with vegan restaurants and stores . Isn't that exactly like HappyCow? Yes, but it would have been London specific! It made sense to me at the time (and, in some ways continues to make sense to me).
In creating the app, one of the issues that we had was how we would create the initial list of places - was there some way I could scrape from other services to populate names and phone numbers? I turned to the Google Maps API to see if I could get a list of places from them, but found I was limited to 60 places for any API call. I didn't make it further.
This year in March, my sister Maria asked me if I could get a list of resturaunts for her for a coordination effort she was working on for coronavirus in her area. I said I could help, and turned again to the Google Maps API as my solution. The api can return you a list of places (up to 60) surrounding a latitude / longitude coordinate, up to 60 places. An example call in node looks like this:
Which returns son JSON that gives you information on ratings, phone numbers and other information.
The response from the api gives me all the information I need, but I need to figure out how to actually get all of the places within a given target area. To make the following discussion more clear, let's define some terms.
Let's consider the ideal solution, a circumscribed target area. This would work if our call to the api returned all of the results for the area. Alas, it does not. Instead, we can implement the method that searches use in the real world - we can divide up our target area with a grid, and search the smaller grid target areas.
My inspiration for using a search grid came from my childhood obsession with the Titanic. The Titanic was a White Star Line passenger ship that sank in the North Atlantic in April, 1912. My birthday is the same day as the Titanic disaster (April 14th), and I always felt a connection to the event. When Robert Ballard discovered the Titanic in 1985, he used two different remotely operated vehichles to search his target area for the wreck of the Titanic, SAR and Argo. The "mowing the lawn" pattern (sweeping back and forth the target area) that the searchers employed sparked the idea in me to do something similar two decades later.
I wrote an algorithm to split the target area into a grid of coordinates on which I can center searches, and by covering the grid with search circles, I can safely cover the whole of the original target area.
Fantastic! We have a nice solution. But what if a API call still returns 60 results?
It's simple, as it turns out. If we get back more than 60 results, we can just create a new target area, and divide it into it's own grid. We can then repeat this as necessary until all searches return less than 60 results.
So, finally, I was able get the api to return all of the places in my target area. I cleaned the data, turned it into a CSV, and sent it to my sister, who now has a list of all of the resturaunts in her area.
This was a fun project for me. It was nice to approach a problem that was slightly outside of my normal constraints, although it was a bit of an artificial one. If you want to check out the code that came out of the project, you can see it at the below link:
I named the project argo in homage to the video sled that discovered the Titanic, also name Argo.
I would be remiss to not mention that I did not come up with this method alone - it has been written about before, which I discovered in the course of writing this article. There are, I'm sure, many more, but these were two that I found immediately.
Ilia, a datascientist in London, used the same method: https://iliauk.wordpress.com/2015/12/18/data-mining-google-places-cafe-nero-example/
This stack overflow answer points to the same solution as well: https://stackoverflow.com/questions/13070477/how-can-i-get-more-than-60-results-from-the-google-places-api