How and why I built a scraper for 3 local public bike systems

About me

Mathematician with python expertise

The problem I wanted to solve

Creating the best map covering all three public bike systems in Santiago, Chile

What is A scraper for 3 local public bike systems?

Two station-based systems were queried, covering half of the city. Then, based on these and other locations, I hit the Mobike API to complete the task. 5000 bikes are stored in my database every time I run this code. This allows me to study biking patterns in my city.

Tech stack

Python, Flask, requests, pandas and folium.

The process of building A scraper for 3 local public bike systems

I first discovered the URL feeding the BiciLasCondes map with the help of a JavaScript friend. Using pandas+requests, this allowed me to retrieve the current info for all 89 stations with one line of python: blc = pd.read_json('https://www.bicilascondes.cl/availability_map/getJsonObject'). For every station, this gives me location and the number of available bikes and slots. Then for BikeSantiago, which has 175 stations, the following call does the same trick: js = requests.get('https://api.citybik.es/v2/networks/santiago').json(). The difference here is that the system url does NOT get updated, so we can only use the locations.

Finally, for Mobike, the system API only gives us the location and IDs of the nearest bikes from a given location. So I decided to run this call from a sample of the above 264 stations belonging to the parts of the city where we know Mobike is operational. This gave me 5000 stations to study. To visualize this, folium as the interface to Leaflet.js saved me a lot of headaches and allowed me to create this: http://quant.cl/static/BIKES/3sistemas.html

Challenges I faced

Using curl from python was not trivial, but it can be done.

Key learnings

Pandas makes development a breeze, however if I wanted to run this every minute, I would have to ditch pandas and work directly with the JSON data provided by every interface.

Tips and advice

Try harder! Stack overflow is your friend.

Final thoughts and next steps

I want to add cyclepaths to my map, as well as bike repair shops, in order to build the ultimate biker's map, and package the service in Amazon Lambda, to be eventually released as an app.