× {{alert.msg}} Never ask again
Receive New Tutorials
GET IT FREE

6 Useful Python Libraries Recommended by #1 Stack Overflow Answerer

– {{showDate(postTime)}}
Martijn Pieters, Stack Overflow Python Legend

Codementor Python expert and Stack Overflow legend Martijn Pieters sat down with us during Office Hours and shared some of the less-well known libraries that he likes to use for web development or daily tasks.

The text below is a summary done by the Codementor team and may vary from the original video and if you see any issues, please let us know!


 

Requests: HTTP for Humans

I am a huge fan of the requests library. If you have to do anything with http requests to other servers, this library goes straight past your urllib that comes with Python, and install requests. It is a fantastic tool and really has a much more intuitive API, which makes handling web requests much easier.

Beautiful Soup

Another great tool for web scraping is Beautiful Soup. When you’re web scraping and looking at web pages to extract information from, you should check this tool out. Beautiful Soup is a great html parsing library that makes and sends off the html “soup”.

Robobrowser


I recently came across a new library that marries the aforementioned two tools—Robobrowser. It is based on the requests library and beautiful soup to replace an older tool called mechanize, which does similar things. Namely, the Robobrowser lets you simulate a browser that goes out to the web, fetches html pages, lets you fill in forms, and submits those. Therefore, if you have to crawl a web page that requires forms or follow links to get to the data you want to get to and extract, take a look at Robobrowser, which can be also found on GitHub. Though it’s a relatively young project, I like the way the API works and the tool it uses.

Autobahn


On the server side of web development, Autobahn has piqued my interest. This tool makes programming web sockets easy, which is very helpful to me as a web developer. It’s also interesting as it makes use of Python 3.4’s new library.

Flake 8 Lint


For daily development, I like to use flake8 lint, which is based on Flake8. This Sublime Text plugin combines Pep8, which is for the Python style guide testing, together with Pyflakes, which detects common errors in your code (e.g. imports you forgot to add or imports you have too many of). Flake8 makes writing code much easier for me, and it integrates quite nicely with modern IDEs. I use sublime text and I have a plugin that puts flake8 error messages right at my fingertips. This is not a new project, but it is certainly something I use every day.

Scrapy

I haven’t actually come across a project where I use Scrapy, but to my knowledge Scrapy is excellent at extracting patterns for specific data. For example, if you want to scrape the latest football scores of Britain’s major leagues, you’d have to go to multiple sites and pages to extract the same information. Scrapy is good for such a task, as it can automate searching and walk through a site to extract very structured data you can specify through expressions. I’d love to have a project for me to use Scrapy that but I haven’t had the chance yet.


Other posts in this series with Martijn Pieters:

Martijn Pieters

Need Martijn’s help? Book a 1-on-1 session!

or join us as an expert mentor!



Author
Martijn Pieters
Martijn Pieters
5.0
I'm no longer available for mentoring, sorry!
Hire the Author

Questions about this tutorial?  Get Live 1:1 help from Python experts!
Martin Foley
Martin Foley
5.0
CTO at Draft AI | #1 CodeMentor for Python, JS, Node, React for now - still working on the others ;)
I'm a Scotsman living in London. I have over 10 years experience coding. I have a first in Computer Science, worked at one of the top cyber...
Hire this Expert
Harshul jain
Harshul jain
5.0
Expert Python developer with 3 yrs of experience
You give me anything, I will try my best to reach to the solution. > I am a Python tutor, and active open source contributor and a...
Hire this Expert

Or Become a Codementor!

Live 1:1 help from expert developers

Codementor is your live 1:1 expert mentor helping you in real time.

comments powered by Disqus
Codementor is your live 1:1 expert helping you in real time