This post is my notes on how to develop python projects easily and thoughts on python project environment management. If anything better and newest, please let me know in comments!

Why we need a better environment to do development?

For every project, we need to build up a project environment to manage the packages of a project, building process, and how to test it. With a fantastic environment, developer can get benefit with following advantages:

Advantage 1. Speed Up Development

• with a good environment, developers can quickly build/develop/test/deploy code on any machine
• quick deployment is very important on CI/CD
  • easy to release
  • agile testing

Advantage 2. Control dependencies of projects

• different projects will not share same packages they don’t use
• avoid dependency conflict between different projects
  • For example, if package A uses a package with latest version and another one uses older version. We can develop and manage the dependency by project independently and avoid hidden conflicts between two versions.
• keep projects smallest size

Though good environment makes developers happy, building a good development environment needs huge effort and challenging. Here we introduce some common practices to build up python development environment.

How to manage your python environment?

Tip 1. Manage Python Version

Why

Python has different versions (python 2.6, python 2.7, python 3.x … ). For each project, you need to decide which version in your project or you need to develop an old project which build on python 2.6. Quickly changing your environment can help you know which version of python you are using!

How

• pyenv: https://github.com/pyenv/pyenv
• Like rbenv in ruby if you are a ruby user XD
• commands:
  • pyenv versions: list all python versions in pyenv
  • pyenv global {python_version}: setup global environment of python.
  • Provide support for per-project Python versions.

Tip 2. Separate environment of each project with virtual environment

Why

To develop a project, we need to build up the packages structures that this project uses. But if we use builtin python, we cannot separate requirement packages by project. Think about the case: now we use pyenv and choose python 3.6 as project python version and have 2 projects to develop, how to manage the package dependencies independently? To solve this case, virtualenv provides a way to build up a empty python environment of a project and you can check in any environment to do related development easily.

How

• project github: https://github.com/pypa/virtualenv
• commands:
  • virtualenv ENV: start a new python virtual environment in ENV directory
  • source ENV/bin/activate: start working in ENV python environment
  • deactivate: checkout current environment

Tip 3. Manage package dependencies with pip

Why

Every language has similar package management tools to help developers manage their projects easily (like gem for ruby, npm for javascript, maven / sbt for java …). A package management tool can manage the dependencies and solve of projects, for example:

• flask dependencies

    'Werkzeug>=0.14',
    'Jinja2>=2.10',
    'itsdangerous>=0.24',
    'click>=5.1',

and these 4 packages also have their requirement packages, a package management tool will to expand the requirement packages recursively and install it one by one. If there is version conflict (like A package needs C package with version >= 3.0 and B package needs C package with version < 3.0), it should have alert to notify users to fix the conflict.

How

• https://github.com/pypa/pip
• commands
  • pip install {package_name} : install a package
  • pip freeze: list all packages and versions in current environment
  • pip install -r {requirement_path}: install all packages in requirement_path
  • pip uninstall {package_name}: uninstall a package

Tip 4. Latest tool to build python development tool: pipenv

Why

To make experience better, pipenv can let user manage packages and environment more easily. In general, this package is like virtualenv + pip.
Also, this package also solves some problems if developers use virtualenv + pip. As official documents mentioned, the problems that pipenv wants to solve is:

The problems that Pipenv seeks to solve are multi-faceted:

• You no longer need to use pip and virtualenv separately. They work together.
• Managing a requirements.txt file can be problematic, so Pipenv uses Pipfile and Pipfile.lock to separate abstract dependency declarations from the last tested combination.
• Hashes are used everywhere, always. Security. Automatically expose security vulnerabilities.
• Strongly encourage the use of the latest versions of dependencies to minimize security risks arising from outdated components.
• Give you insight into your dependency graph (e.g. $ pipenv graph).
• Streamline development workflow by loading .env files.

How

• https://docs.pipenv.org/
• commands
  • pipenv install: install from Pipfile
  • pipenv install {package}: install a package
  • pipenv graph: list project dependency in graph
  • pipenv shell: enter projects python environment

The flow practice to start a new python project

By compositing above tools with simple commands, developers can easily establish python environment for their projects. Following are the simple summary of the steps:

1. Choose python version
2. Create package environment

• using virtualenv + pip
• using pipenv directly

3. Start writing code!

But yes, there are many advanced commands and usages about the packages. Still lots of things to learn …