Deploy Private Github Python Packages on Heroku without Exposing Credentials in Code

Published Nov 07, 2018Last updated May 05, 2019

Recently, we met a deployment problem in heroku python environment.

In heroku python deployment, it will execute pip install requirements.txt and install packages in the file. But when you have a private package, everything goes complicated.

Goal

What we want to do is to install our private package which is on the github. Just making one of the following commands success:

pip install git+https://github.com/my_account/myrepo.git
pip install git+ssh://github.com/my_account/myrepo.git
pip install git+https://{username}:{password}@github.com/my_account/myrepo.git

The complicated thing here is that you don't want to commit any credential or password in git because it causes to many security issues. So how to let heroku know the credential in build is the key-point of this problem.

Third-Party Buildpack Solutions

In fact, there are some third-party buildpacks supporting setting github tokens or ssh-keys in the environments:

But after our discussion, we decide not to use third-party buildpacks because the following reasons

maintenance: everyone who joins related projects needs to know the buildpack and maintain them if anything changes.
security: we don't know what things happen if we do not review the code.

(I've reviewed the code, both of them are pretty simple and pretty useful. You can config your credentials in environment variables and the buildpacks will set for your purpose.)

Solution without Using Third-Party Buildpacks

With some survey on the heroku buildpacks, we found a good solution that helps us solve this problem. To learn the solution, we need to know more about how heroku python buildpack works first. Herkuo python buildpack will execute the following commands in order:

bin/detect
bin/pre_compile
bin/compile: in this step, it will execute pip install -r requirments.txt here.
bin/post_compile
bin/release

With the knowledge, we tried the steps:

Setup environment variables GITHUB_USER and GITHUB_PASSWORD
Put the file bin/pre_compile with link generation code like:

# !/bin/bash

echo "Generate myrepo to requirements.txt"
MY_REPO_GIT="git+https://${GITHUB_USER}:${GITHUB_PASSWORD}@github.com/my_account/myrepo.git"

echo MY_REPO_GIT >> requirements.txt

and it will generate private repos link with user information to requirements.txt before executing bin/compile. Finally it can install all things your want with this solution.

Failed Try

We've tried to use setup.py to do customization in installation but it didn't work because heroku team thinks setup.py can do everything and it's unsafe if exposing all build environment settings on it.

Thoughts

It's a pretty interesting journey when knowing how the buildpack works and figuring out the code for me haha. But pre_compile seems a deprecated feature in heroku. Not sure when it will be removed.