Your personal knowledge base with JupyterLab

Long long time ago I was using Evernote as a my personal knowledge base. I was
collecting snippets of knowledge from my projects I was working on. One of the
constant challenges was ensuring that I do not paste accidentally sensitive data
as a part of the snippets. From other hand I was using only part of Evernote
offering — mostly tree like notes organization and full text search to filter
cards.

Since few years now I am having pet projects with extensive use of Jupyter
notebooks. One day I asked myself — why not use Jupyter notebook as a fully
owned personal knowledge base? One day it transformed into reality.

Nowadays I have jupyterlab environment integrated into my dotfiles:
https://github.com/Voronenko/dotfiles/tree/master/notebook
— on each of my working places I have virtual environment with JupyterLab up,
and I can easily invoke it with shell shortcut

alias znotes=’workon project_notes && cd ${ZNOTES_PATH-~/z_personal_notes} &&
jupyter lab’

I was missing feature to quickly filter notebook when I know it name — and
that’s successfully solved by jupyterlab extension
https://github.com/parente/jupyterlab-quickopen

Another missing feature was support for templates — my notes usually follow
specific pattern for easier search and analysis. This also can be resolved with
custom plugin, this time
https://github.com/timkpaine/jupyterlab_templates

One of the noticeable bonuses is possibility to embed diagrams related to my
software projects as well as interactive code snippets.

The only missing feature now is full text search. I am mostly console guy, thus
so far I am using workaround with nbgrep:

One batch searches for entries in code cells

#!/bin/bash

# usage: nbgrepmd 'pattern'
# Searches for entries of type markdown

SEARCHPATH=${ZNOTES_PATH-~/z_personal_notes}
SEARCHFILTER=".cells[]? | select(.cell_type==\"markdown\") | .source[]?//.input"

# Colours
readonly C_NOC="\\033[0m"    # No colour
readonly C_RED="\\033[0;31m" # Red
readonly C_GRN="\\033[0;32m" # Green
readonly C_BLU="\\033[0;34m" # Blue
readonly C_PUR="\\033[0;35m" # Purple
readonly C_CYA="\\033[0;36m" # Cyan
readonly C_WHI="\\033[0;37m" # White
### Helper functions
print_red () { local i; for i in "$@"; do echo -e "${C_RED}${i}${C_NOC}"; done }
print_grn () { local i; for i in "$@"; do echo -e "${C_GRN}${i}${C_NOC}"; done }
print_blu () { local i; for i in "$@"; do echo -e "${C_BLU}${i}${C_NOC}"; done }
print_pur () { local i; for i in "$@"; do echo -e "${C_PUR}${i}${C_NOC}"; done }
print_cya () { local i; for i in "$@"; do echo -e "${C_CYA}${i}${C_NOC}"; done }
print_whi () { local i; for i in "$@"; do echo -e "${C_WHI}${i}${C_NOC}"; done }

# 'jq' technique lifted with gratitude 
# from https://gist.github.com/mlgill/5c55253a3bc84a96addf

# Break on newlines instead of any whitespace
# IPython Notebook files often have spaces in it
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

if ! type mdfind > /dev/null 2>&1; then
    # Use find from findutils
    echo find $SEARCHPATH -name '*.ipynb'
    FILES=$(find $SEARCHPATH -name '*.ipynb')
else
    # mdfind uses OSX's spotlight search, so it's almost instant
    # generate a list of all the ipynb files in any of the directories
    FILES=$(mdfind -onlyin $SEARCHPATH -name '.ipynb')
fi

# On the command line we get the argument to search for
PATTERN=$1

for f in $FILES
do
    # Use 'jq' to filter out only the code in input cells
    # Then remove quoting
    # Colorize it with pygments (give it the most context possible to get color right)
    # And finally, search the remainder for a given pattern

    OUTPUT=$(jq $SEARCHFILTER $f \
        | sed 's/^"//g;s/"$//g;s/\\n$//g;s/\\"/"/g;s/\\\\/\\/g;s/\\n/\n/g' \
        | pygmentize -l python 2>/dev/null \
        | grep $PATTERN)


    # If the grep matched anything, print it
    if [ $? -eq 0 ]; then
        echo -e "${C_CYA}$f${C_NOC}:\n\n$OUTPUT\n\n"
    fi
done

IFS=$SAVEIFS

and another shell script watches for entries in markdown cells

#!/bin/bash

# usage: nbgrepmd 'pattern'
# Searches for entries of type markdown

SEARCHPATH=${ZNOTES_PATH-~/z_personal_notes}
SEARCHFILTER=".cells[]? | select(.cell_type==\"markdown\") | .source[]?//.input"

# Colours
readonly C_NOC="\\033[0m"    # No colour
readonly C_RED="\\033[0;31m" # Red
readonly C_GRN="\\033[0;32m" # Green
readonly C_BLU="\\033[0;34m" # Blue
readonly C_PUR="\\033[0;35m" # Purple
readonly C_CYA="\\033[0;36m" # Cyan
readonly C_WHI="\\033[0;37m" # White
### Helper functions
print_red () { local i; for i in "$@"; do echo -e "${C_RED}${i}${C_NOC}"; done }
print_grn () { local i; for i in "$@"; do echo -e "${C_GRN}${i}${C_NOC}"; done }
print_blu () { local i; for i in "$@"; do echo -e "${C_BLU}${i}${C_NOC}"; done }
print_pur () { local i; for i in "$@"; do echo -e "${C_PUR}${i}${C_NOC}"; done }
print_cya () { local i; for i in "$@"; do echo -e "${C_CYA}${i}${C_NOC}"; done }
print_whi () { local i; for i in "$@"; do echo -e "${C_WHI}${i}${C_NOC}"; done }

# 'jq' technique lifted with gratitude 
# from https://gist.github.com/mlgill/5c55253a3bc84a96addf

# Break on newlines instead of any whitespace
# IPython Notebook files often have spaces in it
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

if ! type mdfind > /dev/null 2>&1; then
    # Use find from findutils
    echo find $SEARCHPATH -name '*.ipynb'
    FILES=$(find $SEARCHPATH -name '*.ipynb')
else
    # mdfind uses OSX's spotlight search, so it's almost instant
    # generate a list of all the ipynb files in any of the directories
    FILES=$(mdfind -onlyin $SEARCHPATH -name '.ipynb')
fi

# On the command line we get the argument to search for
PATTERN=$1

for f in $FILES
do
    # Use 'jq' to filter out only the code in input cells
    # Then remove quoting
    # Colorize it with pygments (give it the most context possible to get color right)
    # And finally, search the remainder for a given pattern

    OUTPUT=$(jq $SEARCHFILTER $f \
        | sed 's/^"//g;s/"$//g;s/\\n$//g;s/\\"/"/g;s/\\\\/\\/g;s/\\n/\n/g' \
        | pygmentize -l python 2>/dev/null \
        | grep $PATTERN)


    # If the grep matched anything, print it
    if [ $? -eq 0 ]; then
        echo -e "${C_CYA}$f${C_NOC}:\n\n$OUTPUT\n\n"
    fi
done

IFS=$SAVEIFS

If you have better option to run full text search over set of JupyterLab
notebooks — please comment under the article — I would be very grateful.

And, finally, the last opportunity: KeyBase. If you make use of
https://keybase.io/ — you can put your knowledge base on
a mounted keybase folder (either git baked or just file based) — in that way
your knowledge base becomes fully portable.

In current setup — it covers my workflow for project notes, but I constantly
looking for additional plugins that could improve usage experience.