A Brand New Git Repo

Apr 28, 2019

I remembered recently that this blog has been sitting around idle for a while. I figured that the continuous deployment solution I’d set up (CodeShip + Amazon S3) might’ve died by now (well, the CodeShip part anyway) but no, it’s still around. Those folks seem to be going strong, much to my surprised and delight.

However, a lot has changed in the past two years. For one, Github now supports unlimited private repositories with a free account. I had originally hosted the code (I use Hugo) on BitBucket because back then they had support for private repos and Github didn’t. However, all of my other repos are on Github, and since Github now supports private repos for free too, I figured that I might as well move this thing over too.

This was slightly complicated by the fact that CodeShip doesn’t allow you to change the backend used for one of their pipelines. You can switch from one BitBucket repo to another, or from one Github repo to another, but not from BitBucket to Github or vice versa. So, I cloned the repo locally, created a new one on Github, added that as a remote, duplicated my pipeline and pushed, and it worked! Of course, I also cleaned up some of the backend mess I had made the first time around. For instance, I had just straight up given CodeShip access to my main AWS account’s Access and Secret keys. Now, I have a custom policy that only grants write access to the one bucket it needs, and CodeShip has the keys to an IAM role with solely that policy attached. Much cleaner (and much, much safer!).

Some things changed on the Hugo side of things too, such as one function being removed and another being on the deprecation path, fixed those nits too.

Install OpenCV 3.1.0 with Python 3.5+ in Ubuntu 14.04

Apr 19, 2016

Getting OpenCV 3 up and running with Python 3.5+ on Ubuntu 14.04 is a pain in the ass. For the most part the instructions here are the same as in this guide, with a few changes needed specifically for Python 3.5+. I’m going to assume that you already have pyenv and pyenv-virtualenv installed for the purpose of this guide.

Install prerequisites

sudo apt-get update
sudo apt-get install -y build-essential cmake git pkg-config
sudo apt-get install -y libjpeg8-dev libtiff4-dev libjasper-dev
sudo apt-get install -y libpng12-dev libavcodec-dev
sudo apt-get install -y libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install -y libgtk2.0-dev libatlas-base-dev gfortran

Install python headers

This is the slightly tricky part. We want OpenCV to work with Python 3.5+, but Ubuntu 14.04 only comes with Python 3.4.x. So, we’ll have to turn to an external repo:

sudo add-apt-repository ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install -y python3.5-dev

This will download and install the headers and libraries for the latest release of Python 3.5.x. This number will vary as this post gets older, and I don’t know of a way to check exactly which version is installed. However, for OpenCV to work with virtualenv, we’ll need to know the exact version of python3 headers that are installed. I’ll explain how to do this next.

Fetch opencv and opencv_contrib

The 3.1.0 in the commands below was the latest released version of OpenCV at the time of this post. Check OpenCV News for the latest release number when you run this, and substitute below if it’s different.

cd /tmp
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 3.1.0

cd ..
git clone https://github.com/Itseez/opencv_contrib.git
cd opencv_contrib
git checkout 3.1.0

Try (and fail) to build OpenCV

We first need to modify the OpenCV build configuration to use python3.5 instead of python3.4.

gedit /tmp/opencv/cmake/OpenCVDetectPython.cmake

In the file that opens, find a line that starts with find_python(3.4 and change it to find_python(3.5. Save the file and quit the editor.

At this point, OpenCV configuration will fail because we haven’t set up python and numpy yet. The only reason for the following step is to find out the version of python3 headers and libs that was installed.

cd /tmp/opencv
mkdir build && cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=/tmp/opencv_contrib/modules \
-D BUILD_EXAMPLES=ON ..

cmake will succeed, print a long list of configuration variables. Look for lines that look like this:

--   Python 3:
--     Interpreter:  /home/amey/.pyenv/shims/python3.5 (ver 3.5.1)

The (ver 3.5.1) is the part we’re interested in. That’s the python version we’ll have to use.

Create python environment and install numpy

pyenv virtualenv 3.5.1 cv
pyenv activate cv
pip install numpy

Build OpenCV for real

cd /tmp/opencv/build
rm -rf *
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=/tmp/opencv_contrib/modules \
-D BUILD_EXAMPLES=ON ..

If everything goes according to plan, the output will change and you’ll see something like:

-- Python 3:
--   Interpreter:    /home/amey/.pyenv/shims/python3.5 (ver 3.5.1)
--   Libraries:      /usr/lib/x86_64-linux-gnu/libpython3.5m.so (ver 3.5.1)
--   numpy:          /home/amey/.pyenv/versions/cv/lib/python3.5/site-packages/numpy/core/include (ver 1.11.0)
--   packages path:  lib/python3.5/site-packages

As you can see, cmake has picked up a lot more info this time around. If you still only see an Interpreter line and nothing else.. well, I’m sorry I guess.

Edit - Try this alternative command, sourced from the guide for anaconda:

cmake -DBUILD_TIFF=ON -DBUILD_opencv_java=OFF -DWITH_CUDA=OFF -DENABLE_AVX=ON -DWITH_OPENGL=ON -DWITH_OPENCL=ON -DWITH_IPP=ON -DWITH_TBB=ON -DWITH_EIGEN=ON -DWITH_V4L=ON -DWITH_VTK=OFF -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=$(python3 -c "import sys; print(sys.prefix)") -DPYTHON3_EXECUTABLE=$(which python3) -DPYTHON_INCLUDE_DIR=$(python3 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") -DPYTHON3_PACKAGES_PATH=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") ..

After this, build and install OpenCV:

make -j4
sudo make install

Optional - Create a symbolic link

(Edit - This is optional because if you built and installed OpenCV using the Anaconda command above, the library has been installed in the pyenv directory already.)

All that remains is to create a symlink to the OpenCV library that was (or should’ve been at least) installed to /usr/local/lib.

cd ~/.pyenv/versions/cv/lib/python3.5/site-packages
ln -s /usr/local/lib/python3.5/site-packages/cv2.cpython-35m-x86_64-linux-gnu.so cv2.so

If you get an error while running the ln command, check the /usr/local/lib/python3.5/site-packages/ directory. It’s possible that your library might have been built with a different name. If that directory is empty, well, sorry again.

Test

If nothing went wrong, you should now be able to test the OpenCV installation. Activate the virtualenv if it isn’t already active and try the following:

python3
>>> import cv2
>>> cv2.__version__
'3.1.0'
>>>

Of course, the '3.1.0' version number is what I installed, yours should reflect the version that you installed.

Setup Ubuntu 14.04 with Cuda and Lasagne

Apr 5, 2016

Installation of prerequisites

    # Install common stuff
    sudo apt-get install build-essential
    sudo apt-get install zsh git

    # Install SciPy dependencies
    sudo apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran

    # Install pyenv
    curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash

    # Install python 3.5.0
    env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.5.0

    # Install cuda
    wget http://developer.download.nvidia.com/compute/cuda/7.5/Prod/local_installers/cuda-repo-ubuntu1404-7-5-local_7.5-18_amd64.deb
    sudo dpkg -i cuda-repo-ubuntu1404-7-5-local_7.5-18_amd64.deb
    sudo apt-get update
    sudo apt-get install cuda
    popd

    # Create theanorc
    echo "[global]" > ~/.theanorc
    echo "floatX = float32"
    echo "device = gpu"

Create pyenv and install lasagne

Choose a name you see fit. I picked “lasagne”.

    pyenv virtualenv 3.5.0 lasagne
    pyenv activate lasagne
    pip3 install --upgrade https://github.com/Theano/Theano/archive/master.zip
    pip3 install --upgrade https://github.com/Lasagne/Lasagne/archive/master.zip

Wrap up

Reboot your system so that the installed nvidia drivers will load

Usage

Prior to using python, activate the environment with

    pyenv activate lasagne

Pyplot in Jupyter inside pyenv on El Capitan

Jan 25, 2016

With the new root protection feature in El Capitan, it’s hard to install pip packages into the system python. Besides, it’s generally considered bad practice to clutter up your system python’s library with packages. Most developers prefer to set up their own virtualenv for python and then install packages into that local environment. I did the same for setting up Jupyter Notebook and everything worked as expected. Well, mostly everything.

When I tried to import matplotlib.pyplot, I got a rather long error:

RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see ‘Working with Matplotlib in Virtual environments’ in the Matplotlib FAQ

So it seems that pyplot doesn’t work well with python inside a virtualenv, at least on OS X. The FAQ describes a way to work around this by adding a script to your virtualenv and then invoking that script as your interpreter instead of invoking python directly. That’s half the problem solved, the other half being getting IPython to use this script. A little googling provided a solution to that as well.

So here’s the combined solution, including how to create and use the virtualenv.

Install required packages

# Install pyenv

brew install pyenv

# A convenient way to create virtualenvs
brew install pyenv-virtualenv

# Install the desired version of python
pyenv install 2.7.10

# Create a virtualenv, I'm calling mine "jupyter"
pyenv virtualenv 2.7.10 jupyter

# Activate the virtualenv
# You'll have to do this everytime before running "jupyter notebook"
pyenv activate jupyter

# Install all the packages your heart desires
pip install notebook matplotlib numpy scipy sklearn

Create the wrapper around python

Find the path to your virtualenv’s python binary:

# Only run this if you're not already in the jupyter virtualenv
pyenv activate jupyter

# Where's python?
pyenv which python

This will give you a path that looks like /Users/amey/.pyenv/versions/jupyter/bin/python. Create a new file named frameworkpython (or whatever you want, this is the name suggested by the matplotlib FAQ) and put the following inside:

#!/bin/bash

# what real Python executable to use
PYVER=2.7
PATHTOPYTHON=/usr/bin/
PYTHON=${PATHTOPYTHON}python${PYVER}

# find the root of the virtualenv, it should be the parent of the dir this script is in
ENV=`$PYTHON -c "import os; print os.path.abspath(os.path.join(os.path.dirname(\"$0\"), '..'))"`

# now run Python with the virtualenv set as Python's HOME
export PYTHONHOME=$ENV
exec $PYTHON "[email protected]"

Remember to use the correct version in the line PYVER=2.7 and the correct path to your system’s python in the line PATHTOPYTHON=/usr/bin/. The system python directory can usually be found by running which python in a new shell session. If you get a path to pyenv (like /Users/amey/.pyenv/shims/python), you’ll have to go hunting. It should be at /usr/bin/python or /usr/local/bin/python.

Mark the new file as executable with chmod +x /path/to/frameworkpython, and that’s the wrapper taken care of. Next we need to create a new kernel for jupyter and tell it to use the newly-created frameworkpython as the interpreter. For this, we need to find out where jupyter is reading its configuration files from. Run jupyter --paths inside the virtualenv, you should get output similar to the following:

config:
    /Users/amey/.jupyter
    /Users/amey/.pyenv/versions/jupyter/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /Users/amey/Library/Jupyter
    /Users/amey/.pyenv/versions/jupyter/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
/Users/amey/Library/Jupyter/runtime

The first data directory is usually the one we’d like to use. So create a directory there for the new kernel. I decided to call my new kernel matplotlib to make it easy to identify, and so I ran

mkdir -p ~/Library/Jupyter/kernels/matplotlib

And then create a configuration file named kernel.json for the new kernel inside the directory you just created. For me, this was ~/Library/Jupyter/kernels/matplotlib/kernel.json. Paste the following inside this file:

{
 "argv": [ "/Users/amey/.pyenv/versions/jupyter/bin/frameworkpython", "-m", "ipykernel",
          "-f", "{connection_file}"],
 "display_name": "matplotlib",
 "language": "python"
}

Tweak the path inside argv and the display_name variable according to where your wrapper is and the name of your new kernel respectively. And that should do it. When you activate the virtualenv and launch jupyter, you should see an option to create a notebook with your new kernel and Matplotlib and Pyplot should work fine inside it if everything went according to plan.

Tunnel vision

Oct 25, 2015

For the past few years I’ve been looking forward to an extended stay in the States. I had a long list of things I’d do, people I’d meet, places I’d see, dishes I’d try, etc.

I’ve been here for about six weeks now, and the only thing (well, one of two things) that’s consistently on my mind is.. the next quiz. The next homework. The assignment that’s due three weeks from now. The homework that I did poorly in. The career fair I didn’t go for. The https://www.google.com/foobar tab that’s been sitting open in my browser for two weeks now. Internships I need to apply for (and land) before December.

It’s easy to get tunnel vision. While all of the above are undoubtedly important, I’m falling into the same trap I fell in a few years ago. Instead of living my life, I’m worrying about everything else so that I can live a better life sometime in the future. And when that future gets here, I’ll still be busy making my future life better. Hence I’m making this post to remind myself: while worrying about tomorrow, don’t forget about today.