First, make sure your Homebrew formulas are all up to date:

brew update

Install Colima along with the Docker and Kubernetes command-line tools:

brew install colima docker kubectl

If you work on a network controlled by an organisation that uses TLS stripping security appliances you’ll probably need to install additional CA root certificates before you can pull container images from the Internet, etc. You can put them in the usual place in your home directory and Colima will automatically install them in the VMs it starts:

mkdir -p ~/.docker/certs.d
cd ~/.docker/certs.d
curl -o proxy-cert.crt https://insecurity.my.corp/proxy-cert.crt

(Do make sure you put each certificate in a separate file; if they are concatenated you’ll need to split them.)

With Colima installed, you should be able to start a Colima instance. There are a handful of options to control the CPU, disk, and memory allocation for the VM, the runtimes to configure on it, etc.

colima start --memory 8 --kubernetes

Check that the Docker and Kubernetes command-line tools have been configured to talk to the new Colima instance:

kubectl get pods -A
docker ps

If you installed custom certs, you’d better check that it’s all working correctly by pulling an image:

docker pull python:3.12-slim

Easy.

Speaking of process, here is what we’ll be doing:

1. Download the source code of an official Python release.

2. Configure the build appropriately for our machine.

3. Compile the software.

4. Test the software to make sure it works properly.

5. Install the software.

6. Configure our system to make sure we can use the software easily.

Getting a Python release

Go to the Python source downloads and choose a version to download (I’ll use 3.5.1 throughout this document but the process should be similar for any other version). Download the “Gzipped source tarball” to your server. You can do this by copying the URL in your web browser and using the wget command like so:

wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz

This will download the release and save it in a file based on the URL. In our case it’ll be called Python-3.5.1.tgz. If you don’t have wget installed (maybe you get a command not found error) you can install it using your OS package manager:

sudo yum install wget

Once the file is downloaded you can unpack it using the tar command. Originally tar was used with magnetic tape drives (the name stands for tape archive) but it also works for archives stored in plain old files on disc and it’s still the standard archive format in UNIX environments. tar is often used together with gzip compression to reduce the size of file but all modern tar commands can handle compression and decompression transparently for you. tar archives usually have the .tar file extension and often have another extension for the compression format like .gz, .bz2, or .xz (for GZipk, BZip2, or XZ respectively). These extensions can be piled on top of each other like .tar.gz or squashed together into a single extension like .tgz. You can extract a tar archive file using the following command:

tar xf Python-3.5.1.tgz

tar’s argument handling is ancient and unlike almost all other UNIX programs so you probably needn’t bother learning how the work; just remember tar xf means “extract from a file”. Like most UNIX programs, tar will not produce any output unless it encounters an error or you specifically ask it to. If you’d like it to give you some feedback as it works use tar xvf instead (“extract verbosely from a file”).

You should now have a directory called Python-3.5.1/ which contains the release files. From now on I’ll assume we’re working inside this new directory:

cd Python-3.5.1/

Building Python

Most UNIX software uses a very similar build process:

• first you configure the build with the install location, where to find any libraries it needs, etc.

• then you compile the software

• then you test the software

• then you install the software

Each step is usually a single command and the whole process usually looks a little like this:

./configure
make
make test
sudo make install

The ./configure command runs the configure script which is included in most software packages. This is usually an enormous, automatically generated shell script which figures out how to call your compiler, where to find the libraries the software needs, where the new software should be installed, etc. If ./configure cannot find a compiler, a required library, etc. then it will report an error. You can usually tell which command or library is missing from the error message but it can be tricky to tell what to do to fix the problem. Usually it’s a missing library or header file and you can install the, e.g., foo and foo-devel packages and try to run ./configure again. When I’m feeling too lazy to read the requirements for a package it’s not unusual to go through this cycle a few times.

One of the things ./configure produces is another large, automatically generated script called Makefile (often there will be several of these scattered throughout the subdirectories of the software). The Makefile contains instructions on how to achive various tasks like “build” or “make the executable called python” or “run the test suite”. The Makefile script is executed with the make command. You can run make to execute the default task (i.e. the first one in the file) or, e.g., make test to run the “test” task.

The default task for Python, like most packages, is to compile the software so running:

make

should kick off the build process. For a widely used, well tested package like Python you should expect make to succeed whenever ./configure does.

Once the build is finished you should run the test suite to make sure that it all works as expected:

make test

Again, official Python releases are exceptionally reliable and the tests can take a long time to run so you might like to give this a miss.

Installing

Once it’s built you should install the software into some permanent location. Almost all packages that follow the ./configure, make, make install convention will install under the /usr/local/ directory by default: they’ll put their executables in /usr/local/bin/, data files in /usr/local/share/, etc. The /usr/local/ directory is properly considered part of the operating system (as opposed to part of your user account) so you should require root privileges to create files and folders there. This is why make install us usually run under sudo:

sudo make install

This will create the directories and copy the compiled files into place as the root user.

You might like to install it somewhere else. Perhaps you don’t have root access or you’d like to install multiple versions side-by-side? In this case you’ll need to change the ./configure command you run at the start to have a --prefix argument:

./configure --prefix=/some/other/directory

If you’ve already used ./configure and/or make with the default settings you can use make clean to wipe out any work it did and start again. For example, let’s install into the py-351 directory under your account home directory you could run the following command:

make clean
./configure --prefix=$HOME/py-351
make
make test
make install

Note that you don’t need (and shouldn’t use) sudo to run make install this time because you already own your $HOME directory. If you run ls $HOME/py-351 you should see all the installed files.

Using the installed software

Now that the software is installed there are a few ways you can use it; in order of difficulty:

• Specifying the full path

• Changing your $PATH temporarily

• Changing your $PATH permanently

You can always run a command simply by specifying the full path to the executable to run. Following on from the $HOME/py-351 example above:

$HOME/py-351/bin/python3 -V

should run the newly installed Python interpreter and output the version message.

If you want to use the new install in just this shell session you can change your $PATH temporarily:

PATH="$HOME/py-351/bin:$PATH"

The $PATH variable contains a : separated list of directories to search for an executable when you run a command in the shell. The first executable file with the right name is the one that will be executed. The which command can do this search and tell you which file will be executed or even give a list of all the options:

which python3
which -a python3

Notice that we are set PATH to be our installation directory followed by the current value of $PATH? If we leave the :$PATH off the end we’ll replace our existing $PATH and the shell will be able to find only our newly installed Python commands!

If you want your new $PATH setting to be used by all programs that get started by you shell session (i.e. not just the commands you manually type in but ones that they start in turn) you’ll need to export the variable:

export PATH

For the rest of your current shell session every mention of python3 in the commands you type in or the commands that programs you run execute for you mean $HOME/py-351/bin/python3 (if that file exists).

To make this change permanent you need to change your shell configuration files. The name of these configuration files and whether or not they already exist can change from system to system but they are always in your $HOME directory and are called one or more of the following:

• .bashrc
• .bash_rc
• .bash_profile
• .profile

If any of those files already exist in your $HOME directory (files beginning with a . are considered “hidden” by most UNIX tools, you’ll need to use ls -a instead of ls to see them) then you should edit that file. If not, .profile is probably a safe bet.

Create or open the chosen file in a text editor (I’ll describe using vi or vim) and add the lines you used above to the end of the file:

PATH="$HOME/py-351/bin:$PATH"
export PATH

To edit a file with vim (or vi, for our purposes they are identical) just pass it the name of the file as an argument:

vim .profile

Unlike most other editors vim has modes. The most important modes are command mode (where you give the editor commands like “save” or “open a new file” or “quite”) and insert mode (where you insert text into the file). By default vim starts in command mode but you can enter insert mode by pressing the i key. When you are in insert more you should see an -- INSERT -- on the bottom line of your terminal. While you are in insert mode you can use the keyboard arrow keys to navigate and any characters you type should be inserted into the file. Scroll to the bottom of the file and enter the two new lines as before. To leave insert more press the escape key on your keyboard; the -- INSERT -- should go away. There are plenty of keyboard shortcuts you can use in command mode but the easiest way to enter a command is with the command line which appears when you type a :. When you do you should notice the cursor jump to the bottom line of your terminal after a : character. The command to write your changes to disc and quit vim is wq (just like the shell you end a command by pressing the enter key).

You should now be able to restart your shell and, if all goes well, run the python3 command without specifying the full path $HOME/py-351/bin/python3. If it does not work after starting a new shell (e.g. starting a new terminal or logging in again) try looking at the output of echo $PATH. If it does not have new value at the start your shell is probably using one of the other files I listed above. Try making the same changes to each of them until it does work.

In this section I used the $HOME/py-351 prefix but all of these steps will work for the /usr/local prefix too and, on many systems, you will need to do them.

• Create a system group exchangefiles.

• Create a /home/exchangefiles/ directory and files/ directory within it.

• Allow users in the exchangefiles group to connect to the server using SFTP (but not SSH).

• Lock users in the exchangefiles group into the /home/exchangefiles/ directory using a chroot.

• Restrict some other options for users in the exchangefiles group.

So we’ll allow these users to connect to the SSH server and use SFTP to access a specific directory, and nothing else.

Preparations

First, lets create the new group:

sudo addgroup exchangefiles

Then create the new directories:

# Create the chroot directory
sudo mkdir /home/exchangefiles/
sudo chmod g+rx /home/exchangefiles/

# Create the group-writable directory
sudo mkdir -p /home/exchangefiles/files/
sudo chmod g+rwx /home/exchangefiles/files/

# Give them both to the new group.
sudo chgrp -R exchangefiles /home/exchangefiles/

SSHD configuration

The OpenSSH server configuration is typically called something like /etc/ssh/sshd_config. Find this file and open it in an editor. First we’ll make sure it will support SFTP in a chrooted environment. Search for a existing Subsystem sftp statement or insert it if it’s missing:

# Enable to built-in implementation of SFTP.
Subsystem sftp internal-sftp

Here internal-sftp means “use the SFTP server built into sshd”. The default on many platforms is to use an external SFTP implementation (/usr/libexec/sftp-server on Mac OS X or /usr/libexec/openssh/sftp-server on RedHat-y Linux distros) but this won’t exist inside the chroot. (Previous versions of this post omitted discussion of Subsystem; thanks to Matthew Saltzman for suggesting the improvement.)

Next we’re going to add a section to the end of the file using the Match directive which applies to users in our group:

Match Group exchangefiles

After that we specify the configuration directives which apply the matched connections:

# Force the connection to use the built-in SFTP support.
ForceCommand internal-sftp
# Chroot the connection into the specified directory.
ChrootDirectory /home/exchangefiles

Let’s lock down some of the additional capabilities of the OpenSSH server so these people can’t, e.g., forward connections through the server and into our private network:

# Disable network tunneling
PermitTunnel no
# Disable authentication agent forwarding.
AllowAgentForwarding no
# Disable TCP connection forwarding.
AllowTcpForwarding no
# Disable X11 remote desktop forwarding.
X11Forwarding no

So the whole block looks like this:

Match Group exchangefiles
  # Force the connection to use SFTP and chroot to the required directory.
  ForceCommand internal-sftp
  ChrootDirectory /home/exchangefiles
  # Disable tunneling, authentication agent, TCP and X11 forwarding.
  PermitTunnel no
  AllowAgentForwarding no
  AllowTcpForwarding no
  X11Forwarding no

Testing

To apply the configuration change, just restart your SSH server, create a user and try to access the site.

# On the server:
sudo adduser --ingroup exchangefiles testfiles
sudo service ssh restart

# On a test machine:
sftp testfiles@server.example.com
ssh testfiles@server.example.com

Connecting with sftp should result in a connection, but ssh should return an error message:

This service allows sftp connections only.

Connection to server.example.com closed.

When connected you should be able to list, upload and delete files under /files/.

TODO

If you’re doing something like this, you probably want the users to be able to access each other’s files – otherwise why bother sharing a space? To do this you’ll need to set the umask so that files are created with the correct permissions.

The internal-sftp command has a -u option which might help, but I haven’t tested this.

Updates

• 2015-01-08: Tweaked group write permissions on /home/exchangefiles after feedback from “Nik”. Thanks Nik!