CUCC Expedition Handbook

Troggle - software development laptop

python logo

Software development machine

For troggle itself, you need a linux machine. We all use Debian or Debian-derived machines (Debian itself, Ubuntu, Xubuntu etc.) but other forms of linux may work. Running Debian or Ubuntu under WSL on Windows 10 works fine.

If you want to work on the troggle source code and be able to commit to the server git repo, your account will need to be added to the troggle project members list. Contact wookey at wookware dot org to get this set up.

Before you start

Make sure you are familiar with the debugging tools already built into troggle. You may not need to write any new software for what you need to do. Look at the tools and reports listed on the Troggle Control Panel page (needs 'expoadmin' login password.)

Maybe what you need is simply some fixes or workarounds in the imported data files, not in the python code.


You need to already know really quite a lot about what troggle does, and how it is used in practice. Also you will have installed quite a lot of the software such as tunnel, therion, survex etc. as documented in your bulk update laptop configuration. In particular read the information there about VS code and git.

This page is a work in progess. Text will be moved here from
debian logo

Installing linux

If you don't already know how to do this, then you should probably not be attempting to work on the troggle code. But in case you are an experienced linux user who has always had someone else set up the system for them, then Ubuntu is the easiest and more forgiving to install, either directly on the computer or inside WSL. Ubuntu installs python and various essential python dev tools by default.

The server is running fairly old, stable releases of Debian and Django, but you will probably want Ubuntu-22.04 which is compatible.

There are two scripts in the troggle folder which will do semi-automatically what is described below. Have a look at

You will run just once to install the basics, but you will run every time you fire up a new python version/django version combination or play with the versions of the imported packages as listed in requirements.txt. takes a few minutes initially, but then about an hour when it installs therion and tunnel as these drag in a huge number of dependencies.

Files and directories

Do familiarise yourself with the directory structure on the expo server, which we will be duplicating (partly) as docmented in The Expo Server. We are not here doing a full install of all the software and scripts on the server, just the minimum to run, test and debug troggle on Django. These will be setup for you by

WSL on Windows

[On 23 November 2022 a new way of installing and using WSL was announced, see Windows Subsystem for Linux now packaged as a Microsoft Store app so all documentation on this expo site needs to be revised.]

If you are using an old laptop, you may have to run Ubuntu in WSL1 rather than WSL2. WSL2 requires virtualisation features that your hardware may not support if it is more than about 5 years old.

The standard documentation for Ubuntu or debian below all works, but you should first skim the Windows expo laptop configuration too. So far as expo software is concerned, WSL1 and WSL2 behave identically.

...Except for file permissions, which can cost you a day of frustration if you are unlucky. The trick is to make sure that all the files in your development folders, e.g. C:\expo\ which contains your repos e.g. C:\expo\troggle\ are owned in the Windows system by the default Windows user e.g. MACHINENAME\philip and owned in the Linux system e.g. /mnt/c/expo/ by the default Linux user, e.g. philip:philip using
sudo chown -Rhv philip:philip *
and then reboot your machine as this doesn't seem to properly take effect until you do that.

See also WSL File Permissions.

Why no Docker container?

Yes, it is true that this would greatly speed up on-boarding new programmers.

But there is the significant danger that containers would get copied around and deployed without being properly cleaned up: resulting in configuration drift and a snowflake server situation. File permissions are a big issue.

We should do both: create a Docker system for getting started, then transition programmers to script-based or recipe-based provisioning so that systems are rebuilt cleanly. CUYC (who also use Django) have a bash script which sets up a new django development system. We should copy that in the first instance. Alas, we haven't got around to doing any of this yet.

Configuring git

On a new machine you need to configure your git identity:

git config --global ""
git config --global "Your Name"
git config --global pull.rebase true

We use pull.rebase as this seems to work best with our repos and the way we work.

Installing python

XKCD python install

Python is not installed by default on Debian, and in any case we need specific versions to be installed. For Ubuntu 20.04 the default installation is python3.9 whereas Ubuntu 22.04 gives you python3.10.

If you are planning on eventually helping the migration of troggle from debian Bullseye (v11) to Bookworm (v12) to then you will also want the appropriate later versions of python e.g. 3.10 on Ubuntu 20.04 like this:

sudo apt install python3 python3-pip -y
sudo apt install sqlite3 sqlite3-doc -y
sudo apt install survex -y
sudo apt install software-properties-common -y
sudo apt install git gitk -y
sudo apt install default-jdk -y
sudo apt install binutils binfmt-support -y

#sudo add-apt-repository ppa:deadsnakes/ppa
#sudo apt install python3.10 python3.10-venv python3.10-doc 
#sudo apt --fix-broken install

cd /usr/bin
sudo ln -s python3 python
sudo ln -s pip3 pip 

You will also definitely need sqlite3 even if you are planning to use another database. Sqlite3 is used by the test harness system.

Note that when you install survex it installs a shed load of packages that it needs.

We do not install Django at this point. We will be installing Django in a separate virtual environment (a 'venv'), not in the main linux system.

Buggeration: The sudo add-apt-repository ppa:deadsnakes/ppa line won't work after Debian 11 or Ubuntu 22.04. So this handbook page will need to be changed to use the new method. And this way of installing python3.10 is fine on Ubuntu but not on Debian, where you are stuck with 3.9 on Debian 11 (Bullseye). Note that default-jre is Java which is used by tunnel and gpsprune.

We need to have the folder structure that troggle expects. On your own machine you will have your own logon id, and you do not need to create an 'expo' user, but you do need to create folders where and an 'expo' user would have them. Make links there to wherever you have actually installed the repos. So if you have installed all your repos in /mnt/c/EXPO/, then you would need to do this:

cd ~
cd ..
sudo mkdir expo
cd expo
sudo ln -s /mnt/c/EXPO/expoweb expoweb
sudo ln -s /mnt/c/EXPO/troggle troggle
sudo ln -s /mnt/c/EXPO/loser loser
sudo ln -s /mnt/c/EXPO/drawings drawings
sudo ln -s /mnt/c/EXPO/expofiles expofiles
sudo ln -s /mnt/c/EXPO/expowebcache expowebcache
sudo mkdir expowebcache/3d
cd ..
ls -tlA expo


If you do not have a local copy of the 40GB /expofiles/, don't worry. Later on we can set 'EXPOFILESREMOTE = True' in the file and your test system will use the live expofiles on (read only).

If you do have 'EXPOFILESREMOTE = True' then the forms which upload scans and photos to the server will not work as you expect. They will upload to your local machine, but read the status of the folders from So you will get confusing and apparently inconsistent behaviour: e.g. you will upload a file but then be unable to see it.

For development, you mostly only need a local copy of the wallets and scanned survey notes and sketches in expofiles/surveyscans which is less than 5GB.

Installing Django and troggle

django logo

The important point to note here is that unless you are doing something fairly trivial, or you are a git genius, it is sensible to set up a python virtual environments to hold duplicate copies of both troggle and Django code. Then you will be able to check very quickly that your edited version of troggle runs with old, current and pre-release versions of python and of Django; and you will more easily be able to manage problems with incompatible versions of Django plugins as installing and upgrading the dependent packages is very fast.

Installing a venv

We set up a venv specifically for python 3.9 (which is the standard version on our server which is running unmodified Bullseye (debian v11) and Django 3.2.12. See the standard python documentation on venv for python 3.9.10. Create a new venv for each version of python you are using. Get that venv installed right by explicitly stating the python version to create it python3.9 -m venv py39d32. Note that we are creating it as a sibling folder to /expoweb/ . Note also that up to now we have been using 'sudo ..' but for installing things inside the venv we do not use 'sudo ..':

cd ~
cd ../expo
python3.9 -m venv py39d32
cd py39d32
source bin/activate
pip list -o

The last command lists the default packages installed in the venv. This is for comparison later, after we have installed troggle, Django and dependencies. You will get a warning that you have an out of date version of pipbut this is as we want: we are using a version of pip appropriate for the older version of python within the venv. troggle logo

Installing the troggle dependencies

The first time you do this on a new machine you can't complete the pip installation of Django as you have not yet got the dependencies appropriate for troggle - because you have not yet cloned the troggle repo. So the first time it is easiest to just create requirements.txt yourself with a text editor. Without using git yourself, you can get the file from the website at requirements.txt. If you have already cloned all the repos, then just copy it.

cp ../troggle/requirements.txt .
where requirements.txt (note the capitalisation of the listed packages) is:
This will pick up the latest Django 3.2.x available, but all the other dependencies are pinned. These dependencies are as-standard on debian Buster (10) and you will want to experiment with upgrading them for Bullseye (v11).

Once you have the file, install the listed dependencies like this:

pip install -r requirements.txt
pip list -o

Pillow (not currently in that package list) is an image handling package used to make the prospecting map (currently disabled, see /prospecting_guide/).
tinymce is the wysiwyg in-browser editor (disabled pending reinstatement)

This is also documented in the updating Django for troggle page where it describes upgrading and testing with later versions of Django.

If you have not used pip before, read this

Installing troggle

The :troggle: repo is the python source code for troggle. This is what you will be editing. There are over 9,000 lines of python code (excluding comments and blank lines) and nearly 3,000 lines of HTML/Django template code. This is over 600 files in over 400 folders, but only 49MB in size (plus a .git repo of 32MB).

key exchange

You need this so that you can upload your edited code to the git repo on the server.

Follow this link to register a key with the expo server to get git access if you have not already cloned the :troggle: repo.

git clone troggle

You will do a git clone to create a folder /troggle/ containing all the troggle code. You need to clone the 'python-3' branch, if you an see multiple branches.

cd ~
cd ../expo
git clone ssh://

Now we create soft links within the venv to all the repo folders and wherever you have a local copy of /expofiles/ and /expowebcache/:

cd py39d32
sudo ln -s /mnt/c/EXPO/expoweb expoweb
sudo ln -s /mnt/c/EXPO/troggle troggle
sudo ln -s /mnt/c/EXPO/loser loser
sudo ln -s /mnt/c/EXPO/drawings drawings
sudo ln -s /mnt/c/EXPO/expofiles expofiles
sudo ln -s /mnt/c/EXPO/expowebcache expowebcache

If you have not been using git on this machine until now, you will need to identify yourself before you go any further.

git config --global ""
git config --global "Your Name"

do the basic Django health checks

This all checks that the installation has completed properly.


The first line django-admin will complain that it has not got a SETTINGS file, but that's fine. See django-admin initial output for what you should expect to see at this point. If it crashes though, you have not managed to install the software completely. If you get an error when running

Then run django-admin like this (using the python version you have installed in the folder underneath /lib/):
python /usr/local/lib/python3.8/dist-packages/django/bin/
which should fix the paths, but this means that you installed Django directly on your machine and not in a venv. From now on you should be able to run django-admin from within any folder on your machine.

Now try

django-admin --version
The version number is the version of Django you have installed. Double check that it is the one you meant to install and check with our Django versions page.

do the basic troggle health checks

Now try

You will get an error: No module named 'localsettings'. Fixing this is described below, but for now try:
python check -v 3 --traceback

The most important is the python check. If this works, then you have installed the software correctly.

The next task is to edit the SETTINGS files to match your machine and folder structure. So find the appropriate copy of the localsettings in /_deploy/ and copy it into the main troggle folder:

cp _deploy/wsl/
The git repo copies have got munged passwords. is not stored in git, but the copy on the server does have the correct passwords. So use sFTP to download from to get these.

We have at one time made localsettings in /_deploy/ appropriate for

WARNING: only the WSL and debian variants are current in December 2021. All the others are so old that they will need serious work to be useable. Copy what you need from WSL and debian variants of

Now edit and insert useful values for EXPOUSERPASS [e.g. cavey:beery], EXPOADMINUSERPASS [e.g. beery:cavey], SECRET_KEY. SECRET_KEY can be anything, it just has to be unique to each installation and invisible to anyone not a developer.

Set EMAIL_HOST and EMAIL_HOST_PASSWORD to an email account you control that can send email. Then troggle can email you when some things go wrong. This may mean having to set EMAIL_PORT and MAIL_USE_TLS too (this is not used in troggle currently). Set EXPOUSER_EMAIL and EXPOADMINUSER_EMAIL to your own email address while you are doing software development. All these will be different when troggle is deployed on the public server.

Now you need to edit the following settings in your file to match your development machine, e.g. if you have /expofiles/ mounted on another disc:

FILES = Path('/mnt/f/expofiles/')
EXPOFILES = Path('/mnt/f/expofiles/')

All the other settings (drawings, expoweb etc.) will work fine if they are parallel directories to the directory you installed troggle into. The troggle code can find out itself where it is living.

If you do not have a local copy of /expofiles/ (40 GB), you can use the expo server copy if you set:

and then the FILES and EXPOFILES setings will be ignored. (Except for the upload forms which will 'upload' files to your local disc. )

Now try this again:

and in addition to the [django] command list, you will now gets command lists for [auth], [contenttypes], [core], and [sessions]. These are the modules (plugins) loaaded into django. [core] is the core of troggle (but not all of it: the input file parsers are not in [core]).


python -Wall check 
This loads the settings files and checks that all the Django packages and python libraries imports all work. It gives warnings of deprecated Django which should be fixed. You should not get any warnings on a fresh install on a new machine but you will see warning and error messages when you are trying to upgrade troggle to use later versions of Django.
python check -v 3 --traceback
Ideally this will list settings imports and then say
System check identified no issues (0 silenced).

If you get an error your python sys.path is probably not set correctly yet. Do

python -m site
when you are in your troggle directory to see the list of paths python looks for when it is searching for packages (both django and troggle).
Ensure that the path to the troggle/ directory is in the list. It should be at the top, which is where the current working directory is.


python check -v 3 --deploy
which will give security warnings for deployment. You will get various middleware settings because we have not got https:// properly configured everywhere (we have unresolved issues with the Django admin control panel and https:// in October 2021) and a warning not to use DEBUG=true in deployment. We always have DEBUG=True set for troggle as otherwise the users get useless error messages. This is completely contrary to Django official security advice.
Click triangle to see security warnings in detail
?: (security.W001) You do not have '' 
SECURE_BROWSER_XSS_FILTER, and SECURE_SSL_REDIRECT settings will have no effect.                         

?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only 
session cookie makes it more difficult for network traffic sniffers to hijack user sessions.                                                                                           
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your 
MIDDLEWARE, but you have not set CSRF_COOKIE_SECURE to True. Using a secure-only 
CSRF cookie makes it more difficult for network traffic sniffers to steal the CSRF token.              

?: (security.W018) You should not have DEBUG set to True in deployment. 

If you got an error traceback with

then the settings registration of troggle with django is incomplete. Delete all your cached .pyc files and try again. You probably have a mistake in your or files.
python diffsettings
This last one shows everything set in global settings, settings and localsettings. Anything different from global settings (django built-in) has '###' appended.
python help migrate
python migrate
Tests the uptodateness of your sqlite database. The help option explains what this does and gives extra command line options.

check full functionality

Now run the test suite:

python test -v 3 --traceback

This will run the entire troggle test suite of ~80 tests (it takes only a few seconds).

If you get an error, and you probably will, have a look in the source code of the test, e.g. for this error:

FAIL: test_page_folk (troggle.core.TESTS.tests.PageTests) 
look in the file troggle/code/TESTS/ in the class PageTests. It will also say:
  File "/mnt/c/EXPO/troggle/core/TESTS/", line 266, in test_page_folk
which means that the asssert failure in on line 266 of troggle/code/TESTS/ and that the failure function is test_page_folk() . If you look at this you will see that line 264 is:
 response = self.client.get('/folk/index.htm')
so this file is missing. Duh. Of course it is. We downloaded troggle from git but we didn't run the standalone script to generate the folk list. The quickest thing to do when installing troggle is to simply download a copy from the server:
 wget -O ../expoweb/folk/index.htm
and run the tests again:
python test -v 2
Click on the triangle to see how to run the folk script. You don't need to do this now. The folk generation script is top of the list in

So do this:

cd ../expoweb/folk
python ../scripts/ <folk.csv >index.htm
cd ../../troggle

The test suite now tidies up after itself, so there should not be any temporary files left behind or local git commits that you will need to clean up.

The test suite has ~80 tests but does not cover all of what troggle does and does not use any real data. You need to manually test these too, after you have done a full data import:
- http://localhost:8000/pathsreport
- http://localhost:8000/stats
- http://localhost:8000/people (takes a minute or so)
- http://localhost:8000/wallets/year/2019l
- http://localhost:8000/survexfile/caves/
- http://localhost:8000/expofiles/training-info/Idiots guide to accessing expo git.pptx
- http://localhost:8000/1623/291/291
- http://localhost:8000/caves
- ttp://localhost:8000/admin/doc/models/core.expedition/ (admin login required)
- http://localhost:8000/survexfile/204

Check parsing and importing

Nearly half the code deals with importing and parsing data, so you need to test that a full data import works. Run the full data import troggle$ python reset R000. It should take about 5 minutes to import everything. See Troggle Full Import for what you should expect to see.

Use git to commit your edits

You need to know git. Sorry, but there it is. See our git repositories and our git cheat sheet.

Helpful database tools and scripts

The public server uses a MariaDB SQL database and development is usually done using a single-user sqlite database which is a standard Django option.

You will find it very, very useful to see what is going on if you look directly at the data in the database (just a single file in the sqlite case) and browse the data in the tables. This is vital when doing Django migrations between Django versions. A light-weight, simple db browser is DB Browser for SQLite. Connecting directly the the MariaDB database with a control panel or workbench gives even more tools and documentation capabilities. See the troggle server documentation for how to install MariaDB.

When Ubuntu is running on WSL, it does not use systemctl. So you need specific instructions for installing MariaDB under WSL, do what it says in these instructions first (sudo apt install ...etc.):

Create a new dedicated administrative MariaDB user 'expo' who can access all databases. Log in to the MariaDB command with

sudo mysql
and execute these commands:
GRANT ALL PRIVILEGES on *.* TO 'expo'@'%' IDENTIFIED BY 'my-secret-password-schwatzmooskogel' WITH GRANT OPTION;
SET PASSWORD FOR expo=PASSWORD('my-secret-password-schwatzmooskogel');
and you will need to set this user and password in your
    'default': { 
        'ENGINE': 'django.db.backends.mysql', # 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME' : 'troggle',                   
        'USER' : 'expo',                      
        'PASSWORD' : 'my-secret-password-schwatzmooskogel',       
        'HOST' : '',                          # Set to empty string for localhost. 
        'PORT' : '',                          # Set to empty string for default. 

But it still does not work

That is because we need to install the python tools that talk to mariadb. And while it 'will just work' for python3.9, the standard installed on the distro, with python3.10 it is a bit more work. but note that there is a problem with using python 3.10 in that some bits of pip are not correct and you will get a ImportError: cannot import name 'html5lib' error. TEMPORARILY use this hack from bootstrap:

curl -sS | python3.10
pip install mysql-connector-python
installs the correct pip for python3.10 on Ubuntu. Which works, then
sudo apt-get install python3.10-dev
sudo apt install libmariadbclient-dev
pip install mariadb
wwhich now has installed mariadb python stuff, but seems to have trashed my django installattion. Hmph. And pip.
Go on to: Troggle architecture
Return to: Troggle programmers' guide
Troggle index: Index of all troggle documents