CUCC Expedition Handbook

Troggle - programmers' guide

git logo

Programming Guide

If you want to roll up your sleves and fix some horrifying annoyance that is bugging the hell out of you then you have come to the right place.

Issue Tracking

In the firm belief that agreement on the problem must precede implementation of a solution, the Troggle git repository has been mirrored by Charlie Crossley. You may find it on their server at git.xle.sh/expo/troggle. This is an instance of Gitea, and includes an issue tracker (a la GitHub but minus the Microsoft). Please browse issues and feel free to submit new ones for discussion as well as to add comments to ones already opened.

you will already have tried all these..

and you will have dismissed them as too trivial for your needs. You will have skipped over all that nasty confusing Django stuff:

and you will also also have merely skimmed as irrelevant or trivial:

You want to dive into the real stuff and add missing fields to the core data model:

We admire your drive and enthusiasm. We were like you once.

First, you need to get the "expoadmin" password from a nerd. Then you will be able to dive into the online admin system where you can explore and poke the live system and database using the system admin control panel. This has a data structure browser for the classes and fields of the system as live and in-use: all the foreign keys and relationship cardinalities. (This is not the same thing as the troggle control panel which is for deeper data management: bulk uploading and export of data files.)

And, you may be surprised to discover, we have given a lot of thought into how to make it easier for you to get involved in the programming while not screwing up this multi-decade software project.

Kill it with Fire

dumpster fire

Everyone at some point feels like giving up and wonders if we should just rewrite it all from scratch using 'modern' techniques. Several of us have given this a lot of thought, see 'A kinder, simpler troggle', the 2030 plan, architecture constraints and most importantly, the 4-layer stack any future technology with some public URLs would need to emulate.

We have also been learning from the global software community when you really do need to 'Kill it with Fire': what are the characteristics of code and of organisations that mean that it is a good idea to give up, and if so how to manage the rewrite. The answers are "almost never" and "don't": as explained in this summary of the book 'Kill it with Fire'.

Replacing old systems, and changing their architecture, while keeping them operating continuously, is now a well-understood software engineering process comprising a dozen separate techniques. With troggle, we can probably fairly easily replace the cross-referencing between different types of things (e.g. logbook entries, survex files, persons, expedition days) but we will find it tricky to replace the sychronisation and concurrency that SQL databases give us 'for free'. We are waiting to see if Django comes up with something cunning as it steadily converts to a more asynchronous architecture using python asyncio.

Perhaps an external add-on?

Perhaps you can program something external to troggle, in JavaScript say, using troggle data?

Creating your own system development machine

It's a simple sequence which steadily builds your expertise:

  1. Use the online forms and reports.
  2. Read documentation to get the concepts and structures into your head, so that you can see how it all fits together and works.
  3. Explore the public pages of the 4 git repositories including all the source code
  4. Use the online forms and reports together with 'survex' and 'therion' installed on your own laptop.
  5. Get the 'expoadmin' password and explore the live system online: watch the data change in real time
  6. Configure an ssh key and Filezilla or scp/sFTP.
  7. Configure a basic expo laptop
  8. Configure cave survey software on your laptop
  9. Configure a troggle development machine - a laptop where you have cloned the repos and installed django. Installing is the same as updating Django. See Troggle: updating Django.

Fixing broken URL dispatch

This happens all the time. If ever troggle is crashing on you very mysteriously, it is almost certainly due to a typo in troggle/urls.py . Odd but true.

Fixing a broken HTML template

[to be written]

Running databaseReset

Running the test suite

This manual is very incomplete - notes

[Note that /survey_scans/ is a url published by troggle and is not the same thing as the real directory /expofiles/surveyscans/ at all.]

Only a small part of troggle's original plan was fully implemented and deployed. Many of the things it was intended to replace are still operating as a motley collection written by many different people in several languages (but mostly perl and python; we won't talk about the person who likes to use OCamL).

Writing python

We aspire to a coding standard, and the Google python coding standard is a pretty good one. We don't use their TODO system though (para 3.12) and the function (re)naming in troggle is still work in progress (para. 3.16.4).

We don't have any type hints in troggle. It is not a big enough project to be worth it - probably.

We have run much of the code through black in 2022 so that is our default formatting standard.

We have also used ruff, isort and chill to trim and deduplicate the various pip installation packages and to fix the order of python packages in import statements.


Archived updates

Since 2008 we have been keeping detailed records of all data management system updates in the version control system. Before then we manually maintained a list of updates which are now only of historical interest.

A history of the expo website and software was published in Cambridge Underground 1996. A copy of this article Taking Expo Bullshit into the 21st Century is archived here.


Go on to: Troggle architecture
Return to: Troggle intro
Troggle index: Index of all troggle documents