tests:
 ADD TESTS when we are redirecting /expofiles/ to a remote file-delivering site

- Add test for running cavern to produce a .3d file


forms:
 


middleware:
 SmartAppendSlashMiddleware(object) Not Working. 
It needs re-writing to be compatible with Django v2.0 and later


models/caves:
 
- Find out why we have separate objects CaveSlug and why
  these are not just a single field on the Model. This was Martin's idea,
  but we are using text aliases now so we only need one slug in the data model
  
- Can we rewrite things to eliminate the CaveSlug and objects? Surely
  foreign keys work fine ?!
  
- Why do we have CaveAndEntrance objects ? These do not need to be explcit for a many:many relationship these days

- move the aliases list from the code and put into an editable file
  
- Restore constraint:   unique_together = (("area", "kataster_number"), ("area", "unofficial_number"))
  or replace by a unique 'slug' field, better.


models/logbooks:
 
- Can we rewrite things to eliminate the CaveSlug and objects? No
  Surely foreign keys work fine ?! No
  
  Foreign keys do not allow for there being multiple ways to refer to a cave, eg 1623-1999-03 aka 1623-204
  Having slugs allows for much more loose coupling to caves, which removes alot of the need to reset the database, which interupts work flow.
  It also means we do not have to be creating temporary cave objects in the database, where we do not have the underlying file in cave_data.

  To Do move Cave Slug back to troggle.core.models
	


views/caves:
 
- in getCaves() search GCavelookup first, which should raise a MultpleObjectsReturned 
  exception if no duplicates
  
- Learn to use Django .select_related() and .prefetch_related() to speed things up
  especially on the big report pages 
  https://zerotobyte.com/how-to-use-django-select-related-and-prefetch-related/
  
- Remove all the URL rewriting which is there because we have not yet edited all the caves to use
  our new (2023) standard addressing of /16xx/NNN/NNN.html where *all* caves are assumed to have their
  own directory 16xx/NNN/ even if they have no images to put in it.


views/drawings:
 - Need to check if invalid query string is invalid, or produces multiple replies
and render a user-friendly error page.


views/logbooks:
 - Fix the get_person_chronology() display bug.

- Fix id= value preservation on editing


views/other:
 
- [Low priority] Fix Login page so that it produces the frontpage or
  redirects to the page which produced the login prompt requirement.


views/scans:
 
- one of these views serves files as binary blobs, and simply set the mime type based on the file extension,
as does the urls.py dispatcher which sends them here. Here they should actually have the filetype checked 
by looking inside the file before being served.

- need to check if inavlid query string is invalid, or produces multiple replies
and render a user-friendly error page.


views/survex:
 
- filter out the non-public caves  from display UNLESS LOGGED IN

- Never actual uses the object for the survexfile, works entirely from the filepath! Make it check and validate

- the primary survex file in each cave directory should be in a configuration? not buried in the code
  and implicit in the order of *import statements ? It is in the cave_data file.

- overlapping and cross-calling when things fail make this hard to undersand, e.g. svx() and 
  survexcavessingle() can get called for a survex file depending on whether the URL ends in ".svx" or not,
  but each tries to handle the other case too.
  


views/uploads:
 
- Ideally we should validate uploaded file as being a valid file type, not a dubious script or hack
  Validate image files using a magic recogniser in walletedit()
  https://pypi.org/project/reportlab/   or 
  https://stackoverflow.com/questions/889333/how-to-check-if-a-file-is-a-valid-image-file

- Write equivalent GPX upload form system, similar to walletedit() but in expofiles/gpslogs/
  Need to validate it as being a valid GPX file using an XML parser, not a dubious script or hack
  
- Validate Tunnel & Therion files using an XML parser in dwgupload(). Though Julian says 
  tunnel is  only mostly correct XML, and it does fail at least one XML parser.
  
- parse the uploaded drawing file for links to wallets and scan files as done 
  in parsers/drawings.py

- Enable folder creation in dwguploads or as a separate form

- Enable file rename on expofiles, not just for /surveyscans/ (aka wallets)

- Make file rename utility less ugly.


views/wallets_edit:
 
- Nasty bug in navigating to 'previous wallet' when we have a 2-year gap in expos
  The xxxx#00 wallet is not getting edited correctly. Something is off by one somewhere..

- Register uploaded filenames in the Django db without needing to wait for a reset & bulk file import

- Refactor walletedit() as it contains all the wallets 'complaints' code from the pre-2022 
  script 'wallets.py'

- We should validate uploaded file as being a valid image file, not a dubious script or hack?


parsers/caves:
   
- When reading cave data, to start off wit we do not know the cave id (slug) so we can't give a useful url in
  the error message, but we do have the filename. Systematize this, and the same thing with reading entrance files.
  
- Cannot use Edit This Page for pendingcaves.txt_edit as Edit This Page is expecting an html file.
   So we will need a separate file-editing capability just for this configuration file ?!
   
- we want to overwrite a PENDING cave if we are now importing the 1623-xxx.html file for it

- rewrite archaic regex
  re.findall("<%(itemname)s>(.*?)</%(itemname)s>" % {"itemname": itemname}, text, re.S)
  in modern form and pre-compile it. 
   
- crashes on MariaDB in databasereset.py on server when deleting Caves and complains Area needs a 
  non null parent, But this is not true.   The only solution we have found is to let it crash, then 
  stop and restart MariaDB (requires a logon able to sudo)   and then restart the databasereset.py 
  again. (status as of July 2022). May not happen now that class Area is removed (Sept.2023).


parsers/drawings:
 
- Rename functions more consistently between tunnel and therion variants

- Refactor to use pathlib instead of whacky resetting of loop variable inside loop
  to scan sub-folders.

- Recode rx_valid_ext to use pathlib suffix() function

- Recode load_drawings_files() to use a list of suffices - not the huge if-else monstrosity


parsers/locations:
 
-    


parsers/logbooks:
 
- check cross-references in other logbooks and other HTML frahments
e.g. cave descriptions

- Most of the time is during the database writing (6s out of 8s).

- profile the code to find bad repetitive things, of which there are many.

- attach or link a DataIssue to an individual expo (logbook) so that it can be found and deleted

- rewrite to use generators rather than storing everything intermediate in lists - to 
  reduce memory impact [low priority]
  
- We should ensure logbook.html is utf-8 and stop this crap:             
            file_in = open(logbookfile,'rb')
            txt = file_in.read().decode("latin1")
            


parsers/survex:
 
- Obscure bug in the *team inheritance and rootblock initialization needs tracking down,
  probably in the team cache which should NOT be global, but should be an instance variable of 
  LoadingSurvex
  
- Lots to do to cut down on unnecessary .save() calls to avoid hitting the db so much. Should
  speed it up noticably. 
  
- Learn to use Django .select_related() and .prefetch_related() to speed things up
  https://zerotobyte.com/how-to-use-django-select-related-and-prefetch-related/
        
- LoadSurvexFile() Creates a new current survexfile 
        The survexblock passed-in is not necessarily the survex parent. FIX THIS.
                
- When Olly implements LEG in the  'dump3d --legs' utility, then we can use that to get the length of
  all the legs in a survex block instead of adding them up oursleves. Which means that we can
  ignore all the  Units and offset stuff, that troggle will work with survex files with backsights,
  repeated readings from distox etc.. Not actually useful for pre 2022 survey data, 
  but good future-proofing.
  Also it will be a tiny bit more accurate as these leg lengths are after loop closure fixup.


urls:
 
- Replace more re_path() with modern and simpler path(). Careful: some have to stay as re_path()

- The admin and logout paths need to stay using re_path() as they 
  have to be locked to the start.

- The final _edit and CATCHALL also have to use re_path().

- Test VERY CAREFULLY for each change. It is fragile.