pgeu-system.git - PostgreSQL Europe Conference and Membership management system.

Age	Commit message (Collapse)	Author
2022-01-15	Fix crash on job scheduler status page if never run	Magnus Hagander
	When the job scheduler had never run any jobs, the status page would crash instead of showing so (despite having code that was supposed to handle that situation). Spotted by Vik
2021-12-22	Fix name returned from ScheduledJob	Magnus Hagander
	This clearly never worked, but there is no code path in the general code that ever sees it.
2021-12-22	Pass in skip checks parameter, required by django 3.2	Magnus Hagander

2021-01-23	Implement an inotify based command reloader for daemon commands	Magnus Hagander
	Django used to have this, but at some point replaced it with a dependency on facebook/watchman. Since we don't require this (and it's a heavy dependency including runnning a separate daemon -- which is not packaged on at least Debian stable versions), and didn't have it, the django implementation would fall back on polling all files in a loop once per second. And since there are thousands of files to depend on in django environment, that could use a substantial amount of CPU. We don't use this for the webserver itself, as that runs under smoething like uwsgi, but we have two daemons at this point (scheduled task runner and social media poster) that does. So implement a local reloader based on inotify which is of course a lot more efficient. We don't try any fancy detection magic, and instead just watch every .py and.pyc file in a configurable set of directories (which would typically consist of the code checkout and the virtualenv root, but can be adapted). If used, this adds a dependency on pyinotify, but the default configuration is still to fall back on the django implementation.
2020-10-31	Ensure scheduled jobs runner quits on global exceptions	Magnus Hagander
	Most exceptions are handled by the runner itself, but if an exception happened at the global level only the main thread would exit, leaving the watcher thread running alone, meaning in reality the service hung. Since the system is designed to have systemd (or whatever else is used) automatically restart it on exit, this change ensures that the process exits in this case triggering that behaviour. This can happen for example if the database is restarted at the wrong moment.
2020-10-31	Remove unused import	Magnus Hagander

2020-07-13	Remove unused imports	Magnus Hagander

2020-07-06	Allow job schedule triggers to specify a delay as well	Magnus Hagander
	Immediate doesn't have to mean immediate, can be for example delayed 30 seconds.
2020-05-20	Add button to reset last-run-was-failure state of scheduled jobs	Magnus Hagander
	For jobs that run very infrequently, it can be useful to reset the failure state so they don't show up in an alerting red color in the list.
2020-05-11	Pass and store request in backend forms	Magnus Hagander
	This makes it possible to use it to for example indicate messages in the django messaging framework from inside methods in the class
2020-05-11	Support a job-specified default for notify on success	Magnus Hagander

2020-04-04	Set both no_color and force_color when calling jobs	Magnus Hagander
	New django requires us to both specify that we don't want color and that we don't want to force color...
2020-04-04	Update for django 2.2 autoreloader	Magnus Hagander
	Seems the interface to autoreloader has changed completely, attempt a new set of code off the new one. NOTE! This change makes django 2.2 a requirement!
2020-04-01	Specify on_delete for all ForeignKey and OneToOneFields	Magnus Hagander
	This was already done once in 8289e05cd but had not been properly maintained.
2020-03-12	Replace datetime.now() with timezone.now()	Magnus Hagander
	As a step on the way to better timezone support, use the django function timezone.now() instead of datetime.now(). As long as we haven't enabled timezones globally this becomes a no-op and does exactly what it did before, but once timezones are enabled it will generate datetimes that are aware of this. No functionality change but gets a lot of boiler-plate out of the way making the verification of the rest of the timezone work easier.
2020-02-09	Verify that integer parameters are integers at an early stage	Magnus Hagander
	Previously we'd in many places pass down the value directly from get or post requests to a lower layer, only to have that layer throw an exception because it wasn't an integer, or we'd ust wrap it in int() which also causes a hard exception when it's not an integer. Instead create a small wrapper for get_int_or_error() which can be called with a parameter that's supposed to be integer, and will then just return a 404 if the parameter doesn't exist or is not an integer. These are all "should never happen" scenarios, so not generating hard crashes and stackdumps are an improvement. None of these were places where the actual bad data would get anywyhere, they would all just cause an ugly exception, but should get fixed regardless. One or two instances spotted by Daniel Gustafsson, and then a lot of grep to try to find most of the rest.
2019-11-26	Don't run jobscheduler trigger until fully migrated	Magnus Hagander
	In particular, if doing a partial migration of for example the auth module, the trigger would simply crash since the job scheduler tables didn't exist yet.
2019-09-30	Add function to trigger immediate run of a scheduled job	Magnus Hagander

2019-09-30	Make "command" field of scheduled jobs unique	Magnus Hagander

2019-07-09	Create setting for SCHEDULED_JOBS_EMAIL_SENDER	Magnus Hagander
	This controls the sending address for status from the scheduled jobs runner. If not set, it will be automatically set to SCHEDULED_JOBS_EMAIL.
2019-01-30	Fix missing newline in log message.	Magnus Hagander

2019-01-29	Don't show "first run" under the headline of "last run"	Magnus Hagander
	oops...
2019-01-29	Write all job runner log output to stderr	Magnus Hagander
	That way it gets caught in journald
2019-01-28	Add documentation for a number of previously undocumented modules	Magnus Hagander
	Since the documentation structure now allows it, write docs for elections, meetings, membership, scheduled jobs and payments. More is definitely needed, but it's a good starting point.
2019-01-28	Create an internal jobs scheduler	Magnus Hagander
	Right now it's too complicated to set up the huge amount of cronjobs necessary to get the system working, and it's very easy to miss one or three. So create an internal jobs runner that will know which jobs to run, and when to run them. Since the data about the jobs come from the app itself, it will automatically update when a newer version is deployed. This is handled by a post_migrate hook that enumerates jobs. All jobs are the existing django management commands. When a member class called ScheduledJob is added to a management command it automatically becomes managed by the job scheduler. A job can either be scheduled to run at an interval ("every 30 minutes") or at a fixed time ("at 23:37"). If time is chosen, multiple different ones can be used, but only in the form of a time, so at least once per day. A job can also be defined without a schedule (for manual runs), and finally one job can trigger the immediate run of a different job after it. All jobs can be enabled/disabled through the web interface, though normally they should all be enabled. It is also possible to override the scheduling of individual jobs in the web interface. Jobs can be defined as internal, meaning that they only call internal functions and database accesses in django, or external (default), meaning they do things like call external APIs. Internal functions will be run in the same process using the same database connection as the manager, and external jobs will be executed in a subprocess. External commands can be given a timeout (default = 2 minutes) after which they will get killed. Each job can optionally be given a static method called should_run, which will be executed before the job. If this returns False, the job will be skipped. This can typically be used to avoid the fork+exec+potentiallyexpensivecheck of some external commands for jobs that need to react reasonably quickly but that are expensive to run. This function will be called internally in the runner even for external jobs. All jobs are run through one django management command that is intended to run as a systemd service. That means they are "single threaded" and there is no risk of overlapping jobs. By default notification emails are sent for all failed runs. It is also possible to on a per-job basis configure notifications to be sent on successful runs as well. A history of all jobs is kept in the database, including the output from both successful and failed jobs.