Four changes to notifications since the fork: one new trigger, and three you only notice at the wrong moment.
Notify when something changed
Job templates and organizations have a fourth trigger, Changed, beside start, success and failure. A run that reported a change on any host notifies whatever is attached to it, on top of the trigger for how the run ended: a job that changed something and succeeded fires Success and Changed both, and one that failed fires Failure and Changed. The message is customizable like the others.
The reason it exists is the playbook that has nothing useful to say through success or failure. A hardening or compliance run in check mode succeeds whether or not it found drift, so the interesting outcome is not how it ended but that something reported as changed. Check mode counts toward the trigger, which is the whole point of it.
Ad hoc commands answer the same way, so an organization's Changed templates cover them as they cover its jobs. Project updates, inventory updates, system jobs and workflow jobs record no per host results and never fire it. Start never fires it either, since nothing has run yet.
The hosts a job ran against
Custom notification bodies can use job.hosts, the sorted names of the hosts the job ran against, capped at a thousand so one notification from a large inventory does not turn into a wall of text. It is a data point beside the rest of the job metadata, so it drops into whatever body you already write.
A timeout on outgoing requests
None of the HTTP backends passed a timeout to its request, and requests has no default, so a receiver that accepted the connection and then went quiet held the notification open for as long as the socket stayed up. Notifications are sent from a dispatcher task, so the worker was held with it, and the webhook backend made it worse by following redirects up to its retry limit. One unresponsive endpoint was enough.
AWX_NOTIFICATION_REQUEST_TIMEOUT now bounds the webhook, Mattermost, Rocket.Chat and Grafana backends, defaulting to 30 seconds, which is what the email backend already did. It is a plain setting rather than a field on every template, so it can be raised or lowered in a settings file without an API change.
Unsetting a secret
Notification templates carry password and token fields, and once one was set there was no way to put it back to blank. They have revert buttons now, the same ones credentials and the other forms have.
The Changed trigger and the timeout are mine, #696 and #670. The host list and the revert buttons are @cigamit's, #617 and #331, the first of them answering issue #612.
Changed is the one that could go further. It sits on job templates and organizations, not on workflow job templates, because a workflow runs against no hosts itself. If you would want it to fire when any job inside a workflow changed something, say so and it can be designed properly rather than guessed at.
