Re: Systemd Journald and audit logging causing journal issues

Hey Steve, Just wanted to follow up on this and say we are still seeing services across the board that have: Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable basically created a script to check all unit file services/targets and grep status -l for Journal, it is effecting a large range of service.target's, service.service's and service.socket's If we restart the service or reboot we no longer see those messages about journal and everything appears to run as expected.

On 10/19/2017 04:13 PM, Steve Grubb wrote: > On Thursday, October 19, 2017 1:08:22 PM EDT Brad Zynda wrote: >>>> grep perm_mod /etc/audit/audit.rules >>>> -a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F auid>=1000 >>>> -F auid!=4294967295 -k perm_mod >>>> -a always,exit -F arch=b32 -S chmod -S fchmod -S fchmodat -F auid>=1000 >>>> -F auid!=4294967295 -k perm_mod >>>> -a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -F >>>> auid>=1000 -F auid!=4294967295 -k perm_mod >>>> -a always,exit -F arch=b32 -S chown -S fchown -S fchownat -S lchown -F >>>> auid>=1000 -F auid!=4294967295 -k perm_mod >>>> -a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -S >>>> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F >>>> auid!=4294967295 -k perm_mod >>>> -a always,exit -F arch=b32 -S setxattr -S lsetxattr -S fsetxattr -S >>>> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F >>>> auid!=4294967295 -k perm_mod >>>> >>>> grep delete /etc/audit/audit.rules >>>> -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat >>>> -F auid>=1000 -F auid!=4294967295 -k delete >>>> -a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat >>>> -F auid>=1000 -F auid!=4294967295 -k delete >>>> -a always,exit -F arch=b64 -S init_module -S delete_module -k modules >>>> -a always,exit -F arch=b32 -S init_module -S delete_module -k modules >>> >>> These rules are well formed. Meaning no obvious holes that would cause >>> unintended events. The other ausearch/aureport commands I gave you >>> should >>> show you what is causing the events and to which files. This information >>> may be used to create some kind of "never" rule that limits what gets >>> audited. If you do create some exclusion rule, it needs to be above the >>> perm_mod rules because audit is a "first match wins" system. >>> >>> -Steve >>> >>> -Steve >> >> Here is a peak report (at least in the last 24 hours) didnt include the > >> file summaries as that would make this email HUGE: > Well, the idea is to look for something that's getting hit a lot. What it > sounds like is that things are getting deleted and modified quite a bit > all > over the place. Does the executable report show a pattern such as one > application doing a lot? For example, with the rule you have, doing a yum > update will delete a whole lot of stuff and modify a whole lot of stuff. > > Its possible to put exceptions in the rules so that one program does not > flood the logs. But looking at the quantities below, the audit system can > easily handle that. > > Its also possible to exclude directories from auditing if the pattern is > that you have a daemon receiving and modifying files and then deleting > them. You should look at the executables & files and see if you can do > with auditing what they are doing because its not interesting. > > If this is causing you problems on the journald side where its causing > other tasks to fail. Then I'd break the link between auditd and journald. > The amount of event you show is highish but well within range of what > auditd can do. Just make sure flush is set to incremental_async and freq > is 100 or 200. You should be OK. > > -Steve > >> Key Summary Report >> =========================== >> total key >> =========================== >> 8170 perm_mod >> 5390 delete >> 1073 access >> 56 time-change >> 26 session >> 12 privileged >> 7 logins >> >> >> Syscall Summary Report >> ========================== >> total syscall >> ========================== >> 4250 fchmodat >> 1613 chmod >> 831 fchmod >> 521 fchown >> 97 chown >> 12 setxattr >> >> >> Syscall Summary Report >> ========================== >> total syscall >> ========================== >> 2887 unlink >> 2189 rename >> 186 unlinkat >> >> >> so from the list the top 2 in each perm_mod and delete from the above >> list seem to be the culprits..