Hey Steve,
Just wanted to follow up on this and say we are still seeing services
across the board that have:
Warning: Journal has been rotated since unit was started. Log output is
incomplete or unavailable
basically created a script to check all unit file services/targets and
grep status -l for Journal, it is effecting a large range of
service.target's, service.service's and service.socket's
If we restart the service or reboot we no longer see those messages
about journal and everything appears to run as expected.
Thanks,
Brad
On 10/19/2017 04:13 PM, Steve Grubb wrote:
On Thursday, October 19, 2017 1:08:22 PM EDT Brad Zynda wrote:
>>> grep perm_mod /etc/audit/audit.rules
>>> -a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F auid>=1000
>>> -F auid!=4294967295 -k perm_mod
>>> -a always,exit -F arch=b32 -S chmod -S fchmod -S fchmodat -F auid>=1000
>>> -F auid!=4294967295 -k perm_mod
>>> -a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -F
>>> auid>=1000 -F auid!=4294967295 -k perm_mod
>>> -a always,exit -F arch=b32 -S chown -S fchown -S fchownat -S lchown -F
>>> auid>=1000 -F auid!=4294967295 -k perm_mod
>>> -a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -S
>>> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F
>>> auid!=4294967295 -k perm_mod
>>> -a always,exit -F arch=b32 -S setxattr -S lsetxattr -S fsetxattr -S
>>> removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F
>>> auid!=4294967295 -k perm_mod
>>>
>>> grep delete /etc/audit/audit.rules
>>> -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat
>>> -F auid>=1000 -F auid!=4294967295 -k delete
>>> -a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat
>>> -F auid>=1000 -F auid!=4294967295 -k delete
>>> -a always,exit -F arch=b64 -S init_module -S delete_module -k modules
>>> -a always,exit -F arch=b32 -S init_module -S delete_module -k modules
>>
>> These rules are well formed. Meaning no obvious holes that would cause
>> unintended events. The other ausearch/aureport commands I gave you should
>> show you what is causing the events and to which files. This information
>> may be used to create some kind of "never" rule that limits what gets
>> audited. If you do create some exclusion rule, it needs to be above the
>> perm_mod rules because audit is a "first match wins" system.
>>
>> -Steve
>>
>> -Steve
>
> Here is a peak report (at least in the last 24 hours) didnt include the
> file summaries as that would make this email HUGE:
Well, the idea is to look for something that's getting hit a lot. What it
sounds like is that things are getting deleted and modified quite a bit all
over the place. Does the executable report show a pattern such as one
application doing a lot? For example, with the rule you have, doing a yum
update will delete a whole lot of stuff and modify a whole lot of stuff.
Its possible to put exceptions in the rules so that one program does not flood
the logs. But looking at the quantities below, the audit system can easily
handle that.
Its also possible to exclude directories from auditing if the pattern is that
you have a daemon receiving and modifying files and then deleting them. You
should look at the executables & files and see if you can do with auditing
what they are doing because its not interesting.
If this is causing you problems on the journald side where its causing other
tasks to fail. Then I'd break the link between auditd and journald. The amount
of event you show is highish but well within range of what auditd can do. Just
make sure flush is set to incremental_async and freq is 100 or 200. You should
be OK.
-Steve
> Key Summary Report
> ===========================
> total key
> ===========================
> 8170 perm_mod
> 5390 delete
> 1073 access
> 56 time-change
> 26 session
> 12 privileged
> 7 logins
>
>
> Syscall Summary Report
> ==========================
> total syscall
> ==========================
> 4250 fchmodat
> 1613 chmod
> 831 fchmod
> 521 fchown
> 97 chown
> 12 setxattr
>
>
> Syscall Summary Report
> ==========================
> total syscall
> ==========================
> 2887 unlink
> 2189 rename
> 186 unlinkat
>
>
> so from the list the top 2 in each perm_mod and delete from the above
> list seem to be the culprits..