Re: Systemd Journald and audit logging causing journal issues

Hey Steve, No problem you guys are busy with updates.. So I kind of stepped into a known issue with a current disagreement between the 2 maintainers?

what can be done to resolve this going forward as it is killing services in production environments?

I agree with the need not to remove auditing as this is a slippery slope and should not occur but the decision was based on little documentation in regards to the problem and loss of service, I will look at further checking in hopes to find the specific rule. Though I think the latter to fix the issue is the appropriate avenue.

The rules have been put in place across many organizations that check with tools like CIS-CAT and OSCAP, so a lot of rules and a point of possible single failure.

In regards to the audit.socket what is the expected outcome of masking this service?

On 10/17/2017 11:25 AM, Steve Grubb wrote: > Hello, > > I apologize for the late reply...just found the message. > > On Monday, October 2, 2017 1:30:19 PM EDT Brad Zynda wrote: >> I am sending along an issue brought to the systemd-journald dev list >> initially: >> >> On 10/02/2017 11:40 AM, Lennart Poettering wrote: >>> On Mo, 02.10.17 11:25, Brad Zynda (bradley.v.zynda(a)nasa.gov) wrote: >>>> Sep 28 13:50:03 server systemd-journal[565]: Suppressed 73244 messages >>>> from /system.slice/auditd.service >>> >>> The question is: why does auditd even log to the journal? > > It doesn't. I have had many arguments with the systemd people about > polluting syslog with audit events. If we wanted audit events there, we > would have wrote them there. The journal is listening on a multicast > audit socket that was created just for them and using a posix capability > that was created just for them. And journal also turns on auditing even > if you didn't want it. In short, they have, with intention, created your > problem. > >>>> Now we are required to have full audit rules and does this look like at >>>> rate limiting issue or an issue of journal not able to handle the >>>> traffic to logging? >>> >>> journald detected that it got flooded with too many messages in too >>> short a time from auditd. if this happens then something is almost >>> certainly off with auditd, as auditd is not supposed to flood journald >>> with messages, after all it maintains its own auditing log database. > > No...that's the way it works. If you want the audit stream, you have to be > able to handle it. My suggestion is that we have a separation of duties. > Auditd has audit events, journal has syslog. Besides, mixing audit and > syslog data means the security officer and system admin roles have been > combined. I think there is an audit.socket unit file that can be masked. > >>> Please ping the auditd folks for help > > They created the problem of audit events in syslog. That said, its been my > experience that whenever you get lots of events, there may be something > wrong with your rules. > > The normal technique to figure out what wrong is to run aureport --summary > --key during the time range of the flood to see what rule is triggering. > Then we can look at that rule to see if there's something wrong with it. > > More below... > >> Hey Everyone, >> >> Not sure if this is a bug so: >> >> systemctl status -l systemd-journald.service >> ● systemd-journald.service - Journal Service >> >> Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; >> >> static; vendor preset: disabled) >> >> Active: active (running) since Tue 2017-09-26 20:01:16 UTC; 5 days ago >> >> Docs: man:systemd-journald.service(8) >> >> man:journald.conf(5) >> >> Main PID: 565 (systemd-journal) >> >> Status: "Processing requests..." >> CGroup: /system.slice/systemd-journald.service >> >> └─565 /usr/lib/systemd/systemd-journald >> >> Sep 28 13:50:03 server systemd-journal[565]: Suppressed 73244 messages >> from /system.slice/auditd.service >> Sep 28 13:51:03 server systemd-journal[565]: Suppressed 98979 messages >> from /system.slice/auditd.service >> Sep 28 13:52:03 server systemd-journal[565]: Suppressed 109433 messages >> from /system.slice/auditd.service >> Sep 28 13:53:03 server systemd-journal[565]: Suppressed 99788 messages >> from /system.slice/auditd.service >> Sep 28 13:54:03 server systemd-journal[565]: Suppressed 111605 messages >> from /system.slice/auditd.service >> Sep 28 13:55:03 server systemd-journal[565]: Suppressed 111591 messages >> from /system.slice/auditd.service >> Sep 28 13:56:03 server systemd-journal[565]: Suppressed 107947 messages >> from /system.slice/auditd.service >> Sep 28 13:57:51 server systemd-journal[565]: Suppressed 32760 messages >> from /system.slice/auditd.service >> Sep 28 17:21:40 server systemd-journal[565]: Suppressed 210 messages >> from /system.slice/auditd.service >> Oct 01 02:16:01 server systemd-journal[565]: Suppressed 1333 messages >> from /system.slice/auditd.service >> >> journalctl --verify >> PASS: /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system.journal >> PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-000000000097f6c7-0005596b745b4d1c.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-000000000096a587-00055966f35ae59a.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-00000000009554f1-000559629c4cdb7e.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-0000000000940591-0005595e1811a2d1.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-000000000092b500-00055959f2de5ede.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-0000000000916479-0005595573137b74.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-0000000000901337-00055950d80cc3d8.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-00000000008ec2fb-0005594cad14b07a.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-00000000008d7373-0005594838683e58.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-00000000008c238e-00055943fe2072e3.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-00000000008ad1d9-0005593ff64a4f69.journal PASS: >> /run/log/journal/d28b0080ffe0432a974f36e4fb4bfa9b/system@0d49221d68d04ef0 >> b95 d8203c5e96a46-0000000000897f32-0005593e18c5758b.journal >> >> >> journalctl --disk-usage >> Archived and active journals take up 1.1G on disk. >> >> >> Initially we saw: >> 16733 PATH >> 5070 SYSCALL >> 5024 CWD >> 3765 AVC >> 323 CRYPTO_KEY_USER >> 223 USER_START >> 222 USER_ACCT >> 222 CRED_ACQ >> 220 LOGIN >> 220 CRED_REFR >> 218 USER_END >> 218 CRED_DISP >> 46 USER_LOGIN >> 12 EXECVE >> 4 USER_AUTH >> 2 CRYPTO_SESSION >> 1 USER_ROLE_CHANGE >> 1 USER_CMD >> 1 SERVICE_STOP >> 1 SERVICE_START >> 1 BPRM_FCAPS >> >> so we blocked type PATH in audit.rules > > This is not the right thing to do. If a security officer asks what is > being > accessed, you got rid of the information. The right thing is to figure out > which rule is being hit and see if something is wrong with it. For > example, I have seen people do this: > > -a always,exit -S open,openat -F exit=-EPERM > > The problem is that they did not restrict the rule an architecture and > they > were getting lots of events for the wrong syscall. I've also seen people > add -F success 0 to an open syscall. This also results in a large number > of events. > > So, I'd recommend making sure all rules have keys added and the running > the > key summary report to see what rule needs inspection. > > If you find the rule that's causing the problem and you want an opinion, > send it to the mail list. > > -Steve > >> But we are still seeing 100K of dropped/suppressed messages. >> >> Note: systemloglevel = INFO >> >> Centos 7 1708 3.10.0-693.2.2.el7.x86_64 >> >> systemd.x86_64 219-42.el7_4.1 >> >> >> Now we are required to have full audit rules and does this look like at >> rate limiting issue or an issue of journal not able to handle the >> traffic to logging? >> >> Error we are seeing from services that have silently failed, in this >> case glassfish.. >> >> systemctl status -l glassfish >> ● glassfish.service - SYSV: GlassFish start and stop daemon >> >> Loaded: loaded (/etc/rc.d/init.d/glassfish; bad; vendor preset: >> disabled) >> >> Active: active (exited) since Tue 2017-09-26 20:01:36 UTC; 5 days ago >> Docs: >> man:systemd-sysv-generator(8) >> >> Process: 1328 ExecStart=/etc/rc.d/init.d/glassfish start (code=exited, >> >> status=0/SUCCESS) >> >> Warning: Journal has been rotated since unit was started. Log output is >> incomplete or unavailable. >> >> Eventually glassfish will fail but it wont kill the service so we never >> get an nms service down trap from the OID. >> >> Please let me know if further info is needed or if certain limits need >> to be adjusted. >> >> Thanks, >> Brad Zynda >> >> >> -- >> Linux-audit mailing list >> Linux-audit(a)redhat.com >> https://www.redhat.com/mailman/listinfo/linux-audit