Re: Handling -ENOBUFS
On Wednesday 05 November 2008 11:30:16 Lucas C. Villa Real wrote:
I'm facing a situation where -ENOBUFS is returned from both
audit_send() and audit_get_reply(). The system is under high stress,
with 250k files being created and having creat() and chmod() syscalls
audited.
Is this what you really wanted to audit? :)
Looking the code at lib/netlink.c, I saw that audit_send()
doesn't
handle -ENOBUFS. Would it be possible to replace the condition from
"while (retval < 0 && errno == EINTR)" to "while (retval < 0
&& (errno
== EINTR || errno == ENOBUFS))" to fix the problem when sending
packets from userspace to kernel?
Have you tried that? Does it fix the problem or just hang the utility?
My understanding for the problem in audit_get_reply() is that the
I/O
buffers are all full and auditd was just not scheduled at the expected
rate, causing these buffers to overflow. Does that make sense?
If you go over the backlog limit, you get a syslog message about that unless
you have it set to ignore. My guess would be that you have a general network
memory pool depletion and is not related to audit specifically.
If it does, do you have a suggestion about the best way to approach
this
problem, besides changing auditd's priority?
Increase the backlog and increase auditd's priority. I have not played with
running auditd with a different scheduler policy than whatever the default
is. But you may want to see if one of the other scheduler polices treat audit
better. or maybe you want to tune /proc/sys/kernel/sched_granularity_ns.
One interesting thing which I noticed is that 'auditctl -s'
doesn't
report that messages were lost,
They weren't lost by the audit system so it doesn't know they didn't arrive.
This is happening with an old kernel, 2.6.16.46 + a bunch of
patches,
and audit 1.7.4. I cannot completely upgrade it to a new release, but
I can certainly backport audit specific bits if you remember having
fixed something similar since then.
Well, that proc tunable is only available for the CFS scheduler. Not sure what
you have for older kernels.
-Steve