Subject: Re: [Syslog] Re: Last Call: draft-ietf-syslog-protocol (The syslog Protocol) to Proposed Standard

From: Balazs Scheidler <[email-masked]>
Date: Thu, 01 Feb 2007 10:57:17 +0000

On Thu, 2007-02-01 at 08:31 +1100, Mark Andrews wrote:
> > - 'The syslog Protocol '
> >    <draft-ietf-syslog-protocol-19.txt> as a Proposed Standard
> 
> 	draft-ietf-syslog-protocol-19.txt recommends using a reliable
> 	protocol.  Existing implementations of syslog do this and
> 	deadlock with nameservers which are logging via syslog.
> 
> 	I'm very wary of this recommendation.

I think this is not the fault of the new standard, but an operational
issue, that probably has a fix/workaround in implementations. (for
example syslog-ng permits disabling DNS, or latest versions can use a
list of internal IP->hostname mappings)

-- 
Bazsi

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: Sam Hartman <[email-masked]>
Date: Thu, 01 Feb 2007 08:09:29 -0500

>>>>> "Mark" == Mark Andrews <[email-masked]> writes:

    >> - 'The syslog Protocol ' <draft-ietf-syslog-protocol-19.txt> as
    >> a Proposed Standard

    Mark> 	draft-ietf-syslog-protocol-19.txt recommends using a
    Mark> reliable protocol.  Existing implementations of syslog do
    Mark> this and deadlock with nameservers which are logging via
    Mark> syslog.

Please explain the deadlock in more detail.  One of the primary
reasons for the syslog working group is reliable syslog, so I think we
need to focus on how to avoid the deadlock in other ways rather than
avoiding reliability.

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 1 Feb 2007 14:19:28 +0100

I am thinking on how to come this by on the protocol basis, but so far I
have no good clue (except that Baszi is probably right and it is a
purely operational issue). However, I know some problems in this
relation and though it is not a deadlock issue, I would like to share
the scenario for sake of completeness:

It occurs on a typical *nix system that runs both syslogd and the dns
daemon. So it could potentially occur on almost all *nix DNS servers.
The problem exists if syslogd contains rules for remote forwarding AND
these rules do contain hostnames which need to be resolved by DNS. A
typical problem is that syslogd starts before the dns daemon. When it
intend to send a message to the remote host, it must do a dns lookup,
which fails because dns daemon is not yet active. The stock linux
syslogd contains some logic which simply defers the connection request,
but at the cost of some message loss. Other ways to avoid this problem
is to use IP addresses in syslog configuration, host files or, as Baszi
said, special configuration options for static address mapping in
syslgod.

I do not, however, know actual deadlock situations (that is neither
syslogd nor dnsd can proceed because they are waiting on each other). I,
too, would appreciate any details on that.

Rainer

> -----Original Message-----
> From: Sam Hartman [mailto:[email-masked]]
> Sent: Thursday, February 01, 2007 2:09 PM
> To: Mark Andrews
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: [Syslog] Re: Last Call: draft-ietf-syslog-protocol (The
> syslogProtocol) to Proposed Standard
>

---

From: Sam Hartman <[email-masked]>
Date: Thu, 01 Feb 2007 18:17:11 -0500

>> >> - 'The syslog Protocol ' <draft-ietf-syslog-protocol-19.txt>
    >> as >> a Proposed Standard
    >> 
    >> reasons for the syslog working group is reliable syslog, so I
    >> think we need to focus on how to avoid the deadlock in other
    >> ways rather than avoiding reliability.
 
    Mark> 	nameserver logs to syslog.  syslog trys to resolve a
    Mark> address which requires the nameserver to succeed.  syslog()
    Mark> uses a reliable transport to talk to syslogd.  This pipe
    Mark> fills up.  syslog() then blocks waiting on syslogd which is
    Mark> waiting on the nameserver ....

This is mostly out of scope for the current document which focuses on
syslog communication between systems not within a system.

Additional ways to break the deadlock:

1) Provide syslogd with at least one non-locally-logging nameserver

2) Provide enough buffering that the probability you run into this is acceptable.

3) Use an asynchronous DNS library in syslogd

I do think a note about deadlocks would be useful in the protocol
document.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 02 Feb 2007 10:01:09 +1100

> need to focus on how to avoid the deadlock in other ways rather than
 
	nameserver logs to syslog.  syslog trys to resolve a address
	which requires the nameserver to succeed.  syslog() uses a
	reliable transport to talk to syslogd.  This pipe fills up.
	syslog() then blocks waiting on syslogd which is waiting
	on the nameserver ....

	There are two way to break this.
	1. use a lossy transport
	2. don't attempt to resolve names/addresses in syslogd.

	You can reduce the problem by always logging using
	IP addresses.  However it doesn't remove the problem
	completely as you still need to look up addresses
	for forwarding purposes.

	Similarly if syslogd is using a reliable transport
	to talk to another syslogd.  That too can block which
	will eventualy lead to blockages to applications /
	memory exhaustion.
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: "Rainer Gerhards" <[email-masked]>
Date: Fri, 2 Feb 2007 15:24:25 +0100

> > >>>>> "Mark" == Mark Andrews <[email-masked]> writes:
[snip]

> 	will eventualy lead to blockages to applications /

*That* is a different beast, not yet discussed. I know that it exists
but it is not related to DNS. If it happens, it is really bad and
typically results in a complete system hang. There are some situations
where a lossy transport is preferable. For example, I have written a
syslog/TCP implementation which, if forced to run in single-threaded
mode, favours discarding messages over blocking as the later could
indeed lead to fatal problems on a typical linux system.

The root cause, however, is not reliable syslog per se. The root cause
is the implementation. A reliable syslogd actually needs to be
implemented with a non-blocking, de-coupled, buffered design. In almost
all cases, that means separate receiver threads, a to-be-processed
message queue and separate sender threads. Still, you have to decide
what to do if the queue size is exhausted. But that scenario is much
less likely. In that case, I'd still favour loosing some messages over
potentially loosing all AND the system the syslogd runs on. As far as I
see it, this is an app-layer issue out of scope for the underlying
protocol. The problem, of course, is that syslog is simplex and
hop-to-hop, so the original sender will never know that the message was
lost. Protocol-wise, I do not see any fix to that except integrating
some asynchronous notification mechanism. IMHO, this is outside the
scope of syslog.

It may be useful to add some hint to implementors pointing to this
blocking condition. But I am not sure if it is really justified.

Rainer