Subject: [Syslog] FW: draft-ietf-syslog-protocol: "Reliable delivery considered harmful."

From: "David Harrington" <[email-masked]>
Date: Wed, 7 Feb 2007 12:55:59 -0500

Forwarded from [email-masked] mailing list.
Please ensure that responses also go to the [email-masked] list.

dbh

-----Original Message-----
From: David W. Hankins [mailto:[email-masked]] 
Sent: Thursday, February 01, 2007 1:09 PM
To: [email-masked]
Subject: draft-ietf-syslog-protocol: "Reliable delivery considered
harmful."

On Thu, Feb 01, 2007 at 08:09:29AM -0500, Sam Hartman wrote:
> >>>>> "Mark" == Mark Andrews <[email-masked]> writes:
>     >> - 'The syslog Protocol ' <draft-ietf-syslog-protocol-19.txt>
as
>     >> a Proposed Standard
> 
>     Mark> 	draft-ietf-syslog-protocol-19.txt recommends using a
>     Mark> reliable protocol.  Existing implementations of syslog do
>     Mark> this and deadlock with nameservers which are logging via
>     Mark> syslog.
> 
> 
> Please explain the deadlock in more detail.  One of the primary
> reasons for the syslog working group is reliable syslog, so I think
we
> need to focus on how to avoid the deadlock in other ways rather than
> avoiding reliability.

If you have 50,000 syslog lines to put out, and only enough
network/disk/something bandwidth for 5,000 within the same time
frame, that's a problem.

If you insist on keeping all 50,000 lines of output, there is no
solution to that problem.  If you block, that's a big problem as
it ultimatley totally disables the service attempting to log
information.  If you write to a growing backing store, well you'll
run out of space eventually (even disk is not infinite).
Compression can only get you so far.

Ultimately you have to drop something, and if it's not the
syslog output, it's the service's output.

Reliability is the problem, and the advice we give our users is
not to log reliably (or even to configure servers to log to a
local file, circumventing syslog entirely).

Note that reliable network transport of syslog information is
equally damaging as reliable local storage of syslog output (eg
by forcing disk synchronization of each line).  At least one
of today's syslog implementations insists that you prefix your
filenames with a "-" if you /don't/ want it to fsync() every
line.  This default cripples DHCP servers.

The current words in the draft;

   It may be desirable to use a transport with guaranteed delivery to
   mitigate congestion.

May be adequate to the point of suggesting that reliable delivery
might not be desirable.  But on the whole the draft doesn't read
that way, and it doesn't state the truth: reliable delivery of
syslog output is always harmful.  The point of bothering with
reliable syslog delivery, if there is one, is for those very
rare cases where losing the data is more harmful than harming
system services.

-- 
David W. Hankins	"If you don't do it right the first time,
Software Engineer		you'll just have to do it again."
Internet Systems Consortium, Inc.	-- Jack T. Hankins
_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: "David Harrington" <[email-masked]>
Date: Wed, 7 Feb 2007 13:01:02 -0500

Forwardede from [email-masked]; please include [email-masked] in
responses.

dbh

-----Original Message-----
From: Pekka Savola [mailto:[email-masked]] 
Sent: Thursday, February 01, 2007 3:30 PM
To: David W. Hankins
Cc: [email-masked]
Subject: Re: draft-ietf-syslog-protocol: "Reliable delivery considered
harmful."

> Compression can only get you so far.

As an operator, I would find a reliable syslog useful.

However, I do not see this as a problem.  95% of the problem (from our

perspective) is solved if no messages are lost _on the wire_ between 
the sender and the recipient of syslog messages.

It's acceptable for the syslog sender to replace overflowing lines of 
syslog (if some messages need to be dropped due to lack of resources) 
with a message about rate-limiting, messages being dropped, or 
whatever -- just the same way as messages might get dropped when 
syslogging to a local media.

As long as a) there is a message about syslogs getting dropped, and b)

this is infrequent in a well operated system (i.e. the system log 
levels are set so that typically the amount of logging is OK), this 
should be no problem.

> rare cases where losing the data is more harmful than harming

IMHO, "reliable delivery of syslog output is always harmful" is very 
much an over-statement.

It seems your concern is limited to the corner case where the syslog 
sender would already have a full syslog buffer, and the sender would 
get even more syslog to send. While this is a serious problem when it 
occurs, it should be easily solvable: just drop the messages (with a 
suitable note in syslog or in the local log) that exceed the buffer 
size or prune messages from the buffer using some more advanced 
strategy.

As a particular example, we'd be very interested in getting reliable 
syslog from our routers to cover for messages lost due to network 
outages (fixed usually quickly with rerouting). We are talking of 
1-200 messages being lost within a 10 minute period -- a very 
reasonable packet rate in average.  A 1,000 or 10,000 line buffer in 
the router should be very reasonable in recent control processors. 
I'd rather the control processor reserve 0.1% of its memory (or CPU) 
to store and transmit these messages rather than try to use the last 
0.1% when it's already chugging at 99.9%; the last 0.1% would not make

a meaningful difference anyway for whatever it's using almost all of 
its resources.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

_______________________________________________
Ietf mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf

---

From: "David Harrington" <[email-masked]>
Date: Wed, 7 Feb 2007 13:16:43 -0500

Forwarded from [email-masked]; please include [email-masked] in any
responses.

dbh 

-----Original Message-----
From: Harald Tveit Alvestrand [mailto:[email-masked]] 
Sent: Sunday, February 04, 2007 3:44 PM
To: David W. Hankins; [email-masked]
Subject: Re: draft-ietf-syslog-protocol: "Reliable delivery considered
harmful."

Daring to rush in without having read the documents....

it seems to me that somewhere one needs a NOTE, something along the
lines 
of:

NOTE: In some situations, for instance when a destination disk is full
or 
damaged, a syslog facility may be unable to process all messages,
despite 
the message transport being reliable. In such a case, it is reasonable
for 
the logger of a message to have the option of either not logging more 
messages or ceasing its own operation. This document does not specify
which 
option to take.

Or words to that effect.

                  Harald

--On 2. februar 2007 09:59 -0800 "David W. Hankins"
<[email-masked]> 
wrote:

> On Fri, Feb 02, 2007 at 08:31:49AM +0100, Stephane Bortzmeyer wrote:
>> Wether it is a bug or a feature depends on your requirments. On
some
>> high-security environments, people prefer to suspend the service
>> rather than not being able to log it. (Otherwise, an attacker could
>> easily attempt many attacks, fill in the hard disk and then perform
>> the real attack unlogged).
>
> I'd just like to point out that you're choosing one bug over
> another.  A DOS in preference to lack of observance of events.
>
> In my opinion, that's a bad selection, but it's your selection to
> make.
>
> That kind of preference, that kind of choice, is a good thing to
> have, but it would be unwise to apply to the general case a
> systematic selection of DOS over observation.
>
> --