Subject: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX?

From: Jeroen Massar <[email-masked]>
Date: Tue, 15 Feb 2011 12:17:28 +0100

Hi,

As the subject states, for both this cloud[1] and CEE[2] proposals, why
not use IPFIX instead for structured logging data!?

Greets,
 Jeroen

[1] http://www.ietf.org/id/draft-cloud-log-00.txt
[2] http://cee.mitre.org/

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 16 Feb 2011 11:35:07 +0100

On 2011-02-16 06:21, Heinbockel, Bill wrote:
> From what I understand, IPFIX is for expression of IP flows from network sensing
> devices.

For a short bit forget about the history of IPFIX, it indeed comes from
NetFlow, and thus is used quite in a network centric way, but
effectively it is a structured streaming data format.

> Could you please explain how IPFIX is relevant to event and cloud logging data?
> I understand how CEE and IPFIX may overlap for describing networking events, but
> it is unclear to me how IPFIX could handle things like Windows Event Logs and
> RHEL audit logs.

There are two parts to IPFIX: Templates + Data

The template describes how the data looks like, for instance, lets take
an Apache CLF log entry:

66.249.66.174 - - [16/Feb/2011:10:48:11 +0100] "GET /robots.txt
HTTP/1.1" 200 2629 "-" "Googlebot-Image/0"

We can make an IPFIX template for that

[
	{4, IPv4_SRC },
	{4, TIMESTAMP},
	{4, HTTP_METHOD},
	{v, URL},
	{v, HTTP_PROTOCOL},
	{2, HTTP_RESULT},
	{8, OCTETS},
	{v, HTTP_REFER},
	{v, HTTP_USERAGENT},
]

The 'v' markers indicate variable fieldlengths, the others indicates the
number of bytes such a field takes. The data is then just encoded in the
above format, presto.

The above is a simple example, one can also have repeating lists and of
course you could make a variable template which just includes the fields
that you actually want to look at or you could already do some
aggregation and add other fields. Templates are only sent every now and
then, as they should not change. The data is the important bit.

The fieldnames are actually numbers in the data, thus very compact, and
are mapped to descriptions, data types etc, per a nice XML file
 http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt for
a more human readable version ;) for the official IANA list and with the
help of Enterprise IDs any others can easily be added.

The big advantage is that you can more or less do static templates if
you want and you only need one single parser on the collector side, thus
one does not have to create another parser and collector again for
decoding other protocols, just one, the IPFIX one, and you can optimize
that really well for all kinds of scenarios.

Greets,
 Jeroen

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 16 Feb 2011 11:39:43 +0100

The SIP CLF WG has just recently rejected IPFIX for it being binary and
chosen indexed ASCII instead for their format. Their reasoning (after a long
struggle) is probably educating:

http://www.ietf.org/mail-archive/web/sip-clf/current/msg00364.html

I don't think that IPFIX is a good solution *in the syslog context*. It is
very far from what people expect. Other than that, I'd probably need to
re-iterate the arguments made on the SIP CLF mailing list, so it probably is
better to refer to their archive ;)

Rainer

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Jeroen Massar
> Sent: Wednesday, February 16, 2011 11:35 AM
> To: Heinbockel, Bill
> Cc: Sam Johnston; [email-masked]; [email-masked]
> Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX?
> 
> _______________________________________________
> Syslog mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/syslog

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 16 Feb 2011 11:56:41 +0100

> better to refer to their archive ;)

Why would they expect anything about the *DATA* format of a protocol?

Note that the whole point that IPFIX (or any other structured data
format for that matter) 'solves' is that one has to make a parser for
every single log file format out there. Doing this at the meter tends to
be cheaper due to the ability to distribute that than at the aggregated
part. (then again sFlow as an example does it exactly the other way
around, just pushing packets and letting the collector do the hard
parsing part, but we are talking about sampled flows here thus you will
miss out on events which is not a decision you can make at the meter if
you are looking at say breaking attempts or failures ;)

I think the pro-ascii versus binary argument comes effectively primarily
from organizations who process large amounts of variable-string ascii
data already and who do not really care about a few extra bits or a bit
more overhead in processing data as they have large global clusters of
hosts already doing that work. Their programming languages tend to be of
a scripted-style too which tend to make it harder / less efficient to
work on binary data but work great with ascii-alike data.

Nevertheless, I've a generic logline parser which simply converts syslog
and other log file formats into IPFIX. The problem with the whole ascii
thing though is that one has to teach the parser what fields are what,
and in the case of for instance the Apache CLF teach it the weird
delimiters that are present. These are all special cases, something that
one would really like to avoid if one wants to keep it speedy.

My model partially solves that as I only have to do the special casing
at the edge, where the log file gets converted into IPFIX. As those are
considered 'meters' I just deploy more and more of those, while I can
keep the collector side generally either a single box and otherwise
easily distribute the data amongst them.

And of course, the conversion goes the other way too, it can spit out
reformatted 'ascii' again if needed.

Greets,
 Jeroen

 (who finds it funny to see ASCII btw, as there is this thing called
  UTF-8 that makes it possible to express things in all languages of
  the world. I guess those people have to live with punycode etc...)

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 16 Feb 2011 12:34:06 +0100

Well I really don't like to restart that discussion in this context here
again, but let me note that what you are doing with your converter is very
useful. It actually normalizes data into a canonical format. This is
something that CEE tends to do, but in a protocol agnostic format (and I do
similar things in my projects as well). The general utility is unquestioned.
The question is if such an effort must be bound restricted to a single
protocol. My PoV is that this is counter-productive.

You definitely have a point in that IPFIX may be superior than syslog in many
regards. I do not intend to argue against this. But often a simpler solution
is able to draw more attention, and thus deployments, than a (potentially or
actually) technical superior one (shouldn't we all use the OSI stack by now,
just as one example...). 

I don't think it is useful to include IPFIX in syslog. But it may be an
option that IPFIX makes syslog obsolete. I think you should take that later
route.

But as I said -- I do not intend to spawn another iteration of this lengthy
discussion. It has occurred sooo often in the past years.

Rainer

PS: You are right in one more thing "ASCII" is the wrong term. Most folks
(including me) seem to be sloppy and say ASCII when they actually mean
printable text data, of course including UTF-8.

> From: Jeroen Massar [mailto:[email-masked]]
> Sent: Wednesday, February 16, 2011 11:57 AM
> To: Rainer Gerhards
> Cc: Heinbockel, Bill; Sam Johnston; [email-masked]; [email-masked]

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 16 Feb 2011 13:54:16 +0100

On 2011-02-16 12:34, Rainer Gerhards wrote:
[..]
> protocol. My PoV is that this is counter-productive.

I might be missing the parsing here, but can you explain what part is
counter-productive to have a single protocol and having to write only a
single parser versus having a lot of different protocols who require
parsers which can resolve ambiguity?

> actually) technical superior one (shouldn't we all use the OSI stack by now,

I wonder what the "E" in IETF stands for if I see the above statement.
Why not simply use CSV or hey just pack it in XML! :)

Please note that IPFIX has gathered quite some attention already.

As for the OSI stack argument, the world is not ready for IPv6 either.

I like to also quote:
http://www.ietf.org/mail-archive/web/sip-clf/current/msg00430.html

8<-----------------------------------------------------------------------------
1. Shall we use a TAB or a SPACE (or any LWS) as field delimiters?
Considerations and points raised:

- TABs don’t survive Telnet or web pages very well, especially when
copy/pasting.
- spaces can appear inside fields (as can most other delimiters we choose)
- how do TAB and SPACE delimiters interact with shell tools (rep, cut, awk)
----------------------------------------------------------------------------->8

The moment you have to worry about a delimiter, you have big
problems.... that is why char format is not a good idea. Yes, you can
use all the standard unix command tools to process them, but are you
really going to do this on the big scale? I certainly hope not.

The same with Apache logs, very useful to have to parse the IP address
again to get the information you want. Especially with IPv6 where the
address can have quite a number of formats based on how the address is
represented, not even going talking about the point.

Oh and those unix tools, they do not support UTF-8 in a lot of cases,
thus they are suddenly quite useless.

> option that IPFIX makes syslog obsolete. I think you should take that later

If there is going to be new effort for syslog, is that then not the
route it should be taking? Structured data is so much more useful than
unstructured data, and that is the big problem that

> discussion. It has occurred sooo often in the past years.

Is there a summary of this discussion with a clear listing of pro/cons?
Would be good to have that in a draft, is there one? If there isn't
please write one with those arguments so that it can referred to instead
of stating 'we did this before, EOD'...

Greets,
 Jeroen

---

From: "Heinbockel, Bill" <[email-masked]>
Date: Thu, 17 Feb 2011 11:35:19 -0500

Thanks for that, I did not realize that IPFIX had been extended beyond its netflow 
past.

I don't have the time now, but I am interested in looking into it further. It does 
kind of remind me of ASN.1/SNMP, where we need to worry about the names/OID 
translation

Also, a lot of vendors and users seem to prefer the ease of text-based protocols 
like Syslog for logging. I am not saying this is good or bad, but it seems to be 
the sweetspot -- binary is not natively human readable and XML has too much 
overhead.

William Heinbockel
The MITRE Corporation

>Sent: Wednesday, 16 February 2011 05:35
>To: Heinbockel, Bill
>Cc: [email-masked]; Chris Lonvick; Gene Golovinsky; Sam Johnston; Common
>Event Expression

---

From: Dan Schlitt <[email-masked]>
Date: Thu, 17 Feb 2011 12:20:58 -0500

On this topic it might be well to look at Tom Limoncelli's article in 
the February Communications of the ACM. He is highly respected in the 
system administrator community and speaks for many of us.

/dan

-- 

Dan Schlitt
[email-masked]

---

From: Gene Golovinsky <[email-masked]>
Date: Thu, 17 Feb 2011 12:47:33 -0600

Is there a link to the article?

--Gene

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Dan Schlitt
Sent: Thursday, February 17, 2011 11:21 AM
To: Heinbockel, Bill
Cc: Sam Johnston; Event Expression; [email-masked]; [email-masked]
Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX?

On this topic it might be well to look at Tom Limoncelli's article in the February Communications of the ACM. He is highly respected in the system administrator community and speaks for many of us.

/dan

-- 

Dan Schlitt
[email-masked]

On Thu, 17 Feb 2011, Heinbockel, Bill wrote:

> Thanks for that, I did not realize that IPFIX had been extended beyond 
> its netflow past.
>
> I don't have the time now, but I am interested in looking into it 
> further. It does kind of remind me of ASN.1/SNMP, where we need to 
> worry about the names/OID translation
>
> Also, a lot of vendors and users seem to prefer the ease of text-based 
> protocols like Syslog for logging. I am not saying this is good or 
> bad, but it seems to be the sweetspot -- binary is not natively human 
> readable and XML has too much overhead.
>
>
> William Heinbockel
> The MITRE Corporation
>
>
>> Cc: [email-masked]; Chris Lonvick; Gene Golovinsky; Sam Johnston; 
>> Common Event Expression
>>> From what I understand, IPFIX is for expression of IP flows from 
>>> network
>> sensing
>>> devices.
>>
>> For a short bit forget about the history of IPFIX, it indeed comes 
>> from NetFlow, and thus is used quite in a network centric way, but 
>>> Could you please explain how IPFIX is relevant to event and cloud 
>>> logging
>> data?
>>> it is unclear to me how IPFIX could handle things like Windows Event 
>>> Logs
>> and
>>> RHEL audit logs.
>>
>> The template describes how the data looks like, for instance, lets 
>> take an Apache CLF log entry:
>>
>> The 'v' markers indicate variable fieldlengths, the others indicates 
>> the number of bytes such a field takes. The data is then just encoded 
>> in the above format, presto.
>>
>> The above is a simple example, one can also have repeating lists and 
>> of course you could make a variable template which just includes the 
>> fields that you actually want to look at or you could already do some 
>> aggregation and add other fields. Templates are only sent every now 
>> and then, as they should not change. The data is the important bit.
>>
>> The fieldnames are actually numbers in the data, thus very compact, 
>> and are mapped to descriptions, data types etc, per a nice XML file 
>> http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt 
>> for a more human readable version ;) for the official IANA list and 
>> with the help of Enterprise IDs any others can easily be added.
>>
>> you want and you only need one single parser on the collector side, 
>> thus one does not have to create another parser and collector again 
>> for decoding other protocols, just one, the IPFIX one, and you can 
>> optimize that really well for all kinds of scenarios.
>>
>> Greets,
>> Jeroen
>