syslog performance: scaling up before scaling out
The other day I was reading a blog post on handling syslog at scale back on cribl.io’s blog. As you can imagine, syslog-ng has been used to solve syslog related challenges for a while now (24 years to be exact) and with that expertise I wanted to point out a few things in relation to that blog post. This might even become a series.
The blog post linked above, gives some advice how to scale syslog, in the section titled: Scaling Syslog the Right Way. Read the blog post for more details, but here’s my summary:
- place the receiver (e.g. Cribl Stream) right next to your log sources (e.g. in the same data center)
- scale out the receiver over many nodes so it becomes a cluster
- make sure to deploy a load balancer in front
What this means in practice is that you will need a sizeable infrastructure to consume the logs of a syslog producing device. Since my assumption is that all data centers have such appliances, you will need to deploy this infrastructure in each of your data centres (to be close to the sources).
While I can see that load balancing clusters to process log data are important in some scenarios, I don’t think this use case should be one of them. A single node, potentially in a failover High Availability setup should suffice.
There’s a choice between scaling out vs scaling up a workload. Cribl recommends scaling it out. My take is that it should be possible to scale it up before scaling it out.
syslog-ng has a bag of tricks to make it fast even on a single node, thereby reducing hardware costs and operational complexities and the need for a load balancer.
- it is implemented in C (compared to Cribl’s choice of TypeScript, fluentd and logstash are Ruby IIRC)
- it avoids/minimizes copying of data while processing them, its log routing implementation uses copy-on-write semantics for cases where multiple paths potentially change the message in parallel paths
- it avoids/minimizes memory allocations, in the simplest case it allocates 1 block of memory for 1 message
- it uses an efficient asynchronous architecture, using epoll and one thread per CPU core
- it uses a message representation where fields (aka: name-value pairs or properties) are stored in a packed block of memory, that is efficient to look up & serialize.
- it uses internal queueing mechanisms that avoids lock contention and allows back-pressure to be applied selectively
- it offers alternatives to regular expressions, as regexps are pretty slow to evaluate at volume
syslog-ng offers a domain specific language to route and manipulate messages, Cribl uses JavaScript.
I understand that this is an apples to oranges comparison. Cribl seems to have good UX. syslog-ng has good performance.
This is on my ~2018 laptop (Intel(R) Core(TM) i5-7440HQ CPU @ 2.80GHz, 4 cores), with a single destination writing all messages to disk, with syslog parsing enabled.
# single threaded sender
$ loggen -S -r 10000000 -s 300 –active-connections=1 -I 20 -P localhost 2000
average rate = 305506.81 msg/sec, count=6111711, time=20.0052, (average) msg size=304, bandwidth=90697.33 kB/sec
# sending on 10 threads
$ loggen -S -r 10000000 -s 300 –active-connections=10 -I 20 -P localhost 2000
average rate = 561537.95 msg/sec, count=11533347, time=20.5389, (average) msg size=304, bandwidth=166706.58 kB/sec
syslog-ng config:
@version: 3.38
log {
source { tcp(port(2000) log-iw-size(10000) log-fetch-limit(1000) flags(syslog-protocol)) ; };
destination { file("/install/foobar" log-fifo-size(10000)); };
};
Recent Comments