Select Page

syslog-ng relaunch

by | Jan 19, 2022 | background | 14 comments

syslog-ng has been around for decades: I started coding the first version of syslog-ng in September 1998, circa 24 years ago. The adoption of syslog-ng skyrocketed soon after that: people installed it in place of the traditional syslogd across the globe. It was packaged for Debian, Gentoo, SUSE and even commercial UNIXes. It became a default logging daemon in some of these Linux distributions. Commercial products started embedding it as a system component. Over the years however I feel that syslog-ng has become a trusted piece of infrastructure, few people really care about. I set out to change that.

The use of syslog-ng has become so widespread and dominant, needing minimal maintenance, that after a point, people stopped noticing its existence. It became like the printer sitting in an office corner: we know it’s there, we use it regularly, we appreciate the function but we don’t really know or care about the details or the brand providing us with given service.

I see syslog-ng regularly in this spot today: its deployment might have been a big project in its time with its own challenges, but it has been a solved problem ever since.

Not that log management and log processing would be a static, boring field of IT & IT Security. Like all other fields of enterprise IT, there’s been tremendous activity in the last 10-15 years.

Markets and relevant trends:

  • SIEM & User Behavior Analytics(LogLogic, ArcSight, QRadar, Splunk, …)
  • Big Data (Hadoop, Kafka, Storm, Spark, NiFi)
  • Enterprise SaaS services (Office365, Google Workspace, etc.)
  • Containers and orchestration (Kubernetes, OpenShift, cloud & on-prem)
  • Cloud Native Applications

All these changes naturally resulted in an equal frenzy in the tools processing and managing log data. New tools and services emerged, old tools gained new features. I could probably go on and get into details on these trends but that’s not why I am here today.

I started this blog as I wanted to show two things:

  1. That syslog-ng has not been the stoic figure in the corner and has incorporated important improvements over the years that are not widely known and unfortunately not even assumed.
  2. To solicit feedback on my future plans and with that help guide the development of syslog-ng to the future.

The intent behind this blog is to address the 2nd point.

The first point might sound a little strange at first: if there are indeed functionality in syslog-ng that its users don’t know or care about, that can only mean one of two things:

  1. Those features were not needed in the first place.
  2. The marketing/communication of syslog-ng as a project has not been very good.

As one of the engineers behind the changes I firmly believe #1 is not true. The features we added to syslog-ng over the years are important. I believe these features enable syslog-ng to address problems that only few people assume it could address. But I am not here to go into details on those features either.

My take on the marketing issue is different: other projects, open source or commercial, have been better at communicating their value propositions. They were more successful at communicating their release-by-release improvements and with that gained a more significant traction in the marketplace.

The reason behind this failure is an entire post on its own (let me know if you are interested!), my short and simple summary is a single word: focus.

I am the founder of the syslog-ng project. I founded a company that sponsored the syslog-ng project. But neither my or my company’s primary focus has ever been syslog-ng. Some of you may remember that syslog-ng was hosted on balabit.com. Balabit was a player in the Privileged Access Management space (e.g. the likes of CyberArk, BeyondTrust, e-DMZ, Wallix etc). Albeit we made an effort to combine log management with PAM, but truth be told we never really succeeded in doing so. syslog-ng grew from being my personal hobby to become the 2nd product in the Balabit portfolio.

This situation handicapped syslog-ng compared to those projects and companies that had logs as their primary focus.

Balabit was acquired 4 years ago: I spent my sabbatical, I learnt a couple of new hobbies (electronics mainly, welding is something I still want to learn), implemented home automation in my house (see http://bazsi.blogspot.com/), became a hobby angel investor and a management consultant. With all that I am somewhat bored. I love spending time with my family all these new things, but at the same time I need new challenges. There are too many “small” things I spend my time with and I have an itch to do something “bigger”.

I want to give syslog-ng a chance it never had: I want to make it my primary focus. The foundations and the technology are already there, let’s put the spotlights on, blow the dust off. Engage with users, understand their needs and communicate value. Understand things that are missing and fix them.

In a nutshell, I would like to relaunch syslog-ng as a project. Let’s reboot the process that keeps a product able to adapt to a changing market and continue to be relevant for more decades to come.

I am inviting you to be a part of it. Feedback, new use cases, feature requests and even bug reports are welcome. Strong points that you like, weak spots that you would like to see improved are very interesting.

Subscribe below and help me in this endeavour.  Stay tuned!

Subscribing to this blog shows interest, interest brings motivation, motivation brings features and bugfixes to syslog-ng. Please show your appreciation and interest by subscribing. Thanks.

14 Comments

  1. In our infrastructure, syslog-ng is responsible for collecting syslog from our Cisco routers and UNIX systems (different flavours), some apps. We then use syslog-ng filters to sort the incoming data into various local files in the filesystem.

    This data is then polled by Splunk Heavy Forwarders and transferred to a local Splunk we use as an SIEM.

    We maintain the syslog-ng config file manually, which has grown quite large over time. It has become a challenge to navigate in it. The syntax is something that we need to teach to newcomers.

    This is an RHEL7 system and we are using syslog-ng 3.5.6 from EPEL. We have 1-2k msg/sec.

    We are generally happy, syslog-ng just works. It would be great if the maintenance of the configuration could be simplified, especially to less experienced staff.

    Reply
    • My use of syslog-ng is roughly equivalent, though in a home lab. I suggested its use in my workplace once in exactly this capacity but the boss went with rsyslog as he was more familiar with it. I found that rsyslog was just as straightforward for this purpose so that was ok. Neither is particularly friendly to new starters so improving the accessibility to newbies would be one way of making it a preferred choice over rsyslog.

      Having said that, in this specific use case I do wonder how much the containerized Splunk Connect 4 Syslog package will negate the need for learning either? I haven’t had a chance to use it yet but it looks promising.

      Obviously that only applies to people using Splunk specifically. A more approachable syslog-ng setup is still a good thing for everyone else! p.s. thank you Balázs for making it in the first place, it has served me well 🙂

      Reply
      • Thanks for your feedback and sorry it took me so long to answer. I once posted an answer but WordPress ate it.

        Good point on being more user friendly to newcomers. The documentation is more like a reference, and does not make it particularly easy for people to start. I am thinking of posting HOWTOs and tutorials on this site to change that.

        SC4S is a great example of how syslog-ng can provide a platform while SC4S is an application of said platform. Basically SC4S is a “best-practice” configuration file that gives you out-of-the-box features, focusing on a specific use-case (e.g. feeding Splunk). All of those features were made possible by syslog-ng, however Ryan and Mark did a great job of providing an out-of-the-box experience.

        And it’s not just the experience, SC4S contains support for 100+ applications. The logs for these applications are transformed in a way that works out-of-the-box with their accompanying Splunk TA.

        There’s another similar project, I recently learned about: https://github.com/peterviskup/syslog-ng-container

        What I see about these is that syslog-ng gives you reusable componens via its SCL (syslog-ng configuration library), that can then be plugged into these “applications”. Essentially syslog-ng is a platform for these apps.

        If you are using Splunk and don’t need much more SC4S is the right choice for you.

        Reply
  2. Firstly, congrats on finding your focus!

    We currently use just rsyslog as a central log repository.

    Our needs are sort of simple:
    – compliance (and good sense) require us to log “everything” centrally
    – access to logs is controlled
    – alerting and monitoring is mostly taken care of by systems like Sentry, prometheus, and alertmanager
    – We still need to watch the kernel OOM killer, uwsgi harakiri, unexpected ufw log entries, etc.

    Additionally, we’ve been thinking of having two separate destinations (or signing logs?) to ensure no-one can tamper with the logs after they were originally emitted.

    I should note that we certainly don’t want an “applicance” for this. Yuck.

    So, I’m wondering what does syslog-ng provide that rsyslog doesn’t? Originally, I actually thought about setting up syslog-ng because I thought it might be better. But it wasn’t just an “apt-get install”, or at least there was no website to tell me what to do. All I could find were websites with marketing webcasts and products-solutions-resources corporate bullshit. Rsyslog was easier, so that’s what I went thing.

    Reply
    • Thanks Ville for your input. It is unfortunate that syslog-ng’s packaging situation has deteriorated over time in the past. One of my initial posts on this site I prepared for launch is touching on this topic: https://syslog-ng-future.blog/syslog-ng-distribution-and-support-bottleneck/

      On the other hand, syslog-ng is usually regarded very good by its users in these areas:
      – configuration language is easier to read and understand, still very powerful
      – the quality of the documentation
      – good acceptance of syslog data broken in various ways (e.g. non standard formats).
      – feeding SIEMs like Splunk directly (over HTTP)

      And a solid, stable and performant solution that is improved continuously at an 8-10 weeks release cadence.

      Generally the “gap” that I would like to address is to be the log “midlayer” that connects sources and SIEMs (+other consumers) in an easy to deploy fashion.

      Applications (and OSs) mess up logging far too often, they are logging for many different purposes (monitoring, observability/troubleshooting and security) and they are unable to adhere to any logging standards out there.

      In many cases a log emitted by an application makes it quite hard to use it in something like Splunk or another SIEM. For instance timestamps are messed up quite often, use formats that make it difficult to extract fields, a single event spanning a lot of messages and so on. There is often an “impedance mismatch” between the apps and the SIEMs, which can be straightened out by something like syslog-ng. Kafka or other queueing mechanisms are “dumb”, they will not change the messages they carry. syslog-ng can do that if needed.

      But, logs by a single application is just one thing, it is also a problem that the syslog message stream is a mixture of logs from all kind of applications, identifying the application alone is complex. Recognizing the application, device/host or even the proper timezone is difficult.

      I would like to focus on making these much easier than today AND at the same time to focus on the deployment/infrastructure site as well.

      If you use rsyslog and it fulfils its purpose, I might not be able to convince you to change. If you hit a complexity however that stems from an application + SIEM combination that you don’t want to fix manually yourself, I might 🙂

      Reply
  3. Please put udp-balancer() in the OSE!!!!

    Reply
    • Hi Chris. This is something on my radar as performance is an important strength of syslog-ng. You can get almost to udp-balancer() by creating multiple udp() sources yourself bound to the same port, then specify the so-reuseport(yes) option for each of them.

      This would give you load balancing across multiple UDP sockets, as implemented by the Linux kernel. See this article how so-reuseport() works: https://lwn.net/Articles/542629/

      udp-balancer() as far as I know (I wasn’t implementing it myself), does a bit more, it uploads an epbf program to the kernel but so-reuseport() should give you a sizable improvement.

      Reply
  4. Your passion about syslog-ng spread to us. The last couple of years were hard without you in the company, but you still managed to find time to help us out with exciting features and useful insights, which is highly appreciated.

    I can safely say that your enthusiasm and new plans are exciting for all of us! I hope we can help you with making your ideas come true. 🙂

    Cheers!

    Reply
    • Thanks Attila! I miss those times of us working together. It’s a recent realization for me that nothing prevents us working together, again.

      Open Source is great because companies come and go but Open Source projects can survive as long as they are relevant and a stable user/dev community exists and remains active.

      I am here to find out if the syslog-ng user community is indeed a thing. I want to believe it is, but at the same time I’d like to see it confirmed. Thus this site.

      Working on something that noone cares about is not my thing at all.

      Reply
  5. Let me add my congratulations for returning to focus on what must have been one of your first loves!

    We use syslog-ng OSE extensively as both a gateway to Splunk and as a platform for analysis of incoming events in ways that Splunk can’t (using program destinations.) We handle large amounts of traffic and in our virtual environment we appreciate the multi-threading, multi-processing that syslog-ng can give you when properly configured – though it took us a while to understand how to do that.

    A good part of our workload is still UDP, and we still struggle at times to catch all the incoming messages. We’ve implemented multiple UDP sources and so-reuseport, and still occasionally lose messages at peak. We also would like to see the UDP load balancer become part of the mainstream syslog-ng to get that last little bit of throughput.

    There are some other improvements we’d like to see… better documentation of log clauses for new sysadmins (the current doc is heavy on embedded log statements but is short of examples of clear, simple layouts); better documentation of program and python destinations, perhaps with examples; a better way to forward messages unmodified from one syslog-ng to another, regardless of source.

    We have found that syslog-ng does not use enough threads. By default it will only use as many as you have logical CPUs on the system… but by modifying the source to use more threads we have seen increases in throughput, as most of the threads will be waiting on I/O at any given time. It would be nice to see this as a build option.

    In general, we have found the configuration of syslog-ng to be a fairly complex task… but it does lots of complex operations, its flexibility is unmatched, and it is incredibly reliable.

    Thank you for starting the syslog-ng project and making it such a valuable tool!

    Reply
  6. We started using syslog-ng because of the patterndb. We have created an entire platform around this where our patterndb matches and enriches every log message we have. The enriched data has information about identified problems, related messages, inventory data such as serial number and physical installation to mention a few. This enriched data is sent to a back end event processor that uses the identified problems and related messages to create/update incident tickets, send SMS alerts depending on severity and alert different teams depending on areas of responsibility and calendar rotation. We are accomplishing this with logs from 2500+ systems/devices at message rates of 15,000+/sec. It all depends on the very high performance of the patterndb.

    Reply
    • Thanks Evan for the detailed answer. db-parser() is very fast, however it is a bit too difficult to use. A bit more modern database format (instead of XML) could help there a lot, I think.

      Reply
  7. First of all I’m happy you are back to roots

    The most painful missing feature for me as the follow -I think all of them since Balabit ages 🙂

    – windows support. Not neccesarly by an agent, it can be wec as well 🙂
    – some kind of central management or kind of API interface, however it can be reached using Ansible or so, but for a huge config this maybe not the ideal method 🙂
    – maybe logstore in OSE 🙂

    I think these two

    Reply
    • Hi Vlad, thanks for the feedback. I’d love to see these in OSE too 🙂

      Some might be more realistic than others, I’ll see what I can do.

      Reply

Trackbacks/Pingbacks

  1. ” #syslogng has been around for decades: I started coding the first… | Dr. Roy Schestowitz (罗伊) - […] " #syslogng has been around for decades: I started coding the first version of syslog-ng in September 1998, circa…
  2. Links 2/2/2022: Red Hat CFO Quits (Many Top-Level Resignations Lately) and GNU Screen Has New Release | Techrights - […] syslog-ng relaunch […]
  3. The syslog-ng Insider 2022-01: Reboot; Sequence; Monterey; CentOS 9; - Blog - syslog-ng Community - syslog-ng Community - […] Read the rest of the blog at https://syslog-ng-future.blog/syslog-ng-relaunch/ […]

Leave a Reply to Evan Rempel Cancel reply

Your email address will not be published. Required fields are marked *