Select Page

syslog-ng future: the path to syslog-ng 4

by | Feb 28, 2022 | plan, technology | 2 comments

syslog-ng 3.0.1 was released 17th February 2009, almost exactly 13 years ago. The key feature at that point was to add support for RFC5424, the new “syslog” protocol. The 3.0 release marked a significant conceptual change in syslog-ng as this was where we introduced support for generic “name-value pairs”, a means to encode application or organization specific fields (aka name-value pairs as we named them) associated with a log message.

The 3.x release train has been a long and a busy one. We are right now at 3.35.1 with 3.36.1 right around the corner. Not counting bugfix releases, that’s ~4 releases per year on average. This pace was slower initially (~1 release/year) which then increased due to all the engineering practices that we implemented in the last decade: syslog-ng is a very well tested application today, covered both in terms of unit tests and functional, end-to-end testing. In the last years, the syslog-ng project has produced 5-6 releases per year (every ~2 months), in a rolling model. Apart from features and bugfixes we also had a sharp focus on compatibility and avoiding regressions.

When I started to draft this post, I compiled a list of noteworthy features that were created since 3.0.1 in 2009. My intention with the list was to include it here to back up my previous claim that there are lot of undiscovered and under-communicated aspects of syslog-ng. However, when I finished with the list, I had to realise that even if I trim it down, it is still too long to discuss it in a blog post at one go. For now, I’ve uploaded my raw notes here. I am probably going to use that list to publish technology pieces on the blog or create a survey to map out which are the more interesting items to syslog-ng users. I don’t know yet.

This post however, is not about the past, the title says it all: it is about the path to syslog-ng 4. With the relaunch taking place, I was thinking what else could be better to symbolize a restart than a new major version? With that we can take a moment to reflect on the 3.x series and start anew with fresh energy.

It is very important to state that syslog-ng 4 is not the revolutionary, break-everything kind of release that we see too often in the software world. Rather it is an evolutionary change that will be produced similarly to previous releases, that is:

  • the release will contain both features and bugfixes
  • if a change in behaviour is unavoidable, we keep being compatible using the config version mechanism, e.g. the “@version:” tag in the front of the config file
  • compatibility with old config versions are retained long term (e.g. we are compatible back to 3.0, with compatibility back to 2.0 dropped just a couple of years ago)

But why the fuzz, you may ask, about a new version number if nothing changes and we do exactly as before?

Well, there are some plans scheduled for 4.0 (more on those later), but I consider this release to be an opportunity to set up new, long term objectives. Objectives that will cover the upcoming releases as well and not just 4.0 itself. With the launch of this blog and through interactions with the community, I already have some thoughts of my own, still, I would like to allow community members to contribute even on the strategic level. Let’s find the mission statement for syslog-ng that covers the next 10 years and then guide the project towards those goals with a step in each release. I am posting the specifics and the mechanism of this work in an upcoming post. Until that post, please continue to send me feedback (via Email, gitter.im, GitHub, Reddit, LinkedIn whatever you like), I am truly enjoying each and every one of these interactions and make an effort to respond to all your queries. Also, the syslog-ng project started to use GitHub’s discussion feature, so if you have a suggestion with regards to syslog-ng 4, feel free to submit it here.

Release management and Support

So how would the release of 4.0 happen? Is this a new branch over 3.x? How long would we support 3.x?

These are all valid questions, however the answer is simple: syslog-ng 4 is nothing more than a 3.x release in this respect. We will add features and bugfixes and compatibility will be provided using the config version feature (ie. @version). We will make no breaking changes that we cannot continue to be compatible with. There will be no separate 3.x and 4.x releases going in parallel. If we break something, fixes would be pushed out in upcoming versions (either the scheduled one or an emergency one if the problem is critical). We are confident that our current test coverage gives us a safety net that allows us to use this release strategy.

At the same time, we are scheduling some larger-scale changes that will probably not fit into a normal 8 week release cycle we do these days. We don’t want to stop doing our 3.x releases and we don’t want to publish half-baked features. So how are we going to resolve this conflict?

The regular bugfix/feature flow of 3.x will continue to operate as before. Any 4.0 related functional change will be merged to master (and thus make it into 3.x releases) but any functional change will be disabled.

Once all 4.0 related changes are merged, a 4.0.1 release will be created, effectively turning on the new behaviours, except if the user operates in `@config: 3.x` mode, which is the usual method  to tell syslog-ng to operate in compatibility mode.

All of this basically means the following:

  • the 3.x feature and bugfix flow operates as normal
  • the 4.x related changes get merged and can be evaluated if someone is interested (by using “@version: 3.255” at the top of your configuration file)
  • no half-baked functionality is exposed, even if they take longer to bake than the 8 week release cadence.
  • all protected by our testing infrastructure

Up until now, only the versioning framework was merged with some more queued for merging. Details on some of the plans for 4.0 are coming in separate posts. Stay tuned!

Subscribing to this blog shows interest, interest brings motivation, motivation brings features and bugfixes to syslog-ng. Please show your appreciation and interest by subscribing. Thanks.

2 Comments

  1. Many projects use semantic versioning, including my own.
    In your post you refer to “major” versions as the second number in 3.x.y.
    Also, with rolling releases the 3d number doesn’t make much sense.

    What are your thoughts on this ?

    Reply
    • I’ve noted a word “major” in the text I used incorrectly above, which I have removed, because indeed 3.x releases cannot be considered “major” in the semantic versioning context. It now reads: “… that’s ~4 releases per year on average.”

      Going back to your points: syslog-ng uses something similar to semantic versioning, however it does not apply directly, as semantic versioning is more about library versioning, and not product versioning.

      Still, we use our version numbers in a similar spirit. Quoting semver.org:

      1. MAJOR version when you make incompatible API changes,
      2. MINOR version when you add functionality in a backwards compatible manner, and
      3. PATCH version when you make backwards compatible bug fixes.

      Responding to those above:

      1. Whenever we publish new MAJOR versions we remain compatible. For instance, we removed v1.0 and v2.0 compatibility in 3.13.1 in 2017 (8 years after the original 3.0 release).

      2. We always add functionality in a backwards compatible manner and these are published in MINOR versions.

      3. PATCH versions are only published in case of release critical errors. These do happen sometimes. We had 4 of these during the 3.x timespan, last it happened at 3.33.2 for instance. It is true that in most cases all minor versions have only one release and no followup PATCH releases are published.

      This happens both because our testing regime is pretty good but also because the next MINOR version is maximum 8 weeks away.

      With that said, version numbers all become symbolic. With the policy described above, we could go on with 3.x all eternity. But that does not feel right either (not to mention that some internal representation of the version would overflow eventually 🙂

      My take on this is that product version numbers often become a means of communication instead of being technical identifiers. With syslog-ng 4, I want to communicate that it is a fresh start, and that there are still important changes happening in syslog-ng. It does not say much else.

      On the other hand, I’d have trouble defining what it takes to name something “MAJOR” vs “MINOR”. All of our releases are produced by a roughly 8 week long timespan, so their content is roughly the same size all along. The similar question in the Linux kernel developer community was decided quite similarly.

      https://kernelnewbies.org/Linux_4.0#Arbitrary_version_change

      Reply

Trackbacks/Pingbacks

  1. Syslog-ng 3.36 news: better TLS 1.3, basic MacOS support, and many more - Blog - syslog-ng Community - syslog-ng Community - […] Development of syslog-ng 4.0 is already under way, and syslog-ng 3.36 already contains some under-the-hood changes for 4.0. You…
  2. The syslog-ng insider 2022-03: syslog-ng 4; MQTT source; Zinc; Elastic Cloud; 3.36; - Blog - syslog-ng Community - syslog-ng Community - […] Read the rest of the blog at https://syslog-ng-future.blog/syslog-ng-future-the-path-to-syslog-ng-4/ […]

Submit a Comment

Your email address will not be published.