I find that a lot of syslog-ng deployments are lagging behind and are using ancient versions. It has become difficult for me to get these deployments to more recent versions. No product is able to improve and cover new ground in a situation like this…
Maintaining this change velocity in the log management space is not feasible. Deploying a log management and processing infrastructure from scratch can literally take years just one time. Swapping out technologies every now and then on a whim would mean that the project never reaches the goals it was set out to achieve.
With that I said I still think that being able to regularly push out updates to deployments is an important bottleneck to solve for any product to be sustainable. This is needed for both the feature front (e.g. addressing new use-cases) and on the support perspective (e.g. fixing bugs).
I often get questions about syslog-ng 3.5.6. This release was originally published 5th August 2014, roughly 8 years ago, and happens to be part of EPEL7. syslog-ng is included in BMW i3 vehicles, this video shows the listing of open source components on the infotainment screen, The BMW Open Source DVD contains syslog-ng 3.4.7, a whooping fresh release from December 2013. There are similar stories with syslog-ng included in products or an OS release, usually with pretty old versions.
Why does this happen?
Due to the early adoption of syslog-ng, it was included in a number of Linux distributions and BSDs/UNIXes, even became default in some of them. I considered this a great success.
For none of these distributions however is log management a central question. They each need some kind of log daemon, but that’s it. Whether that log daemon is syslogd, rsyslog or syslog-ng does not really matter. Neither matters their actual version number. So even though distributions helped initial syslog-ng adoption, they have become a bottleneck in delivering new releases to users.
Users can still upgrade, right?
Enterprise users (and products that embed Linux and syslog-ng) pick an OS version and plan with it for ~10 years. Unfortunately they deploy syslog-ng as a part of the OS and expect the OS vendor to provide support. Often, the sysadmin responsible for log management is not even allowed to upgrade. Some claim that upgrading syslog-ng would violate their support terms, causing the entire OS to become unsupported.
So even though more recent versions of syslog-ng includes functionality or fixes they need, they stick to the old version and try to work around any issues they find.
The support from the OS vendor for the logging component is questionable at best and is restricted only for the most basic use-cases, not cases where syslog-ng would play an important role in one’s infrastructure. Just as log management is not a central focus for the OS, neither is it for the support team behind the OS. They would fix security issues, should they be reported, but otherwise they will just continue to use what they have.
Solution: state of the art binaries to pick from
Building the latest version of syslog-ng for your enterprise distro on your own is not for the faint of heart. Even though 20 years ago, building your own kernel or application was an essential part of a sysadmin’s job on any UNIX, this is not true any more.
Also, it was a lot easier to build syslog-ng in 2001, today we have so many integrations that pulling all the build dependencies (and the right versions) is far from trivial.
We worked hard in the past years to resolve this issue and today syslog-ng is not only available in source format. There are a number of options today to pick from, should you want to use the latest and greatest:
- Debian packages produced by the project itself: https://github.com/syslog-ng/syslog-ng#installation-from-binaries
- List of binary packages for various enterprise/community Linux distributions & BSDs: https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx
- Production docker image (for docker-compose or Kubernetes deployment): https://hub.docker.com/repository/docker/balabit/syslog-ng
- Downstream products, for example the open source SC4S to feed syslog data to Splunk using syslog-ng: https://splunkbase.splunk.com/app/4740/
- Premium Edition and commercial support from One Identity (the company which acquired Balabit): https://www.syslog-ng.com/products/log-management-software/
Over time, the building of bespoke/customized packages has become much easier too, this blog post explains it all.
So what’s your excuse? I am really interested if the options above suffice. Do you still use an old syslog-ng version? Why? Would any of the above work for you? If not, What would YOU need to upgrade syslog-ng to recent versions? And what would you need to change your processes to plan for upgrades regularly?
If you have a response to any of these questions, please post it as comment below or drop me an email. Thanks.
As a developer, I am really pleased with how easy it is to create packages with dbld (syslog-ng’s packaging and development tool). You can effectively create a package to your desired distro with one command. They are also automatically generated for GitHub Pull Requests, which came handy multiple times for us and users, too, while trying to validate a fix even before it was merged.
The https://ose-repo.syslog-ng.com APT repository became live last November and multiple users have said they started using it. It already has stable releases and nightly releases are not far to come. I think it is worth to try out for those who use a DEB based distro and do not want to create packages by themselves.
And as mentioned, most users of RPM based distros are covered by https://copr.fedorainfracloud.org/coprs/czanik/
I think the majority of users are covered by these repositories. I hope that more and more people will recognize this, so they can enjoy the new features in syslog-ng. And if we missed something, it would be interesting to know.
As long as Peter Czanik maintains those builds… Do not get me wrong. I know he is quite enthusiast about it but still the repo relies on a single person. It’s okay for a hobby project but an enterprise might requires more.
The rpm/deb packaging is part of the source tree and as such are part of the syslog-ng project.
These packages are automatically built by GitHub actions as a part of our CI pipeline. You can even download those builds as well.
You can trivially set up your own build environment on your premises if that’s not enough, you only need an OS with docker.
Peter’s repositories are using the same packaging files and his contribution is that he makes them available in EPEL and Copr.
I can imagine a case where he would cease doing the packages but you still have options in that case.
Also, I don’t see how any other open source project would be different or better than this.
Package creation must be transitioned to community. There should be a community team working together to create the packages. This removes the dependency on one individual and will provide continuity if any member becomes less or completely inactive.
A repository of packages should only contain packages that are backwards compatible and the repository needs to have a specified support life. Having a single update stream that regularly creates warnings or issues about configuration file format changes is not an easy hurdle when managing 1000’s of systems.
We deal with a few community repositories for packages. To ensure control of a testing pipeline we copy repositories locally and ensure that all of our development hosts get the packages installed for testing before our pre-production hosts are updated and finally our production hosts. This also ensures that when we do an update, random outages on the internet do not prevent the download of the latest package. It is already on site.
Hi Evan! Long time no see 🙂 Sorry for responding somewhat late.
Package creation is being transferred to the project, at the moment
* Debian packaging is in the source tree, https://github.com/syslog-ng/syslog-ng/tree/master/packaging
* .deb packages are automatically created for every pull request, for example: https://github.com/syslog-ng/syslog-ng/actions/runs/1859437583
* We do have nightly builds of them: https://github.com/syslog-ng/syslog-ng/actions/runs/1855676464
* Releases are built as deb packages and published in an APT repository: https://github.com/syslog-ng/syslog-ng#debianubuntu
An automated test for packaging is in the works, here: https://github.com/syslog-ng/syslog-ng/pull/3903, with that we would automatically pull and upgrade syslog-ng as a package and test if it starts up.
IIRC you are using RPM packages and doing the same for those is would be great.
Right now the status with RPM is:
* we do have Fedora + RHEL (and derivatives) packaging in the source tree: https://github.com/syslog-ng/syslog-ng/tree/master/packaging
* which is automatically built for pull requests: for example: https://github.com/syslog-ng/syslog-ng/actions/runs/1859437583
* we also build nightlies of the same.
* We don’t have a yum repository and the automated test suite for packages, yet.
Creating a yum repository is not that far in the future. To push packages to EPEL however is a manual process and I can’t see how that could be automated, and until it can, it remains the responsibility of a single person.
We can get to a point where the open source project offers RPM packages on its own, automatically on every release, after all tests on the code have been executed. Does this cover your needs?
You have another interesting point, which is: “single update stream that regularly creates warnings or issues about configuration file format changes is not an easy hurdle when managing 1000’s of systems.”
I guess you are referring to the “config file versioning” scheme syslog-ng uses to introduce changes. Would a configuration option that tells syslog-ng to ignore all future changes help you?
I mean syslog-ng IS compatible with older versions of the configuration (it is compatible back to syslog-ng 3.0) and we rarely remove those compatibility features: We removed 2.0 compatibility in 2017, while syslog-ng 3.0 was released in 2009, so we are supporting a span of 13 years with the current version. We do nudge people to upgrade their config format, but that could potentially be disabled with a flag, at least until the compatibility with your specific version is dropped completely.
Would this resolve this pain point?
Thanks for your input.
I think people stick with the older versions for two simple reasons:
1. They work and are rock solid.
2. There are no new features they need.
I agree that sometimes the “it ain’t broke don’t fix it” attitude has merits, however should a bug surface in those ancient versions ever, you can get into trouble.
Fixing bugs in decade old software without test coverage is risky and leaping ahead to a new version of the software is risky too.
These risks are what you can mitigate by upgrading in smaller steps, regularly.
So I think it all boils down to support and supportability at the end.
Of course if you don’t need said support or you can fix potential bugs yourself, that works too.