[Confirmed Bug] [Fixed] Logs broken ???

Just did an upgrade of an old m0n0wall box - and am seeing same, i.e., no logs - any suggestions to mitigate?

Thanks,

Version 1.8.2 built on Mon Jun 8 01:20:18 CDT 2015
Platform Soekris net5501

Fyi: it looks like the log files are binary (or damaged), exec.php and 'cat /var/log/filter.log' displays 'CLOG' and then a few characters:

When I win the lottery and have free time left

I thought I would post what happened to create this bug and how it slipped through. After all, if someone learns from my mistake, perhaps it isn't so bad.

The 1.8.2 and 1.8.3 releases are based on 1.8.2b6 which was unreleased, and has very few changes over 1.8.2b5. It was tested by several people and worked perfectly. (well, it did not have this bug) So, on the 8th, I moved it to "tags" and changed the version file and the build script only, and created an image. Since it had been throughly tested already, all I did was install it to make sure it worked. (I did not look at each page) I also made some custom images for partners and sent it to them, and they installed as well. (And apparently did not check logs) The next day, after hearing nothing, I released 1.8.2 publicly, and many people downloaded it. (And also did not check logs)

On the 12th, we hit slashdot. This was a good thing, and there was more traffic on Saturday then there has been the entire time the website has been up! And many more people running SmallWall! And some of them checked logs... Ooops!

Building SmallWall is a complex process, and it is mostly scripted. Several of the tools are modified either to run in the smaller environment, or to work with the php and xml based system, or both. So clean source tools from FreeBSD are patched to build the custom binaries. In my case, /usr/src/user.sbin/syslogd/syslogd.c had a patch applied and not removed. So when run again, it was removed, and not patched... So it failed to run, and there were no logs.

Figuring out what happened was a challenge, but the fix was simple. After known good builds 1.8.2b6 and 1.8.2b5 also had the problem, I knew it was the build environment, not the image. So I installed a new system in a VM and started the build. Then I compared the file systems and found the corrupt file. I went to all of this trouble because "fixing" the problem without understanding why it happened does not really "fix" the problem.

Moving forward, I will be working on the build scripts to make sure the build environment is clean (Or at least that the patches are not already applied...) before starting. It won't fix everything, but it will fix this!

And most of all, thank you to the new users who provided very good bug reports, and made fixing it much easier on an early Saturday morning when I was still half asleep.

I'd also like to send my appreciation out for the quick bug fix too - having problems with VPN I needed to see the logs. Waking up this morning to find this stuff fixed was just exceptional.

These things happen. At least you rolled up your sleeves and took on a project which I personally wouldn't have the time to do, and a project I've relied on for a number of years now. I was dreading looking at alternatives and evaluating them and perhaps having to throw away my ALIX firewalls for something that could handle the requirements of those other firewalls.

You've helped me and a lot of people dodge a bullet - and for that I'm grateful.