Faulty generation of log files after reducing the value of MaxParts

[ekicholong]

Hi Ray,

I am doing some log files rotation using the MaxSize & MaxParts properties of the TLogFileDestination object and I recently discovered the following issues:

Issue 1:
Assuming the basic LogFileDestination properties are set as follows: FileName=Sample.csl, FilePath=D:\Logs, MaxSize =40 and MaxParts=6

If I reduce the value of MaxParts to e.g. 3, new log files are generated on top of the existing ones starting from Sample_7.csl (assuming Sample_6.csl was full) up to log file n = 10 (i.e. Sample_10.csl). Thereafter only the 10th log file is used to store new log messages and it’s file size exceeds the limit set by MaxSize.

Issue 2:
Assuming the basic LogFileDestination properties are now set as follows: FileName=Sample.csl, FilePath=D:\Logs, MaxSize =40 and MaxParts=12

As is the case with issue 1, after reducing the MaxParts value to e.g. 8, new log files are also genarated on top of the existing ones. But this time the number of new files continue to grow without a limit.

My expectation is that after reducing the value of MaxParts, newly generated log files never exceeds the specified value of MaxParts even if there are existing log files with a similar name within the same folder. Also, no file should exceed the size limit set by the MaxSize whatsoever.  Or maybe I am missing something here? 🙂

Current codeSite version in use: 5.4.0 (CodeSite Studio)

Your support is much appreciated

[Ray Konopka]

Hi,
For issue 1, is it correct to assume that with the MaxSize=40 and MaxParts=6 that before you change the MaxParts to 3, you have 6 log files that already exist in the directory, and then you start doing file logging with the new MaxParts value of 3? I’m trying to understand the sequence of steps so that I can try to reproduce. And please confirm that all you changed was the MaxParts value to 3–that is, you did not change any other properties.

In regards to Issue 2, you state “new log files are also generated on top of the existing ones.” Can you be more specific about what exactly you are seeing in this situation?

Also, your comment “no file should exceed the size limit set by the MaxSize whatsoever” is not valid. Since each CodeSite message is variable in size, it is possible that the last message saved to a log file part will result in the log file part being larger than the MaxSize value. The MaxSize value specifies whether or not the next CodeSite message is allowed to be added to the file. If the current log file part is less than MaxSize, then the message is saved to the log file.

We considered looking at the size of the next message to be saved, and then only save to the current log file part if the resulting size will be less than MaxSize. However, then you need to consider what happens if the next message is larger than MaxSize, which can happen with screenshots, for example. Then you end up with a log file part larger than MaxSize anyway, because you have save the message is the log file part. We did not see the benefit outweighing the added complexity.

Ray

[ekicholong]

Thank you for your reply.

For issue 1, yes it is correct to assume that there were 6 log files in a folder before i changed MaxParts from 6 to 3 and started a new logging session.  And also for issue 2, there were 12 log files before changing MaxParts from 12 to 8. I can confirm that i changed the value of MaxParts property only.

By “new log files are also generated on top of the existing ones.” I meant that after the last file part is full, new log file parts are generated on top of the existing ones using the name numbering pattern. If for example sample_6.csl was the last file part, first its filled up and once full new files are created as follows: sample_7.csl, sample_8.csl etc.  In the case of issue 1, new log file parts are created up to sample_10.cls and then all new logs are saved only in this one file until i stop the logging session. And in the case of issue 2 where MaxParts > 10, new log file parts are continously created until i stop the logging session.

My comment “no file should exceed the size limit set by the MaxSize whatsoever” was only in reference to issue 1 where by sample_10.csl continously grows in size waaay above the set MaxSize value. But I agree with you and I am aware of what you are saying, that the last file part can slightly outgrow the value of MaxSize in order to accomodate the last message. That is ok as long as all other new messages are not saved in the same file therefore increasing it’s size as is the case with sample_10.csl.

I do hope am a bit clear now and am willing to offer more information if required to.

Cheers

[Ray Konopka]

Hi,

Thank you for posting the additional details and clarifications. I am able to duplicate the behavior that you reported. The crux of the issue is that the Dispatcher looks in the specified folder for all log file parts, and ends up selecting the “last” part file. The assumption is that the last part file in the directory is either the MaxParts log file or an earlier log file part. So the last part file is selected and checked for MaxSize. If the file exceeds MaxSize, then the next part number is calculated from the “last” part file.

The problem is that the assumption above is incorrect in the situation that you describe. The “last” log file represents a log file part that already exceeds MaxParts, which is why the parts increase. Well, the part files increase until the number of digits needed to represent the parts changes to a new order of magnitude. For example, in Sample 1, the parts increase to 7, 8, 9, and then at 10 the file size just grows forever. In Sample 2, the parts increase to 13, 14, 15, etc. all the way to 98, 99. Then at 100 the file size just grows forever.

Part of the fix for this is to NOT use the “last” part file name if it exceeds the MaxParts value. However, this is fine if the MaxParts value does not cross a power of 10 value. For example, in Sample 1, MaxParts went from 6 to 3. No problem. However, for Sample 2, MaxParts changes from 12 to 8, which crosses 10. The problem is that the file parts are named differently when MaxParts is set to 12 versus 8. When MaxParts is set to 12, the file parts will have names like:
Sample_01.csl
Sample_02.csl
. . .
Sample_12.csl

However, with MaxParts set to 8, the file parts will have names such as:
Sample_1.csl
Sample_2.csl
Sample_8.csl

The fact that the format of the log file parts is different makes recovering from this situation rather complicated. There are a couple ways in which this could be handled. First, the code that looks for the “last” part needs to be more specific about finding a match and consider the difference between the two cases above. So, in Sample 2 when the MaxParts is 8, the code would look for Sample_1, etc. and NOT consider Sample_01 as a matching part. In this case, after switching MaxParts from 12 to 8, the log file parts would essentially start over at Sample_1.csl, and the existing Sample_01.csl to Sample_12.csl files would remain unchanged and be there forever, even after the “new” Sample_1.csl to Sample_8.csl files get rotated out.

The second method of handling things would be to change the way the log file parts are named and simply use 4 digits for all log file parts. So, the file names would look like:
Sample_0001.csl
Sample_0002.csl
. . .
Sample_0012.csl

This is the solution that I am leaning towards, but I’m not sure it makes sense to do this for CodeSite 5 because of the implications it would have on existing deployments. Essentially, all log file parts would reset to *_0001.csl, which is no different than the problem I described earlier.

This type of change is more likely for CodeSite 6, when there are more significant changes in the product and how destinations are managed, and resetting log file parts for an application would be more easily facilitated during an upgrade.

With all of this in mind, I am curious to gain a better understanding of how you are using the MaxParts property. Specifically, is the MaxParts property something that gets changed a lot for your application? What is the process that results in the property value changing? Do you provide access to change this on an end user machine, or is it simply set internally by you (i.e. developer)? How often does the value change?

In all the years that CodeSite has supported log file parts, this issue has never been reported. I suspect because the MaxParts value is typically not changed (at least not decreased), and if the property is changed, the existing log file parts are simply deleted when deploying the application. Hence, I am curious to understand the implication of this particular issue on your deployment.

Ray

[ekicholong]

Hi Ray,

Thank you very much for the feedback. Any solution is very much appreciated.

To answer your questions;

I have a configuration file which contains MaxParts, MaxSize, FileName, FilePath amongst other properties for configuring logFileDestination. Only a developer, a support-team and a test-team member should be able to edit this file and change the values of these properties.  The file is being used by an application generating quite a large number of logs using codesite.

The idea of later changing values of especially MaxParts and MaxValue via a configuration file came about as I am yet to determine the optimal values to use for the end-users. And so during the testing phase I want to experiment with different values until i get an optimal one, which means that there could be an instance where it might be necessary to for example increase or reduce the value of MaxParts (and or MaxSize) depending on the size of logs generated – without having to recompile the application. Perhaps it is worth to note that the application testing is done by a separate test-team who only get the compiled binaries and one of my test request would be to observe the size and the number of logfiles generated. In addition, during a support session for an end-user (customer), a support-team member might realise that there is need to increase or decrease the value of MaxParts (and or MaxSize) depending on the number of logs generated which vary between users. In both cases you want to ensure that tester’s or user’s computer is not littered with uneeded log files.

 

[Ray Konopka]

Hi,
Thanks for explaining how you are using the MaxParts and other values. Given that the primary objective is to determine optimal values for the properties, I believe it would be helpful to instruct the testing team to clear out the log files after changing the values and then re-evaluate the sizing. This would eliminate the unwanted behavior that can occur when reducing the MaxParts values when there are already more parts in the directory.

Ray

[Author]

Posts