Archive and Delete files
Closed - This job posting has been filled and work has been completed.
We have a problem with our QA boxes where:
- Some components fill up the drives with log files quickly.
- We shouldn't delete all the files. We need to save the log files that have our messages in them so that we can go back and see what is going on. ... but the Market Data logs become overwhelmingly big pretty quickly.
1. We need some overnight automated process (below). This would need to be run using the Windows 2008 Task Scheduler on some box.
2. In addition to the automated system, we would like to find some tools that simplify management of the disk usage. We would like to have some utility that lets us quickly zero in on the subdirectory that is using the most disk space.... Kind of like a pie chart of where your disk is being used.
Step 1: Copy QA Log Files from QA Boxes to Y: (\\220.127.116.11\share)
Copy all log files from the previous day or older to Y. A log file is any *.txt file under any subdirectory of c:\Raptor.
Once you know they have been copied to Y drive, then delete them on the CA Box.
Step 2: Efficient Retention
Build a Utility: QAArchiver.exe that is run from the command line.
Usage is: QAArchiver [TopLevel Directory of Archive]
The objective of this utility is to store the last 30 days of "meaningful" log files for each adaptor and/or RemoteMarketSub Status Log.
This utility is only applicable to QA log files. Production log files must be retained forever.
The utility recursively searches directories and purges old files based on the following rules:
A meaningful log file is a *.TXT file that has a size over 20 kilobytes AND:
Category A: contains at least one of these strings: 35=D, 35=8, 35=R
Category B: contains 35=X, 35=S
A FIX Filename Suffix is the name of the adaptor. For example the file:
the part of the file name in Yellow is the Suffix.
Note that files of this type are also important:
For any single sub-directory AND FileName Suffix (treat StatusLogs as their own group as well), we will only keep on the Y drive:
Up to 1 GB of the most recent files in Category A
Up to 1 GB of the most recent files in Category B (or the entire last file if it is over 1 GB)
For any file that we are not going to keep, it should be deleted from the archive drive.