Log format improvement

[Willy2]

First of all, I have installed Cleanmem (CM) two days ago and I am already impressed. Programs that I regularly need and that keep running in the background now occupy much less memory. And its seems CPU usage has gone down as well. No surprise there, because (it seems ???) Windows doesn't need to write info to the swapfile that often any more. Currently I am in the process of fine tuning the settings, e.g. which applications do I want to be curtailed in their memory usage. And that's why the log file feature is such a wonderful thing.

But reading the log file, in its current format, is a challenge on its own. The format of the text/info can - IMO - be improved dramatically. I suggest the following text format. This is what had in mind (in italics). And please do add a blank line between two entries/two sets of lines.

Cleanmem task: 11-12-2009, 17.52 PM
iexplorer.exe: Memory: 50,65 MB ==> 3.66 MB
explorer.exe: memory,  20.41 MB ==> 5.72 MB
................
................
.................

Cleanmem task:  12-12-2009 ........
................

[Shane]

Should only take me 5 min to make the changes.
When I get back from some jobs with my customers today I will get it done for ya 

Shane

[Shane]

Since the program makes a standard txt file I cant do any formats. How does this look?

Cleanmem Memory Stats: 7/8/2010 6:56:52 PM
--------------------------------------------------------------
explorer.exe (PID: 1816): Memory: 10.67 MB ==> 464.00 KB
notepad.exe (PID: 1600): Memory: 132.00 KB ==> 132.00 KB
explorer.exe (PID: 3588): Memory: 7.77 MB ==> 360.00 KB
firefox.exe (PID: 3952): Memory: 44.07 MB ==> 356.00 KB
--------------------------------------------------------------

Cleanmem Memory Stats: 7/8/2010 6:57:03 PM
--------------------------------------------------------------
explorer.exe (PID: 1816): Memory: 6.38 MB ==> 452.00 KB
notepad.exe (PID: 1600): Memory: 132.00 KB ==> 132.00 KB
explorer.exe (PID: 3588): Memory: 5.32 MB ==> 360.00 KB
firefox.exe (PID: 3952): Memory: 20.41 MB ==> 352.00 KB
--------------------------------------------------------------

If you noticed I also added the process ID in case there is more than one process with that name running. 

Shane

[Willy2]

Yes, this is a dramatic improvement over the previous log file format. Personally, I DO prefer a *.txt file over e.g. a *.html file. A *.txt file simply loads much faster with e.g. Notepad than a *.html file with e.g. IE 8.

The PID is also good because on my PC, e.g. IE 8 (always ??) starts two processes. However I would suggest an additional modification along the lines as provided below. (see below, in italics)

Cleanmem stats .....................
Processes monitored - Memory usage.
---------------------------------------------------------
InCD.exe : ............ , 2.54 Mb
iexplore.exe : .......... , 50.43 Mb ==> 5.34 Mb
explorer.exe : .......... , 23.76 Mb ==> 4.21 Mb
defraggler.exe : ........... , 4.28 Mb
ccleaner.exe : .......... , 5.76 Mb
firefox.exe : ............ , 20.38 Mb

Or something like this (see below):

Cleanmem stats .....................
Processes monitored - Memory usage.
---------------------------------------------------------
InCD.exe :
iexplore.exe : .......... , 50.43 Mb ==> 5.34 Mb
explorer.exe : .......... , 23.76 Mb ==> 4.21 Mb
defraggler.exe :
ccleaner.exe :
firefox.exe :


In the log settings file the user can specify which processes are to be monitored. But which processes are curtailed in their memory usage must be specified in the only file. And the (potential) difference between the two could be expressed in the log file (see above). The log file shows that six processes are monitored but only two processes (iexplore.exe and explorer.exe) that are curtailed in their memory usage by Cleanmem.

[Shane]

Cleanmem skips all processes in the ignore list, so having it logged would require rewriting it all, I prefer not to lol.

Also if a person has more than one process with the same name but in different paths I thought the path of the process would be good as well. This is how I have cleanmem making the log now, how does this look? I haven't released anything yet till I am done with you 

Cleanmem stats: 7/9/2010 8:33:11 PM
Processes Monitored - Memory usage.
--------------------------------------------------------------
explorer.exe - Memory: 6.68 MB ==> 456.00 KB (PID: 1808)(Path: C:\WINDOWS\Explorer.EXE)
explorer.exe - Memory: 8.29 MB ==> 368.00 KB (PID: 1312)(Path: C:\WINDOWS\Explorer.EXE)
notepad.exe - Memory: 152.00 KB ==> 132.00 KB (PID: 2568)(Path: C:\WINDOWS\system32\notepad.exe)
firefox.exe - Memory: 10.36 MB ==> 472.00 KB (PID: 3700)(Path: C:\Program Files\Mozilla Firefox\firefox.exe)
--------------------------------------------------------------


Shane

[Shane]

Took it one step further

Instead of the file path, I have it pull the command line used to start the process.
I find this will be even more useful in helping trace memory leaks for apps

Cleanmem stats: 7/9/2010 9:29:53 PM
Processes Monitored - Memory usage.
--------------------------------------------------------------
explorer.exe - Memory: 10.89 MB ==> 444.00 KB (PID: 1232)(Path: C:\WINDOWS\Explorer.EXE)
firefox.exe - Memory: 62.86 MB ==> 632.00 KB (PID: 2840)(Path: "C:\Program Files\Mozilla Firefox\firefox.exe" )
explorer.exe - Memory: 13.21 MB ==> 384.00 KB (PID: 120)(Path: "C:\WINDOWS\Explorer.EXE" /IDLIST,:2736:1232,/S)
notepad.exe - Memory: 1.49 MB ==> 144.00 KB (PID: 916)(Path: "C:\WINDOWS\system32\NOTEPAD.EXE" C:\Documents and Settings\User\Desktop\test.txt)
--------------------------------------------------------------

Shane

[Willy2]

Below your latest suggestion (in italics):

1. Cleanmem stats: 7/9/2010 9:29:53 PM
2. Processes Monitored - Memory usage.
3. --------------------------------------------------------------
4. explorer.exe - Memory: 10.89 MB ==> 444.00 KB (PID: 1232)(Path: C:\WINDOWS\Explorer.EXE)
5. firefox.exe - ......................
6. explorer.exe - .......................
7. notepad.exe - ......................
8. --------------------------------------------------------------

My latest suggestions:
-- How important is the PID ? The PID is - IMO - less important than the path of the program. (I am not a software developer). So, perhaps you can omit every PID code in lines #4 through #7.
-- Change line #2 to ""Process monitered - Memory usage - Path"". Then you can omit the words ""Memory"" and ""path"" in lines #4 through #7. Then line #4 can become ""explorer.exe - 10.89 MB ==> 444.00 KB - C:\WINDOWS\Explorer.EXE"". The three items in the line are separated by a ""-"" and one or more spaces only. That's much more eye friendly. But I don't know if instead of the word ""memory"" another word (e.g. ""Systemcache"") can show up in a line in the log file.

-- If you really want to keep the PID code then e.g. line #4 could become ""explorer.exe - 10.89 MB ==> 444.00 KB - 1232 - C:\WINDOWS\Explorer.EXE"" and line #2 could become ""Process monitered - Memory usage - PID - Path"" or ""Process monitered - Memory usage - Program ID - Path"".

I know you don't want to rewrite Cleanmem now but perhaps you can include the suggestions below in a future version.

-- Concerning post #3: I think there's a (small) misunderstanding. I have added 4 processes to the ""only"" list and 13 processes in the ""log settings"" file. But I didn't add any process to the ""ignore"' list. When a process is running AND included in the ""log settings"" list then it shows up in the log file. And the log file shows how much memory a process uses. Up to this point I like what I see. My suggestion was that ONLY if a process was curtailed by Cleanmem then info could be added to the log file showing the memory usage reduction.

-- Perhaps you could include a bit more information in either the header or in a line (or both) of the log. I was thinking of which account (e.g. Administrator, User, Shane or Owner) and/or what kind of process (Owner, System, User) as displayed in Taskmanager.

[Shane]

I think pid can be important when watching the log file while the programs are still running. The point of the PID is for the processes that have the same file name and running multiple. such as IE and explorer and many others.

The command path is very helpful for the same reason.

Total info that I can pull for a process is

Caption -          explorer.exe
Command Line -          C:\WINDOWS\Explorer.EXE
Computer -          COMPUTER309
Creation Date -          Saturday, July 10, 2010 at 9:57:08 AM
Description -          explorer.exe
Executable Path -          C:\WINDOWS\Explorer.EXE
DOS Executable Path -       C:\WINDOWS\Explorer.EXE
Handle -             1376
Handle Count -          664.00
Kernel Mode Time -       155625000
Max Working Size -       1.348 MB
Min Working Size -          200.000 KB
Name -             explorer.exe
Other Operation Count -       297,597.00
Other Transfer Count -       22,887,588.00
Page Faults -          46,367.00
Page File Usage -          25.324 MB
Peak Page File Usage -       26.313 MB
Peak Virtual Size -          132.145 MB
Peak Working Set -       24.781 MB
Priority -             8 (Normal)
Private Page Count -       26,554,368.00
Process ID -          1376
Quota NonPaged Pool Usage -    18496
Quota Paged Pool Usage -       200532
Quota Peak NonPaged Pool Usage -    22816
Quota Peak Paged Pool Usage -    237684
Read Operation Count -       1662
Read Transfer Count -       5758866
Thread Count -          22.00
User Mode Time -          29218750
Virtual Size -          121.813 MB
Windows Version -          5.1.2600
Working Set Size (Memory Usage) -    9.527 MB
Write Operation Count -       28.00
Write Transfer Count -       4,672.00

I don't want the log to get out of hand though. Last thing I want is the log file growing to fast on someone.
So far I want to keep the new log file as

Cleanmem stats: 7/9/2010 9:29:53 PM
Processes Monitored - Memory usage.
--------------------------------------------------------------
explorer.exe - Memory: 10.89 MB ==> 444.00 KB (PID: 1232)(Path: C:\WINDOWS\Explorer.EXE)
firefox.exe - Memory: 62.86 MB ==> 632.00 KB (PID: 2840)(Path: "C:\Program Files\Mozilla Firefox\firefox.exe" )
explorer.exe - Memory: 13.21 MB ==> 384.00 KB (PID: 120)(Path: "C:\WINDOWS\Explorer.EXE" /IDLIST,:2736:1232,/S)
notepad.exe - Memory: 1.49 MB ==> 144.00 KB (PID: 916)(Path: "C:\WINDOWS\system32\NOTEPAD.EXE" C:\Documents and Settings\User\Desktop\test.txt)
--------------------------------------------------------------

with some minor changes. 

Shane

[Willy2]

No, no. I agree. Don't put too much info in the logfile/in a line. All the suggestions I made in previous posts were specificly meant to reduce the amount of characters/info/words in a ""process""-line in the log file and to increase the readability of the log file. And that would - IMO - be achieved by moving a number of recurring words/info from each line to the header of a separate entry, as described/suggested above. And the characters ""("" and "")"" are not so eye friendly. (perhaps the suggestions I made are those ""minor changes"" you talked about ?)
 
I read how the API called ""EmptyWorkingSet"" works and if I read it correctly (from the webpage of MSDN) it ""takes care"" of one process at a time. I was hoping that a user, in future version, could select all the processes that were activated by the user and be taken care of by Cleanmem and e.g the processes activated by the system weren't touched at all. At least, Taskmanager is able to pull up which process was started by which account (User, Owner, or System). But I guess I'll have to manually add those Owner-processes to the ""Only"" file.

[Shane]

I will have to dig more into the api to find out how to pull the user of a process. I just have cleanmem take care of everything, it hurts nothing. I'm on windows 7 64 bit 8 gb of ram. Yes I have plenty of ram but there is something about a system keeping the unused, left over memory cleaned up that seems to keep things moving smoothly. (Example is explorer.exe will climb to over 100mb of memory for odd reasons, cleanmem keeps it in line)

I know the argument that memory is a resource and should be used not free'ed. And I agree, I just want the memory used when it needs to be is all. Left over memory not being cleaned up is what I want gone

Anyways back to the log, how does this look for readability?

Cleanmem stats: 7/9/2010 9:29:53 PM
Processes Monitored - Memory usage.
--------------------------------------------------------------
explorer.exe - Memory: 10.89 MB ==> 444.00 KB (PID: 1232)
   (Path: C:\WINDOWS\Explorer.EXE)
firefox.exe - Memory: 62.86 MB ==> 632.00 KB
   (PID: 2840)(Path: "C:\Program Files\Mozilla Firefox\firefox.exe" )
explorer.exe - Memory: 13.21 MB ==> 384.00 KB
   (PID: 120)(Path: "C:\WINDOWS\Explorer.EXE" /IDLIST,:2736:1232,/S)
notepad.exe - Memory: 1.49 MB ==> 144.00 KB
   (PID: 916)(Path: "C:\WINDOWS\system32\NOTEPAD.EXE" C:\Documents and Settings\User\Desktop\test.txt)
--------------------------------------------------------------

I am off to take the kids to see a movie. when I get back, once we both agree on the new log format I will release the new version

Shane

[Evan]

Shane,

While you are making changes to the log file why not add a deliminator (a comma or semi-colon) to each "field" so that users can easily import the data into a database program or a spreadsheet program to analyze at a later point or to just look for an easy trend?

Just my quick 2 cents.

-Evan

[Evan]

I think that is good for readability but I still think you should have it easily exported/imported as well.  Now mixing those two...that's when I'll give my thumbs up. 

-Evan

[Willy2]

It seems you're determined to keep the PID info in the log file. Well, in that case I would suggest the following sequence for each process line:

1) process name + memory usage
2) PID code. e.g. "" - PID: 2463 - ""
3) new line (carriage return) + 2, 3 or 4 spaces
4) path of the process (+ switches)

e.g.

explorer.exe, 35.43 Mb ==> 1.73 Mb, PID: 2463 (newline)
    c:\windows\explorer.exe

(a lot of characters less and much friendlier for the eyes)

That indentation after PID and before the path certainly increases the readability of the log. Because the full path, including switches, can be a (very) long sequence of characters.

Over here Cleanmem is running on a laptop with a mere 512 MB Ram and Windows XP. Sometimes Windows is very busy copying files/info from/to the harddisk. And then every bit of performance improvement (e.g. Cleanmem)is VERY welcome.

It's my impression, when I open Taskmanager, that processes started by the user (e.g. explorer and IE 8) are using more CPU time and - at least - (much) more memory than System processes. That's the reason I came up with that ""User"" or ""System"" related suggestion. But that 's something for a future version.

[Shane]

Just got back from the movie, kids had a good time

So we want the ability to have the file easily imported into other programs, using a comma can do this, BUT it would limit how easy the log would be to read.

Example: (With the goal of importing the log to excel)

name, mem, after, pid, path
explorer.exe, 10.89 MB, 444.00 KB, 1232, C:\WINDOWS\Explorer.EXE

To have excel import it in right we cant have any extra text, to labels or anything, the data has to be separated by the comma. So I am not to sure if this is the best way or not.

How ever if we want to keep my own formatting with the goal of easy to read, then how about this for the final?

Cleanmem stats: 7/9/2010 9:29:53 PM
--------------------------------------------------------------
explorer.exe, 10.89 MB ==> 444.00 KB, PID: 1232
    C:\WINDOWS\Explorer.EXE
firefox.exe, 62.86 MB ==> 632.00 KB, PID: 2840
    "C:\Program Files\Mozilla Firefox\firefox.exe"
notepad.exe, 1.49 MB ==> 144.00 KB, PID: 916
    "C:\WINDOWS\system32\NOTEPAD.EXE" C:\Documents and Settings\User\Desktop\test.txt
--------------------------------------------------------------

Shane

[Willy2]

I DO like your latest proposal. Very eye friendly. Frankly and personally, I don't see why the user should be able to import these data into a spreadsheet. It's simply a log file that enables the user to see of what's going on in Cleanmem and not a row of very important figures that needs to be processed in a spreadsheet.

A latest proposal: Add that second header line back in, right after ""Cleanmem stats ........ "".

e.g.

Process monitored - Memory usage - PID - Process path""

But this latest suggestion of a second header line is not a matter of ""Make or break"". I would appreciate it if you would add such a line but if you don't then: so be it.   

Right now, that's my personal opinion. You've got my thumbs up. I am looking forward to see the new log file in the new version of Cleanmem. Thanks for all the efforts.