Bookmark and Share

Author Topic: Log format improvement  (Read 34949 times)

0 Members and 1 Guest are viewing this topic.

Offline Willy2

  • PcWinTech Full Member
  • **
  • Join Date: Jul 2010
  • Posts: 277
  • Karma: 7
Log format improvement
« on: July 08, 2010, 01:23:21 AM »
First of all, I have installed Cleanmem (CM) two days ago and I am already impressed. Programs that I regularly need and that keep running in the background now occupy much less memory. And its seems CPU usage has gone down as well. No surprise there, because (it seems ???) Windows doesn't need to write info to the swapfile that often any more. Currently I am in the process of fine tuning the settings, e.g. which applications do I want to be curtailed in their memory usage. And that's why the log file feature is such a wonderful thing.

But reading the log file, in its current format, is a challenge on its own. The format of the text/info can - IMO - be improved dramatically. I suggest the following text format. This is what had in mind (in italics). And please do add a blank line between two entries/two sets of lines.

Cleanmem task: 11-12-2009, 17.52 PM
iexplorer.exe: Memory: 50,65 MB ==> 3.66 MB
explorer.exe: memory,  20.41 MB ==> 5.72 MB
................
................
.................


Cleanmem task:  12-12-2009 ........
................
« Last Edit: July 08, 2010, 01:59:01 AM by Willy2, Reason: typos »

PcWinTech.com Forums

Log format improvement
« on: July 08, 2010, 01:23:21 AM »

Offline Shane

  • Top Geek, err uh Dog.
  • PcWinTech Administrator
  • PcWinTech Guru
  • *******
  • Join Date: Jul 2008
  • Posts: 16,846
  • Location: USA
  • Karma: 523
  • "Knowledge should be shared not hidden."
Re: Log format improvement
« Reply #1 on: July 08, 2010, 08:39:55 AM »
Should only take me 5 min to make the changes.
When I get back from some jobs with my customers today I will get it done for ya  :wink:

Shane
(My weekends belong to my wife and kids, I will try my best to answer all posts daily during the work week)

(About Shane)
Site Owner, Top Admin, Lead Programmer, Wife & 5 kids, Needs a lot more coffee.

When people ask "Why fix what isn't broken?" I reply "To make it better."
"Only a life lived for others is a life worthwhile"
Honor & Respect is all that matters.

Owner & Programmer of: www.pcwintech.com & www.tweaking.com

Offline Shane

  • Top Geek, err uh Dog.
  • PcWinTech Administrator
  • PcWinTech Guru
  • *******
  • Join Date: Jul 2008
  • Posts: 16,846
  • Location: USA
  • Karma: 523
  • "Knowledge should be shared not hidden."
Re: Log format improvement
« Reply #2 on: July 08, 2010, 06:58:09 PM »
Since the program makes a standard txt file I cant do any formats. How does this look?

Cleanmem Memory Stats: 7/8/2010 6:56:52 PM
--------------------------------------------------------------
explorer.exe (PID: 1816): Memory: 10.67 MB ==> 464.00 KB
notepad.exe (PID: 1600): Memory: 132.00 KB ==> 132.00 KB
explorer.exe (PID: 3588): Memory: 7.77 MB ==> 360.00 KB
firefox.exe (PID: 3952): Memory: 44.07 MB ==> 356.00 KB
--------------------------------------------------------------

Cleanmem Memory Stats: 7/8/2010 6:57:03 PM
--------------------------------------------------------------
explorer.exe (PID: 1816): Memory: 6.38 MB ==> 452.00 KB
notepad.exe (PID: 1600): Memory: 132.00 KB ==> 132.00 KB
explorer.exe (PID: 3588): Memory: 5.32 MB ==> 360.00 KB
firefox.exe (PID: 3952): Memory: 20.41 MB ==> 352.00 KB
--------------------------------------------------------------

If you noticed I also added the process ID in case there is more than one process with that name running.  :wink:

Shane
(My weekends belong to my wife and kids, I will try my best to answer all posts daily during the work week)

(About Shane)
Site Owner, Top Admin, Lead Programmer, Wife & 5 kids, Needs a lot more coffee.

When people ask "Why fix what isn't broken?" I reply "To make it better."
"Only a life lived for others is a life worthwhile"
Honor & Respect is all that matters.

Owner & Programmer of: www.pcwintech.com & www.tweaking.com

Offline Willy2

  • PcWinTech Full Member
  • **
  • Join Date: Jul 2010
  • Posts: 277
  • Karma: 7
Re: Log format improvement
« Reply #3 on: July 08, 2010, 11:25:14 PM »
Yes, this is a dramatic improvement over the previous log file format. Personally, I DO prefer a *.txt file over e.g. a *.html file. A *.txt file simply loads much faster with e.g. Notepad than a *.html file with e.g. IE 8.

The PID is also good because on my PC, e.g. IE 8 (always ??) starts two processes. However I would suggest an additional modification along the lines as provided below. (see below, in italics)

Cleanmem stats .....................
Processes monitored - Memory usage.
---------------------------------------------------------
InCD.exe : ............ , 2.54 Mb
iexplore.exe : .......... , 50.43 Mb ==> 5.34 Mb
explorer.exe : .......... , 23.76 Mb ==> 4.21 Mb
defraggler.exe : ........... , 4.28 Mb
ccleaner.exe : .......... , 5.76 Mb
firefox.exe : ............ , 20.38 Mb


Or something like this (see below):

Cleanmem stats .....................
Processes monitored - Memory usage.
---------------------------------------------------------
InCD.exe :
iexplore.exe : .......... , 50.43 Mb ==> 5.34 Mb
explorer.exe : .......... , 23.76 Mb ==> 4.21 Mb
defraggler.exe :
ccleaner.exe :
firefox.exe :


In the log settings file the user can specify which processes are to be monitored. But which processes are curtailed in their memory usage must be specified in the only file. And the (potential) difference between the two could be expressed in the log file (see above). The log file shows that six processes are monitored but only two processes (iexplore.exe and explorer.exe) that are curtailed in their memory usage by Cleanmem.
« Last Edit: July 09, 2010, 01:43:15 AM by Willy2 »

Offline Shane

  • Top Geek, err uh Dog.
  • PcWinTech Administrator
  • PcWinTech Guru
  • *******
  • Join Date: Jul 2008
  • Posts: 16,846
  • Location: USA
  • Karma: 523
  • "Knowledge should be shared not hidden."
Re: Log format improvement
« Reply #4 on: July 09, 2010, 08:36:02 PM »
Cleanmem skips all processes in the ignore list, so having it logged would require rewriting it all, I prefer not to lol.

Also if a person has more than one process with the same name but in different paths I thought the path of the process would be good as well. This is how I have cleanmem making the log now, how does this look? I haven't released anything yet till I am done with you  :wink:

Cleanmem stats: 7/9/2010 8:33:11 PM
Processes Monitored - Memory usage.
--------------------------------------------------------------
explorer.exe - Memory: 6.68 MB ==> 456.00 KB (PID: 1808)(Path: C:\WINDOWS\Explorer.EXE)
explorer.exe - Memory: 8.29 MB ==> 368.00 KB (PID: 1312)(Path: C:\WINDOWS\Explorer.EXE)
notepad.exe - Memory: 152.00 KB ==> 132.00 KB (PID: 2568)(Path: C:\WINDOWS\system32\notepad.exe)
firefox.exe - Memory: 10.36 MB ==> 472.00 KB (PID: 3700)(Path: C:\Program Files\Mozilla Firefox\firefox.exe)
--------------------------------------------------------------


Shane
(My weekends belong to my wife and kids, I will try my best to answer all posts daily during the work week)

(About Shane)
Site Owner, Top Admin, Lead Programmer, Wife & 5 kids, Needs a lot more coffee.

When people ask "Why fix what isn't broken?" I reply "To make it better."
"Only a life lived for others is a life worthwhile"
Honor & Respect is all that matters.

Owner & Programmer of: www.pcwintech.com & www.tweaking.com

Offline Shane

  • Top Geek, err uh Dog.
  • PcWinTech Administrator
  • PcWinTech Guru
  • *******
  • Join Date: Jul 2008
  • Posts: 16,846
  • Location: USA
  • Karma: 523
  • "Knowledge should be shared not hidden."
Re: Log format improvement
« Reply #5 on: July 09, 2010, 09:32:08 PM »
Took it one step further :-)

Instead of the file path, I have it pull the command line used to start the process.
I find this will be even more useful in helping trace memory leaks for apps :-)

Cleanmem stats: 7/9/2010 9:29:53 PM
Processes Monitored - Memory usage.
--------------------------------------------------------------
explorer.exe - Memory: 10.89 MB ==> 444.00 KB (PID: 1232)(Path: C:\WINDOWS\Explorer.EXE)
firefox.exe - Memory: 62.86 MB ==> 632.00 KB (PID: 2840)(Path: "C:\Program Files\Mozilla Firefox\firefox.exe" )
explorer.exe - Memory: 13.21 MB ==> 384.00 KB (PID: 120)(Path: "C:\WINDOWS\Explorer.EXE" /IDLIST,:2736:1232,/S)
notepad.exe - Memory: 1.49 MB ==> 144.00 KB (PID: 916)(Path: "C:\WINDOWS\system32\NOTEPAD.EXE" C:\Documents and Settings\User\Desktop\test.txt)
--------------------------------------------------------------

Shane
(My weekends belong to my wife and kids, I will try my best to answer all posts daily during the work week)

(About Shane)
Site Owner, Top Admin, Lead Programmer, Wife & 5 kids, Needs a lot more coffee.

When people ask "Why fix what isn't broken?" I reply "To make it better."
"Only a life lived for others is a life worthwhile"
Honor & Respect is all that matters.

Owner & Programmer of: www.pcwintech.com & www.tweaking.com

Offline Willy2

  • PcWinTech Full Member
  • **
  • Join Date: Jul 2010
  • Posts: 277
  • Karma: 7
Re: Log format improvement
« Reply #6 on: July 10, 2010, 03:34:14 AM »
Below your latest suggestion (in italics):

1. Cleanmem stats: 7/9/2010 9:29:53 PM
2. Processes Monitored - Memory usage.
3. --------------------------------------------------------------
4. explorer.exe - Memory: 10.89 MB ==> 444.00 KB (PID: 1232)(Path: C:\WINDOWS\Explorer.EXE)
5. firefox.exe - ......................
6. explorer.exe - .......................
7. notepad.exe - ......................
8. --------------------------------------------------------------

My latest suggestions:
-- How important is the PID ? The PID is - IMO - less important than the path of the program. (I am not a software developer). So, perhaps you can omit every PID code in lines #4 through #7.
-- Change line #2 to ""Process monitered - Memory usage - Path"". Then you can omit the words ""Memory"" and ""path"" in lines #4 through #7. Then line #4 can become ""explorer.exe - 10.89 MB ==> 444.00 KB - C:\WINDOWS\Explorer.EXE"". The three items in the line are separated by a ""-"" and one or more spaces only. That's much more eye friendly. But I don't know if instead of the word ""memory"" another word (e.g. ""Systemcache"") can show up in a line in the log file.

-- If you really want to keep the PID code then e.g. line #4 could become ""explorer.exe - 10.89 MB ==> 444.00 KB - 1232 - C:\WINDOWS\Explorer.EXE"" and line #2 could become ""Process monitered - Memory usage - PID - Path"" or ""Process monitered - Memory usage - Program ID - Path"".

I know you don't want to rewrite Cleanmem now but perhaps you can include the suggestions below in a future version.

-- Concerning post #3: I think there's a (small) misunderstanding. I have added 4 processes to the ""only"" list and 13 processes in the ""log settings"" file. But I didn't add any process to the ""ignore"' list. When a process is running AND included in the ""log settings"" list then it shows up in the log file. And the log file shows how much memory a process uses. Up to this point I like what I see. My suggestion was that ONLY if a process was curtailed by Cleanmem then info could be added to the log file showing the memory usage reduction.

-- Perhaps you could include a bit more information in either the header or in a line (or both) of the log. I was thinking of which account (e.g. Administrator, User, Shane or Owner) and/or what kind of process (Owner, System, User) as displayed in Taskmanager.
« Last Edit: July 10, 2010, 05:49:21 AM by Willy2, Reason: typos »

Offline Shane

  • Top Geek, err uh Dog.
  • PcWinTech Administrator
  • PcWinTech Guru
  • *******
  • Join Date: Jul 2008
  • Posts: 16,846
  • Location: USA
  • Karma: 523
  • "Knowledge should be shared not hidden."
Re: Log format improvement
« Reply #7 on: July 10, 2010, 11:28:25 AM »
I think pid can be important when watching the log file while the programs are still running. The point of the PID is for the processes that have the same file name and running multiple. such as IE and explorer and many others.

The command path is very helpful for the same reason.

Total info that I can pull for a process is

Caption -          explorer.exe
Command Line -          C:\WINDOWS\Explorer.EXE
Computer -          COMPUTER309
Creation Date -          Saturday, July 10, 2010 at 9:57:08 AM
Description -          explorer.exe
Executable Path -          C:\WINDOWS\Explorer.EXE
DOS Executable Path -       C:\WINDOWS\Explorer.EXE
Handle -             1376
Handle Count -          664.00
Kernel Mode Time -       155625000
Max Working Size -       1.348 MB
Min Working Size -          200.000 KB
Name -             explorer.exe
Other Operation Count -       297,597.00
Other Transfer Count -       22,887,588.00
Page Faults -          46,367.00
Page File Usage -          25.324 MB
Peak Page File Usage -       26.313 MB
Peak Virtual Size -          132.145 MB
Peak Working Set -       24.781 MB
Priority -             8 (Normal)
Private Page Count -       26,554,368.00
Process ID -          1376
Quota NonPaged Pool Usage -    18496
Quota Paged Pool Usage -       200532
Quota Peak NonPaged Pool Usage -    22816
Quota Peak Paged Pool Usage -    237684
Read Operation Count -       1662
Read Transfer Count -       5758866
Thread Count -          22.00
User Mode Time -          29218750
Virtual Size -          121.813 MB
Windows Version -          5.1.2600
Working Set Size (Memory Usage) -    9.527 MB
Write Operation Count -       28.00
Write Transfer Count -       4,672.00

I don't want the log to get out of hand though. Last thing I want is the log file growing to fast on someone.
So far I want to keep the new log file as

Cleanmem stats: 7/9/2010 9:29:53 PM
Processes Monitored - Memory usage.
--------------------------------------------------------------
explorer.exe - Memory: 10.89 MB ==> 444.00 KB (PID: 1232)(Path: C:\WINDOWS\Explorer.EXE)
firefox.exe - Memory: 62.86 MB ==> 632.00 KB (PID: 2840)(Path: "C:\Program Files\Mozilla Firefox\firefox.exe" )
explorer.exe - Memory: 13.21 MB ==> 384.00 KB (PID: 120)(Path: "C:\WINDOWS\Explorer.EXE" /IDLIST,:2736:1232,/S)
notepad.exe - Memory: 1.49 MB ==> 144.00 KB (PID: 916)(Path: "C:\WINDOWS\system32\NOTEPAD.EXE" C:\Documents and Settings\User\Desktop\test.txt)
--------------------------------------------------------------

with some minor changes.  :wink:

Shane
(My weekends belong to my wife and kids, I will try my best to answer all posts daily during the work week)

(About Shane)
Site Owner, Top Admin, Lead Programmer, Wife & 5 kids, Needs a lot more coffee.

When people ask "Why fix what isn't broken?" I reply "To make it better."
"Only a life lived for others is a life worthwhile"
Honor & Respect is all that matters.

Owner & Programmer of: www.pcwintech.com & www.tweaking.com

Offline Willy2

  • PcWinTech Full Member
  • **
  • Join Date: Jul 2010
  • Posts: 277
  • Karma: 7
Re: Log format improvement
« Reply #8 on: July 10, 2010, 01:44:01 PM »
No, no. I agree. Don't put too much info in the logfile/in a line. All the suggestions I made in previous posts were specificly meant to reduce the amount of characters/info/words in a ""process""-line in the log file and to increase the readability of the log file. And that would - IMO - be achieved by moving a number of recurring words/info from each line to the header of a separate entry, as described/suggested above. And the characters ""("" and "")"" are not so eye friendly. (perhaps the suggestions I made are those ""minor changes"" you talked about ?)
 
I read how the API called ""EmptyWorkingSet"" works and if I read it correctly (from the webpage of MSDN) it ""takes care"" of one process at a time. I was hoping that a user, in future version, could select all the processes that were activated by the user and be taken care of by Cleanmem and e.g the processes activated by the system weren't touched at all. At least, Taskmanager is able to pull up which process was started by which account (User, Owner, or System). But I guess I'll have to manually add those Owner-processes to the ""Only"" file.

Offline Shane

  • Top Geek, err uh Dog.
  • PcWinTech Administrator
  • PcWinTech Guru
  • *******
  • Join Date: Jul 2008
  • Posts: 16,846
  • Location: USA
  • Karma: 523
  • "Knowledge should be shared not hidden."
Re: Log format improvement
« Reply #9 on: July 10, 2010, 02:01:04 PM »
I will have to dig more into the api to find out how to pull the user of a process. I just have cleanmem take care of everything, it hurts nothing. I'm on windows 7 64 bit 8 gb of ram. Yes I have plenty of ram but there is something about a system keeping the unused, left over memory cleaned up that seems to keep things moving smoothly. (Example is explorer.exe will climb to over 100mb of memory for odd reasons, cleanmem keeps it in line)

I know the argument that memory is a resource and should be used not free'ed. And I agree, I just want the memory used when it needs to be is all. Left over memory not being cleaned up is what I want gone :-)

Anyways back to the log, how does this look for readability?

Cleanmem stats: 7/9/2010 9:29:53 PM
Processes Monitored - Memory usage.
--------------------------------------------------------------
explorer.exe - Memory: 10.89 MB ==> 444.00 KB (PID: 1232)
   (Path: C:\WINDOWS\Explorer.EXE)
firefox.exe - Memory: 62.86 MB ==> 632.00 KB
   (PID: 2840)(Path: "C:\Program Files\Mozilla Firefox\firefox.exe" )
explorer.exe - Memory: 13.21 MB ==> 384.00 KB
   (PID: 120)(Path: "C:\WINDOWS\Explorer.EXE" /IDLIST,:2736:1232,/S)
notepad.exe - Memory: 1.49 MB ==> 144.00 KB
   (PID: 916)(Path: "C:\WINDOWS\system32\NOTEPAD.EXE" C:\Documents and Settings\User\Desktop\test.txt)
--------------------------------------------------------------

I am off to take the kids to see a movie. when I get back, once we both agree on the new log format I will release the new version :-)

Shane
(My weekends belong to my wife and kids, I will try my best to answer all posts daily during the work week)

(About Shane)
Site Owner, Top Admin, Lead Programmer, Wife & 5 kids, Needs a lot more coffee.

When people ask "Why fix what isn't broken?" I reply "To make it better."
"Only a life lived for others is a life worthwhile"
Honor & Respect is all that matters.

Owner & Programmer of: www.pcwintech.com & www.tweaking.com

Offline Evan

  • Global Moderator
  • PcWinTech Guru
  • *****
  • Join Date: Feb 2009
  • Posts: 1,306
  • Karma: 45
Re: Log format improvement
« Reply #10 on: July 10, 2010, 02:03:17 PM »
Shane,

While you are making changes to the log file why not add a deliminator (a comma or semi-colon) to each "field" so that users can easily import the data into a database program or a spreadsheet program to analyze at a later point or to just look for an easy trend?

Just my quick 2 cents.

-Evan
About Evan:Site Admin, Tester, Editor

Offline Evan

  • Global Moderator
  • PcWinTech Guru
  • *****
  • Join Date: Feb 2009
  • Posts: 1,306
  • Karma: 45
Re: Log format improvement
« Reply #11 on: July 10, 2010, 02:05:33 PM »
I think that is good for readability but I still think you should have it easily exported/imported as well.  Now mixing those two...that's when I'll give my thumbs up.  :tongue:

-Evan
About Evan:Site Admin, Tester, Editor

Offline Willy2

  • PcWinTech Full Member
  • **
  • Join Date: Jul 2010
  • Posts: 277
  • Karma: 7
Re: Log format improvement
« Reply #12 on: July 10, 2010, 02:13:44 PM »
It seems you're determined to keep the PID info in the log file. Well, in that case I would suggest the following sequence for each process line:

1) process name + memory usage
2) PID code. e.g. "" - PID: 2463 - ""
3) new line (carriage return) + 2, 3 or 4 spaces
4) path of the process (+ switches)

e.g.

explorer.exe, 35.43 Mb ==> 1.73 Mb, PID: 2463 (newline)
    c:\windows\explorer.exe

(a lot of characters less and much friendlier for the eyes)

That indentation after PID and before the path certainly increases the readability of the log. Because the full path, including switches, can be a (very) long sequence of characters.

Over here Cleanmem is running on a laptop with a mere 512 MB Ram and Windows XP. Sometimes Windows is very busy copying files/info from/to the harddisk. And then every bit of performance improvement (e.g. Cleanmem)is VERY welcome.

It's my impression, when I open Taskmanager, that processes started by the user (e.g. explorer and IE 8) are using more CPU time and - at least - (much) more memory than System processes. That's the reason I came up with that ""User"" or ""System"" related suggestion. But that 's something for a future version.
« Last Edit: July 10, 2010, 05:27:49 PM by Willy2 »

Offline Shane

  • Top Geek, err uh Dog.
  • PcWinTech Administrator
  • PcWinTech Guru
  • *******
  • Join Date: Jul 2008
  • Posts: 16,846
  • Location: USA
  • Karma: 523
  • "Knowledge should be shared not hidden."
Re: Log format improvement
« Reply #13 on: July 10, 2010, 05:54:12 PM »
Just got back from the movie, kids had a good time :-)

So we want the ability to have the file easily imported into other programs, using a comma can do this, BUT it would limit how easy the log would be to read.

Example: (With the goal of importing the log to excel)

name, mem, after, pid, path
explorer.exe, 10.89 MB, 444.00 KB, 1232, C:\WINDOWS\Explorer.EXE

To have excel import it in right we cant have any extra text, to labels or anything, the data has to be separated by the comma. So I am not to sure if this is the best way or not.

How ever if we want to keep my own formatting with the goal of easy to read, then how about this for the final?

Cleanmem stats: 7/9/2010 9:29:53 PM
--------------------------------------------------------------
explorer.exe, 10.89 MB ==> 444.00 KB, PID: 1232
    C:\WINDOWS\Explorer.EXE
firefox.exe, 62.86 MB ==> 632.00 KB, PID: 2840
    "C:\Program Files\Mozilla Firefox\firefox.exe"
notepad.exe, 1.49 MB ==> 144.00 KB, PID: 916
    "C:\WINDOWS\system32\NOTEPAD.EXE" C:\Documents and Settings\User\Desktop\test.txt
--------------------------------------------------------------

Shane
(My weekends belong to my wife and kids, I will try my best to answer all posts daily during the work week)

(About Shane)
Site Owner, Top Admin, Lead Programmer, Wife & 5 kids, Needs a lot more coffee.

When people ask "Why fix what isn't broken?" I reply "To make it better."
"Only a life lived for others is a life worthwhile"
Honor & Respect is all that matters.

Owner & Programmer of: www.pcwintech.com & www.tweaking.com

Offline Willy2

  • PcWinTech Full Member
  • **
  • Join Date: Jul 2010
  • Posts: 277
  • Karma: 7
Re: Log format improvement
« Reply #14 on: July 10, 2010, 10:44:09 PM »
I DO like your latest proposal. Very eye friendly. Frankly and personally, I don't see why the user should be able to import these data into a spreadsheet. It's simply a log file that enables the user to see of what's going on in Cleanmem and not a row of very important figures that needs to be processed in a spreadsheet.

A latest proposal: Add that second header line back in, right after ""Cleanmem stats ........ "".

e.g.

Process monitored - Memory usage - PID - Process path""

But this latest suggestion of a second header line is not a matter of ""Make or break"". I would appreciate it if you would add such a line but if you don't then: so be it.   :undecided:

Right now, that's my personal opinion. You've got my thumbs up. I am looking forward to see the new log file in the new version of Cleanmem. Thanks for all the efforts.  :cheesy:
« Last Edit: July 10, 2010, 10:59:33 PM by Willy2, Reason: typos »

PcWinTech.com Forums

Re: Log format improvement
« Reply #14 on: July 10, 2010, 10:44:09 PM »

 

email