Cleaning System Cache

[marcob]

I downloaded CleanMem today and I just gave it a try after copying some big files, with a total size of about 4GB.

Unfortunately CleanMem doesn't seem to work when it comed to cleaning the System Cache, as it only decreases by about 10Mb(System Cache was about 2500Mb, after running CleanMem twice it went down to about 2490Mb).

Since I've copied the files with Total Commander, I thought TC had the System Cache locked somehow so I closed it and ran CleanMem again. Unfortunately nothing improved, the System Cache was still around 2.5Gb afterwards.

After deleting the copied files, the System Cache went down to 100Mb, so it's quite clear what made the bulk of it.

System:
Windows XP Pro with SP2 32-bit
4Gb RAM(only about 3.3 available to Windows)
LargeSystemCache is enabled(HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory ManagementLargeSystemCache == 1)
DisablePagingExecutive is also enabled

I use LargeSystemCache because the machine is a notebook and this way it ensures the HDD gets less I/O, meaning the battery lasts longer and the machine runs a bit cooler.

CleanMem was executed manually not via a scheduled task. I was using the Administrator account. CleanMem had all the rights, it was not limited by any firewall, security software. Both lists(ignore, only) are the default ones, so they are empty.

[Shane]

Well cleanmem just calls a windows api to clean it up. Do me a fav. download cacheset http://technet.microsoft.com/en-us/sysinternals/bb897561.aspx
I have cleanmem cleaning like his (or at least I try to hehe). And if you haven't seen Mark Russinovich work before try out some of his other tools as well. The man knows his stuff

Use it and see what happens. Let me know the results of that as well. When I test on mine it works like it should, so I am very curious if your settings might play a part in it not working (LargeSystemCache).

Let me know

Shane

[marcob]

Hi Shane,

thanks for the very fast reply.
I disabled LargeSystemCache for the sake of testing without it and it's the same.

Here are some test data, step by step, with LargeSystemCache disabled.

Legend:
CS = Cacheset
SI = Mark Russinovich's Process Explorer System Information

1. Before copy
==============
CS Current size: ~10Mb
CS Peak size:     81784 Kb
SI System Cache: ~1.6Gb

2. After copy
=============
CS Current size: ~22Mb
CS Peak size:     81784 Kb
SI System Cache: ~3.1Gb

3. 30 seconds after CS "Clear"
==============================
CS Current size: ~10Mb
CS Peak size:     81784 Kb
SI System Cache: ~3.1Gb

4. After deleting the copied files
==================================
CS Current size: ~11Mb
CS Peak size:     81784 Kb
SI System Cache: ~1.7Gb


It is clear to me that what CS shows is not the real file system cache amount.
Why? Because if I copy 1.5Gb of data followed by another copy of the same data to another destination that second copy takes place much faster, about as fast as the HDD can write the data(60-70Mb/s) while the first copy only manages to get around 30-35Mb. So it is clear that SI System Cache shows the real System File Cache value, and that's the one I'm interested on.

Is there any way to clean it properly?

[Shane]

ok try this with his program, instead of doing clear what happens if you set the min and max? does that have any different effect?

Shane

[marcob]

Changing Working set min and max doesn't seem to have any effect on the SI System Cache.
If I set too low a value for max(tried 12000kb), it gets somehow ignored, CS Current size could reach 15Mb easily.

It's also impossible to set a max value higher than 2Gb, although SI reported a System Cache higher than 3Gb at times.

So, no, it doesn't change anything, no different effect, it is ignored.

[Shane]

I need to confirm this but I believe that the SI System cache is the total system cache itself, not just the file cache. the CS peak size is the max the file cache had climbed to and of course the current size is the current file cache size.

but looking at your stats the current and peak barely climbed while the system cache nearly doubled. Which makes a lot of sense, you have your system set to keep things in memory more to use less disk IO, which would explain why the file cache isn't climbing and the system cache is.

ugh I need to do some googling and pull some proper info on all this hehe

http://technet.microsoft.com/en-us/library/cc757875(WS.10).aspx
http://technet.microsoft.com/en-us/library/cc784562(WS.10).aspx

you disabled LargeSystemCache can you also disable DisablePagingExecutive and reboot and see what test you pull as well?

Shane

[marcob]

LargeSystemCache disabled
DisablePagingExecutive disabled

Legend:
CS = Cacheset
SI = Mark Russinovich's Process Explorer System Information

1. Before copy(see picture)
==========================================
CS Current size: ~10Mb
CS Peak size: ~82Mb
SI System Cache: ~300Mb

2. After copy(see picture)
==========================================
CS Current size: ~38Mb
CS Peak size: ~82Mb
SI System Cache: ~3.1Gb

3. After CS.Apply and CS.Clear(see picture)
==========================================
CS Current size: ~10Mb
CS Peak size: ~82Mb
SI System Cache: ~3.1Gb

4. After the copied files were deleted(see picture)
==========================================
CS Current size: ~23Mb
CS Peak size: ~82Mb
SI System Cache: ~68Mb

I have also tried Cacheset under Windows 7 32-bit which was running under Vmware and it's the same. It doesn't have any influence on System Cache.

[Shane]

good job on those pics.
One thing I did notice though as the system cache spiked and the available memory didn't go down.

http://support.microsoft.com/kb/243325
found this on windows 2000, should be the same for the later versions of windows as well. this does explain why the memory usage isn't changing when the system cache does.

"Because the Windows 2000 Task Manager System Cache field includes the size of the transition list, this field is a more accurate report of caching information. The transition list is memory which has been reclaimed by the operating system from working sets, including the file cache, memory that was used by programs, and other operating system memory usage. The transition list is not counted by Windows NT 4.0 File Cache.

The File Cache field only counts the size of the file cache working set, or that amount of memory which is currently in use by the cache manager. The File Cache field does not count the number of cached memory pages that are in transition."


From the looks of it the system cache shows a lot more than file cache. I don't think watching the system cache if a fair comparison to the file cache. But I'm very interested how deep this runs, I will search for more in depth info on the matter. (also good to know

Shane

[marcob]

Well, the System Cache does contain the files I copied, all of them, if they can fit insithe the System Cache.
It can be very easily seen by copying(in my case, with more memory) 2Gb of files, then copying them again some place else. They are not read anymore, they are written directly from memory(System Cache).

To me this is file caching.

What is really bad is the fact that Windows transfers data from RAM into VM in order to make more RAM available for "System Caching".

[Shane]

that's the part that stands out to me though. If the 2gb file is going into memory why isn't the available memory dropping since it should be in use? Thats the part that has me wanting to look into it deeper.

Shane

[Evan]

Marcob,

It sounds like what you are really wanting to clear is the "StandByList" and not the FileCache.  Just so you know, the SystemCache that both Task Manager and System Information (via Process Explorer) report is a the total of the StandByList and the WorkingSet.

FileCache is part of the WorkingSet and Sysinternals' CacheSet lets you control the WorkingSet parameters.  The StandByList consist of memory pages that were recently part of the WorkingSet but have been moved out of the WorkingSet and are still stored in RAM.  The large file copy you made would be cleared from the WorkingSet and moved to the StandByList...which would explain why your second attempt at the same copy is much faster.  The StandBy Pages are added to the WorkingSet Total because they are cached pages for the WorkingSet.

Unless I've missunderstood, what you are wanting to have "cleaned" is the StandByList which CleanMem and CacheSet do not touch.

You can read more about this over at Microsoft's NT Degubbing Blog here.

Hope that helps you out.

-Evan

[marcob]

The goal is to stop Windows from paging/swapping.

I mean, I've got myself 4Gb RAM(Ok, Windows only sees ~3.3Gb) and that should be, at least theoretically, enough to be able to get by without lots of swapping. Usually the programs don't use more than 1.5Gb RAM and it almost never exceeds 2Gb.

But that's not actually what happens. After lots of HDD I/O Windows starts paging data from RAM in order to make RAM available for more System File Caching.

The real major goal is to have as little HDD I/O as possible.

To achieve that, I was thinking about regularly cleaning the System File Cache, but Cachemem, CleanMem, FreeRAM XP Pro and cFreemem just don't do it right. There's no way a tool that allocates lots of RAM will help.

Any idea what else I could try?

[Evan]

Can you give us any more details of what you are wanting out of your system?  For instance, do you just want to use your computer for normal Home/Office use with the occasional large file transfer; are you using the computer to act as some sort of server; or are you just trying to squeeze as much performance as you can (like Shane and I try to do)?

Have you been able to verify if, after a large file copy and prior to using CleanMem, your system's HDD I/O spikes during "regular use".

I'll be happy to look more into this and discuss it with Shane and see what we come up with.  I can't promise that I can provide a satisfactory answer today...but I won't give up till I do find an answer.

Also, is your desire to lower HDD I/O related specifically to file transfer? or are you using this Notebook for "regular" use and a large transfer just cripples your system?

I've also read that the registry change you made can be ignored if you are on a Domain rather than just a home workgroup.

[Shane]

What he is looking to do is take as much away from the hard drive swapping as he can, this is because the hard drive is the bottle neck of a system, it is the slowest component in today's systems.

But I'm afraid I don't know of anyway to give the control he wants. With all my tools I ask windows to do it, that's the great thing about API's. But I can only do what the api allows. In this case there is no api to give that kind of control. But MS is known for having a lot of undocumented apis to do things. But Im afraid I dont know this one.  To be honest the only person I can think of who would have an answer is Mark Russinovich himself.

Shane

[marcob]

The notebook is used for development, office and the standard web apps(lots of browsers, Thunderbird) and every now and then some Vmware instances running whatever I need for testing. I never play games on it, it doesn't act as a server.

So, what I would like to achieve, is this(sorted by priority):
1. cool running machine, with least possible amount of noise
2. general performance
3. battery time, when running on batteries

Everything is set right with the proper tools(RM CPU Clock).
Except for Windows messing up with the HDD, starting paging when it shouldn't.

Next thing I'll try is putting the page file on a RAM Disk. I already used this trick in the past when I had an X40 with a 1.8" HDD which was really really slow.

Just tried O&O CleverCache, it's useless for what I need, it does what CacheSet does, give or take. Somehow I start thinking this is a Windows problem which cannot be fixed.

The problem starts with Windows not being able to run properly without a page file and continues as Windows pages when it really shouldn't do it.

> Have you been able to verify if, after a large file copy and prior to using CleanMem,
> your system's HDD I/O spikes during "regular use".
Yes it does. Windows pages a lot.

> Also, is your desire to lower HDD I/O related specifically to file transfer?
No. The problem is that after large transfers the HDD I/O skyrockets, Windows really starts swapping.

> or are you using this Notebook for "regular" use and a large transfer just cripples
> your system?
No. While copying files and the like, I usually don't do anything else.