Windows memory usage

• How can I force Windows to allow an application (Stata) to allocate itself more memory, when I have plenty of unused physical memory? I recently upgraded to 2Gb of physical memory, and I'd like Stata to be able to use as much of this as possible so that I can work on large data files. The memory allocated for data is set manually with a command in Stata (http://www.stata.com/help.cgi?set%20memory), but Windows (XP Home Edition) will only provide a maximum of about 745Mb before Stata reports "op.sys. refuses to provide memory". I was able to use up to 450Mb when my machine had only 768Mb of physical memory, so surely I ought to be able to get far more than 745Mb out of 2Gb. I have no other applications running, and no memory-intensive processes, so ought to have plenty of physical memory free.

• Answer:

  Memory in windows is split in 2 regions, kernel space and user space, both are 2gig if you don't use the /3gb hack which increases the user space to 3gb (can cause you problem because you have less kernel space, especially with your video drivers but people have been known to work with it). Applications allocate memory from windows using calls to VirtualAlloc which maps a region a virtual address space (all process have their own 2gb virtual address space) to physical memory (the actual thing on your motherboard). So address 0x10000000 that is returned by VirtualAlloc can be mapped to addresses 0x0000F000 on the actual RAM but you don't have to worry about it Windows will take care of it for you. So essentially your application can always allocate memory until it hits the 2gb barrier (if your system has less than 2gb of memory it'll be paged to disk and be very slow). The reason an allocation can fail is either because the 2gb space is all allocated or because it's trying to allocate a contiguous chunk that is too big (ie, you have 5 megs free in 2 blocks of 2 and 3 megs and you want a continuous block of 4 megs, this will not work). Now the main causes of memory fragmentation are inefficient heap allocators (lots of small blocks mapped and unmapped in virtual memory, your user space is now a swiss cheese), allocations of inefficient size (on most win32 systems the minimum size that should be allocated is 64k not 4k, 4k vallocs will waste 60k due to internal constraints in the virtual memory system) or dlls being loaded in the middle of your address space thus creating fragmentation. Now it seems that stata isn't doing such a great job at managing memory since you still have user space available (maybe... check the VM Size column in task manager, avalaible by selecting it in View | Select Columns... ) and it can't allocate more memory. This might be because it's trying to allocate it's memory in one big chunk (this is a bad programming practice, but out of your control). The only thing you can (maybe) control is the dlls base address, if those are creating fragmentation you can rebase them with EDITBIN (comes with Visual Studio, probably available elsewhere). Now to do that you first have to determine if the dlls are causing the fragmentation and which ones to move. You can use Dependency Walker to explore which dlls are used by stata and see their preferred base address (no guarantee they're loaded there). You can use graph paper (or excel) to graph them and move all those that fall in the middle of the user space to the lower regions (make sure they don't collide with other dlls or they'll be rebased elsewhere when loading). This is a fairly involved process though. I'd try the /3gb first it might be easier, I don't see why it wouldn't work with Home Edition since the differences between windows skus are mostly in the packaged applications and default UI settings, the kernel is the same. But they might have actually disabled it to make sure people who needed it bought the Professional version. Also consider that if you have 2gb of memory used by stata your system will probably swap to disk excessively and be very unresponsive, but you can probably get more than 754mb of memory used.