How to reduce memory usage per virtual server?

Memory usage in Windows Server

  • Why might a process not use all available physical memory under Windows Server 2008 r2 standard? I have 32Gb of RAM acknowledged by Resource Monitor, of which 19.4 are allocated to a process (the IBM solver CPLEX running in command line form from Power Shell) that has just thrown a controlled OOM error despite there being another 12 Gb marked as available. I'm going to assume (to start with) that the program itself isn't the culprit, but is there a per-user limit switch, a per-process limit switch or something similar in 2008 r2? Any other ideas?

  • Answer:

    I know nothing about CPLEX and suspect few people here do. A quick Google search for http://www.google.com/search?q=cplex%20memory%20usage turns up a "workmem" parameter that can be configured. Its default is 512kb, so it must be some sort of small working set size and not the full system usage. A search for http://www.google.com/search?gcx=c&sourceid=chrome&ie=UTF-8&q=cplex+workmem has many promising leads. More generically, my first thought was it was a Java program with a fixed memory limit, but CPLEX is not Java. zengargoyle's advice on a 4G process limit is possible but I'm a bit skeptical. But be sure you're running a 64 bit system and program. (Sorry if this answer is not useful; your best bet is to find other CPLEX users and ask them.)

cromagnon at Ask.Metafilter.Com Visit the source

Was this solution helpful to you?

Other answers

One possibility is that the process requested more than the remaining amount of memory. I see this happen often with the version of malloc that I program against. It grabs an initial block of memory from the heap. When that all gets allocated, it grabs a second block the size of the first. When that gets used up, it requests a block that size of what is in use, doubling the size of memory under management each time. Eventually, it requests more than is available, even though all memory isn't in use and my program might not ever need that much. Alternatively, there could be enough memory available, but not in a single piece due to fragmentation, so malloc can't return a single piece large enough so it fails.

procrastination

How many cores? How many processes does CPLEX use? Is every bit of everything CPLEX uses true 64 bit code? Spidey sense tingles with 4 worker and 1 aggregator processes each hitting their 4G 32 bit memory limit minus 4 or 5 chunks of memory that aren't actual allocated memory but shared mapped memory for IPC. 4 workers @ 4G plus the master at 4G but 4x128M of that is shared to communicate with the workers == ~19.5G. (could also be a 4G per process limit with 5 processes instead of a 64/32 bit limit).

zengargoyle

Thanks all - solution is a bit of everything suggested here. CPLEX is absolutely 64-bit throughout. The workaround seems to be a suite of memory efficiency routines of which workmem is one, and which effectively stop a progress trace being held in RAM. Forcing the trace to be written to disk is dog slow, but it does at least run. It's parallel code, and it does look as though there's a preemptive survey of available RAM to allow graceful failure ahead of one core actually failing, which explains the odd limit of RAM observed - it's not an OOM error so much as a HAL-style warning. Much obliged!

cromagnon

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.