An impasse with hooking calls to HeapAlloc for a memory tracking application.

I am writing a memory tracking application that hooks all the calls to HeapAlloc using IAT patching mechanism. The idea is to capture all the calls to HeapAlloc and get a callstack.

However I am currently facing a problem with getting the callstack using DBGHELP Apis. I found that the dbghelp dll itself is linking to MSVCRT dll and this dependancy results in a recursive call. When I try to get a callstack for any of the calls from the target application, dbghelp internally calls some method from MSVCRT that again calls HeapAlloc. And since I have already patched MSVCRT it results in an infinite loop.

Has anyone faced this problem and solved it ? Is there any way out of this impasse?

From stackoverflow

• What about using some real memory tracking products like GlowCode?

• This is a standard problem in function interception code. We had a similar issue with a logging library that used shared memory for storing log level information, while the shared memory library had to log information.

  The way we fixed it could be applied to your situation, I believe.

  In your intercept code, maintain a static flag that indicates whether or not you're in the middle of an intercept. When your intercept is called and the flag isn't set, set the flag then do what you currently do, including calling DbgHelp, then clear the flag.

  If your intercept is called while the flag is set, only call the back-end HeapAlloc code without doing any of the other stuff (including calling DbgHelp which is what's causing your infinite recursion).

  Something along the lines of (pseudo-code):

  function MyHookCode:
      static flag inInterceptMode = false
      if inInterceptMode:
          call HeapAlloc
          return
      inInterceptMode = true
      call DbgHelp stuff
      call HeapAlloc
      inInterceptMode = false
      return
  
  function main:
      hook HeapAlloc with MyHookCode
      : : :
      return
  

  Canopus : Thanks for the idea, static may not help me as the calls can come from multiple threads. I think i can use TLS for this purpose.

  paxdiablo : Assuming that TLS is the Windows variant of thread-specific data (i.e., a 'static' per thread), then yes, I would think so. But is HeapAlloc and DbgHelp thread-specific or process-wide? If the latter, you really need a flag that crosses thread boundaries (and is mutex-protected).

  Canopus : Thanks again for the hint. I just found that DbgHelp is not thread-safe. So I also need to have a guard.