Cheating for fun and profit

Yesterday I had one of my weirdest debugging sessions in the last few years. Background: for some time we’ve been having problems with a third party library that comes without source code, only .lib/headers (no names given to protect innocent). It seems like not all objects were deleted after deinitialization. After basic investigation it turned out to be some kind of problem with reference counting, objects couldnt be removed because they were still referenced. It was literally hours before launch, so best I could do was to hack some Release() here and there and pray for the best. It “solved” it in most common scenarios, but obviously couldnt be considered satisfying, so I decided to give it another shot now that we’ve more time.

As said before - it was only binaries (w/o debugging info), documentation wasnt really very detailed, some of the used functions werent documented at all, library was no longer supported and it was even hard to find someone who had rights to it. To make the picture complete, programmer who coded this subsystem on our side left like 3 years ago.

We werent increasing reference counters explicitly and it looked like we were releasing them all properly, so most probably it was the problem with internal references. Library was pretty much black box, most of the objects were identified with handles, so there was no easy way to check what’s exactly increasing the reference count. All we had was an information that object A has non-zero reference count and wont be deleted. No details about who’s actually still referencing it. Sidenote: if you use RC in your commercial engine, please DO make sure there’s an option to see tree of object relationships. In my **home **engine I can easily get the complete callstack of every call to AddRef, it helps immensely in debugging this kind of problems.
Best bet would be to set data breakpoint on the counter, but for that I needed exact memory address. It sounded a little bit like stuff I was doing when I was younger – trying to find memory address where life or money of my hero in game were stored. Back then I was using Game Wizard (if memory serves) for that, I searched the web for something similar and voila! - found Cheat Engine. It does everything GW did and much more! Armed with that tool everything turned out much easier. I scanned memory searching for current reference count, then increased it few times, re-scanning every time and got the address.

After that I could play with the counter the way I wanted and observe what happens, I could even freeze it (oh, good memories of “cracking” games with this one), see when it changes (it can be traced in Cheat Engine, but it’s more useful to set data breakpoint in Visual Studio, that way full callstack can be obtained). It was a little more complicated, because at the first try I caught the counter too late. I had to find offset of my object in its parent, now knowing relative offset of the counter in the child structure set the breakpoint on that and go through the entire load/initialization process. It’ll need some further, more complicated tests on Monday but it seems like I finally found a clean solution (which, as usual turned out to be very trivial – you just dont try to manage low level objects by yourself, just let the library own them… even if dont own them, it’ll still increase ref count, but wont release it(?)). Lesson learned: Cheat Engine will stay on my HDD just for the moments like that, another tool in my box (I only scratched the surface, intend to play with it more during the weekend).

I made another interesting observation during this session. For the whole Friday I was alone in the room, all the other guys were at some Microsoft Conference. Now, there are normally 7 programmers in our room, it’s rather quiet… or so I thought. When sitting alone my productivity just skyrocketed. It was virtually 9 hours, non-stop debugging session, totally focused, no interruptions, “in the zone” the whole time. I realize it’s not possible or desired for everyone to work like that 100% of the time, but it was an interesting experience nonetheless. Another thing is that I felt almost exhausted at the end of the day, there was a feeling of good job done, but at the same time I felt it’s enough, I need to go home (even if some more tests need to be performed).

Old comments

addref 2010-03-28 20:12:14

[…] … I assume that if QI is not addref'ing, then if I use a smart pointer once, the reference …Cheating for fun and profit | .mischief.mayhem.soap.Yesterday I had one of my weirdest debugging sessions in the last few years. Background: for some […]