So last week I fixed Bug 1282606 which has resulted in a bit of a performance improvement for bugzilla.mozilla.org:
Apache2::SizeLimit
is configured to kill processes once they use more than 700M and this happened about every 7 minutes.
About two weeks ago, while working on some performance issues relating to BMO’s new show_bug ui, I discovered that the problem could get worse: running out of memory every 60 seconds. Should everyone switch to the new UI (which is intended to put less load on the server) a lot more load would be on the server. That’s pretty bad, since we want everyone using the new UI as soon as possible. 🙂
This memory leak isn’t new, and I had filed an investigatory bug about it last year. Memory leaks in perl are caused by having cyclic references, and the solution is to not have cycles, use weak references, or to break the cycle when you’re done with whatever data structure it is part of.
I understand the problem, and I know how to fix it… but maybe I don’t know where the problem is?
Thankfully there is a tool for this on CPAN: Devel::MAT.
Using Devel::MAT, it is possible to dump the address space of a perl program and explore it in great detail in a GUI.
I didn’t set out to remove all the memory leaks this time, just the ones that were the biggest or grew the fastest. This meant the TrackingFlags extension Flag objects, the Bug object, and the Comment object.
The changes are on github for the curious,
and the resulting charts below speak for themselves.
snippets 