BMO ❤️ Carton

Back when I started working on BMO
we couldn’t add new dependencies without having someone build an RPM. For no particularly good reason, this made it so in general we didn’t add new dependencies often.

However, about a year ago I started poking at carton and came up with a process to run carton in a docker container that mirrors production, and tar up the resulting local/ directory.

For the last 6 months or so we have been able to add dependencies whenever we want. We can also track changes to the
full dependency tree.

The code for this is on github as mozilla-bteam/carton-bundles and it is a little ugly, but packaging code is rarely elegant.

Optimizing BMO: Part 1, An Easy Fix

One way of squeezing performance out of apache, as noted in this blog post by Hayden James is to disable htaccess files – which are not needed when you have control over the httpd’s config files. Doing this allows the web server to spend less time calling stat() for .htaccess files that don’t even exist – for instance a request to https://something.something/foo/bar/baz is at least 4 calls to stat() .htaccess (once at /, then /foo, then /foo/bar, and finally /foo/bar/baz assuming that baz is also a directory.

As it turns out for BMO this is even easier: bugzilla already sends configuration to the apache process.

Because of this, we can search for .htaccess files at apache startup time, load them using the server->add_config() method and tell apache to not bother looking for them during subsequent requests.

This change was quite small and the performance gain in production should be noticeable (but not large). As it turns out, some of those stat() calls also hit NFS, which will be a discussion for Part 2.

Bigger and better search box on BMO

There is a lot of power hidden in quick search and my data suggests it is under-utilized.

For instance, searching for all review flags is the literal search tag review?
Similarly you can do needinfo?dylan@mozilla.com to find all bugs with needinfos directed at me (actually, this query performs a slightly broader search but I have code to fix that).

There are dozens more examples. The quick search help is a long read, and most people won’t bother.

A long time ago glob suggested stealing the UI from DXR, where you get a little quick-help on the operator
syntax for DXR searches. It’s a pretty simple change, right?

Well, our search box is small. So the first thing it needs is to be bigger.

bigger quicksearch box

More room to work with. This required replacing the table-based layout with some flex boxes. The top-line is nearly pixel perfect
to its previous table-based implementation.

We can also hide some things and begin making the UI responsive

portrait view of bmo quicksearch.

I hope to post a followup showing the quicksearch syntax helper,
but this is at the moment just a side task.

(Although it ties well into the goal of implementing elasticsearch on BMO).

Sorry, I meant I changed it from 226s to 184ms

About twenty days ago it came to my attention via my colleague Ed Morley that BMO’s bzapi was very slow.
It turns out he had reported the same issue the prior year as well!

Performance problems are very enjoyable to work on, I find. Especially when they are reproducible.

I lost most of the day on this, but in the end I was able to take the slowest function from executing at a leisurely 226 seconds to a very fast 184 milliseconds

Perl + Bugzilla in Outreachy Round 13!

I am very proud to be mentoring in the 13th round of Outreachy.

We have a a list of ideas
and a growing list of bugs that would make for good “small contributions”.

I’m open to emails or irc discussions and I’ll try to answer
any and all questions.

email: dylan [at] mozilla [dot] com
or join irc.mozilla.org #bugzilla, I’m in IRC from
13:00 UTC until about 22:00 UTC.

Here’s are project blurb:

Perl is a highly capable, feature-rich programming language with over 28 years of development, making it one of the longest standing FOSS projects. The Perl Foundation is funding a position working on Bugzilla, a widely used, Perl-based issue tracker. In 18 years of development, Bugzilla has grown into a complex application that is used in many different workflows by organizations including Mozilla, the GNOME Project, Red Hat, and freedesktop.org. Some of this complexity is particularly evident in the search functionality, both in implementation and in user interface. We have several proposals to simplify and improve searching, which will positively impact Bugzilla sites around the world.

Fixed some memory leaks in bugzilla.mozilla.org

So last week I fixed Bug 1282606 which has resulted in a bit of a performance improvement for bugzilla.mozilla.org:

Restarts per Hour

Apache2::SizeLimit
is configured to kill processes once they use more than 700M and this happened about every 7 minutes.

About two weeks ago, while working on some performance issues relating to BMO’s new show_bug ui, I discovered that the problem could get worse: running out of memory every 60 seconds. Should everyone switch to the new UI (which is intended to put less load on the server) a lot more load would be on the server. That’s pretty bad, since we want everyone using the new UI as soon as possible. 🙂

This memory leak isn’t new, and I had filed an investigatory bug about it last year. Memory leaks in perl are caused by having cyclic references, and the solution is to not have cycles, use weak references, or to break the cycle when you’re done with whatever data structure it is part of.

I understand the problem, and I know how to fix it… but maybe I don’t know where the problem is?

Thankfully there is a tool for this on CPAN: Devel::MAT.

Using Devel::MAT, it is possible to dump the address space of a perl program and explore it in great detail in a GUI.

I didn’t set out to remove all the memory leaks this time, just the ones that were the biggest or grew the fastest. This meant the TrackingFlags extension Flag objects, the Bug object, and the Comment object.

The changes are on github for the curious,
and the resulting charts below speak for themselves.

Average Request Time

Requests Before Restart

Age of Process Before Restart

Bugzilla on heroku? And hacking on Memcached::libmemcached

I thought I’d take this long weekend to do something fun: get Bugzilla running under Heroku.

Steps:

  1. fork the perl/psgi buildpack to run bugzilla.
    bugzilla buildpack
  2. swear at Gerv for adding login name support and swear at
    checksetup.pl for spinning in an infinite loop when it can’t prompt for a missing config Bug 1284021.
  3. start writing support for storing the “params” data in the db instead of the filesystem, as the filesystem in heroku is ephermal.
  4. realize that you’ll want to memcache this, so might as well add memcache to heroku
  5. realize MemCachier (one of the herkoku memcached providers) requires username and passwords and that the perl bindings don’t support this

After some research… I realize this feature requires support for the binary protocol and is based on SASL. Fine. I’ll learn XS (perl’s FFI) and contribute code to Memcache::libmemcached. How hard can it be?

It turned out to be not very hard
but it’s a work in progress (and definitely leaks memory right now).

Oh yeah, and bugzilla does run on heroku, but it won’t be useful until the params stuff can be stored in the DB.

Developing Bugzilla with Plackup and cpanminus

(note: these notes will be incorporated into bugzilla documentation after some revisions)
(updated: Fri, May 13 2016 at 10:24 US/Eastern with additions from dkl)

Firstly, we need some system dependencies:

System Dependencies for Debian and Ubuntu users

$ sudo apt-get update
$ sudo apt-get install git perl-modules build-essential cpanminus 
   libssl-dev libexpat1-dev

System Dependencies for CentOS users

TODO

Bugzilla Code & CPAN Modules

A checkout of bugzilla is required, and everything we do
will be from that directory.

$ git clone https://github.com/bugzilla/bugzilla
$ cd bugzilla

Next, we can run checksetup.pl with its fancy new –cpanm flag to install all our dependencies into local directory. Note that we do not need to be root for any of these commands, and nothing (from this point on) needs to be installed to the system.

$ perl checksetup.pl --cpanm

If all goes well, that should complete with a message saying you need to edit localconfig and re-run checksetup. If you’re fine just using SQLite, you do not need to edit localconfig. Just re-run checksetup.pl, as below:

$ perl checksetup.pl

The above will prompt you for an email, username, realname and password. After you provide those details, it will configure the database.

Now we can run a development server, with the following perl command below:

$  perl -Ilocal/lib/perl5 local/bin/plackup -R Bugzilla -R Bugzilla.pm -R templates app.psgi

This will start an http on http://localhost:5000.

Navigate to it, and login using the email/password you used above.

Bugzilla will direct you to set the urlbase – which you can set to to the same “http://localhost:5000” above.

Any changes made to files in Bugzilla and templates will trigger a restart. Currently changes to .cgi files might not be picked up, but that should be solved shortly.

Note that these are draft instructions, I appreciate testing and feedback on them. I will fill the todo items in as time allows, and hopefully we get this into the documentation for the Bugzilla 6.0 release.