An apology for impetuous tweeting

I’d like to apologize to this tweet. The events were playing out, and it felt urgent that I act and so I did. I think now I would have said nothing, but still written the letter I ended up writing. That tweet and a few others1 felt impetuous and not how I would like to present myself.

I’m also aware that I hurt at least one close friend, and I am sorry for that. I will try harder to think before I tweet.

  1. There is one where I insulted someone in a mean way that I deleted, but my mistakes remain published.

Open Letter to European Perl Conference

To the organizers of the European Perl Conference in Riga.

I have been involved in the perl community for 22 years. I feel a moral obligation to ensure people that I have introduced to it can continue to feel welcomed and safe. This letter is not meant to shame anyone, but it is public to provide some level of proof that people in the community care about enforcing codes of conduct.

Understanding

There’s actually two issues at hand. The first one is that a person violated the Standards of Conduct (Code of Conduct) at the last Perl Conference in Pittsburgh and is slated to be a keynote speaker for the Perl Conference in Riga.

UPDATE: The keynote speaker issue has been resolved. The rest of this letter remains accurate.

The second issue is that after some prominent people raised concern, there was a perhaps hastily-written blog post that ended with a transphobic joke. The blog post was subsequently edited to remove the joke, along with some tweets relating to the issue.

Adding to the first issue, some may question the acceptability of deadnaming if the victim does not care. It would appear that is still a violation of Standards of Conduct.

Some have already pointed out the bad optics around this, and I’m not here to talk about that. I do care about how this looks to the world at large. This is an open letter precisely because handling this issue in secret would be worse than talking about it.

We’re going to cover Safety and Gender, and at the end some ways which we can resolve this issue. I am not the best person to cover these topics, but I am doing so because that lifts the burden from members of the minority having to explain again and again why something hurt because people want facts.

Safety

People will report Code of Conduct violations. The important question is, how will you handle those incidents and enforce your CoC?

Sage Sharp, 2016-01-25

The underlying issue is one of Safety. Vulnerable individuals attending conference must have faith that the code of conduct will be upheld. No one should be above reproach, and if a person violates the code at an event, it should inform decisions towards that person at other events.

This does not necessarily mean that the person need to be barred from the event — I do not hold an opinion on that matter — but some obviously feel that having the person be a keynote speaker does send a signal that perhaps the conference does not take ensuring the safety of attendees seriously.

You do not have to agree with me on this issue, but you also cannot argue about what trade-offs people make with their personal safety.

For additional information about safety, there is no better source than this collection of FAQs about Codes of Conduct

Gender

Assuming you’re still with me, you agree that people need to feel safe. But you’re struggling because you don’t think the behavior was serious enough.

we’re not talking about serious stuff! He just used the wrong name and pronouns!

some random internet person

This belief is false. Misgendering trans people causes harm.

In the AP News article Misgendering is not a lightweight ‘mistake’ Karolyn Wilson explains that empathy can inform sympathies:

I can’t speak for transgender men and women, but empathy can inform my sympathies: if I feel insulted and demeaned when I am misgendered, how much worse is it for someone who has had to work so much harder than I have to make their outsides match their insides, for someone who is in so much more danger of being discounted as a person or persecuted for who they are?

Karolyn Wilson, Misgendering is not a lightweight ‘mistake’

If you’re reading this and you’re cisgender, presumably you can think back to a time when you were misgendered. If that is not the case, perhaps imagine what it would feel like.

These situations, imagined or otherwise to this tend to have reactions that fall somewhere in a spectrum, but with typical reactions being:

  • You could feel as Karolyn Wilson did — as less of a person, less solid, and less seen
  • Or perhaps you will feel as I did when I first thought about this experience: Indifferent.

If you feel indifferent, and are not yet capable of understanding why gender identity is important you must start believing the lived experience of trans people. You should watch this video by Vi Hart (5 minutes). A quote from this video resonated with me when I first saw it:

My condescending teenager attitude came from a false belief that other people are basically like me.

Vi Hart

I think watching that video and internalizing the line of reasoning Vi Hart uses can help those that are cisgender but not strongly attached to the concepts or expectations of gender.

Now at this point, if you’re still reading I hope you’re with me. Perhaps you’ve read to this point, but you take issue with the joke being characterized as transphobic.

The reason people had a negative reaction to the comment about the perl conference identifying with a different name is because the conference is not a person and drawing this absurd parallel is an attack. This is quite similar to saying ”I identify as an attack helicopter” which is also transphobic and an attack on the concept of gender identity.

There isn’t space to go into this, but this video about gender and this video transphobia are not very long and cover things much better than I can.

I guess I’m done with this topic. As after this some people may label me as a SJW, I’ll pre-emptively tell you to watch The Straight White Man’s Guide to Feminism and Social Justice.

Resolution

Right now people are upset. Several prominent people have called for a boycott of the conference. I’m sure this is not intended. Organizing a conference is very hard, it’s difficult to get the right kinds of help. It’s very stressful and then this happens and people are making demands and I know it must feel like you’re being attacked.

When people say things like “I am appalled by something you have done” it doesn’t mean they are attacking you. It coveys some amount of surprise, and they’re telling you this because they think you are better than the action and can decide to do something different.

Let’s walk it back. There are concrete steps that can be taken to alleviate this problem, and in fact shine.

  1. The SoC violating keynote speaker cannot be a keynote speaker.
  2. An apology for the removed blog post. It was a natural mistake — people will easily forgive such as thing if the apology is genuine.
  3. A commitment to safety must be made. It is not apparent if this conference has a Code of Conduct. The Glasgow conference did, but I don’t see this mentioned initially but I did not look very closely.

The Binding of libcmark-gfm: Segfaults and Debugging

So for various reasons — including “I want bugzilla.mozilla.org to support markdown” I have been working to get a binding to github’s fork of cmark.

For the first part of this, I got some help in #native
writing a Perl module in the Alien:: namespace,
namely Alien::libcmark_gfm.

Armed with this module, I’ve been seeking to make CommonMark
work against the GitHub fork of libcmark.

So far things have been going well, and I decide to just be dumb. The API for libcmark-gfm is a bit different, so I’ll rename the packages from CommonMark to CommonMarkGFM.

Of course, this was the first problem: I was getting
errors about a package not existing, a package named CommonMarkGFM::N. What the hell does that mean? I haven’t changed much yet!

The problem was this bit of C code in the newly-renamed CommonMarkGFM.xs:

stash = gv_stashpvn("CommonMarkGFM::Node", 16, GV_ADD);

Okay, now I don’t know perlguts very well.
I don’t know what gv_stashpvn does (but I can find the docs for gv_stashpvn
and the name is a hint at what it does, in the terse nomenclature of Perl’s internal APIs)

The old string was 16 bytes long. Now it should be 19,
and that perfectly explains why I saw CommonMarkGFM::N.

So I get past that. and now the test suite segfaults.

1..10
ok 1 - use CommonMarkGFM;
ok 2 - markdown_to_html
ok 3 - 'parse_document' isa 'CommonMarkGFM::Node'
Segmentation fault

Hey, maybe this is the same as the first problem I fixed?

So I go looking for that problem, and I find it!
We have some lengths hard-coded in the typemap file
(no, aside from the fact it maps types, I don’t know what the typemap file does. I’m not usually hacking in perlapi).

T_NODE
    $var = (cmark_node*)S_sv2c(aTHX_ $arg, \"CommonMarkGFM::Node\", 19, cv,
                               \"$var\");
/* more omitted */

So I fix those problems, but they were not my problem.
I’m still getting a segfault…

I’m really quite excited at this moment! I have a problem that I can apply things I learned about from this wonderful blog by Julia Evans.

I’ve already been using a Dockerfile to try to compile and test this code so I just need to install Valgrind (and maybe gdb too) and see what happens.

So I run valgrind:

==16==  Access not within mapped region at address 0x88
==16==    at 0xEF4685C: cmark_render_html_with_mem (in /usr/local/lib64/perl5/auto/share/dist/Alien-libcmark_gfm/lib/libcmark-gfm.so.0.28.3.gfm.12)
==16==    by 0xED0A11D: XS_CommonMarkGFM__Node_interface_render (CommonMarkGFM.c:898)
==16==    by 0x4ED6814: Perl_pp_entersub (pp_hot.c:2888)
==16==    by 0x4ED4B05: Perl_runops_standard (run.c:40)
==16==    by 0x4E7D0D7: perl_run (perl.c:2435)
==16==    by 0x400E73: main (perlmain.c:117)

Huh, interesting. Okay, maybe I can use gdb to set a breakpoint there.

(gdb) b cmark_render_html_with_mem
Function "cmark_render_html_with_mem" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (cmark_render_html_with_mem) pending.

Our function doesn’t exist yet as it’s in a shared object that will get loaded later. This is fine — except it isn’t. My breakpoint never happens.

Huh! I guess (as it turns out, wrongly) that maybe I need to change my compilation options. And I also assume the segfaulting is because of something in the Perl extension code.

So maybe it’s that we compile with -02. My gcc is too old to support -Og, so let’s try -O0.

At this point, I’m just copying the line from make’s output and changing it. I just want to get some details in gdb damn it!

So I run the following:

perl Makefile.PL
make
gcc -c  -I/usr/local/lib64/perl5/auto/share/dist/Alien-libcmark_gfm/include -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O0 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic   -DVERSION=\"0.280301\" -DXS_VERSION=\"0.280301\" -fPIC "-I/usr/lib64/perl5/CORE"   CommonMarkGFM.c
make install

Now I can run perl t/03_render.t again, under gdb, and see if I can get more details.

1..10
ok 1 - use CommonMarkGFM;
ok 2 - markdown_to_html
ok 3 - 'parse_document' isa 'CommonMarkGFM::Node'
ok 4 - parse_document works
ok 5 - render_xml
ok 6 - render_man
ok 7 - render_latex
ok 8 - render_commonmark
ok 9 - render functions return encoded utf8
ok 10 - render functions expect decoded utf8

My attitude thus far is clear in the tweet that followed:

Now I proceed to have fun.

I spent the time trying to figure out what -O1 vs. -O0 did, and I wrote a script to repeatedly re-compile that one file
with different options. Along the way, I learned how to make gcc spit out what options it is compiling with (gcc -Q -v ...).
I had some false positives, and then I went to sleep.

After a period of sleep, figured out I wanted the list of flags as a difference between -O0 and -O1. I cleaned up my compile.pl script
and ran it.

The answer is: all of them are fine. -O0 and all the feature flags of -O1 result in no segfault either. Adding -O1 back brings back the segfault. After some more searching of the gcc docs, it is implied some optimizations are just directly tied to the O level.

My fun is now over, and I’ll do the more boring task of figuring out why my code is broken.

Staring at my from gdb’s output is this:

warning: Error disabling address space randomization: Operation not permitted

After a bit of searching, I find a fix for this to run the docker image
with --security-opt seccomp=unconfined.

And suddenly, breakpoints work.

and I can debug the root variable that is passed to cmark_render_html_with_mem… and nothing is wrong there.
Probably I need to re-compile libcmark-gfm with more debugging, I think. Suddenly, I realize that cmark_render_html_with_mem takes three arguments, and the Perl XS code is only passing it two.
How does this work? Well, it appears to cast a pointer to a function pointer, and call it. Calling a function pointer with fewer arguments than it is declared to with is undefined behavior, and I guess the rest of the behavior I observed was nasal demons.

(as an FYI, this argument difference is an API change between upstream libcmark and libcmark-gfm).

Finally, this third argument is a linked list of syntax extensions,
and it’s not clear yet how I will need to pass that back and forth between perl and C. This is also indicative that CommonMarkGFM will need to be a fork of CommonMark

BMO ❤️ Carton

Back when I started working on BMO
we couldn’t add new dependencies without having someone build an RPM. For no particularly good reason, this made it so in general we didn’t add new dependencies often.

However, about a year ago I started poking at carton and came up with a process to run carton in a docker container that mirrors production, and tar up the resulting local/ directory.

For the last 6 months or so we have been able to add dependencies whenever we want. We can also track changes to the
full dependency tree.

The code for this is on github as mozilla-bteam/carton-bundles and it is a little ugly, but packaging code is rarely elegant.

Sorry, I meant I changed it from 226s to 184ms

About twenty days ago it came to my attention via my colleague Ed Morley that BMO’s bzapi was very slow.
It turns out he had reported the same issue the prior year as well!

Performance problems are very enjoyable to work on, I find. Especially when they are reproducible.

I lost most of the day on this, but in the end I was able to take the slowest function from executing at a leisurely 226 seconds to a very fast 184 milliseconds

Perl + Bugzilla in Outreachy Round 13!

I am very proud to be mentoring in the 13th round of Outreachy.

We have a a list of ideas
and a growing list of bugs that would make for good “small contributions”.

I’m open to emails or irc discussions and I’ll try to answer
any and all questions.

email: dylan [at] mozilla [dot] com
or join irc.mozilla.org #bugzilla, I’m in IRC from
13:00 UTC until about 22:00 UTC.

Here’s are project blurb:

Perl is a highly capable, feature-rich programming language with over 28 years of development, making it one of the longest standing FOSS projects. The Perl Foundation is funding a position working on Bugzilla, a widely used, Perl-based issue tracker. In 18 years of development, Bugzilla has grown into a complex application that is used in many different workflows by organizations including Mozilla, the GNOME Project, Red Hat, and freedesktop.org. Some of this complexity is particularly evident in the search functionality, both in implementation and in user interface. We have several proposals to simplify and improve searching, which will positively impact Bugzilla sites around the world.

Fixed some memory leaks in bugzilla.mozilla.org

So last week I fixed Bug 1282606 which has resulted in a bit of a performance improvement for bugzilla.mozilla.org:

Restarts per Hour

Apache2::SizeLimit
is configured to kill processes once they use more than 700M and this happened about every 7 minutes.

About two weeks ago, while working on some performance issues relating to BMO’s new show_bug ui, I discovered that the problem could get worse: running out of memory every 60 seconds. Should everyone switch to the new UI (which is intended to put less load on the server) a lot more load would be on the server. That’s pretty bad, since we want everyone using the new UI as soon as possible. 🙂

This memory leak isn’t new, and I had filed an investigatory bug about it last year. Memory leaks in perl are caused by having cyclic references, and the solution is to not have cycles, use weak references, or to break the cycle when you’re done with whatever data structure it is part of.

I understand the problem, and I know how to fix it… but maybe I don’t know where the problem is?

Thankfully there is a tool for this on CPAN: Devel::MAT.

Using Devel::MAT, it is possible to dump the address space of a perl program and explore it in great detail in a GUI.

I didn’t set out to remove all the memory leaks this time, just the ones that were the biggest or grew the fastest. This meant the TrackingFlags extension Flag objects, the Bug object, and the Comment object.

The changes are on github for the curious,
and the resulting charts below speak for themselves.

Average Request Time

Requests Before Restart

Age of Process Before Restart