I joined my first Internet community in the waning years of the 1900s. (OK, it was 1998 or so, but it’s fun to say it that way.) It was a programming-centric web forum, geared around offering advice and assistance with whatever random code thing people asked about. Forums were, in that era, kind of the dominant precursor to things like Stack Overflow - but much less formalized. One step past a mailing list, really.

I dropped in and out of that community for a few years before moving on more or less forever in 2002 or so. By that point I had joined a couple of other notable forums and had spread out into other online communities.

Even then, it was clear to me that something about certain forums made them stick so much better than others. I couldn’t articulate it then, and I can barely identify the components now, but the general principle is that some communities resonate with me more than others. It has a lot to do with the internal culture and attitudes and general temperature of how people interact with each other within the community - and outward facing interactions, as well.

That first forum I joined turned into a cesspit. Or maybe it always had been a cesspit, and it just took me four years to catch on. Either way, I decided to move on from it because I just wasn’t having a good time there anymore.

Culture is hugely important to me - personally, professionally, philosophically - and I find that online communities have some very extreme cultures in a lot of cases. Over the years I’ve wandered into (and, frequently, right back out of) a lot of places on the web, and one of the most reliable predictors of me leaving a community is a deteriorated culture.

Much of this is coming to a head right now because of my own personal circumstances, but a lot of it is a sign of the times as well. Current events being what they are, a lot of communities that used to be friendly, engaging, and enjoyable to participate in…​ well, they’re turning into polarized battlegrounds littered with meme-ridden "arguments" where people basically just sling rhetoric (or outright insults) at each other until someone gives up.

Don’t get me wrong, people have been terrible to each other since before fire was invented. This is not news. What’s news is that the terrible behavior has finally overtaken a handful of places I used to like to hang out in, and now I don’t like to hang out there anymore.

I want to specifically point out a couple of examples.

I joined Twitter originally because I thought it seemed like a cool way to stay in touch with a bunch of game development friends and acquaintances. And for a while, it was good for that; I could follow the people I liked to hear from, and anyone who wanted to listen to my inanity could follow me in return.

Slowly, the platform evolved, and eventually turned into a monster. Curated feeds actually prompted me to quit Twitter once before. I discovered much later that third-party apps could display the uncurated feed, and joined back up; but things were already palpably different.

The community feel of Twitter is gone. Rampant retweet-sprees and quote chains have destroyed the sensation of being able to control what viewpoints you have to be subjected to. This is a touchy issue for me, because I genuinely believe three things about viewpoints:

I liked Twitter once upon a time because it gave me the ability to hear things I otherwise would not have heard, but I could still lock away the really egregious garbage and ignore it.

For a myriad of reasons, that ability has been lost.

I dread opening Twitter now because I don’t want to read another thread about how gun control is against the Constitution. I don’t want to read another thread about how women don’t deserve to play video games, or make video games for that matter. I don’t want to have to hear echoes of my friends and colleagues fighting an endless war against people who quite honestly have some reprehensible opinions and habits.

I believe in listening to the opposing side. I believe in finding common ground. I believe in the spirit of equitable compromise.

And yet it hurts intensely to watch yet another conversation scroll by where someone I respect and care about in my industry has to defend her right to be a part of that industry at all; where good people who just want the endless stream of shootings to stop already are drowned out by expressions of hate and disdain simply because they had the audacity to say guns are probably something we should take a little more seriously.

I have no intention of walking away from the issues or even the debate, such as it is, in any area really. But I do feel, quite keenly, the lack of a social space where I can just engage with people in a healthy and enjoyable way. I don’t need it to be "safe." I don’t need it to align exclusively with my own biases and preconceptions. I just want a place to talk about stuff - any stuff - where everyone is a rational adult and acting otherwise gets you a nice shiny boot back to whatever hellhole you came from.

For the things I see in my feed these days, the ratio of "interesting idea that I want to think about" to "oh no why" is perilously low. Twitter no longer delivers pleasant, thoughtful discourse at a sufficient volume. (Maybe it never did, but that’s beside the point.)

So I’m phasing out my Twitter account. I will probably hang on to it so people can reach me until I have alternative arrangements made, but I’m actively cutting it out of my life.

This one is a lot more emotionally charged for me. I’ve been lurking around and participating in the GDNet community since 2002. I’ve held a moderator position there for a long time. I used to sink a lot of effort into helping people with projects and questions there. I attended my first couple of GDCs under the banner of GDNet, doing "media coverage" - i.e. I’d write an article about every session I went to, and in exchange, I got a free pass to the conference.

The "Bag of Holding" started as my journal on GDNet. I wrote a lot of stuff there - ranging from the flippant and silly, to the highly technical, to the intensely personal.

Much like Twitter, I’ve simply stopped having fun on GDNet. But the reasons are much less clear to me. It isn’t an outright toxic or hostile place to hang out. There’s still plenty of good people there and plenty of interesting discussion.

I think that in the case of GDNet, it isn’t so much the site that has changed, as it is myself that’s changed. To be clear, there has been a lot of change over the past decade and a half on that site. Some of it was harder to swallow than others. But ultimately, I don’t think my desire to leave GDNet is the "fault" of that community.

Rather, I think I’ve simply had my time, and now it’s time to move on. These things happen.

I’ve been using Reddit for nearly 12 years - most of its existence. I predate subreddits and a lot of other features of the community. When I first started hanging out there, it was a pretty niche sort of place. It wasn’t exclusively about programmers or software developers, but the vast majority of the population had some connection to software.

To me, my desire to leave Reddit is a hybrid of my desire to leave Twitter and my desire to leave GDNet. It is partially a failing of the community. Back in the early days, if you were a jerk, you got called out on it and the problematic behavior was strongly discouraged in a number of ways. Since most people on the site had some common ground (i.e. programming or one of the other early interest groups) it was easy to defend each other from hostility. It was easy to want to be nice.

Now it’s become a very different place. And that’s fine; I don’t pretend to be entitled to it staying a particular way indefinitely. The community has moved on and evolved.

I just don’t want to be part of it anymore.

There are still a lot of fantastic people on Reddit, and a lot of great discussion and exploration happens there. But it’s also exquisitely easy for moderation to slip. Once a sub gets a certain critical mass, if the mod staff is not ready and willing to tackle the volume, bad behavior becomes rampant.

What made me step back and think about all this was not a feeling of being attacked or wronged. What made me step back was realizing that I was also starting to act like a jerk - and very frequently. Not only was I doing things I found distasteful, but there was minimal response or reaction from the community itself to discourage more of the same.

I’m not leaving Reddit because it’s devoid of merit. I’m leaving Reddit because I can’t enjoy it anymore, and my own behavior is part of that problem.

I’m not going to cite the "new" places I’ve taken to hanging out in - partially because I like the idea of keeping them pristine, but mostly because I’m not ready to commit to them just yet. Until I actually settle somewhere, I may be a little less visible and a little harder to get ahold of.

On a general level, though, there are a couple of things I think are interesting to consider.

Culture, as I cited earlier, is hugely important to me. The fundamental structure and even the UX of a community can have massive impacts on the culture. I’m interested in communities where the culture is generally positive, supportive, introspective, self-aware, and self-healing. I think some technological approaches to "Internet community" are more likely to produce these qualities. But again, I’m really early in this process, and I want to gather more data before making any strong statements.

Another interesting factor is what I’ve been mentally referring to as "taxonomy." Some communities are organized around sets of ideas - like subreddits, for example. If you care about Subject X, you go into the section of the community that talks about Subject X, and you hopefully encounter people who are fun to talk to. This is a totally legitimate way to build a community. In fact, I know that some people actually prefer this approach.

The other angle is to organize around individuals. Twitter and Facebook are much more oriented around people than subjects, for example. It wasn’t always the case, but for me now, at this point in my life, this organizational taxonomy is much more appealing.

I would like to join communities where I know interesting people hang out. I want to know what those people have to say. I want to know who they listen to and find interesting. I want to use the network effect to expand my own sphere of awareness and perhaps even influence.

I think a community that focuses on people and culture would really hit the spot.

Arguing for reuse is easy: if you only have to write and debug the code once, but can benefit from it multiple times, it’s clearly better than writing very similar code five or six times…​ right?

Yes and no. Premature generalization is a very real thing. Sometimes we can’t even see reuse potential until we’ve written similar systems repeatedly, and then it becomes clear that they could be unified. On the flip side, sometimes we design reusable components that are so generic they don’t actually do what we needed them to do in the first place.

This is a central theme of the story of Design Patterns as a cultural phenomenon. Patterns were originally a descriptive thing. You find a common thread in five or six different systems, and you give it a name.

Accumulate enough named things, though, and people start wanting to put the cart before the horse. Patterns became prescriptive - if you want to build a Foo, you use the Bar pattern, duh!

So clearly there is a balancing act here. Something is wrong with the idea that all code should be reusable, but something is equally wrong with copy/pasting functions and never unifying them.

But another, more insidious factor is at play here. Most of the time we don’t actually reuse code, even if it was designed to be reusable. And identifying reasons for this lapse is going to be central to making software development scalable into the future. If we keep rewriting the same few thousand systems we’re never going to do anything fun.

Here’s a real world use case. I want to design a system for handling callbacks in a video game engine. But I’ve already got several such systems, built for me by previous development efforts in the company. Most of them are basically the exact same thing with minor tweaks:

Easy. Except Guild Wars 2 alone has around half a dozen different mechanisms for accomplishing this basic arrangement. Some are client-side, some are server-side, some relay messages between client and server, but ultimately they all do the exact same job.

This is a classic example of looking at existing code and deciding it might be good to refactor it into a simpler form. Except GW2 is a multi-million line-of-code behemoth, and I sure as hell don’t want to wade through that much code to replace a fundamental mechanism.

So the question becomes, if we’re going to make a better version, who’s gonna use it?

For now the question is academic, but it’s worth thinking about. We’re certainly not going to stop making games any time soon, so eventually we should have a standardized callback library that everyone agrees on. So far so good.

But what if I want to open-source the callback system, and let other people use it? If it’s good enough to serve all of ArenaNet’s myriad uses, surely it’d be handy elsewhere! Of course, nobody wants a callback system that’s tied to implementation details of Guild Wars 2, so we need to make the code genuinely reusable.

There are plenty of reasons not to use an open-source callback library, especially if you have particular needs that aren’t represented by the library’s design. But the single biggest killer of code reuse is dependencies.

Some dependencies are obvious. Foo derives from base class Bar, therefore there is a dependency between Foo and Bar, for just one example. But others are more devilish.

Say I published my callback library. Somewhere in there, the library has to maintain a container of "things that care about Event X." How do we implement the container?

Code reuse is the name of the game here. The obvious answer (outside of game dev) is to use the C++ Standard Library, such as a std::vector or std::map (or both).

In games, though, the standard library is often forbidden. I won’t get into the argument here, but let’s just say that sometimes you don’t get to choose what libraries you rely on.

So I have a couple of options. I can release my library with std dependencies, which immediately means it’s useless to half my audience. They have to rewrite a bunch of junk to make my code interoperate with their code and suddenly we’re not reusing anything anymore.

The other option is to roll my own container, such as a trivial linked list. But that’s even worse, because everyone has a container library, and adding yet another lousy linked list implementation to the world isn’t reuse either.

The notion of policy-based architecture is hardly new, but it is sadly underused in most practical applications. I won’t get into the whole exploration of the idea here, since that’d take a lot of space, and I mostly just want to give readers a taste of what it can do.

Here’s the basic idea. Let’s start with a simple container dependency.

This clearly makes our nifty class dependent on std::vector, which is not great for people who don’t have std::vector in their acceptable tools list.

Let’s make this a bit better, shall we?

Slightly better, but now clients have to spell a really weird name all the time (which admittedly can be solved to great extent with a typedef and C++11 using declarations).

This also breaks when we actually write code:

This works provided that the container we give it has a method called push_back. What if the method in my library is called Add instead? Now we have a compiler error, and I have to rewrite the nifty class to conform to my container’s API instead of the C++ Standard Library API. So much for reuse.

You know what they say, you can solve any problem by adding enough layers of indirection! So let’s do that real quick.

Let’s pull this apart and see how it works.

First, we introduce a template "policy" which lets us decouple our nifty class from all the things it relies on, such as container classes. Any "reusable" code should be decoupled from its dependencies. (This by no means the only way to do so, even in C++, but it’s a nice trick to have in your kit.)

The hairy parts of this are really just the syntax for it all. Effectively, our nifty class just says "hey I want to use some container, and an adapter API that I know how to talk to. If you can give me an adapter to your container I’ll happily use it!"

Here we use templates to avoid a lot of virtual dispatch overhead. Theoretically I could make a base class like "Container" and inherit from it and blah blah vomit I hate myself for just thinking this. Let’s not explore that notion any further.

What’s cool is that I can keep the library code 100% identical between projects that do use the C++ Standard Library, and projects which don’t. So I could publish my callback system exactly once, and nobody would have to edit the code to use it.

There is a cost here, and it’s worth thinking about: any time someone reuses my code, they have to write a suitable policy. In practice, this means you write a policy about once for every time you change your entire code base to use a different container API. In other words, pffffft.

For things which aren’t as stable as containers, the policy cost may become more significant. This is why you want to reuse in only carefully considered ways, preferably (as mentioned earlier) when you have several use cases that can benefit from that shared abstraction.

One last idea to consider is how the performance of this technique measures up. In debug builds, it can be a little ugly, but optimized builds strip away literally any substantial overhead of the templates.

So runtime performance is fine, but what about build times themselves?

Admittedly this does require a lot of templates going around. But the hope is that you’re reusing simple and composable components, not huge swaths of logic. So it’s easy to go wrong here if you don’t carefully consider what to apply this trick to. Used judiciously, however, it’s actually a bit better of a deal than defining a lot of shared abstract interfaces to decouple your APIs.

I’ll go into the specific considerations of the actual callback system later. For now, I hope the peek at policy-based decoupling has been useful.

Remember: three examples or you don’t have a valid generalization!

Attaining this functionality was not easy, but it was definitely worth the investment. It all started almost exactly a year ago, when I decided that being unable to debug the self-hosting process for 64-bit Epoch was unacceptable.

Initially debug information was generated via piping some bogus line numbers into LLVM and then routing the generated block of CodeView symbols into MSPDB140.dll to generate a somewhat-working PDB file on disk. This implementation took about two weeks.

That wasn’t enough, though; it introduced a heavy dependency on Visual Studio (something I’ve been keen to avoid, despite strongly encouraging use of VS with Epoch) and also had limitations via the API of MSPDB140.dll that were…​ inscrutable, to say the least.

So I set out in search of a complete understanding of the PDB file format and how to generate my own debug information for it. The intervening year wasn’t all dedicated to PDB work; a fair amount of time went into Visual Studio integration and other tidbits of self-hosting effort. (Not to mention there were a few major spans of downtime. This gets exhausting after a while!)

The Epoch repo commit log shows the gory details of how everything came together, but the high-level is pretty simple; using a suite of tools, I reverse engineered large sections of the PDB format and developed an Epoch implementation of code to write them out.

Noteworthy projects:

Target debuggers have been VS2015 and WinDbg, both of which work now with source-level breakpoints and stepping. x64dbg also sort-of works, although it doesn’t like to display source for some reason; everything else seems to be fine, so I don’t know the tool well enough to say if it’s my bug or theirs.

As alluded to above, my next major project is to get variable data inspection working. This is a dark corner of the PDB format that seems poorly understood in the community, so it should be exciting to try and forge ahead here.

At a minimum, I’ll need to start generating types data in the PDBs, so that debuggers know how to interpret various memory addresses correctly. There’s probably some other voodoo too, but that’ll have to wait until I discover what’s in store.

Sometimes when feeding data into a black box like MsDia140.dll it can be hard to know what code paths are taken and why. For example, let’s look at the function GSI1::readHash (see here to follow along in the source).

This function does some stuff I still don’t fully understand, so let’s walk through the process of gaining more understanding.

First we need a partially malformed PDB. This is easy to do since PDB files are sensitive to tiny changes, often in non-obvious ways. In particular, I’m going to work on the Publics stream. This is a fragment of a Multi-Stream File (aka MSF) which contains, among other things, publicly visible debug symbols for some program.

At the beginning of the stream, there is a structure which llvm-pdbdump is sadly cryptic about. Thankfully, llvm-pdbdump contains some sanity checks which seem to align well with the checks made by Microsoft’s code, so it’s at least easy to use the tool to verify what we’re spitting out.

readHash is responsible for decoding part of this data structure, which appears to be some kind of hash table for accelerating symbol lookups. Inside the code for readHash (see link above) there is a call to a pesky function called fixHashIn. By attaching WinDbg to a running copy of Dia2Dump.exe and setting liberal numbers of breakpoints, I traced a failure in my PDB generation code to this single function. fixHashIn is vomiting because I’m feeding it data it doesn’t like.

The first thing to note is that fixHashIn begins with a decrement instruction to decrease the value of one of its parameters. This parameter is supposedly the number of buckets in the hash table, or so I extrapolate from the source.

In my case, the parameter has a value of zero! Clearly I don’t want my hash table to have zero buckets, so it becomes apparent why fixHashIn is choking. What I don’t immediately understand is why it thinks zero is the number of buckets…​ I had thought that I was passing a value in (8 bytes per entry * 16 entries) that would work. Clearly I was wrong, but where was the zero coming from?

A little more background is in order. In an MSF file (MSF being a superset of PDB files), data is divided into streams, each of which is built up of one or more blocks. A block can be different sizes, but I’m using 1KB (1024 bytes) for convenience. Data not used is filled with junk bytes.

Crucially, I pad my blocks with zeroes. If somehow the PDB interpreter is reading one of my padding bytes, it might be incorrectly assuming I want to feed it a zero-size hash table…​ obviously a problem. So what to do?

And now the meat of everything!

Instead of padding my file with zeroes, I use carefully crafted poison values. For my purposes I’m working with 32-bit data, so a poison value is usually 4 bytes long. A good example is 0xfeedface which is a funny but valid hex number that happens to be the right size.

The important thing is that we can’t just pad every 32-bit slot with 0xfeedface. Instead, we want to make permutations of the poison value - one unique permutation per slot. Every possible 4-byte sequence of my PDB’s "padding" is now a unique string of digits.

Here’s the magic part: when I run this in the debugger, I can walk into the fixHashIn function, and look at its parameters.

My first run of this process is surprising - despite poisoning a bunch of data around where I thought this zero was coming from, the value is still zero when we reach the fixHashIn function! This indicates one of two things.

To rule out the possibility that I’m not poisoning enough, I expand the poison to the entire file instead of just one block’s worth of padding bytes. The debugger still stubbornly shows the parameter as zero, meaning that the zero is being computed from some other data being fed in, not read directly from the file on disk.

This line of the Microsoft PDB source is illuminating…​ but this line even more so. At line 65 is a comment stating that fixHashIn is called from two places…​ one of them is the loader for the Publics stream, but one is for a totally unrelated stream called Globals!

It turns out I’ve been hitting breakpoints all evening in fixHashIn, but the call stack is wrong. The calls I’ve been seeing are from a totally different stream of data.

This post may not have a cheerful ending, but I hope the value of poisoning data is clear: I may have taken days to realize my mistake without having 100% proof that the evil zero was not coming from my Publics stream.

In any event, I use the poison technique a lot, and this is just one sampling of my adventures with the PDB format. Maybe I’ll have a better story of success tomorrow!

LLVM divides code into sections called basic blocks. A single basic block represents a linear sequence of instructions, i.e. every instruction in it is executed exactly once. The way to accomplish branching flow control is to end a basic block with a branch instruction, likely a conditional branch of some kind. Branches target other basic blocks to allow different code paths to execute based on the branch.

The dreaded error "Terminator found in the middle of a basic block!" means that the constraints have been violated. Someone tried to issue a branch instruction in the middle of a block, which ruins the idea that every instruction in the block executes exactly once.

In concrete terms, this error signals a bug in the code generation process. It means that somewhere along the line, the Epoch compiler lost track of a basic block being terminated, and continued shoving instructions into it after a branch.

Thankfully, LLVM barfs a "label" when it emits this error, and that label is sufficient to locate the offending basic block:

Sure enough, there are two branches being attempted here. The larger context is uninteresting (it’s a nested if-statement inside a binary tree insertion routine) but the specific failure appears many times, meaning that it’s probably a small number of actual code-generation bugs to solve.

As with any good software, robustness in a compiler happens only when enough bugs have been fixed while simultaneously ensuring that no new ones are introduced. The best tool I know of for doing this is automated testing. Now that a compiler bug has been identified, the objective is to replicate it in as tiny a program as possible.

This "test case" provides two things: a way to reproduce the bug on-demand so a fix can be tested, and a way to detect if the bug ever reappears. The Epoch compiler test suite is still small, but invaluable for addressing this sort of problem. I will add this particular code to the test suite and hopefully have a fix in short order.