Schrödinger’s Memory Manager

Besides the practice of writing software it’s self, there are few activities more engaging for a software developer, than to watch other developers argue about software development in social media. I’ll admit that while I usually abstain from taking part in such arguments, I am often tempted, and sometimes actually do stoke the fire a little. You see, in a field so varied and complex as writing software, it’s actually quite rare to find common ground, and all too easy to be blind to alternative perspectives.

It’s all a matter of perspective.

Lets take the ‘goto’ instruction as an example. It was determined in the late 60’s that the ‘goto’ instruction is harmful, and quite elegantly demonstrated to be so, by a very highly respected computer scientist named Edgar Dijkstra. This view was backed by none other than the also highly regarded computer scientist ‘Nicklaus Wirth’, who edited and published Edgars famous letter on the subject. Yet, it is still one of the most hotly debated subjects more than five decades later! But why?

Well, the answer of course is in the nuance.
You see, like most of the software development world, I too consider the ‘goto’ instruction to be harmful, but under certain specific conditions. Edgar clearly demonstrated that ‘goto’ is an instruction which refuses to be diagrammed, at least without ever increasing context information, proportional to its use. The more the ‘goto’ instruction is used, the more it complicates digramatic documentation, and by extension, the more confusing the code that uses it. If you’re not following what I’m saying here, that’s fine, this can take effort to wrap ones mind around, but for the sake of this post it does not matter. What matters is that you accept the argument that ‘goto’ leads to confusing code, even if you agree only tentatively at this time. I’m working on another, far more detailed post on this subject to help explain the position.

I mentioned above that I agree that ‘goto’ is harmful, but I also mentioned conditions, and nuance. One such condition is that the programming language in which you use the ‘goto’ instruction is a structured language, and that it has sufficient ‘abort’ clauses as to make the instruction unnecessary.

I recently watched a YouTube video on the channel “Low Level Learning” which gives excuse for using the ‘goto’ instruction in the C programming language, in order to essentially ‘exit early with clean-up’. Well, this actually I do see as a valid excuse for using the ‘goto’ instruction, at least in so much that my understanding of the ‘c’ programming language, is that it does not have sufficient ‘abort’ clauses. What the presenter (who’s name I don’t know beyond “Ed”) explains is a technique for using ‘goto’ as a substitute for what another language might provide a ‘defer’ or ‘finally’ clause. Where-in I’m using ‘defer’ and ‘finally’ to express the same idea, not making an argument for exception handling, which is another hotly contested subject.

So in a programming language which does not offer any kind of abort clause for cleaning up resources which were allocated before an early exit, then yes, ‘goto’ can be used for that purpose. If following the ‘dogma’ of not using ‘goto’ of which I’ve been accused, then we have a new rule “Thou shalt not use goto, unless you have to.”

Another nuance of the ‘goto’ instruction is performance. You see, even in a structured language without sufficient clauses to remove the need for ‘goto’ entirely, there is often another way around the problem. For example, in the above example of exiting early, you could call a function which takes a ‘context’ containing references to everything that might have been allocated, and have that function dispose of the now unnecessary resources. This technique will work, and would satisfy those that truly are dogmatic, as an alternative to using the ‘goto’ instruction, however, it comes at a performance cost. A ‘goto’ is essentially just a simple, single branch instruction when compiled to machine code, and therefore it is cheap and fast. A function call on the other hand, has a lot of overhead to prepare the stack frame for the call, and to restore the stack frame on return. In most line of business applications, this likely isn’t too much overhead, but in a performance critical loop for example, the cost is high. ‘Goto’ ought to be excused in performance critical scenarios also. And here-in lies the problem of perspective.

An application developer is one that works as part of a team. They would like to be good citizens and follow “best practices”, which includes avoiding the confusing ‘goto’ instruction. They would like to write neat structured code, and will often excuse performance penalties, in order to make use of features which promote readable code.

I have a belief that what makes readable code can be defined quite objectively, however, for the most part it is considered a subjective subject. There are those for example that argue that interface-based abstraction, or object abstraction, makes code less readable, and those that will argue that it makes the code more readable. The truth of this (having been on both sides of the fence as I now see it), is that this too is a case of perspective. For example, when everyone knows and understands the abstract model being used in an application, it promotes a common understanding, which is great for teams working together. It provides a common frame of reference. When someone joins the team and does not understand that abstract model however, the extra effort to understand the abstraction has the opposite effect for them.

We haven’t even mentioned the “what you’re used to” argument. A procedural programmer using concrete data-types, having become used to this style of programming, will be bewildered in a highly abstracted OOP code base. The same is true the other way around. I spent many years programming OOP code only to find myself frequently maintaining procedural code bases. While I originally learned programming before OOP was a common feature of compilers, the years of OOP training on my mind made it quite an exercise to return to a procedural mind-set. Some years of training myself to work within an ARC memory model, have had me accidentally forgetting to free resources.

Computer Science today is actually very quickly dismissed by a great many software developers. The rigorous practices of diagramming code are long since forgotten by the industry, in which every developer, regardless of training, background, or education, may each have an opinion, and they believe their opinions to be incontestable fact. With the advent of social media, we now have public figures or ‘influencers’ held up as the respected authorities on what is good or bad practice in programming. Unfortunately, many of them will talk confidently from their own perspective, giving very little consideration to ideas outside their own field of knowledge.

I’ll give some examples (as mentions), but I’d like you to understand and take these examples with an appropriate pinch of salt. I am not accusing any of these people of being wrong, but rather, I’m trying to understand the perspective that they are coming from, and in what places their arguments don’t seem to fit.

Low Level Learning

The example that I’ve already given regarding the ‘goto’ instruction. It’s important to understand that the presenter is a cyber security expert, frequently working on low-level code. That means, often in assembler, machine code, or C. To someone who’s mind is trained on low level code, a ‘goto’ instruction is merely a branch, there’s no harm in using it right? It’s necessary in assembler, and so using it to excuse an otherwise missing feature in the ‘C’ language is fine, no harm done, and it produces elegant and optimized results.

What I don’t believe he pays credit to, is an entire industry which has developed high-level structured programming practices over decades, and have worked against the use of ‘goto’ for their coding-standards reasons. The argument against ‘goto’ is not dogma, but science, computer science, and has had it’s place in application development. I believe I explained this sufficiently above as to not labor on it here.

Johnathan Blow

I agree with Johnathan on just about everything he says regarding software development, but I do also have to take many of his positions with context. For example, Johnathan points out that developers are far too quick to reach for abstraction, which leads to VMT’s, which use indirection, which has significant performance costs. I agree, however, Johnathan is a video game developer that works at a low level also, and with critical performance considerations in mind. Johnathan is also not wrong that all software can benefit from being written in an optimal way, however, his position does not take into consideration those benefits of abstraction that I’ve already touched on.

You see, something that is far more important to a line-of-business application software company than performance, is being able to make rapid changes to a code base. Abstraction driven decoupling of parts of a large, and long-lived code base, enable the company to make business decisions that impose large changes on that code base, with reduced effort.

Solid, well optimized and therefore tightly coupled source code is very resistant to such change. In talks, Johnathan has mentioned essentially writing a new video game engine for each game, bespoke to its needs. Sure, he can re-use pieces of his existing code, but for the most part, the product of his work is a video game, which will not have to change. Barring a rework to re-release a game title for newer hardware, or to repair a truly serious bug, there is a definite time at which his source code is finished, complete, and will not need to be touched again. This is not the case for most line of business software, which is frequently changing to keep up with the changing needs of its customer industry.

The Primagen

Prime disagrees with Uncle Bob on raising/throwing exceptions and catching them, vs using a ternary operator on a return value. Having throw-able exceptions in a language is another highly debated subject, and I can see both sides.

I’m aware of a member of the Delphi programming community (a community that uses exception throwing and handling extensively), that will tell you to simply never bother catching exceptions unless you really need to handle a given error state. This lets the exception bubble up the stack until it is either handled, or presented legitimately to the end user as an error message. The Prime will tell you, ( if I have any justification for speaking on his behalf ) that exceptions are evil, because they bubble up the stack and therefore break the locality of where you handle your error states… and that you *should* handle all error states locally. So from where does this disparity between the two takes come?

Well, the Delphi developer that I mentioned, works on binary-native desktop applications in line of business, in which an error message making it to the end-user may be ugly, but it is none-the-less, a valid outcome. His hope is that the end user will report that error to their support representative to have it investigated. This might seem like a horrifying outcome, and there are ways to mitigate it which I’ll mention in a moment, however in truth when you consider the user base for the application, is not too bad at all…

Most line of business applications have a few hundred, or maybe a few thousand users, but generally not counts in the millions. These software products generally have very large feature sets, of which, many of the customers will use only a few features. Line of business software companies generally have a polished, active support network, to support their customers directly. Therefore, a customer, or even a few customers, calling their support representative, will get dedicated attention, and escalation of the problem that they are reporting, right back up to the development team. In some cases, with no more than a single call.

A support call may even be mitigated in many cases, by the addition of self-reporting system. Such a system will have an outer level trap for exceptions, to capture the call-stack and report home to the development team directly. Using this call stack information, the development team are able to track down the cause of the uncaught exception, and release an update, without their customer ever having to know it occurred. In short, the ideal case for these software products is that they never display a message to the end user, but it’s not the end of the world if they do. With large feature sets to maintain, which again may be tightly coupled, using exception handling instead of localized error handling allows the software team to move more quickly with the changing needs of their customers.

On the counter perspective, The Primagen is also not wrong. Rather, he’s coming from a different type of application entirely. Until recently he worked for Netflix. This is a company that I’m sure you’re aware of, with millions of end users world wide. I don’t even have a grasp on how many end users they have, I can only say “millions” as a best guess. Can you imagine the harm it might do to a company like Netflix if say, 2 Million customers all turned on their televisions to be presented with a message that reads “Access Violation 0x000000C” or “SigSegV”, and then their television software hangs? This would be an absolute disaster of an outcome, and worse, despite the size of the organization, I doubt that Netflix could have a support representative walk each of those customers through the process of updating their software to get the patched build!

While I’m certain that the back-end operations of Netflix are feature rich and complex, with a need to scale for the scale of their customer base, they are also in an entirely different class of software. Code must not only be tested, but be rigorously verified, run through staging environments and load tests, before ever making it into production. It’s also possible to quickly respond to such a back-end issue, repairing it perhaps even before it is noticed by the end user. On the front-end, the applications are relatively light in terms of features. They must work, and must work without disrupting the customer, but the number of features is small.

The point?

Debates between software developers online are often fierce, and of course being the arguments of humans, rarely reflect on alternative perspectives. I love having prominent figures to entertain me on YouTube, but they too are human, and subject to their own perspectives. Software development is a field in which these debates are just a fact of life. I often wonder how many other professions get so heated?

I can’t imagine two painters getting into serious debate online about the merits of one low-tac masking tape over another, with a third chiming in to say “learn to cut-in your edges by hand, that’s what real painters do!” Perhaps there’s a fierce social media echo system of auto mechanics getting bent out of shape over the use of a particular brand of wrench?

The problem for me, is cutting through the noise. I decided to write this post because I’m in the process of developing my own language compiler. It’s through investigating other recently developed languages, such as Zig, Odin, Jai and to some extent Rust, that I came across many of the names I mentioned above.

My compiler project is intended for writing video games, just as is Jai which is being developed by Johnathan Blow and company. While the intended purpose has some overlap, the projects are quite different. For starters, Johanthan has seen success in releasing some Indy game titles and therefore has experience which I lack in the space. His compiler is intended to replace C++ for this purpose, and he has been able to employ a handful of people to work on the compiler.

My experience has been in application development for the past two and a half decades, and while writing video games was always an interest for me as a child, I’ve not done so commercially. I also have no intention of replacing an existing and established programming language, but rather, initially this compiler was intended to be a kind of ‘scripting’ engine for a game engine. I know, scripting engines are generally a bad idea, but in my defense, it’s an insufficient word to explain my intention. The language is to compile to native binary, not to be interpreted, is the language that the ‘game engine’ will be written in, and is really intended for actually making games, not just scripting them – still, it’s intentions are more limited than to attempt to replace the established leading compiler in the space. Finally, my project is a solo-developer project, it’s all down to me, and that’s just the way I like it.

So, Schrödinger’s Memory Manager?

With the hotly contested “memory safety” argument floating around right now, the memory management model for my compiler is on my mind. I shared some of my thoughts with a friend, including the problem of what to do for memory management, if anything. Ultimately, I concluded that there is no correct answer to my problem. My friends response was to describe the problem as “Schrödinger’s Memory Manager”, which I felt was an interesting enough response to turn it into a blog post title.

The point of this post was really to share my views on being “pragmatic” in the developer community, rather than to explain my conundrum to you. That said, now that we’re here, I think it will nicely round out this post to do so.

The problem is to decide if the compiler should manage memory at all. In the “good old days” of software development, compilers did not implement any form of memory management. We the developers had to be careful to ensure that if we allocated resources, we also disposed of them at an appropriate time. If we were going to use abstractions at all, we’d need to be certain to never reference instances of the abstraction after those instances had been disposed. It was something of a “cowboy” era from an application development position, but, from the perspective of a game developer it was not unruly at all, and still is not.

Consider for a moment a game engine generating say, 60 frames per second of on-screen action. This means that the code must generate its output for each frame in 16.6 milliseconds! Allocating and disposing of memory on a heap is an expensive operation, which could seriously impact performance. The last thing one would want in a performance critical rendering loop is for the code to be off allocating or freeing memory resources. The ideal would be that any memory required in this loop is allocated in advance, and disposed of only when the critical loop has ended.
In practice, in order for the game simulation to be dynamic, there must be some way to allocate resources during the running of the game. Perhaps not every frame, but at least, while the critical loop is running.

The ideal memory management mechanism for such a scenario then, is no memory management mechanism at all. The developer using the compiler should be in full control of when to allocate memory, and when to free it. My problem is that I don’t feel I can trust the developer quite that much!

Okay, it may well be that I am the only person that ends up using my compiler. I’m under no illusions that it’ll take the world by storm, and that’s not really my intention anyway. None-the-less, no one writes a compiler hoping that it’ll never be used by others. I of course hope that some day, others will see the value in this compiler, and use it. Therefore, while I’m currently writing it for my own entertainment and my own purposes, I am forced to consider what other developers might do to use and/or abuse it. As with any software product, I’m forced to consider how my end-user might break things in ways that may not be obvious. Memory management is a hot area for this consideration.

Now, garbage collection is out. Period. It’s just not the right fit for a game development tool due to its non-deterministic nature, and it’s very costly on performance to run the mark-and-sweep operations required to make it work. It may be great for line of business applications, or even in high scale environments if sufficiently distributed, but it’s not right for high performance media applications.

Automatic Reference Counting, while I actually really enjoy using it in application development, has its complications too. It is deterministic, in that you can carefully craft code to specify where memory is disposed, but from my experience of talking with others, it’s not easily understood. The cyclic reference problem (which prevents allocations from being freed) is actually not a huge problem with a full understanding of the memory model, but in watching developer debates online (and having been involved in a few), I don’t believe that ARC is well understood, and have decided that while it seems intuitive to me, it must not be to many.

So I came up with another option. It’s in some ways similar to ARC, but not quite. It’s that of an owner reference. I’m lead to understand that other languages have such things as “owner pointers”, which may well be the same feature that I’ve come up with. I’ve not yet researched these in detail, but my idea is simply this. When you assign an instance of some allocation to a reference variable, that reference variable becomes its owner. When the owning reference variable goes out of scope, regardless of you having taken a copy of it, the allocation will be freed.

In order to keep an allocation in memory under this model, you must use a custom assignment operator to pass ownership from one reference to another. Never will two references to the allocation both own it. This mechanism could still cause the ‘cyclic reference’ problem, if you pass ownership of references incorrectly, but it significantly reduces the risk because essentially all references other than the ‘owner’ reference, become “unsafe” (not quite weak, but essentially non-binding).

As an additional bonus, it goes some way to reducing the complexity of memory management in a multi-threaded environment, because no thread that doesn’t “own” a resource is able to free it. This encourages the “passing” of resources between threads, rather than the “sharing” of resources, which will only ever lead to pain. So while it does not solve every problem in multi-threading, it at least lends it’s self to a more cautious crafting approach.

Now, the conundrum.
When explaining the memory model of my compiler to others, should I opt for

A) There is no memory manager, be responsible and manage memory yourself.
or
B) If you want to keep the thing allocated at the end of scope, pass its ownership to an outer scope.

In both cases, there is more responsibility placed on the developer than a typical memory management system would aim to allow. To me, and in the space of developing games, this is a good thing. Writing games is a difficult practice, which demands that the developer take responsibility for everything the code does. Unfortunately, debates over such things as memory managers being what they are, they’re also both models that many will not like.

Many will favor the ownership reference, because ultimately it’s “safe” in that its references are testable, and that it will not allow the developer to forget to free anything. Others will favor no memory management at all, because they like the precise control which requires no additional understanding. There is no right answer, and so ultimately, I’m going to have to make up my own mind about what I want the compiler to do.

Hence: Is there a memory manager or not? It’s Schrödinger’s memory manager.

Conclusion

This post was not a request for help with my memory manager problem, nor was it a tutorial post of any kind, it was indeed intended as an opinion piece on being pragmatic when engaging with the software development community. Try to remember, to quote the great Obi Wan “Only a Sith deals in absolutes.” It’s not that there is no “correct” answer, nor is it true that every one of these debates is entirely subjective. Rather, we learn not by arguing our point at all costs, but by hearing other perspectives.

Thanks for reading!