I’m still quite busy these days, but I did find time in the last month to start a new project:
As you can imagine, this was pretty similar to the Sonic CD port that I did last year: not only do they run on the same engine (RSDK – Mania just uses a newer version), but they also have the same dependencies (SDL2 and libtheora).
The process of porting the game wasn’t quite as simple though: while Sonic CD just required some endianness and thread-safety fixes, Sonic Mania had major performance issues. In particular, Competition Mode, the Special Stages, the pinball table, and the Stardust Speedway Zone Act 2 boss fight would all lag severely. The reason for this is that Sonic Mania is software-rendered, and its fancier effects are simply too much for the Wii U’s CPU to handle.
But that wasn’t the only problem that I had to deal with: this port also exposed issues in the Wii U’s SDL2 port as well!
But I’m getting ahead of myself… let’s start at the beginning:
Timeline
Getting it to Boot
In the middle of August, the Sonic Mania decompilation was released, and news of it reached a small Discord server that I was in. One thing led to another, and I accidentally gave people the impression that I would port the decompilation to the Wii U, as I had done previously with Sonic CD. Eventually, word spread to GBAtemp, and I soon had people messaging me, asking about the port that I was allegedly working on. Realising that I’d kind of done this to myself, I figured that I might as well give it a shot. I had an afternoon all to myself – just me, my Wii U, a copy of the decompilation, devkitPPC, and no internet – so I got to work. Within a few hours, I had the game running, albeit very rough and laggy.
Building the game was simple enough: the build system is not complex, so I was able to quickly write my own CMake script from scratch (the decompilation only comes with a Makefile, but CMake scripts are much easier to target the Wii U with). The Wii U already had a port of SDL2, and I was able to compile my own Wii U port of libtheora, so the dependencies were quickly satisfied as well. But once I had produced a Wii U executable, I found that running it would immediately result in an endless black screen – the game was crashing upon boot.
Luckily, the game has built-in error reporting, and I found that the game was failing to locate a file. The game keeps all of its files in a single archive blob, with each one being represented by an MD5 hash of its original filename. To load a file, the game hashes the filename at runtime and then uses that to locate the file in the archive. But here’s the thing: the MD5 hasher that the game uses is garbage: it’s literally just some random code grabbed off a random website.
This hasher’s code commits every sin:
- Tiny, incomprehensible variable names (because why would anyone want to understand what a variable does or is for?).
- No documentation (no, really, who cares about code being readable and maintainable?).
- Uses dynamic memory for no reason (MD5 has absolutely no need for dynamic memory: everything can be done on the stack with a small buffer).
- Doesn’t handle malloc failure (who doesn’t like null pointer dereferences?).
- Is dependent on implementation-defined integer type sizes (because, as we all know, ‘int’ and ‘long’ are the same size on x86-64 Linux as they were on DOS, am I right?).
- Uses signed ints for representing the size of arrays (ah yes, an array of -0x1000 chars, please).
- Only works on little-endian CPUs (because screw basic portability, especially in code that’s, you know, shared online and intended to be used by people in a variety of projects).
While the other points could be considered nit-picks, that last one prevents the game from running on the Wii U, which has a big-endian CPU. How is it not compatible with big-endian CPUs? It casts an array of chars to an int, instead of just bit-shifting and OR-ing them together like sane code would.
Because of this bug, the calculated hashes would be incorrect, causing the game to fail to find vital files and crash.
Anyway, with that fixed, the game can now run, right? Nope: it still crashes on boot. It’s not just the MD5 hasher that only works on little-endian CPUs: the entire game engine is the same way! Oddly, there is some degree of explicit big-endian support in the engine’s code, but I guess due to code-rot, other parts of the engine lack it, such as the UTF-16 string reader. After patching this all up, the game could finally boot.
RGB565
The game was now running, but there was one pretty big problem: it was very, very laggy in anything but the main levels. Special Stages, animated cutscenes, pinball, Competition Mode, they would all lag to half or even a third of the game’s usual speed.
As I explained earlier, this is largely due to the game being software rendered, and the Wii U’s CPU being a dinosaur that’s still compatible with GameCube code, but it’s also caused in part by SDL2: you see, the game outputs an RGB565 framebuffer, but the Wii U port of SDL2 doesn’t support RGB565, causing SDL2 to convert it to a variant of RGBA8888 and use that instead. This adds a massive amount of overhead to the game’s rendering. Despite this, the Wii U actually does support RGB565. Indeed, there is even code in the SDL2 port that tries to use it, but it’s dummied out with a comment explaining that it doesn’t work. The cause of this is that there’s a catch to RGB565 support: each 16-bit RGB565 pixel is little-endian, not big-endian. This isn’t too hard to work around, though: just byte-swap the bitmap as it’s passed to the GPU.
With this done, the game started to perform a lot better, but it would still drop to half speed a lot. This struck me as odd, since it would drop to half speed even if it should only be lagging slightly. Looking into SDL2’s code, I found the cause: the port uses regular V-sync instead of adaptive V-sync. For those not in the know, adaptive V-sync is a variant of V-sync that doesn’t wait until the next frame if the game is lagging. This means that if the game only barely missed the frame that it was meant to wait for, then it wouldn’t wait for the next one – it would just continue. The result is that the game only lags as much as it has to, instead of dropping to a fraction of its framerate. Making the V-sync adaptive was a simple change to make to SDL2.
With these two fixes combined, the game’s performance was massively improved: many instances of lag were eliminated, and the cases where the game still lagged at least ran at a framerate that was much closer to 60FPS.
YUV420
There was still one part that consistently lagged, however: the animated cutscenes. Unlike the rest of the game’s RGB565, the animations render in YUV420. This is a weird format that encodes colour as a single luminance channel and two chrominance channels. Unlike RGB, these channels are not interleaved, and they’re not even the same resolution: the chrominance channels are half the vertical and horizontal resolution of the luminance channel. It’s really weird.
Like with RGB565 before, SDL2 didn’t support YUV420 on the Wii U, so it was converting the YUV420 image to something else every frame. Unlike RGB565, the Wii U doesn’t support YUV420 natively… however, it can be emulated with a fragment shader.
This is a trick that I learnt from SDL2’s OpenGL renderer: if you create three textures, you can assign one to each of the YUV channels. Then, these textures can be sampled in a fragment shader, and dotted with some lookup tables to produce an RGB pixel. This works out to be much faster than SDL2’s CPU-based conversion, allowing the cutscenes to finally run at full speed.
Controller Support
Oddly, the game could only read input from the Wii U Gamepad: the Wii U Pro Controller and Wii Remote didn’t work at all. Looking into it some more, I found that this was because SDL2 only supported those controllers in its legacy ‘joystick‘ API, and not its newer ‘game controller‘ API. The difference between the two is that the ‘game controller’ API binds each button to an XInput-style layout, avoiding the need for the user to configure their own button mappings like in an old PC game. To achieve this, SDL2 contains a database of known controllers and which buttons map to their equivalent XInput buttons. All I had to do was add the Wii U Pro Controller, Wii Classic Controller, and Wii Remote (with Nunchuk) to this database, and now these controllers can be used. This is great for Competition Mode!
Random Crashing
Ah yes, my worst nightmare: after releasing the first few versions of my port, I began experiencing random crashes. There was no pattern to when they’d occur, so I couldn’t reproduce them at will: all I could do was playtest the game for hours and hope that it would eventually crash. At first, I had no leads to go on, but a handy YouTube comment tipped me off that unlocking achievements would consistently result in a crash. From this, I found that the memory allocator was faulty.
Yes, Sonic Mania has its own memory allocator. It’s neat: it performs automatic garbage collection and defragmentation. However, the decompilation broke it in two key areas: the allocator would increment the total number of allocations even if the allocation failed, and the allocation duplicator would fail to increment the allocation total at all, resulting in memory being destroyed by the garbage collector while still in use. Once these were addressed, the memory allocator was back to functioning as intended.
Awesome! No more random crashes, right? Nope: there were more!
At this point, I was stumped: my only lead was a dead end, leaving me with no other ways of diagnosing what the cause of these crashes was. That was, at least, until a Wii U homebrew developer called Gary informed me that the Wii U’s operating system keeps crash logs in its /storage_slc/sys/logs directory. This was an enormous help: suddenly, I had a full stack trace and register dump of the last 100 crashes that my Wii U had experienced! From this, I found that the crashes were occurring in the audio mixer. A null pointer dereference; strange, as there was only one pointer in the audio mixer (the sample pointer), and it should never be null.
This pointer is obtained from a struct which is only read if a flag in it is set, but, whenever this flag is set, the pointer is set to a non-null value. This could only mean one thing… a thread-safety issue.
Eventually, I found the culprit: the StopSFX function. Most audio functions make sure to lock the audio thread before running, but this one neglects to. I’m not sure if this is the case in the real Sonic Mania, or if this is just a mistake in the decompilation, but either way this failure was resulting in a race condition where the audio thread was running after the flag had been set but before the pointer had been set, resulting in a null pointer being read and processed by the audio mixer, crashing the system. With this fixed, the random crashing was finally a thing of the past.
Aroma
During the development of this port, the Aroma Wii U Homebrew Environment was released. In particular, Aroma sports one very interesting feature: running homebrew from the Wii U Menu, bypassing the Homebrew Launcher. This meant that homebrew could use the Home button to open the Home Menu, whereas with the Homebrew Launcher it would just abruptly exit the homebrew without warning. Additionally, it meant that homebrew could include a cool splash screen that would be shown during start-up. It’s also just cool to have homebrew displayed alongside official Wii U software and games. Naturally, I updated my port to take advantage of this, making it look and feel extra official.
File IO
The Sonic Mania decompilation supports mods, but I was surprised to find that these mods would load extremely slowly on the Wii U. For instance, a level-replacement mod would take upwards of an entire minute to load. What’s going on? Is the Wii U’s SD card IO really that slow?
At first, I thought that this was an issue with devkitPPC’s implementation of fopen and related functions: perhaps it’s doing some weird slow low-level interaction with the Wii U’s SD card slot? I figured that I could try using the Wii U OS’s native file reading functions instead, only to find out that those are exactly what devkitPPC uses.
Eventually, I began trawling through devkitPPC’s code to see if it was using the native file reading functions incorrectly. On the contrary, however, I found that the code was very well written: there were already clever tricks that exploited the properties of the CPU’s cache to optimise file reading. Higher up the software stack, in the C standard library (newlib), there was even complete IO buffering, greatly reducing the number of IO accesses. What could possibly be going wrong here?
Welp, after examining the code for hours, I realised that I was looking at the wrong branch. After switching to the devkitPPC branch, I found the cause: newlib’s IO buffering was being manually disabled by a hack in the code, causing every byte of read data to result in an IO poke. No wonder it took a minute to load a couple of megabytes. This change had recently been reverted, but an update had yet to be released that contained this change. Still, the C standard library has a function for re-enabling IO buffering (setvbuf), so I made the game use that, and now mods load just as fast as the rest of the game!
Wrapping Up
After just over a month of work, the port is now quite complete, running fast and stable while providing near feature-parity with official releases. This project has not only led to improvements in the Sonic Mania decompilation, which should benefit all ports and not just this one, but also improvements to the Wii U port of SDL2, which should benefit many Wii U homebrew programs and games which depend on it, granting them support for more controllers, improved support for certain texture formats, and even some bugfixes.
If you’re interested in trying my Sonic Mania port, you can find it here.
If any fancy updates are made to the port, I’ll be sure to talk about them here. I’m still trying to think of a way to port the screen filter shaders, and it would be nice to add full hardware-accelerated rendering to eliminate the remaining lag in the Special Stages. We’ll see. In the meantime, though, I’ve taken an interest in my Mega Drive emulator once again…
Clownacy's Corner 
One thought on “Porting Sonic Mania to the Wii U”