I was digging through some old Sonic hacks of mine when I stumbled across these ancient relics from 2014 and 2016.
Remember that Project Sand/Sonic Aftermath hack that I have a fewoldvideos about? Here’s the one level of it that was worked on before the project died: Sand Zone from Cave Story.
There’s not much that I can tell you about it: this one test level was made before I came up with any ideas for gimmicks and the sort. That purple mess in the middle of the stage is supposed to be spikes, but its graphics are overwritten with the level tiles.
I believe that the music is a straight port from Cave Story: the Organya music was converted to XM with the org2xm tool, then that was converted to SMPS with xm2smps, and then that was converted from binary to ASM with SMPS2ASM, allowing it to be installed in my hack’s custom sound driver. This hack was either rocking my Sonic 2 Clone Driver v2, or Flamewing’s Flamedriver. After all this time, I can’t remember which.
This footage is of an earlier build of my hack than the one seen in the other videos. The reason for this is that the hack was remade from scratch after this build, and this level was never reintroduced afterwards.
Here’s a cutscene that was one of the last things that was worked on before the project died.
This was meant to be the opening cutscene to Knuckles’ story: the hack was meant to be a ‘what if’ scenario where Sonic 2 ends differently, leading to a different series of events in what would be Sonic 3. This cutscene depicts the Death Egg landing in Angel Island’s volcano directly after the events of Sonic 2, instead of the lake like it normally does. Following this, Knuckles would have gone to investigate it. At the time, I didn’t have Knuckles ported into the game, so I used Sonic as a placeholder.
Between this cutscene, Sand Zone, and the custom title screen, this is all there ever was to Project Sand/Sonic Aftermath.
Sorry for the drought of blog posts lately: I’ve been busy with work and a lot of other IRL stuff. Still, if there’s one thing I can give an update on, it’s that two of my projects have finally seen a release: clownmdemu and clownassembler have been released on Sonic Retro, Sonic Stuff Research Group, and Mega Drive Developers Collective.
clownassembler was released less than an hour ago, so there’s not much that I can comment on feedback-wise. However, clownmdemu was released at the end of June, giving it plenty of time to receive feedback.
Surprisingly, people were quite enthusiastic and welcoming of the emulator despite its unfinished state. If anything, people seemed more bothered by the hardcoded key bindings than its inability to boot certain games. I can only hope that clownassembler gets an equally warm reception, considering that it is similarly unfinished.
With the addition of FM support, my Mega Drive emulator came much closer to being able to provide a complete experience for certain games such as Sonic 1. Unfortunately, there was still one major missing feature: drums, voice clips, and sometime even all audio entirely, were inaudible. What gives? Sonic 1 plays most of its audio, but 2 and 3 don’t play any at all?
The reason that this is the case lies in the architecture of these games: Sonic 1 uses a sound engine that runs on the 68k CPU, while 2 and 3 use one that runs on the Z80 CPU. Up to this point, my emulator has not emulated the Z80 CPU, which is why no sound plays in those two games. Additionally, Sonic 1 uses the Z80 CPU for its drums as well as the famous Sega chant, which is why those are missing as well.
However, this problem is no more: new to clownmdemu is Z80 CPU emulation. It doesn’t implement 100% of the Z80’s feature set, but it’s enough to get at least the Sonic games to output all of their audio. Heck, it’s even enough to get Sonic ROM hacks to play their audio, including the ones that use fancy custom Z80 code:
This ROM hack in particular, Sonic 2 Recreation, uses Z80 code that was written from scratch by ValleyBell and is able to apply a variety of effects to its PCM samples.
Writing another CPU emulator was nice because it gave me a chance to apply what I’ve learnt since writing the 68k CPU emulator. In particular, instead of making each step of execution a giant switch statement, there’s only one switch statement: one for each instruction. This does result in some duplicate code, but the hope is that this is outweighed by avoiding the overhead of going through 6 or so switch statements.
I’ve also used this opportunity to write the machine code decoder in such a way that its job can be replaced with a lookup table. You see, the task of breaking a byte of machine code down into a struct which describes the operation to be performed has been given to a function: this function takes a byte and returns a struct. Because of this modularised design, I can execute a loop on start-up which executes this function for every possible byte, from 0 to 0xFF, and then caches the resulting structs in a big lookup table. Then, during emulation, the machine code to be executed can be used as an index into this table to retrieve its corresponding struct. By doing this, the expensive step of manually decoding the machine code is skipped entirely. According to my tests, this doubles the performance of the Z80 emulator. Though, for RAM-limited platforms, I’ve left a compile-time option to revert back to using the function instead of a lookup table. One day, I would really like to refactor the 68k CPU emulator to bring over these improvements.
The Zilog Z80 itself is a weird thing: it’s an 8-bit CPU with a 4-bit Arithmetic Logic Unit which mimics the instruction set of another CPU (the Intel 8080) while adding extra instructions of its own. The extra instructions were bolted-on in a very ugly way that clashes with the ‘neatness’ of the base 8080 instruction set. It also contains some outright hacks that aren’t intuitive to emulate at all: for instance, there’s a certain mode that you can make the Z80 enter where the output of an instruction can be written to both a register and memory at the same time. It essentially hijacks the operand getters and setters.
Anyway, with this implemented, Sonic 1 and 2 are now finally fully emulated: there isn’t a single feature in those games that isn’t being emulated (to my knowledge anyway). Sonic 3 still has some strange bugs, and it makes use of unimplemented features such as SRAM and the Window Plane, so that may be what I work on next.
There are still a number of things that I want to add before I give this emulator a proper release: a libretro core, controller rebinding, SRAM, the Window Plane, 68k instruction durations, 68k exceptions, YM2612 LFO, YM2612 SSG-EG, YM2612 Timers… though, maybe I should just release this anyway: it’s not like this emulator will ever play every title perfectly or anything.
Oh, right. I should also list the resources that I used when making this Z80 emulator. First there’s this blog post which gave a general overview of the Z80, and explained why and how the Z80’s instruction set is the way it is. Then there’s this follow-up which explains the Z80’s timings. There’s also this useful table which visualises the various opcodes. Finally, there’s this webpage which explains how to effectively decode Z80 opcodes. These resources were invaluable to me, and hopefully they can be to others too.
It finally happened! With university over, I decided to tackle what is perhaps my greatest challenge yet in writing this emulator: emulating the YM2612.
The YM2612 is the Mega Drive’s primary audio chip. Apparently, it is a cost-effective, stripped-down version of the YM2608: while the YM2608 featured FM, SSG, Rhythm, and ADPCM modules, the YM2612 is just a standalone FM module with basic DAC output slapped on it.
For the longest time, the only documentation that was available to emulator developers was the “Genesis Software Manual“, which was a document that Sega made available to developers that described the console’s hardware. Unfortunately, this document went into very little detail about how the sound hardware worked. Still, it was apparently good enough for a number of emulators to be made back in the 1990s and 2000s.
Later, in 2008, Nemesis obtained a copy of the official YM2608 manual. Unfortunately, this document was in Japanese, but he was able to produce a mostly-coherent machine-translation. This document answered many questions that emulator developers had about the YM2612, but it still failed to go into detail when it came to certain subjects that were essential for emulator developers to understand.
When I began implementing my own FM emulation, I decided that I would try to stick to this manual early on, and only seek out additional information once I get stuck. With just this YM2608 document and my own knowledge of the YM2612, I was able to produce an extremely basic emulator that produced one sine wave per channel. These sine waves could have their volume and frequency adjusted. While this wasn’t nearly enough to produce authentic audio, it did at least make it possible to hear music and sounds.
Unfortunately, there wasn’t much more that I could do than this: I knew from the manual that there were three core components of the YM2612 that were essential to how it produced audio: the Operators, the Phase Generator, and the Envelope Generator. However, while the manual described what they are and what input they take, it did not describe what output they create from that input. To give an example, a ‘detune’ value can be supplied that offsets the frequency, but neither the Genesis Software Manual nor the YM2608 manual describe how much it offsets the frequency.
At this point, I decided to find some more information on the YM2612. I remembered that the SMPS devkit which was found a few years ago had a YM2612 manual, but unfortunately it too was in Japanese and with seemingly no translation available. It turns out that the manual for the YM2612’s CMOS equivalent (the YM3438) was also found, but yet again it was in Japanese. This wasn’t a huge loss though, as those didn’t appear to contain anything particularly useful that wasn’t already in the YM2608 manual.
What was useful, however, was a thread on SpritesMind that I’ve had bookmarked for years. It’s 58 pages of discussion and discoveries regarding YM2612 emulation, including some incredible documentation that was produced by Nemesis. In particular, he created three massive write-ups of exactly how the YM2612’s Operators, Phase Generator, and Envelope Generator all work. This information is utterly invaluable, as it provides most of the ‘missing pieces’ to the YM2612’s functionality that the manuals lack. Somehow, with nothing more than a Mega Drive, an oscilloscope, and an extensive array of tests, he was able to even figure out details as nuanced as the exact values of the chip’s internal sine wave lookup table.
It took me about four hours straight to read through that whole thread, but it was worth it! I’ll probably have to read through it again to catch any details that I missed the first time around. If you’re interested in making your own Mega Drive emulator, or are just curious about the YM2612, then I cannot recommend that thread enough.
Unfortunately, Nemesis wasn’t able to complete his documentation of the YM2612, meaning that there was still missing information on three key components: the accumulator, Operator feedback and modulation, and the Low Frequency Oscillator.
For the time being, I passed on implementing the Low Frequency Oscillator because Sonic 1 (the game that I was using to test audio) doesn’t use it. The accumulator, I had already produced my own implementation of through guesswork. This left the Operator feedback and modulation.
So what is Operator feedback and modulation? To understand that, you have to understand how the YM2612 produces sound. So, what the heck: here’s an overview of how the YM2612 works:
The YM2612 has six channels. Each channel is composed of four sine waves, dubbed ‘Operators’. Each Operator has its own Phase Generator and Envelope Generator. The Phase Generator advances the sine wave, and the Envelope Generator produces an ADSR envelope. The Phase Generator manages the frequency of the sine wave, and thus is responsible for producing notes, while the Envelope Generator is responsible for shaping the sine wave into a more complex waveform, and thus creating basic ‘instruments’ or ‘voices’.
However, the Operators allow for even more advanced ‘instruments’ to be made through Operator modulation: rather than output its waveform to the speakers, an Operator can instead feed directly into another Operator, modulating its sine wave in a process known as ‘phase modulation‘.
‘Operator feedback’ is the process of the first operator of a channel feeding into itself, which is a feature unique to the first operator.
In the spirit of emulation development, I decided I’d find out for myself how to implement Operator modulation: with the help of the cycle-accurate Nuked OPN2 YM3438 emulator, I compared the output of my emulator to what a real Mega Drive would sound like, and tweaked my own Operator modulation implementation until it sounded correct.
Despite this, the audio still sounded far from accurate. I was able to track one source of distortion down to an incorrect implementation of the Phase Generator’s multiplier, and while that did fix the channels sounding like whistles, it still left the audio sounding like this:
Clearly the envelope generator was running too fast… and yet its code matched Nemesis’s notes exactly. I spent hours debugging this, creating custom FM instruments to test specific parts of the emulator against Nuked OPN2, but nothing made sense: the envelope generator was absolutely working as intended. I then figured that perhaps I had given the emulator’s frontend the wrong sample rate, and that it was somehow playing the audio back twice as fast.
The truth was a lot dumber: I’d accidentally given the emulated YM2612 a 7x overclock.
The YM2612’s clock is derived from the Motorola 68000’s clock, which is derived from the master clock. The 68000’s clock is the master clock divided by 7, and the YM2612 clock is the 68000’s clock divided by 6. My emulator had the YM2612 clocked at the master clock divided by 6.
Once that was corrected, (and I properly implemented ADSR envelope rate-scaling which I somehow glossed over when reading the YM2608 manual 100 times), I finally had this:
At last: it worked! And it came so suddenly too: one minute everything’s a whistle, and the next it sounds like a real Mega Drive!
And that takes us to where we are now. There are still many things left for me to implement in my YM2612 emulator, such as per-operator frequencies, Timer A and Timer B, the Low-Frequency Oscillator, SSG-EG (a second Envelope Generator), and possibly even the debug registers.
You might be asking yourself how I’m going to replicate all of those when there’s so much missing documentation. Well, the truth is that the YM2612 and YM3438 have actually been completely documented for years now. I just figured that it would be too easy to use that documentation. What is that documentation? Nuked OPN2’s source code.
You see, Nuked OPN2 isn’t just a cycle-accurate emulator: it’s a cycle-accurate emulator that’s based directly on a die-shot of a de-capped YM3438. Essentially, Nuked OPN2 is a conversion of the YM3438’s circuitry to C. With this, there are no mysteries about how the YM3438 works: everything is documented in such a way that you can verify it just by running it. What other form of documentation doesn’t just say ‘dude, trust me’, but ‘here: I’ll prove it’?
While I do plan on using Nemesis’s documentation to implement SSG-EG, Nuked OPN2 can be used to implement any details that aren’t explained elsewhere.
Surprisingly, some parts of my YM2612 emulator happen to function exactly how a real YM2612 does, in contrary to how documentation suggests they should function. For example, the note octave is encoded as a number between 0 and 7, which expresses the following behaviour:
Divide note frequency by 2.
Leave note frequency as-is.
Multiply frequency by 2.
Multiple frequency by 4.
Multiple frequency by 8.
Multiple frequency by 16.
Multiple frequency by 32.
Multiple frequency by 64.
Both the manuals and Nemesis’s notes suggest that this should be implemented as a left-shift by the number of the octave minus 1, with special-case logic for octave 0 that does a right-shift by 1 instead. In my emulator, however, I just left-shift by the number of the octave, and perform a single right-shift afterwards, creating the same result with more-efficient code. According to Nuked OPN2, this is exactly what a real YM3438 does as well.
As much as I’d like to continue rambling about this stuff, there’s only so much to write about, and only so many hours in the day. As usual, you can find clownmdemu’s source code in its Git repo. Hopefully I’ll have more progress on my emulator to talk about soon. In the meantime, I’ll leave you with a video of a fun bug:
Today, I looked into adding support for the Mega Drive’s interlaced video output to my emulator. It didn’t go how I planned, and I eventually realised that it was worthless to pursue. Still, I think this makes for a fun story.
First, I should probably go over the basics of how old CRT TVs would display. Basically, the screen is split up into 480 lines, but they are not all drawn at once. Rather, the even lines are done first, and the odd lines are done on the next frame, or vice versa. You could think of it as the TV rendering 240 lines at 60FPS or 480 lines at 30FPS.
With that in mind, we can begin to understand the Mega Drive’s interlacing. It has three modes:
Mode 0, which is the non-interlaced mode. This mode is plain 240p. It uses a trick to prevent the odd lines from ever being drawn, meaning that two sets of 240 even lines are drawn instead.
Mode 2. This mode is very interesting: it is like mode 1, except the odd lines will not display the same graphics as the even lines. Basically, the Mega Drive’s vertical resolution will double, being 320×480 or 256×480. However, because only even or odd lines are displayed in a single frame, this means that the image will be downsampled back down to 320×240/256×240 when displayed.
Regardless of the interlace mode, it is always 240 lines that are output in a frame. I wanted to implement this in my emulator, to perfectly replicate the rendering of interlace mode 2, which until now has been rendering at the native internal resolution of 320×480.
However… always rendering a 320×240 image wouldn’t be correct. After all, old TVs have 480 lines, not 240. Since mode 0 disables odd lines, there should theoretically be empty black spaces between each line, creating a ‘scanline’ effect.
To recreate this, I set about making my emulator always render a 320×480 image, and having the emulated console simply skip lines. This is accurate to how a real Mega Drive displays on a real CRT TV. However, having leftover lines from a previous frame mixed in with lines from the current frame produced an ugly ‘comb’ effect:
Because of this, and knowing that the lines of a CRT fade when they’re skipped, I decided to simply make the skipped scanlines black.
In Mode 1 and Mode 2, the lines which are black and the lines which are actually drawn alternate every frame, causing the screen to ‘jitter’. It looked pretty cool and authentic to how I remember the two modes looking on a real Mega Drive. I ended up doing some extended playtesting with this, to soak in the nice scanline effect. When I was done, I closed the emulator and- oh dear:
My monitor had severe image persistence:
It turns out that this is a terrible idea: apparently, much like the waveform of a sound, the electricity going through an LCD pixel must go up and down, positive and negative. It alternates every frame, just like how a wave alternates every sample. By rapidly flickering a pixel between colour and no colour every frame, however, the pixel never gets to go negative, always remaining positive, or vice versa. The result is that the pixel becomes a capacitor, storing charge and refusing to release it, causing it to display colour when it shouldn’t.
Thankfully, this appears to be temporary, as the built-up charge will dissipate naturally. Still, this was bloody terrifying: I thought I’d just ruined the display of my fancy new laptop.
Clearly, this interlacing emulation had to go. The last thing I need is a wave of heated bug reports from furious users who think their monitors have been destroyed.
Thinking about it, I realised that there’s no point to emulating the Mega Drive’s interlacing in the first place besides authenticity: in every use-case, interlacing is either an annoying side-effect or an irrelevant technical detail.
Sonic 2 uses mode 2 for its split screen multiplayer, taking advantage of the doubled vertical resolution. The interlacing does nothing but halve the game’s vertical resolution and introduce an ugly jitter effect.
Mode 2 could be used to display a static 320×480 image, in which case the interlacing wouldn’t be visible at all.
As a result of this, interlacing emulation is completely pointless: my emulator has been doing ‘the right thing’ the entire time by just not emulating it. Better yet, my emulator goes above and beyond by rendering mode 2 at its native 320×480 resolution instead of halving its resolution back down to 320×240, meaning that it renders mode 2 better than a real Mega Drive.
I suppose this is where the story ends. Interlacing has been a strange thing to research, but I’m glad that I now understand more about how the Mega Drive delivers images to the display. I just wish that I didn’t nearly suffer heart failure in the process.
With my honours project complete, I decided to put my newfound free time into a project that I’ve been meaning to get around to for almost five years: disassembling Knuckles in Sonic 2.
In case you don’t know, Knuckles in Sonic 2 (which I’m just going to call ‘KiS2’ from now on) is a version of Sonic 2 that lets you play as Knuckles instead of Sonic and Tails. Sonic hackers like to port Knuckles from this version back into regular Sonic 2, but, in the process, they effectively undo the huge number of changes that KiS2 made to Sonic 2’s codebase. This ranges from simple alterations for Knuckles, to bugfixes that have gone undiscovered to this very day.
You might be asking yourself why I want to disassemble this game, since a disassembly for it already exists. Well, the reason is that the existing disassembly is completely separate from the Sonic 2 disassembly that also already exists. Not only does this mean that it is horrifically outdated in comparison to the Sonic 2 disassembly, but this also makes it extremely difficult to compare the two games and find differences between them.
As of writing, this task is finally done, and I have a modified Sonic 2 disassembly that produces a perfect copy of KiS2. With this disassembly more or less complete, I figured I should explain everything I’ve learnt about KiS2 here:
Obviously, Knuckles has replaced Sonic. This is actually surprisingly tacked-on: Knuckles is just a lightly-modified Sonic with all of the gliding and wall-climbing behaviour wrapped in a single function call. I suppose this isn’t surprising, but I was under the impression that the whole Knuckles object was copied from an in-development version of Sonic & Knuckles. I think I got that idea from the Sonic 3 Unlocked blog, but I could just be misremembering.
Notably, Knuckles’ graphics are loaded from the Sonic & Knuckles cartridge: the tiles are recoloured at runtime to suit Sonic 2’s palette. The sprite mappings and dynamic tile loading data are also loaded from the Sonic & Knuckles cartridge. Sonic hackers may find this surprising, since Sonic & Knuckles uses a different sprite mapping format to Sonic 2. This leads me into my next point…
All of the game’s mappings were converted to Sonic & Knuckles’ format. This strikes me as very odd, as this means that the mappings now have to be included in the KiS2 ROM, instead of being loaded from the Sonic 2 cartridge, wasting space. Maybe it was considered too much effort to go through the whole game and split the mappings? This conversion was universal: even unused mappings were converted. Heck, even unreferenced parts of mappings were converted. This suggests that the mappings were created using assembly macros, and the macro itself was modified to convert the mappings to Sonic & Knuckles’ format.
The difference between Sonic 2’s and Sonic & Knuckles’ sprite mapping format is that Sonic 2’s has extra data for the game’s two player mode, which uses a fancy rendering mode of the Mega Drive’s VDP. This leads me onto yet another point…
Two Player Mode
Two player mode was removed, but not entirely. It appears that the developer(s?) were struggling to fit the game to the size they wanted, so they began removing code related to two player mode, and once they reached their desired size, they stopped. In the end, they scraped by with only 680 bytes to spare.
There are plenty of leftovers from two player mode in the game: the variable used to detect two player mode (dubbed ‘Two_player_mode’ in the disassembly) still exists, and is referenced frequently in the game’s code. For example, the level title card object still makes heavy use of the flag.
Being a Sonic hacker, I’ve removed two player mode from Sonic 2 before, and I’ve done it much more thoroughly than in KiS2. With that in mind, I know how complex removing two player mode is, so it doesn’t surprise me that the developers didn’t go all the way with it.
This won’t be a surprise to most people reading this, but KiS2, despite being a version of Sonic 2, doesn’t have many of Sonic 2’s assets in it. Instead, it copies them from the attached Sonic 2 cartridge. You see, KiS2 isn’t a standalone game: it’s actually a bonus mode in Sonic & Knuckles. Sonic & Knuckles’ cartridge has a cartridge slot on top of it, allowing you to plug other cartridges into it, with KiS2 being the result of plugging in Sonic 2’s cartridge.
The way Sonic 2’s assets were removed from KiS2 is pretty basic: at the end of Sonic 2 is a massive block of assets (including the game’s music, sounds, drum samples, enemy graphics, player graphics, player sprite mappings, level graphics, level layout, level object placements, and more), and it is simply removed in KiS2. Notably, assets that aren’t part of this giant block were not removed, such as the title screen’s ‘1 PLAYER’ and ‘2 PLAYER VS’ text.
As mentioned earlier, some assets are loaded from the Sonic & Knuckles cartridge, such as Knuckles’ assets. However, those aren’t the only things that are loaded from that cartridge: KiS2 features modified level object placements, which reward the player for exploring with Knuckles’ wall-climbing. Strangely, the data for this is in the Sonic & Knuckles portion of the cartridge instead of KiS2. It’s possible that this was done to free-up space in KiS2, with Sonic & Knuckles having room to spare.
KiS2 contains a surprising number of bugfixes:
Perhaps most notably, KiS2 removed the air speed cap, which appears to be a leftover from Sonic 1. This is significant because it has always been unclear whether the air speed cap was deliberately retained in Sonic 2 as a feature, or leftover as a bug. The air speed cap is responsible for at least two areas in Sonic 2 not working as intended: the red spring that leads to the ‘monkey island’ in Emerald Hill Zone Act 2, and the launcher that flings you over a large gap in the floor in Wing Fortress Zone. In both cases, the speed cap causes the player to undershoot their target if they press left or right on the D-pad while moving through the air. The removal of this speed cap in KiS2 suggests that it was indeed an unintentional leftover all along.
One of the most well-known bugfixes in KiS2 is the correction of a bug that causes the bottom two lines of the screen to appear incorrectly in Emerald Hill Zone. I wonder how this bug was discovered, since televisions were especially prone to overscan hiding the edges of the screen back then.
One type of bugfix that KiS2 contains is taking the player’s character out of their ‘roll-jumping’ state, where their controls are basically locked. Being left in this state at a bad time can result in the game soft-locking, as the player is unable to move their character. Times when KiS2 makes the character exit their roll-jumping state is when they enter a wind-tunnel and when hovering over a propeller in Wing Fortress Zone.
Sonic 2 suffers from a particularly glaring bug, where entering the cheat to gain 15 Continues causes the game to play Oil Ocean Zone’s music forever. The cause is a nonsensical sound ID being submitted to the sound driver. This is corrected in KiS2. This bug was also fixed in the version of Sonic 2 included in Sonic Mega Collection.
The title card appears to have had a bugfix applied to it which prevents odd behaviour if the graphic of the name of the zone goes too far to the left of the screen, causing its X coordinate to drop below 0. This bugfix works by replacing some unsigned conditional branches with signed conditional branches, and only drawing the sprite if it is within 48 pixels of the screen’s left side.
The bumpers in Casino Night Zone have their own layout data. This data needs to be terminated with special byte patterns that prevent the bumper manager from reading beyond them and parsing surrounding code as data. One of these termination patterns is missing from the very start of Act 1’s layout data. In a stroke of good luck, the code before the data happens to resemble the terminating byte pattern, preventing the bumper manager from processing invalid data. In KiS2, however, this is no longer the case. A proper data terminator was added at the start of the data, fixing this problem. Fun fact: this bug appears to have not been fixed in the earliest prototype of KiS2, causing the game to crash if you go to the top left corner of the level.
There are also some modifications to the game’s collision code, which may be an attempt to fix bugs in it. Unfortunately, I haven’t figured out the point of these modifications yet, so I can’t say for sure what bugs, if any, they’re trying to fix. One bug that it appears to be trying to fix is the bug in Sonic 2 where collision with an object from below doesn’t properly push the player out, sometimes resulting in them phasing straight through the object. This fix does not work correctly, however, and cancels-out the player’s inertia when it shouldn’t. You can read more about it here.
One rather funny bug is that if you’re moving at a high speed towards a wall, and then start moving in the other direction at last second, Sonic will impact the wall and then start moving away from it while playing his pushing animation. KiS2 appears to fix this bug as well, preventing Knuckles from entering his pushing animation if he is not facing towards the object that he pushed against.
In Sonic’s movement code, a register that holds his speed is unintentionally partially overwritten before being used later on to decide whether Sonic is moving fast enough to skid or not. This creates an asymmetry in what speed Sonic needs to be in order to skid when attempting to move in the opposite direction. This too is fixed in KiS2. You can read more about this bug here.
Another bug fixed by KiS2 is that, when the player turns Super, a ring is instantly drained. This is due to a counter never being initialised. Now, the game waits a second before draining the first ring, which is consistent with how it drains every ring afterwards.
In Mystic Cave Zone, it’s possible for the player to become ‘detached’ from a hanging vine switch, appearing suspended in the air away from the vine itself. KiS2 addresses this by forcefully updating the player’s coordinates to match the vine every frame.
Speaking of Mystic Cave Zone, the boss of that zone has a nasty bug where, apparently due to a copy-paste error, the wrong address register is used at one point, causing a random byte of memory to be overwritten. Somehow, KiS2’s developers noticed this and fixed it.
And… that’s it. That should be the last of the bugfixes that I’ve found in KiS2. So, what other changes were made in KiS2?
Yep, JmpTos again. They always find an excuse to crop up when I do this kind of thing. For those not in the loop, ‘JmpTo’ is the nickname given to branch extensions that are present through Sonic 2’s codebase. If a branch is too short to reach its destination, it instead branches to a long-range jump instruction in order to reach it. In the first two revisions of Sonic 2 (REV00 and REV01), they appear to have been generated by the assembler. In the third revision – REV02 – they changed significantly, presumably because the developers switched to using a different assembler. They’ve once again changed quite a bit in KiS2.
What’s interesting about the JmpTos in KiS2 is that they appear to be hand-made, as opposed to the obviously-automated JmpTos in Sonic 2 REV00 and REV01. You see, it appears that the developers went through much of the game’s code, ‘tidying’ the JmpTos: rather than being messily mixed into code, as they were in REV02, they were grouped and moved to the end of their respective blocks of code. Additionally, redundant branches to JmpTos were eliminated: in Sonic 2 REV02, it wasn’t uncommon to see unconditional branches that branched to JmpTos, when they could have just been jump instructions that jumped straight to the intended destination – KiS2 removed many, if not all, of these.
Further adding to the idea that REV02 and KiS2’s JmpTos were hand-made is the fact that one of the JmpTos in REV02 (‘JmpTo13_MarkObjGone’) is completely unused. It was removed in KiS2.
Restored Debug Features
Invisible objects, such as plane-switchers and invisible walls, become visible in Debug Mode in KiS2. One object in particular is made visible with code that was previously only in REV00. This suggests that the code may have existed in REV01’s and REV02’s source code in a dummied-out form that was simply un-dummied-out in KiS2. Perhaps these debug features were hidden behind a build-time flag?
Removed Development Code
In Sonic 2, after the ‘loadLevelLayout’ function is some leftover code. The first chunk of code is the level layout loading function from Sonic 1, modified to repeat the background layout. This was used in some of Sonic 2’s prototypes.
After that is a function that converts a level’s chunks from Sonic 1’s 256×256 format to Sonic 2’s 128×128 format, and after that is a function for eliminating duplicate 128×128 chunks. These were likely used to convert Green Hill Zone’s chunks to 128×128 for Sonic 2’s “Nick Arcade” prototype.
After surviving through numerous prototypes, all three revisions of the final Sonic 2, Sonic Classics, and the Mega Play arcade version, this code was finally removed in KiS2. RIP.
Also known as ‘attract mode’, the game will play some demos if you leave it on the title screen. The developers of KiS2 attempted to preserve compatibility with Sonic 2’s demos, reenabling things like the air speed cap and giving Knuckles Sonic’s jump height when a demo is playing. Unfortunately, the result is not perfect, and the demos still manage to desynchronise at points. The developers went so far in their attempts to keep the demos working that they manually edited the inputs for the Emerald Hill Zone demo.
I could talk about the modified title screen, Wing Fortress Zone cutscene, ending, and logo after the credits, but honestly I can’t think of anything noteworthy about them. Maybe I’ll go over them in a follow-up post, if I can think of anything interesting to say.
As an experiment in what is possible with this disassembly, I’ve added an option to build a ‘standalone’ version of KiS2 that doesn’t rely on Sonic 2 or Sonic & Knuckles in order to run. This is similar to the ‘Sonic 3 Complete’ mode of the Sonic & Knuckles disassembly, which produces a version of Sonic 3 & Knuckles that doesn’t rely on Sonic 3. You can find a built ROM of this standalone KiS2 here. The intention of this, in addition to just being a tech demo, is also to make it feasible to produce ROM hacks of KiS2, which is practically impossible whilst it is dependant on two other ROMs.
Personally, I’ve learnt a lot about KiS2 from this disassembly, and I hope others will learn a lot from it too. KiS2 has always been a mysterious black box to me: its many changes and fixes always being out of reach and beyond our understanding, with no easy way to find the new in a sea of old. Every change and every fix was a needle in a haystack… but not anymore. Maybe now we can see a *complete* port of Knuckles to Sonic 2, title screen, ending, compatibility adjustments, and all!
This will never not be weird to me: for some reason, there’s a Sonic monitor lookalike in Microsoft Office. To see it, go to the ‘View’ tab and select the ‘Zoom’ option:
I don’t get it. What came first: Sonic’s monitor or Microsoft Office’s monitor? Who was copying who? Were they even copying each other to begin with?
There seems to be multiple variations of this little sprite: I have an old screenshot from 2015 that shows a version which is much closer to Sonic’s monitor sprite:
Here’s a comparison of the three:
Somebody please tell me that I’m not alone in thinking that these look uncannily similar: they’re both 30×30, they use similar greys, and even the image in the Sonic monitor perfectly matches the size and position of the image in the Office monitor. They’re so similar that I can literally put it into a ROM-hack of Sonic 1 and it works perfectly:
Somebody please help: this has been driving me nuts for the last 7 years.
To my understanding, not much is commonly known about the source code of the original Sonic the Hedgehog games. As someone who’s been obsessed with the programming of these games for almost ten years, I believe that I know a lot more than most people do. Unfortunately, my memory is awful, and I’m not going to be a Sonic hacker forever, so I want to preserve this information however I can before it’s lost again.
As of writing, the source code for the ‘classic’ Mega Drive Sonic the Hedgehog games has never been found. We do have an exhaustive list of disassemblies, however those do not capture all information that proper source code would give us. For example, while the logic of code is recovered by these disassemblies, the meaning of it might not be. Sometimes, the logic isn’t all you need to understand code: it may do something that seems pointless or nonsensical, leaving nearby labels and comments to explain it fully. However, a disassembly cannot reproduce the original labels and comments: those are lost. Likewise, a disassembly cannot reproduce code that was never in the ROM to begin with, such as disabled (or “commented-out”) prototype code. Because of this, a disassembly cannot truly replace source code.
However, small snippets of source code have surfaced over the years. The most obvious example I can think of is a fully-intact copy of a single source file (likely “EDIT.ASM”) in Sonic 2’s “Nick Arcade” prototype.
Sonic 2 “Nick Arcade” prototypes’s source file
Inside one of Sonic 2’s prototypes is a fully intact copy of the source file responsible for the game’s “edit mode” (commonly known as “debug mode”):
This source code contains everything: code, comments, labels, and even disabled prototype code that we never knew even existed before. Notably, these labels give the internal names of enemies and objects used throughout the game, while the comments elaborate on the intended level order. The source code snippet even gives some insight into the exact assembler used to build the source code into a ROM.
That’s not the only useful data in this prototype however…
Sonic 2 “Nick Arcade” prototype’s symbol list
Starting at ROM address 0x418A8 is what appears to be a partial copy of the assembler’s symbol table. Oddly, it doesn’t appear to match the ROM, as labels that appear in the above source code are recorded in this symbol table at different addresses to where they are in the actual ROM. It can be assumed, based on this, that this symbol table is leftover from a previous execution of the assembler, rather than the one that produced the ROM that we have.
From roughly 0x418A8 to 0x47B14, the symbol table follows this format:
The first four bytes denote the length of the symbol identifier divided by four, and rounded up. That is to say, it denotes the number of longwords needed to hold the identifier. Notably, this length is stored in big-endian format.
The following longwords contain the label. Unused bytes are set to 00.
The following four bytes contain the value that is assigned to that symbol. This value is also in big-endian.
Here’s an example:
00 00 00 02 65 64 69 74 69 6E 69 74 00 01 AB 12
The first four bytes are 00000002, indicating an identifier that is 2 longwords (or 8 bytes) long.
The next eight bytes are 65646974696E6974. When interpreted as ASCII, it is “editinit”, which is one of the labels seen in the above source code.
The following four bytes are 0001AB12, which is the ROM address that was assigned to this label when the assembler that produced this symbol table was ran. In the ROM we have, however, “editinit” is located at 0001BABE.
Past 0x47B14, the format of the symbol table changes. Unfortunately, the entries don’t seem to specify what each symbol’s value is, instead containing what appears to be three pointers. This format continues to 0x50000, when actual game data resumes.
The symbol table begins again at 0x50A9C, but in another format that seems to just contain identifiers and pointers.
[EDIT: I’m a dummy: apparently you *can* extract meaningful data from the later parts of the symbol table, as someone else was able to and produce this massive list.]
While the later symbols don’t contain any useful information, they are still useful for determining the original names of various bits of code. For example, the label ‘random’ is certainly what we’ve been calling ‘RandomNumber’ in the Sonic 2 disassembly. Additionally, ‘bgmset’ and ‘soundset’ are more than likely ‘PlayMusic’ and ‘PlaySound’. What’s notable about that last one is that it answers the question of whether ‘PlayMusic’ really was intended to play music, since sometimes it’s used to play sounds instead, which caused people to question whether it actually was a dedicated music-playing function, or simply a generic ‘play something‘ function.
One might be wondering why there’s an intact source file and a partial symbol table in the middle of a ROM. Well, the space that this data occupies is what would normally be padding: towards the end of the ROM, the game’s data is spaced-out for some reason, leaving big gaps. My theory is that whatever tool the devs were using to produce the final ROM simply malloc‘d a huge buffer, and pasted data where it was needed, never initialising the unused space. This resulted in the unused space containing garbage data, leftover in memory from other programs.
This isn’t the only time that symbol data appears in a Sonic game:
Sonic & Knuckles Collection’s symbol list
Unfortunately, I can’t comment too much on this due to not having a copy of Sonic & Knuckles Collection on hand. However, this game contains another instance of symbol data, inside its main EXE.
For the most complete set of symbols, however, one need not look further than…
Sonic CD’s unstripped ELF files
Sonic Gems is a compilation of various emulated classic Sonic titles. Its version of Sonic CD, however, is not emulated: rather, it is a port of Sonic CD’s PC port.
One might be wondering how a game written in Motorola 68000 assembly could have ever been ported to anything with a different CPU architecture. The answer to that is that its code was machine-converted from assembly to C. Notably, this process preserved the original labels, and possibly even the original comments.
Within the files of Sonic Gems, you can find a series of ELF files, which parallel the DLL files of Sonic CD’s original PC port. What’s special about these ELF files is that they are unstripped, containing huge amounts of symbol data.
When loaded into something like IDA or Ghidra, this symbol data is taken advantage of while disassembling or decompiling the code. This reverse-engineered code will have its original labels intact, giving a great look into the original source code. Unfortunately, the code you’ll be looking at is the disassembled/decompiled output of PowerPC/MIPS assembly which was generated from C which was converted from 68k assembly. That is to say, it’s hideous and practically unreadable. Still, if you’re able to draw parallels between this code and the original 68k assembly, you effectively have a pre-labelled disassembly.
These ELF files also contain some debug data, such as the paths of various source files. I was able to extract a bunch of them back in 2018:
This file list gives you a look at the layout of Sonic CD’s source code. Notably, you can see ‘EDIT.C’, which is very likely the file that originally contained the code seen in Sonic 2’s “Nick Arcade” prototype.
Sonic Gems isn’t the only game to contain unstripped executables like this. For instance, The Legend of Zelda Collector’s Edition contains an unstripped executable for its Nintendo 64 emulator (or “simulator”, as it calls itself. Yeah, sure, Nintendo).
Yuji Naka’s video
A few years back, Yuji Naka found some old footage of him working on Sonic 1 back in February 1990. In this video, he scrolls through a portion of the game’s source code, giving us yet another sighting of authentic source code:
Some of these filenames may seem familiar, as they survived into Sonic CD. Note that the items ending with the Yen symbol are actually directories, and not files.
This footage is also noteworthy for showing that Sonic 1 was developed on DOS. This is something that I’ll come back to later.
A tiny snippet of source code can be found in a patent that Sega filed around the time of Sonic 2’s release. It contains two tables that are responsible for Emerald Hill Zone’s spiral loops. While it’s not all that insightful, it does give you a look at the original formatting of these tables, and how the data was grouped.
The developers of the J2ME version of Sonic 1 appear to have had access to Sonic 1’s source code, or at least its assets: this is evident through how its filenames reflect labels found in the original source code. For instance, ‘scdtblwk.scd’, matches the label ‘scdtblwk’ that is found in Sonic 2’s “Nick Arcade” prototype.
Most notably, however, some of the files in this version appear to be ‘raw’, unprocessed versions of the data found in the ROM of the Mega Drive version. For example, there’s an unused function in Sonic 2 that converts the collision data from a previously-unknown format to the format that is seen in the ROM itself, and it was discovered that the collision files in the J2ME version are in this mysterious format.
There are other file format oddities as well, such as block priority being stored in its own file, instead of being embedded in the chunk data. It’s possible that this too is how the data was originally formatted, before being converted into its final form for inclusion in the ROM.
So we have all this information about the source code itself, and the game’s assets, but what about the development environment? Well, for starters, from the footage we saw earlier, we can tell that Sonic 1 was developed on DOS computers.
According to LazloPsylus, Sonic 1 was very evidently assembled with the 2500AD assembler, X68k, based partly on the fact that the syntax of the code snippet seen in Yuji Naka’s footage is unusual, and unlikely to have built with any other assembler.
However, it does not appear that Sonic 2 was assembled with X68k, as the “Nick Arcade” source code snippet uses a different syntax. It’s plausible that Sonic 2 was assembled with SN 68k, also known as asm68k, however the presence of an ‘addsym’ directive, which is not supported by asm68k, calls this into question.
Further complicating matters is something that I discovered while writing this very blog post: the “Nick Arcade” symbol table uses big-endian integers. DOS PCs are x86, and store their integers in little-endian. This suggests that Sonic 2 wasn’t assembled on a DOS PC at all, unlike Sonic 1, and rather that it was assembled on a big-endian platform like a 68k-powered Macintosh. Another indication of this is that the “Nick Arcade” source snippet uses Unix-style line endings (0x0A), instead of DOS-style line endings (0x0D 0x0A).
There’s some evidence to suggest that Sonic 2 used a different assembler between REV01 and REV02: cross-object-file function calls behave differently, ‘dc.b’ directives are automatically padded, some ‘addi’ and ‘subi’ instructions were optimised to ‘addq’ and ‘subq’ instructions, and some ‘lea’ instructions were unoptimised from PC-relative addressing to absolute long addressing. The cross-object-file behaviour of REV00 and REV01 matches that of the “Nick Arcade” prototype, so it can be assumed that Sonic 2 used the same assembler throughout development up until REV02.
Something I just discovered while writing this blog post is that Sonic 1’s machine code is similar to Sonic 2 REV02: in code that is shared between Sonic 1 and Sonic 2, the same ‘addi’ and ‘subi’ instructions are ‘addq’ and ‘subq’ in both Sonic 1 and Sonic 2 REV02. Likewise, the same ‘lea’ instructions are unoptimised. With this in mind, it would appear that Sonic 2 was migrated back to X68k during the development of REV02, and it would continue to use this assembler as it was used to develop Sonic 3, Knuckles in Sonic 2, and the Mega Play arcade version of Sonic 2.
Sonic 2’s source file boundaries
While we don’t have an exact list of Sonic 2’s source files, we do have a way of determining where its original source files began and ended: the assembler used by Sonic 2 before REV02 would resolve cross-object-file function calls in a clunky way, basically proxying them to little ‘JmpTo’ functions. To put it literally, a ‘bsr’ instruction that referenced a label which was outside of the current object file would instead branch to a single-instruction function that was appended to the end of the object file, which would itself jump to the ‘bsr’ instruction’s original destination. Because these ‘JmpTo’ functions are inserted at the end of the object file, they can be used to tell where an object file (and thus a source file) ended.
From this, we can determine things like that the ring object (object 0x25), the scattered ring object (object 0x37), the big Special Stage ring object leftover from Sonic 1, and the Casino Night Zone ring prize object (object 0xDC) were all stored in the same source file.
Sonic 2 “Simon Wai” prototype’s unassembled Kosinski file
Like the “Nick Arcade” prototype before it, Sonic 2’s “Simon Wai” prototype includes another source code file. This one, however, is not entirely complete, but it contains more than enough useful information:
‘Kosinski’ is the nickname of a compression format used by various Mega Drive games, and this source file contains data compressed in this format. The header, when translated, says…
; Before compression $8000 After compression $2c00 Compression ratio 34.4% Number of cells 1024
‘Cells’ refers to Mega Drive tile graphics, which are 32 bytes each. It doesn’t make such sense in this context, since it’s not tiles that are being compressed, but rather Aquatic Ruin Zone’s level chunk data.
Anyway, what we can learn from this source file is that Kosinski-compressed data wasn’t included into the ROM as binary data, but rather assembly data for the assembler to process. This is unusual, as every assembler that I’ve worked with supports including binary data directly, without requiring this odd shim.
Regardless, this assembly file also possibly indicates why Kosinski files are always padded to 0x10 bytes: each ‘dc.b’ directive in the source file is 0x10 values long, so perhaps the tool that produced this assembly file was unable to output a ‘dc.b’ that was less than 0x10 values long, and so it would instead output dummy 0 values until the ‘dc.b’ was ‘full’.
Aaaaaaand… that’s it. I think that’s all I know about Sonic the Hedgehog’s source code. Hopefully this information is of use to someone.
In addition, the assembler is now case-insensitive. Symbol case-insensitivity is user-configurable, just like in asm68k.
Also like asm68k, the assembler can now output a “symbol file”. This is useful because symbol files can be used by Vladikcomper’s Advanced Error Handler. However, the error handler itself cannot be assembled with my assembler, because of its use of advanced asm68k features that I have yet to implement.
One particular difference in how I’m developing this assembler now compared to how I was doing it before is that I’m now consulting asm68k’s manual (SATMAN.pdf) for details on unimplemented features, whereas previously I was avoiding official documentation, and just relying on my own knowledge and guesswork. The reason for this didn’t have anything to do with legal paranoia, just that I didn’t want to make things too easy for myself early on. Now that I’m implementing features that I didn’t even know existed, I feel that using the manual is justified.
One area where using the manual has helped is operator precedence: previously, it matched that of C, but operator precedence is actually different in asm68k, so my assembler was recreating it incorrectly.
Speaking of the operators, I’ve been able to implement my own assembler extension: asm68k doesn’t provide any logical operators, only bitwise ones, so I’ve taken the opportunity to add them to my assembler. Now, in addition to ‘&’ and ‘|’, there’s ‘&&’ and ‘||’. There’s also ‘!’ to complement the bitwise unary ‘~’. I’ve also implemented C-style equality and inequality operators: ‘==’ and ‘!=’, to complement the usual ‘=’ and ‘<>’. This makes it possible to use C-style expressions in the assembler, which is a great fit for me. It’s nice that there’s at least one area where my assembler is already superior to asm68k.
It happened sooner than I was expecting, but here it is: my assembler can now assemble the Sonic 1 disassembly.
As I guessed in the last part, there really weren’t many things left to implement before this was possible: it turns out that the Sonic 1 disassembly doesn’t use variables at all, nor does it use ‘org’ or ‘fatal’ directives. It does use string literals in ‘dc’ directives though, as well as ‘even’ and ‘end’. It also uses if/else/endc blocks, which I somehow completely forgot to mention in the previous post.
Once those were implemented, it was just a matter of ironing out a bunch of bugs before my assembler was eventually outputting a binary file that exactly matched the one produced by asm68k. Not only that, but my assembler detected various mistakes and ambiguities in the disassembly’s code, enabling me to improve the disassembly’s readability. You can see these changes here, here, and here.
I’ve got to say, it’s so cool to assemble a ROM of Sonic 1 with my own assembler, and then play it in my own emulator. It’s like I’m creating an entire software ecosystem or something. Soon enough, it won’t just be Sonic 1: I’ll be assembling my own Mega Drive homebrew too! But, before I can do that, I’ll need to make my assembler at least partially-compatible with AS.
Right, AS. The Macroassembler AS. If I want to assemble Sonic 2, Sonic & Knuckles, or my homebrew, I’m going to have to add support for AS’s features. That’s going to be tricky, because AS is a multi-assembler: it can assemble multiple different assembly languages at once. The Sonic 2 and Sonic & Knuckles disassemblies use this for assembling those games’ soundengines, which are written in Zilog Z80 assembly. On top of that, AS has extensive macro facilities (that ‘Macroassembler’ title is well-earned) which I also need to replicate. Overall, AS support won’t be easy, and it definitely won’t be happening soon.
I’d publicly release this assembler, possibly with a fork of the Sonic 1 disassembly that includes a copy of it, but I’m not sure if that’s the wisest thing to do right now: I think I should keep it private until my honours project is over to avoid any issues regarding “plagiarism” (code contributions) and “human testing” (getting feedback from users). The assembler needs some polish anyway. But don’t worry: once my honours project is over and this becomes a hobby project, it’ll be ‘release early, release often‘ all the way!