bsnes News - The Future of bsnes (2010-10-19)

Talking about the future requires elaborating on the past.

Up to the present
I initially started bsnes out of frustration with all other SNES emulators showing virtually no interest in emulation accuracy. This affected me directly as a ROM hacker, when I would discover that my ROM hacks would not run on real hardware. I also felt it would be a good opportunity to create a really powerful debugger, something that was sorely lacking.

It is important to note right off the bat: I never desired for bsnes to be the most popular SNES emulator, I simply wanted people to move to better emulators, and introduce some friendly competition to get all emulators to improve, as simply submitting patches and asking them to improve was not working at all. Simple accuracy improvements even when they caused no performance penalties were rejected solely because they would break fan translations that relied upon said inaccuracies to work. The emulators had become entire platforms unto themselves. And so I set off to create my own SNES emulator ...

As it turns out, the problem of accuracy was not linear as I once naively believed, but exponential. That is to say, if emulating 90% of games requires 200MHz, then emulating 95% requires 400MHz, 97.5% requires 800MHz, etc. In other words, there are rapidly diminishing gains.

This did not seem to be much of a problem, as even six years ago the kind of hardware needed to run even what bsnes has become was very cheap and accessible to just about everyone.

I was further inspired by software such as Nestopia. Here was an emulator that came out of nowhere, and managed to become one of the most successful NES emulators around. It gave me the impression that most people cared about accuracy.

Niceties aside and truth be told, the most popular SNES emulators are to this day still less accurate than NESticle was for the NES. The hacks are just added inside the emulators instead of onto the ROM images directly these days.

Unfortunately, correlation is not causation. I failed to understand what it was that made Nestopia popular. One has to look at the time period: NESticle had very poor Windows support, very few people were able to get any audio out of it at all with Windows XP and onward. People weren't switching for the accuracy, they were switching because NESticle was no longer usable.

Speed could have also played a small part here. While NESticle could run on a 50MHz system, Nestopia needed 800MHz. But 800MHz was nothing even then, so it was overlooked. Couple this with most peoples' inability to comprehend that accurate emulation overhead is based around processor synchronization and not raw clock speeds, and what emulation accuracy even means (it is essentially a buzz-word these days), and you are put in a frustrating situation where people wonder why dynarec/HLE N64 emulation requires less resources than an accurate SNES emulator, yet they never question why an NES emulator also requires more processing power than UltraHLE and its ilk.

Rather than being understanding and rational as I suspected, it turns out that most people are instead highly aggressive and defensive when they discover that their computers are not fast enough to run certain software programs. Rather than simply admit that perhaps their hardware is showing its age, they lash out at the software developers, completely oblivious to the realities of what emulation means, and blind comparisons are made to other software programs with completely opposing philosophies as justification as to why said software programs should run faster.

But this still wasn't obvious to me. I kept trying, perhaps I could win them over on features. I compromised on some of the very ideals I set out to improve in the first place. I added support for countless compression formats, even ones that nobody else supported. I even helped fix JMA to support 64-bit systems, something the format author wouldn't even do. I added support for the defective-by-design IPS patching format. I added all of the filters, all of the fancy GUI features, save states, rewind, movie recording, pixel shaders ... truth be told, at least half of my time developing bsnes was probably sunk into improving the GUI and feature set.

But that was all in vain. I then came to the belief that perhaps it was just speed. Now seriously going against my own philosophies, further becoming my enemy to defeat him, I created alternate cores to push speed forward as much as possible while still keeping accuracy above anyone else. This was a blessing and a curse as it allowed me to also go back to my roots a bit and work on my long discussed clock-based PPU video renderer. But I was now essentially maintaining three emulators in one: the accurate version, the previous version, and a new faster version.

And all along, I was supporting that trainwreck GUI code with so many ridiculous features, and working around all of the bugs incurred by making the terrible decision to go with Qt for the GUI toolkit.

And despite all of these sacrifices, and all of my efforts, nothing has changed. Sure, maybe I attracted a few hundred extra end users, but they didn't help report emulation bugs, it was still the same core group of people who cared right from the start that helped me improve bsnes.

But at this point, the latest bsnes has 7,000 downloads, while the latest ZSNES, the worst of the worst in terms of accuracy, and no better since I started on bsnes back in 2004, has 26,000,000 downloads. By any measurement, I have failed miserably to advance the SNES emulation scene at all.

ZSNES and Snes9X did not have the problems of compatibility that NESticle had. Combined with peoples' familiarity, unwillingness to try something new, and newfound sense of complacency when it comes to upgrading computer equipment, there was never any chance for me to carve a new niche or improve anything here.

I've come to the realization that there is absolutely nothing I can do to convince people of the merits of accuracy and standardization to superior file formats. Even if I were to get bsnes to be faster than ZSNES, and more portable than Snes9X, it would make no difference. The name recognition and familiarity is already there.

Motivation
So this leaves me with the realization that I have wasted years of my life chasing something impossible.

Again, I want to reiterate that it wasn't my goal to become the most popular emulator. I'll elaborate on exactly what I wanted.

First, as I said, accuracy. Why do I care about accuracy when most people do not? Because I spent eight years prior to bsnes as a ROM hacker, tirelessly hacking and translating games never released in English so that others could play them. I successfully translated Dragon Quest V and Der Langrisser, and failed at many more. But to me, it was really important that my work was playable on real hardware. It went to reason that eventually emulators would improve (and I still believe that eventually that will happen, even if it requires the complete abandonment of the current popular emulators and the architectures they are compatible with), and these translations would no longer run.

I didn't just care about my own hacks. I bought a copier and got those running on real hardware easily enough. No, I cared about everyone's hacks. There were a half-dozen translations at that point which had problems running on real hardware. It was the norm, in fact. The ones that worked fine on real hardware were the exception to the norm. That, or very simplistic in nature.

I knew how much it'd bother me if my own work didn't run on real hardware, and I didn't want other people to run into this problem where their hard work would eventually be lost to the sands of time as older emulators were finally deprecated.

The commercially released games themselves were also important, of course. Most people may not notice that Starfox and Star Ocean battles ran at twice the speed of real hardware, or that Earthworm Jim 2 was not playing any sound effects. In fact, most people only care about the dozen or so games they play. The rest need not even exist. So long as it's somewhat close to what their fuzzy memories of decades past recalls, they are happy.

But I really cared about the games, and wanted these gems to be perfectly preserved. The work of ROM hackers was no better than the work of the original game programmers. They were just as entitled to perfect preservation of their efforts.

I thought we had hope with Snes9X v1.52, but this was for naught. The focus was placed on using a cycle-accurate DSP core, causing a significant speed hit, but this was pointless as a cycle-accurate DSP core is useless without cycle accurate CPU or SMP cores. Arguably the least important chip required to be perfect was the first goal. The rationality was apparent: sound emulation is often complained about, so do the least amount of work possible to improve it. Yet it didn't work. Despite APU timing hacks for nearly 50 games, no amount of hacks can render Earthworm Jim 2 sound effects playable, despite the fact that I was able to produce perfect sound in this game with a sample-based (32x less accurate) DSP core from anomie, years earlier. At this time, the Snes9X team has no interest in replacing the aging CPU or SMP cores with more accurate ones.

Another goal of mine was to standardize and improve the way we stored cartridges. Other emulators, which only cared about popularity, would simply add support for every ROM format extension ever used. There are seventeen different extensions still used out there in the wild because of this. One of them is ".bin" ... it has made a mockery out of the ROM loading windows, with constant false positive matches. The most popular extension is still ".smc", which was the extension used by one of the most obscure SNES copier devices made, the Super Magicom. Virtually no one uses ".sfc", which stands for Super Famicom, the official Japanese name of the SNES. And since the Japanese created the Super Famicom, it goes to reason what the extension should be.

When I asked the ZSNES developers if we could at least be somewhat reasonable and remove the most ridiculous extensions, ".bin" included, I was told this would not happen. But hey, at least they were slowly removing copier interleave file formats. Not all of them, mind you. Just one more cause of problems. But at least they were finally removing a few of them.

And then there were the ROM copier headers. These were 512-byte blobs of copier-specific data that various copiers would use to load dumped games off of floppy disks. They are completely useless to emulators. In fact they aren't really even needed by copiers, the copier developers were just being lazy when it came to loading the games.

The ridiculouslessness of this is aparrent when you realize that maybe 1% of SNES gamers in the world even own a copier. And for the 1% that do, there are dozens of copier header formats. The odds of downloading a game, and having it have a copier header for your particular game are next to none. You pretty much have to have a tool, like ucon64, to add your desired copier header and split the games to fit on several floppy disks anyway. There was never any reason for the raw cartridge data to store copier headers. All these copier headers do is cause problems, the most obvious being that raw file checksums and file sizes are meaningless since the copier header can contain any data it wants, and may or may not even exist.

Combine these headers with ROM hackers and tool developers who could not agree on whether to rely on copier headers, and Cowering who couldn't decide whether his GoodSNES ROM verification tool should use them or not, and it's a total crapshoot whether your ROM has a copier header, whether your IPS patch requires one or not, or whether your ROM hacking tool requires one or not. Of course, it only takes someone with a brain five minutes to realize there should not be any copier headers, but that's asking too much I suppose.

When I suggested we remove these headers, I was bombarded by posts of people thinking I was the new Hitler, by forcing them to remove their dearly beloved and valuable copier headers, or else. No dissenters allowed. It boggles my mind how people can be so willfully ignorant.

And then there was NSRT. Rather than help fight against copier headers, Nach chose to co-opt them, storing NSRT data where copier headers appeared, to give the emulators some special information about the games, like its opinion on what the official names of games should be (not even Nintendo has a consistent named list of games), and what types of controllers it supported.

I don't doubt that it is useful information, and that emulators probably should be able to take advantage of such information to improve the user experience, but storing this data in binary format in front of ROM data is a stupid idea. Put it in a separate file.

If there was one thing that should have been stored inside the ROM file itself, it would be ROM mapping data. You see, these ROM dumps have no information on what memory mapping chips were actually used by the games themselves. So as a result, emulators have created these elaborate hacks that scan all kinds of arbitrary bits to create the memory map, and you get this kind of hodge-podge map that doesn't really match what the real carts were like. One more problem for ROM hackers who would eventually like to see their games running on real cartridges.

Some would convince you that there is enough data to recreate these maps from the internal headers required by Nintendo. Sure, but you're effectively checksumming specific portions of the file to pick the right maps. Some will say the data tells you exactly what is needed in a standardized format, but there are two major problems with this: one, real hardware does not depend on what is in the internal header ROM data, that can be anything. And frequently is with those beta ROM dumps floating around like with "Batman vs The Joker". And two, looking at the case of "Wanderers from Ys" versus "Fire Emblem: Thracia 776", you cannot deny that the SRAM mapping trick is not part of any standard. It's a blatant hack.

But even still, I knew that it was irresponsible to put this data into the copier header. I felt that the best course of action would be an internal database to recognize validly dumped games (which would also help prevent false bug reports), along with external mapping definition files for ROM hacks and translations. I even compromised to remove the idea of an internal database entirely, and made the external mapping format fully flexible to support any known mapping configuration. I spent weeks trying to discuss this format, and the furthest it went with the ZSNES devs was nitpicking what we should call each memory mapping mode ... who gives a fuck about nomenclature? Pick something that is reasonably good, standardize on it, and move on.

Unfortunately it was a lost cause. Even after creating a standard and trying to take into consideration the very small amount of input anyone would give me, it went nowhere. No other emulator supports any form of external memory map specifications.

Next there was patches. IPS patches are a miserable file format, and copier headers only confound their problems. It's basically a coin toss with IPS: you have a one in two chance that your patch will even apply to the right areas in the file, because if the IPS patch was made on a headered ROM, your ROM also needs to have a header and vice versa.

This was another trivial problem for other emulators to solve. Simply remove the header prior to applying the patch. Problem solved, people would make their patches without headers, and they'd work everywhere. But no, that'd be too easy.

But even then, more problems were exposed: hard-patching was not reversible, and there was still no way to tell if the patch even applied successfully, as there was no checksum to compare against in IPS to tell. Given that many games have multiple revisions, this further decreased your chances of getting a patch to work.

I spent over a year working with Nach to fortify his NPS patching format. I spent a year on the most popular ROM hacking forum discussing this new format, UPS, asking everyone to please give input so that we could make it a standard. Virtually no input. Finally, after over a year, I released the finalized spec, and immediately got some ridiculous complaints about applying multiple patches, something that is just as easy with UPS as it is with IPS, and that was it. Virtually nobody distributes UPS patches in the SNES scene. It's only really caught on in the GBA scene where ROMs larger than IPS can support are common.

It is a source of great frustration that the person who initially came up with the file format has so far done absolutely nothing with it. No tools, no support in his emulator for it, nothing.

There is UPS support in NES, Game Boy, Game Boy Advance and Genesis emulators, but no support in SNES emulators, where patches are the most common. It only exists in Snes9X v1.52 because I finally broke down and added the support myself; and zones, who was never made an official mainline developer, was nice enough to add the patch once the rest of the Snes9X team abandoned the project, leaving him holding the bag.

And then there were cheat codes. Here is a format that limits you to very short, ANSI-only strings for descriptions, and has no support for grouping cheat codes. Ever see those "3CA7-1BF2+2A1D-98F3" multi-part codes? Sure would be nice to group them so you only have to toggle one to turn a code on and off. ZSNES is even worse off, as the emulator only supports a 5x5 all-caps font due to its legacy DOS design, you can't even use lowercase in your descriptions. Further, these files were binary-based, so you couldn't even edit them yourself outside of the emulator.

I tried to work with the other emulator authors to come up with a sane format that eliminated all of the old problems, again to no avail. I had to push on and create an XML format that supported UTF-8 text and unlimited-length descriptions on my own.

And it just goes on and on. All I wanted was to make everything better, more enjoyable for everyone. But I was fought every step of the way by just about everyone, absolutely and completely resistant to any kind of change from the status quo.

Wasted effort and the straw that broke the camel's back
So back to my original point about ensuring games work in the future on all emulators. Even after five years of bsnes being out there, and me preacing these virtues everywhere I could, we had the Lennus II translation in 2009. A fan translation that only ran in ZSNES.

I really tried, I put my vitriol aside, stepped in, and offered to help. I offered to work on the game, without even having the ROM hack source code, to track down and fix the problem. Only to come across a post by one of the authors on another forum some weeks later essentially calling me an asshole for being pedantic about it not running anywhere else, all the while putting "accuracy" in quotes as if it was a made-up term. Talk about gratitude. When I responded, it was of course denied that he was referring to me, the only person who ever brought it up. But I wasn't born yesterday.

Still I pressed on. Until recently. I came across yet another Chrono Trigger hack. This one had it all. Apparently the developers wanted to add CD-quality music. Hey, what about MSU1? That's a great, simple way to add support which can then be added to other emulators trivially. Nah, they instead decided to create a ZSNES fork that monitored RAM variables to play music externally.

Not only does this permanently lock the hack to ZSNES alone in a way that you can't possibly add this support to any other emulator, it also makes it impossible to maintain an accurate pitch when interacting with the emulator. Fast forward, rewind, pause, entering menus, etc ... forget about it.

Okay, well at least the patch will be UPS right? Nope, IPS. And it requires a ROM copier header, too.

Well, at least by this point people will be upset that it doesn't work anywhere else but ZSNES? Nah, maybe one person, who was subsequently ignored. "Who cares? ZSNES is perfect, use it" (even though you may not be running on an x86 32-bit compatible operating system, of course.)

And you know what? Despite me being one of the most vocal people out there, I didn't even care. I didn't even have the slightest ounce of concern to bother responding. I had had enough. I've done everything I could do by this point. All for nothing. These people are fucking retarded. You don't care? Then you know what, neither do I. When nobody can run your game in ten years, that's your problem, not mine. You can't say I didn't do everything in my power. But some people just can't be helped.

Toward the future
That led me to my next epiphany. I've been wasting years of my life chasing some foolish dream, all for nothing. The people who could ever care already do, the rest never will. All I've done was get upset and depressed at the constant barrages about how bad my emulator is because it won't run on their eMachines PC from 1999, and what a jerk I am for posting about why accuracy matters. All I've done was compromise my own values by trying to build bridges and meet at the middle ground. All in vain.

Why? Why bother anymore? I don't have it in me to care anymore. I've given up. bsnes is no different from any of my other projects: doomed to obscurity. But you know what drives me to spend seven years on Der Langrisser, only to have about 1,000 people download it? What drives me to work on bsnes for six years only to have about 7,000 people download it? It's because I do it for myself, and not for anyone else. I personally enjoy it.

And that is why I'll never give up, no matter what is said, and no matter how much people try to hurt me. Despite the fact that bsnes has always been 100% free software, and given away without asking for anything at all in return. Peoples' sense of entitlement truly frightens me.

So fuck it all. Enough with the compromises. The bsnes/phoenix port that removes Qt is a new beginning. I have not added compressed archive support, rewind support, movie support, IPS patching support, software filter support, alternate ROM extension support or even copier header support. And I'm not going to. I created the GUI I personally wanted in two weeks, not three years.

But even with this, looking at bsnes, I have to be honest. It's pretty much finished now. Every single processor, even the PPU and DSP, are emulated at the clock level. Even the add-on processors, the SuperFX and SA-1 ... all of the caches are emulated, all of it is emulated right down to the individual clocks. With 100% compatibility and no known bugs, it's about as perfect as it's ever going to get. The things I am improving now will never even be noticed by anyone.

That's not to say it is 100% perfect. New bugs continue to pop up now and again, and I'm very grateful to anyone who will use bsnes to spot them, report them, and help get them fixed. I'll continue working on bsnes to keep fixing any discovered issues, and to keep improving even those things that nobody notices, for as long as I possibly can.

But I'm done catering to others. I'm done trying to convince the world of the merits of accuracy and superior + unified file formats. It's like talking to a wall, and I don't need the pain and disappointment when people go on the defensive attacking me for it.

Going forward, I may or may not release Qt and/or phoenix-based Windows binaries. Who knows. I'm not going to destroy the Qt port, it would be stupid to throw away something already made. I'm just no longer going to keep improving it to add requested features.

Instead, I'm just going to keep focusing on the emulation core. The source code is free software and open source. People can do whatever they want with it: continue the Qt port on their own, distribute their own compiled binaries, create their own GUIs, add all the features they want, and that's cool with me. Or they can create their own emulators to compete. I don't care anymore.

I will say one thing to any potential future competition that indicates that their emulator is more accurate: they have either stolen my core, or they are full of shit. I've spent six years performing countless tests, and had the help of a half-dozen geniuses during that time like anomie, blargg and Jonas Quinn. There is nothing short of one-off contrived examples that would pass on another emulator but not on bsnes. And nothing above what I've done already will ever make any visible difference in any commercial games. Perfection may very well be impossible, but I'm as close now as one can ever realistically get. I've taken accuracy from probably 70-80% all the way up to 99.99%. Debating one-thousandth of one percent of a difference is pretty asinine, even for me. So there is really no point in engaging in competition anymore. Still, I do hope more people will step up and offer emulators that can equal bsnes. I do still want the scene as a whole to improve, I'm just done with trying to do so myself.

In closing
I formally withdraw from any and all efforts to promote bsnes and/or its ideals in the public space. From this point on, the only forum you will see me participating on is my own, board.byuu.org. bsnes the public project is dead. Insult me and my project all you want, I'm not bothering anymore. Do so elsewhere and I won't even see it, do so on my forum and get banned.

bsnes is now my personal project, and I am only uploading the source code for those who do care about the things I care about. The rest of you who would complain about my efforts, can go straight to hell. I owe the world nothing at all.

If you want my help for something SNES-related, or if you want to help improve bsnes by testing it and reporting bugs, please don't hesitate to stop by my forums.

And lastly, I do want to apologize for all of my anger of the years. It's been a direct result of the things I've discussed above. It's because I cared so much that I let it get so personal, and get me so riled up in the first place. And that is why I am stepping down from the public space.






Sourcs: byuu.org