Extracting: Railroad Tycoon

Hier könnt ihr euch selbst, eure Homepage, euren Entwicklerstammtisch, Termine oder eure Projekte vorstellen.
Forumsregeln
Bitte Präfixe benutzen. Das Präfix "[Projekt]" bewirkt die Aufnahme von Bildern aus den Beiträgen des Themenerstellers in den Showroom. Alle Bilder aus dem Thema Showroom erscheinen ebenfalls im Showroom auf der Frontpage. Es werden nur Bilder berücksichtigt, die entweder mit dem attachement- oder dem img-BBCode im Beitrag angezeigt werden.

Die Bildersammelfunktion muss manuell ausgeführt werden, die URL dazu und weitere Details zum Showroom sind hier zu finden.

This forum is primarily intended for German-language video game developers. Please don't post promotional information targeted at end users.
Antworten
Alexander Kornrumpf
Moderator
Beiträge: 1662
Registriert: 25.02.2009, 14:37

Extracting: Railroad Tycoon

Beitrag von Alexander Kornrumpf » 31.07.2019, 23:44

tl:dr; Wall of text about a very old game with no current relevance. And about this Forum. And yet you are here ...

Introduction

Long story short, I have managed to convert some of the assset files that come with "Sid Meier's Railroad Tycoon Deluxe" into a modern file format (png) and have a pretty good idea of what the other files are about. In the end this is not the tale of a great feat I would have hoped it would become. My moderate success was not due to some sudden spark of genius but due to more sleepless nights than I care to admit in which I banged my head at the disassembly.

Over years I have revisited and left the problem again many times, Years with pauses of several months between shorts bursts of renewed interest, the latter often coinciding with updates in Krishty's "Extracting Ace Combat" Thread here.

I don't even know what I did wrong over all that time. One evening without consciously doing anything different than before, I "just knew" I was, finally, looking at the disassembly of the right function (Figure 1). I ported the assembly verbatim to C (think: uint16_t ax = 0x0;) , and I was done. Subsequent restructuring of the code in a more high-level style did not reveal any hidden secrets, at least not to me. Totally anticlimactic.

_______________________________________________________________________________________________________________________

Code: Alles auswählen

_071B:0010
_071B:0010 ; =============== S U B R O U T I N E =======================================
_071B:0010
_071B:0010 ; Attributes: bp-based frame
_071B:0010
_071B:0010 sub_56E0        proc far                ; CODE XREF: sub_1EE2+B2↑P
_071B:0010
_071B:0010 arg_0           = dword ptr  6
_071B:0010 arg_4           = dword ptr  0Ah
_071B:0010 arg_8           = word ptr  0Eh
_071B:0010
_071B:0010                 push    bp
_071B:0011                 mov     bp, sp
_071B:0013                 push    si
_071B:0014                 push    di
_071B:0015                 push    ds
_071B:0016                 push    es
_071B:0017                 push    bx
_071B:0018                 cld
_071B:0019                 mov     ax, [bp+arg_8]
_071B:001C                 cmp     ax, 1
_071B:001F                 ja      short loc_56F3
_071B:0021                 out     0A6h, al        ; Interrupt Controller #2, 8259A
_071B:0023
_071B:0023 loc_56F3:                               ; CODE XREF: sub_56E0+F↑j
_071B:0023                 les     di, [bp+arg_4]
_071B:0026                 lds     si, [bp+arg_0]
_071B:0029                 mov     dx, 10h
_071B:002C                 lodsw
_071B:002D                 mov     bp, ax
_071B:002F
_071B:002F loc_56FF:                               ; CODE XREF: sub_56E0+2C↓j
_071B:002F                                         ; sub_56E0+76↓j ...
_071B:002F                 shr     bp, 1
_071B:0031                 dec     dx
_071B:0032                 jnz     short loc_5709
_071B:0034                 lodsw
_071B:0035                 mov     bp, ax
_071B:0037                 mov     dl, 10h
_071B:0039
_071B:0039 loc_5709:                               ; CODE XREF: sub_56E0+22↑j
_071B:0039                 jnb     short loc_570E
_071B:003B                 movsb
_071B:003C                 jmp     short loc_56FF
_071B:003E ; ---------------------------------------------------------------------------
Figure 1: The Disassembly looks like this. You knew that, of course, because all disassembly somewhat looks like this. That exactly is the problem with finding anything meaningful in it.
_______________________________________________________________________________________________________________________

This post will probably make very little sense without the context of Krishty's thread as linked above. I suggest you read that thread first. It is the more interesting read anyway. I will reference that thread a lot, maybe to the point where it gets awkward. To avoid misunderstandings: Krishty and I never met. The number of PMs we exchanged over the years can be counted on one hand, I think. He does not know about this project until now and I do not know anything about him other that he did a cool thing and I wanted to do the same. I will go into my motivation some more now but part of me thinks, that is all hindsight rationalisation and basically this is about competition. "If Krishty can do it, I will be damned if I can't." Maybe I'm overthinking this.

Motivation

"Sid Meier's Railroad Tycoon Deluxe" (what a title! let's call it RRT) was not the first videogame I played but the first one I owned. The first physical disc someone bought in a shop and gifted to me. The first disk that was inserted into the newly bought family computer to install a game, or probably any software, with little to no concept among the gathered users what that was: installing software. It was not a new game when I got it, I think the first release was 1990 - without the "Deluxe" back then - and this anecdote takes place ca. 1996. Many now classic gaming franchises were founded in that year, you can google for yourself, but the 1996 release surely most relevant for this story is "Sid Meier's Civilisation II", arguably marking the passage of a whole generation in the genre. I was late to the party.

Ten more years later, 2kgames made RRT available for free as part of the marketing campaign for "Sid Meier's Railroads!" an otherwise unremarkable game that does not run on Windowses newer than Windows XP. They bundled RRT with a Dosbox and a .pdf of the copy protection sheet that was so usual in these days. In preparation for this post, I checked and the link is now dead. There is no longer any sign on the 2K Homepage that "Railroads!", let alone RRT ever happened. So 2K seems to have abandoned RRT almost 30 years after its first release but RRT never again left me after downloading that full digital copy.

You can still get RRT at https://www.pcwelt.de/downloads/Sid-Mei ... 73103.html apparently, which seems legit, but I haven't tried it and you would proceed at your own risk.

As it is probably clear from the above, nostalgia is a main driver of what follows. Krishty speaks of "nostalgische Verstimmung" (an untranslatable play of words along the lines of "nostalgic moodiness" and the desire for conservation (here I won't even try to translate his exact words).

For me it is less about conservation and more about evolution. The interest in games, kindled by the one we discuss here, led to an interest in game development - who would have guessed, given were we are here - led quickly to the realisation that I am unable to produce even passable "programmers art". There are many reasons why I never produced a single game, however small, but this is the one reason which I would have put forward at age 14 or so. An age at which admitting any other reason would have been too much for a developing ego. One day around that time I realised that one of my games (not RRT) stored its assets as bitmaps which I could read. Since that day I had this fantasy that I could use the assets of an existing game to make a new one. Like I said, this never happened, but the idea somehow sticks with me until this day.

The second motivator, which I apparently also share with Krishty, is the urge to do something comparatively hard with no conceivable utility other than my own entertaiment just because it is possible. The analogy of a jigsaw-puzzle comes too mind. Simply knowing that it had to be possible to read these files because, of course, the original game did just that, helped me over many dead ends.

What next?
To manage expectations: Given how long it took to write down only this part, I am not sure I will ever complete the write-up. Given that it took me years to get to the point were I had something to write about, what are a few month for the write-up? If I get around to it you might eventually learn how I obtained Figure 2. For now you can find Part II here.

_______________________________________________________________________________________________________________________

screen1.png
screen1.png (6.55 KiB) 401 mal betrachtet
Figure 2: Teaser. Here is something you won't see in the game. A tile masked "Err", probably for debugging purposes.
_______________________________________________________________________________________________________________________

Benutzeravatar
Krishty
Establishment
Beiträge: 6863
Registriert: 26.02.2009, 12:18
Benutzertext: state is the enemy
Kontaktdaten:

Re: Extracting: Railroad Tycoon

Beitrag von Krishty » 01.08.2019, 21:10

Krishty Krishty Krishty … cool stuff about RRT … Krishty Krishty Krishty ;)

Great work and very lovely tiles you mined there! I’m keen on details – I wrote decoders for some 90s image codecs (PackBits & Co.) and always like to learn about new ones. I didn’t bother to decode the disassembly, but it looks hand-written – no modern compiler would emit such dense encoding, and I doubt compilers in the 90s were any better.
One day around that time I realised that one of my games (not RRT) stored its assets as bitmaps which I could read. Since that day I had this fantasy that I could use the assets of an existing game to make a new one.
Well written … this probably got me on ZFX :-)
seziert Ace Combat, Driver, und S.T.A.L.K.E.R.   —   rendert Sterne

Alexander Kornrumpf
Moderator
Beiträge: 1662
Registriert: 25.02.2009, 14:37

Re: Extracting: Railroad Tycoon

Beitrag von Alexander Kornrumpf » 02.08.2019, 00:03

Krishty hat geschrieben:
01.08.2019, 21:10
I didn’t bother to decode the disassembly, but it looks hand-written – no modern compiler would emit such dense encoding, and I doubt compilers in the 90s were any better.
Yes, I'm pretty sure the image decoding was directly written in ASM. The basic unit it operates on is a bit (not a byte) the way this works is to shift and then jump based on the carry flag. I am not aware of a way how this can be expressed in C directly. The algorithm is actually easier to express in ASM than it is in C I think.

In C, to my best knowledge I can do no better than this:

Code: Alles auswählen

cf = bp & 0x01;
bp = bp >> 1;
if (cf) 
where in ASM I have this:

Code: Alles auswählen

shr bp, 1
jb somewhere
I wasn't able to determine if compilers understand that these are equivalent but in any case the C version is a very indirect way of expressing this. Usually it is the other way round, I assume.

Benutzeravatar
Krishty
Establishment
Beiträge: 6863
Registriert: 26.02.2009, 12:18
Benutzertext: state is the enemy
Kontaktdaten:

Re: Extracting: Railroad Tycoon

Beitrag von Krishty » 02.08.2019, 08:06

Yeah, no way you can do this in C. I learned it the hard way while trying to implement integer overflow checks – carry/overflow is totally out of reach for compilers if you are not using the according intrinsics (_mm_addcarry_u32() & Co.). Must be hand-written. Very cool trick, though it probably fails micro-op fusion on modern CPUs.
seziert Ace Combat, Driver, und S.T.A.L.K.E.R.   —   rendert Sterne

Alexander Kornrumpf
Moderator
Beiträge: 1662
Registriert: 25.02.2009, 14:37

Re: Extracting: Railroad Tycoon

Beitrag von Alexander Kornrumpf » 15.08.2019, 23:04

tl:dr; Lengthy setting of the stage and revelling in nostalgia. Some minor technical points but mostly backstory.

Versions, Documentation, and other Quirks

In Part I, I abbreviated "Sid Meier's Railroad Tycoon Deluxe" to "RRT". This might cause some confusion to potential readers. Only after writing I learned, that among some fans "RRT" is reserved for the original game (without the "Deluxe") and the latter is commonly referred to as "RDX". But to me it always will be just "RRT" and this thread is only about the "Deluxe" version.

Sadly, I never played the original. I would love to, but chances of ever getting my hands on a copy are pretty small. Some consider the original version the better game and from what I read in retrospect some things that seemed to be improvements at the time might indeed have not aged as well as the original. In any case, when I first got the game I did not have this context and so to me it was the Railroad Tycoon, not a second version.

As a curious side-note I think the developers themselves were a bit conflicted whether the "Deluxe" version was actually an improvement. The manual, likely approved by some marketing department, has one version of the story to tell, but there is a readme1.doc (I wonder where the 1 comes from) with a slightly different phrasing. I think it is a fascinating document of its time and it probably doesn't hurt to cite it here in its entirety, ASCII-Art and all (Figure 3a). While we are at it, there is another, slightly more official looking readme file called readme.now. Among other things it contains an apology for using the wrong data types (Figure 3b).
______________________________________________________________________________________________________________________

Code: Alles auswählen

      ______             _   _  ___   __             __
 \/___|    |    | | |   | | | |  |   |_   /\  |\  | |__
 |     [][]|    |-| |   |-  |-   |   |   /__\ | \ |    |
 |         |    | | |   | \ | \  |   |  /    \|  \| ___|
/ oo--OOO-o


Release Notes:  Update for Railroad Tycoon Deluxe to version 1.01

Bug Fixes:

  1)    Save game files are now fixed.  The bug did not show up in
        our QA process.  For those of you who are versed in programming
        we forgot a parameter in a create call.  Since the create call
        was using a return address to fill in the blank it would only
        fail if your machine config was just right. Or wrong in this
        case. (I took time to explain this because someone out there
        was pretty vocal about how we could miss this.  Well now you
        know.)

  2)    Center Key now works 'C'.

  3)    Roland sound bug fix for 486 33MHz and above.


History:
   I thought some of you would like to know how we came to publish
Railroad Tycoon Deluxe.  Our Japanese development group released
Railroad Tycoon in November of 1991.  The game was a direct port to
the NEC 9801 from the IBM.  Our Japanese customers looked at the EGA
graphics and said "Uggh!" or the Japanese equivalent.  This motivated
us to rework the graphics system for a 640x400 16 color graphics
mode, (NEC Format.)

   In the process of reworking the graphics, we decided to add new
theaters and trains.  After we released the Japanese version of the
game we were asked if we could release this product for the US/UK. 
Because of the interest we had on the BBS's we decided to include it
in our Spring 1993 release.

We have concentrated on new worlds and economies and a sound and art
upgrade.  I have read your posts.  Please continue sending us your
comments and suggestions.  Understanding what you are looking for
will help us include appropriate suggestions in future products.

Thank You for purchasing
    Railroad Tycoon Deluxe


    Jeff Billings
Figure 3a: We listen to our fans: On BBS's!

Code: Alles auswählen

APOLOGIES:
        We are unable to fix a money roll over problem that manifests
    in a couple of different ways.  Your cash may mysteriously become
    negative, or you may end up with spikes in your stock graph, or 
    if your Railroad net worth rolls over, you may be kicked out of
    office.  This is an indication that you have done so well as to
    exceed the design parameters of the game.  
        
        We were also unable to raise the restrictions on the number
    of stations and trains available without completely rewriting the
    entire game.  There is room in the future for a bigger, better,
    faster game in the Railroad Tycoon line and we hope you will be
    with us at that time for more enjoyment.

                      
                      Thank-you for choosing RAILS DELUXE,

                            The RAILS DELUXE Team.
Figure 3b: Did these people learn nothing from Y2K? Oh ... [Aside: is it still safe to assume all readers are familar with the Y2K-Problem?]
_______________________________________________________________________________________________________________________

Being a mixture of old code from 1990 (original) and slightly newer code from 1993 (deluxe), and given the huge leaps of technology in that time, it is interesting to see that parts of the game are fighting for every bit -- using half bytes instead of bytes at places, having hardcoded limits for things at convenient powers of two -- while others parts are incredibly wastefull -- fullscreen renders with very little value to gameplay, redundant content, advertisements for other, long forgotton games.

There is another reason back from 1996 for which I call the game "RRT" and which is strangely relevant to my endevours years later. Consider this gem from the game manual:
Loading
This assumes your machine runs under DOS 5.0 when it boots, which is true of most IBM and compatible machines with hard disks.
(1) Turn on your machine. If it is already on, exit all programs and return to the root directory with the “CD” (change directory) DOS command. For example, if your hard disk is C: then “CD C:\” does this.
(2) Set Speed: If you have a “turbo” or multi-speed computer, use your normal speed setting.
(3) Enter MPS directory: You can enter the MPS directory with the “CD” (change directory) DOS command. For example, if you are in your root directory C:, then “CD MPS” will take you to the MPS directory.
(4) Load Program: Type “RDX” and press return. The simulation will begin loading.
For some readers this will bring back fond memories, others will probably be to young to have any idea what it is about. Both groups can be helped by watching retro computing videos on youtube, here is one about the Turbo Button. For me, when first starting the game, these instructions were at the same time much needed as they were confusing. What wasn't on the Windows 95 start menu didn't exist for me back then and Windows 95 machines did not boot into "DOS 5.0" whatever that was.

When following these instructions the game would greet you with a about 90 seconds of credits which, unlike in modern games, didn't even help to hide loading times. Figure 4 shows a screenshot from DosBox. It could be slightly shortned, but not interrupted completely, by hitting <ESC> at just the right time, but it got boring quickly.

______________________________________________________________________________________________________________________
intro_010.png
intro_010.png (8.33 KiB) 183 mal betrachtet
Figure 4: Animated Credits
_______________________________________________________________________________________________________________________

Somehow I got my hands on an old manual for MS-DOS and learned that the RDX command, faithfully typed in per the instructions from the manual, was in fact not a command at all but rdx.bat and could be viewed in an editor. That revealed that two .exe were called, intro.exe and rrt.exe. Finding this out probably is the first thing I did that remotely resembled "hacking" in the traditional, technophile sense of the word.

From that day on, I always called rrt.exe directly and never had to watch the boring intro again. And therefore, for me the game is called RRT. Until ...

What Next?
intro.exe, disregarded for many years, loads the same media file formats as the full game but is much smaller (34 KB vs 263 KB) figuratively turning a 1000 pieces jigsaw-puzzle into one with 128 pieces. Anybody who has actually solved a jigsaw-puzzle knows that this is a huge difference. In the following I will finally describe finding the image decompression routine in intro.exe. But "Extracting: Intro.exe" would have made for a boring title.

Part III is online but it is not what I thought it would be.

Benutzeravatar
Schrompf
Moderator
Beiträge: 3848
Registriert: 26.02.2009, 00:44
Benutzertext: Lernt nur selten dazu
Echter Name: Thomas Ziegenhagen
Wohnort: Dresden
Kontaktdaten:

Re: Extracting: Railroad Tycoon

Beitrag von Schrompf » 16.08.2019, 11:03

Good story. Thanks for explaining!
Häuptling von Dreamworlds. Baut an was Neuem. Hilft nebenbei nur höchst selten an der Open Asset Import Library mit.

Alexander Kornrumpf
Moderator
Beiträge: 1662
Registriert: 25.02.2009, 14:37

Re: Extracting: Railroad Tycoon

Beitrag von Alexander Kornrumpf » 21.08.2019, 21:11

Schrompf hat geschrieben:
16.08.2019, 11:03
Good story. Thanks for explaining!
Thanks for the kind words.

Alexander Kornrumpf
Moderator
Beiträge: 1662
Registriert: 25.02.2009, 14:37

Re: Extracting: Railroad Tycoon

Beitrag von Alexander Kornrumpf » 22.08.2019, 00:06

tl;dr: Sid Meier's Railroad Tycoon Deluxe .pic files are LZSS-encoded

QuickTime (tm?) won't Open my Files

To most laypersons "PDF" (the portable document format) ".pdf" (the filename extension commonly associated with it) and "Acrobat Reader" are probably vaguely the same thing. Does this make sense? You decide. Is it understandable, in terms of how Windows displays the underlying information? I think it certainly is. In that respect ".pdf" works completely different from let's say ".py" (python code). The ".py" filename extension tells us nothing about the actual file format (UTF-8? ANSI? UNIX-Lineendings? Tabs? Spaces?) and it is probably wise to have Windows associate the extension with something like notepad++ even though one does most of the actual editing in an IDE.

All of this is completely obvious once you stop and think about it, but it took me way to long to actually do stop and think. I grew up with the misconception that the filename extension is there to tell Windows with what application to open a file, whereas it is actually there to tell the user what, roughly, they might expect the content of the file to be. I think it is interesting how Windows doubles down on the misguiding interpretation by hiding the filename extension from the user, which, of course, is a completely terrible idea. So while I turn the filename extension display back on as the first thing whenever I use a new Windows installation, at the same time, unless I am very careful, I find my thoughts about these things being confused by the wrong intuition Windows has been feeding me for more than 20 years now.

The point being that I have no name for the file format which was used by RRT for its pictures. The files themselves do not tell. They do seem to start with with these four bytes 00 48 38 00 which might hint at a format indicator but that is impossible to google. I'm incredibly tempted, as a result of the confusion I just described, to call the format ".pic-file-format". I had to remind myself several times that this is nonsense, and fight my own confusion. The only thing that ".pic" really tells us, is that the file is probably a picture, which is correct.

In my research I came across maybe half a dozen applications which prefer to call their pictures .pic and might have fitted the bill, including no less than three other games from Microprose from that area. However, either I did not understand the format descriptions or they are different formats. This is to say I am not aware of any other application that uses exactly this file format. (Of course it would be embarassing now if I post the code and someone identifies that as a straightforward variant of DEFLATE or the like, but I'm pretty sure that is not the case).

Writing all of this down a memory came back to me. Back in the day, several games (Myst, I think, but also others) used QuickTime which, when installed saw it fit to register itself as the application to at least one of the filename extensions which RRT used, I think it was said ".pic". I now remember the disappointment when QuickTime could not actually open the file. So in a way, I'm making good on that loss by decoding the files myself.

What's next?

Ironically, the shortness of the section above makes me feel obliged to a) provide at least one piece of information that brings us closer to the actual point -- I did that right in the tl;dr -- and b) give a more extensive outlook what might come up next.

When I began to write this I knew it would take a long time to write it up and that I might loose interest before I finished. Several abandoned attempts at blogging taught me that much.

I planned to mirror the structure of Krishty's thread, which is why I referenced it so much in the first post. Like that thread, and a scientific article, I thought, I would of course begin with previous work. I would discuss how the .pic files from other Sid Meier Games are described on the internet how that raised hopes and how it would then not quite fit the files from RRT. Maybe I will get around to discussing that some day but seeing how this series of posts turned out, it seems less relevant to me now. Is it really previous work if it's about a different file format? I wouldn't regard other formats of .pic files (like the one QuickTime apparently knows) previous work just because of the filename extension, so why bother with these ones? I would love to compare the different formats and I probably should at least link and backlink to and from those other forums with a few sentences, but this is not my priority right now.

Anyway. According to my plan I would then point out the differences to Krishty's project and where my experiences differed from those he describes and my thoughts on why. This I will probably do next. If you are waiting for the actual format or rather decompression algorithm I'm afraid they are pretty boring. But I promise I will try to get there eventually. Until then I hope the backstory is mildly entertaining.

Benutzeravatar
Krishty
Establishment
Beiträge: 6863
Registriert: 26.02.2009, 12:18
Benutzertext: state is the enemy
Kontaktdaten:

Re: Extracting: Railroad Tycoon

Beitrag von Krishty » 22.08.2019, 08:04

LZSS, cool – like in Ace Combat 3, but probably not in separate streams.

It’s pretty impressive to see that games in the 90s had just a dozen different compression algorithms in use. That’s probably due to the fact that search engines weren’t quite mature and instead, game programmers all read the same books. (The same reason quaternions were largely unknown to game programmers until the early 2000s.)

Nowadays we have PNG (DEFLATE), JPEG (DCT + Huffman) and Webm derivatives (Wavelets) for textures on the hard drive, DXT (three different families) for textures on the GPU, Zip/gzip (DEFLATE again), BZip (BWT + Huffman) and 7-Zip (LZMA) for game archives, Mermaid/Selkie/Kraken (LZdontknow) for high-performance archives, H.265 (DCT + Motion detection + arithmetic coding) for videos *plus* the in-house stuff for things like terrain compression in flight simulators. Sucks to be a reverse engineer in the 2030s :)
They do seem to start with with these four bytes 00 48 38 00 which might hint at a format indicator but that is impossible to google. I'm incredibly tempted, as a result of the confusion I just described, to call the format ".pic-file-format".
48 38 is H8 in ASCII, so I’d just go by 0-H8-0 or the like.

On the other hand, TFX’s SPR files are entirely different from id’s SPRs, but I call them SPR anyway.
seziert Ace Combat, Driver, und S.T.A.L.K.E.R.   —   rendert Sterne

Antworten