Icewind Dale 2 - Searching the Sourcecode

EE are scams for gullible sheep.

Fluent said:

It's weird, though, because a lot of musicians from way back still have their masters, mixes, etc.. Not sure why code in gaming isn't kept the same way.

Click to expand...

This is the part of the show where I get to say (in all kindness) from an american commercial:




Three things were common back in the 90s that caused a lot of problems: tape backups, really expensive versioning controls, and studio instability.

Tape backups are often more volatile than other methods such as optical media, HD backups, and especially modern data centers. Expect a certain amount of corruption over time even in good storage conditions. Also expect that the purpose of tape backup was never for posterity, but for emergencies during and just after development.

Versioning is in really good shape right now, but systems during that decade, if you had one at all, like SVN used to be a much bigger pain to work with. Code integration and merging could sometimes suck if your dev practices weren't tuned to perfection (and they often weren't in old skool studio environments). We have excellent practices now, but it's a more recent development than people realize. The gold standard for versioning, git, only came out in 2005. And many other systems iterated on the standard git set.

Third, studio volatility, is something that's still with us. It still affects source code now, but with advances in backup and versioning, it's not nearly as bad. Also publishers and studios have realized that it's in their best interests to have best practices. Picture the chaos going on in Interplay as multiple groups start getting laid off, transitioning to new studios, or leaving the industry entirely. Motivations and personal interests start diverging and it's very possible for things like code backups to get lost in the mix *even though it's essentially the primary product they sell*.

Anyhow, your analogy is understandable but not really correct. Music studio work comes from a different set of traditions than code development. The art is at the forefront of things. Code development extends from electronics engineering and because it was originally understood to be "the instructions to make the computer run", devalued what code could do. It's grown to take over society since then and glommed on a bunch of very human practices (like readability, modular chunking we call "objecty orientation", variable naming, and comment practices) that are a kind of business management, sociology, math, and linguistics hybrid. Art came late to the scene.

What we see in game development, except for the single indie auteur like ConcernedApe (Stardew Valley), is honestly more like movie making: an absolute mess of conflicting interests that swirl together like a tornado under the guidance of a director and producers. Music studios do this too, but remember that there's only one product at the end: the music. Movies and games bring together multiple artistic and not-artistic (with gasps from programmers who do view their work as art) disciplines. Lots of stuff is happening to get these things out the door and naturally involves particular interests from publishers with money that don't often think much of historical disciplines like archival work and preservation and focusing on things like not letting code and trade secrets loose on the world. Some like Richard Stallman hate this, but I'm sympathetic.

Incidentally, the movie industry has nearly as bad a problem with preservation such that people like Martin Scorsese have devoted much of the latter half of their lives to recovering old decaying films.

In the end, things are better now, but not perfect. Expect that a brand new art form that originally required big bucks and investors would have trouble with this sort of thing. Partly from technology issues, partly because people back then just didn't know any better and didn't have formal guiding theories for how dev should properly be done, and partly because the nature of the beast involved money which changes incentives pretty quick.

Yeah, I get that. I think we're in an age where things like code should be archived and kept all the same.

luj1 said:

EE are scams for gullible sheep.

Click to expand...

Oh give me a break. Stick a sock in it and grouse about something more productive. You tempt me to buy more EE copies to spite you. Beamdog did not pee in your cereal.

The work done on the Infinity Engine games is not remotely trivial and entirely welcome because it makes part of game history accessible to new audiences.

Fluent said:

Yeah, I get that. I think we're in an age where things like code should be archived and kept all the same.

Click to expand...

We know that now (mostly). Part of the problem was that inflection point where costs to keep code around dropped precipitously. If you think about how often and by how many people code is changed, it's amazing that back in the bad old days of sort-of versioning people thought of keeping things for posterity at all. Changes just build and build and build.

I have the fortune of being in the gap between four major engineering disciplines (electrical, systems, aerospace, software). Software engineering is a much more formal practice now, but even still I'm floored at how iterative it is vs. say, designing and fabricating a printed circuit board. Physical engineers are adopting lots of practices from software engineering (see: SpaceX for an example of success), but it's always naturally slower. So you'll nearly always find schematics and drawings for an Airbus A320 where way back when, a full set of all code changes ever in a control system was a bit touch-and-go.

@Bedwyr: Good points in post #22 but as far as backups are concerned, well, Icewind Dale 2 is old but it is not a game of the 1990s. It came out in 2002. It was the time when optical media were slowly squeezed out by mobile hard drives.
I vividly remember my work colleagues bringing their external HDDs to work every day to exchange (pirated) movies so even in the private sector people were waving good-bye to error-prone optical media and making the switch to more reliable (and higher capacity) storage formats.

Tape backups were still a thing but mostly in places with ancient technology like government administration facilities, banks, hospitals, insurance companies etc.

Everyone else had moved on towards SCSI RAID HDD storage and I think that many companies also did offsite backups even back then (even if it required physically taking a couple SCSI RAID HDDs home after work or having someone drive the day's work/updates over to a safe, rented place).

So, on a whole I find it very puzzling indeed that not a single person is supposed to have some DVDs, an external (or formerly internal) HDD or maybe even USB (USB 2.0 came out in 2003) or any other flash/smart card format with the Icwind Dale 2 source.
I really hope that someone can help Beamdog out here. It would be a real shame and tragedy if the source code is truly lost for good.

I think this is more a result of the writing being on the wall for Interplay, rather than a lack of media for archival.

Straight from wikipedia:

By 2001, Titus Software completed its acquisition of majority control of Interplay. Immediately afterwards, they shed most of Interplay's publisher functions and signed a long-term agreement under which Vivendi Universal would publish Interplay's games. Interplay founder Brian Fargo eventually departed to found InXile Entertainment as Fargo's plan to change Interplay's main focus from PC gaming to console gaming failed.

Click to expand...

If your job is going up in smoke, you're likely not going to care about backing up source code for a game you got thrown into the wringer for, on a production schedule of ~12 months.

The problem with the source code is too many of these companies went bankrupt. I thought they were acquired but cant' remember the history of who acquired whom and if they even cared about the source code.

The fact that some source code is being recovered from original developers and NOT corporate entity is probably a big no-no 'cept that the there isn't a surviving corporate entity to sue them (or worse throw the law at em). I don't intend to sound anal here but some companies (esp when the original folks who ran them are long gone) can be quite anal on principal (nature of corporate lawyers - sue first - call the cops second - ask questions later).
-
Anyway back in 2002 they should have had some sort of source control and tape backup was fairly common. I think I first used cvs on a regular basis around 1990-1992 period so I know its been around for a period of time. Even back in the early 90's tape was fairly cheap and dense enough to save hard disks fairly painlessly (today is a different story as hard-disk density has outpaced tape).
-
Hum. One thing that occurred to me is that these folks came from the window/dos world so perhaps that had an impact (I was using mostly solaris in those days).

Bedwyr said:

Tape backups are often more volatile than other methods such as optical media, HD backups, and especially modern data centers. Expect a certain amount of corruption over time even in good storage conditions. Also expect that the purpose of tape backup was never for posterity, but for emergencies during and just after development.

Click to expand...

This is all pretty wrong. The expected lifetime of data on tapes is generally way longer than on optical discs or HDDs. And tapes have always been the preferred method for archiving, they aren't intended just for backups.

I've been doing tape backups as part of my job at different companies for the last 25 years, both for backups (ie "emergencies") and also for archival ("posterity"). Not once have I ever loaded an old tape, even after many years, that had any data corruption whatsoever. It's quite possible I've been a little lucky there, but my point is, it's not something that's expected as you're making it out to be. Not at all.

Hard drives often won't even start up if they've been turned off for several years, and obviously if kept on, its mechanical components can and will fail at any time - no such issue with tapes.

As for optical, it's not really competitive with tape for archival because the capacities and speeds are too low compared to tape. A modern tape (LTO7) stores 6TB per tape and can write at speeds up to 300 MB/s (both those numbers are assuming no compression, real world capacities and speeds are even higher because the tape drives do compression). The largest Blu-rays (BDXL) can only store 128GB per disc and they're only able to write at roughly 30-60 MB/s. Also, historically, data on CDs/DVDs decayed a lot faster than data on tape. Blu-rays supposedly do have a very long lifetime, even longer than tapes (though time will tell), but that's a pretty new development and wouldn't have been pertinent at the time of IWD2 release anyway.

luj1 said:

EE are scams for gullible sheep.

Click to expand...

I thought they did a good job with them. BG2 class kits in BG1 and its worth getting already.

Stingray said:

This is all pretty wrong. The expected lifetime of data on tapes ..

Click to expand...

A man that knows his tapes!

Did some work on DDS tapes many years ago (helical scan), but LTO seems zooming along (with many parallel tracks and impressive linear density) and surprised to know that BaFe has come back following metal particles.

Optically, I think that there was one company that used phase-change material to make disks for archival (their disk was huge and looked like a magnified version of a 3.5 floppy) - I think called Plasmon - but went out of business many years ago.

So yes, tape is king for high volumetric storage density.

Morbus said:

But Interplay probably threw it in the bin like it did with everything else.

Click to expand...

I still have this in my signature in the Larian forums :

"Interplay. Some zombiefied unlife thing going on there" - skavenhorde at RPGWatch

Click to expand...

Remember when Gearbox said that they can't find the code for Homeworld Cataclysm?! Some of you might be aware of the fact that GOG managed to land a digital release recently, though the game was renamed as Homeworld Emergence, allegedly due to the copyright on the WORD cataclysm being owned by Blizzard. Which raises the question: who in their sane mind would give copyright on one common word.

I wouldn't be surprised to discover that this whole "we could not find the code" covers some legal dispute, too.

Do you need the source code to do a digital release even if you're not making any fundamental changes to the game?

Unless you're bundling it in an emulator, yes you do.

Well, that is a good question. One that someone with better knowledge in programming could answer.

And seeing the number of mods for IWD2, my guess is that at least some of the modders could probably reverse-engineer the code.

Beamdog's alleged problem is the cost and they probably hope that they could get for free the code and won't have to pay for reverse-engineering. Not exactly sure if i should condone such attitude.

You can release products without code. It has been done before and will be done again .

I'm starting to wonder whether this whole thing is just another Grimoire-like tempest in a teapot. In 2014 Beamdog told PCGamer:



So have things actually changed with Beamdog's current call out for the IWD2 source code?

Apparently not. Oster's tweet, quoted in the OP, simply says that Beamdog needs the IWD2 source code in order to "start accessing" whether IWD2 EE would be a "doable" project. Here's Oster's tweet in its entirety:



__