A Special Kind of Hell

I've spent the majority of today attempting to get Synergy to compile.

Basically, as far as I can tell, after I formatted my computer some time ago and reinstalled MSVC++ 2008, reinstated all the libraries that Synergy uses (Allegro, AllegroGL & SOIL), at some stage something went wrong.

I loaded up the .sln file, and hit Build, and I was confronted with one of the strangest errors I've seen in a while: something about an inline assembly code error, expected an opcode and found a data type, or something bizarre like that.

I started to systematically compile each individual .cpp file, and found that all of them except 1 compiled just fine. The troublemaker that caused the problem was gam_Entity.cpp, which is weird because that's a very simple source file, with very few dependencies and very straight-forward class-definition-style code.

So I kind of fiddled around in that file, commenting out sections of code and recompiling, trying to find a particular line that spouted the error, and I found that no line caused it, since I ended up commenting out the entire .cpp and .h files! By this point I was just downright flabbergasted, and I turned to google for help.

I found a couple of pages that mainly went through fiddling with project properties, cleaning out .obj itermediary files, that sort of thing. But none of those seemed to fix it. So then I started thinking maybe it was something about where Allegro was being included, so I started including allegro.h and alleggl.h at different points in the header file, commenting out includes, trying #include instead of #include "allegro.h", random trial and error things like that.

I don't remember exactly at what point it worked, but somewhere it worked. As far as I can tell, no one thing that I did fixed it, it just seemed to `come good', which is troubling.

Anyway, after that assembly error stopped coming up, I had a new set of errors coming up; a tonne of link errors. Welcome to the next nightmare. This was basically about 180 LNK2001 errors saying various identifiers had already been included in .obj files. I did more scanning of the net and went a little crazy, and then I realised I had actually gone through all the functions in m_common.h and removed the inline from the beginning of each non-template function definition in an attempt to fix the previous problem. Dur. So I replaced the inlines and bam! No more link errors.

But then I got a single error; something about P1 and P2 being different versions. Scanning the net more, I found out this relates to the compiler having incosistent versions of itself, or that libraries that I'm using have been compiled with different version compilers, something quite likely. So then I kind of started losing my mind, and tried bringing in different versions of each library to MSVC++ 8.0, downloading and installing MSVC++ 9.0 and installing the libraries for that, and even going back to MSVC++ 6.0 and trying there.

What I ended up with was uninstalling 8.0 sort of out of some irrational belief that it would help to quell the curse upon my computer, but also (more rationally) that I should try reinstalling it at some point and carefully version checking each library I bring into it.

MSVC++ 9.0 overcame the P1 != P2 error, but now SOIL.lib was spouting a couple of link errors to unresolved external symbols __alloca() and something else. So now I'm quite lost. My guess is some version inconsistency between the 9.0 compiler and the compiler that SOIL was written for, ie SOIL only compiles on 8.0 or earlier, or something like that. That's something I mean to test when I reinstall 8.0.

I then tried to comment out the SOIL code and I was left with a single link error about MSVCRT not being able to locate _main. So I followed some instructions on a website to fiddle with the MFC settings in the project settings, which just spawned more errors so I reverted back to EXACTLY how the settings were, and now I have 7 errors. It's absolute madness!

MSVC++ 6.0 finally managed to compile and execute an older version of the code once I commented out all reference to SOIL. This is basically the best result that I've achieved. At this point I'm looking at bringing in all the code that I've written since this old version, getting it to function, and then (hopefully!) compile and run. At that point I'll be back to where I was, except in 6.0 instead of 8.0, to which I say good riddance. This might also be a good time to rid myself of AllegroGL for good. One lesson I take from this is `the less dependencies, the better'.

So there it is, my special kind of hell. To be honest, this should never have happened. Part of the function of this blog is to act as an engineering journal, where I can record version changes and things I do to my projects which might end up leading to something like this current scenario in the future. I should have had logged which version all of my libraries, IDE and compiler were and the directory of my last known successful build of Synergy.

I won't make the same mistake again.

--------------------------------------------------

Synergy Version: 0.0.0
IDE Version: MSVC++ 6.0
Allegro Version: 4.2.2
AllegroGL Version: 0.4.2
SOIL Version: 2008/07/07

Current Status:

• Compiling & Executing
• SOIL producing two LNK2001 errors; all SOIL references have been commented out
• Thus, textures currently not loading
• Old demo version; multiple smoothed 3D objects floating in space

To Do:

• Eliminate all AllegroGL dependency
• Fix texture loading
• Implement multicore functionality
  
• Reintegrate following features:

1. multiple material meshes
2. the `Big Room' demo
3. lightmaps
4. unified lighting attenuation system

• Implement uniform lightmap patching system
• Implement physics library integration

Multi Yodel Scope

So, I've been on holidays for two months or so, and have been partaking in eclectic activities. Most of these are not significant to the focus of this blog, however I have managed to glean some constructive work from this time.

First up, I've given some attention to restructuring Synergy to support multicore processing. I had the design goals to allow the engine to scale with cores at run time, to have next to zero locking mechanisms in the code, and to have the multicore functionality as transparent as possible.

What I ended up with is largely a combination of two materials I read. The first is an introductory article to writing multicore games for the Xbox 360, which gives a good overall breakdown of an engine's structure into parallel processes. The second is Valve Corporation's presentation on Source Engine's multicore concepts.

Taking these two together, I basically have two `Delegator' threads that run through the structure outlined in the 360 article as 2 parallel processes. Every significant independently executable function within those threads is called with a function wrapper task(), which takes a pointer to the function to be run, and up to four parameters caste to void type. The task() function takes the information passed to it and builds a linked list with it, and then pushes that linked list onto a lock-less task queue.

All the while this is happening, there is a thread pool with some number of threads all executing a peon() function. As the name might suggest, peon functions continually check the task queue for a task, and if they can grab it they do so, break the linked list up into a function call, and then execute the function representing the task to be done. Upon completing the function, the thread returns to its peon() function to continue checking for more work to do. As you might guess, the number of peon threads in operation is related to how many CPU cores are visible to the engine. The more cores available, the more peons can be running.

Within all of this is a simple semaphore system whereby a task in the queue can be associated with a semaphore. This allows the delegator threads to wait until a certain cluster of tasks in the queue are completed before adding more tasks. This is a necessary evil simply due to the serialised structure of games themselves, and at the moment is the most logical solution I can think of. This is, in fact, a locking mechanism, but it's only operating when the actual logic of the program cannot proceed until the current tasks are completed, and it should minimise any kind of thrashing, since the peon threads never rely on the semaphores.

So that's Synergy's multicore structure at the moment. Nothing too sophisticated, and still yet to actually be tested! But at least in theory it meets all of my design goals for the moment.

Next up in my projects is a Team Fortress 2 map! I've been itching to do some mapping for a while, but struggled to settle upon a good idea until I was reading over the Valve Developer Community Team Fortress 2 pages, and there was an article about Circular Control Point map types. This is a game mode currently not implemented in any official map, but has the potential to be an extremely dynamic game type if implemented appropriately.

After reading about it, I had a rapid explosion of ideas for a map, and begun to sketch out every area in photoshop, paying careful attention to attack/defend dynamics, flanks, killzones and setup areas. As well as the obvious considerations for each class.

After about a day I had the whole map sketched out and detailed with all major routes, spawns, setup areas, rocket jump access, etc, and began to actually rough the map out in Hammer. As of writing this I have the entire map roughed out in seperate map files and am currently bringing them all together and playing a juggling act to get them to all fit together. Some points have considerably larger areas than others, and so alot of scale harmonising needs to be done, which is a little tedious.

I'm hoping to have a tight, playable alpha of the map ready by the next tf2mappers.net game day, so it can have some testing done by fellow mappers. Anyway, though it's hardly eye-candy material at this point, here's some screen shots:

Battle area below D

Attacker's far staging area at C

Primary indoor contesting area of C

Oh and here's some random sketch I half-finished up the other day:

What if the mountain defeats you?

Well, I've been largely preoccupied with the struggle between procrastination and obligation that is typical of my student career. For the most part, the only things productive I have been working on have been my Systems Programming assignment, which essentially involves developing a shell for UNIX.

Outside of that (and largely useless activities such as online gaming), I have tried to find time and motivation to work on some kind of creative activities. I'm trying to avoid diving into my large projects (such as Synergy) for a few reasons:

- it is mentally draining and involving to work on them, and I need to maintain my mental stamina for academic pursuits.
- I'm learning a lot of very pertinent programming methods in my study this semester. Methods which can apply directly to improving the operation of Synergy. As such, I want to wait until I have grasped this knowledge more fully before I apply it to my personal projects.

So my focus has been towards finding creative activities which can still contribute to the development of my overall skill set and ultimately the development of my portfolio of work, but are light-weight activities that I can pick up, work on for a set amount of time and then put down and return to my study.

Ultimately, this has lead me doing a lot more artistic activities, such as drawing. Nick introduced me to a very useful site called `Posemaniacs'. Head over there if you want to see what it's about. You don't want to? Fine, I'll explain: it is a collection of browser-embedded tools for human figure sketching and study. My link leads to the 30-second drawing section, which as you might suspect, involves the presentation of figures in poses for only 30 seconds at a time before changing, promoting rapid sketching and utmost line economy.

I've only attempted a couple of sessions, undertaking a more reasonable [for me] configuration of 60 seconds. Here's how they've turned out:

And a slightly less deplorable collection:


I also partook in a competition over at GeekFanGirl, where entrants had to either draw a scene or character from an active role play, or write short story about it/them. Turns out, I was the only one to do an art entry. I drew one of the main NPCs from my own RP I GM, the wild druid, Taech:


There was a deadline for entries, and mine made it with 40 seconds to spare. To put it another way, this entry was largely rushed. This was a positive thing, in my mind, because I learned a lot about being economical with my work flow. I'm irritated with a lot of things in the piece though, such as the messed up lines and artifacts around the silhouette, the skull structure being a complete deviation from what I wanted, and the completely schizophrenic rendering styles across the image.

Still, I'm pretty proud of the results, either way. I refuse to alter it, leaving it as a record of what I could accomplish in the time I had for working on it.

So that's about it. I'm hoping to make the 60 second sketches a regular, if not daily activity. If I can manage that, then I believe I am well on my way to becoming a competent artist.

Lightmaps cont.

Well, it's been a little while since there has been any progress of note on any of my primary projects, but today I finally felt motivated enough to jump back in to work on Synergy.

What have I been up to in the mean time? After having a game of a WarCraft III map called `Warchasers' at Ben's place, I felt compelled to make a much improved clone of its gameplay. I worked obsessively on this clone, titled Netherworld Odysseys. It has become quite sophisticated, with a generalised random loot system, random monster spawning, character saving, charged item stacking, multiple difficulty levels including difficulty scaling with number of players, door/key system and Diablo 2-style mercenaries. Here's a couple of screenshots just for fun:

Assaulting the Spider Warren

The derelict wastelands of the Prison Realm

Unfortunately the map was foremost designed as a multiplayer experience, but due to a number of technical shortcomings saving and load of characters became a necessary process to keep the map running, and saving/loading of data only works for single player. This in effect rendered the map a single-player only experience, killing my enthusiasm for it.

However, it's not a complete loss. The triggers I developed for it are highly modular in nature and much of the cool features of the map can be reused in other warcraft maps of mine. If I find time and enthusiasm I might return to it and see if I can improve some of the technical issues crippling it currently and bring it back to a playable state for multiple players, get it to a satisfying play length and release it. Until then it festers on my hard drive...

So! What did today bring? I admit that I had become horrificly bogged down with the lightmap functionality of Synergy and thus suffering a fetid case of Coder's Block. The lightmaps needed to be divided into correct patches over the level geometry to give an unstretched, unwarped surface for the lighting to exist over.

Such functionality is non-existent in Synergy in its current state, and my coder's block prevented me from proceeding further with the matter. Thus, today was spent researching methods for patching, and other methods of lighting a scene.

I ended up reading up on radiosity methods, and the literature revealed a great deal of insight to me. Such was the extent of my new-found understanding that I went back to the problem of patching with much clarity and began to sketch out a patching algorithm for Synergy (based largely upon this example).

I'm still going through it, and will continue work on it tomorrow, but as it stands currently it seems most satisfactory. Upon implementation I expect Synergy's light mapping to be devoid of artifacts, uniform and correctly aligned. From there it's a very simple extension to add in some brute radiosity.

Here's hoping there's no hitches.

Let There Be Lightmaps

Lightmaps are officially a nightmare to implement. They seem easy, but I have found otherwise. The last four days have been a battle, but I have made significant progress in implementing this feature in Synergy.

I began with the code outlined in this tutorial. A number of modifications needed to be made, including accounting for multiple light sources and using triangular surfaces instead of quads. I also factored lambertian calculation into the formula. Adhering to the tutorial strictly, plus the minor modifications I made to the formula, I ended up with this strange result:

Puzzle #1

The streaks were caused by a value overflow with my premature byte casting. Working in dwords and then casting to byte after the clamp got me to this (after adding a couple more lights):

Puzzle #2

So it's obvious to see here the mapping coordinates are wrong. After systematically trying every single combination of texture coordinates for each vertex I discovered the closest result was this:

I smell victory close at hand...

This implied the texture needed to be flipped on one axis, so I changed the order the data was input into the bitmap, and lo and behold I had this:

Victory is mine!

Although its close to a correct solution, it's easy to see there are a number of artefacts present in the lightmap, such as the lit borders and holes in corners. All of the problems become much more visible viewing the lightmap in isolation:

Oh god damnit

Higher resolution lightmaps such as 64x64 mitigate the problem, but cut deeply into the framerate. I'm still considering how I can overcome this problem, but I discovered a number of documents and papers on the subject of lightmaps, so I'll be taking a read of those and hopefully gain a bearing.

I needed to implement a basic light management system in order to support all of this process, and it has lead to some rather nice results for the lighting overall in the demo. The main improvement is the addition of attenuation constants for each lightsource which coincide with those used to calculate the lightmap, leading to nice consistent lighting:

Yes, the trees are still around

And now for something that isn't to do with Synergy engine! I was talking to Nick online and he suggested we have a go at doing some sketching together using opencanvas. I've had my head so deeply in coding territory the last few days that it was a little difficult jumping back into drawing mode, but after roughly 15-20 minutes I had this horrible looking sketch of a person that resembles a damn night elf:

Mr Cumbersome

It's ok in some respects, very mediocre in others. Working with the construction was extremely cumbersome, and has highlighted the fact that I need to engage in more studies and revision of facial construction and anatomy theory. Still it was a good process to work through, and having Nick on hand to point out issues as they manifested was a new and interesting experience. We've agreed to spot each other's work (presumably studies) from now on, which I'm looking forward to.

Bits and Pieces 2

More relatively unremarkable tasks undertaken for Synergy today. Most of it is restructuring of code, generalising demo code into specific classes. This results in a fleshing out of the mesh/model class family, including the 2 new classes SynStaticModel and SynDynamicModel.

I've also added some code in SynMesh in preparation for implementing dynamic light map functionality in meshes' rendering and an animation skeleton data member for loading in animation information from 3DS files.

I also created the almighty SynEntity class, of which many other classes are derived. It's just a position and orientation in space with some wrapper functions for doing rotations easily, but that's something that's used over and over and it has tidied up the code considerably.

I also generalised texture loading functionality into 2 functions in the GfxCore class. This has made setting up textures a breeze.

That's about it, I suppose. I'm eager to begin implementing light maps as soon as possible. Hopefully tomorrow night.

Bits and Pieces

I've been sucked into Diablo 2 and other distractions lately (damn you, Blizzard), so progress has been meagre. However, I was determined today to continue progress on Synergy. I have been deliberating over integrating a physics engine into the code, and the task is daunting.

I've decided to make the task as easy as possible by setting up everything the physics tests will eventually need and fleshing out the 3DS loading more. This entailed implementing some functionality that supports environments. There's alot of different features that fall under this umbrella, including lightmaps, materials, entity spawning, and many others.

I decided decent ground work for a number of these things was beginning work on extracting material information from the 3DS files. This task proved to be another work in cryptography, as only relatively vague information is at my disposal regarding the file format.

However, after a number of hours stumbling around in the dark, I was successful: the engine loaded in a model of a simple room that had separate materials for the walls and floor. I decided to manipulate the number in the demo code somewhat to make sure all the objects sit within this new room, in preparation for implementing the physics/collision detection. Here's how it all shaped up:

Almost starting to look like a game?!

There's a lot more material information in the 3DS file I can potentially tap into, but it is not relevant at this stage in the engine. Next I will be focusing on loading in lights from 3DS files, and maybe, just maybe take a look at calculating light maps...

I've also devised an algorithm for welding points within a certain distance threshold of each other. Big deal, right? Well as far as my searching on google went, the approach appears to be brute-force O(n^2) algorithms, from what I could find (excerpt from game programming book). My approach was very simple; apply a 3-dimensional closest-pair algorithm to the vertices, with one modification: when the algorithm reaches an input of only 2 vertices, test if their distance is below the given threshold, if it is, weld them and return infinity for the distance.

This should in effect weld all vertices within the threshold in O(n log n) time, but I have yet to implement it and test its correctness.