I've finally setup a nice decompression thread that takes care of everything (almost) without any hiccups from the rendering side.

Right now I'm only decompressing textures in full size.
When a texture it's loaded, the average color (or 1x1 mip-map) is loaded right away (the 1x1 mip is stored in the file header without any need for decompression). Then a task for decompressing the full size texture is created and the decompression starts right away.
Right now it's either 1x1 or full-size, next I'll have to pick actual intermediate resolutions depending on the image being rendered.

One nice thing that I found out is that DX10 is thread-safe by default (no need to set special flags, making one fear for degraded performance). So, the actual texture creation and decompression is all happening in a thread separate from the main loop.
As I mentioned before, a texture decompressor will unpack data in slices using OpenMP. This allows me to use whatever cores I have to decompress one texture at the time as fast as possible (more cache coherent than trying to decompress different textures at once).
Currently, during load I can see all 4 cores being used to the maximum (more or less).. it's a nice feeling 8)

To minimize initialization times, I've also implemented a simple Direct3D texture object cache.
Every time a texture is released by the engine, it doesn't actually get immediately released in Direct3D but rather added to a free list (unlike the Wikipedia article, I'm actually not using linked-lists) where it can be picked up again if there is another request for a texture with the same characteristics.. otherwise it will be released from D3D after a few frames being unused.
Cache aside, I'm mostly counting on the multi-threading to cover up for all those potential stalls incurring from resource management in D3D.

The next big step will be to continuously "resize" textures depending on the needs of the frame being rendered.
This should happen lazily to avoid too much work by the decompression system.
The reason why one would want to resize textures is to economize on memory, but that doesn't have to happen at every frame for every texture. It makes more sense to keep a texture at 1024x1024 even if the next frame only needs a 512x512, rather than putting to work the decompression. The 1024x0124 may be needed soon again and the rasterizer is going to pick the right mips regardless of the maximum resolution.

In general I like the idea of progressive quality. I like the idea of drawing geometry with whatever texture resolution I have. This way I can theoretically keep a consistent frame rate and the quality of the textures will change depending on how fast the decompression can happen (which depends on the CPU/GPGPU power).

Next next I'll have to worry about geometry.. that's trickier especially when using a lot of complex materials.. but hopefully some coworkers can help there ;)

wooooo
zzzzzzzzzzz

Flash news.. I'm very busy at work 8)

I could work less, but I want to produce something good. I like the idea to take a more general approach to problems and make something bigger out of it.

One of my current goals is to develop in a scalable manner. In order to do that, things need to be rethought in a more generic form.
For example one can have a triangular mesh, or could develop a system to do a remeshing to turn the original geometry into a semi-regular data structure that can easily be compressed and streamed progressively.

I think that scalability is really a key to the much needed productivity improvement in game development.

At work we talk every day about how to go about some solution, and there the key question is always: "can we use a scalable and generic solution ?".
This is usually about development pipeline.. not about actual code. The idea of code reusability is less straightforward. I actually aim more at providing simple implementations, to modularize code so that it can easily be grabbed without too many dependencies.. rather than trying to fit all in a supposed grand scheme of hierarchy of objects and whatnot.

In the end the harder problems are really those about how to organize data and how to transform those data across the development pipeline.

On the side, I also used OpenMP for the first time. After a few odd results, I managed to parallelize a loop that uncompresses images in that progressive-JPEG-like format that I've been working on.
Like for JPEG, the image is processed by 8x8 sub-blocks. Using OpenMP pragmas I set the parallel section to happen on rows of blocks.
Parallelizing every row of blocks makes sense, but I could probably try to do multiple rows at once to see if I can reduce overhead of context switching and potential cache trashing. Parellelizing every block instead turned out to be overkill.
As a rule of thumb, if I think that I could wrap some code into a function with practically no overhead, then perhaps I can make parallel section out of it. In fact, I think that OpenMP eventually grabs that section and makes a function out of it anyway...

Aside from some early decoding artifacts due to my inability to share some variables from outside the "parallel for" (see the example at the bottom here), using OpenMP was really easy. Definitely much simpler than manually creating and reusing threads, also less involved than using Threading Building Blocks because one doesn't need to create functor objects and also because OpenMP is readily available with modern compilers with minimal effort.

cool
zzzzzzzzzz

I found a paper on Intel's site. It's from someone at Id Software.
I think it's pretty close to what's behind the cool MegaTexture name.

..basically a very optimized progressive-JPEG-like (edit: actually streams but it's not progressive in the "progressive JPEG" sense) streaming and decompression in real-time. Something very close to what I'm doing recently (as a task in a project, not as a full blown research).

For my implementation I decided to put an extra effort and make it easy to use outside the main application. This means that I'm going hide all my "nice" support classes and types and only expose the bare minimum for anyone to use the proposed functionality.

I set the goal of making a DLL out of this progressive CODEC, but more DLLs for the future.
The reason for a DLL rather than a LIB is that static linking can be pretty tricky. For example I have a global new and delete overload because I normally need 16/64 bytes aligned memory.
It's nice to be able to use a lot of my common library, but it wouldn't be nice to force headers and symbols onto other potential users.

Thinking in a DLL way also makes it easy to write at least one clean class that is easy to understand and document. The class exported in the DLL header is a bare-bones interface with a pointer to an actual hidden implementation class (aka PIMPL).

...basically I'm talking about modularity ! ..back from the dead.. saving the day where the OOP abuse complicates APIs.

cool

Italy lost to Spain in a match for the European Soccer Cup (whatever it's called 8).

On one side I wish Italy won.. on the other side, it's good that Italians understand that they need to get busy and get a culture (practical and scientific, not humanistic !).. fuck soccer ! (Affanculo il calcio !!)

Italians, move your asses ! Kick out the lame politicians and then kick yourself in the butt.. because it's the culture that is wrong.

Change yourself !

wooooo

P.S. Coglioni Del Mondo !

I'm a bit lost. The older (wiser ?) I get the more I realize how everything in life is so unstable and relative.
Most things are what we make of them. You could be stupid and become the president of a big country, start a few wars, change history for better or for worse. People then may or may not realize how stupid you are.. but it will still be a bit win for chaos. Chaos rules modern civilizations, big time.

At work things are going well.. relationships with coworkers improve everyday. As a senior employee, I feel that younger coworkers have certain expectations towards me.. which makes sense, but at the same time one could be wrong on something at any age.
Some people will challenge the status quo more than others. Sometimes I'm challenged myself, though I'm a person that challenges things... challenge the challenger !

You can challenge something, demonstrate the opposite and possibly be successful in your job. Or you can go along and still be successful.

Basically I think that nothing is really well defined. I can see how I could take a path at work and focus on something or focus on something else.. there is a lot of potential everywhere. I could be rich and famous if I cared for that more than other things.. then people complain that they are not rich, but I can't see how it's not their fault.

Same is with personal relationships. A recurring pattern:

10 find a girlfriend
20 take her for granted and get tired of her
30 break up
40 regret for a while
50 find a new girlfriend
60 stay in limbo for a while
70 lose interest for the previous girlfriend
80 goto 20

.. in the meantime there is this ghost.. the ghost of the "M" word.
My opinion is that if I didn't leave Italy, I'd be "M" by now ("M" is pronounced: "fucked !") .. because life would have been much more static. I'm not a wild adventurer, but you don't move to California and then to Tokyo only to get stuck in some gear.
I sometimes wonder how fashion and female customs affect economy. With all those cute girls wearing sexy clothes everyday.. perhaps affects how men think and behave. Perhaps men would be less motivated at work if there wasn't so much "fish in the sea" (though not easy to catch !) constantly teasing the hungry worker.. work harder, get richer, buy me an expensive bag !
And those sexy girls definitely influence other girls.. it's like a race to who's cuter.. or at least it's a reminder that if you don't act cute or dress sexy, then some other girl will and she'll take the better man.

Basically a competitive environment..
Of course all this has to be taken with a grain of salt. I think I see the general rule to get people going and working. It's true everywhere, but perhaps slightly accentuated in Japan. To what degree and to what extent I don't know, but I can definitely see why women in Rome and in Los Angeles dress a lot more casual than their Japanese counterparts.

..but where does all this lead ? I'm supposed to feel arrived, to want to settle, to want to go home early. Instead I want to do my programming, I want to code graphics, mostly, but anything else.. I like the freedom to go about projects at work or at home.. and I find it hard to believe that I will ever want to get into a relationship that is destined to become affection and little or no passion in the best case. But more like fighting, cheating, separation in the more common case.

Every time the "program counter" gets to 50 I'm glad that I'm still in the loop, because every cycle is a bit different, every time I learn something new and I fear the day when I will have to break that cycle.

ummmmmmmmm !!!!

Two days ago I rented 3 movies from the local store. At somebody's request I also rented The Helix... Loaded (Ahorix in Japanese (aho = idiot)) a parody of The Matrix.
Not being a big fan of the original movie and knowing how rare can be a good parody, I rented it with some skepticism.
The movie however beat all my expectations, in a negative way. I couldn't force myself past the first 10 minutes.
This parody is completely pointless, it looks like some home movie, but a lot less funny, because it also includes some decent (by home-movie standard) special effects. At least it could have been funny for being blatantly cheap, but instead it's sad for showing that someone actually put any money into it (and me for accepting to rent this lemon).
I'm just glad that I only wasted 10 minutes into it... the whole thing is 1 hour and a half. I actually think that if I had to waste 97 minutes of my life for something so void, I would probably enjoy more playing with the actual DVD disc for its intriguing prismatic effects.

Some movies are so good that one must see them, some are so bad that one has to see them.. this movie is so empty and unfunny that it really has no point. Or maybe one could create the category of movies that are so unintentionally unfunny that one has to rent them to try watch the first 5-10 minutes to feel what is like to be lobotomized.

Very sad indeed.

One of the reasons why I left Italy was that Italians talk and argue about politics constantly, as if all you had to do was to cast a vote to some mythical "right party" to fix all the problems at once.

Recently I've been reading the Italian news again, between the plain ansa.it and the opinionated beppegrillo.it, the blog of Beppe Grillo, a controversial standup comedian banned by TV for his political jokes and that later on gained some serious creed by predicting one of the biggest corporate scandals in Europe.
Mr. Grillo's blog is one of the most frequented in the World, now being translated in English and Japanese. But I doubt that many people outside Italy actually follow it as only the Italian version gets between 3000 and 4000 comments per post.. too many to make any sense of them, and frankly most of those comments are just random venting.

With Naples drowning in its own trash for almost half a year now, there is definitely a lot to talk about.
It seems that Italy managed to stop funds to one of our few Nobel Prize scientists, Carlo Rubbia, which is famous for nuclear research, and that has been working on using solar energy. The research was first founded in Italy, but then he had to move to Spain... while the Italian politicians are pushing for Nuclear energy (banned by a referendum 20 years ago) and at the same time burning trash with cancer-facilitating incinerators, that for the occasion have been renamed as "thermo-valuers".

At the root of all this there is culture. Italians are not altruistic at all and are relatively ignorant on how things go about in the rest of the World.
One could argue that the United States of America is also a fairly closed country where news feeds are rather limited (see Anna Miller's 2008 talk on TED).
USA however is a country with so much active research and development. It gathers the brightest minds from all over the World and to some extent it can afford to have the general population not being too concerned or too educated about what happens outside (there goes the Iraq war..).

Back to the Italian situation. I believe the country needs people to simply go to school, study a lot, study scientific matters, develop logical skills and also study abroad.
We need more and more people to stick their necks outside and see on a practical level how things work in the rest of the World. You can't improve much if you have no practical experience of what other countries are doing right.
In the short term however, before the average cultural level grows, we need to start making sure that people that end up with political power are not your "average Luigi".
People need to face that for a special job one needs special people.
A person with power to govern should have, first of all, a basic moral integrity. A stable personality, great intelligence and motivation (aside from getting rich and powerful !).
Not everyone needs to be a physics Nobel Prize, but that would be ideal.

Most of all, politicians mental abilities and economic activities should be constantly monitored. There should be periodical reviews in which politicians' goals and achievements are being shown to the greater population (leaving some room for national security, but not too much).

Basically, I want to know that whoever is taking decisions for me is actually more fit than me, if not, then I should be doing his job !
Politics shouldn't be a career, it should be a service. Granted that experience is a key factor in everything, but experience is also an excuse for people to get lazy and to make huge mistakes on the behalf of millions and millions of people.
Being a politicians is a job of great responsibilities, it ought to employ the brightest minds and put them constantly under scrutiny !

..in the meantime, Italian parliament and senate includes convicted criminals and shady individuals.
Get those out first !

I've been coding at home.. with my iMac.. only, using Windows XP 8)

While learning new OSes and new development environments or even new languages (Objective-C !) can be a great thing, my current focus is really mouse about graphics that application interfaces or multi-platform development.
I'm also so much used to Visual Studio (.NET 2005, using VC6 key mapping) and Visual Assist X.

I'm only now managing to get a decent setup and here is a list of things that are helping me:

VMWare Fusion

Fusion can actually display single application windows mixed with the OS X windows, but personally I prefer to leave it in a single window, better if full-screen.. because I can get OS X windows on top of it (basically full-screen means "maximized", though there is no "maximize" in OS X).

One problem I had, is that the cursor stays the Mac one, meaning that I get a black caret on my black background code editor windows 8)
For that I went to change cursor type on the Windows side.

Another key factor is having 3GB of RAM..

A Real Mouse !!

With one button one has to use the Control key to simulate right click. Which is fine, until you try to do Control+LMB 8)

Only today I discovered that Apple's Mighty Mouse (standard for iMac) can actually handle a right click !! Great ! ..not really: it's very easy to accidentally do a right click while trying to do a left one. A very frustrating experience, possibly worse than the unwanted side buttons clicks, the scroll ball/nipple getting stuck and sometimes the inability to click at all. What a crappy mouse !

My solution was to plug a regular 2 buttons + Scroll Wheel mouse.. much better now.

Key and mouse mappings

I set the middle mouse button (clicking on Scroll Wheel) to show "All Windows" (see Expose' video). The "All Windows" view is actually very useful.

Then I swapped Control and Alt (Option on Mac), because on the iMac keyboard Alt takes the place of Control which is too much an important key to try attempt relearn it's position (when I'm not at work !)

...And ?

My smallish "computer desk" is still lacking. Must get rid of it, but trashing old furniture and housing large furniture does require a certain will power if you live in Tokyo 8)

I wish I had two smaller monitors rather than a large 24 inches one. In the office, I like how I can maximize Visual Studio in the left monitor while doing something else in the right monitor.

Still, I'm much happier now. I can use the same computer using applications from both OS X and Windows.
Interoperability is great. Copy and paste, drag and drop works both ways.
Keyboard and mouse are just too important, and while I can tolerate XP not recognizing the Japanese layout (could overwrite a DLL for that, but I prefer the English keyboard layout for code anyway), I definitely needed to remap the Control key and to get a decent mouse.

I'm writing this with an editor on the Mac while the Windows screen saver runs in the background. Putting an x86 processor on the Mac is the best thing Apple could have done for the machine at least for those of us that can't get into serious relationships (with operative systems 8)

woooo !!

Yesterday night I went to ageHa club in Tokyo to listen to Ferry Corsten.
I'm a big fan of Trance music.. it was great.. if only I weren't thinking all the time that I wanted to go back home and code !!

As I pretty much still develop on Windows, I've been using my Vaio laptop at home (not laptop from the company, they promised it once, but then took it back 8( ).
At home I only have a small desk and that's for the iMac. So when I use the Vaio it's on the kotatsu, with it's limited monitor and keyboard... not the best environment for developing (it's important to have a full size keyboard to write code !).

So, yesterday I decided to try installing Visual Studio on the iMac under VMWare Fusion. It works surprisingly well (I have 3GB of RAM). DX9 support doesn't seem to be there, but it's probably a setting or an upgrade. In any case recently for me it's either DirectX 10 (ewww) or software rendering (slooww).

Speaking of software rendering, I started with the idea of implementing a RenderMan state machine and borrow from RenderMan Shading Language. But the tendency is more towards Mental Ray.. which interestingly implements shaders as DLLs compiled in C/C++ (I've heard that Pixar itself does use all sort of languages to write shaders).

Writing shaders in C++ is one of my goals. HLSL/Cg kind of shaders are rather limiting, but for a good reason, those limits make it easier to write compilers that can parallelize operations. Still those limits are becoming unbearable and if shaders are written in C++, they can potentially run at the same level of a 3D engine on the same platform without having to create all those silly contractual interfaces like it happens with HLSL for example.
There will always be a problem with optimizing the interface between engine and modular shaders or any kind of module, but still, not having to worry about heavy state changes, buffering hardware display lists, etc etc, is going to make it easier to solve certain problems in different ways.

I see HLSL/Cg shaders like the (Macromedia) Flash of real-time 3D graphics: an excellent platform that can bring to great results with minimal effort, but a very limiting platform, eventually forcing a whole mindset to down the throat of an entire generation of graphics programmers.

It's important to be nonconformist sometimes !

woooooooo

It's really fucked up to think that to draw a quad one has to has to create vertex buffer resource, an "input layout" and whatnot.

It's lame but I must do it, but it's really lame !

Fack those ill conceived abortions (oxymoron ?).. started with Direct3D 1.0

Everyday I see the design mistakes I make when I write any code.. and then I think that it's the same mistakes that every programmer does.. only that most of us have to deal with the bullshit of few.

Direct3D 10 is bullshit... and so is Vista.. but that's nothing new 8)

Oh and by the way, I managed to work around the nVidia driver crash bug in Vista... all I had to do was to scale down the geometry 8)
I had position values going beyond 160000.0f, I scaled down to 50000.0f and the crash is gone.

Pretty fucked up 8)

UPDATE: It turns out that the problem arises with the size of the object relative to the far plane. I guess it's basically a far plane clipping issue.
In fact I had the driver crash again when pulling closer the far clipping plane on a scene that would work fine before.
So there you have it, now fix that driver, please !