Choose the right DXTC compression algorithm

--Blade9722 07:33, 23 November 2006 (EST)

Preface.

Altough most of what I'm describing should appear obvious and trivial to those skilled in the arts, I noticed there are some common mistakes which regularly appears also in the work coming from well-experienced texturers, so I think there's a need for some clarifications.

Brief description of most common DXTC compression formats.

DXT1C/DXT1A.

Channels bit depth: 8:8:8:1 RGBA.

Compression ratio: 8:1.

The DXT1A is often described as "RGB texture with one-bit-depth alpha". Actually, for each pixel there are two available configurations: if the alpha channel is set to 1 (white), RGB channels store usual color info. Instead if the alpha channel is set to 0 (black), upon saving the texture RGB channel are set to 0 as well. In other words, each pixel could contain color info only if the alpha channel is set to white. This makes a sense, as a black alpha channel usually means that the pixel is completely transparent (invisible), so there's no need to store color info. This algorithm is tipically used for icons, decals, blood, tatoos, i.e. texture with a completely transparent part.

Instead, DXT1C is often described as "RGB texture withouth alpha channel". This is not true: the alpha channel is still present, but is completely white. Actually, there's no physical distinction between this and DXT1A, simply upon choosing DXT1C as compression algorithm the alpha channel is set to be completely white just before saving. This is tipically used for common completely opaque textures.

DXT3.

Channels bit depth: 8:8:8:8 RGBA.

Compression ratio: 4:1.

This is often described as "RGB texture with explicit alpha". It's commonnly used for partially transparent textures. There application field is the same of DXT5, theoretically you should use this when alpha channel is featuring steep variations. However, I must point out that from my personal experience in 99% of cases DXT5 will preserve better alpha channel detail, so if you don't have time enough to make a comparison, I recommend using DXT5 instead.

DXT5.

Channels bit depth: 8:8:8:8 RGBA.

Compression ratio: 4:1.

This is often described as "RGB texture with interpolated alpha". The application field is the same of DXT3, theoretically this should be chosen if alpha channel is featuring slow gradients. Again, from my personal experience this algorithm is better than DXT3 in most cases, so I recommend using DXT5 when a comparison is not available.

Notice that DXT1 compression ratio is double than DXT3,DXT5, and so it generates half size dds files. The three algorithms are different only for the alpha channel compression, the RGB part is exactly the same for all three. So if you don't need alpha blending you should use DXT1, as using DXT3 or DXT5 will only double dds file size without any improvement in visual quality. Instead, I regularly find out in texture replacement lots of DXT5 compressed textures with the default white alpha channel.

The most common mistake is this: some modders probably thinks that DXT5 is increasing RGB visual quality, so they systematically use it without caring whether alpha channel is used or not

Uncompressed dds

Channels bit depth: 8:8:8:8 RGBA.

Well, I think there is no need to explain this

Texture Types

Now, in the Oblivion engine there are three common texture types:

Color map

(aka base texture or diffuse map)

RGB Channels: texture pattern (i.e what you see without effects or lights)

Alpha Channel: the function is dependent on what is the texture function

If the texture is an icon or a decal: The alpha channel stores the transparency info, based on greyscale (White: completely opaque, black: completely transparent). All the pixels outside the icon/decal shape contour should be completely black, to make them completely transparent. If the alpha channel is one bit depth (i.e all pixels are white or black, no grey pixels), choose DXT1, otherwise choose DXT3 or DXT5.

If the texture belongs to a mesh: You have to check the mesh in order to understand the function of the alpha channel.

- If parallax shader is enabled (look here for the details lnk), the alpha channel greyscale act as a heightmap for the parallax shader (Black: the pixel is carved, white: the pixel is popping out). Choose DXT3 or DXT5 (recommended the latter).

- If NiAlphaproperty is present, the alpha channel is used for transparency. Choose DXT3 or DXT5 (recommended the latter).

- Otherwise, the alpha channel is being ignored, in this case choose DXT1

If it's a landscape texture(i.e. is inside the textures\landscape folder): I'm not 100% sure about that, but it seems the alpha channel is useless, as tranparency info is stored directly in the esp. So I recommend using DXT1. This even if in some Bethesda landscape textures alpha channel is present, as (courtesy from Qarl who discovered this) it seems it's only because they are a copy of other mesh related textures, where the alpha channel is really used.

If you don't know how or simply don't have time to check this: if you're replacing a Vanilla Oblivion texture, simply mantain the original choice.

Normal map

(identified by the _n.dds suffix)

RGB Channels: store the normal map info

Alpha Channel: if specular map is enabled, stores the reflectivity info (white: shine, Black: dull)

How to enable specular mapping: It's based upon compression algorithm

- DXT3,DXT5,uncompressed: specular map is enabled

- DXT1: specular map is disabled.

Notice that the common mistake of choosing DXT5 instead of DXT1 in this case is even worse, as the completely white alpha channel will make the texture shining like a star!!. For the same reason, if you decide to store the normal map as uncompressed to get rid of the DXTC blockiness, remember to change the alpha channel into black if the texture is meant to be dull.

Glow map

(identified by the _g.dds suffix)

RGB Channels: Stores the minimum amount of lighting used on the texture. White = full lighting, Black = no lighting (the same as if there were no glow map), #2080FF = a blue light, etc. Note that regular light brighter than the glow map will override it, so if you put a dull red glowmap on a white texture, it will appear red in the dark but normal white under good lighting conditions. Also note that it is the minimum amount of light, so a green glowmap on a texture which has nothing in the green channel will appear black.

Alpha Channel: from my knowledge, is never used, so I recommend using DXT1 in all cases.