Better Retextures : Detailed Guide For NifSkope and the GIMP
Still in progress - pictures to be added.
This is a very detailed reference tutorial. It's mostly aimed at making things easier for the modder who is tired of looking at retexes where everything is the same shade, including metal parts that shouldn't be. Hopefully, this will allow you to use GIMP and NifSkope to make better retextures with more color variety, higher contrast, and unique additional images. You may not need all the parts of the tutorial at once, but I hope this will serve as a useful resource. I've also covered normal and glow maps, since I haven't seen many resources for those.
Important Preliminaries: The Modding Toolbox
1. You'll need a BSA unpacker in order to get at the default game meshes and textures. Get one here: http://letalka.sourceforge.net/morr/index.htm
2. You'll need the GIMP, a powerful image editing program. You can use Photoshop, but GIMP is free and I will be discussing commands and features of that particular program. I'm currently on version 2.2.13. Get GIMP and the GTK runtime environment (you need it to run GIMP) here: http://gimp-win.sourceforge.net/stable.html
3. You'll need a .dds plugin for GIMP to be able to open the game's texture format. Get it here: http://nifelheim.dyndns.org/~cocidius/download.php?filename=gimp-dds-win32
4. You'll need NifSkope in order to retexture game meshes. This wonderful mesh viewing and editing tool will also allow you to see your texture change with each save as you go along. Get it here:
http://sourceforge.net/project/showfiles.php?group_id=149157&package_id=170735
Important Reminder about GIMP
GIMP has a wonderful feature that some other programs, including Blender, do not have. You can use the "undo" command (ctrl-z or click edit-undo) even after you've saved a file. The undo history only disappears after a file has been closed and reopened. This means you can try things out, look at them in NifSkope, and then cheerfully undo 25 steps in a row if you like.
So, in the immortal words of Douglas Adams, don't panic.
Choosing the Mesh to Retexture
It's completely up to you. I'm going to use the Leyawiin cuirass for an example, because it's part of my current project with WarKirby's Huge Castle mod (available if you search for "Castle" on the Oblivion section of tessource mods). In this case, I open the folder with all my unpacked BSAs in it and open the folder called "meshes," then "armor," then something like "leguardcuirass." Then I should see two folders that say "m" and "f." Since I'm female, I'll start with the cuirass under 'f', but since the same texture goes on both it doesn't really matter. Double-clicking the mesh opens it in NifSkope.
You might see the mesh looking all white, even though the "nisourcetexture" node in NifSkope has a texture there. This is because the texture is in a different folder than NifSkope expects. Rename it to the correct one (BSAS/textures/armor/leyawiincuirass.dds or whatever) and the texture should appear. It looks a little flat now, but that's because NifSkope doesn't show the normal map the way the game engine does:
PICTURE HERE
Leave that window up on the screen, but don't save yet.
Opening the Texture with GIMP
Now click through the bsas folder until you find textures/armor/ and the leyawiin cuirass. If you click on it directly you'll get an error message. Right click and choose the option that says "open with" and "gimp-win-remote" to open the file with the GIMP.
PICTURE HERE
Now we're looking at the texture in that lovely gray interface so beloved of open source folks. Don't worry, you'll get to love it, too. You might get an error message complaining that the image is a volume map, but just click "OK" and ignore it.
Hmm. The texture looks nothing like the mesh. There's a good reason for this, and it has to do with UV maps, which are a longer subject for another tutorial. Different parts of this texture are being placed on different parts of the mesh, and there's no connection between where they are in the texture and where they will be on the final project. Some bits of the flat texture will even be reused on more than one part of the mesh. That's why it's good to look at our project in NifSkope as we go, so we can see what we're doing.
Saving A New Texture and Mesh
Use File-Save As to make a copy of the texture with a new name, say MyCuirass.dds in the folder data\textures\armor\YourName. Be sure and type the extension or GIMP won't save it. I usually save with no compression and don't generate mipmaps, and that's always worked fine, but you may have your own preferences in this area.
For more info on the .dds file format and what mipmaps mean, look here:
http://cs.elderscrolls.com/constwiki/index.php/DDS_Files
Now go back to your NifSkope window and change the texture path there to where your texture is, instead of the game one. It should look exactly the same in the window. Make sure all the slashes face backward \ and not forward / or Oblivion won't read the texture. When that's done, save the mesh as something new too, such as data\textures\armor\YourName\mycuirass.nif.
This is important, because if you mess up the original unpacked .nif and .dds it's hard to get them back without unpacking all the .bsas again.
Removing an Emblem: Rectangular Select, Cut, and Paste
Okay! Now back to our texture. I chose this one for a good reason - most of its parts are rectangular, so we can easily select them. Right now I'm going to use the Rectangular Select tool (press r, click and drag) to select the horse emblem on the cuirass front.
PICTURE HERE
We're going to be using Rectangular Select quite a lot. This command is your best friend when you're retexturing, trust me.
Now I use edit-cut to make the selection disappear. Yikes! There's a hole in the texture now, with a grid showing behind it. That's no problem, because we'll fill it back in. Use the rectangular select again to choose a longer, narrower area on the left side of the hole, then alt-edit-c and alt-edit-p to copy and paste it. Now you can press "m" to move this selection over to cover up part of the hole. When it's where you want it, go to Layer-Anchor Layer to anchor it in place. Repeat with the right side, and if necessary top and bottom, to fill in the hole completely.
PICTURE HERE
If you want to make this area look more continuous, take a look at the "wrinkles" painted on either side of the new blank space. If you copy them, then drag them down and diagonally, you can make it appear that they continue right down into the selection. This helps to make it 'fit in' and not scream 'Something else was here!' You can also use the smudge command (s key) to blend the edges, but do it sparingly or you'll have a very blurry texture. Remember, any tool you use can be resized in the main gray GIMP area. Click on the little round circle or the little black square, depending what tool you're using, and a drop-down menu will appear with all the possible brushes.
PICTURE HERE
When this part looks satisfactory to you, save the .dds. Now go back to NifSkope and look at it. Voila, you can see your new cuirass with no horse on the front.
PICTURE HERE
Isolating Parts of a Texture: Paste as New and Colorize
Now, in order to make different parts of our texture look different, we could just use the rectangular select and color tools, but this method will give us a little more control.
Let's start with a nice bit of detail. There's a long, narrow piece running down the center of the Leyawiin cuirass. Rectangular select it and copy it. It might take a couple of tries to get that shape and that shape only. Feel free to zoom in and out using the "View - zoom" menus. I use "View-shrink wrap" a lot, too.
Now go to Edit - paste as new. The rectangular piece is now in its very own file that we can edit all by itself. On this new file, go to Tools-Color Tools-Colorize. Now you have three sliders, and yikes! The rectangle seems to have gone black. That's okay, it's not permanent. The sliders in front of you say something like color, saturation, and lightness. "Color" tells you what color the selection will turn, "saturation" is how intense the color is, and "lightness" is just what it sounds like. Play with the sliders until you get a color that interests you.
PICTURE HERE
Other color tools will allow us to make a selection more than one color or change it in more than one way, but we'll do the simplest way first. If you want to make your color even more vivid, or perhaps a little subtler, you can also use Tools-Color Tools-brightness and contrast. This one also has only two sliders and is very easy to use.
Once the new selection is a new color - I'll make mine a nice bright red - go to Select-Select All and then Edit-Copy.
Now go back to your original texture and click Edit-Paste. The rectangle is back in your texture. Drag it back over to where you copied it from and paste it over the original bland one. Now anchor it and save the file.
PICTURE HERE
Look at NifSkope again. Wow! That insignificant little strip actually covers several different parts of the cuirass, giving us a nice red lining peeking out here and there. This is why we need NifSkope when any retex program would do - we can see exactly what our changes actually do on the mesh itself.
Helpful Note: Parts of the Texture and What to Do With Them
You now have two files open, mycuirass.dds and the unnamed file with the pasted red rectangle in it. If you don't think you'll want it again, you can just exit the pasted one without saving it, and it will disappear forever. If you think you might need to make several different retexes of this cuirass in future, it's not a bad idea to save it as something different, such as cuirassdetail1.dds. I sometimes do this when I know I'll be making several versions of something for a project. I have saved versions of just the fabric part of the Leyawiin cuirass and another file with all the small detail parts pasted separately, just for example.
Now let's look at our main texture again. Notice how it's almost all divided into neat squares and rectangles? This means the world is your oyster, because you can recolor any of these parts separately from the others. You can recolor just the cuirass fabric, and then paste a new emblem onto it. You can give it black metal parts or a shiny belt. Anything you like. This is the answer to all those bad retexes with bizarrely-colored metal on them - cut the metal parts off and tweak them separately.
You can also paste a new emblem or insignia onto a cuirass or a shield, like this:
PICTURE HERE
For those parts that aren't strictly rectangular, here's another method.
Isolating Parts of a Texture: Erase, Fuzzy Select, and Color Select
If you need to copy a texture part that isn't quite rectangular, go ahead and select it in a large enough rectangle to contain all of it. This will mean it includes part of the rest of the texture, but that's all right, we'll fix it. Copy and paste as new again. Now press shift-e to bring up the eraser brush. Make sure it's a size you can use comfortably for the small section you need to erase. Now use it to erase the part of the texture you don't want. You can erase in a straight line by pressing down the left mouse button, then holding down shift and dragging the mouse. A line should trail out behind it. Click again when you're ready to create the line of erasure.
PICTURE HERE
You can also use this with parts of a texture that aren't remotely rectangular at all. It takes longer, but it's sometimes worth it to get exactly the color effect you want. You can also use the Tools-Selection Tools-Fuzzy select to copy and remove parts of a texture that are oddly shaped, but this takes a little practice. Remember that if you're using fuzzy select, you can hold down the shift key and keep clicking and dragging to select a bigger area.
If you want to select parts of a mesh that are mostly one color, like the yellow-gold parts of the archmage's robe, you can also use Select-Select by Color and click on that color. Then click Select-Invert to select everything BUT that color, then Edit-cut. Now everything but the yellow you selected is gone. This is also tricky, because any given texture contains many subtle color shades that may seem similar to you but very different to GIMP. In this case, remember that you can select more than one color by holding down shift and clicking away with the color selector. This feature alone has made retexing much easier for me, and I hope it does for you, too.
More Advanced Color Tools: Hue/Saturation/Value and Color Curves
Here's where we get a little fancier.
Hue/Saturation/Value
Now this can be fun. I'd use it sparingly, because often you can get results just as well with color select and brightness/contrast, but there are times when it's quite a powerful tool. You can use this to make changes to the entire picture, or use color select to make changes to just one color.
Let's start with the whole picture. Suppose we're looking at the Leyawiin cuirass again, and we'd like the parts we made bright red to be blue instead. Click on Tools-color tools-hue/saturation.
PICTURE HERE
Now you're looking at a puzzling wheel of colors and three sliders under it. The wheel lets you choose a color to edit in the picture, based on what color value GIMP thinks each pixel has. Right now the wheel is set to "master," which means changes you make apply to ALL the colors in the picture. If we want to affect just red - the color I made the details earlier - click on the red color box, and the circle under it will be filled in.
There's a tiny checkbox in one corner of the dialogue that says "live preview." Make sure it is checked, so you can see what effect your changes will have before you make them "real."
With red selected, try dragging the sliders back and forth. "Value" makes the color lighter or darker, while "Saturation" makes it more or less intense. Turning the saturation to 0 will make all the red look gray, because it won't have any color any more, but it will still have a light/dark value. "Hue" is mostly what we want right now. Drag the slider until the red in the box looks nice and blue. If you have the preview checked, you can see the results on your cuirass.
PICTURE HERE
You can also see why we have to use this tool carefully. Every single red pixel in the picture is affected, not just one shade. This can result in strange spots or effects that we don't want. Sometimes you can avoid this by using color select to select a specific shade and then using HSV on just that shade.
This also gives us another possible way to change around the entire picture's color scheme. You can click back to master and desaturate the entire picture down to 0, then use the selection tools (as in the above sections of the tutorial) to recolor it one section at a time. This isn't always necessary, but it can help when you've got an overly complex color scheme that doesn't easily lend itself to tweaking with sliders.
Remember, if you want to change everything BUT a particular color, you can click edit-select by color, choose that one color, and then click select-invert. Then do whatever you like. Play around with these things and you may be amazed what you can do.
Color Curves
Now, this one is tricksy, Precious.
I've pointed out one of the shortcomings of HSV, which is that it tends to recolor every shade of a given color. Color Curves lets us change shade by shade, but it's a subtle tool and it takes practice to make it work well.
To work on the entire picture, go to Tools-color tools-curves.
PICTURE HERE
Eek! A scary exponential graph came up. Not to worry, we're not going to do calculus, the computer is there to do that for us. Click on the little box that says "value" and you'll see you have several choices, including the colors and alpha. Just for an example, click on Red.
The curve looks the same, but look what happens when you click on it! Little points appear. You can drag one particular point up and down to alter one particular shade of red, although the fewer points you have, the more shades are affected. When you drag upward or downward, the color becomes more or less intense, and dragging sideways (which is only slightly possible) tends to alter the shade. Try it and see what happens to your picture. Make sure you have "preview" enabled again.
PICTURE HERE
With a light touch, this allows you to tweak around all the colors in your texture separately. The "value" graph affects how light or dark the colors are, and different parts of it deal with different hues. Dragging the "alpha" may not appear to make much of a change in your picture, but it will make it more or less shiny when you're looking at it ingame, so be careful with this.
Again, this is one I use sparingly, but it can provide some complex and subtle effects if you know how to use it.
It also has another benefit: you can save each individual curve. This is useful if you're making the same changes to several different textures, say retexturing a whole suit of armor. Click on "save" and give your curve a useful name, like "leyawiinretexalpha," so that you can tell later where it belongs. Now if you click on "open" from this curves dialogue, it will bring up your saved curves and let you apply them directly to your texture. This can save a lot of time. Just remember that if you have separate curves for red, green, blue, AND value AND alpha, you have to save five curves. It won't save the entire curves dialogue for you.
Normal Maps
Why do I Need a Normal Map?
Oblivion is different from Morrowind in that every texture needs to have a normal map. Otherwise your mesh may look black or pink ingame. The normal map can basically be any copied .dds that is renamed to the texture name plus _n, as in mycuirass.dds and mycuirass_n.dds, but that's not the best way to do it.
With a flat normal map, your texture will look flat. With a lumpy normal map, your texture will look lumpy. With a translucent normal map, your texture will be very shiny.
How Do I get One?
The easiest way to do this in GIMP is using the GIMP normal map plugin (remember that download link up above?) so make sure you have it installed. Then open your original texture. If the plugin is installed correctly, you should be able to go to filters - map and see a list that includes the word "normalmap." Click on it, and you get another dialogue with sliders. We really like our sliders in the GIMP.
Again, make sure the preview is enabled. With the default settings, you should see some sort of purplish thing in the window (it won't show a preview of the whole file with this one). This is because the normal map doesn't affect the color of the texture, but it tells Oblivion how much the texture should stand out or sink in over the mesh.
The Copout Normal Map
If you just don't want to deal with this, leave the menu up top at "four sample," and the scale at 1, and set the conversion below to "max rgb."
PICTURE HERE
Then click "OK." This should give you a usable not-overly-shiny map. Then save the result as mycuirass_n.dds, or whatever an _n.dds version of your texture would be. Make sure it's in the same folder as your texture, so the game can find it. You don't have to change anything in NifSkope for this to work, it just has to be in the right folder.
The Better Normal Map
Here are some useful techniques.
The Lumpy/Worn Look
To get a look wherein you can see every scratch on a sword and every dent in the armor, try using Sobel 3x3 and setting the scale higher than 1. (Increasing the scale makes the texture's highs and lows more pronounced.) Look at the preview until you get something that looks like you want it. You can use Sobel 5x5 for something really extreme, but I've seldom found that necessary.
Carved Runes or Bas Relief
If part of a normal map seems to be popping out when you want it to look carved in (or vice versa, for that matter), try checking "invert x" and "invert y" before you hit the "okay" button. This will affect the whole normal map, so you want to be careful in your use of it. Remember, you can cut and paste parts of a normal map just as you can with a texture, so if necessary you can cut off the runes, normal map them separately, and paste them back on.
Smooth and Shiny
Go down to the dropdown that normally says "alpha = unchanged" and change it to "alpha channel = height." Most of your normal map will seem to disappear, with only the highlights standing out. You can get a little more depth by increasing scale, but you'll most often use this one on things that need to be shiny or translucent more than they need to stand out. Some fabrics, gems, and the occasional sword blade or ultrasmooth armor use this setting.
Don't make the mistake of setting "height source" to "alpha" in the dropdown that's just above the "alpha=unchanged" one, as that seems to make everything flatten out. (I'm still not sure why.)
Glow Maps
Do I Really Need a Glow Map?
Not every texture will need one, and in some they're a bad idea. In general, glow maps are what you use when you want part of a texture to be very shiny and glowing but not the entire thing. This is used on the glass parts of the glass armor and weapons, for example, which is what gives them their look of glowing depth. It's also used to make parts of the daedric weapons glow in the dark.
I use it myself on weapons and selective parts of armor that I want to look slightly unearthly. This is also a good way to make things show up even at night or in dark places. Fiery runes are another fun use.
How Do I Get One?
You don't need a special plugin to make glow maps, because a glow map is just a black and white picture made from the texture. The very simplest glow map of all is just the original texture that you've completely desaturated (see the Hue/Saturation/Value section) of all color. Then you save it as mycuirass_g.dds and make sure it's in the same folder as the texture and the normal map.
Of course, that won't look very good. The game engine reads the glow map as "white=glow a lot, black=not at all," and gray areas often look wrong. Notice, if you open the glass cuirass glow map (something like glasscuirass_g.dds in your bsas under textures\armor\glass), you see white swirls on black, with the swirls being where the glass parts of the armor are. This keeps the chain mail or leather parts of the armor from glowing, which would just look wrong.
What I usually do first is to select the parts of the texture I want to glow. This is easier than it sounds, since it's usually one color or one set of shades that can be chosen using Edit-select by color. Then I do select-invert to select everything BUT that color or area. Now I go to Tools-color tools-brightness/contrast. Turn the contrast all the way up and the brightness all the way down. Now everything non-glowing is black. Select-invert again to go back to your intended glow areas. Now do the same thing, but turn the brightness UP instead of down. This should make your glow areas all white.
Do Select-none, and now "save as." Name the glow map mycuirass_g.dds, or whatever your texture name is with _g on the end. Again, make sure it's in the same folder as the texture.
I've recently found out that you CAN use a colored picture as a glow map. This can be useful if you want something to be one color and glow a different one. Here's a thread on the Niftools forums with some useful information:
http://www.niftools.org/forum/viewtopic.php?t=907&start=45&sid=e0388c949d1db181563a833a5937c959 Niftools thread
NifSkope Reminder
NifSkope won't show normal or glow effects, so you can basically only see those in a playtest.
Conclusion
If anyone has other techniques in the GIMP that they feel should be part of this tutorial, please feel free to create another heading and add them. GIMP is a wonderful program, and I'm sure I haven't even come near all the useful tricks it can play.