3ds Max: Vertex Painting

Download the images and example files here: Vertex Shading Tutorial Scene

Read a version of the tutorial with linked thumbnails here: STF Thread

WHAT YOU NEED FOR THIS TUTORIAL:

- 3D Studio Max

- Gundalf's Nif exporter

The numbers in the article correspond with the pictures in the linked 7-Zip archive (which are also linked below). 7-zip is a freeware zip program which can be downloaded from 7-zip.org. It can achieve superior compression compared to any other zip program out there.

Vertex painting is a technique with three primary uses:

1) Letting you add fake shadow detail to a model

2) Allowing smoother transitions between a model and the ground

3) Blends two tiled textures without the need for a border texture

Ground transitions are highly useful if for instance you're modelling a rock or a tree, while fake shadow detail is a great touch-up on any exterior models. Bethesda uses it all the time. For example, think about the rocks around Cheydinhal. They look fantastic, right? :D Well, what if I told you all Cheydinhal rocks uses two tileable textures, and that their great look is achieved purely through vertex painting? It's true, and if you take away the vertex paint part of the equation, the rocks doesn't look very good at all.

Similarly, it's not always a viable option to set up a border texture every time you want to transition to a new texture on your model. By using vertex painting on both sides of the transition, the effect is much less noticeable. Again, Bethesda uses that all the time. Whenever you saw dark parts between two textures, that was Bethesda smoothing a bad joint. You may be wondering in which cases you would need to use vertex painting for this. It's not hard to set up border textures, and they look pretty too. That's true, but how would you transition on a rock? On a tree?

Compare, if you will, these two images of the model "rockgoldcoast2080glichen01" - a big rock found just outside Anvil (Tamriel worldspace : AnvilArenaMain : -46, -7). It's a good example because it uses all three applications of the technique.

With vertex paint in the CS - pretty. If you raise the rock higher you'd see how Bethesda painted the base area for ground transition I mentioned above, and you study it you see how they painted fake shadow detail all over it. Also note how they used vertex painting to create a transition from the green texture on top and the grayish rock on the sides.

Without vertex paint in Max - GAAH! What the heck happened to it?! Well, that's the magic of vertex paint in the above screenshot, and the naked truth of what the model really looks like without it. So, you see why this is an important part in creating nice-looking models.

Odds are you are already familiar with vertex painting without even knowing it, because it's the exact same thing as painting the ground in the Construction Set. The ground is made up of vertices and polygons like any other 3d object, and that part of the CS can be likened to a watered-down and specific variant of the technique used in 3D Studio Max.

Right, now that you know what vertex painting is good for, let's get started :)

Image01

1. I've created a simple box for this tutorial. I'm using a lot of polygons for it because the more polys, the more control you have over the shading. If it had been a real object I would have optimized and tweaked it afterward to keep the vertex shadow effect while nixing the superflous faces.

Image02

2. Add a VertexPaint modifier to your mesh. The panel pops up on its own whenever selecting that modifier in your modifier tree.

Image03

3. Unroll it and select one of the options. A mistake commonly made by beginners of this technique is to start painting right away. That won't work. You have to first *select* the parts of the mesh you want to paint.

Image04

4. Personally I prefer clicking somewhere in the viewport and pressing CTRL+A. As you can see, all the faces of the mesh lights up, indicating that they are selected.

Image05

5. Initially your model doesn't have any vertex paint at all. Sometimes it looks like it does, but it doesn't. So the first thing we want to do is to flood fill the whole model with a pure white color. You can use any flood fill color you like, and that can create some rather cool effects, but pure white is the only color that doesn't affect the texture.

Image06

6. After you've selected pure white, press the flood fill button. I've circled it.

Image07

7. Notice the icons at the top? These lets you see the mesh in different ways. Personally I like to start with turning the texture off (4th button) and only showing vertex shades (1st button). Then select a dark grey color from the color picker.

Image08

8. Paint the bottom of it with dark grey. The effect created by doing that is that of a "fake shadow". It's not the perfect solution, but it's a solution that makes an object look more real while resting on any surface. Perfect for rocks, for example. You can also use a dark green texture, which looks particularly good when used on tree trunks.

Image09

9. After you've painted around the whole base area, you'll probably notice it looks artificial, so pick a light grey color next.

Image10

10. Then paint around the base area again, higher up on the Z plane than the first time.

Image11

11. You can go on to add more fake shadow detail on the rest of the model. It often gives a subtle visual improvement that helps in keeping the player from noticing repeating patterns in tiled textures.

Image12

12. There, all done. Don't ask me why, but I get locked in after using this method and can't select stuff in the viewport, so I have to save, file > reset, and reload the scene.

Image13

13. Create an object similar in shape to your real object. Make it as optimized as you possibly can while still covering the whole mesh.

Image14

14. Since it's the same size and has the same pivot, just placing it on the same XYZ coordinates work perfectly in this case.

Image15

15. I like to give it a transparant material. All that does is make it easier to see the real model.

Image16

16. Thusly.

Image17

17. Make sure you've installed Gundalf's NIF exporter. Then select your new mesh, go to the Utilities panel, click Reset XForm then Reset Selected. Now click NifProps, tick "Is Collision Mesh", select Stone as the material and keep it as Static. Now you have a model with both vertex shading and collision.

Image18

18. Here's how it looks in the Construction Set.