Cel-Shading By Sami "MENTAL" Hamlaoui

Seeing as people still e-mail me asking for source code to the article I wrote on GameDev.net a while ago, and seeing as the 2nd version of that article (with source for every API out there) isn't even close to being halfway finished, I've hacked together this tutorial for NeHe (that was actually going to be the original intention of the article) so all of you OpenGL gurus can play around with it. Sorry for the choice of model, but I've been playing Quake 2 extensivly recently... :)

Note: The original article for this code can be found at: http://www.gamedev.net/reference/programming/features/celshading.

This tutorial doesn't actually explain the theory, just the code. WHY it works can be found at the above link. Now for crying out loud STOP E-MAILING ME REQUESTS FOR SOURCE CODE!!!!

Enjoy :).

First of all, we need to include a few extra header files. The first one (math.h) is so we can use the sqrtf (square root) function, and the second (stdio.h) is for file access.

#include <math.h>						// Header File For The Math Library
#include <stdio.h>						// Header File For The Standard I/O Library

Now we are going to define a few structures to help store our data (saves having hundreds of arrays of floats). The first one is the tagMATRIX structure. If you look closely, you will see that we are storing the matrix as a 1D array of 16 floats as opposed to a 2D 4x4 array. This is down to how OpenGL stores it's matrices. If we used 4x4, the values would come out in the wrong order.

typedef struct tagMATRIX					// A Structure To Hold An OpenGL Matrix
	float Data[16];						// We Use [16] Due To OpenGL's Matrix Format

Second up is the vector class. This simply stores a value for X, Y and Z.

typedef struct tagVECTOR					// A Structure To Hold A Single Vector
	float X, Y, Z;						// The Components Of The Vector

Third, we have the vertex structure. Each vertex only needs it's normal and position (no texture co-ordinates). They MUST be stored in this order, or else when it comes to loading the file things will go horribly wrong (I found out the hard way :(. That'll teach me to hack my code to pieces.).

typedef struct tagVERTEX					// A Structure To Hold A Single Vertex
	VECTOR Nor;						// Vertex Normal
	VECTOR Pos;						// Vertex Position

Finally, the polygon structure. I know this is a stupid way of storing vertexes, but for the sake of simplicity it works perfectly. Usually I would use an array of vertexes, an array of polygons, and contain the Index number of the 3 verts in the polygon structure, but this is easier to show you what's going on.

typedef struct tagPOLYGON					// A Structure To Hold A Single Polygon
	VERTEX Verts[3];					// Array Of 3 VERTEX Structures

Pretty simple stuff here too. Look at the comments for an explanation of each variable.

bool		outlineDraw	= true;				// Flag To Draw The Outline
bool		outlineSmooth	= false;			// Flag To Anti-Alias The Lines
float		outlineColor[3]	= { 0.0f, 0.0f, 0.0f };		// Color Of The Lines
float		outlineWidth	= 3.0f;				// Width Of The Lines

VECTOR		lightAngle;					// The Direction Of The Light
bool		lightRotate	= false;			// Flag To See If We Rotate The Light

float		modelAngle	= 0.0f;				// Y-Axis Angle Of The Model
bool    	modelRotate	= false;			// Flag To Rotate The Model

POLYGON		*polyData	= NULL;				// Polygon Data
int		polyNum		= 0;				// Number Of Polygons

GLuint		shaderTexture[1];				// Storage For One Texture

This is as simple as model file formats get. The first few bytes store the number of polygons in the scene, and the rest of the file is an array of tagPOLYGON structures. Because of this, the data can be read in without any need to sort it into any particular order.

BOOL ReadMesh ()						// Reads The Contents Of The "model.txt" File
	FILE *In = fopen ("Data\\model.txt", "rb");		// Open The File

	if (!In)
		return FALSE;					// Return FALSE If File Not Opened

	fread (&polyNum, sizeof (int), 1, In);			// Read The Header (i.e. Number Of Polygons)

	polyData = new POLYGON [polyNum];			// Allocate The Memory

	fread (&polyData[0], sizeof (POLYGON) * polyNum, 1, In);// Read In All Polygon Data

	fclose (In);						// Close The File

	return TRUE;						// It Worked

Some basic math functions now. The DotProduct calculates the angle between 2 vectors or planes, the Magnitude function calculates the length of the vector, and the Normalize function reduces the vector to a unit length of 1.

inline float DotProduct (VECTOR &V1, VECTOR &V2)		// Calculate The Angle Between The 2 Vectors
	return V1.X * V2.X + V1.Y * V2.Y + V1.Z * V2.Z;		// Return The Angle

inline float Magnitude (VECTOR &V)				// Calculate The Length Of The Vector
	return sqrtf (V.X * V.X + V.Y * V.Y + V.Z * V.Z);	// Return The Length Of The Vector

void Normalize (VECTOR &V)					// Creates A Vector With A Unit Length Of 1
	float M = Magnitude (V);				// Calculate The Length Of The Vector 

	if (M != 0.0f)						// Make Sure We Don't Divide By 0 
		V.X /= M;					// Normalize The 3 Components 
		V.Y /= M;
		V.Z /= M;

This function rotates a vector using the matrix provided. Please note that it ONLY rotates the vector - it has nothing to do with the position of the vector. This is used when rotating normals to make sure that they stay pointing in the right direction when we calculate the lighting.

void RotateVector (MATRIX &M, VECTOR &V, VECTOR &D)		// Rotate A Vector Using The Supplied Matrix
	D.X = (M.Data[0] * V.X) + (M.Data[4] * V.Y) + (M.Data[8]  * V.Z);	// Rotate Around The X Axis
	D.Y = (M.Data[1] * V.X) + (M.Data[5] * V.Y) + (M.Data[9]  * V.Z);	// Rotate Around The Y Axis
	D.Z = (M.Data[2] * V.X) + (M.Data[6] * V.Y) + (M.Data[10] * V.Z);	// Rotate Around The Z Axis

The first major function of the engine... Initialize, does exactly what is says. I've cut out a few lines of code as they are not needed in the explanation.

// Any GL Init Code & User Initialization Goes Here
BOOL Initialize (GL_Window* window, Keys* keys)

These 3 variables are used to load the shader file. Line contains space for a single line in the text file, while shaderData stores the actual shader values. You may be wondering why we have 96 values instead of 32. Well, we need to convert the greyscale values to RGB so that OpenGL can use them. We can still store the values as greyscale, but we will simply use the same value for the R, G and B components when uploading the texture.

	char Line[255];						// Storage For 255 Characters
	float shaderData[32][3];				// Storage For The 96 Shader Values

	FILE *In = NULL;					// File Pointer

When drawing the lines, we want to make sure that they are nice and smooth. Initially this value is turned off, but by pressing the "2" key, it can be toggled on/off.

	glShadeModel (GL_SMOOTH);				// Enables Smooth Color Shading
	glDisable (GL_LINE_SMOOTH);				// Initially Disable Line Smoothing

	glEnable (GL_CULL_FACE);				// Enable OpenGL Face Culling

We disable OpenGL lighting because we do all of the lighting calculations ourself.

	glDisable (GL_LIGHTING);				// Disable OpenGL Lighting

Here is where we load the shader file. It is simply 32 floating point values stored as ASCII (for easy modification), each one on a seperate line.

	In = fopen ("Data\\shader.txt", "r");			// Open The Shader File

	if (In)							// Check To See If The File Opened
		for (i = 0; i < 32; i++)			// Loop Though The 32 Greyscale Values
			if (feof (In))				// Check For The End Of The File

			fgets (Line, 255, In);			// Get The Current Line

Here we convert the greyscale value into RGB, as described above.

			// Copy Over The Value
			shaderData[i][0] = shaderData[i][1] = shaderData[i][2] = atof (Line);

		fclose (In);					// Close The File

		return FALSE;					// It Went Horribly Horribly Wrong

Now we upload the texture. As it clearly states, do not use any kind of filtering on the texture or else it will look odd, to say the least. GL_TEXTURE_1D is used because it is a 1D array of values.

	glGenTextures (1, &shaderTexture[0]);			// Get A Free Texture ID

	glBindTexture (GL_TEXTURE_1D, shaderTexture[0]);	// Bind This Texture. From Now On It Will Be 1D

	// For Crying Out Loud Don't Let OpenGL Use Bi/Trilinear Filtering!

	// Upload
	glTexImage1D (GL_TEXTURE_1D, 0, GL_RGB, 32, 0, GL_RGB , GL_FLOAT, shaderData);

Now set the lighting direction. I've got it pointing down positive Z, which means it's going to hit the model face-on.

	lightAngle.X = 0.0f;					// Set The X Direction
	lightAngle.Y = 0.0f;					// Set The Y Direction
	lightAngle.Z = 1.0f;					// Set The Z Direction

	Normalize (lightAngle);					// Normalize The Light Direction

Load in the mesh from file (described above).

	return ReadMesh ();					// Return The Value Of ReadMesh

The opposite of the above function... Deinitialize, deletes the texture and polygon data created by Initalize and ReadMesh.

void Deinitialize (void)					// Any User DeInitialization Goes Here
	glDeleteTextures (1, &shaderTexture[0]);		// Delete The Shader Texture

	delete [] polyData;					// Delete The Polygon Data

The main demo loop. All this does is process the input and update the angle. Controls are as follows:

= Toggle rotation

1 = Toggle outline drawing

2 = Toggle outline anti-aliasing

= Increase line width

= Decrease line width

void Update (DWORD milliseconds)				// Perform Motion Updates Here
	if (g_keys->keyDown [' '] == TRUE)			// Is the Space Bar Being Pressed?
		modelRotate = !modelRotate;			// Toggle Model Rotation On/Off

		g_keys->keyDown [' '] = FALSE;

	if (g_keys->keyDown ['1'] == TRUE)			// Is The Number 1 Being Pressed?
		outlineDraw = !outlineDraw;			// Toggle Outline Drawing On/Off

		g_keys->keyDown ['1'] = FALSE;

	if (g_keys->keyDown ['2'] == TRUE)			// Is The Number 2 Being Pressed?
		outlineSmooth = !outlineSmooth;			// Toggle Anti-Aliasing On/Off

		g_keys->keyDown ['2'] = FALSE;

	if (g_keys->keyDown [VK_UP] == TRUE)			// Is The Up Arrow Being Pressed?
		outlineWidth++;					// Increase Line Width

		g_keys->keyDown [VK_UP] = FALSE;

	if (g_keys->keyDown [VK_DOWN] == TRUE)			// Is The Down Arrow Being Pressed?
		outlineWidth--;					// Decrease Line Width

		g_keys->keyDown [VK_DOWN] = FALSE;

	if (modelRotate)					// Check To See If Rotation Is Enabled
		modelAngle += (float) (milliseconds) / 10.0f;	// Update Angle Based On The Clock

The function you've all been waiting for. The Draw function does everything - calculates the shade values, renders the mesh, renders the outline, and, well that's it really.

void Draw (void)

TmpShade is used to store the shader value for the current vertex. All vertex data is calculated at the same time, meaning that we only need to use a single variable that we can just keep reusing.

The TmpMatrix, TmpVector and TmpNormal structures are also used to calculate the vertex data. TmpMatrix is set once at the start of the function and never changed until Draw is called again. TmpVector and TmpNormal on the other hand, change when another vertex is processed.

	float TmpShade;						// Temporary Shader Value

	MATRIX TmpMatrix;					// Temporary MATRIX Structure
	VECTOR TmpVector, TmpNormal;				// Temporary VECTOR Structures

Let's clear the buffers and matrix data.

	glClear (GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);	// Clear The Buffers
	glLoadIdentity ();					// Reset The Matrix

The first check is to see if we want to have smooth outlines. If so, then we turn on anti-alaising. If not, we turn it off. Simple!

	if (outlineSmooth)					// Check To See If We Want Anti-Aliased Lines
		glHint (GL_LINE_SMOOTH_HINT, GL_NICEST);	// Use The Good Calculations
		glEnable (GL_LINE_SMOOTH);			// Enable Anti-Aliasing

	else							// We Don't Want Smooth Lines
		glDisable (GL_LINE_SMOOTH);			// Disable Anti-Aliasing

We then setup the viewport. We move the camera back 2 units, and then rotate the model by the angle. Note: because we moved the camera first, the model will rotate on the spot. If we did it the other way around, the model would rotate around the camera.

We then grab the newly created matrix from OpenGL and store it in TmpMatrix.

	glTranslatef (0.0f, 0.0f, -2.0f);			// Move 2 Units Away From The Screen
	glRotatef (modelAngle, 0.0f, 1.0f, 0.0f);		// Rotate The Model On It's Y-Axis

	glGetFloatv (GL_MODELVIEW_MATRIX, TmpMatrix.Data);	// Get The Generated Matrix

The magic begins. We first enable 1D texturing, and then enable the shader texture. This is to be used as a look-up table by OpenGL. We then set the color of the model (white). I chose white because it shows up the highlights and shading much better then other colors. I suggest that you don't use black :)

	// Cel-Shading Code
	glEnable (GL_TEXTURE_1D);				// Enable 1D Texturing
	glBindTexture (GL_TEXTURE_1D, shaderTexture[0]);	// Bind Our Texture

	glColor3f (1.0f, 1.0f, 1.0f);				// Set The Color Of The Model

Now we start drawing the triangles. We look though each polygon in the array, and then in turn each of it's vertexes. The first step is to copy the normal information into a temporary structure. This is so we can rotate the normals, but still keep the original values preserved (no precision degradation).

	glBegin (GL_TRIANGLES);					// Tell OpenGL That We're Drawing Triangles

		for (i = 0; i < polyNum; i++)			// Loop Through Each Polygon
			for (j = 0; j < 3; j++)			// Loop Through Each Vertex
				TmpNormal.X = polyData[i].Verts[j].Nor.X;	// Fill Up The TmpNormal Structure With The
				TmpNormal.Y = polyData[i].Verts[j].Nor.Y;	// Current Vertices' Normal Values
				TmpNormal.Z = polyData[i].Verts[j].Nor.Z;

Second, we rotate the normal by the matrix grabbed from OpenGL earlier. We then normalize this so it doesn't go all screwy.

				// Rotate This By The Matrix
				RotateVector (TmpMatrix, TmpNormal, TmpVector);

				Normalize (TmpVector);		// Normalize The New Normal

Third, we get the dot product of the rotated normal and light direction (called lightAngle, because I forgot to change it from my old light class). We then clamp the value to the range 0-1 (from -1 to +1).

				// Calculate The Shade Value
				TmpShade = DotProduct (TmpVector, lightAngle);

				if (TmpShade < 0.0f)
					TmpShade = 0.0f;	// Clamp The Value to 0 If Negative

Forth, we pass this value to OpenGL as the texture coordinate. The shader texture acts as a lookup table (the shader value being the index), which is (I think) the main reason why 1D textures were invented. We then pass the vertices position to OpenGL, and repeat. And Repeat. And Repeat. And I think you get the idea.

				glTexCoord1f (TmpShade);	// Set The Texture Co-ordinate As The Shade Value
				// Send The Vertices
				glVertex3fv (&polyData[i].Verts[j].Pos.X);

	glEnd ();						// Tell OpenGL To Finish Drawing

	glDisable (GL_TEXTURE_1D);				// Disable 1D Textures

Now we move onto the outlines. An outline can be defined as "an edge where one polygon is front facing, and the other is backfacing". In OpenGL, it's where the depth test is set to less than or equal to (GL_LEQUAL) the current value, and when all front faces are being culled. We also blend the lines in, to make it look nice :)

So, we enable blending and set the blend mode. We tell OpenGL to render backfacing polygons as lines, and set the width of those lines. We cull all front facing polygons, and set the depth test to less than or equal to the current Z value. After this the color of the line is set, and we loop through each polygon, drawing it's vertices. We only need to pass the vertex position, and not the normal or shade value because all we want is an outline.

	// Outline Code
	if (outlineDraw)					// Check To See If We Want To Draw The Outline
		glEnable (GL_BLEND);				// Enable Blending
		// Set The Blend Mode		

		glPolygonMode (GL_BACK, GL_LINE);		// Draw Backfacing Polygons As Wireframes
		glLineWidth (outlineWidth);			// Set The Line Width

		glCullFace (GL_FRONT);				// Don't Draw Any Front-Facing Polygons

		glDepthFunc (GL_LEQUAL);			// Change The Depth Mode

		glColor3fv (&outlineColor[0]);			// Set The Outline Color

		glBegin (GL_TRIANGLES);				// Tell OpenGL What We Want To Draw

			for (i = 0; i < polyNum; i++)		// Loop Through Each Polygon
				for (j = 0; j < 3; j++)		// Loop Through Each Vertice
					// Send The Vertices
					glVertex3fv (&polyData[i].Verts[j].Pos.X);

		glEnd ();					// Tell OpenGL We've Finished

After this, we just set everything back to how it was before, and exit.

		glDepthFunc (GL_LESS);				// Reset The Depth-Testing Mode

		glCullFace (GL_BACK);				// Reset The Face To Be Culled

		glPolygonMode (GL_BACK, GL_FILL);		// Reset Back-Facing Polygon Drawing Mode

		glDisable (GL_BLEND);				// Disable Blending

You see now Cel-Shading isn't that difficult. Of course the techniques could be enhanced a lot. A good example is the game XIII /data/lessons/http://www.nvidia.com/object/game_xiii.html, which makes you think you are in a cartoon world. If you want to get deeper into cartoon rendering techniques, you could look into the book Real-time Rendering (Möller, Haines) on the chapter "Non-Photorealistic Rendering". If you prefer reading articles from the web, a huge link list can be found here: /data/lessons/http://www.red3d.com/cwr/npr/

Sami Hamlaoui (MENTAL)

Jeff Molofee (NeHe)

* DOWNLOAD Visual C++ Code For This Lesson.

* DOWNLOAD Borland C++ Builder 6 Code For This Lesson. ( Conversion by Christian Kindahl )
* DOWNLOAD Code Warrior 5.3 Code For This Lesson. ( Conversion by Scott Lupton )
* DOWNLOAD Delphi Code For This Lesson. ( Conversion by Michal Tucek )
* DOWNLOAD Dev C++ Code For This Lesson. ( Conversion by Warren Moore )
* DOWNLOAD Euphoria Code For This Lesson. ( Conversion by Evan Marshall )
* DOWNLOAD JoGL Code For This Lesson. ( Conversion by Abdul Bezrati )
* DOWNLOAD Linux / GLut Code For This Lesson. ( Conversion by Kah )
* DOWNLOAD Linux/GLX Code For This Lesson. ( Conversion by Patrick Schubert )
* DOWNLOAD Linux/SDL Code For This Lesson. ( Conversion by Sean Farrell )
* DOWNLOAD Mac OS X/Cocoa Code For This Lesson. ( Conversion by Bryan Blackburn )
* DOWNLOAD Visual Studio .NET Code For This Lesson. ( Conversion by Grant James )


< Lesson 36Lesson 38 >