openGL shaders, vertex attributes, fbos, vbos etc...

Today I continued working on a little project for a demonstration I want to give and I needed a place to store everything I read as a reference, so here we go...

This demonstration will show flying cards that you can kick around. I want these cards to have a nice bright shiny look so I implemented light rays using a shader. I'n this article I'll describe how I created the light rays, but first a bit about the physics I'm using.

After coding lots of physics simulations myself I decided to give Bullet Physics a try. Bullet Physics is used in a couple of big hollywood productions and commercial games and has proven to be a good library. The API is clear, well designed and there is good documentation. I'm not sure how active the community actually is but it seems to be ok. 

When I started using physics engines and 3D, long time ago, I was confused how they are used with things one draws on screen, like game characters.  Having done many Flash applications I was used to working with pixels. In 3D you don't really work with pixel units. You just use "units". For example, the boxes you see in the image below have a size of 0.4, 0.4, 0.01 (x,y,z) units. Because it's a 3D scene the  boxes, near are bigger than boxes which are further away. Well, when you work with a physics engine you tell the engine what kind of shapes you're using and the physics engine tells you where to position and orient these elements.

Debug render of bullet physics objects

The physics engine calculates the position and rotation of the shapes you added to the physics simulation and you use these to update the things you draw on screen. Bullet Physics has a great helper function called "getOpenGLMatrix" which gives you a matrix that you can use directly for each model.

But enough about the physics engine. As I told you, I'm writing this article so I have a reference for myself how to implement light rays, FBOs, VAOs, etc... There are lots of things you need to remember like enabling vertex attributes at the right time, drawing the VBO into a plane of the same size of the screen when you apply a effect with a shader. So the rest of this article describes how I created the light rays based on the article of Fabien Sanglard. 

This image below shows you the physics simulation to which I applied a texture and the light rays as described in the rest of this article and on Fabiens page.

The source which shows how I created this with a couple of high level wrappers around shaders, vbos, fbos, textures etc... can be found here.

The first step is to create a buffer for the vertices and texture coordinates (not shown in the example code) you want to send to your GPU and use in your shader. Old school openGL uses glBegin(), glVertex3f(), glTexCoord() etc... Although these functions are fantastic to use when debugging, they are also incredibly slow. For each position, texture or color you need a function call. To improve performance you simply upload all your vertex data once when your initialize the application. Then you only tell openGL where to draw and what to draw. This code shows you how to create a simple quad and uses immediate mode (glBegin() etc..) to draw. 

When running this code I get something like the image below. I'm using my custom debug draw function which adds colors to vertices it draws. 

Okay so I'm drawing a quad to the screen using immediate mode. Next step is to move everything into a Vertex Buffer Object (VBO) and create a shader so I can apply the light ray effect. If you've never done anything with shaders or VBOs this can become a bit difficult especially because it's a lot to grasp at once.

But hang on, I'll describe each step to make it as clear as I can and will be simplyfing some openGL concepts. Look at a VBO as a big array with all the information you need to draw, like vertices, texture coordinates or normals. You only need 3 lines of code to create a VBO. First you ask openGL to generate a VBO, then you tell openGL what kind of buffer it is. You do this by binding the generated buffer with GL_ARRAY_BUFFER using glBindBuffer. Next you upload all data using glBufferData. See the openGL reference for what the parameters mean.

Next step is creating a shader so we can actually do something with this VBO. At this point I'll introduce a new concept: vertex attributes. A vertex attribute is something you can set for each vertex you want to draw. The quad in the example has a position and a color vertex attribute for each of the four vertices. Keep in mind that a vertex is not the same as a position. A vertex is drawn at a position and can have many other attributes.

In this example I also use a view and model matrix. Often the concepts of view and camera matrices are mixed up. There is a big difference. The view matrix is the inverse of the camera matrix. In this example I manipulate the view matrix by translating into the negative Z. By default, we look into the negatize Z axis in openGL.     

Now that we have our basic shader we need to change our code:

• load and compile the shaders
• add uniforms 
• enable vertex attrib arrays 
• tell opengl how to use the buffer and vertex attributes

I'm using a wrapper to load, compile and link the shaders. In the source you can see I'm load()ing the vertex and fragment shaders. I'm adding a uniform for the projection_matrix and view_matrix. Uniforms are a bit like vertex attributes. But as vertex attributes are used for each vertex an uniform is used for a group of vertices. The values of the uniforms in this shader are shared by the four vertices. You can see uniforms like "global" variables.

Next step is to enable the vertex attribute so openGL will provide your shader with attribute data from the VBO. The function glEnableVertexAttribArray is used for this. Then the last thing openGL needs to know is where it can find the data for this attribute in the VBO. Did you remember a VBO is just a big array with vertex information? Because you can store position, texture coordinates, colors etc.. all in one VBO you must explain how to map this array to the specific attributes in the shader. For this you use glVertexAttribPointer. I'm using offsetof to tell where in the data the attribute is found.

When you want to draw a VBO you use glDrawArrays. Before calling glDrawArrays you make sure that the the VBO is bound (1), the correct vertex attribute pointer information is set (2) and the vertex attribute array (3) is enabled. When using multiple VBOs, this means you have to call (1), (2) and (3) each time in your draw function (read more info about this). Check out the full source. 

Ok great you made it up to this point (O_o). Now we're getting to the interesting part: vertex array objects (VAO) and the light ray shader. I'm going to speed up through the VAOs. A VAO is a way for openGL to remember the state of vertex attributes: if they are're enabled and how the VBO data maps to specific vertex attributes. openGL.org has a very good wiki page about VAOs.  Using a VAO couldn't be easier and you can check out this code to see how we use the VAO:

• Create a new Vertex Array Object
• Bind the VAO before specifying the attributes

Okay now that we explained how VAOs work we can start looking at the light rays. These light rays are based on this excellent article by Fabien Sanglard. Read his article to see how this works, I'm describing the separate stages here.

First step is rendering the scene into a texture using a FBO and then draw the texture onto a quad which is the same size of the window and "stretch" the pixels to get the light rays. I'm using a wrapper class for the FBO but there are plenty of articles desribing how to use FBOs. We rendered the quad you see in the image above into this texture. To use this texture we need to change the shader and buffer a bit. We need texture coordinates and a sampler2D so we can use the texture in our shader.

Great, now that we have a FBO into which we draw the scene (that pink quad) and we're able to draw the "render to texture" result into another quad.   

Finally, let us add the light ray shader code from Fabien. This shader basically stretches the pixels of the texture. It uses texture coordinates and a light position to create a vector pointing from the  "fictional" light source to the current texture coordinate (deltaTexCoord). Then it smudges the texture along this vector. I'm saying vector, but in practice they're texture coordinates.

To add the light rays we have to add the uniforms to our shader and set the values. I downloaded Fabiens' source code and got the values from the file "Engine.cpp".

An important point to remember here is the way we use the FBO. We used the FBO to render the scene into a texture. This texture contains exactly the same thing that you see on the screen when we didn't use it. When we want to use this texture and render it on a quad in such a way that it shows exactly the same thing as the scene which was rendered we need to create a quad with the same size as the window. There is a nice trick to do this. You use an identity matrix for your projection matrix and a identity matrix for the view matrix. Then you render a quad with normalized device context sizes, which means a quad with top left (-1,-1) and bottom right (1,1). 

After rendering the original scene over the light rays we're done! You can see the complete source code at this gist. When running the application we get something like this:

openGL vertexAttributes

Whenever I use raw openGL I've to google around to figure out when to call which function so the openGL states are correctly set. This morning I was trying to get a shader and FBO to work for which I'm using two VBOs and two VAOs.  There is more to it, but a VBO is a big array you can send to your GPU at once. Together with vertex attributes which you define in your shader you draw to your screen.  openGL needs to know how to map your VBO (read array) to vertex attributes.  You do this by first with the following steps which must be performed for each VBO:

• Bind your VBO 
• call glEnableVertexAttribArray()
• call glVertexAttribPointer()

Caching generated images with PHP

For a project we need to perform some image operations on lots of dynamically generated images and then serve these images to thousands of people. For this I had to find a way to minimize server load and traffic.  Somehow caching, which seems like a trivial task, means lots of reading because most information you'll find when googling isn't correct.  Most people mix up headers, use wrong date formats, etc..  I found one correct way to enable client-side caching using the Cache-control header. 

/*
 * Send an png file to the browser enabling client-side cache which works on:
 * Mac OSX:  Firefox 9.01,  Safari 5.1.2, Opera 11.5, Chrome 16.0.9
 * Windows: IE 9 (8,7 not tested), Chrome 16.0.9, Firefox 7.0, Firefox 9.01
 * Does not work on Windows Safari 5.01
 */
function sendFileToBrowser($imageFile) {
	header('Content-type: image/png');
	header('Content-transfer-encoding: binary');
	header('Content-length: '.filesize($imageFile));
	header('Cache-control: public, max-age=30');
	readfile($imageFile);
	exit;
}

I created this function to send png images which are generated when necessary back to the user with the correct headers. I tested if the caching worked by monitoring the HTTP response and requests to my apache server. When you test this yourself, do not use CTRL+R (Windows), CMD+R (Mac) to reload the page as this invalidates the cache. You should open a new tab/window and enter the URL there. Only Safari 5.0 on Windows seems to be having problems with this.  In short when you need to cache generated images (by i.e. PHP) you only need to send the Cache-control header.  You can set some flags in this header, like public, which means that clients/browsers must also cache the data for secure pages (https).  The must-revalidate is tricky with safari; it doesn't seem to work. Check the specs for more information about this.

Matrix & Quaternion FAQ

Just stumbled upon this amazing FAQ on matrices and quaternions. (Got a backup here)

Mushrooms

Working on something I've been wanted to created for a long time now.... Using openFrameworks I'm generating "mushrooms" for which I can tweak the width, size, roundness etc... I'm positioning the mushrooms randomly using basic dart throwing.  I've exported the generated meshes to the .r3f format and imported it into Blender.

Top view of openFrameworks app which I used to generated some mushrooms. The colors are representing the vertex orders which I use when debugging. 

This image above is rendered in Blender and the bottoms sides of the mushrooms have a emission material applied to them.  I'm trying to give this a bit more a mushroomish look but for now Cycles has no subsurface scattering which I could really use for the mushroom material.

Render using basic diffuse material and sky lighting. I always like this kind of simple look but it's kind of missing lights to add some ambience. 

Normal mapping

Just some thoughts for myself as a reminder on how to work with normalmapping and shaders

• When you use varying variables in your shader (i.e. normals: binormal, tangent, normal) make sure to normalize them in your fragment shader as varyings are interpolated between vertex and fragment shader which mess up normalization done in the vertex shader.
• I found it easiest to debug/test my shaders by not using a diffuse texture but just using a very dark color for the diffuse part. I.e. 0.01 for RGB. 
• Also, use a normal map which does not have very fine details but use a coarse one
• I was using the shaders below though I still have some questions like: do I need to normalize the tangent and bitangent for the tangent matrix; it doesnt seem to matter though. Also I had to put the lights at a negative z position (-50), when using positive z values I didn't see any lighting. This can be due to the light direction vector... this was exactly the case; I had to transform the light position into tangent space as well.
• Also use at least two lights which makes testing the normalmap a lot clearer
• Use a hight specularity value so it has a sharp fallof (50-128)
• Use a moving light so you can better see if shading works; sometimes it's difficult to notice shading changes because they are minimal

Vertex shader

#define LIGHT_COUNT 2
 
attribute vec4 pos; 
attribute vec2 tex;
attribute vec3 norm;
attribute vec4 tan;
 
uniform mat4 modelview;
uniform mat4 projection;
uniform mat4 modelview_projection;
 
struct Light {
	vec3 position;
	vec4 diffuse_color;
	vec4 specular_color;
};
 
uniform Light lights[LIGHT_COUNT];
 
varying vec2 texcoord;
varying vec3 normal;
varying vec3 tangent;
varying vec3 binormal; 
varying vec3 light_directions[LIGHT_COUNT];
varying vec3 eye_position;
varying vec3 eye_normal; 
 
void main() {
	gl_Position = modelview_projection * pos;
	eye_position = vec3(modelview * pos);
	texcoord = tex;
	normal = norm;
	eye_normal = normalize(vec3(modelview * vec4(normal,0.0)));
	tangent = normalize(tan.xyz);
	binormal = cross(eye_normal, tangent) * tan.w;
	binormal = normalize(binormal);
	mat3 tangent_space = mat3(tangent, binormal, eye_normal); 
 
	for(int i = 0; i < LIGHT_COUNT; ++i) { 
		vec3 lp = (vec3(modelview * vec4(lights[i].position, 0.0)));
		light_directions[i] = normalize(tangent_space * (lp - eye_position)); 
	}
}

Fragment shader

#define LIGHT_COUNT 2
 
uniform sampler2D diffuse_texture; 
uniform sampler2D normal_texture; 
 
struct Light {
	vec3 position;
	vec4 diffuse_color;
	vec4 specular_color;
};
uniform Light lights[LIGHT_COUNT];
 
varying vec3 eye_position;
varying vec3 normal;
varying vec3 eye_normal;
varying vec2 texcoord;
varying vec3 tangent;
varying vec3 binormal;
varying vec3 light_directions[LIGHT_COUNT];
 
void main() {
	vec4 texel_color = vec4(0.0, 0.0, 0.0, 1.0);
	vec3 final_normal = normalize(eye_normal);
	vec3 normal_color = texture2D(normal_texture, texcoord).xyz * 2.0 - 1.0;
	normal_color = normalize(normal_color);
	mat3 tangent_matrix = mat3(tangent, binormal, final_normal);
	final_normal = normalize(tangent_matrix * normal_color);	vec4 diffuse_color = texture2D(diffuse_texture, texcoord);
	texel_color += diffuse_color;
	for(int i = 0; i < LIGHT_COUNT; ++i) { 
		float n_dot_l = max(dot(normalize(final_normal), light_directions[i]), 0.0); 
		if(n_dot_l > 0.0) {
			texel_color += ( 0.4*(n_dot_l * lights[i].diffuse_color));
			vec3 reflection = normalize(reflect(-normalize(light_directions[i]), final_normal));
			float spec = max(0.0, dot(final_normal, reflection)); 
			float fspec = pow(spec, 352.0);
			texel_color += (fspec * lights[i].specular_color)  ;
		}
	}
	gl_FragColor = texel_color;
}

Streaming video with openFrameworks

During the Art & Code 3D conference in Pittsburg we've been working on a streaming video implementation. Arturo was able to get this to work using GStreamer on Linux. I was already working on an application using the Ogg format (using theora to encode the images).  Finally after a couple of days I got my first video stream. I just updated the ofxOgg github repository with a working version which can stream and record your openFrameworks application into ogg.   This addon is using only the ogg and theora libraries. You do not need to install any other application.  There is one issue with the addon when you open the video stream in Chrome...i'm working on this.

Bare bones openGL in openFrameworks

During our last openFrameworks freakDay meeting we discussed the usage of VBOs, vertex attributes and shaders in openFrameworks.  To explain the openGL concepts I created a demo application and shader which shows how to use openGL only without the need of ofShader, ofVBO.

// Error.h
/*
**  error.h
**
**  Copyright (c) 2008 Max Rupp (feelgood@cs.pdx.edu) All rights reserved.
*/
 
#ifndef __ERROR_H__
#define __ERROR_H__
#define DEBUG
#ifdef  DEBUG
 
#include <stdlib.h>
#include <assert.h>
/*
#define eglGetError( )\
        {\
            for ( GLenum Error = glGetError( ); ( GL_NO_ERROR != Error ); Error = glGetError( ) )\
            {\
                switch ( Error )\
                {\
                    case GL_INVALID_ENUM:      printf( "\n%s\n\n", "GL_INVALID_ENUM"      ); assert( 0 ); break;\
                    case GL_INVALID_VALUE:     printf( "\n%s\n\n", "GL_INVALID_VALUE"     ); assert( 0 ); break;\
                    case GL_INVALID_OPERATION: printf( "\n%s\n\n", "GL_INVALID_OPERATION" ); assert( 0 ); break;\
                    case GL_STACK_OVERFLOW:    printf( "\n%s\n\n", "GL_STACK_OVERFLOW"    ); assert( 0 ); break;\
                    case GL_STACK_UNDERFLOW:   printf( "\n%s\n\n", "GL_STACK_UNDERFLOW"   ); assert( 0 ); break;\
                    case GL_OUT_OF_MEMORY:     printf( "\n%s\n\n", "GL_OUT_OF_MEMORY"     ); assert( 0 ); break;\
                    default:                                                                              break;\
                }\
            }\
        }
*/
#define eglGetError()
#define eglCheckFramebufferStatus( )\
        {\
            switch ( glCheckFramebufferStatus( GL_FRAMEBUFFER ) )\
            {\
                case GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT:         printf( "\n%s\n\n", "GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT"         ); assert( 0 ); break;\
                case GL_FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT: printf( "\n%s\n\n", "GL_FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT" ); assert( 0 ); break;\
                case GL_FRAMEBUFFER_INCOMPLETE_DRAW_BUFFER:        printf( "\n%s\n\n", "GL_FRAMEBUFFER_INCOMPLETE_DRAW_BUFFER"        ); assert( 0 ); break;\
                case GL_FRAMEBUFFER_INCOMPLETE_READ_BUFFER:        printf( "\n%s\n\n", "GL_FRAMEBUFFER_INCOMPLETE_READ_BUFFER"        ); assert( 0 ); break;\
                case GL_FRAMEBUFFER_UNSUPPORTED:                   printf( "\n%s\n\n", "GL_FRAMEBUFFER_UNSUPPORTED"                   ); assert( 0 ); break;\
                case GL_FRAMEBUFFER_INCOMPLETE_MULTISAMPLE:        printf( "\n%s\n\n", "GL_FRAMEBUFFER_INCOMPLETE_MULTISAMPLE"        ); assert( 0 ); break;\
                case GL_FRAMEBUFFER_UNDEFINED:                     printf( "\n%s\n\n", "GL_FRAMEBUFFER_UNDEFINED"                     ); assert( 0 ); break;\
                default:                                                                                                                              break;\
            }\
        }
 
#define eglGetShaderInfoLog( Shader )\
        {\
            GLint   Status, Count;\
            GLchar *Error;\
            \
            glGetShaderiv( Shader, GL_COMPILE_STATUS, &Status );\
            \
            if ( !Status )\
            {\
                glGetShaderiv( Shader, GL_INFO_LOG_LENGTH, &Count );\
                \
                if ( Count > 0 )\
                {\
					Error = (GLchar *)malloc(Count);\
					glGetShaderInfoLog( Shader, Count, NULL, Error );\
                    \
                    printf( "%s\n", Error );\
                    \
                    free( Error );\
                    \
                    assert( 0 );\
                }\
            }\
        }
 
#else
 
#define eglGetError( )
 
#define eglCheckFramebufferStatus( )
 
#define eglGetShaderInfoLog( Shader )
 
#endif /*   DEBUG     */
#endif /* __ERROR_H__ */

// testApp.h
#pragma once
 
#include "ofMain.h"
#include "Error.h"
 
class testApp : public ofBaseApp{
 
	public:
		void setup();
		void update();
		void draw();
 
		void keyPressed  (int key);
		void keyReleased(int key);
		void mouseMoved(int x, int y );
		void mouseDragged(int x, int y, int button);
		void mousePressed(int x, int y, int button);
		void mouseReleased(int x, int y, int button);
		void windowResized(int w, int h);
		void dragEvent(ofDragInfo dragInfo);
		void gotMessage(ofMessage msg);
 
		// shader 
		GLuint shader_vertex;
		GLuint shader_fragment;
		GLuint prog;
		GLint pos_attrib;
		GLint col_attrib;
		GLint custom_attrib;
 
		// vbo
		GLuint buffer;
		string readFile(string file);
 
};

// testApp.cpp
#include "testApp.h"
#include <stdlib.h>
#include <fstream>
 
struct VertexData {
	ofVec3f pos;
	ofVec3f col;
	float custom;
};
 
//--------------------------------------------------------------
void testApp::setup(){
	ofBackground(33);
 
	// Compile and link shaders into a program.
	shader_vertex = glCreateShader(GL_VERTEX_SHADER); eglGetError();
	shader_fragment = glCreateShader(GL_FRAGMENT_SHADER); eglGetError();
 
	string vert_source = readFile("program.vert");
	string frag_source = readFile("program.frag");
	const char* vert_ptr = vert_source.c_str();
	const char* frag_ptr = frag_source.c_str();
	glShaderSource(shader_vertex, 1, &vert_ptr, NULL); eglGetError();
	glShaderSource(shader_fragment, 1, &frag_ptr, NULL); eglGetError();
 
	glCompileShader(shader_vertex); eglGetError();
	glCompileShader(shader_fragment); eglGetError();
 
	prog = glCreateProgram(); eglGetError();
	glAttachShader(prog, shader_vertex); eglGetError();
	glAttachShader(prog, shader_fragment); eglGetError();
	glLinkProgram(prog); eglGetError();
	glUseProgram(prog); eglGetError();
 
	// create buffer
	float s = 4;
	VertexData points[4];
	points[0].pos.set(-s, s, 0);
	points[1].pos.set(s,s, 0);
	points[2].pos.set(s,-s,0);
	points[3].pos.set(-s, -s, 0);
	points[0].col.set(1,0,0);
	points[1].col.set(0,1,0);
	points[2].col.set(0,0,1);
	points[3].col.set(1,0,1);
	points[0].custom = 0.5;
	points[1].custom = 1.0;
	points[2].custom = 1.0;
	points[3].custom = 0.8;		
	glGenBuffers(1, &buffer); eglGetError();
	glBindBuffer(GL_ARRAY_BUFFER, buffer); eglGetError();
	glBufferData(GL_ARRAY_BUFFER, sizeof(VertexData)* 4, &points[0].pos.x, GL_DYNAMIC_DRAW); eglGetError();
 
	// enable vertex attributes.
	pos_attrib = glGetAttribLocation(prog, "pos"); eglGetError();
	col_attrib = glGetAttribLocation(prog, "col"); eglGetError();
	custom_attrib = glGetAttribLocation(prog, "custom"); eglGetError(); 
 
	glEnableVertexAttribArray(pos_attrib); eglGetError();
	glEnableVertexAttribArray(col_attrib); eglGetError();
	glEnableVertexAttribArray(custom_attrib); eglGetError();
 
	// where can opengl find the values in the buffer?
	glVertexAttribPointer(pos_attrib, 3, GL_FLOAT, GL_FALSE, sizeof(VertexData), (GLvoid*)offsetof(VertexData, pos)); eglGetError();
	glVertexAttribPointer(col_attrib, 3, GL_FLOAT, GL_FALSE, sizeof(VertexData), (GLvoid*) offsetof(VertexData, col)); eglGetError();
	glVertexAttribPointer(custom_attrib, 1, GL_FLOAT, GL_FALSE, sizeof(VertexData), (GLvoid*) offsetof(VertexData, custom)); eglGetError();
}
 
string testApp::readFile(string fileName) {
	stringstream ss;
	ifstream frag_file;
	frag_file.open(ofToDataPath(fileName.c_str(),true).c_str());
	if(!frag_file.is_open()) {
		printf("Cannot open vertex shader.\n");
		::exit(1);
	}
	string line;
	while(getline(frag_file, line)) {
		ss << line << "\n";
	}
	return ss.str();
}
 
//--------------------------------------------------------------
void testApp::update(){
 
}
 
//--------------------------------------------------------------
void testApp::draw(){
	// reset gl.
	glMatrixMode(GL_PROJECTION);
	glLoadIdentity();
	gluPerspective(45, ofGetWidth()/ofGetHeight(), 0.1, 1000);
 
	glMatrixMode(GL_MODELVIEW);
	glLoadIdentity();
	glTranslatef(0,0,-10);
 
	// and draw our vbo
	glBindBuffer(GL_ARRAY_BUFFER, buffer); eglGetError();
	glDrawArrays(GL_QUADS, 0, 4); eglGetError();
}
 
//--------------------------------------------------------------
void testApp::keyPressed(int key){
	if(key == ' ') {	
		float s = 4;
		float t = (ofGetElapsedTimef());
		float v = (1.0+sin(t)*0.5);
		VertexData points[4];
		points[0].pos.set(-s, s, 0);
		points[1].pos.set(s,s, 0);
		points[2].pos.set(s,-s,0);
		points[3].pos.set(-s, -s, 0);
		points[0].col.set(v,0,0);
		points[1].col.set(0,v,0);
		points[2].col.set(0,0,v);
		points[3].col.set(v,0,v);
		points[0].custom = v;
		points[1].custom = 1.0 * v; 
		points[2].custom = 1.0 * v;
		points[3].custom = 0.8 * v;
		glBindBuffer(GL_ARRAY_BUFFER, buffer); eglGetError();
		glBufferData(GL_ARRAY_BUFFER, sizeof(VertexData)* 4, &points[0].pos.x, GL_DYNAMIC_DRAW); eglGetError();
	}
}

Art & Code 3D

Friday

Start of conference! We just got the "Live 3D Stream for Art and Code 3D, thanks to Golan Levin, arturoc, mr.doob, roxlu, binx, joelgethinlewis, all the Art+Code 3D Labbers, openFrameworks Community, Microsoft Research and Microsoft Kinect Team"  working now! Last week this amazing group of artists and hackers got live streaming of point cloud data working in the browser!  We're using gstreamer to together with openFrameworks to stream data through your browser with the help of the great guys over at www.giss.tv.  I've been working on a live stream addon which will stream directly to the web without the need of  gstreamer so you can start hacking yourself!

Websockets 

One of the first thoughts to get this live streaming of 3D point cloud data working was using websockets; a way to connect to an application over a network.  So an openFrameworks would stream the point clouds through a websocket into your browser.  Because websockets cannot do binary sockets this was to slow as all RGB data has to be converted using base64. 

Monday

Just started the dev week at the studio of creative inquiry! Hacking around with everyone.. and wow it's so amazing crazy nice here! One big laugh !  There are some really interesting concepts flying around here.  Everyone started coding and working on demos.

Kyle doing art&code

JGL....

Joshua showing compressed point cloid data

Matt working with his 3D printers

Sunday

Arrived at Pittsburgh International! Heading over to the Carnegie Mellon Uni together with @companje.  

BSD sockets on Mac

So now and then I need to have send some data from one application to another one. Having a simple test server around which you easily debug is very nice to have. Therefore I past this snippet of code which creates a tcp server which working on unix/linux. It's purpose is to be used as a snippet for a test server, not for a fully fledged server.  I'm working on some demo applications for a iPhone class I'll be teaching which involve communication with a server. Stay tuned for more info on this! I'm using the Poco threading classes (which you can remove if you want to)

LEDServer.h

#ifndef LEDSERVERH
#define LEDSERVERH
 
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/select.h>
#include <netdb.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include "Poco/Thread.h"
#include <vector>
#include <map>
 
using std::vector;
using std::map;
class LEDClient;
class LEDServer : public Poco::Runnable {
public:
	LEDServer();
	~LEDServer();
	void start();
	void run();
private:
	int server_socket;
	int server_port;
	struct sockaddr_in server_addr;
	Poco::Thread thread;
	map<int, LEDClient*> clients;
};
#endif

LEDServer.cpp

#include "LEDServer.h"
#include "LEDClient.h"
 
LEDServer::LEDServer() 
:server_socket(0)
,server_port(8888)
{
}
 
LEDServer::~LEDServer() {
	thread.join();
}
 
void LEDServer::start() {
	thread.start(*this);
}
 
void LEDServer::run() {
	server_socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
	if(server_socket == -1) {
		printf("Can't create server socket\n");
	}	
 
	int result = 0;
	int val;
	result = setsockopt(server_socket, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val));
	if(result < 0) {
		printf("Cannot reuse address\n");
		exit(1);
	}
 
	// bind.	
	bzero(&server_addr, sizeof(server_addr));
	server_addr.sin_family = AF_INET;
	server_addr.sin_addr.s_addr = htonl(INADDR_ANY);
	server_addr.sin_port = htons(8888);
	result = bind(server_socket, (struct sockaddr*)&server_addr, sizeof(server_addr));
	if(result != 0) {
		printf("Cannot bind...\n");
		exit(1);
	}
 
	// listen
	result = listen(server_socket, 5);
	if(result < 0) {
		printf("Cannot set listen mode to socket\n");
		close(server_socket);
		exit(1);
	}
 
	// some sets
	fd_set socket_readset, test_set;
	FD_ZERO(&socket_readset);
	FD_SET(server_socket, &socket_readset);
 
 
 
	while(true) {
		// get notified by readset.
		test_set = socket_readset;
		result = select(FD_SETSIZE, &test_set, NULL, NULL, NULL);
		if(result < 1) {
			printf("Server error\n");
			exit(1);
		}
 
		for(int i = 0; i < FD_SETSIZE; ++i) {
			if(FD_ISSET(i, &test_set)) {
				if(i == server_socket) {
					// listen for a new connection.
					struct sockaddr_in client_addr = {0};
					int child_socket = 0; 
					int client_addr_len = sizeof(client_addr);
					child_socket = accept(server_socket, (struct sockaddr*)&client_addr, (socklen_t*) &client_addr_len);
 
					if(child_socket == -1) {
						printf("Cannot accept connections\n");
						close(server_socket);
						exit(1);
					}
 
					// store our new client.
					FD_SET(child_socket, &socket_readset);
					LEDClient* lc = new LEDClient(child_socket);
					clients.insert(std::pair<int,LEDClient*>(child_socket, lc));
				}	
				else {
					map<int, LEDClient*>::iterator it = clients.find(i);
					if(it != clients.end()) {
						it->second->readFromSocket();
					}					
				}
			}
		}
	}
 
	// clean up..
	map<int, LEDClient*>::iterator it = clients.begin();
	while(it != clients.end()) {
		delete it->second;
		++it;
	}
}

LEDClient.h

#ifndef LEDCLIENTH
#define LEDCLIENTH
 
 
#include <iostream>
#include <vector>
 
using namespace std;
 
class LEDClient {
public:
	LEDClient(int clientSock);
	void readFromSocket();
	int sock;
	vector<char> buffer;
};
 
#endif

LEDClient.cpp

#include "LEDClient.h"
LEDClient::LEDClient(int clientSock)
	:sock(clientSock)
{
}
 
void LEDClient::readFromSocket() {
	char tmp_buf[512];
	ssize_t bytes_read = read(sock, tmp_buf, 512);
	std::copy(tmp_buf, tmp_buf+bytes_read, std::back_inserter(buffer));
 
	// show what we got so far.
	cout << "Now in buffer (" << buffer.size() << ") = " ;
	for(int i = 0; i < buffer.size(); ++i) {
		cout << buffer[i];
	}
	cout << endl;
 
}

Face swap redux

After seeing the amazing stuff by Arturo and Kyle using the facetracking library of Jason Saragih and having a free midday in a long long time I let my vertices dance on the screen to create this... Next step is finishing up my hemesh implementation and apply surface smoothing... 

Communicating between Flash and openFrameworks with AMF

At Apollo Media we work (besides building websites) on interactive intallations for museums.  These projects often involve multiple computers, multiple operating systems and are always networked. For each project we decide what programming environment and language suits the needs of the specific project.  Mostly we have a mix between some c++ coded applications and flash. When we create C++ applications openFrameworks is a nice solution because it has lots and lots of very usefull addons, has a extremely active community with nice, knowledge sharing people.  

Last year we used several different  ways to communicate between the computers/applications we created.  We tried OSC (which is awesome btw), stdin techniques, custom sockets, xml sockets etc..  All these solutions are well suited for networked communication, but they all need extra code on both Flash and C++/openFrameworks sides.  

For this project I wanted a solution for which I did not need to use other libraries;  The right solution for this is using the serialization protocol which is standard in Flash, called AMF. This is a binary protocol with a tiny overhead for encoding different data types; it has some smart ways of not sending the same object, array or byte array multiple times over the socket.  Instead it uses references to already transfered objects, array and byte arrays. 

To make this possible I added ofxAMFServer to the ofxFlashCommunication addon which creates a fast, tiny and simple AMF remoting server. Again, as the same with ofxFlashCommunication you need to add three lines of code to setup this scalable, multi threaded server. When you connect to this ofxAMFServer from Flash, you'll receive an ofxAMFEvent in your application.  This event object has a member function called getParams() which contains the parameters sent to it from Flash.  

void testApp::onAMFEvent(ofxAMFEvent& ev) {
	cout << "Request uri:" << ev.getTargetURI() << endl;
	cout << "Params (JSON): " << ev.getParams().toJSON() << endl;
	cout << "Params (XML):" << endl << ev.getParams().toXML() << endl;
	last_amf_params = ev.getParams().toXML();
	cout << "Data from flash:" << (string)ev.getParams()["data"] << endl;
 
	Dictionary result;
	result["name"] = "Some result";
	ev.setReturnValues(result);
}

These parameters are stored in a new type I created in my ofxMissing addon. It's called "Dictionary" and its main purpose is to create a generic "multi purpose, multi value" variable. This Dictonary type can store strings, integers, arrays etc.. It can transform the data to JSON and XML. 

To return data back to your Flash application you need to call setReturnValues(Dictionary) on the ofxAMFEvent object. (note: ofxAMFServer is really young and there will probably change some things in the future as this way of using events in multi threaded applications can lead to problems). 

Have a look at the example code which starts a basic AMF server. Then use the flash demo/test application I added to the repository to send some data to the demo app.

NOTE: You need both ofxFlashCommunication and ofxMissing in your project.

Windows network card settings via script

Another day of windows reminds me how great Mac OSX is, even when it has it own peculiar things.  I had to set lots of IPs, switch them often etc.. so I used this script to automatically change the ip address of a network card.

netsh int ip set address name="Local Area connection" source=static addr=10.64.10.84 mask=255.255.255.0  gateway=10.64.10.1
netsh in ip set dns name="Local Area connection" source=static addr=10.64.0.1

Another thing which I had to do is start some applications in a sequential order. This script will first start a database server minimized, then causes a dely by using a ping. After that it starts my the desired application.

start /min /d "C:\Users\U\Desktop\APP_QR_SERVER\" qr_server.exe
ping -n 6 www.google.com
start /d "C:\Users\U\Desktop\APP_STREAMING\" streaming_app.exe

Remove permission popups in Windows Vista

So once in a while I have to work on a Windows Vista PC when porting applications we create at Apollo Media. Windows is fine, but one thing which is really, really, really annoying are those permission popups when you want to rename or copy something. Normally I can live with those popups but yesteday it got worse: I wasn't allowed to rename a directory (?!). So I asked google how to fix it... and yes this is the solution:

• Start > Control Panel
• Search for "User accounts"
• There you see a link at the bottom "Turn User Account control on or off", click on it
• Then make sure the checkbox there is unselected!
• Reboot... and you're happy

ofxKeylogger

For a project we're working on for the "Amsterdam Museum" with Apollo I had to catch system wide keypress events.  At my github page you can download the keylogger addon for openFrameworks which works both on Mac and Windows.  It does what I want but I'm aware that the windows version needs a bit work to be a fully capable keylogger as it's not yet figuring out what modifier keys are pressed. 

To test the ofxKeylogger addon on Mac, I created a simple application which counts how many keystrokers it receives. And wow this is crazy.. The count is on "22.201" at this moment which are the number of key strokes it received in about 4 hours.  This roughly means I press my keyboard 12.000.000 times a year? 

Custom application icon with XCode

For a project I had to figure out how one can use a custom application icon using openFrameworks and XCode. I created a keynote which explains step by step how to add a custom icon (which is pretty easy). I also describe how to create a custom application name. 

appname=PhotoUnit.app
if [ ! -d ] ; then 
	mkdir -p $TARGET_BUILD_DIR/${appname}
fi
 
cp -r $TARGET_BUILD_DIR/$PRODUCT_NAME.app $TARGET_BUILD_DIR/${appname}

Testing half edge noise fields modifier

After spending a whole sunday trying to compile and link against some libraries I had to see some colors ;). Just added a mesh generator to my half edge implementation for openFrameworks. I generate a plane and apply a new modifier called "noise" to it.  The noise modifier moves the positions of the vertices, based on simplex noise.

openFrameworks 007 presentations

After months and months of work it's finally there! The biggest release of openFrameworks so far! Great work everyone! Below are some of the presentations from our last meeting in Antwerp, with info on some of the new stuff in openFrameworks 007. The keynotes are not yet 100% ready and I see these as an ongoing work to create a set of keynotes which help people to understand the awesome world of creative programming. If you have additions to these presentations and/or suggestion feel free to contact me.

Half Edge Wowza!

Recently I met the bright and creative Jan Vantomme . He showed me some stuff using the Hemesh library for processing made by the friendly Frederik Vanhoutte (w:blut). After reading up on the core data structure which is used by Hemesh I picked up my project from a half year ago where I started implementing Half Edge. I've been interested in generative form for a while and used custom solutions to apply basic operations to meshes; but now, finally I implemented/ported most parts of the Hemesh library of Frederik.  Half Edge is actually a really nice, simple structure which makes it easy to perform operations on 3D meshes. Especially with the great, clean written Hemesh library I was ready to go in a couple of hours... and a sleepless saterday night.  As soon as I've got little more time.. (I'm in the middle of deadlines at the moment)... I will finish the library and will probably release it as a addon for openFameworks. 

Tron Tribute

This week I had some great discussions with Josh about the application he worked on for the effects in Tron using openFrameworks.  Yesterday I sat down to do some coding and trying to "rebuild" this amazingly beautifull effect.  I'm far from there, but it's kind of in the direction I want to go.  There are just a couple of screenshots form the application.