A little bit of colour

Today I decided to try out what would happen when projecting different colours onto different coloured material. Predictably, the only difference was a tint in the clothing, although I did learn that the camera picks up images much better on black material, but it means I will never be able to project certain levels of white onto the black material. Black material is very good for good black levels though, but bad for showing high levels of white. I don't think the different colours will be much as I can simulate the same levels through the projector anyway, and having everything projected onto it being tinted is a very big limitation.

Most projection mapping projects tend only use still objects like cars or buildings. In fact, all of the projection mapping I have seen will only use a still object. I imagine this is because it's so much harder to go to such an impressive level of detail on a moving object, since you'd have to make sure the object lines up perfectly in every single frame, which would be near impossible. This means that if I want to show off any fancy projection mapping, all the detailed stuff needs to be done at the beginning, when I've had plenty of prep time to get the object into exactly the right place it needs to be. After I move it, I obviously won't be able to go through all the trouble of setting it up and testing it in the middle of the performance. It may be possible to do if I disguise it cleverly by using elements to tell me where I need to place the object which would be used as part of the story, or I could just keep the object there for the entire story. The mask I used in a previous test has proven to be an excellent prop for projection mapping, I could give it an entire personality with this.

I really think what I'm doing here is quite unique, but hopefully that's because it's original, not because it's impossible. Projecting vague shapes rather than detailed designs onto my body should allow me to make small mistakes without it being obvious. Another possibility is to use a very bold, cartoony outline which overlaps the outside of my body and the inside so I have a larger room for error. The final test I will do is where I will see if I can animate and perfect the synchronization of a short sequence, if I succeed then I will be able to incorporate myself into the animation, if I fail then I will have to base the story around still objects.

Initial experiments in mapping and some ideas

Before I can come up with a story, I need to experiment to find out what's possible first, so I can write the story around what I can actually do. I had already gotten a few ideas on what to do before I started these experiments. The area which would be projected onto my eyes needed to be kept black, so I wouldn't be blinded and the projector must be kept in a consistent position so the distortion will be the same each time so it would match up correctly. 

 Using a white mask and my projector, I loaded up Photoshop and started digitally painting over the mask. This enabled me to have an instant preview of what I was drawing onto the mask as well as being able to see what sort of distortion was caused in the 2D image afterwards. On still objects it's possible to create all sorts of intricate designs to be projected onto them. At first I just doodled on the mask but then I created a grid to see the distortion more clearly. This showed me that the angle of the projector and the object were both equally important, and that the sides of the object require less pixels as the image is stretched further on the sides of the object. Also since the object was tilted backwards slightly, the 2D image is shorter.

After that last little experiment I created a rainbow grid in flash, then duplicated into a larger grid and darkened other squares so I could see roughly what big square part of me was in if the smaller squares then narrow it down with the smaller squares.

This showed me that the 2D image wasn't being distorted that much on an actual person, I'm guessing this is because I was roughly a flat surface. The worse of the distortion came from folds and creases in clothes, meaning that my costume design should be as smooth as possible using either skin tight clothing, or hard surfaces, like armour or cardboard boxes. Mainly what I have to watch out for is that objects closer to the projector will be bigger than the background.

One big problem, which should be obvious, is that the webcam I'm using, as well as my phone-camera, does not record projected light very well, and ill often appear blurry or glowing, even though the picture is very crisp and clear in real life. Unfortunately my old phone was lost at a party, but I have previously recorded footage of a projection without the blurring or glowing with it, so it is possible to record, meaning I will have to borrow a good digital camcorder eventually. Another possibility is to use a dark colour palette, as it will glow and blur less in recording.

One limitation I've run into is the size of my room. The further away the projector is from the screen it's projecting on, the bigger the screen, so the size of the screen is then limited from the size of my room. Although I'm sure I will be performing this in a larger room than my room, I am still limited to the smaller size as I need to practice at home, since I can't be lugging around the expensive, irreplaceable, fragile projector around constantly, and I need as much time as possible to practice; even though, I still think it's an acceptable size, although it does limit me as it means I can't do anything with my whole body, only my upper body.

Another problem which has occurred to me is that the projector can only focus on one thing, so objects further away or closer to the projector than the object which the projector is focused on, will have a more blurry texture. Since the background will have most of the animation, and is the place where the animation will be the clearest anyway, I will focus the projector onto the background. This means I can't have any objects too far away from the background without losing the crispness of the projected image. Solid colours and non-detailed textures will be largely unaffected however.

Before the recording I went through a short practice run and it was impossible as I had incorrectly timed the animation so it was slower in the middle of the walking, which kept tripping me up and I went ahead of the animation. This has shown me that the animation of the character must be as realistic and well timed as possible, otherwise it will be a lot harder than it needs to be to keep in sync. 

The other thing I noticed when recording this is that I would constantly look at the monitor for the preview so I knew I was keeping in sync. This is a problem when I'm not facing the monitor, especially since I often crank my neck to see, resulting in my head going outside of  the projected area with the character in it. Recently I have accidentally stepped on a mirror, ironically this is lucky since it has given me the idea to use the broken shards as background props, and to place them strategically around the performance area so I can always see the monitor and the preview. Another problem I had was that the preview I was using was only taking up a small portion of the screen, making it difficult to see, this can be solved easily by having a full-screen preview, which is not available using YouTube's webcam upload feature, although I am sure I will be able to find and use software which does have a full-screen preview.

This is the second test I did, just to see if the level of synchronization I needed was possible. The low FPS I used to quickly test this out was a major hindrance, so my final animation needs to be a reasonably high FPS, at least for the character. At first it was very difficult to keep in sync, but after a few tries it gets a lot easier, this shows that I should be able to be perfectly in sync with my animation as long as I get a lot of practice in and plan the animation out well.

One thing which inspired me was the shadow created by the hood I recorded with in the previous video. I use the hood so it reflects the projection better and doesn't alter the colour. A few of the stitches have come out the top so instead of a head shape, it's got a little bit of a spike at the top, behind my head, like the Freakazoid character, Candlejack. I think this is can add a bit to my character, it makes the character look ghost like and more slick and cool, like the candle jack character. This has many uses for character design which will  inspire the story.

The third test video I created was a more detailed version of the previous one. I used the footage from a previous test and distorted it in After Effects with the corner pin effect, to align it with the original grid I used. From here I noticed that I could trace over the shadow to create the distorted image, this means I can easily create all the outlines by recording myself going through the animation I intend to make, while the grid is being projected onto me, and then trace the shadow afterwards.

The main problem I had was aligning the eyes correctly. Instead of projecting two black eyes, I projected one black opening in the face, like a ninja hood, but even with that, it was still very hard to keep in sync with it. Drawing the eyes in the first place is difficult as well, since the shadow doesn't show where the eyes are and it's hard to see the eyes on the low resolution, webcam recording which blurs the grid.

Something which may help is having an additional camera to the side of the scene facing the background so I have an unobstructed view of the shadows. This would have to be a high resolution camcorder since the usable area of the recording would be very thin, since it's facing the background at an angle, and I'd need to stretch it out to align over the grid.

The video also shows how the shadows affects the performance. This means that I should not bring my arms too far away from my body as there will be a shadow across my face, removing whatever texture was being projected onto it, and, of course, this will affect other things like the background as well. Mostly I should focus on keeping each "layer", or object as close to each other and the background as possible to minimize the distance of the shadows cast. The problem is minimized when the projector is placed further into the middle, but unfortunately I have no mount for the projector and I am unable to afford one for this performance, so I will have to keep the projector on a table. The table will have to be consistent during animation, practice and the final performance since, as I mentioned before, the position of the projector effects the alignment and distortion of the 2D image with the real world.

Another problem I found was that after the third or so attempt, my arms were rather tired, since I had kept them raised for a continuous period of time. This shows that my animation will be limited by my physical ability. I have had experience in holding poses and my arms up for long periods of time before in my Drama course, where I had to hold a face and keep my arms up while I pretended to be a cupboard. From that experience I learnt that practice and repetition was all which was needed to train body parts so you could hold them for long periods of time. Even so, this has still shown me that my character animation will still be heavily limited by physical motions, when usually animations are limitless. I will try to limit my character animation to slow movements and motions which I can easily predict so I can keep in sync as much as possible.

Through all of this, I have been getting a few ideas, which I should really jot down. The main theme I am looking to explore is probably video games. I think the story will revolve around the idea of virtual reality, and that we will eventually have an artificial reality completely indistinguishable from reality, that we can shape and mould as we please.

To make the most of this, and to be as impressive as possible, I should really show off some things which can only be shown with projection mapping. I can make myself bigger by moving closer to the projector, a common perspective trick. Immediately after coming up with the idea of changing size I thought of the idea where I could play Mario, get a mushroom and move closer to the camera to be bigger. Through this I could use a title screen saying, "New New Super Mario Brothers", a play on the title which came out suggesting that the animation is set in the future where Nintendo have brought out a new, "New Super Mario Brothers", which is a new version of the classic series, "Super Mario Brothers". 

The whole idea of me acting in a projection mapping animation reminds me of Kinect, a new device for the Xbox 360 which allows the user to use their body as a controller. One of the criticisms of Kinect racing games is that they auto correct the user too much to make up for the lack of precision in Kinect's motion detection, this has lead to some funny YouTube videos where a person has sat completely still and still came in 3rd place. I could make fun of the Kinect, although I don't want to alienate the part of my audience who are unaware of the flaw or even the Kinect. Should I choose to make references I should include jokes or make it entertaining in other ways to those who do not understand or get the references.

The Nintendo Wii is another console which includes motion controls using the Wiimote and nun chuck peripherals. Common criticisms are inappropriate use of motion controls, little effort actually required for the motion controls and that the motion controls really only consist of waggling the wiimote and not much else. This would be quite easy to make fun of, I could start by playing a game with a lot of enthusiasm, then put in less and less effort, while the character I'm playing is still going full force, and eventually I'll give up and use a game pad.

Sound is also an important part, and I can use my voice as well a recorded sounds. Switching between my voice and a recorded, maybe even distorted, voice can be a helpful tool in highlighting the contrast between two situations. For example I could be talking using a normal voice, but then I put on my computer hood or something and I instantly switch to a voice coming out of a speaker instead.

My face can also be easily changed to anyone's and anything. Using this idea I could, for instance, have a scene where I'm creating an avatar and pretend to be someone I'm not in a chat room, or I could use photos of people's faces which I switch to during the credits, the photos would be of those who helped me.

Initial research

This is the blog for my final major project at the University of Creative Arts where I will be showing the progress on creating an animation which is going to projected onto a live action actor and set. This is called 3D projection mapping and has been quite popular in advertisements.

Other 3D projection animations are usually purely designed to show of the visual effects which come from projection mapping. My animation will aim to integrate animation and reality in order to portray a story to an audience.

My initial inspiration for this was actually a few years ago when I saw a mobile phone advert by the French creative agency, "SUPERBIEN". Which you can watch on vimeo at http://vimeo.com/10692284 . My animation won't interact with as many objects and it will be a lot simpler, but hopefully as visually impressive as possible. What SUPERBIEN do is called 3D projection mapping, which I have said before. They project a 2D image which is distorted in such a way so that the light hits the object in a way where it undoes the distortion and the 2D image is wrapped around the 3D objects. If something is closer to the projector then the part of the 2D image hitting that closer object needs to be smaller, since the closer something is to the projector, the larger the projection is on the object, so you basically get an inverse 3D model of the actual scene being projected, this is something I will definitely be using a lot in my animation.

In my project I intend to only use a single live action actor with a very simple design so complex algorithms are not needed to precisely map the 2D image like in the majority of  projection mapping, which are usually designed to play with the faces and edges of the objects like with Toyota's brilliant 3D projection, which you can see at http://www.youtube.com/watch?v=UJ7E7uEZN00.