This is a map I will use for animation. Using a projector, I projected a fullscreen Photoshop 7 onto the set I will use, which includes a door, a cardboard panel and two masks. Unfortunately, due to technical difficulties and the fact that I didn't want to get locked in the studio over the weekend, I only had a few hours to set this all up and complete the map, which is why it is somewhat rough.
What I didn't think of before was using the objects already at the set in my animation. I can definitely incorporate the door, the glass and the frame in my animation, as well as the two masks which I have been focusing on so much. There is a lot of area around the masks as well that I can use. Mainly I want to focus on the masks but I can use the distant area for some subtle effects and to make the projection area blend in more to the surroundings.
These are just some doodles I made to practice facial expression, which is a vital part of my animation. People's heads and faces stretch and contort with every movement and emotion, which may be a hurdle as I have a physical, rigid mask which has to remain stationary which my faces will be projected on to so I might have to break out the binds of the physical world and animate slightly outside of the masks. Hopefully I will be able to keep it subtle enough that the audience only notices the lively facial expressions.
Above is just one of the videos I have been watching recently for ideas on techniques to play with the audience's perception on the physical world using projection.
After some thinking, I have come to realize that the story will have to be quite short, and not just to minimize the animation time, but because people will be walking in and out, looking at everyone's art piece at the show. Even if I make an incredibly interesting and gripping animation, if it's long then people can come in half way and not want to bother to wait until the animation repeats itself.
Another idea I came up with is a visual indicator to show the viewer how far in animation is. Unless I want to create an animation loop with no story which can be watched at any point, which I don't, because I feel animation projection projects always lack a story, and I want to change that. What I thought to do is have the general colour scheme fade slowly from one colour to another and tell the viewer outside what the colours meant and how long the animation is. This way it doesn't impact or distract the viewer from the animation and story while still letting anyone know how long to wait until the animation starts again so they can watch it from the beginning. The colour scheme itself could play an integral part in the story and show the emotion progression.
A theme is something I really need to focus my story around so it remains on point, non-convoluted and easy to follow. After a bit of thought and exploration, I decided on basing the story around loneliness, something I know a lot about, mainly because this project has forced me into a small room for months on end all alone so I can finish it in time, and for a few more reasons which are irrelevant.
The theme needs to be consistently shown throughout the animation, but I want it to not be obvious until the end. I want this so that the end has a greater impact, making you realize how depressing the entire story really is, picking up things which seemed innocent at first glance, but once you think about, have a different meaning with knowledge gained from the ending. This would also encourage the viewer to watch again, which will be a completely different experience now that they know everything which seemed happy on the first watch, was actually indicating the theme of loneliness and maybe depression.
Looking through internet and books hasn't gotten me anywhere. There is virtually no information about 3D projection mapping, I assume because it's a rather new technique which has only recently surfaced. The only way I can discover more about 3D Projection mapping is by experimenting and learning for myself.
Recently, a researcher in computer vision at the University of Surrey has come up with an excellent method for tracking objects via a webcam. I could potentially use this to track my body parts movements and overlay a 3D character over my actual body via projector. This would eliminate the need for tedious trial and error training in keeping in sync. I could even have audience participation by having it track symbols on an outfit. Unfortunately the source code hasn't yet been released and likely will not be until way after my deadline. Even so, I may revisit this project in the future and use some of Zdenek Kalal's work.
The video above shows me at the end of an extensive trial and error run on trying to keep in sync with an animation, which was based upon a previous recording of myself in front the grid I previous made. After the initial video was recorded, I distorted it to overlap the original grid so I could reference it and trace the shadows and create an animation which would be projected onto me, if I were able to keep in sync.
From this test I've come to the conclusion that projection onto a live actor is something which is impossible for me at the moment. The light is blinding so it becomes very difficult to see where I am in relation to the character by looking at the screen, and by looking at the shadow I still can not keep in sync properly. Maybe if I had a year to perfect this it would be possible and amazing, but for now I should simplify my project so that the final piece will look more professional and polished. Plus, using only still props means that I can reproduce this performance with ease, and many more people will be able to see it, instead of just the few people who can see the one performance.
These are just some rough sketches I made for character and set design. Initially I wanted to explore sets which were obviously reminiscent of traditional theatre sets. Particularly I wanted to use a thrust type of staging, which is the classic theatre set everyone should be familiar with, where a platform comes out from the background. I wanted to use the thrust staging as it linked the animation with traditional media and entertainment, showing the progress we have made since then and showing how everything can be emulated, which I want to be a big part of my animation. Unfortunately, the area where my animation will be projected is unsuitable for this and will look rather silly, as the set will be small and high up due to the limitations of the placement of the projector.
I've decided that I don't want any particular character models, as I want the two character to be able to morph into whatever fits their mood, showing the fact that people can easily change what they look like as they can change their avatar in an online game. Courage the Cowardly dog has inspired me somewhat with this as he changes forms to communicate and it's that cartoony, energetic feel I want to capture in my animation. Using paper and paint, I've filled in the eyes in two white masks I've had, as well as going over them again in white paint to ensure they reflect the light better. By filling in the eyes I can project my own eyes onto the masks and have a higher level of control over the emotion shown in them. The only problem I'm worried about is the texture I have after the painting, where before they were smooth, they now have thin lines all over them and are very hard, although I don't expect anyone to be touching them.
In the set design I've tried to keep in mind how the light will bounce off the set. The angles of the objects are intentionally set so that shadows don't become a hindrance, as anywhere there are shadows, I can't project, thus can't manipulate. The other thing I have to be aware of is portability of the props. I need to be able to transport the entire set to and from my house and to the University where it will be shown, and since the projection has to be so specifically set, it can't be too complex to fix in specific places, perhaps I should create a mechanism to ensure props stay in perfectly the same place. Because of this, and various other issues I should have already mentioned, I have decided to keep the set as simple as possible, and this also gives a minimalistic, futuristic, "apple" type of feel to it. This will also be an excellent contrast to the colourful, exciting animation being projected onto it.
The masks should be slightly off centre, and not facing head on to the audience to create a more organic feel to show that the masks are alive and make them seem more human. There will also be a board, I have recently painted white, just for an extra, simple prop which will let me show off projection mapping a bit more.
Over the next few days I will be looking into getting feedback from peers in order to improve my animation. I need to do this soon as I will be starting the animation as soon as I finalize the story, storyboards and create a map of the are where my animation will be projected.
Today I decided to try out what would happen when projecting different colours onto different coloured material. Predictably, the only difference was a tint in the clothing, although I did learn that the camera picks up images much better on black material, but it means I will never be able to project certain levels of white onto the black material. Black material is very good for good black levels though, but bad for showing high levels of white. I don't think the different colours will be much as I can simulate the same levels through the projector anyway, and having everything projected onto it being tinted is a very big limitation.
Most projection mapping projects tend only use still objects like cars or buildings. In fact, all of the projection mapping I have seen will only use a still object. I imagine this is because it's so much harder to go to such an impressive level of detail on a moving object, since you'd have to make sure the object lines up perfectly in every single frame, which would be near impossible. This means that if I want to show off any fancy projection mapping, all the detailed stuff needs to be done at the beginning, when I've had plenty of prep time to get the object into exactly the right place it needs to be. After I move it, I obviously won't be able to go through all the trouble of setting it up and testing it in the middle of the performance. It may be possible to do if I disguise it cleverly by using elements to tell me where I need to place the object which would be used as part of the story, or I could just keep the object there for the entire story. The mask I used in a previous test has proven to be an excellent prop for projection mapping, I could give it an entire personality with this.
I really think what I'm doing here is quite unique, but hopefully that's because it's original, not because it's impossible. Projecting vague shapes rather than detailed designs onto my body should allow me to make small mistakes without it being obvious. Another possibility is to use a very bold, cartoony outline which overlaps the outside of my body and the inside so I have a larger room for error. The final test I will do is where I will see if I can animate and perfect the synchronization of a short sequence, if I succeed then I will be able to incorporate myself into the animation, if I fail then I will have to base the story around still objects.
Before I can come up with a story, I need to experiment to find out what's possible first, so I can write the story around what I can actually do. I had already gotten a few ideas on what to do before I started these experiments. The area which would be projected onto my eyes needed to be kept black, so I wouldn't be blinded and the projector must be kept in a consistent position so the distortion will be the same each time so it would match up correctly.
Using a white mask and my projector, I loaded up Photoshop and started digitally painting over the mask. This enabled me to have an instant preview of what I was drawing onto the mask as well as being able to see what sort of distortion was caused in the 2D image afterwards. On still objects it's possible to create all sorts of intricate designs to be projected onto them. At first I just doodled on the mask but then I created a grid to see the distortion more clearly. This showed me that the angle of the projector and the object were both equally important, and that the sides of the object require less pixels as the image is stretched further on the sides of the object. Also since the object was tilted backwards slightly, the 2D image is shorter.
After that last little experiment I created a rainbow grid in flash, then duplicated into a larger grid and darkened other squares so I could see roughly what big square part of me was in if the smaller squares then narrow it down with the smaller squares.
This showed me that the 2D image wasn't being distorted that much on an actual person, I'm guessing this is because I was roughly a flat surface. The worse of the distortion came from folds and creases in clothes, meaning that my costume design should be as smooth as possible using either skin tight clothing, or hard surfaces, like armour or cardboard boxes. Mainly what I have to watch out for is that objects closer to the projector will be bigger than the background.
One big problem, which should be obvious, is that the webcam I'm using, as well as my phone-camera, does not record projected light very well, and ill often appear blurry or glowing, even though the picture is very crisp and clear in real life. Unfortunately my old phone was lost at a party, but I have previously recorded footage of a projection without the blurring or glowing with it, so it is possible to record, meaning I will have to borrow a good digital camcorder eventually. Another possibility is to use a dark colour palette, as it will glow and blur less in recording.
One limitation I've run into is the size of my room. The further away the projector is from the screen it's projecting on, the bigger the screen, so the size of the screen is then limited from the size of my room. Although I'm sure I will be performing this in a larger room than my room, I am still limited to the smaller size as I need to practice at home, since I can't be lugging around the expensive, irreplaceable, fragile projector around constantly, and I need as much time as possible to practice; even though, I still think it's an acceptable size, although it does limit me as it means I can't do anything with my whole body, only my upper body.
Another problem which has occurred to me is that the projector can only focus on one thing, so objects further away or closer to the projector than the object which the projector is focused on, will have a more blurry texture. Since the background will have most of the animation, and is the place where the animation will be the clearest anyway, I will focus the projector onto the background. This means I can't have any objects too far away from the background without losing the crispness of the projected image. Solid colours and non-detailed textures will be largely unaffected however.
Before the recording I went through a short practice run and it was impossible as I had incorrectly timed the animation so it was slower in the middle of the walking, which kept tripping me up and I went ahead of the animation. This has shown me that the animation of the character must be as realistic and well timed as possible, otherwise it will be a lot harder than it needs to be to keep in sync.
The other thing I noticed when recording this is that I would constantly look at the monitor for the preview so I knew I was keeping in sync. This is a problem when I'm not facing the monitor, especially since I often crank my neck to see, resulting in my head going outside of the projected area with the character in it. Recently I have accidentally stepped on a mirror, ironically this is lucky since it has given me the idea to use the broken shards as background props, and to place them strategically around the performance area so I can always see the monitor and the preview. Another problem I had was that the preview I was using was only taking up a small portion of the screen, making it difficult to see, this can be solved easily by having a full-screen preview, which is not available using YouTube's webcam upload feature, although I am sure I will be able to find and use software which does have a full-screen preview.
This is the second test I did, just to see if the level of synchronization I needed was possible. The low FPS I used to quickly test this out was a major hindrance, so my final animation needs to be a reasonably high FPS, at least for the character. At first it was very difficult to keep in sync, but after a few tries it gets a lot easier, this shows that I should be able to be perfectly in sync with my animation as long as I get a lot of practice in and plan the animation out well.
One thing which inspired me was the shadow created by the hood I recorded with in the previous video. I use the hood so it reflects the projection better and doesn't alter the colour. A few of the stitches have come out the top so instead of a head shape, it's got a little bit of a spike at the top, behind my head, like the Freakazoid character, Candlejack. I think this is can add a bit to my character, it makes the character look ghost like and more slick and cool, like the candle jack character. This has many uses for character design which will inspire the story.
The third test video I created was a more detailed version of the previous one. I used the footage from a previous test and distorted it in After Effects with the corner pin effect, to align it with the original grid I used. From here I noticed that I could trace over the shadow to create the distorted image, this means I can easily create all the outlines by recording myself going through the animation I intend to make, while the grid is being projected onto me, and then trace the shadow afterwards.
The main problem I had was aligning the eyes correctly. Instead of projecting two black eyes, I projected one black opening in the face, like a ninja hood, but even with that, it was still very hard to keep in sync with it. Drawing the eyes in the first place is difficult as well, since the shadow doesn't show where the eyes are and it's hard to see the eyes on the low resolution, webcam recording which blurs the grid.
Something which may help is having an additional camera to the side of the scene facing the background so I have an unobstructed view of the shadows. This would have to be a high resolution camcorder since the usable area of the recording would be very thin, since it's facing the background at an angle, and I'd need to stretch it out to align over the grid.
The video also shows how the shadows affects the performance. This means that I should not bring my arms too far away from my body as there will be a shadow across my face, removing whatever texture was being projected onto it, and, of course, this will affect other things like the background as well. Mostly I should focus on keeping each "layer", or object as close to each other and the background as possible to minimize the distance of the shadows cast. The problem is minimized when the projector is placed further into the middle, but unfortunately I have no mount for the projector and I am unable to afford one for this performance, so I will have to keep the projector on a table. The table will have to be consistent during animation, practice and the final performance since, as I mentioned before, the position of the projector effects the alignment and distortion of the 2D image with the real world.
Another problem I found was that after the third or so attempt, my arms were rather tired, since I had kept them raised for a continuous period of time. This shows that my animation will be limited by my physical ability. I have had experience in holding poses and my arms up for long periods of time before in my Drama course, where I had to hold a face and keep my arms up while I pretended to be a cupboard. From that experience I learnt that practice and repetition was all which was needed to train body parts so you could hold them for long periods of time. Even so, this has still shown me that my character animation will still be heavily limited by physical motions, when usually animations are limitless. I will try to limit my character animation to slow movements and motions which I can easily predict so I can keep in sync as much as possible.
Through all of this, I have been getting a few ideas, which I should really jot down. The main theme I am looking to explore is probably video games. I think the story will revolve around the idea of virtual reality, and that we will eventually have an artificial reality completely indistinguishable from reality, that we can shape and mould as we please.
To make the most of this, and to be as impressive as possible, I should really show off some things which can only be shown with projection mapping. I can make myself bigger by moving closer to the projector, a common perspective trick. Immediately after coming up with the idea of changing size I thought of the idea where I could play Mario, get a mushroom and move closer to the camera to be bigger. Through this I could use a title screen saying, "New New Super Mario Brothers", a play on the title which came out suggesting that the animation is set in the future where Nintendo have brought out a new, "New Super Mario Brothers", which is a new version of the classic series, "Super Mario Brothers".
The whole idea of me acting in a projection mapping animation reminds me of Kinect, a new device for the Xbox 360 which allows the user to use their body as a controller. One of the criticisms of Kinect racing games is that they auto correct the user too much to make up for the lack of precision in Kinect's motion detection, this has lead to some funny YouTube videos where a person has sat completely still and still came in 3rd place. I could make fun of the Kinect, although I don't want to alienate the part of my audience who are unaware of the flaw or even the Kinect. Should I choose to make references I should include jokes or make it entertaining in other ways to those who do not understand or get the references.
The Nintendo Wii is another console which includes motion controls using the Wiimote and nun chuck peripherals. Common criticisms are inappropriate use of motion controls, little effort actually required for the motion controls and that the motion controls really only consist of waggling the wiimote and not much else. This would be quite easy to make fun of, I could start by playing a game with a lot of enthusiasm, then put in less and less effort, while the character I'm playing is still going full force, and eventually I'll give up and use a game pad.
Sound is also an important part, and I can use my voice as well a recorded sounds. Switching between my voice and a recorded, maybe even distorted, voice can be a helpful tool in highlighting the contrast between two situations. For example I could be talking using a normal voice, but then I put on my computer hood or something and I instantly switch to a voice coming out of a speaker instead.
My face can also be easily changed to anyone's and anything. Using this idea I could, for instance, have a scene where I'm creating an avatar and pretend to be someone I'm not in a chat room, or I could use photos of people's faces which I switch to during the credits, the photos would be of those who helped me.
This is the blog for my final major project at the University of Creative Arts where I will be showing the progress on creating an animation which is going to projected onto a live action actor and set. This is called 3D projection mapping and has been quite popular in advertisements.
Other 3D projection animations are usually purely designed to show of the visual effects which come from projection mapping. My animation will aim to integrate animation and reality in order to portray a story to an audience.
My initial inspiration for this was actually a few years ago when I saw a mobile phone advert by the French creative agency, "SUPERBIEN". Which you can watch on vimeo at http://vimeo.com/10692284 . My animation won't interact with as many objects and it will be a lot simpler, but hopefully as visually impressive as possible. What SUPERBIEN do is called 3D projection mapping, which I have said before. They project a 2D image which is distorted in such a way so that the light hits the object in a way where it undoes the distortion and the 2D image is wrapped around the 3D objects. If something is closer to the projector then the part of the 2D image hitting that closer object needs to be smaller, since the closer something is to the projector, the larger the projection is on the object, so you basically get an inverse 3D model of the actual scene being projected, this is something I will definitely be using a lot in my animation.
In my project I intend to only use a single live action actor with a very simple design so complex algorithms are not needed to precisely map the 2D image like in the majority of projection mapping, which are usually designed to play with the faces and edges of the objects like with Toyota's brilliant 3D projection, which you can see at http://www.youtube.com/watch?v=UJ7E7uEZN00.
