Gobelins 2d 3d integration

From bernie's
Jump to: navigation, search

The task is to integrate a 3d rendering into a 2d style background with a 2d hand drawn character, or vice versa, integrate a 2d character in a 3d rendering.

We'll be using a (superbly done animation) and (a superbly animated) 2d 'blob':



Preparing maya for animation rendering

Prepping maya: set project

It's a good idea to keep your workfiles contained in a 'project'. You don't have to, but I heavily recommend it otherwise your files often end up rendering in `C:\Users\you\Documents\maya\projects\default\images`

Once again it's not mandatory it's just something nice (and in production we have tools that set these projects for you)

If you've already done your animation, no worries. Open your file with File>open, then once opened go to File>Project window


Now when you will render a sequence your files will be in the images subfolder

It's also easier to manage your textures because maya will look for them in the 'sourceimages' folder like so:


And as a plus, when you work in a project, all your files are relative. Meaning you can actually zip the whole folder and deploy it somewhere else.

When you open a new maya (or when it has crashed as it often does) remember to go to File>Recent Projects>Your Project or File>Set Project>(choose folder) before opening up your file.

Prepping maya: render settings for animation

Make sure your Arnold is loaded, there should be an arnold shelf and in the renderview you should be able to start a rendering with arnold. If it ain't showing up go to Windows>Settings and Preferences>Plugin Manager and check mtoa.mll

Let's set up your maya so it can actually render a file sequence (which we will load in after effects afterwards). Open up the render settings, make sure you are using arnold, and change the file extension to the one i've highlighted:


Underneath the File Output menu, you can also change the start and end frame and the camera that is used for your scene (it defaults to persp1).

I also changed the file output to be 8 bit png because we might not need that much data (EXRs are often preferred for photo-realistic renders)

You should render out EXRs as arnold doesn't like adding alpha channels to PNGs. You can check 'half precision'.

Rendering animation sequences out of maya

Now in the top menu you can render your sequence with Render > Render Sequence


And your computer should start to render out images (which will probably be black if you haven't added any lights!) Don't worry about specific render settings yet, we will get there.

Texturing your character (UVs and painting)

You'll want to texture your character (and its props if there are any), most likely by painting them (as opposed to a more photo-realistic approach were you 'shade' or 'surface them', choosing the metallic/transparent look of your object. For now let's pretend we want a 2d/cell-shade look.

Your animated character needs 'UVs' to be able to tell maya (or other software) how to apply a 2d image to a character in 3d.

Your characters should have UVs already; creating UVs is a little beyond the scope of this tutorial (or look at Nicolas's file on the google drive, or youtube tutorials). It gets even more complicated if you've already animated your character.

Anyways, if you go to Windows>Modelling Editors>UV Editor you should see the 'pelt' of your character:


(with a silly inception-like image of itself that I've added so you can see how the 2d image is transferred to the 3d model)

Ok nice enough, but will I have to paint on the flat, weird looking blueprint ?

Well yes and no. Usually we use software like zBrush, Mari, Substance Painter or other tools - even Photoshop can import and paint 3d models but it's clunky in my opinion (tutorial).

So I propose we use maya and its old-ass tools to start painting/roughing out colors, and finish with photoshop to get what you want. The idea is to use maya's paint tools.

Let's start painting!

We'll start by adding a new 'lambert' material to our character. In my file, the rigged character is 'locked' so I can't select the mesh, so I have to toggle on 'Geo Selectable' in the main controller. Your character might have the same 'feature' (maybe the character is locker in the layer manager at the bottom right too.

Anyhow, I also made sure to only display the geometry in my scene so that the controllers don't get in the way (you could also isolate the mesh with shift-i):


Right click and add a new lambert material (we'll get to the fancy Arnold toon stuff later on)


We can start the paint tool by changing the interface to 'Rendering' context if it isn't already so, and choosing Texturing>3D Paint Tool


And now to make the actual tool show up if it hasn't popped up (or if it disappears), click the icon on the left (1), it will open up the paint window.

You should now add a new color texture (2) that maya can paint to. I chose PNG at 2048x2048 (3) because I don't think I'll need much more (you can scale it in Photoshop later on if you don't have enough details)


Highlighted in blue are two options you can use to 'flood' the character with your base color, and then you can use your tablet to draw on the character with your favorite color. Press 'b' and the left mouse button to scale your brush size up or down and paint to your heart's wishes.

Note on colors: the colors you pick might not match the ones you paint. That's because there's a whole story about Linear workflow and Color Spaces in texture management. Once again this is a little beyond the scope of this tutorial; don't trust the maya colours too much; we'll open them up later in Photoshop. Alternatively, you can also uncheck 'Color Management' when you choose a color.

Another handy tip if you have the mesh wires (green by default) in your way is to go to Show>Selection Highlighting. Remember to toggle it back on when you are done.


Hit save textures from time to time to bake it to disk; the path will be shown in the script editor output line at the bottom. Usually its Project Folder/sourceimages/3dPaintTextures/scene_name


Flattened look:


Before we start painting in Photoshop, let's export the UV map of our mesh so we can paint precisely in Photoshop. Select your mesh, Windows>Modeling Editor>UV Editor then in the UV editor, Image>UV Snapshot


You can now overlay the UV snapshot to your painted image and start painting precise details with photoshop. Rinse and repeat for all your objects/meshes.


Using arnold to render out cartoon/cell shade colors

Ok I'm going to go on a bit of a fuzzy path here, but I'll show you in that order:

  • how to render
  • short intro on arnold shaders
  • short intro on lights
  • using the aiToon (cell) shader with our textures
  • adding outlines/strokes
  • splitting your character and your background as well
  • undoing everything and using render layers and splitting strokes and colors to comp them in after effects

how to render

I showed you at the beginning how to render a sequence, but to test things out you can have the viewport be a live rendering. In the viewport window just go to Render>Arnold Render and it should pop up a window where you can press play and it will render out the viewport. Alternatively, you can go to the top menu bar and Arnold>Open Arnold Renderview; this will pop up a window where you can test render your scene (top right play icon), and save snapshots (bottom right camera icon).


Short intro on arnold shaders

Shaders define how your 3d objects reacts to a light before it reaches the camera, how transparent, rough, metallic, illuminated, reflective they are.

Arnold use an 'uber' shader for pretty much all its realistic materials, and the other ones (except the toon shader) we're not going to talk about because they are beyond the scope of this exercise. The arnold doc is pretty comprehensive and well furnished so if you have time you should definitely read through it: https://docs.arnoldrenderer.com/display/A5AFMUG/Understanding+Physically+Based+Rendering+in+Arnold


Is the 'uber' shader that does everything. To apply it to your object, right click the mesh, Assign New Material > (Arnold Tab) aiStandardSurface. Once you have added your shader, you can edit it (and rename it to keep things clean), the fasted way is to right click the mesh and hit Material Attributes. The standard shader comes shipped in with presets meaning if you go for a realistic render, chances are there's already the foundation for your shader here.

The presets (which can be found on pretty much any node) can be applied like so:


With default rendering parameters and a couple of lights you can get a realistic render straight away:


Here I used presets (Balloon, Honey, Velvet and Orange Juice)

and my scene viewed from further out:


arnold lights


There are a few sources of light used in Arnold that are in order in the Arnold shelf:

  • area light: your main work horse, we'll see it in more details below
  • mesh lights: like turning your tube object into a neon light
  • photometric lights: used for architecture mainly, they use ies files to shape the way your light looks
  • skydome: all-emcompassing light that simulates an environment, big spheres to which we apply high dynamic range images, which store more light information than a standard JPEG would.
  • light portal: used for faking window light sources, don't worry too much about it
  • physical sky: to create an outdoor lighting including a sun and blue sky

the area light

It's the main light you will use, mainly because it's the one that works with the toon shader. There are a few options that you can change:


  • Intensity and exposure: change the strength of the light (I almost always set my intensity to 100 and just change the exposure). Note that you can also add a texture in the light color to create a light gobo/mask.
  • Spread: changes the, well, 'spread' of the light, the less spread the more it will behave like a focused light much like a theater 'chase' light.
  • Samples: if you have noisy/grainy renders, you can add more 'samples' in your render settings, but it's often better to start adding more samples to specific light sources first.
  • Visibility: this can be useful if you want the lights to just show up as reflection (specular to 1 and diffuse to 0), or not show up in reflections but illuminate the scene (specular to 0 and diffuse to 1)

There's a few things that I'd like to talk about but that would take too much space here so I invite you to look at tutorials online: light blockers (you can 'shape' where you light goes) and light linking (you can tell a light to just illuminate one specific object in the scene).

I won't go in the full details about render options but you should notice here that my light 'bounces' around a lot. While it's awesome for photorealistic renders, maybe you don't want that. The quickest way to turn that off is to change the number of 'bounces' in the render options by setting 'diffuse' to 0



This is the main shader we will use for a 2d 'cell shade' look and add outlines/strokes. Once again the Arnold doc is fantastic and will answer more questions that I will: https://docs.arnoldrenderer.com/display/A5NodeRef/Toon#Toon-StylizedHighlight

But let's take a look at it.

Assign a new toon shader to your character via right click>Assign New Material>(Arnold Tab)aiToon

There are a couple of options but for now we'll try to get the colors acting nicely. I assume you've created a texture, so you can go to the Base tab of your toon shader, click the little checker at the right of the color slider, and then go pick File Texture (in the maya dropdown). In the file texture go choose your beautifully painted texture.


Ugh ? But it's exactly like before !

Yes it is. You can now go to tonemap slider and select Ramp, and in your ramp change it so that it is stepped like so (notice I turned interpolation to 'None'):


aiToon outlines

To create a stroke/drawn line, we'll first need to activate outlines by going to the Render settings>Arnold Tab>Sampling>Filter and picking 'Contour' and a maximum line size (here I chose 5 pixels)


Now in each shader you will have mainly two parameters you can play with to get a nice outline, the Edge Detection Angle Threshold (set it to 60 to start with, it will reveal differences in curvature of your model) and the Width Scaling, which will be a percentage of your main silhouette size (here set at .65 so 65%):


Rince and repeat and you're done right ?

Omg this is so tedious!


Which is why we're going to go a little more technical and use render layers!

Render layers

To make it simple a render layer is a saved state of your current scene where you've for example:

  • picked which objects are visible
  • which objects have which shader
  • which light is on
  • whether or not such object should have shadows
  • changed render settings to that it renders faster
  • pretty much anything you want in maya!

Each render layer will be rendered separately. While it will take a little longer to render your scene over and over; it can be a real time saver because you don't have to set up a 'perfect' scene and you can finish things of in compositing. Render layers are like saving several versions of your file where you apply different shaders each time. Except you can continue to work on your animation!

For now we'll output a contour, a 'toon' shadow (which could be done solely in compositing, mind you) and the base color:


It gets a little technical so please follow along!

the render layer editor

Let's take a look at the Render layer editor that you can access next to the render button; this is the setup that I used to output the three files above.


  • In red is the masterLayer area: this is the 'standard' layer; this is what gets rendered by default. We will turn it off using the yellow icon on the right.
  • In blue is the layers setup area: here you can create layers (with the plus icon just top left). You should rename your layers as they will be the file names later on.
  • In green are collections and overrides: a collection is a selection of meshes, lights, animation curves, and pretty much anything you can select in the outliner. You can have as many collections as you want in layers, and copy paste them between layers.
    • overrides are options that you wish to change in that parent collection, in that current layer. For instance here in the layer toonLight you can see that I have one or more objects in the collection (__SET_animatedSphere... where I've chosen for the object not to cast shadows. We'll get into overrides later.
  • The purple and yellow icons are respectively the buttons to make the current layer visible; ie show up in the viewport (because we might have only a subset of our objects in the collections) and the other button is a button to toggle on or off if it will be rendered when we hit render sequence. Here I turned off masterLayer because I only want my layers to render.

creating a render layer and adding a shader override

So let's create our render layers. We want our character and the lights to be present. It's kind of hard to show in screenshots, so a gif is worth a thousand words (and several megabytes):


I created a new layer, renamed it, created a new collection, selected my lights and character geometry (just the geometry otherwise the nurbs controller would render), added it to my collection, and made this layer the current layer (so I can see it in the viewport).

(a small note, I added my lights manually because in the render layer editor I toggled off Options>Include all lights in each render layer by default)

So let's create our first render layer which will be the cell shade light pass. Since we've created our render layer and our character collection, we can right click the collection and hit create layer override.


We can now add an aiToon shader in the override, and this will cover all our meshes in the collection with this toon shader. I'm specifically not putting the outlines in this shader for now.


the outlines layer and some complicated override parameters

All's good. I can ctrl-c ctrl-v the previous render layer and rename it 'outlines'. You can actually remove the lights as the outlines don't need them. I'll keep them for now. Now here is where it gets technical: Remember when earlier we were looking at the outlines in the toon shader and we had to change the filter in the render options to 'contour' ? Well we have to tell maya that for this specific render layer, we want to change the filter to 'contour' (by default it's Gaussian width of 2).

So we need a new collection that contains our render options, specifically the filter options


Create a new collection, change 'Collection Filters' to All, write defaultArnoldFilter in the box and hit add. You can now add a filter override by right clicking the collection>Create Absolute Override. In the override menu 'Type Attribute', write 'aiTranslator' (what maya calls what we call 'filter', don't ask me why.). Hit create and change 'gaussian' to 'contour'.

We're not done yet! We also want to change the filter width of the contour (our max contour width which I will put at 5). I'll show you the other way to add layer overrides (because you don't want to have to find out every attribute by name)

Open up the render options, and find the filter size (since we already over-rode the filter type should show up as 'contour'). You can right click the filter size directly and hit 'Create Absolute Override for visible layer' . It will turn the parameter orange (all parameter names that turn orange in maya are layer overrides) and you can change its value.


Try changing visible layer now to see the values change !


The color layer

The last layer we want is the color layer. Ideally pure 'flat' colors. We could add arnold shaders and make the emission at 1 so it's like a light, but it's a little complex to set up. So we will use a feature of arnold (and other renderers) called AOVs, or arbitrary output values.

Arnold AOVs

AOVs are basically images that the renderer uses internally to create the final image. We get them for 'free' (things like light just the shadows, just the light going through skin, just light that bounces around, just the refections) and more importantly we can grab the AOV called 'direct albedo' which is the raw, flat textures.

The color layer

You can copy another render layer and delete the 'material override'. It should go back to your normal character with the textures on.'

You could also go back to the main master layer and create a new render layer and render collection from here


Alright, now you can enable the AOV by going to the AOV tab of the render options, and choosing diffuse_albedo from the left column to the right:


Now by default it will create a diffuse_albedo pass for all the render layers, but we only need it for this one. So you can switch to the other render layers and toggle off the checkbox (highlighted in blue in the image above) by right clicking the checkbox, Create Absolute Layer override for visible layer and turning it off. When you switch to/from the layer, it should change the toggle on/off state.

You can turn on the AOVs will you render your image interactively like so:


You can take a look at other AOVs. I will also render a 'depth' AOV (called Z) and for our purposes this will be it.

TBD (to be done) 3d/maya/arnold

(i'm linking to youtube tutorials mileage may vary)


Alright. Finish prepping up your scene, and hit sequence

2d comp

We've finished rendering and here's the output of all my footage:


(note here, my Z pass comes out all 'white'. It's mapping a distance between the camera (0) and your scene (infinity) so I used a 'levels' effect to make show up.

After Effects, the overview

If you know after effects, you can probably skip this part! And the whole tutorial, or maybe just check the few tips and tricks at the bottom (TBD) For your information the current machine I use there's only an old version of After Effects so things might have improved since then, sorry if I'm not up to date on everything. But for our purposes, it should stay the same.

After effects if very much like Photoshop with a timeline added. You will be working on 'compositions' which are assemblies of your footages.

General UI


AE's default UI should look more or less like this. You can move around panels or reset the UI if you've f*ed up at the top-right where it says Workspace>Standard (right click and 'reset')

  • The 'Project window' is the library where you import your sequences, backgrounds, sounds and where the 'compositions' are created. Keep it tidy!
  • The viewer is pretty self-explanatory. It shows your current work. A common beginner's mistake is if you've double-clicked a footage, you will see the actual footage here (not a composition) and won't be able to edit it
  • The panels are little modules that you can dock, usually I have my effects window here, and type/text options
  • The timeline is where you will be working a lot, it's where you stack your footage, add key-frames and edit animation

Project window


As mentioned, keep this area tidy. I like to have my 'final' composition at the top, and the rest in organized folder. You can drag and drop footage here or double click. A good tip is to drag recently imported footage (you can select multiple) on the composition icon at the bottom, it will create a composition that is appropriately sized. Another tip is if you are missing files when you open your after effect file, you can write 'missing' in the search textbox to make them show up.



I only show the few things I toggle on/off here. Most often people work with 'adaptive' resolution, but sometimes I like to work with 'Final' quality, even if I'm zoomed out, because effects don't always behave correctly when resolution is small. In the 'preview preferences' I often toggle on Off>Final Quality, even though it's slower, it shows final quality effects when changing values.



There's a lot of buttons here, so you should try them on! The ones I use a lot are the 'solo' toggle button. Otherwise I use mostly shortcuts. Another tip for beginners is that sometimes your missing some column (for instance here the blending modes). Hit the buttons at the bottom left or F4, or right click and change which colums show up.


Compositions are the main work items of After Effects, think of them as photoshop files. In fact, there's almost a one-to-one correspondence between a photoshop psd and an after effect composition, as you can see if we load a PSD file into after effects (and tell after effects to keep layers) we can see the same hierarchy applies:


The layer group in Photoshop is what we call a composition in After Effects.

Some of the effects and layer styles haven't copied over properly mind you. You an of course edit your photoshop file to update your after effects file. Just be aware in this specific case, if you add a new layer in photoshop it won't reload automatically in After Effects.

In each composition you can stack have layers, and move them around in the timeline to change when they appear in time.


Are the base elements that fill in your composition (which is by default transparent).

There are several different types of layers:

  • compositions can be nested, they can be
  • solids and shapes are 'filler objects' native to after effects, you can edit and modify them via masks (masks are avaible on any kind of layers, weather it be text, solids, footage etc..)
  • adjustment layers are 'transparent' layers, that you can apply effects to (blurs, levels etc) that will apply effects to layers situated below
  • null layers are, well, 'null' objects which are mainly used as 'parent' objects that you can attach to children objects to scale or move them accordingly
  • text objects which you can animate
  • footage which you've imported, whether it be a sequence of images, a video file or a still image.
  • sound file
  • cameras and lights are used to create '3d' comps where you can fake a '3d' environment (mostly on flat planes) - we will not use these for now

Layer examples TBD

Last things before we start/prefs/exports

Before we start here's a few "gotchas". By default After Effects might not be configured for your intended frame rate (usually 24 or 25fps if you're at Gobelins). Go to Edit>Preferences>Import and set your sequence import frame rate to one of your choosing. This will save you headaches.

A note on bit depth as well. Even though you will output your final movie in 8 bits (like a quicktime for youtube), you might want to work in 16 or even 32bit per channel in after effects. It might slow things down, but it will also make some effects work much better. Bit depth is a little complex to get into but you can read about it here: https://helpx.adobe.com/after-effects/using/color-basics.html

Exports used to be done straight in After Effects (in any composition, hit Ctrl-M to render out), but nowadays you should use Adobe Media Encoder, a specialized tool to create your videos from After Effects or Premiere

3d comp

Stacking layers

Ok let's load our sequences from 3d and drop them into the little composition icon at the bottom left of the project window - or alternatively, create a new comp (Ctrl-N) with the proper size and length


It's the barebone comp. Here are a few notes:

  • I renamed my layers in the timeline because the filename isn't really nice to look at. I click the 'layer name' bar to change between the name i gave the layer and the real name.
  • I changed the mode of the '3d_shadow' layer to multiply so it darkens my color
  • I toggled off the visibility of the Z layer because I don't need it.
  • Finally, I noticed my '3d_color' pass is missing an alpha, and I won't be able to see through it! Damn! Let's fix this and learn along the way


So how do I fix my 3d_color layer ? Well I could rerender it but I already have the proper alpha in my '3d_shadow' layer, so we'll use that one. To do this, two options:

  • use a 'set matte' or 'set channel' effect, and go grab the alpha
  • use the 'Track Matte' column. It's easier to understand when you look at the timeline, so I'll use this.

Duplicate the layer called '3d_shadow', and place it just above the layer you want to mask. On the layer you want to mask, go to the column called 'TrkMat' and choose for the layer '3d_color' Alpha Matte


Note that this could also be any other layer, like a sequence or a black and white mask if you choose Luma Matte:


Due to time constraint I've decided to finish the tutorial using marvelous youtube technology: