You are viewing an archive of the old fxhome.com forums. The community has since moved to hitfilm.com.

My Alpha mask is too sharp...Apparently

Posted: Fri, 28th Nov 2008, 2:35am

Post 1 of 14

pscamm

Force: 2225 | Joined: 5th Nov 2006 | Posts: 300

VisionLab User VideoWrap User Windows User

Gold Member

Hello all,
Quickie question for any masters out there

Ref a test i done once - With a background vid in place and a CG layer on top that is masked by an alpha channel to only show the required object, I have had several comments saying that the edge going around the object (defined by the alpha) is too sharp against the background footage !

I have tried all sorts of grading filters like defocus and Gaussian blur on the upper layer but this appears to soften the alpha (and my whole object) which ends up showing more from the upper layer instead of mixing the pixels of the upper layer with those from the layer below.

Is there a filter in visionlab that will (for want of a better expression) only Dither the 'border' around my object and the lower layer by 'x' pixels ??

If you zoom in close to a vid in VL you notice that pixels kind of mix (dither) together when moving from one colour to another on the plate, if the masked layer above is to look integrated into this background layer then the border between the two layers will need feathering /dithering / mixing just a bit. Doesn't it ?????

Question is......How ?

Cheers All
Paul
Posted: Fri, 28th Nov 2008, 2:41am

Post 2 of 14

Axeman

Force: 17995 | Joined: 20th Jan 2002 | Posts: 6124

VisionLab User VideoWrap User PhotoKey 5 Pro User MuzzlePlug User PowerPlug User PhotoKey 3 Plug-in User FXhome Movie Maker FXpreset Maker MacOS User

SuperUser

What the background consists of will have an impact on what blending technique you use. Light Spill will be effective for cases where the background is bright, and possibly even in some other cases as well. Did you try adding a Gaussian blur from the Key toolset, as opposed to the Grading toolset?
Posted: Fri, 28th Nov 2008, 2:47am

Post 3 of 14

pscamm

Force: 2225 | Joined: 5th Nov 2006 | Posts: 300

VisionLab User VideoWrap User Windows User

Gold Member

Christ...That was quick Cheers Axe

Tried light spill but background was defiantly not bright enough, didn't see any difference at any setting.

HMmmm....Never thought of using the Key tool set, i assumed that would be for if you keyed a green screen clip for example

Hay Ho, i'll try it and let you know

Cheers Axe
Posted: Fri, 28th Nov 2008, 6:10am

Post 4 of 14

Axeman

Force: 17995 | Joined: 20th Jan 2002 | Posts: 6124

VisionLab User VideoWrap User PhotoKey 5 Pro User MuzzlePlug User PowerPlug User PhotoKey 3 Plug-in User FXhome Movie Maker FXpreset Maker MacOS User

SuperUser

Sure thing. Let me know how it goes, I'm curious if that will work as well.
Posted: Sat, 29th Nov 2008, 4:14pm

Post 5 of 14

pscamm

Force: 2225 | Joined: 5th Nov 2006 | Posts: 300

VisionLab User VideoWrap User Windows User

Gold Member

Axe,
Didn't work, had no effect at all. Iv'e got a screen shot of VL with my project loaded up which helps show how i have things set up but i cant seem to find a way of adding it to my post, is there a way i can attach this image to a post ?

Regards
Paul
Posted: Sat, 29th Nov 2008, 4:38pm

Post 6 of 14

pscamm

Force: 2225 | Joined: 5th Nov 2006 | Posts: 300

VisionLab User VideoWrap User Windows User

Gold Member

Ok,
In the mean time, i had a thought ref the comments i had about the edge being too sharp.....

Here's a link to the origonal clip, take a look & see if you agree

http://www.fiddlerstudios.com/misc/

Maybe the comments of too sharp refer more to motion blur than to the edge blur around my 3D object. What i mean is, if you watch the motion of the camera on the background plate in the vid, there is a small amount of directional blur caused by the camera movement but my added creature will stay perfectly solid as theres no motion blur on that object so it will look pin sharp and perfectly focussed all the time.
If you think i have a valid point here then one solution i can think of is to add a grading object to my timeline on top of all the others, load the objects alpha into the 'Mask' channel so whatever i do only affects my object and not the whole plate and finally add the 'Angle Blur' filter, animate its direction and intencity to match what the background plate is doing and all should be fine biggrin

Well, not quite. Tried this also and as my alpha is in place the blur effect which i hoped would spill onto the background plate didn't because it disappeared behind/under my alpha...GRRrrrrr

As you can imaging, this is doing my head in *L*
Posted: Sat, 29th Nov 2008, 5:33pm

Post 7 of 14

Axeman

Force: 17995 | Joined: 20th Jan 2002 | Posts: 6124

VisionLab User VideoWrap User PhotoKey 5 Pro User MuzzlePlug User PowerPlug User PhotoKey 3 Plug-in User FXhome Movie Maker FXpreset Maker MacOS User

SuperUser

Having watched that video clip, in general that's pretty nice work. I understand the 'sharpness' comments, but I think it has to do with a slight difference in lighting on the creature vs. the background plate. The creature has more contrast than the background, to my eye. As far as motion blur, isn't that something that you could add in your animation software? Typically it can calculate the motiomn blur correctly based on the camera's movements.

To add a screenshot, you just have to upload the image somewhere then post a link.
Posted: Mon, 1st Dec 2008, 3:34pm

Post 8 of 14

pscamm

Force: 2225 | Joined: 5th Nov 2006 | Posts: 300

VisionLab User VideoWrap User Windows User

Gold Member

Axeman,
Thanks for taking a look, i trust your eyes a damn site more than i do mine as im a newbie here trying to learn. I'll play around with the contrast as you suggest and see what i can come up with.
Ref the motion blur thing, Yes, i probably could sort any motion blur in the 3D app providing this blur also makes it into the alpha channel but the only problem there is the render time can increase 10 fold or more which isn't ideal.

Ill try some tests and see what happens and post back

Thanks for taking the time to help me out

Much Appreciated
Paul
Posted: Mon, 1st Dec 2008, 3:43pm

Post 9 of 14

Axeman

Force: 17995 | Joined: 20th Jan 2002 | Posts: 6124

VisionLab User VideoWrap User PhotoKey 5 Pro User MuzzlePlug User PowerPlug User PhotoKey 3 Plug-in User FXhome Movie Maker FXpreset Maker MacOS User

SuperUser

No problem. Keep us posted as to how you get on.
Posted: Mon, 1st Dec 2008, 5:42pm

Post 10 of 14

pscamm

Force: 2225 | Joined: 5th Nov 2006 | Posts: 300

VisionLab User VideoWrap User Windows User

Gold Member

Axeman,
Just for interest, i managed to load the pic i said about on the net if you wanted to see how it's all set up

http://img509.imageshack.us/my.php?image=vlshotkl3.jpg

This is actually a trial at a multi pass rendered layer composite

The Track/layer details are as follows:

Monster_Alpha - Speaks for itself really *L*, this is a fully rendered clip which contain's the alpha channel. Only there because the other passes don't contain their own alpha's

UUU - This is the actual background plate for the composite

Monster_Ambient - 1st layer, ambient light pass using the 'normal' blend mode and an 'Object Mask' with Monster_Alpha loaded in the 'mask' channel, the only other thing added to this track is a 'grade - Transparency' filter for adjustment.

Monster_Diffuse - Diffused light pass using 'Add' blend mode, the only thing added to this track is a 'grade - Transparency' filter.

Monster_Specular - Specular light pass using 'Add' blend mode, the only thing added to this track is a 'grade - Transparency' filter.

Monster_Shadow - Shadow pass using the 'Multiply' blend mode also using 'Grade' filters diffuse and transparency.

Leg_Mask - This is a duplicate layer of the UUU background plate with 2 freehand masks for the legs on the chair so the monsters leg appears behind the chair leg - no other filters added

Grade_Object - This also has the 'monster_Alpha' loaded in the mask channel so any grading (which im not very good at yet *L*) i do only affects the CG object layers below and not the whole plate. This is where i imagined placing the 'Blur - Angle' filter but as i said the effect is masked from the background plate because of the loaded alpha.

So, does everything look right here ?

Cheers
PAul
Posted: Tue, 2nd Dec 2008, 3:31am

Post 11 of 14

Axeman

Force: 17995 | Joined: 20th Jan 2002 | Posts: 6124

VisionLab User VideoWrap User PhotoKey 5 Pro User MuzzlePlug User PowerPlug User PhotoKey 3 Plug-in User FXhome Movie Maker FXpreset Maker MacOS User

SuperUser

Yep, looks like you got it set up correctly to me.

I was just experimenting a bit with a quick setup involving an object mask applied to another object, and I got a blur to work decently by applying a blur to the Alpha source layer. When you apply the Grade filter, make sure that it is set to affect RGBA, not just RGB. As long as the blur is set to affect the alpha as well as the RGB channels, it worked ok for me. I just wasn't sure if this setting was something you messed with at all in your experimentation.
Posted: Tue, 2nd Dec 2008, 9:06pm

Post 12 of 14

pscamm

Force: 2225 | Joined: 5th Nov 2006 | Posts: 300

VisionLab User VideoWrap User Windows User

Gold Member

Hello again Axe,
Ok, so i hear what your saying so i tried it. Before i go on, i did make sure all possible settings were set to RGBA.

Open the following pic to hopefully see i interpreted your instructions correctly, it's a VL screen shot again so you can see how i set it up and also to show the result

http://img523.imageshack.us/my.php?image=multilayercompositeblurai2.jpg

Yes, there seem's to be blur spilling onto my background plate but if you note, there is a lot of black mixed in with the blur also, i hiked up the intencity to 49 so you can see it a little better. If i turn the blur on and off i see not a lot of blur on the actual creature but lots against the floor in the back plate. I hope you'll agree that the result isn't exactly what i expected. Notice also i copied the blur up to the 'grade object' to see if that would make any difference but turning this one on added a grey hue to the whole creature and didn't really improve the effect.

If you open the next pic link you'll see the effect as i would expect to see it.

http://img523.imageshack.us/my.php?image=singlelayercompositebluhr5.jpg

This pic as you'll see is just a back plate with a SINGLE layer on top, even though there is a built in alpha to the 'doc 2_alpha' image, it's not being used to mask any other layer. Added simply a blur angle and set it as best as i could to match the stretched blurs on the background in angle and lenght and as you can see the result is completely different to the previous pic. This is exactly what i'd expected to see in the multi layer comp you looked at earlier.
In the other pic, i have no option but to load an alpha channel to the 'grade object' else the blur effects the whole plate including the background. If only there was an option box to click somewhere which would tell the blur angle to blur onto the whole composite as you see it on the screen instead of disappearing behind the loaded alpha sad
That would be my problem solved.

If you see my problem, do you think i should submit a feature request for a future update ?

Regards
PAul
Posted: Tue, 2nd Dec 2008, 11:26pm

Post 13 of 14

Axeman

Force: 17995 | Joined: 20th Jan 2002 | Posts: 6124

VisionLab User VideoWrap User PhotoKey 5 Pro User MuzzlePlug User PowerPlug User PhotoKey 3 Plug-in User FXhome Movie Maker FXpreset Maker MacOS User

SuperUser

Sorry, I should have been more clear in my last post. What I was attempting to achieve, and seemed to do so successfully, was the original idea you asked about, involving adding a bit of blur to the edges of your composite so they blend with the BG more. We had since moved on to discussing motion blur, which I think is what caused the confusion.

If you were to render the alpha into the footage of the creature, you could blur it into the footage just how you want, as in your second example. But when you blur the footage, and then constrain what is visible with an Object mask, that mask is going to force a harder edge onto the top of the blurred footage. Adding the exact same blur to both the Alpha source layer and the image layer may have some success, but I suspect the edges are still going to look weird.

My understanding is that the proper way to do what you are trying to do is to render the motion blur into the footage, but i have very little experience with compositing CGI.
Posted: Fri, 5th Dec 2008, 3:52pm

Post 14 of 14

pscamm

Force: 2225 | Joined: 5th Nov 2006 | Posts: 300

VisionLab User VideoWrap User Windows User

Gold Member

Thanks Axeman,
Sorry too, got you a little confused there didn't I *L*. One thing i have noticed is a very thin black line around my creature using the Alpha. I think this may be the fault of my 3D package output rather than a fault in VL, i'll contact them over this & see what they find, im sure that this issue isn't going to help it look integrated into the scene.
I understand you haven't got a lot of experience in this specific area, Hey, nor have i *L* biggrin but im trying to learn so i need to make contact with someone who HAS got experience so i can learn the rules. Are you aware of anyone who uses Fxhome products who has dabbled in this sort of thing or maybe there are some doing work for the Fxhome movie project as i recall the guy's asking for CG ppl to do post work for it.

Cheers
Paul