This looks like a limitation of the keying engine.
Basically, the key is based on the difference between the background color channel (blue or green) and the other two channel values for any given pixel. This works well generally, but falls apart a bit for very dark/black pixels, since there is little to no difference between the channel values (in fact, the background channel can even have a slightly higher value that the non-background channels) ... which results in the mess you see in the "blacks".
Since we generally shoot with backgrounds that are evenly lit (and therefore have a fairly consistent luma value) a great solution for PK would be to include luma based cutoff sliders that could essentially designate low and high thresholds for the keying engine that would designate pixels brighter or darker than the slider values as being "solid".
Here's a link to where I've asked for this before in the wishlist thread:http://fxhome.com/forums/viewtopic.php?t=39504&postdays=0&postorder=asc&start=23
In the meantime, you can work around the issue in photoshop.
Setup your PK project to be sized "As Foreground", so your export will be the same size as the original image. Key as normal in PK, export canvas to PNG with no background, this keys most of the image for you, but leaves some messiness in the darks of your subject.
- Open your PNG.
- Also open your background image and drag it into your PNG window, then make sure the PNG layer is above the background layer.
- Now, we'll fix the blacks....
- Open your original photo and drag it into your PNG photo so it's the top layer above the PNG and the background.
- Go to the channels panel and select the channel that matches your backdrop (green channel for greenscreen, blue channel if bluescreen) ... you should now see just a grayscale image of your selected channel.
- Drag this channel to the "New Channel" icon at the bottom of the channels panel to make a copy.
- Now, use the Image->Adjustments->Levels command on the result to adjust the image so the bright and mid-grays are completely white, leaving just a tiny bit of gray edge around any black zones in the image.
- Use the Image->Adjustments->Invert command to invert the image so the black areas become white, and visa versa.
- Getting close .... now drag this layer that you've been adjusting down to the "Load Channel As Selection" icon (looks like a circle marquee) at the bottom of the channels panel. The black areas of your image should now be selected.
- Click the RGB channel at the top of the Channels panel to show the full color image again.
- Time to finish! ... go back to the Layers panel and make sure that top layer (with the original image) is selected. Click the Add Layer Mask button at the bottom of the layers panel.
The result should be that only the dark and black areas of your original image are left "solid", revealing the rest of the PK PNG and your background. If needed, you can do a bit of touch-up to the mask with a brush. You can also use the Hue/Sat command to desaturate backdrop color from your original image layer if needed in case some background color spilled in when you filled in the darks.
If you feel like posting a not-keyed-yet copy of your example above, I could try the process out on it for you.
Hope this helps.