Saturday, May 19, 2012

Demosaicing the Fuji X-Pro1 Part 3

Yes. Part three. The previous posts about Demosaicing the Fuji X-Pro1 are here and here.

This post follows on from the previous two by showing how to get demosaicing that is, for practical purposes, as good as SILKYPIX, the best of the  Fuji X-Pro1 raw developers I tested in the previous posts.

In post one, I noted the chroma smearing that was very evident in the Adobe Camera Beta conversion, and to a lesser extent in the SILKYPIX conversion. At the time, I didn't recognize the cause, and asked for input from readers. I didn't get any input, but not being one to be able let a technical puzzle alone till it's solved, I carried on thinking about it. Finally, something clicked. Attentive readers may also remember that I had mentioned in passing that it would be possible to eliminate the highlight artifacts that PhotoRaw showed by median filtering.

What clicked was that there are also simpler forms of filtering that can eliminate those kinds of artifacts - specifically, mean filtering. Mean filtering is considered to be better than median filtering at reducing artifacts, but I hadn't previously really considered mean filtering because it's also considered to be quite primitive. Specifically, it smears chroma and reduces saturation. Duh! I had been thinking in terms of ACR and SILKYPIX using one of the very high-tech anti-artifacting algorithms to be found in academic journals, not something that barely gets a mention.

Mean filtering and median filtering

Before going further, I should briefly explain mean and median filtering. In simplified form, mean filtering is just simple averaging of a series of colors (or numbers) . Median filtering takes the center value of a series.

For example, consider a series of five numbers:
1 2 4 4 5
The mean of those is  3.2, but the median is 4, because if you order the numbers, the number in the middle is 4.

In imaging, median filtering is (usually) considered better because you get a color that actually exists in the image. E.g. if you had an area of blue and red pixels, a "mean" would be green. Problem is, green isn't either red or blue, and will stand out. A median will always give you either red or blue, and so will blend in.

So I did some experiments:

PhotoRaw, ACR and SILKYPIX

Just for reference, here are the three contenders from post one:

PhotoRaw 3.5.4, 400% crop

PhotoRaw does pretty well, but has those pesky artifacts on the paperclips.....

Adobe Camera Raw V7.1 beta

ACR beta 7.1  - Lots of chroma smearing, and the letters are quite desaturated.


SILKYPIX conversion

SILKYPIX - some chroma smearing, saturation down, resolution appears slightly lower than PhotoRaw or ACR

Now for something different......PhotoRaw plus!

What I did as an experiment is to build two modified version of PhotoRaw 3.5.4:
  1. A version with  3x3 median filtering
  2. A version with  5x5 mean filtering

Here's the results:

PhotoRaw 3.5.4 plus median filter, 10 pass, 400% crop

PhotoRaw 3.5.4 plus mean filter, 5 pass, 400% crop

What we see is:

  • The median filter cleans up most (but not all) the artifacts, even run 10 times. But it doesn't leave any significant chroma smearing. As expected.
  • The mean filter cleans up all the artifacts, run 5 times. But it has chroma smearing that's very similar to SILKYPIX(!). In fact, at this point PhotoRaw plus a mean filter starts to look nearly indistinguishable from SILKYPIX - a normal user would be unlikely to spot any difference.
  • The remaining difference between SILKYPIX and PhotoRaw plus the mean filter is that SILKYPIX's mean filter appears to only be being applied close to edges. In addition, there appears to be some local saturation adjustment going, probably to counteract the saturation reduction that mean filtering would cause. If you look closely, the pixels right next to the lettering on the SILKYPIX image are smeared very similarly to the PhotoRaw plus image. But a few pixels away, SILKYPIX has gone back to an unfiltered version. The intent here would be to  give SILKYPIX the best of both worlds - artifact suppression, but relatively little chroma smearing away from edges.
  • ACR also appears to be using some form of mean filter, which also seems to be being applied near the edges, probably again with some kind of saturation adjustment causing the odd lightening effect. But ACR is doing so in a very heavy-handed way. On the face of it, it looks rather as if the ACR engine doesn't much like the X-Pro1's sensor pattern, and the camera raw team have just brute force filtered out any artifacts. Hopefully the team will be able to improve this in a final release. (Update - they didn't - see this post)

Conclusion

Firstly, the addition of mean filtering to the new PhotoRaw "generalized AHD" engine brings it very close in performance to the Fuji recommended raw developer as regards the X-Pro1. Not quite as good, but "PhotoRaw plus" now rates as the second best (and only by a whisker!) raw converter for the X-Pro1. Also, there's a clear development path (edge sensitive mean filtering rather than mean filtering everywhere) to get it all the way.

Secondly, it seems very likely that the way that both SILKYPIX and ACR are handling the X-Pro1 sensor is by some form of mean filtering. In the case of SILKYPIX, it seems well tuned to the sensor. ACR, not so much. But SILKYPIX had the advantage of Fuji help prior to the camera's release.

Finally, there doesn't seem to be any major "secret sauce" to what Fuji/SILKYPIX are doing. In a way, this is somewhat discouraging as it implies that there isn't anything more than what we've seen to be extracted from the sensor.

The reason why mean filtering, rather than median filtering is being used is probably two things. Firstly, as we've seen, mean filtering is better at removing the artifacts that the X-Trans sensor is prone to. Secondly, mean filtering is a lot faster than median filtering - no sample ordering step.

The inevitable question is, where does this leave PhotoRaw? Sadly, a version of PhotoRaw using the mean filtering technology above won't be coming to an App Store near you. It's too slow. This is where the difference between a desktop raw processor and a iPad raw processor shows.

What I will be doing however is looking at whether it is possible to implement an "artifact sensitive" mean filter that only filters where it has to. This would be somewhat similar to what SILKYPIX and Adobe seem to be doing, but would be focussed on improving speed by processing far fewer pixels more than on image quality. Such a version might be fast enough to make the cut into a release version of PhotoRaw.

UPDATE: It's probably the case that while filtering does cause the chroma smearing, it's not median or mean filtering - see part 4 here.

19 comments:

  1. Thank you for this very informative series of posts. While I admit reading them in the hope that you'd find the panacea that rendered the x-trans as useful as hoped, I enjoyed the insight into the problems posed by such an unconventional sensor design. All the best with your future development. Cheers, Ben

    ReplyDelete
  2. Indeed thanks for your very informative posts and your determination to understand how to get optimum results out of this image sensor!

    It really helps me to understand what the strenghts and compromises of this design are (this information is otherwise hard if not impossible to get). It seems that you win some in sharpness and moire resistance by this new design but part of that advantage is lost by limitations in demosacing the RAW data.

    Anyway it's good to see support coming for this really nice camera!

    Regards,

    Alfred

    ReplyDelete
  3. Very informative set of articles.
    Nice to see that someone can actually take the time to explain the complexities of these matters in a clear and understandable way. Thanks very much.
    Jonathan

    ReplyDelete
  4. As others have said, Thank You! I've enjoyed reading your explanations of what is to me, black magic!

    ReplyDelete
  5. Hi Sandy,
    thank you for the informative read. I too thought that aside of lack of AA filter there's really no advantage to X-trans vs. Bayer layout but then something else came to mind:
    in the RGB world the green channel is by far the most important for overall image quality, and obviously green photosites are over-represented in X-Trans pattern. Next comes red channel, and again red pixels are arranged in somewhat regular angled grid which should be a good thing; lastly blue channel is the least important by far (and gets mangled up by jpeg compression the most) and in X-Trans layout the blue is a real red-headed stepchild. So in a lot of ways X-Trans kinda amplifies the RGB ratio already present in a Bayer layout.
    Based on the above I think X-trans output has a potential to deliver better-looking RGB images, processing challenges nonwithstanding.
    Lastly, on a totally separate subject - and I am sorry in advance about a noob question - but your post about demosaicing reads as if the challenge is simply to "fill the holes" - ie. you would process R,G,B channels separately, based on corresponding photosites, and then apply mean/median to a composite image... I wonder if anyone ever came up with an algorithm that would build a perfect monochrome image from all 3 channels? Then a less accurate chroma channel can be constructed, and the final conversion to RGB can be an improvement over the traditional, per-channel approach...

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. (With typo corrected) I substantially simplified the demoasicing discussion - the way the AHD algorithm (and most others) work is by first creating as near perfect a green image (which is quite close to mono) as possible, using both the green and the red/blue pixels. Then red/blue pixels are filled in, based on real and interpolated green pixels, as well as real red/blue pixels.

    So what is (typically) being done is relatively close to what you suggest.....

    In the X-Trans pattern, the green pixels are indeed over-represented related to a normal Bayer patter. In principle, this should allow you to a better job on the first stage green interpolation, which should then flow through to a better red/blue interpolation. In most of the demosaicing algorithms that follow that sort of a model, the greatest determinant of how good your red/blue interpolation is going to be, is how good the green interpolation was. In practice, the uneven grid spacing spacing seems to negate the X-Trans' theoretical "more green pixels" advantage. At least, that's my experience

    ReplyDelete
  8. Enjoyed the informative articles too.

    I'm just curious as to the relative total number of photosites (of all colors) between bayer and XTrans 16MP sensors - is there a significant difference? I haven't found any explicit comparisons of the number of photosites, despite many explanations of the differences in the pixel array.

    ReplyDelete
  9. Within a 6x6 square, a conventional Bayer array would have 18 green photosites, 9 red photosites and 9 blue photosites. The X-Trans sensor has 20 green photosites, 8 red photosites and 8 blue photosites in the same space.

    ReplyDelete
  10. Thanks for the quick reply, and forgive me for one more question: in that typical 6x6 sensor area are the same amount of effective pixels produced for both sensor arrays? The impression I had was that the XTrans takes 36 photosites to make one pixel, whereas the bayer takes only four. This would require many more photosites for the XTrans sensor.

    ReplyDelete
  11. For both the conventional Bayer array and the X-Trans array, each individual photosite becomes one pixel in the final image. The magic of interpolation(!)

    ReplyDelete
  12. Like imagefoundry, I also wondered why you couldn't just take each photosite and make a simple calculation for exposure value (to compensate for the different light levels of each color) and end up with an uninterpolated monochrome image. In other words, G=recorded EV value, R is recorded value boosted by three stops (whatever the actual difference in EV value compared to green), B is boosted by two stops, etc.

    Using all three colors should give you a similar resolution to the new Leica Monochrome, and you'd still have an interpolated color image to overlay with for filter effects. This is where you tell me it's more complicated than that...

    ReplyDelete
  13. Well, take for example an image of a green leaf on a white page. The white page measures as green photosites 100, red photosites 50, blue photosites 20. So green is twice red.

    Now let's look at the leaf. For simplicity, let's say the green photosite measures 100, and the red and blue blue photosites measure 0. (Because its a green leaf!).

    Multiple the red value by 2, and you still have zero. So the mono image would be have speckles - what should be a continuous tone is actually 100, 0, 0

    The problem is that the Bayer array filter is cutting out light in a way that can't be simply recovered.

    ReplyDelete
  14. So, that idea only works for completely neutral gray scenes. Thanks for your patience explanations.

    ReplyDelete
  15. Very interesting.

    I do not use any of these - I use either Aperture or Capture One Pro, neither of which support the XP1 yet. I look forward to when iOS apps can run on the desktop so that I can try yours, as I don't fancy editing on my iPhone which is the only iOS device I have.

    It won't be long before Apple make the apps work in both platforms, I am sure.

    I hate Silky Pix - really horrid interface. Better if you spring for the paid version, but somehow that sticks in my throat!

    ReplyDelete
  16. Looks like you should use some sort of pattern recognition in order to avoid artifacts.

    ReplyDelete
  17. Denis, Adobe may already being that, or something related. especially in the luma channel; the diagonals are a bit "too clean". But pattern recognition needs huge amounts of processing power to avoid problems of its own. There are also specialist interpolation algorithms designed for irregularly spaced data. But again, they take huge amounts of processing power.

    ReplyDelete
  18. Just a quick thought, but what if you combined median and mean filtering in your code -- perform the mean over the kernel but pick the pixel value closest to it in the kernel. (Just from my quick thinking, this may take slightly longer than mean filtering but on average be shorter than median filtering but probably end up with results closer to the later, perhaps?)

    Also, I'm not sure if this is done, as I haven't really looked at the software for raw sensor conversion, but a dictionary code system for the sensor-pattern vector might be promising? (In fact, if you could optimize one for each type of sensor-site pattern on the Xtrans, it may improve both performance and speed -- of course, it would take a decent bit of work to assemble such a dictionary. :-)

    ReplyDelete
  19. Eric,

    I have in fact played with a hybrid mean/median filter. Unfortunately, performance wasn't very good - it tends to latch onto demosiaced intermediate values.

    Pattern sensing might well be a workable option - the problem is that for the X-Trans, I think that it would require a huge number of patterns, and would chew up a huge amount of processing time in pattern recognition.

    ReplyDelete