Bayer Patterns in Digicam CCDs

There's been some discussion recently about the color technology of CCD (charge-coupled device) sensors used in most digital cameras. In particular, the issue of the true resolution of the CCD versus the resolution of the image that it produces has arisen. To understand why these figures are different, you need to know a little about how the CCD in a camera works.

The CCD is a grid of light-sensitive cells representing pixels, 1600 x 1200 of them for the currently in-vogue 2.1M pixel cameras. Each cell is around 5 microns square. Normally we think of a pixel as representing any of the possible colors in an image. For example, whenever you display an image on a Windows TrueColor display, you have 24 bits of color information per pixel (eight each for the red, green and blue components), giving you 16M (where M is 1024 x 1024) colors. That's not how CCDs work, however. The pixel sensors in a CCD are essentially monochromatic. That means that they respond primarily to the intensity of the light falling on them, regardless of the frequency (color) of the light. To make them sensitive to different colors, a filter is placed over each pixel during manufacturing. The sensor is one of the primary colors: red, green or blue1. Since a given pixel location can only have one filter, you can see that in a given CCD, roughly a third of the pixels are used for each color. Another way of putting this is that the full-color resolution of the device is around a third of the number of pixels that is quoted in the specs.

So how are these filters laid out? The first thing to know is that the human eye is more sensitive to green light than to red or blue. It therefore makes sense to devote more pixels to green than the other two colors. Since pixels are discrete elements, we need to think in terms of integer multiples, and the most appropriate relationship is that there are twice as many green pixels as red or blue (or, to put it another way, the total number of greens is the same as the reds and blues combined). Now we just have to come up with a nice geometric pattern to distribute the filtered pixels appropriately. The pattern chosen is the so-called Bayer pattern, and it looks like this:

Every other pixel is a green one. On even numbered rows and columns (counting from zero), the alternate pixels are red, and on odd numbered rows and columns, the alternate pixels are blue.

When you take a picture with a camera that uses a CCD configured as shown above, a set of intensity values is read out from the device. The first value, corresponding to the top-left pixel in the picture, is the red intensity for that pixel. The next value is the green intensity for the pixel to the right of it, and so on until all of the values have been read. Typically the values are read as 8 or 10-bit numbers, so have a range from 0 (no light) to 255 or 1023 (maximum brightness).

The only problem is that we only have a third of the information we need to display a meaningful image. For example, for the top left pixel we have a red, but no green or blue. For the next pixel we have a green, but no red or blue. These 'missing' values are recovered by a technique called interpolation, which is a fancy term for 'guessing'. Say we want to find the green value for the red pixel in the middle of the square above (the one with the white outline). We can see that it's surrounded above, below, left and right by green pixels, so we take the average of those four green and assign that to the green level for our red pixel. Of course, this isn't necessarily the true value, because we have no way of knowing what the green level really was at that point. As we'll see, this can lead to problems in areas of sharp color change.

You can use a similar approximation technique to obtain the missing red and blue colors too. In terms of a C-like programming language, the algorithm for the pixel at (x, y) is:

switch (x % 2 + (y % 2)*2)  {
    case 0: //red
        red = red[x][y];
        green = (green[x][y-1] + green[x-1][y] + green[x+1][y] + green[x][y+1]) / 4;
        blue = (blue[x-1][y-1] + blue[x+1][y-1] + blue[x-1][y+1] + blue[x+1][y+1]) / 4;
    break;
    case 1: //green on RG row
        red = (red[x-1][y] + red[x+1][y])  / 2;
        green = green[x][y];
        blue = (blue[x][y-1] + blue[x][y+1]) / 2;
    break;
    case 2: // green on GB row
        red = (red[x][y-1] + red[x][y+1]) / 2;
        green = geen[x][y];
        blue = (blue[x-1][y] + blue[x+1][y]) / 2;
    case 3: //blue
        red = (red[x-1][y-1] + red[x+1][y-1] + red[x-1][y+1] + red[x+1][y+1]) / 4;
        green = (green[x][y-1] + green[x-1][y] + green[x+1][y] + green[x][y+1]) / 4;
        blue = blue[x][y];
    break;
}
Unfortunately, not many digicams let you see the raw output from the CCD. Most of them process it into an RGB format (usually TIFF or JPEG) before storing it into the camera's flash memory. In order to experiment with Bayer patters, I wrote a couple of filters for Photoshop. These work on any RGB image. The first filter, Bayerize, converts the image to one that uses a Bayer pattern. It does this very simply by setting the color components to zero in pixels where the Bayer pattern doesn't use them. The second filter, Debayerize 1, takes an image in Bayer format and interpolates the missing colors are described above. This gives you a good idea of how a simple color-recovery algorithm works.

You can download the filters by clicking here. Unzip the files into your Photoshop\Plug-Ins\Filters directory, and restart Photoshop. They will show up in the Filter | Bayer Pattern menu.

The Filters In Action

Below I give some examples of what you see using the filters. This should be of interest to those who don't have Photoshop, and it also illustrates exactly what happens during the de-Bayerizing process.

The first example is an image scanned from a photo my lovely wife, Xiaoli (pronounced 'Jennifer'). It was actually scanned from a negative using the HP Photosmart scanner (the original model). The section of the photo I chose has quite a lo of detail, and therefore some rapid color transitions. Here's the original, along with an enlarged detail:

Next we see what happens when the Bayerizer filter is applied:

The Bayer pattern is very clear in the detail image, and the normal-sized image shows a distinct green cast, because of the relatively large number of green pixels. Finally, here's the de-Bayerized version:

The overall picture doesn't look bad, through there's a noticeable loss of resolution (not surprising given that we threw 66% of the information away!), but the enlargement shows distinct color artifacts when compared to the original.

Part II: Grayscale images and Single Pixels

1Some cameras, notable the Canon Pro70, use a subtractive color filtering scheme, and use Cyan, Magenta, Yellow and Green filters.

Home