Fixing a bad column inside an image ?

[wilbert]

I have a digital camera with a CCD sensor which appears to have a column defect (there's a small vertical line on every image which is brighter compared to the surrounding pixels).
Unfortunately the camera doesn't output raw images and it seems to do some processing before outputting an image.
As a result of this, the lines are not exactly straight and the horizontal offset can vary a bit depending on what zoom factor is used.



Does anybody have an idea how to fix the images the camera outputs ?
I tried manually in Photoshop with a clone brush but that's a lot of work.
Ideal would be if an application could detect the brighter line and replace it with interpolated pixels from the left and right.
Unfortunately I have no idea how to detect the line.
The black images above are clear but in a normal photo, the line can be subtle and is mainly visible in parts of the image which are dark.

[#NULL]

Its not exactly a column, but rather a curve spanning multiple columns. Do you have an image containing other content too to see how the lines looks then? is it always bright/white?
I had a little test to detect the line, but maybe others have a real solution, for example using some image processing library.

Code: Select all

EnableExplicit
InitNetwork()

Define url.s = "https://www.w73.nl/pb/zoom_out.jpg"
;Define url.s = "https://www.w73.nl/pb/zoom_middle.jpg"
;Define url.s = "https://www.w73.nl/pb/zoom_in.jpg"

Define filename.s = GetTemporaryDirectory() + "_pb_test_tmp_img.jpg"

Procedure.i absi(v)
  If v < 0
    ProcedureReturn -v
  EndIf
  ProcedureReturn v
EndProcedure

If ReceiveHTTPFile(url, filename)
  UseJPEGImageDecoder()
  Define img = LoadImage(#PB_Any, filename)
  If IsImage(img)
    StartDrawing(ImageOutput(img))
    Define xmax = OutputWidth() - 1
    Define ymax = OutputHeight() - 1
    Define maxDiffSum.f = 0
    Define xLine = -1
    Define x, y
    For x=0 To xmax
      Define diffSum.f = 0
      For y=0 To ymax
        Define color
        Define r, g, b, a
        Define r_, g_, b_, a_
        color = Point(x, y)
        r = Red(color)
        g = Green(color)
        b = Blue(color)
        a = Alpha(color)
        If y > 0
          diffSum + ( (absi(r - r_) + absi(g - g_) + absi(b - b_) + absi(a - a_)) / 4.0 )
        EndIf
        r_ = r
        g_ = g
        b_ = b
        a_ = a
      Next
      If diffSum > maxDiffSum
        maxDiffSum = diffSum
        xLine = x
      EndIf
    Next
    If xLine >= 0
      ;LineXY(xLine, 0, xLine, ymax, $ff0000ff)
      LineXY(xLine-4, 0, xLine-4, ymax, $ff0000ff)
      LineXY(xLine+4, 0, xLine+4, ymax, $ff0000ff)
    EndIf
    StopDrawing()
    
    Define w = ImageWidth(img)
    Define h = ImageHeight(img)
    Define win = OpenWindow(#PB_Any, 50, 50, w, h, "..")
    AddKeyboardShortcut(win, #PB_Shortcut_Escape, 10)
    Define ig = ImageGadget(#PB_Any, 0, 0, w, h, ImageID(img))
    
    Repeat
      WaitWindowEvent()
    Until Event() = #PB_Event_CloseWindow Or Event() = #PB_Event_Menu
  EndIf
EndIf

[wilbert]

Its not exactly a column, but rather a curve spanning multiple columns. Do you have an image containing other content too to see how the lines looks then? is it always bright/white?

Thanks for your code. It works on the black image but unfortunately not on a normal image.

Here's a more normal image.


To be honest, the line is very subtle.
The problem is, that it is the right side image from a stereo camera.
So even if it is subtle, if you see a vertical line with one eye and not with the other, it will be distracting.

[#NULL]

Thanks for your code. It works on the black image but unfortunately not on a normal image.

Yeah, that's what I would have thought. Now with the new image I tried to find the line with the least variations between adjacent pixel, instead of the most variation. But that doesn't either. I have a couple of other ideas though.

[wilbert]

I have a couple of other ideas though.

That would be great.

The exif data of the original image does contain a FocalLength value so in theory I guess, it might also be possible to calculate the position of the curve based on the FocalLength value but I have no idea about the relation between the two. So an image detection based solution is probably the best.

[Little John]

I'm thinking of some code that I wrote some time ago, which might bei usful for solving this problem. Currently I'm travelling, and I'll test that when I'm home again.

A spontaneous idea:

• Only deal with the right half of the picture.
• For each row, store the positions of all pixels whose color ist brighter than the colors of its both neighbours.
• Check which of these positions are equal in all rows.

[#NULL]

Maybe you can try this. I did not tidy up the code yet and there are some things that might not be needed, because I checked for every hit how many other hits are directly above and below, so that I can weight them. But it was just much easier to simply count how many hits there were for a particular x-column and use the x with the most hits. The green box should indicate were the line is.
As for the actual detection:
For each y-row I find the x where the difference is the greatest between the average of its left/right pixels and the average of its top/bottom pixels. The idea is that at the line the horizontal brightness gradient (to the left and right pixel neighbors) should be large, and the gradient to the top/bottom neighbors should be small. So the difference between these gradients is an indicator for being at the line.

Code: Select all

EnableExplicit
InitNetwork()

;Define url.s = "https://www.w73.nl/pb/zoom_out.jpg"
;Define url.s = "https://www.w73.nl/pb/zoom_middle.jpg"
;Define url.s = "https://www.w73.nl/pb/zoom_in.jpg"
Define url.s = "https://www.w73.nl/pb/vline.jpg"

Define filename.s = GetTemporaryDirectory() + "_pb_test_tmp_img.jpg"

Procedure.i absi(v)
  If v < 0
    ProcedureReturn -v
  EndIf
  ProcedureReturn v
EndProcedure

Procedure.f colorToValue(color)
  Static r, g, b, a
  r = Red(color)
  g = Green(color)
  b = Blue(color)
  a = Alpha(color)
  ProcedureReturn (1.0*r + 1.0*g + 1.0*b) / 3.0
EndProcedure

If ReceiveHTTPFile(url, filename)
  UseJPEGImageDecoder()
  Define img = LoadImage(#PB_Any, filename)
  If IsImage(img)
    StartDrawing(ImageOutput(img))
    Define xmax = OutputWidth() - 1 - 1
    Define ymax = OutputHeight() - 1 - 1
    Dim diffMax(ymax)
    Define x, y
    For y=1 To ymax
      Define diffMax.f = 0
      Define diffMaxX.f = -1
      For x=1 To xmax
        Define lef.f = colorToValue(Point(x-1, y))
        Define rig.f = colorToValue(Point(x+1, y))
        Define top.f = colorToValue(Point(x, y-1))
        Define bot.f = colorToValue(Point(x, y+1))
        Define diff.f = Abs(0.5*lef + 0.5*rig) - (0.5*top + 0.5*bot)
        If diff > diffMax
          diffMax = diff
          diffMaxX = x
        EndIf
      Next
      diffMax(y) = diffMaxX
    Next
    
    Dim diffMaxCount(ymax)
    For y=1 To ymax
      If (y>1) And (diffMax(y) = diffMax(y-1))
        diffMaxCount(y) + 1
      EndIf
      If (y<ymax) And (diffMax(y) = diffMax(y+1))
        diffMaxCount(y) + 1
      EndIf
      If (y>2) And (diffMax(y) = diffMax(y-2))
        diffMaxCount(y) + 1
      EndIf
      If (y<(ymax-1)) And (diffMax(y) = diffMax(y+2))
        diffMaxCount(y) + 1
      EndIf
    Next
    
    Dim xCount(xmax)
    For y=0 To ymax
      xCount(diffMax(y)) + 1
    Next
    
    Define maxCount = 0
    Define maxCountedX = -1
    For x=0 To xmax
      If xCount(x) > maxCount
        maxCount = xCount(x)
        maxCountedX = x
      EndIf
    Next
    
    DrawingMode(#PB_2DDrawing_Outlined)
    For y=1 To ymax
      Box(diffMax(y)-1, y-1, 3, 3, $ffffffff)
      If diffMaxCount(y) > 3
        Box(diffMax(y)-6, y-6, 13, 13, $ff0000ff)
      EndIf
      Box(maxCountedX-10, 0, 21, OutputHeight(), $ff00ff00)
    Next
    
    StopDrawing()
   
    Define w = ImageWidth(img)
    Define h = ImageHeight(img)
    Define win = OpenWindow(#PB_Any, 50, 50, w, h, "..")
    AddKeyboardShortcut(win, #PB_Shortcut_Escape, 10)
    Define ig = ImageGadget(#PB_Any, 0, 0, w, h, ImageID(img))
   
    Repeat
      WaitWindowEvent()
    Until Event() = #PB_Event_CloseWindow Or Event() = #PB_Event_Menu
  EndIf
EndIf

[JHPJHP]

Hi wilbert,

Since you were a great help in bringing OpenCV to the forum, it's only right that I provide an OpenCV solution

HoughLines.pb shows one way to find the position of a line; you will see in the next script how this information might be used.
- only tested against the image you provided, and will probably need some tinkering for a more general use

InPaint.pb repairs images with small defects. Using the image you provided, a repaired version of the image is saved to the current folder.
- the only requirement was knowing the X position of the line; hard coded for this example

This was just a 20 minute cut and paste job from PureBasic Interface to OpenCV.
OpenCV offers numerous algorithms for line detection; a lot more could have been done with a little time and effort.

[BarryG]

To be honest, the line is very subtle.

Subtle? It's the total opposite of subtle. You can see this line a mile away; even for myself without my glasses.

I assume the camera is no longer under warranty, or you wouldn't be asking how to fix the line with software? Have you looked into how much it'd cost to repair it? Might be cheaper and more convenient than constantly trying to fix every photo with an app.

Or, since it's a stereo camera, there must a software that can take the same area from the left image and paste it over the right image? That works (I tried it with a stereo image manually) but I don't know of any tools that can do it automatically.

[Michael Vogel]

My first thoughts were also to have a look into the Exif Data - depending on the number of prism in the lens you maybe need only 8 to 10 shots to create an approximate formula for the "exact" position (or at least to calculate a range of some view pixels where to search for the line).

Maybe you can upload also some original images somewhere to have a closer look...

[Mijikai]

Is there a raw format of the image?
The jpg already messes up the pixels to a state where it is much more difficult to repair.

[wilbert]

@#NULL, Thanks. Both your solution and the one of JHPJHP detect the location of the line.

@JHPJHP, Thanks. The InPaint result looks great !!
Could OpenCV also be used to convert a curve on an image into a mathematical representation ?
Since the line can be curved, ideal would be if it could be represented by a mathematical formula.

@BarryG, It's a second hand FinePix REAL 3D W1 of about 10 years old. I bought it from someone in a different country.
Both sending it back or letting it be repaired would be relatively expensive. That's why I'm looking for a software solution.
When objects are at different distances, the left and right image are too different to just copy and paste a part from the left image.

@Mijikai, Unfortunately it doesn't output raw images.

@Michael Vogel, Unfortunately I have no knowledge of optics
If you know how to calculate the position of the line, that would be ideal.

Here's 10 images at real size (3648 x 2736 px) at different focal lengths.
https://www.w73.nl/pb/3DW1_png.zip

Code: Select all

FocalLength   Range x
(mm)	

 6.3          2804 - 2816   (
 7.1          2862 - 2865   (
 8.0          2874          |
 9.1          2889          |
10.3          2916          |
11.6          2937 - 2939   )
13.1          2957 - 2961   )
14.8          2949 - 2953   )
16.7          2932 - 2938   )
18.9          2904 - 2910   )

[Mijikai]

Heres some code to find the general location within a few pixles.
(Tested with all images provided)

Code:

Code: Select all

EnableExplicit

UseJPEGImageDecoder()
UseJPEGImageEncoder()

Global img.i
Global hdc.i
Global w.i
Global h.i
Global x.i
Global y.i
Global min.i
Global pos.i
Global Dim offset.i(0)

Macro ColorDifference(a,b)
  (((Red(a) + (Green(a) << 1) + Blue(a)) >> 2) + ((Red(b) + (Green(b) << 1) + Blue(b)) >> 2) >> 1)
EndMacro

img = LoadImage(#PB_Any,"vline.jpg")
hdc = StartDrawing(ImageOutput(img))

If hdc
  w = OutputWidth()
  h = OutputHeight()
  Dim offset(w - 1)
  For x = 0 To w - 1
    For y = 1 To h - 1
      offset(x) + ColorDifference(Point(x,y - 1),Point(x,y))
    Next
  Next
  For x = 1 To w - 1
    If offset(x) < min Or (offset(x) > 0 And min = 0);<- find the smallest difference
      min = offset(x)
      pos = x
    EndIf
  Next
  
  Debug pos
  Debug min  
  
  For y = 0 To h - 1
    Plot(pos,y,#Red)
  Next
  StopDrawing()
  SaveImage(img,"vline_fix.jpg",#PB_ImagePlugin_JPEG)
EndIf

End

Not sure if it helps as i have no idea what to do from here.
Edit: Just noticed #NULL posted an even better way

[Joris]

My 50 cents...

What about inverting left and putting together with right image to easier find that line/curve ?

[#NULL]

In case the line is curved too much, my last code could be adapted to process the image in horizontal segments/stripes. so you would get the x-position of the line for y 0-10, then the x-position of the line for y 10-20 etc.