Fixing a bad column inside an image ?

[NicTheQuick]

You could use something like this: https://www.nvidia.com/research/inpainting/

You can find lots of codes online, e.g. here: https://github.com/leehomyc/Faster-High ... Inpainting
Or for a bigger overview you can look here: https://paperswithcode.com/task/image-inpainting

[wilbert]

When I opened this thread, I assumed the camera has a free zoom (It has a zoom in / out button).
But when I just tried to take photos on purpose with a different focal distance compared to the reference images I posted, it didn't work.
It seems the zoom always works in steps and not continuously. So it looks like I can use 10 reference images each containing a curve for a specific focal distance.
That reduces the problem to following the curve of the reference image and finding the best inpaint solution.
The OpenCV example JHPJHP posted looks good but I'll also look into the other suggestions posted.

[JHPJHP]

Hi wilbert,

It sounds like you have this sorted, but just in case...

The previous 10 images you posted should work well as mask's for the InPaint example.
Matching the correct mask to the image should be a simple calculation.

If you need any help with OpenCV, please don't hesitate to ask.

JHPJHP

[Michael Vogel]

Just a remark about the line position - best would be to know the lens internals, next possibility is to do some math (Mr. Gauss & Horner as I remember), but there's also a simple way to get an impression for a curve passing given points: try a spreadsheets "Trendline X/Y":
Function Plot

[wilbert]

Here is an updated version of my previous download using one of the images you posted as an InPaint mask:

Thanks for the example.
I improved the quality of the line images. They are now in png format.
Next step is to take a look at the .mpo files the camera outputs. A mpo consists of two jpeg images and it also contains the exif data I need to determine the focal length that was used to take the picture.

Just a remark about the line position - best would be to know the lens internals, next possibility is to do some math (Mr. Gauss & Horner as I remember), but there's also a simple way to get an impression for a curve passing given points: try a spreadsheets "Trendline X/Y":

It looks like I can do with 10 reference images so in that case I don't need to calculate the line position; only get the focal length from the exif data and choose the appropriate image.

[#NULL]

If you need it I can try to improve my line detection code to get the curve as well. Maybe to create a mask image that you can then use with the inpaint method?

[wilbert]

If you need it I can try to improve my line detection code to get the curve as well. Maybe to create a mask image that you can then use with the inpaint method?

The images I got at the moment of the lines are working pretty good as a mask.

[wilbert]

I did make some progress.
I can get the left and right images and the focal length that was used to take the picture programatically.
The corresponding mask fits perfectly.

Here's an example of a photo at the original size and its mask
https://www.w73.nl/pb/ImageAndMask.zip
The photo was taken with ISO 100 when the line is least visible.

The thing I'm questioning at the moment is the use of inpainting.
The results are pretty good but it seems a bit of a shame to throw away useful information.
While the pixels on the affected line are too bright in dark areas, they still contain useful information about color.
The information that is still contained in those pixels, could help to make a better decision of what the actual pixel value should be I guess.

[davido]

@wilbert,
I hope you don't mind a rather naif suggestion:
If this is a fault with the hardware, surely the value of the affected pixels would be reproducible?
This could be tested from examining different black images, perhaps.
If this was so, then the line could be located by software.

[fabulouspaul]

I fiddled around with the image given above and found a solution at least for this image.

The thought behind is, that a whole column is too bright, so that its brightness should peek out if compared to the column before.
When found such a column, calculate the average of each pixel in the previous column and the too bright column and replace the pixels.

Code: Select all

; Recover bright pixel-column

EnableExplicit

UseJPEGImageDecoder()

Enumeration 
  #image
EndEnumeration

Declare.i col_brightness(image.i, column.i)

Global file.s
Global width.i
Global height.i
Global i, j
Global r, g, b
Global color_prev_col.i
Global color_cur_col.i
Global brightness_cur_col.i
Global brightness_prev_col.i

file = OpenFileRequester("Choose JPG...", "", "JPG-File|*.jpg", 0)

If file = ""
  End
EndIf

If LoadImage(#image, file)
  width = ImageWidth(#image) - 1      
  height  = ImageHeight(#image) - 1
  
  For i = 0 To width
    ; get brightness of current column
    brightness_cur_col = col_brightness(#image, i)
    
    ; no previous column
    If i = 0
      brightness_prev_col = brightness_cur_col
    EndIf
    
    ; check for brightnessdifference
    If Abs(brightness_cur_col - brightness_prev_col) > 4
      
      Debug RSet(Str(i), 6, " ") + " : " + RSet(Str(brightness_cur_col), 3, " ") + " (" + RSet(Str(brightness_prev_col), 3, " ") + ")"
      
      StartDrawing(ImageOutput(#image))
      For j = 0 To height
        ; get color of current pixel and pixel in previous column
        color_prev_col = Point(i - 1, j)
        color_cur_col  = Point(i, j)
        
        ; calculate average and darken it by 10%
        r = ((Red(color_cur_col) + Red(color_prev_col)) / 2)     * 0.9
        g = ((Green(color_cur_col) + Green(color_prev_col)) / 2) * 0.9
        b = ((Blue(color_cur_col) + Blue(color_prev_col)) / 2)   * 0.9
        
        Plot(i, j, RGB(r, g, b))
      Next
      StopDrawing()
      
      Debug "  -> " + RSet(Str(col_brightness(#image, i)), 3, " ")
      
    EndIf
    
    brightness_prev_col = brightness_cur_col
  Next
  
  file = SaveFileRequester("Save recovered image as .BMP...", "", "BMP-File|*.bmp", 0)
  If file <> ""
    SaveImage(#image, file)
  EndIf
  
Else
  Debug "Error loading image " + file
  End
  
EndIf

End

Procedure.i col_brightness(image, column)
  ; calculate average brightness of a pixel-column
  Protected width.i
  Protected height.i
  Protected brightness.f
  Protected color.i
  Protected i
  
  width  = ImageWidth(image) - 1
  height = ImageHeight(image) - 1
  
  ; to prevent errors
  If column > width Or column < 0
    ProcedureReturn -1
  EndIf
  
  StartDrawing(ImageOutput(image))
  For i = 0 To height
    color = Point(column, i)
    
    ; calculate brightness and adjust by human reception
    brightness = brightness + (0.299 * Red(color) + 0.587 * Green(color) + 0.114 * Blue(color))
  Next i
  StopDrawing()
  
  ProcedureReturn brightness / height
EndProcedure

[wilbert]

If this is a fault with the hardware, surely the value of the affected pixels would be reproducible?
This could be tested from examining different black images, perhaps.
If this was so, then the line could be located by software.

It is a hardware fault.
Making black images and detecting the line on that black image is exactly what I did.
At first I thought about using them as mask images but later I realized that it would be easier to define things using simple data.
For focal distance 10.3 mm for example, I now have something like

Code: Select all

Data.w 280, 2915,  2176, 2916,  280, 2915,  #Null

Which means 280 times at x position 2915, 2176 times at position 2916 and 280 times again at position 2915.
The total times is always 2736 which is the height of the images.

I fiddled around with the image given above and found a solution at least for this image.

It does indeed make things better. I'll look into the approach as a possible alternative to inpainting.

[JHPJHP]

Hi wilbert,

I just read one of your earlier posts...

The problem is, that it is the right side image from a stereo camera.

I don't know if the following will be of any help, but it may point you in another direction?

Without the benefit of a left side image
Copied main.jpg to compare.jpg.
Red line drawn down the height of main.jpg simulating the defect.

Using OpenCV
Create a template image of the area around the defect in main.jpg.
Match the template image to the area in compare.jpg.
Merge the compare.jpg area with the matching main.jpg area.

[wilbert]

I don't know if the following will be of any help, but it may point you in another direction?

Unfortunately it doesn't work very well on the original images. Here's a zip with both the left and right images without any processing done to them.
https://www.w73.nl/pb/ImagesAndMask.zip
One problem is that the images are a bit different because of the depth of objects, another problem I noticed is that for the left and right sensor also the lighting isn't exactly the same.

[#NULL]

I tried harder and now get better results. But I only have 2 images for testing (the one with the bushes and the one with the tree). Some boxes indicate the detection/line if the debugger is enabled. If disabled ony the result is shown.
The line x-position is first detected approximately for the whole image, then more precisely for separate y-segments (to accout for the curving) and only in an narrow x-range around the approximate position. Then brightness of the line is adapted to the surrounding pixels and neighbor pixel colors are blended into the line etc.
Some artifacts will still be left. In some areas in the tree image when there is not really a line to be detected it will detect other parts of the image and create some artifacts instead of removing them. But overall I have to look very closely to still find the line in the processed image and I quite know what to look for. Maybe give it a try. I didn't try the inpaint so I don't know how it compares to that.
With debugger and a large image it will take some time, because I didn't care about performance yet.

Code: Select all

EnableExplicit
InitNetwork()

Define img
Define url.s = ""
Define filename.s = ""
Define tmpFile.s = GetTemporaryDirectory() + "_pb_test_tmp_img.jpg"

Select 5
  Case 1 : url = "https://www.w73.nl/pb/zoom_out.jpg"        : filename = tmpFile
  Case 2 : url = "https://www.w73.nl/pb/zoom_middle.jpg"     : filename = tmpFile
  Case 3 : url = "https://www.w73.nl/pb/zoom_in.jpg"         : filename = tmpFile
  Case 4 : url = "https://www.w73.nl/pb/vline.jpg"           : filename = tmpFile
  Case 5 : filename = "/media/user/USB16/pb2016/tests/detectLineInImage/ImageAndMask/DSCF0024_r.jpg" ; (tree)
  Case 6 : filename = "/media/user/USB16/pb2016/tests/detectLineInImage/ImageAndMask/FL6_3.png"
  Case 7 : filename = "/media/user/USB16/pb2016/tests/detectLineInImage/vline.jpg"                   ; (bushes)
EndSelect

CompilerIf #PB_Compiler_Debugger
  Define debugApproxLineBox  = #True
  Define debugSegmentLineBox = #True
  Define debugLinePixel      = #True
CompilerElse
  Define debugApproxLineBox  = #False
  Define debugSegmentLineBox = #False
  Define debugLinePixel      = #False
CompilerEndIf

Procedure loadImageFile()
  Debug "loading image"
  Shared img, url, filename
  If (url And ReceiveHTTPFile(url, filename))  Or  ((Not Bool(url)) And filename)
    UseJPEGImageDecoder()
    UsePNGImageDecoder()
    img = LoadImage(#PB_Any, filename)
  EndIf
  If Not IsImage(img)
    MessageRequester("error", "image not loaded")
    End
  EndIf
EndProcedure

Procedure.f getColorBrightness(color)
  Protected r.f, g.f, b.f
  r = Red(color)
  g = Green(color)
  b = Blue(color)
  ProcedureReturn (r + g + b) / 3.0
EndProcedure

Procedure.i setColorBrightness(color, brightness)
  Protected r.f, g.f, b.f, a.f
  Protected bri.f, bri_.f, fac.f
  
  r = Red(color)
  g = Green(color)
  b = Blue(color)
  a = Alpha(color)
  bri = (r + g + b) / 3.0
  fac = 0.99
  
  While bri > brightness
    bri_ = bri
    r = r * fac
    g = g * fac
    b = b * fac
    bri = (r + g + b) / 3.0
    If bri = bri_
      Break
    EndIf
  Wend
  
  While bri < brightness
    bri_ = bri
    r = 255.0 - ((255.0 - r) * fac)
    g = 255.0 - ((255.0 - g) * fac)
    b = 255.0 - ((255.0 - b) * fac)
    bri = (r + g + b) / 3.0
    If bri = bri_
      Break
    EndIf
  Wend
  
  ProcedureReturn $ffffffff & RGBA(r, g, b, a)
EndProcedure

loadImageFile()

If IsImage(img)
  StartDrawing(ImageOutput(img))
  Define xMax = OutputWidth() - 1
  Define yMax = OutputHeight() - 1
  
  Define stdDrawingMode = #PB_2DDrawing_Outlined | #PB_2DDrawing_AlphaBlend
  DrawingMode(stdDrawingMode)
  
  Define yStart, yEnd
  Define xStart, xEnd
  Define x, y
  
  ; ------------------------------------------------
  ; get approximate line x-position for the whole image
  Debug "get line x"
  
  yStart = 2
  yEnd = yMax-2
  
  xStart = 3
  xEnd = xMax-3
  
  Dim diffMaxXForY(yMax)
  Define diffMax.f = 0
  Define diffMaxX.f = -1
  Define diff.f
  Define lefBri.f, rigBri.f, topBri.f, botBri.f 
  
  For y=yStart To yEnd
    diffMax = 0
    diffMaxX = -1
    For x=xStart To xEnd
      lefBri = getColorBrightness(Point(x-1, y))
      rigBri = getColorBrightness(Point(x+1, y))
      topBri = getColorBrightness(Point(x, y-1))
      botBri = getColorBrightness(Point(x, y+1))
      
      ; get the difference in brightness of the top/bottom pixels and the left/right pixels
      diff = ((topBri+botBri)/2.0 - (lefBri+rigBri)/2.0)
      
      If diff > diffMax
        diffMax = diff
        diffMaxX = x
      EndIf
    Next
    ; store for this y the x where the largest difference occured
    diffMaxXForY(y) = diffMaxX
  Next
  
  ; count how many hits for the maximum difference each x has
  Dim xCount(xMax)
  For y=yStart To yEnd
    xCount(diffMaxXForY(y)) + 1
  Next
  
  ; get the x with the most hits
  Define maxCount = 0
  Define maxCountedX = -1
  For x=0 To xMax
    If xCount(x) > maxCount
      maxCount = xCount(x)
      maxCountedX = x
    EndIf
  Next
  
  ; now we have an approximate line x-positon in the image
  Define linePosX = maxCountedX
  Debug "linePosX:"+linePosX
  If debugApproxLineBox
    Box(linePosX-20, 2, 41, yMax-1, $6600ff00)
  EndIf
  
  ; ------------------------------------------------
  ; split/process image in y-segments to find the exact x-position of
  ; the line in each segment (to account for the curving of the line)
  Debug "process segments"
  
  Define segmentSize = 100
  Define nbSegments = yMax / segmentSize
  If yMax % segmentSize
    nbSegments + 1
  EndIf
  Define s
  For s=0 To nbSegments-1
    
    yStart = s * segmentSize + 1
    yEnd = yStart + segmentSize -1
    If yEnd >= yMax
      yEnd = yMax - 1
    EndIf
    
    xStart = linePosX - 10
    xEnd = linePosX + 10
    
    ; ------------------------------------------------
    ; find line x-position in current y-segment
    
    Dim diffMaxXForY(yMax)
    
    For y=yStart To yEnd
      Define diffMax.f = -99999
      Define diffMaxX.f = -1
      
      For x=xStart To xEnd
        Define lefBri.f = getColorBrightness(Point(x-3, y))  ; we use a greater x-distance here (+/-3) than before for some reason
        Define rigBri.f = getColorBrightness(Point(x+3, y))
        Define topBri.f = getColorBrightness(Point(x, y-1))
        Define botBri.f = getColorBrightness(Point(x, y+1))
        
        ; (as before)
        Define diff.f = (topBri+botBri)/2.0 - (lefBri+rigBri)/2.0
        
        If diff > diffMax
          diffMax = diff
          diffMaxX = x
        EndIf
      Next
      diffMaxXForY(y) = diffMaxX
    Next
    
    Dim xCount(xMax)
    For y=yStart To yEnd
      xCount(diffMaxXForY(y)) + 1
    Next
    
    Define maxCount = 0
    Define maxCountedX = -1
    For x=0 To xMax
      If xCount(x) > maxCount
        maxCount = xCount(x)
        maxCountedX = x
      EndIf
    Next
    
    ; now we have a better line x-positon for this y-segment
    Define segmentLinePosX = maxCountedX
    
    ; ------------------------------------------------
    ; do some color shuffling
    
    For y=yStart To yEnd
      
      ; we first process the line as 3 pixels wide ...|||...
      For x=segmentLinePosX-1 To segmentLinePosX+1
        If Not ( (x >= 4) And (x <= (xMax-4)) )
          Break
        EndIf
        Define yOff = 2
        If y < 2
          yOff = y
        EndIf
        If y > (yMax - 2)
          yOff = yMax - y
        EndIf
        
        ; get surrounding pixels and brightness
        Define cen = Point(x  , y)
        Define lefTop = Point(x-3, y-yOff)
        Define lefBot = Point(x-3, y+yOff)
        Define rigTop = Point(x+3, y-yOff)
        Define rigBot = Point(x+3, y+yOff)
        Define cenBri.f = getColorBrightness(cen)
        Define lefTopBri.f = getColorBrightness(lefTop)
        Define lefBotBri.f = getColorBrightness(lefBot)
        Define rigTopBri.f = getColorBrightness(rigTop)
        Define rigBotBri.f = getColorBrightness(rigBot)
        
        ; get average brightness of surrounding pixels
        Define newBri.f = (lefTopBri + lefBotBri + rigTopBri + rigBotBri) / 4.0
        
        Define newColor
        
        ; get average color of center and surrounding pixels
        newColor = RGBA( (Red(lefTop) + Red(lefBot) + Red(cen) + Red(rigTop) + Red(rigBot)) / 5.0, 
                         (Green(lefTop) + Green(lefBot) + Green(cen) + Green(rigTop) + Green(rigBot)) / 5.0, 
                         (Blue(lefTop) + Blue(lefBot) + Blue(cen) + Blue(rigTop) + Blue(rigBot)) / 5.0, 
                         (Alpha(lefTop) + Alpha(lefBot) + Alpha(cen) + Alpha(rigTop) + Alpha(rigBot)) / 5.0 )
        
        ; adapt to surrounding brightness
        newColor = setColorBrightness(newColor, newBri)
        
        ; replace center pixel with new color
        DrawingMode(#PB_2DDrawing_AllChannels)
        Plot(x, y, newColor)
        DrawingMode(stdDrawingMode)
        
        If debugLinePixel And y%16=0
          Plot(x, y, $ff0000ff)
        EndIf
      Next
    Next
    
    ; we now process/smooth the outer edge of the 3 pixels wide line ...|||...
    Define i
    
    ; this defines the x-positions of the edge pixels we want to process, but it's all done for each single y like ..###|###..
    Dim xCen(5)                  ;        0123 
    xCen(0) = segmentLinePosX-3  ;    .#.|||...
    xCen(1) = segmentLinePosX-2  ;    ..#|||...
    xCen(2) = segmentLinePosX-1  ;    ...#||...
    xCen(3) = segmentLinePosX+3  ;    ...|||.#.
    xCen(4) = segmentLinePosX+2  ;    ...|||#..
    xCen(5) = segmentLinePosX+1  ;    ...||#...
    
    ; (lesser version)
;     Dim xCen(3)                  ;        0123 
;     xCen(0) = segmentLinePosX-3  ;    .#.|||...
;     xCen(1) = segmentLinePosX-2  ;    ..#|||...
;     xCen(2) = segmentLinePosX+3  ;    ...|||.#.
;     xCen(3) = segmentLinePosX+2  ;    ...|||#..
    
    Define iMax = ArraySize(xCen())
    
    For y=yStart To yEnd
      
      For i=0 To iMax
        
        ; each iteration the center pixel 'cen' is marked as '#' in the above comments
        x = xCen(i)
        
        ; get neighbor pixels
        Define lef = Point(x-1, y)
        Define cen = Point(x  , y)
        Define rig = Point(x+1, y)
        
        ; mix the color of the left/center/right pixels
        newColor = RGBA( ((Red(lef) + Red(cen) + Red(rig)) / 3.0), 
                         ((Green(lef) + Green(cen) + Green(rig)) / 3.0), 
                         ((Blue(lef) + Blue(cen) + Blue(rig)) / 3.0), 
                         ((Alpha(lef) + Alpha(cen) + Alpha(rig)) / 3.0) )
        
        ; replace the cen/center/# pixel
        DrawingMode(#PB_2DDrawing_AllChannels)
        Plot(x, y, newColor)
        DrawingMode(stdDrawingMode)
      Next
    Next
    
    If debugSegmentLineBox
      Box(segmentLinePosX-10, yStart, 21, yEnd-yStart, $ff00ff00)
    EndIf
    
  Next
  StopDrawing()
  
  ; magnify some area
  Define cropX = linePosX-40
  Define cropY = 0
  Define cropSize = 80
  Define cropScale = 8
  Define crop = GrabImage(img, #PB_Any, cropX, cropY, cropSize, cropSize)
  ResizeImage(crop, cropSize*cropScale, cropSize*cropScale, #PB_Image_Raw)
  
  Define w = ImageWidth(img)
  Define h = ImageHeight(img)
  
  ; main window
  Define win = OpenWindow(#PB_Any, 750, 50, w, h, "..")
  AddKeyboardShortcut(win, #PB_Shortcut_Escape, 10)
  Define ig = ImageGadget(#PB_Any, 0, 0, w, h, ImageID(img))
  
  ; magnify window
  Define win2 = OpenWindow(#PB_Any, 10, 10, cropSize*cropScale, cropSize*cropScale, "..")
  AddKeyboardShortcut(win2, #PB_Shortcut_Escape, 10)
  Define ig2 = ImageGadget(#PB_Any, 0, 0, cropSize*cropScale, cropSize*cropScale, ImageID(crop))
 
  Repeat
    WaitWindowEvent()
  Until Event() = #PB_Event_CloseWindow Or Event() = #PB_Event_Menu
EndIf

[wilbert]

I tried harder and now get better results. But I only have 2 images for testing (the one with the bushes and the one with the tree). Some boxes indicate the detection/line if the debugger is enabled. If disabled ony the result is shown.
The line x-position is first detected approximately for the whole image, then more precisely for separate y-segments (to accout for the curving) and only in an narrow x-range around the approximate position. Then brightness of the line is adapted to the surrounding pixels and neighbor pixel colors are blended into the line etc.
Some artifacts will still be left. In some areas in the tree image when there is not really a line to be detected it will detect other parts of the image and create some artifacts instead of removing them. But overall I have to look very closely to still find the line in the processed image and I quite know what to look for. Maybe give it a try. I didn't try the inpaint so I don't know how it compares to that.
With debugger and a large image it will take some time, because I didn't care about performance yet.

Thanks !
I tried it with a few other images. There was one that didn't detect the line at the right position but that's not that important since I can know the line by looking at the focal length in the exif data.
The important thing is that the result images look very good.
Your method of removing the line seems to work very well.