Memory manipulation

[Baldrick]

I am at the moment obtaining an image from an IP camera via LAN / WAN which I am writing directly to a file as 'jpg' type. The only problem I am having is getting rid of the initial header data from the network stream. I know this sould be quite simple, but I am very rusty at the moment.
Just wondering if any of you have a simple way of losing the header data from the memory allocation prior to writing to file?
In the snippet below, I need to remove all data prior to the start of the image which is seen as ÿØÿÛ

• Cache-Control: no-cache
  Pragma: no-cache
  Expires: Mon, 06 Jan 1990 00:00:01 GMT
  Content-type: image/jpeg
  Content-length: 123506
  Connection: close
  
  ÿØÿÛ

I have tried movememory / copymemory modified to lose the initial header data to a 2nd buffer but this is then doing damage somewhere at the end of the received data section putting everything out of order & resulting in misaligned images. I have also tried only creating the file after the ÿØÿÛ has been found to discard all prior data but this is proving not very reliable & I can't work out why...
Also, fwiw, I did my initial testing with www authenticate (Basic) disabled on the camera & results were excellent. It is only now that the headers have been changed by this extra measure that it has become a problem.

The incoming data looks something like this:
Send:

• GET /img/snapshot.cgi?size=3&quality=1 HTTP/1.1
  HOST:192.168.1.13
  Authorization: Basic ZGVtbzE6ZGVtbzE=

Received:

• HTTP/1.0 200 OK
  
  Cache-Control: no-cache
  Pragma: no-cache
  Expires: Mon, 06 Jan 1990 00:00:01 GMT
  Content-type: image/jpeg
  Content-length: 116053
  Connection: close
  
  ÿØÿÛ
  Kfk¿²¯Œt_‚íñ84^†++‹”k;û9aŽëˆZêÚ(®1Dcnò€](wWÒ_þ,xo^ø/ñËYÔõ_Ÿxþ/
  ¬-áïG©[jrÛly/%.ì8ãmëw“¼„ and so on for all the image data

[/list]

[idle]

not enough info for me to answer
if your just getting an image from a the web server assuming the webcam periodically updates the image and saves it
then you can just use

Code: Select all

 InitNetwork()
 ReceiveHTTPFile("http://www.xyz.com/webcam.jpg","webcam.jpg")

This doesn't answer your question but without any code I can't really answer
so if you did a Get wouldn't it be base64encoded?

Code: Select all

Global gServerID
Global gBusy

Procedure.s GetURL(mUrl.s)
 
  Protected Header.s, *Buffer = AllocateMemory(1200), recString.s,mDoc.s,mServer.s,Timeout.i,pos,pos1,lRet.i,i.i,DataLength.i,TotalRec.i,tdata.s,rlen.i,chunked.i
   
  i=FindString(mUrl,"/",1)
  If i
    mDoc = Right(mUrl,Len(mUrl)-i)
    mServer = Left(mUrl,i-1)
  Else
    ProcedureReturn ""
  EndIf
   
   
  If gbusy = 0
     gServerID = OpenNetworkConnection(mServer,80)
     
           
      If Not gServerID
        MessageRequester("Network Problem","Can't establish connection, please check that you're online")
       
      EndIf
  EndIf
     
  Timeout = ElapsedMilliseconds() + 10000
       
  If gServerID
       
      Header = "GET /"+ mDoc + " HTTP/1.1"+#CRLF$
      Header + "Host: "+ mServer + #CRLF$+#CRLF$
       
      SendNetworkData(gServerID,@Header,Len(Header))
     
      Repeat
        Delay(2)
      Until NetworkClientEvent(gServerID) = #PB_NetworkEvent_Data And ElapsedMilliseconds() < Timeout
     
      If ElapsedMilliseconds() < Timeout 
         DataLength = ReceiveNetworkData(gServerID,*Buffer,1200)
         totalRec + DataLength
         recString + PeekS(*Buffer,DataLength)
         pos = FindString(UCase(recString),UCase("Content-Length"),1)
         If pos
            pos1 = FindString(UCase(recString),#CRLF$,pos+2)
            tdata.s = Mid(recString,pos,pos1-pos)
            pos = FindString(tdata,":",1)
            rlen = Val(Right(tdata,Len(tdata)-(pos+1)) )
           
            Debug "content len " + Str(rlen) 
         ElseIf FindString(UCase(recString),UCase("Transfer-Encoding: chunked"),1) 
            rlen = $7FFFFFFF
            chunked = 1
         EndIf   
         ;Debug recString
         Repeat
           DataLength = ReceiveNetworkData(gServerID,*Buffer,1200)
           totalRec + DataLength
           Debug Datalength
           If DataLength
              recString + PeekS(*Buffer,DataLength)
           EndIf
           Delay(2)
           If chunked
             If FindString(recString, #CRLF$ + "0" + #CRLF$,1)
               DataLength = 0   
             EndIf
           EndIf   
         Until totalRec >= rlen Or DataLength = 0   
         Debug Str(totalRec) + " " + Str(datalength) 
      EndIf     
      CloseNetworkConnection(gServerID)
 
  EndIf 
 
  FreeMemory(*Buffer)
  gbusy = 0
 
  ProcedureReturn recString
   
EndProcedure

InitNetwork()

OpenWindow(#PB_Any,0,0,250,300,"event window",#PB_Window_ScreenCentered | #PB_Window_SystemMenu)

StringGadget(0,5,5,240,20,"www.villavacations.net/cam/cam2.jpg") ;some random webcam 
ButtonGadget(1,5,25,60,20,"Get")
EditorGadget(2,5,45,240,200)


Repeat

ev = WaitWindowEvent()

If ev = #PB_Event_Gadget
    evg = EventGadget()
    If evg = 1
      SetGadgetText(2,"")
      murl.s = GetGadgetText(0)
      doc.s = geturl(murl)
      If doc <> ""
        SetGadgetText(2,doc)
      EndIf   
    EndIf       
EndIf

Until ev = #WM_CLOSE 

[Baldrick]

Thanks for the effort Idle. The authentication is not an issue, that all works very well. I am able to login, retrieve the real time jpeg image data, etc with no problems at all. My problem is that I have not been doing much programming work recently & am having a bit of a mental block atm on how to dump the header files.
( Strange part is that without the authentication active the image works around 95% perfectly with only the occasional miss. But when I activate authentication on the IP camera, the headers change very slightly & I know I am doing something really stupidly wrong, but just cant seem to work it out in 5 minutes.)
I guarantee that when I have that 'Ahaaaa' moment I will feel really silly.
I am on the road atm & wont get a lot of time to play with it this week, though I would like to get my preliminary working properly for demonstration purposes as I will be talking to various wholesale suppliers for IP camera's this week & will be looking quite seriously at what API is available for these various cameras.

( BTW, I havent offered up any code at this stage as it is too ugly, unthought out, etc & is only just me for now working out how to best use the various snapshot, mpeg, h264, etc options with these cameras as I am moving into selling these units. ( I basically want to do a few little applications of my own to make it hard for other competitors in my region to compete with me. )

[idle]

It wasn't any effort, the code was lurking around in archives

So when you say IP camera, do you mean a camera that has a built in webserver?
If thats the case perhaps look up Axis cameras
I have some code to access them but I can't remember how much of the api I implemented

[Pupil]

The HTTP header always ends with the sequence $0d $0a $0d $0a, ie a double CARRIAGE RETURN. You should search for this, everything after this is data and you should be able to just save the buffer from this point to the end of the buffer. I would also check against the 'content-length' entry in the header to make sure that the size is correct..

[Baldrick]

So when you say IP camera, do you mean a camera that has a built in webserver?
If thats the case perhaps look up Axis cameras
I have some code to access them but I can't remember how much of the api I implemented

Yes Idle, thats exactly what i am up too. The Camera I am using for the moment is a Cisco WVC210PTZ.
This week I will be in Sydney doing some training on some IP camera's, network video recorders, NAS, long range WiFi equipment, etc. The NVR's I am training on are hybrid units allowing for installation of a mix of standard composite analogue camera's along side of IP camera's generally for use as video surveillance recording. This will be done through a company here in Aus called Ness Security Products. The main NVR unit I am interested in from them is a re-badged AverMedia controller. I suspect some of their IP camera range may be re-badged Axis camera's also.
My main interest is to be able to make up a few little Windows based app's which can be used by say for example 'Joe Farmer' to enable him to keep an eye on his prize heifer as she is about to drop her 1st calf, maybe a set up to monitor & keep watch on the irrigation equipment, etc around his farm where recording is less of an issue as being able to live view things. I have a cloud based IP alarm monitoring platform running here now which I have been testing for several months with the owner of this service with excellent results over standard broadband connections as well as with WiFi bridged connections back to ADSL fixed connections, 3G connections, so on & will be wanting to include the ability to integrate this into my custom apps.

The HTTP header always ends with the sequence $0d $0a $0d $0a, ie a double CARRIAGE RETURN. You should search for this, everything after this is data and you should be able to just save the buffer from this point to the end of the buffer. I would also check against the 'content-length' entry in the header to make sure that the size is correct..

Thanks Pupil, thats what i thought also, but it appears not to be the case with the camera I am testing. Sometimes it give 2 x CRLF but other times only 1 x CRLF & apart from maybe it is a small bug in the camera I am testing I have no idea why this is the case. ( I also just ran Idles little code snippet on this camera & got exactly the same result of 1 x CRLF as my own code was showing. )

[Baldrick]

@Idle & Pupil,
Thanks for your help here it is good to have ppl to talk to with this sort of thing sometimes, I now have it working nicely. It was 1 of those silly mistakes as I said it would be. I was resetting a variable out of place which meant that even though I was shifting my memory write position, it had been reset to the start of the memory. That meant that I was writing most of the header into the image....

Code: Select all

                                Case    #PB_NetworkEvent_Data
                                        ;Debug "data waiting"
                                        dl=ReceiveNetworkData(cID,*buffer,2048)
                                        AddGadgetItem(#Text_0,-1, PeekS(*buffer,dl))
                                        ;Debug PeekS(*buffer)
                                        ;Debug "data finished"
                                        found=FindString(PeekS(*buffer,dl),"ÿ"+"Ø"+"ÿ"+"Û")
                                        ;Debug Len(PeekS(*buffer))
                                        ;Debug found
                                        If found
                                                If ReadFile(101,"img.jpg")
                                                        CloseFile(101)
                                                        DeleteFile("img.jpg")
                                                 EndIf 
                                                CreateFile(101,"img.jpg")
                                                CloseFile(101)
                                                ;Debug "found @ "+Str(found)                                           
                                       EndIf  
                                               OpenFile(101,"img.jpg")
                                               FileSeek(101,Lof(101))
                                               If found 
; I was resetting found to 0 here... :-(
                                                       WriteData(101,*buffer+found-1,dl-found+1)
                                                       ;Debug "moved data"
                                                       ;Debug PeekS(*buffer+found-1,dl-found+1) 
                                                       ;Debug "****************************************"
                                                    Else 
                                                        WriteData(101,*buffer,dl)
                                              EndIf 
                                               CloseFile(101)
                                               found=0
                                               image1=LoadImage(100,"img.jpg")  
                                        SetGadgetState(#Image_0,image1)

[Pupil]

Glad you got things working! Regarding your little misstake, i can only say -been there, done that... again ... again ... and again

[idle]

yes it helps when your not to reseting your pointer.
I guess you'll just have to scan the buffer until you get to the first alpha numeric chr past CRLF$ + CRLF$

Sounds like a good project Baldrick, I'll dig out that axis code it might be relevant, I'll send you a link to it.

[Baldrick]

yes it helps when your not to reseting your pointer.
I guess you'll just have to scan the buffer until you get to the first alpha numeric chr past CRLF$ + CRLF$

Sounds like a good project Baldrick, I'll dig out that axis code it might be relevant, I'll send you a link to it.

Thanks again Idle, code like that could become very handy for me in the coming months as I delve deeper into these things I think.
I will send you a bit of a pm next weekend with a link to a little demo uC webserver I have made & set up along with the monitoring platform so you can have a bit of a look at how all this monitoring & camera stuff integrates in real time if you wish.

[idle]

yes that would be great.