Pulling Web CSV Data Directly Into PureBasic Program?

[Mark1Up]

Hi,

I want to pull historical stock market data (comma-seperated format) from Yahoo into my PureBasic application. Is there a way I can put it directly into an array or save it to disk first and then read it in? I think I can handle parsing the data once I get it into a general string array or a disk file.

Here is an example of some CSV data from Yahoo Finance (it's free!).

http://table.finance.yahoo.com/table.csv?s=IBM&g=d

Thanks for any help or ideas ,

Mark

[ricardo]

Im not sure if i understand wat you need, but i think that its simple: just use the Network commands to get the webpage and parse it.

This one works to get the data:

Code: Select all

InitNetwork()

ConnectionID = OpenNetworkConnection("table.finance.yahoo.com", 80)

If ConnectionID
  com$="GET http://table.finance.yahoo.com/table.csv?s=IBM&g=d HTTP/1.1"+Chr(13)+Chr(10)
  com$=com$+"Accept: */*"+Chr(13)+Chr(10)
  com$=com$+"Accept: text/html"+Chr(13)+Chr(10)
  com$=com$+"Host: "+host$+Chr(13)+Chr(10)
  com$=com$+"User-Agent: Yahoo CVS Parser"+Chr(13)+Chr(10)
  com$=com$+Chr(13)+Chr(10)
  Res = SendNetworkData(ConnectionID,@com$,Len(com$))

    Repeat
    Delay(10)
    Result = NetworkClientEvent(ConnectionID)
  
    Select Result
  
    Case 2
      Content$ = Space(14500)
      ReceiveNetworkData(ConnectionID,@Content$,14500)
      Ok = 1
      MessageRequester("Done!","Your Data" + Chr(13) + Chr(10) + Content$,0)
      CloseNetworkConnection(ConnectionID)
    EndSelect
    
    Until Ok = 1
EndIf

[Mark1Up]

Thanks Ricardo, that was more straight forward than I thought. I didn't understand how to specify the HTML commands properly. Your example is much appreciated.

Mark

[Karbon]

URLDownloadToFile_() -- I think is the Win API function, pretty much does exactly what the name suggests

Handy for quick stuff.

[ricardo]

URLDownloadToFile_() -- I think is the Win API function, pretty much does exactly what the name suggests
Handy for quick stuff.

But in my experience it fails to much... or maybe im using it badly.

Per example, try this one:

Code: Select all

Result = URLDownloadToFile_(0,"http:\\www.wikipedia.org/wiki/PureBasic_programming_language", "result.html", 0, 0)

Here it faiils all the time and i don't know why, in the same way pages with popups fails. Some advise are welcome

[Paul]

Code: Select all

Result = URLDownloadToFile_(0,"http://www.wikipedia.org/wiki/PureBasic_programming_language", "result.html", 0, 0) 

ricardo, your above example works fine here.
(and it works every time with no problems)

Actually I have used URLDownloadToFile_() quite often and have never had problems with it.

[ricardo]

Hi Paul,

But here it fails ALL the times.
Im not in slow connection and never have problems in internet, just this command, im in XP (could be this the reason?)

And i have detected that webpages with popups fails always

[LarsG]

Maybe it could be your firewall settings?!?!
Or do you have a popupkiller running in the background?!?

[El_Choni]

Result = URLDownloadToFile_(0,"http:\\www.wikipedia.org/wiki/PureBasic_programming_language", "result.html", 0, 0)

Why not try changing http:\\ to http://

[plouf]

i have a similar problem with this command .
i use the URLDownloadToFile() command and works OK
on every computer i have try, except on one's guy
the problem is that either it is not downloaded at all,
or with double header (after i have tweak the code trying to found out what happened)

i guess the problemm sourcing from his firewall (ZoneAlarm)
but he claims that he has shut it down ....

@Ricardo do you use ZoneAlram too ?

[tkruk]

Maybe it is an issue with a Proxy Server.

The code above does not work for me but it is because I have
to enter a proxy server before I can make an external connection.

I wonder if there is a way to do it through code.

Tom

[ricardo]

i have a similar problem with this command .
i use the URLDownloadToFile() command and works OK
on every computer i have try, except on one's guy
the problem is that either it is not downloaded at all,
or with double header (after i have tweak the code trying to found out what happened)

i guess the problemm sourcing from his firewall (ZoneAlarm)
but he claims that he has shut it down ....

@Ricardo do you use ZoneAlram too ?

Nop.
XP has some kind of 'protection' itself... but i can use the command with many urls but not with all.
I try to use a w98 that i have here and the same result, but i have to note that the w98 are connected to the XP one to get internet, so if a firewall is the problem ofcourse my w98 will be in the same.

My conclusion is that its not to reliable something that fails in some PCs, not reliable to use it in a product that you will distribute to any kind of users.

[ricardo]

Result = URLDownloadToFile_(0,"http:\\www.wikipedia.org/wiki/PureBasic_programming_language", "result.html", 0, 0)

Why not try changing http:\\ to http://

Tried without success

[LarsG]

You can switch off the automatic firewall settings on XP by opening your network connections... right click your internet connection and choose properties.. Then go to the advanced tab and deselect the firewall option..

[ricardo]

@LarsG

Yes, but i can't ask all my users to do that.
I use the protection because im permanently connected and the local lan (home lan) gets Internet from my PC.
I think this kind of configuration is very common and i feel its not very reliable to trust in this kind of API if planning to distribute software, thats why i think its a better slution to use Network commands... but i found a problem a couple days ago that i cant try to make many connections in a short period of time.
Im looking for a solution.