Page 1 of 2
Pulling Web CSV Data Directly Into PureBasic Program?
Posted: Thu Jun 19, 2003 9:50 pm
by Mark1Up
Hi,
I want to pull historical stock market data (comma-seperated format) from Yahoo into my PureBasic application. Is there a way I can put it directly into an array or save it to disk first and then read it in? I think I can handle parsing the data once I get it into a general string array or a disk file.
Here is an example of some CSV data from Yahoo Finance (it's free!).
http://table.finance.yahoo.com/table.csv?s=IBM&g=d
Thanks for any help or ideas
,
Mark
Re: Pulling Web CSV Data Directly Into PureBasic Program?
Posted: Thu Jun 19, 2003 11:56 pm
by ricardo
Im not sure if i understand wat you need, but i think that its simple: just use the Network commands to get the webpage and parse it.
This one works to get the data:
Code: Select all
InitNetwork()
ConnectionID = OpenNetworkConnection("table.finance.yahoo.com", 80)
If ConnectionID
com$="GET http://table.finance.yahoo.com/table.csv?s=IBM&g=d HTTP/1.1"+Chr(13)+Chr(10)
com$=com$+"Accept: */*"+Chr(13)+Chr(10)
com$=com$+"Accept: text/html"+Chr(13)+Chr(10)
com$=com$+"Host: "+host$+Chr(13)+Chr(10)
com$=com$+"User-Agent: Yahoo CVS Parser"+Chr(13)+Chr(10)
com$=com$+Chr(13)+Chr(10)
Res = SendNetworkData(ConnectionID,@com$,Len(com$))
Repeat
Delay(10)
Result = NetworkClientEvent(ConnectionID)
Select Result
Case 2
Content$ = Space(14500)
ReceiveNetworkData(ConnectionID,@Content$,14500)
Ok = 1
MessageRequester("Done!","Your Data" + Chr(13) + Chr(10) + Content$,0)
CloseNetworkConnection(ConnectionID)
EndSelect
Until Ok = 1
EndIf
Posted: Fri Jun 20, 2003 2:01 am
by Mark1Up
Thanks Ricardo, that was more straight forward than I thought. I didn't understand how to specify the HTML commands properly. Your example is much appreciated.
Mark
Posted: Fri Jun 20, 2003 2:27 am
by Karbon
URLDownloadToFile_() -- I think is the Win API function, pretty much does exactly what the name suggests
Handy for quick stuff.
Posted: Fri Jun 20, 2003 6:15 am
by ricardo
Karbon wrote:URLDownloadToFile_() -- I think is the Win API function, pretty much does exactly what the name suggests
Handy for quick stuff.
But in my experience it fails to much... or maybe im using it badly.
Per example, try this one:
Code: Select all
Result = URLDownloadToFile_(0,"http:\\www.wikipedia.org/wiki/PureBasic_programming_language", "result.html", 0, 0)
Here it faiils all the time and i don't know why, in the same way pages with popups fails. Some advise are welcome
Posted: Fri Jun 20, 2003 6:33 am
by Paul
Code: Select all
Result = URLDownloadToFile_(0,"http://www.wikipedia.org/wiki/PureBasic_programming_language", "result.html", 0, 0)
ricardo, your above example works fine here.
(and it works every time with no problems)
Actually I have used URLDownloadToFile_() quite often and have never had problems with it.
Posted: Fri Jun 20, 2003 6:46 am
by ricardo
Hi Paul,
But here it fails ALL the times.
Im not in slow connection and never have problems in internet, just this command, im in XP (could be this the reason?)
And i have detected that webpages with popups fails always
Posted: Fri Jun 20, 2003 11:39 am
by LarsG
Maybe it could be your firewall settings?!?!
Or do you have a popupkiller running in the background?!?
Posted: Fri Jun 20, 2003 11:56 am
by El_Choni
Result = URLDownloadToFile_(0,"http:\\www.wikipedia.org/wiki/PureBasic_programming_language", "result.html", 0, 0)
Why not try changing
http:\\ to http://
Posted: Fri Jun 20, 2003 12:31 pm
by plouf
i have a similar problem with this command .
i use the URLDownloadToFile() command and works OK
on every computer i have try, except on one's guy
the problem is that either it is not downloaded at all,
or with double header (after i have tweak the code trying to found out what happened)
i guess the problemm sourcing from his firewall (ZoneAlarm)
but he claims that he has shut it down ....
@Ricardo do you use ZoneAlram too ?
Posted: Fri Jun 20, 2003 6:10 pm
by tkruk
Maybe it is an issue with a Proxy Server.
The code above does not work for me but it is because I have
to enter a proxy server before I can make an external connection.
I wonder if there is a way to do it through code.
Tom
Posted: Fri Jun 20, 2003 7:26 pm
by ricardo
plouf wrote:i have a similar problem with this command .
i use the URLDownloadToFile() command and works OK
on every computer i have try, except on one's guy
the problem is that either it is not downloaded at all,
or with double header (after i have tweak the code trying to found out what happened)
i guess the problemm sourcing from his firewall (ZoneAlarm)
but he claims that he has shut it down ....
@Ricardo do you use ZoneAlram too ?
Nop.
XP has some kind of 'protection' itself... but i can use the command with many urls but not with all.
I try to use a w98 that i have here and the same result, but i have to note that the w98 are connected to the XP one to get internet, so if a firewall is the problem ofcourse my w98 will be in the same.
My conclusion is that its not to reliable something that fails in some PCs, not reliable to use it in a product that you will distribute to any kind of users.
Posted: Fri Jun 20, 2003 7:29 pm
by ricardo
El_Choni wrote:Result = URLDownloadToFile_(0,"http:\\www.wikipedia.org/wiki/PureBasic_programming_language", "result.html", 0, 0)
Why not try changing
http:\\ to http://
Tried without success
Posted: Fri Jun 20, 2003 7:48 pm
by LarsG
You can switch off the automatic firewall settings on XP by opening your network connections... right click your internet connection and choose properties.. Then go to the advanced tab and deselect the firewall option..
Posted: Fri Jun 20, 2003 8:54 pm
by ricardo
@LarsG
Yes, but i can't ask all my users to do that.
I use the protection because im permanently connected and the local lan (home lan) gets Internet from my PC.
I think this kind of configuration is very common and i feel its not very reliable to trust in this kind of API if planning to distribute software, thats why i think its a better slution to use Network commands... but i found a problem a couple days ago that i cant try to make many connections in a short period of time.
Im looking for a solution.