PureBasic Forums - English

Posted: **Sat Sep 23, 2023 9:03 pm**

Hello friends, it's me again

This time I am trying to read data from a website that displays its data with JS - and that is my problem: When I try to get the HTML contents from
'https://www.lbma.org.uk/prices-and-data ... ces#/table' using ReceiveHTTPMemory(), I can apparently not read the prices, this is what I have tried:

Code: Select all

InitNetwork()
*LBMA = ReceiveHTTPMemory("https://www.lbma.org.uk/prices-and-data/precious-metal-prices#/table")
Debug "Content: " + PeekS(*LBMA,MemorySize(*LBMA),^#PB_UTF8|#PB_ByteLength)
FreeMemory(*LBMA)

I need a bit of help here how to get AM fixing and PM fixing of Gold in EUR from that LBMA page. Any ideas?

The required outcome would be that I have 2 variables with .e.g. the values for 22-09-2023 would be
AMEUR$="1810.82"
PMEUR$="1807.08"
so I can do further calculations.

Greetings to you all !

Posted: **Sun Sep 24, 2023 1:09 am**

You can directly query the API which that page utilises:

Code: Select all

EnableExplicit

#LBMA_API_ENDPOINT_JSON = "https://prices.lbma.org.uk/json/"

; Describes the price for a good in each of: USD ($), GBP (£), and EUR (€).
;
; N.B. Floating point values are iffy for currency work. Fine for storing the price of
;      something, but definitely avoid math on balances, etc., given FP error.
Structure MultiCurrencyPrice
  usd.d
  gbp.d
  eur.d
EndStructure

; Describes a snapshot of prices in various currencies associated with a given date for
; morning and afternoon fixes.
; Note that afternoon prices may be zeroed in cases where only the morning fix has been
; done.
Structure DailyPriceSnapshot
  ; Prices for the date's morning fix.
  morning.MultiCurrencyPrice
  ; Prices for the date's afternoon fix.
  afternoon.MultiCurrencyPrice
EndStructure

; Describes raw data as returned by the LBMA prices API.
Structure _PriceRawDataEntry
  is_cms_locked.b
  ; Date in text form. YYYY-MM-DD.
  d.s
  ; Price tuple: [USD, GBP, EUR]
  v.d[3]
EndStructure

; Parses pricing data from the given JSON string consisting of price entries for either
; the morning or afternoon fix, storing results in the supplied map.
;
; @param JSONText: Raw JSON to be parsed as a list of daily price entries for either the
; morning or afternoon fix.
; @param IsMorning: Whether or not the data correlates to the morning fix. When #False it
; is assumed to be the afternoon fix.
; @param OutPrices: Map which should receive pricing data. Keys are YYYY-MM-DD formatted
; dates.
;
; @returns Returns #True when successful. Otherwise returns #False.
Procedure.i ParsePricingJsonTextToMap(JSONText.s, IsMorning.i, Map OutPrices.DailyPriceSnapshot())
  Protected pricingJson = ParseJSON(#PB_Any, JSONText)
  If Not pricingJson
    ProcedureReturn #False
  EndIf
  
  NewList rawPricingData._PriceRawDataEntry()
  ExtractJSONList(JSONValue(pricingJson), rawPricingData())
  FreeJSON(pricingJson)
  
  ForEach rawPricingData()
    Protected *snapshot.DailyPriceSnapshot = OutPrices(rawPricingData()\d)
    Protected *outPrice.MultiCurrencyPrice = @*snapshot\morning
    If Not IsMorning
      *outPrice = @*snapshot\afternoon
    EndIf
    
    *outPrice\usd = rawPricingData()\v[0]
    *outPrice\gbp = rawPricingData()\v[1]
    *outPrice\eur = rawPricingData()\v[2]
  Next
  
  ProcedureReturn #True
EndProcedure

; Fetches precious metal pricing data from the LBMA API.
;
; @param PreciousMetalName: Name of the metal for which pricing data should be fetched.
; Expected to be one of: "gold", "silver", "platinum", or "palladium".
; @param OutPrices: Map which should receive pricing data. Keys are YYYY-MM-DD formatted
; dates.
; @param RetainMap: Whether or not the output pricing map should not be cleared before
; inserting fetched data into it.
;
; @returns Returns #True when successful. Otherwise returns #False.
Procedure.i FetchPricingData(PreciousMetalName.s, Map OutPrices.DailyPriceSnapshot(), RetainMap.i = #False)
  If Not RetainMap
    ClearMap(OutPrices())
  EndIf
  
  ; Normalise metal name to lowercase.
  PreciousMetalName = LCase(PreciousMetalName)
  
  ; Fetch and parse AM data...
  Protected *rawJsonData = ReceiveHTTPMemory(#LBMA_API_ENDPOINT_JSON + PreciousMetalName + "_am.json")
  If Not *rawJsonData
    DebuggerWarning("Failed to fetch AM data for metal: " + PreciousMetalName)
    ProcedureReturn #False
  EndIf
  If Not ParsePricingJsonTextToMap(PeekS(*rawJsonData, MemorySize(*rawJsonData), #PB_UTF8 | #PB_ByteLength), #True, OutPrices())
    DebuggerWarning("Failed to parse AM data for metal: " + PreciousMetalName)
    FreeMemory(*rawJsonData)
    ProcedureReturn #False
  EndIf
  FreeMemory(*rawJsonData)
  
  ; Fetch and parse PM data...
  *rawJsonData = ReceiveHTTPMemory(#LBMA_API_ENDPOINT_JSON + PreciousMetalName + "_pm.json")
  If Not *rawJsonData
    DebuggerWarning("Failed to fetch PM data for metal: " + PreciousMetalName)
    ProcedureReturn #False
  EndIf
  If Not ParsePricingJsonTextToMap(PeekS(*rawJsonData, MemorySize(*rawJsonData), #PB_UTF8 | #PB_ByteLength), #False, OutPrices())
    DebuggerWarning("Failed to parse PM data for metal: " + PreciousMetalName)
    FreeMemory(*rawJsonData)
    ProcedureReturn #False
  EndIf
  FreeMemory(*rawJsonData)
  
  ProcedureReturn #True
EndProcedure


NewMap prices.DailyPriceSnapshot()
Debug "Fetching gold price data..."

If FetchPricingData("gold", prices())
  Debug "Fetch OK!"
  Debug ""
  Debug "Data for 2023-09-22:"
  Debug "  AM: " + StrD(prices("2023-09-22")\morning\eur, 2) + "€"
  Debug "  PM: " + StrD(prices("2023-09-22")\afternoon\eur, 2) + "€"
EndIf

I'm not entirely sure of your use case, so I just crammed everything into a Map.

This could definitely be improved by using a data structure better for dynamic ordered/time-series data, but without knowing exact requirements, I've spared this complexity.

Disclaimer: I've no idea if they impose rate-limiting on this API, but it seems plausible. Since they return a great deal of data with each request, it's best to consume with some restraint. You'll definitely want to store and read from a cache instead of hitting their backend, where possible.

You could send AM + PM requests (and requests for more metals) asynchronously rather than in order, so loading occurs a bit faster. I'd probably avoid that and only do background fetch when prices update, saving into SQLite or similar as a cache, just to hammer the API less.

Posted: **Sun Sep 24, 2023 12:16 pm**

I thank you. A very interesting way to get the data out of that page.

I have therefore inserted 'InitNetwork()' and tried your code, but all I get is zero-values:

Code: Select all

Fetching gold price data...
Fetch OK!

Data for 2023-09-22:
  AM: 0.00€
  PM: 0.00€

Posted: **Sun Sep 24, 2023 1:37 pm**

Can it be that the error results from the fact that you use a PB version that is too old?

StarWarsFan wrote: Sun Sep 24, 2023 12:16 pmI have therefore inserted 'InitNetwork()' ...

Since Version 6.00 LTS (22th June 2022) InitNetwork() is no more needed to use network functions (deprecated)

Posted: **Sun Sep 24, 2023 2:53 pm**

This time I am trying to read data from a website that displays its data with JS - and that is my problem: When I try to get the HTML contents from
'https://www.lbma.org.uk/prices-and-data ... ces#/table' using ReceiveHTTPMemory(), I can apparently not read the prices, this is what I have tried:

This is a recurring question. (Help need a remark)
ReceiveHTTPMemory, ReceiveHTTPFiles, HTTPRequest, HTTPRequestMemory simply downloads the page's source code (text file). A JS code inside HTML page must be interpreted by the browser's JS engine to display results.

Posted: **Sun Sep 24, 2023 6:45 pm**

StarWarsFan wrote: Sun Sep 24, 2023 12:16 pm I thank you. A very interesting way to get the data out of that page.

I have therefore inserted 'InitNetwork()' and tried your code, but all I get is zero-values:
Code: Select all
Fetching gold price data...
Fetch OK!

Data for 2023-09-22:
  AM: 0.00€
  PM: 0.00€

Like @Kiffi said, it's likely a PB versioning difference.

I almost always use the latest stable LTS (6.02 at time of posting), where output has:

Code: Select all

Data for 2023-09-22:
  AM: 1810.82€
  PM: 1807.08€

OS: Windows 11.

If you write your PB + OS versions, I can maybe have a look at what's going wrong and send a workaround.

While I usually dislike being urged into upgrades myself, I'd highly recommend it in this case, if possible. Newer versions have loads of handy new features and bugfixes, and rarely require changes to existing code.

Posted: **Wed Sep 27, 2023 3:11 pm**

That may actually be the case. V 6.02 changed so much in the display of my programs that I decided to uninstall it and stick with V 5.72.
Kiffi, however, only referred to InitNetwork(). I did not think of anything else that could cause your code not to work.
Will try on weekend.

Posted: **Fri Sep 29, 2023 11:46 pm**

StarWarsFan wrote: Wed Sep 27, 2023 3:11 pm That may actually be the case. V 6.02 changed so much in the display of my programs that I decided to uninstall it and stick with V 5.72.
Kiffi, however, only referred to InitNetwork(). I did not think of anything else that could cause your code not to work.
Will try on weekend.

I had a look with 5.73 LTS (not exactly 5.72 but already installed and close enough) and was able to reproduce the same 0.00€ which you saw.

I've narrowed it down to a change in PB 6.0 where accessing any Map element implicitly creates said element if missing, instead of only creating the element in certain cases, for consistency:

https://www.purebasic.fr/english/viewtopic.php?t=77272 wrote: Wed May 19, 2021 10:08 am - Beta 2 is out, all reported bugs have been corrected, don't hesitate to test it with your usual projects ! Change log:
Code: Select all
- Changed the way the map elements are created when using passive syntax, to be more consistent. There is no more a dummy element.
- OSVersion(): added support for OS X 11 and 12

I've made accommodations for this as well as the need for InitNetwork(...) on prior PB versions here:

Code: Select all

EnableExplicit

#LBMA_API_ENDPOINT_JSON = "https://prices.lbma.org.uk/json/"

; Describes the price for a good in each of: USD ($), GBP (£), and EUR (€).
;
; N.B. Floating point values are iffy for currency work. Fine for storing the price of
;      something, but definitely avoid math on balances, etc., given FP error.
Structure MultiCurrencyPrice
  usd.d
  gbp.d
  eur.d
EndStructure

; Describes a snapshot of prices in various currencies associated with a given date for
; morning and afternoon fixes.
; Note that afternoon prices may be zeroed in cases where only the morning fix has been
; done.
Structure DailyPriceSnapshot
  ; Prices for the date's morning fix.
  morning.MultiCurrencyPrice
  ; Prices for the date's afternoon fix.
  afternoon.MultiCurrencyPrice
EndStructure

; Describes raw data as returned by the LBMA prices API.
Structure _PriceRawDataEntry
  is_cms_locked.b
  ; Date in text form. YYYY-MM-DD.
  d.s
  ; Price tuple: [USD, GBP, EUR]
  v.d[3]
EndStructure

; Parses pricing data from the given JSON string consisting of price entries for either
; the morning or afternoon fix, storing results in the supplied map.
;
; @param JSONText: Raw JSON to be parsed as a list of daily price entries for either the
; morning or afternoon fix.
; @param IsMorning: Whether or not the data correlates to the morning fix. When #False it
; is assumed to be the afternoon fix.
; @param OutPrices: Map which should receive pricing data. Keys are YYYY-MM-DD formatted
; dates.
;
; @returns Returns #True when successful. Otherwise returns #False.
Procedure.i ParsePricingJsonTextToMap(JSONText.s, IsMorning.i, Map OutPrices.DailyPriceSnapshot())
  Protected pricingJson = ParseJSON(#PB_Any, JSONText)
  If Not pricingJson
    ProcedureReturn #False
  EndIf
  
  NewList rawPricingData._PriceRawDataEntry()
  ExtractJSONList(JSONValue(pricingJson), rawPricingData())
  FreeJSON(pricingJson)
  
  ForEach rawPricingData()
    ; Reuse an existing element to update or add missing price data, otherwise create a
    ; new one.
    ; In PB 6.0 and later, this can be shortened with implicit map-element creation.
    Protected *snapshot.DailyPriceSnapshot = FindMapElement(OutPrices(), rawPricingData()\d)
    If Not *snapshot
      *snapshot = AddMapElement(OutPrices(), rawPricingData()\d)
    EndIf
    Protected *outPrice.MultiCurrencyPrice = @*snapshot\morning
    If Not IsMorning
      *outPrice = @*snapshot\afternoon
    EndIf
    
    *outPrice\usd = rawPricingData()\v[0]
    *outPrice\gbp = rawPricingData()\v[1]
    *outPrice\eur = rawPricingData()\v[2]
  Next
  
  ProcedureReturn #True
EndProcedure

; Fetches precious metal pricing data from the LBMA API.
;
; @param PreciousMetalName: Name of the metal for which pricing data should be fetched.
; Expected to be one of: "gold", "silver", "platinum", or "palladium".
; @param OutPrices: Map which should receive pricing data. Keys are YYYY-MM-DD formatted
; dates.
; @param RetainMap: Whether or not the output pricing map should not be cleared before
; inserting fetched data into it.
;
; @returns Returns #True when successful. Otherwise returns #False.
Procedure.i FetchPricingData(PreciousMetalName.s, Map OutPrices.DailyPriceSnapshot(), RetainMap.i = #False)
  If Not RetainMap
    ClearMap(OutPrices())
  EndIf
  
  ; Normalise metal name to lowercase.
  PreciousMetalName = LCase(PreciousMetalName)
  
  ; Fetch and parse AM data...
  Protected *rawJsonData = ReceiveHTTPMemory(#LBMA_API_ENDPOINT_JSON + PreciousMetalName + "_am.json")
  If Not *rawJsonData
    DebuggerWarning("Failed to fetch AM data for metal: " + PreciousMetalName)
    ProcedureReturn #False
  EndIf
  If Not ParsePricingJsonTextToMap(PeekS(*rawJsonData, MemorySize(*rawJsonData), #PB_UTF8 | #PB_ByteLength), #True, OutPrices())
    DebuggerWarning("Failed to parse AM data for metal: " + PreciousMetalName)
    FreeMemory(*rawJsonData)
    ProcedureReturn #False
  EndIf
  FreeMemory(*rawJsonData)
  
  ; Fetch and parse PM data...
  *rawJsonData = ReceiveHTTPMemory(#LBMA_API_ENDPOINT_JSON + PreciousMetalName + "_pm.json")
  If Not *rawJsonData
    DebuggerWarning("Failed to fetch PM data for metal: " + PreciousMetalName)
    ProcedureReturn #False
  EndIf
  If Not ParsePricingJsonTextToMap(PeekS(*rawJsonData, MemorySize(*rawJsonData), #PB_UTF8 | #PB_ByteLength), #False, OutPrices())
    DebuggerWarning("Failed to parse PM data for metal: " + PreciousMetalName)
    FreeMemory(*rawJsonData)
    ProcedureReturn #False
  EndIf
  FreeMemory(*rawJsonData)
  
  ProcedureReturn #True
EndProcedure

If Not InitNetwork()
  Debug "Failed to initialise networking library!"
  End
EndIf

NewMap prices.DailyPriceSnapshot()
Debug "Fetching gold price data..."

If FetchPricingData("gold", prices())
  Debug "Fetch OK!"
  Debug ""
  Debug "Data for 2023-09-22:"
  Debug "  AM: " + StrD(prices("2023-09-22")\morning\eur, 2) + "€"
  Debug "  PM: " + StrD(prices("2023-09-22")\afternoon\eur, 2) + "€"
EndIf

(Changes on new side: L57:63, L124)

Posted: **Thu Oct 12, 2023 10:24 am**

This is the very first time I work with JSON and a web-API.
Thank you. I am learning sooo much here. Very much appreciated.

The latter version of the code (by yuki » Fri Sep 29, 2023 11:46 pm)
works nicely with my V5.71. Values are correct.

The map itself was not even required, I need only the newest value, but thanks for that, too.

PureBasic Forums - English

Reading Data From Website - Special

Reading Data From Website - Special

Re: Reading Data From Website - Special

Re: Reading Data From Website - Special

Re: Reading Data From Website - Special

Re: Reading Data From Website - Special

Re: Reading Data From Website - Special

Re: Reading Data From Website - Special

Re: Reading Data From Website - Special

Re: Reading Data From Website - Special