Looking for Suggestions

[swhite]

Hi

I am just looking for ideas on how to best handle the following scenario in Linux. I have 200 remote locations and growing where I need to get data. It can take a few seconds to 10 minutes per location to get the data because the equipment sends the data one line of text at a time. All data must be collected from all locations within several hours. My idea is to launch multiple instances of the capture program to get the data. The number of instances would increase as the number of remote locations increases. This would have to be tested to determine an optimal number of instances.

Now my question is would it be better to use a variable number of threads to capture the data or would it be better to use RunProgram to launch the data capture programs? Each thread or run instance would be passed the parameter indicating which location is to be retrieved and I need to track which locations have been captured and whether the capture was successful. Once a capture is completed I would start a new one for the next location and repeat this until all locations have been captured.

Thanks,
Simon

[Randy Walker]

Hi Simon... This comment makes no sense to me: "the equipment sends the data one line of text at a time".
I have to guess you are polling this data at a specific time of day, so first thing I would do if possible would be to write a bash script to be copied onto each machine and setup a cron to run that script when appropriate so that all the "one line of text at a time" data to be polled is waiting in a single file, so You only have to grab that one file from each machine.

[mk-soft]

Question:
- Local or Internet locations.
- How should the data be transmitted
- Why transferred line by line and not in one block (depending on size)
- Safety

[Axolotl]

Unfortunately, your description is very unclear.
In order to recommend a concept, a number of questions (in addition to mk-soft) need to be clarified.
- Who initiates the communication?
- How many data locations, what amounts of data, and in what time frame?
- How are the various data locations connected?
- Failure scenarios
- tbc.

Sorry, just my spontaneous questions

[swhite]

Hi

The data is transferred as a line of ascii text that must be acknowledged before the next line is sent. This just happens to be the way the equipment operates and I have no control over that. The equipment is spread out all over the country often in remote locations and some with poor internet connections. I am currently doing this using other software running multiple processes as opposed to threads. I was looking for suggestions as I would like to rewrite this application using Purebasic.

The communication starts via my program connecting to the machine via TCP and issuing a command that triggers the equipment to start sending the data. This process continues until I receive "\\" as the end of the available data. Then I disconnect from the machine.

I cannot run a cron job throughout the day collecting the data because if for instance someone decides to send some required data to the equipment it will immediately disconnect me from the equipment and connect to the new request. This means you have to do this task during particular hours when there is no chance of another connections being made. The volume of data is usually not high. There might be no data or perhaps 10kb or 20kb.

The most likely failure scenarios are :
Loss of internet connection
Inability to connect to the site
Incorrect Password for connection
Incorrectly formatted data (causes by equipment failure at the site)

Simon

[benubi]

I think you'd have a central executable that manages the states (probably stores in a local DB) of the connections. Multi process is more robust, in case of a bug the external process will fail and the central coordinator can log errors and schedule an other try. But in my CGI experiments I noticed that on windows (back then XP, not a server version) some RunProgram() commands would fail when there are many called in series; IDK if this was because of PureBasic or a Windows limitation, you may only start X programs per second. I believe using external process is more robust when there are changes at the adapter/network environment (idk exactly where) when you reset your router, or the IP changed; this may invalidate handles and require a restart of the program, but when you use external process they should start & initialize with up-to-date configuration (in case your control program runs for ever).

Threaded should have it's advantages but perhaps you only need multiple threads for the connection negotiation that may be blocking; on success the connection maker thread can send the ascii query, and set a timeout for the no data return case; that thread (per connection) could also directly handle the answer. The max/opt amount of threads could be the number returned by CountCPUs(), or even higher for workers that remain mostly idle waiting for an answer (sleeping with delay command on low activity); on non-server windows platforms you have a limited number of TCP connections you are allowed to open per second, and I think there might also be a maximum of open outgoing connections. Those bottlenecks may make it impracticable to have hundreds of threads+connections, but I have never tried it.

[Axolotl]

Perhaps I haven't fully understood the difficulty yet, but I always think simply.
There could be a/one thread in which the connections are established, data is read, and the connections are closed again. A list of connections (one structure element could be state) that is constantly being run through.... You can control the number of open connections very easy, I guess.
As long as no blocking calls/interruptions are to be expected, I don't think it needs to be any more complicated than that.

[mbierly]

Do you have the ability to install your own program on a computer on the same remote network where the equipment is located?

If so, then install your polling program locally on the remote network. In that window of time when you don't worry about interruptions do the equipment polling and save the data to a local database. Then you could connect and download the "polled" data in more efficient chunks and at a time you would prefer. Remotely keep track of what has been requested by the central location so you can handle drops or internet outages and each remote location accumulates its data until sent back to the home location.

[captain_skank]

add a rPi ( Zero or W ) at each location and allow that to collect the data from the equipment and then let that call home to your collection program on a scheduled basis.

You would have to write a script to poll the quipment on the rPi, but you wouldn't be limited to a line at a time form communicationg with your collection program.

[swhite]

Do you have the ability to install your own program on a computer on the same remote network where the equipment is located?

If so, then install your polling program locally on the remote network. In that window of time when you don't worry about interruptions do the equipment polling and save the data to a local database. Then you could connect and download the "polled" data in more efficient chunks and at a time you would prefer. Remotely keep track of what has been requested by the central location so you can handle drops or internet outages and each remote location accumulates its data until sent back to the home location.

I do not have the ability to install my program on the same network as the equipment.

[swhite]

add a rPi ( Zero or W ) at each location and allow that to collect the data from the equipment and then let that call home to your collection program on a scheduled basis.

You would have to write a script to poll the quipment on the rPi, but you wouldn't be limited to a line at a time form communicationg with your collection program.

I do not always have the ability to install any extra hardware at the site. Some sites require a 6 hour drive one way just to reach the remote locations so sending a technician to such a location is expensive and difficult when the temperature is -40 degrees Celsisus.

Simon

[swhite]

Thank-you for all these suggestions. You have given me something to think about before I attempt to re-write the application in Purebasic.

Simon

[Randy Walker]

Hi Simon,

I can see my thoughts were not well suited to your needs but I'm happy I was able to help Stir up a storm of ideas.