Looking for Suggestions

[swhite]

Hi

I am just looking for ideas on how to best handle the following scenario in Linux. I have 200 remote locations and growing where I need to get data. It can take a few seconds to 10 minutes per location to get the data because the equipment sends the data one line of text at a time. All data must be collected from all locations within several hours. My idea is to launch multiple instances of the capture program to get the data. The number of instances would increase as the number of remote locations increases. This would have to be tested to determine an optimal number of instances.

Now my question is would it be better to use a variable number of threads to capture the data or would it be better to use RunProgram to launch the data capture programs? Each thread or run instance would be passed the parameter indicating which location is to be retrieved and I need to track which locations have been captured and whether the capture was successful. Once a capture is completed I would start a new one for the next location and repeat this until all locations have been captured.

Thanks,
Simon

[Randy Walker]

Hi Simon... This comment makes no sense to me: "the equipment sends the data one line of text at a time".
I have to guess you are polling this data at a specific time of day, so first thing I would do if possible would be to write a bash script to be copied onto each machine and setup a cron to run that script when appropriate so that all the "one line of text at a time" data to be polled is waiting in a single file, so You only have to grab that one file from each machine.

[mk-soft]

Question:
- Local or Internet locations.
- How should the data be transmitted
- Why transferred line by line and not in one block (depending on size)
- Safety

[Axolotl]

Unfortunately, your description is very unclear.
In order to recommend a concept, a number of questions (in addition to mk-soft) need to be clarified.
- Who initiates the communication?
- How many data locations, what amounts of data, and in what time frame?
- How are the various data locations connected?
- Failure scenarios
- tbc.

Sorry, just my spontaneous questions

[swhite]

Hi

The data is transferred as a line of ascii text that must be acknowledged before the next line is sent. This just happens to be the way the equipment operates and I have no control over that. The equipment is spread out all over the country often in remote locations and some with poor internet connections. I am currently doing this using other software running multiple processes as opposed to threads. I was looking for suggestions as I would like to rewrite this application using Purebasic.

The communication starts via my program connecting to the machine via TCP and issuing a command that triggers the equipment to start sending the data. This process continues until I receive "\\" as the end of the available data. Then I disconnect from the machine.

I cannot run a cron job throughout the day collecting the data because if for instance someone decides to send some required data to the equipment it will immediately disconnect me from the equipment and connect to the new request. This means you have to do this task during particular hours when there is no chance of another connections being made. The volume of data is usually not high. There might be no data or perhaps 10kb or 20kb.

The most likely failure scenarios are :
Loss of internet connection
Inability to connect to the site
Incorrect Password for connection
Incorrectly formatted data (causes by equipment failure at the site)

Simon