Redirected URLs

Trond · Post by **Trond** » Wed May 28, 2008 4:31 pm

pdwyer wrote: So there won't be a leak in a sense of an increasing memory drain, just a one time usage. It's not freed deliberately in the proc as it assumes it WILL be reused continually in the same process.

That's true, but if two threads are created at the same time, both may find the static variable empty and create a new regular expression. So we have to regexes to fit into one static variable. One of them will get lost.

AND51 · Post by **AND51** » Wed May 28, 2008 6:05 pm

Trond wrote:you said that it's shorter and faster because you don't free the regular expression

Boy, where did I say that?
I said that my code is faster, because the expression is CREATED only once!!
I have the slight feeling that you don't even understand my code snippet, so go and read the explanation that I gave to pwdyer!!!

> Makes the executable 105 kb bigger on Linux compared to the original code. That's quite a bloat
Yes, sorry. I forgot that my code won't run on your PC with amazing 64 KB RAM and 100 KHz... Even my calculator is faster... :roll:

Anyway, you think it's time to push every argument to the front that comes out of your mind? This would heavily decrease the niveau of this discussion, so... stop this shit!

> false statements like #LF$ is a legal separator
Why do you always repeat this? Are you able to read? I think you must revise the ABC. Call me and I'll help you. For free! I invite you to visit me, then we learn the ABC! I'll serve tea and cake and everyday we learn a letter!

Just a short note: I already looked the right thing up and said that I didn't remember correctly. WHO CARES? NOBODY!!!!! ONLY YOU!!! STFU, my friend! If I would comment every fals satement in these forums, I had to work 16h a day.

> and you can only create 2000 threads
Trond. My friend. If you're tough enough, you may go on reading, otherwise keep away from the following secret.
Are you ready for the truth? Yes? Really? OK, but I've warned you!

I started a thread in the german forum, asking if it was okay to use hundreds or thousands of threads. After a nice discussion I had the idea to test my PC on how many threads I can create:

Code: Select all

Procedure Thread(temp) 
   Sleep_(#INFINITE) 
EndProcedure 
Repeat 
   x+1 
Until Not CreateThread(@Thread(), x) 
Debug "Maximum amount of threads "+StrQ(x)+"."

I was not the onliest one, saying the 2.000 seems to be the maximum. In former times, my personal maximum was 2.050, today with my new PC and another OS it is ~1.600.

_____________________________________________

Now let's calm down and we try to be friendly to each other again.
I had my lesson, you (hopefully) had your lessen.
We both were right and we both were wrong. No one of us is better.
You know facts that I don't know and the other way around.

There is no need to carry out a 20-page-discussion in this forum. Just let us stay friends. Thanks.

Trond · Post by **Trond** » Wed May 28, 2008 9:31 pm

AND51 wrote:
Trond wrote:you said that it's shorter and faster because you don't free the regular expression
Boy, where did I say that?
I said that my code is faster, because the expression is CREATED only once!!
I have the slight feeling that you don't even understand my code snippet, so go and read the explanation that I gave to pwdyer!!!

Hi. It's obvious to me that you can write quite well. However, your inability to read, even what you wrote yourself, is quite baffling. I've never met anyone who can write but not read before. Truly amazing. For other forum users, here's what you wrote:

In this case it's ok to leave out the Free-Command, because the expression is only allocated once. [...]
This is not an act of lazyness, but an act of performance optimizing

If by "an act of performance optimizing" you don't mean that it does faster, then I'm completely stumped.

> Makes the executable 105 kb bigger on Linux compared to the original code. That's quite a bloat
Yes, sorry. I forgot that my code won't run on your PC with amazing 64 KB RAM and 100 KHz... Even my calculator is faster... :roll:

As I said, you're code isn't so bad it hurts, but adding 105 kb to the executable is generally not a performance optimization.

Just a short note: I already looked the right thing up and said that I didn't remember correctly.

If you don't remember something, look it up and then post, instead of the other way around. Sure, we all can slip, but when you criticize other for their mistakes you should be prepared to eat some dust when you were the one who was wrong.

We both were right and we both were wrong. No one of us is better.

My point exactly. None better. Which is why I think it's bad taste to say that your code is in fact better.

And just one more thing: I wasn't wrong about the threads. No need to warn me. I'm at 11000 threads right now (with your code) and see no point in continuing to 50000. You're just running out of virtual memory. As you said, my calculator has more memory than your computer. Or maybe just lowering the thread stack size will help? (Hint: it will.) Peace.

AND51 · Post by **AND51** » Wed May 28, 2008 11:08 pm

Trond wrote:
In this case it's ok to leave out the Free-Command, because the expression is only allocated once. [...]
This is not an act of lazyness, but an act of performance optimizing
If by "an act of performance optimizing" you don't mean that it does faster, then I'm completely stumped.

That's all? I have the feeling that we misunderstood each other heavily!
"This" is related to "allocated only once". In other words: The one-time-allocation is an "act of performance optimizing".
It's only a side-effect that I don't have to free the expression. But The word "this" was NOT pointing to this side-effect!!

Trond wrote:adding 105 kb to the executable is generally not a performance optimization.

Well this might be a misunderstanding, either. "Performance" is primarily ment regarding the time. I want the procedure working as fast as possible.
1.) The memory-consumption is also important, but only a secundary issue.
2.) I don't make the libraries... Surely, 105 KB are more than enough. But with the technique from today (huge HDD capacities, broadband-internetconnection, compression-methods) thse 105 KB don't hurt anybody really.
It depends on your own opinion, wether you prefer performance optimization OR memory consumption optimization.

> I'm at 11000 threads right now (with your code)
Well, then there is only 1 logical reason... The computers of me and my german PB-forum collegues are somehow broken?

Trond wrote:Peace
- Yeah. Come on, let us forget this turbulent discussion.

Trond · Post by **Trond** » Thu May 29, 2008 8:25 am

Well, then there is only 1 logical reason... The computers of me and my german PB-forum collegues are somehow broken?

No, but the default stack size is 1 mb. You can't keep allocating 1 mb chunks for ever without running out of memory. If you make the default stack size smaller, you can make more threads.

AND51 · Post by **AND51** » Thu May 29, 2008 5:11 pm

Ah, and where can I modify this behaviour?
And what is this good for?

Trond · Post by **Trond** » Thu May 29, 2008 5:41 pm

Which behaviour? The default 1 mb stack size, or the fact that after you allocate 1 mb 2000 times you run out of memory? The 1 mb stack size is to allow each thread to use that much stack space. You will rarely need that much unless you have deep recursion. So with passing /STACK:32768 to the linker will give each thread only 32kb of stack space, which is usually plentiful. You should then be able to reach at least 40000 threads without problems.

pdwyer · Post by **pdwyer** » Fri May 30, 2008 12:45 am

Thanks trond,

That should be a tip/trick all by itself.

I wonder if PB could make that an internal command, I remember Powerbasic had a command to set the stack size. In one app I needed to increase it over 1m due to recursion (admittedly in hind sight there was probably a better design to be had).

Trond · Post by **Trond** » Fri May 30, 2008 8:40 am

You can do it with PB, just put /STACKSIZE:whatver in a text file, and set this file as the linker options file in the compiler options.

Little John · Post by **Little John** » Fri May 30, 2008 9:21 am

pdwyer wrote:

Thought you'd never ask !

http://www.purebasic.fr/english/viewtop ... h&start=15

Interesting! Thanks for pointing me to it.

Regards, Little John

Little John · Post by **Little John** » Fri May 30, 2008 9:50 am

Trond wrote:The 1 mb stack size is to allow each thread to use that much stack space. You will rarely need that much unless you have deep recursion. So with passing /STACK:32768 to the linker will give each thread only 32kb of stack space, which is usually plentiful. You should then be able to reach at least 40000 threads without problems.

Wouldn't that trick be useful also for many "normal" programs with only 1 thread (and w/o very deep recursion or w/o recursion at all), because using this option simply considerably reduces memory consumption?

Regards, Little John

Trond · Post by **Trond** » Fri May 30, 2008 10:58 am

Yes, it would.

Little John · Post by **Little John** » Fri May 30, 2008 11:10 am

Very cool, thanks a lot for the tip!

Regards, learning Little John

dige · Post by **dige** » Fri Jun 27, 2008 9:31 am

does anynody knows, how to use this stuff with a proxy?

AND51 · Post by **AND51** » Mon Jul 21, 2008 5:32 pm

A little update, now supporting META refresh tags:

Code: Select all

InitNetwork()

Procedure.s traceURL(URL$, AllowMetaRefresh=#False, RecursionDepth=#INFINITE)
   If RecursionDepth
		Protected location=CreateRegularExpression(#PB_Any, "(?im)^Location: .*$")
		Protected Dim subPattern.s(0)
		If ExtractRegularExpression(location, GetHTTPHeader(URL$), subPattern()) : Debug "URL"
			URL$=traceURL(Mid(subPattern(0), 11), RecursionDepth-1)
		ElseIf AllowMetaRefresh And ReceiveHTTPFile(URL$, GetTemporaryDirectory()+GetURLPart(URL$, #PB_URL_Site)+".tmp") : Debug "META"
			Protected file=ReadFile(#PB_Any, GetTemporaryDirectory()+GetURLPart(URL$, #PB_URL_Site)+".tmp")
			If file
				Protected meta=CreateRegularExpression(#PB_Any, "(?Umi)<meta\s+http-equiv=(\x22|)refresh\1\s+content=(\x22|)\d+;\s*URL=.+\2>")
				Protected url=CreateRegularExpression(#PB_Any, "(?U)URL=.+(\x22|>)")
				While Not Eof(file)
					If ExtractRegularExpression(meta, ReadString(file), subPattern())
						If ExtractRegularExpression(url, subPattern(0), subPattern())
							URL$=traceURL(Mid(subPattern(0), 5, Len(subPattern(0))-5), 1, RecursionDepth-1)
						EndIf
						Break
					EndIf
				Wend
				FreeRegularExpression(meta)
				FreeRegularExpression(url)
			EndIf
			CloseFile(file)
			DeleteFile(GetTemporaryDirectory()+GetURLPart(URL$, #PB_URL_Site)+".tmp")
		EndIf
		FreeRegularExpression(location)
	EndIf
	ProcedureReturn URL$
EndProcedure

Debug traceURL("http://www.pure-board.de", 1)

First, it searches for the Location-header, if no one is found, the entire file is searched for a META-tag.
You can switch off the META search with the parameter.