parser text editor

[fablefox]

i thinking for making "text editor" which is more of a parser actually. its to analyze a totally large text file. and maybe output a smaller ish text file.

i'm thinking of making a linked list where each line is a list. and delete processed line.

is it a nice idea?

-edit-

it doesn't need per character editing. just view lines and process it. maybe get some data too.

[fablefox]

and i'm talking about up to 1 gb in size. but since hundreds of character per lines i don't think it wont take too much list. specially after parsing and remove unwanted lines....

[rmenezes]

Have you considered using perl to handle the edits? It would be faster than trying to write a "text editor" from scratch. Or maybe a text editor like Notepad++? Some of its plugins (I'm thinking TextFx) are quite powerful.

Just a thought.

[IdeasVacuum]

....yes, you can do it by loading the whole file to memory and then parse the memory.
Similar sort of thing: how to get a list of used variables

[fablefox]

thanks for the reply.

apparently, notepad++ can't open 600 mb something text files. neither is several other text editor / IDE. Thank god the latest community visual C++ IDE can open it. and I know I can run VB script inside it.

But the thing is I'm thinking long term. i will add more functionality to help parse certain kind of text files (its a database log).

i need a simple application that can load a 600 mb to 1 gb worth of text file into memory, and automatically parse it to remove useless log (depending on situation / my input) which I can put into library of inputs, but can also looks for certain string. so i know how many time it has been restarted, the date, if it crashed, what, of it have errors (ORA-xxxxx) what is the error, frequency, date, etc.

but stripping the log of useless data that I'm not interested in will generate much smaller file for me to analyze.

so that's the idea, for now.

[Marc56us]

Hi,

Some tools:

AWK is the best tiny and old tool for filtering text data. Exist for Unix and Dos.
You can use it as filtering tool in a batch file.

Edipad Pro (shareware) is THE best editor (IMHO). I use it a as my favorite editor for many years.
"Open and edit files of absolutely any size, including files larger than 4 GB, even on a 32-bit system with a modest amount of RAM. EditPad Pro intelligently swaps out parts of huge files to disk, allowing it to nearly instantly open very large files while using only a small amount of memory. "
http://www.editpadpro.com
It's also the best tool for work with RegEx: It highlights in real-time matching expressions.

Hoo Win Tail (shareware) Very useful tool to monitor log in real time. It can trigger actions on keyword.
http://www.hootech.com/WinTail/

[IdeasVacuum]

Well, if the app can be left unattended, you can run a loop to read the file 1-line-at-a-time, apply your logic to that line that decides if it should be discarded or kept, if it's to be kept, save (append) to a separate file.

[fablefox]

Hi,

Some tools:

AWK is the best tiny and old tool for filtering text data. Exist for Unix and Dos.
You can use it as filtering tool in a batch file.

Edipad Pro (shareware) is THE best editor (IMHO). I use it a as my favorite editor for many years.
"Open and edit files of absolutely any size, including files larger than 4 GB, even on a 32-bit system with a modest amount of RAM. EditPad Pro intelligently swaps out parts of huge files to disk, allowing it to nearly instantly open very large files while using only a small amount of memory. "
http://www.editpadpro.com
It's also the best tool for work with RegEx: It highlights in real-time matching expressions.

Hoo Win Tail (shareware) Very useful tool to monitor log in real time. It can trigger actions on keyword.
http://www.hootech.com/WinTail/

Any open source lookalike for EditPad Pro?

I'm looking at atom.io but I wish it support python/ruby instead. i can't study all programming languages under the sun just to configure / write a tool/plugin for an app

[fablefox]

Komodo Edit (the open source and free version of Komodo IDE) is out, it can't open a 6xx MB log/text file.





still, I got some ideas.... a text splitter tool?

[fablefox]

atom.io is currently limited to < 2mb files only

[Vera]

....yes, you can do it by loading the whole file to memory and then parse the memory.
Similar sort of thing: how to get a list of used variables

That link won't work except for IdeasVacuum
... but here's one to reach that posting

Indeed an interesting subject and I only once crossed an application that claimed to be able to load such huge files. Its trick was to only load those parts of the file that where actually visible.

I too would tend to shrink those files first and see if the left over comes closer to a more reasonable (handy) size.

greets ~ Vera

[IdeasVacuum]