Author Topic: Clean log file  (Read 1235 times)

gonger

  • Hydlaa Citizen
  • *
  • Posts: 292
    • View Profile
Clean log file
« on: September 17, 2019, 06:10:55 am »
Greetings,

Since I changed the log file options (in-game: Options / Interface / Chat - Logs) only after playing for a couple of years, I have huge amounts of combat messages in my log. The file size is about 56 MB by now, and Notepad++ is having a hard time with it.
I am probably not the only one with this problem, so my question is: Does anybody have a small script that would remove combat messages from my main chat log?

Thanks in advance,

Gonger

Shatterkiss

  • Hydlaa Resident
  • *
  • Posts: 92
    • View Profile
Re: Clean log file
« Reply #1 on: September 18, 2019, 04:45:52 am »
I don't have a script, but what I did was search for system lines that I'd never need (combat messages, how much exp I got, etc.) and marked those lines. Then I just deleted all bookmarked lines. Worked pretty well, if not as quickly as a script.

gonger

  • Hydlaa Citizen
  • *
  • Posts: 292
    • View Profile
Re: Clean log file
« Reply #2 on: September 20, 2019, 03:57:22 am »
Hi Shatterkiss,

Thanks for the answer. Could you add a few details? Just the outline to get me started, I will get along afterwards.

Greetings,

Gonger

Migg

  • Hydlaa Citizen
  • *
  • Posts: 223
    • View Profile
Re: Clean log file
« Reply #3 on: September 20, 2019, 06:10:07 pm »
I am guessing you are on Windows (hence Notepad++). I can give you a SED script you could use, I hear Windows nowadays can sport a BASH script so try it to see how it works. I am guessing you need to get rid of both "[Sys]" and "[BSys]" lines, the following command will do it if you have a BASH shell and also SED (or can find a way to somehow run a Linux on your machine that has SED included) and can change to the log directory containing your logfile before running it:

Code: [Select]
sed -i '/ \[B\?Sys\] >/d' *_chat.txt
The command will work with any PS character name, so be careful if you have characters whose logs you don't want to modify. Replace "*_chat.txt" with the actual filename to avoid that.

If you only want to remove "[Sys]" lines use the following command instead:

Code: [Select]
sed -i '/ \[Sys\] >/d' *_chat.txt
Replace "Sys" with "BSys" in the command above to only remove "[BSys]" lines. Good luck!
« Last Edit: September 20, 2019, 06:12:34 pm by Migg »

gonger

  • Hydlaa Citizen
  • *
  • Posts: 292
    • View Profile
Re: Clean log file
« Reply #4 on: September 21, 2019, 05:40:07 pm »
Thanks Migg, but this will not help me. My log lines are not tagged with "[Sys]" or "[BSys]". I just checked and for some reason, under Options / Interface / Chat-Logs, I have no value for Log Bracket.

Some examples:

Quote
(21:52:34) [NPC] Maulberlord attack Gonger Xaraha with aggressive stance
(21:52:35) [NPC] Maulbernaut thinks Gonger Xaraha is weak
(21:52:35) [NPC] Maulberlord thinks Gonger Xaraha is weak
(21:52:37) [NPC] Maulberlord casts Psychic Blow to Gonger Xaraha
(21:52:37) [NPC] Maulbernaut casts Stone Fist to Gonger Xaraha
(21:52:41) [NPC] Imago Consumer thinks Gonger Xaraha is equal
(21:52:41) [NPC] Maulbernaut casts Stone Fist to Gonger Xaraha
(21:52:42) [NPC] Maulberlord casts Psychic Blow to Gonger Xaraha
(21:54:04) [NPC] Imago Consumer emits a cloud of noxious fumes.

(15:19:58) [NPC] Maulbernaut attacks Gonger Xaraha with normal stance
(15:19:59) >You start casting the spell Missile Shower
(15:20:00) >You've gained some practice points in Heavy Armor.
(15:20:00) >Maulbernaut's smash hits Gonger Xaraha and deals 14.75 damage to you!
(15:20:00) >You've gained some practice points in Heavy Armor.
(15:20:00) >Maulbernaut's smash hits Gonger Xaraha and deals 14.88 damage to you!
(15:20:00) >You hit Maulbernaut with a shower of energy beams for 123.96 damage.
(15:20:00) >Missile Shower affected 1 target.
(15:20:00) >You've gained some practice points in Crystal Way.
(15:20:03) >You are too far away to attack!
(15:20:05) >You've gained some practice points in Sword.

But to use the Windows Shell is not a bad idea...

Gonger

Shatterkiss

  • Hydlaa Resident
  • *
  • Posts: 92
    • View Profile
Re: Clean log file
« Reply #5 on: September 21, 2019, 05:44:16 pm »
That's far more advanced than what I did! I just opened the file in Notepad++ in the Search interface went to the Mark tab. Make sure the bookmark line box is checked and that regular expression is selected. Then I marked lines like the following:

Code: [Select]
>You gained \d\d\d experience points and
<You gained \d\d experience points and
<You start to harvest.
<You start to dig.
<You've got some
<You hit

That sort of thing. I leave some system messages (storing/dropping/etc) as they can be useful in tracking down items. If you don't mind clearing all sytem lines, you'll note that they all start with the timestamp and the a '<'. I think you can mark all of those lines with this:

Code: [Select]
\(\d{2}:\d{2}:\d{2}\) >
That will mark every line that comes into the system tab. That includes /who results, the standard emotes like greets, transactions, etc.

Once you've bookmarked everything you want to delete, just go to Search-Bookmarks-Remove Bookmarked Lines

May want to make a backup, just in case.


EDIT: Ah, you posted while I was typing. In that block above, I'd just mark lines containing the following:

Code: [Select]
\[NPC] .* casts .* to Gonger
\[NPC] .* thinks Gonger
\[NPC] .* attacks Gonger

Doing those manually is terribly slow, but shouldn't deleted quest text. You can try this, which will deleted all NPC lines that don't contain the word "says"

Code: [Select]
^\(\d{2}:\d{2}:\d{2}\) \[NPC] (?!.*says).*$
NPCs use emotes though, and those could be relevant.

Other useful lines to bookmark and delete:

Code: [Select]
<You've gained some practice points
<You are too far away
<You made
Sarras
hits you
« Last Edit: September 21, 2019, 06:18:27 pm by Shatterkiss »

gonger

  • Hydlaa Citizen
  • *
  • Posts: 292
    • View Profile
Re: Clean log file
« Reply #6 on: September 25, 2019, 08:22:42 am »
Hi Shatterkiss,

Thanks again for your advice. I did not know about this Bookmark feature in Notepad++, that was most useful.
With your regular expressions and some variants, I could greatly reduce my log file's size. Plus, the latest update to Notepad++ seems to allow better handling of large files, so I should be ok for the moment.

Gonger

Migg

  • Hydlaa Citizen
  • *
  • Posts: 223
    • View Profile
Re: Clean log file
« Reply #7 on: September 26, 2019, 02:05:20 pm »
And remember to enable log brackets for the next time you need to do this  ;)

Shatterkiss

  • Hydlaa Resident
  • *
  • Posts: 92
    • View Profile
Re: Clean log file
« Reply #8 on: October 01, 2019, 11:57:26 pm »
Happy I could help! Had a few 250Mb log files myself, thanks to crafting and combat spam, until I got tired of waiting a few minutes for Notepad to open them and cleaned them up.

Damola

  • Hydlaa Citizen
  • *
  • Posts: 216
    • View Profile
Re: Clean log file
« Reply #9 on: October 06, 2019, 03:13:49 pm »
I do not know how they did it, but Kate under Linux loads the chat log file from Damola, which is about 48 MiB, in a second or so. Kate is also available for Windows. See kate-editor.org
Love,
Damola

Can-ned Food

  • Hydlaa Resident
  • *
  • Posts: 194
  • Many ask what The Can contains, but few will know.
    • View Profile
    • an OpenDocument Spreadsheet for planning your character's stats profile
Re: Clean log file
« Reply #10 on: October 07, 2019, 05:33:27 pm »
I do not know how they did it, but Kate under Linux loads the chat log file from Damola, which is about 48 MiB, in a second or so.
I am using Kate 3.14.2, and it can take almost forever to open something half that size.  Might be how I installed Kio libraries which causes the trouble, but Kate does give me a warning when I open >10 megabytes.

Anyways, I configured PlaneShift to split up my logs to nine or so files.  Gives me some trouble reconstructing events because PlaneShift doesn't quite classify its messages consistently … you might want to split up messages to four files.
Gedundk Kokro, kran
Ailela Belair, nolthrir
Hwokmar Cmar, ynnwn

Damola

  • Hydlaa Citizen
  • *
  • Posts: 216
    • View Profile
Re: Clean log file
« Reply #11 on: October 13, 2019, 09:53:16 am »
@Can-ned Food: The version you are using is way too old. As far as I remember I read developers at some time improved performance *a lot*. I have 19.08.1 and as I said, loading of 48 MiB of chat log file is almost instant.
Love,
Damola

gonger

  • Hydlaa Citizen
  • *
  • Posts: 292
    • View Profile
Re: Clean log file
« Reply #12 on: October 13, 2019, 02:16:30 pm »
@Can-ned Food: The version you are using is way too old. As far as I remember I read developers at some time improved performance *a lot*. I have 19.08.1 and as I said, loading of 48 MiB of chat log file is almost instant.

Hi Damola,

Following your advice I tried Kate 19.08.1 under Windows, and unfortunately I am not impressed by the performance on big files. It is not only the loading, but also the searching inside the file.

Gonger

Damola

  • Hydlaa Citizen
  • *
  • Posts: 216
    • View Profile
Re: Clean log file
« Reply #13 on: October 13, 2019, 03:51:45 pm »
Well searching indeed can take a few seconds in 48 MiB file if it searches over a long distance, which is a bit annoying as it starts searching as you type. Might be good to be able to disable "as you type".

However, I believe there is a limit as to what performance can be achieved and taking some seconds, even 10 to 20 seconds, for searching 46 MiB for a long distance search from the start of the file, is not all that surprising to me. Especially on a ThinkPad T520 with Sandybridge which is a tad bit dated meanwhile. Is Notepad++ faster with that? Searches for shorter distances are much faster.
Love,
Damola

gonger

  • Hydlaa Citizen
  • *
  • Posts: 292
    • View Profile
Re: Clean log file
« Reply #14 on: November 18, 2019, 04:32:08 pm »
I am now using Visual Studio Code (https://code.visualstudio.com/). It loads fast, and searches even faster.
My only trouble is that it refuses to Word Wrap the chat log, maybe because of the control codes in /who or something similar. I will find sooner or later...

But the speed is great.

Greetings,

Gonger