Sysop: | Amessyroom |
---|---|
Location: | Fayetteville, NC |
Users: | 21 |
Nodes: | 6 (0 / 6) |
Uptime: | 31:29:10 |
Calls: | 139 |
Files: | 91 |
Messages: | 42,722 |
https://unix.stackexchange.com/questions/769727/how-do-i-read-the-last-lines-of-a-huge-log-file
"I have a log of 55GB in size.
I tried:
cat logfile.log | tail
But this approach takes a lot of time. Is there any way to read huge
files faster or any other approach?"
:)
https://unix.stackexchange.com/questions/769727/how-do-i-read-the-last-lines-of-a-huge-log-file
"I have a log of 55GB in size.
I tried:
cat logfile.log | tail
But this approach takes a lot of time. Is there any way to read huge
files faster or any other approach?"
:)
On 06.03.2024 10:42, D wrote:[...]
I would "chunk" it. When I was young I had to analyze several GB of logs
due to a security incident and what I did is to cut up the mega-file in
several chunks, and then do what ever I had to do to those chunks, and
then add the result to one file.
Don't know if that is possible or useful in your situation, but for me,
about 18 years ago, it worked beautifully.
The point here is that 'cat' reads the whole file and pipes all
to 'head'. While 'head' is capable of reading the file itself
if called as "tail logfile.log" and can thus directly address
the file from the rear. In other words, you don't need elaborate sophisticated workarounds, just use tail in a straightforward way.
(Axel posted some timing numbers.)
Janis
Best regards,
Daniel
On Tue, 5 Mar 2024, Kaz Kylheku wrote:
https://unix.stackexchange.com/questions/769727/how-do-i-read-the-last-lines-of-a-huge-log-file
"I have a log of 55GB in size.
I tried:
cat logfile.log | tail
But this approach takes a lot of time. Is there any way to read huge
files faster or any other approach?"
:)
I would "chunk" it. When I was young I had to analyze several GB of logs
due to a security incident and what I did is to cut up the mega-file in several chunks, and then do what ever I had to do to those chunks, and
then add the result to one file.
Don't know if that is possible or useful in your situation, but for me,
about 18 years ago, it worked beautifully.
Best regards,
Daniel