Thursday, February 3, 2011

Merge standard in and log file to standard out

I'm looking for a command or a bash script that can perform the following:

(see nice picture here: http://i.imgur.com/4VYOf.png but I cannot post it because I have not enough reputation)

That is: I want a "supermerge" to merge the standard in stream with contents written continuously to a log file (var/log/messages in this example). The result should be written to standard out.

The reason is to scans for certain error messages and these messages can both be written to a log file and to standard out.

For example:

my_strange_program /var/log/messages | supermerge /var/log/messages | my_log_scanner
  • my_strange_program can send errors to standard out and the log file.
  • supermerge - the script I'm looking for
  • my_log_scanner a program that scans for error messages (I've already got that)

Note: The log files can rotate: messages, messages.1, messages.2 etc. So it is not possible to just tail the file, since it can be renamed.

  • There is a tool called multitee. It sounds like the thing you are looking for. I tried to figure out how it works but came to no working result

    Another idea is to use multitail, if it is enough to get the merge on the screen

    multitail -f /var/log/syslog -L "ping 4.2.2.1" 
    
    From krissi
  • Try this:

    my_strange_program /var/log/messages > /tmp/tmpfile | tail -F /var/log/messages /tmp/tmpfile | my_log_scanner
    
    Cristian Ciupitu : If you're redirecting stdout (step1) how will tail (step2) read anything?
    Gilles : @Cristian: step 1 doesn't print anything to the pipe, and tail doesn't read anything from the pipe, so it works — here the pipe is just a confusing way of writing `&`. @Dennis: there's a possible race condition if `my_strange_program` writes more than 10 lines before `tail` is ready. Passing `-n +1` to `tail` will fix this, at the expense of showing `/var/log/messages` in full.
    Cristian Ciupitu : @Gilles: yes, you're right.
    Lennart Schedin : I have tested and this works. The only small problem is that tail does not die when the my_strange_program dies. I guess I can solve this with the tail parameter "--pid".
    Dennis Williamson : @Lennart: Yes, if you change the `|` to a `&` and do `tail --pid=$! ...` it should pick up the correct PID for you.
    Lennart Schedin : Ah! yes! Thanks!

0 comments:

Post a Comment