xargs use stardard output as parameter for another command

October 2nd, 2006 mysurface Posted in cat, pipeline, Text Manipulation, wget, xargs | Hits: 57743 | 10 Comments »

xargs is a command line of findutils package. It is not so common for new user, but it is a very useful tools, let me pick an example to show the usage. Bare in mind, xargs is a kind of combo command use together with other command through pipeline.

I have a list of links to download, to download all, i put those links to a file. When i cat, i puts it line by line to standard output. Then I can pipeline and ask xargs to pass it one by one to wget.

cat list.txt | xargs wget -c

It is recommended to put -0 for xargs for filename, coz sometimes the filename consist of symbol.So

cat list.txt | xargs -0 wget -c

It doesn’t seems to be working, thanks durian for commenting this out. If your links consist of symbol like single quote ( ‘ ) or other reserved symbols, I ll suggest you to edit your link manually by putting escape (\).

Let say my original link look like this


Observered the “You’re”, change it to “You\’re”


10 Responses to “xargs use stardard output as parameter for another command”

    xargs does not call ‘wget’ several times and pass a list of whitespace-delimited items one by one as argument each time.

    In fact, it calls ‘wget’ just once, and passes the list as argument list, similar to this:

    wget -c `cat list.txt`

    To pass several string one by one to a command, here’s a one-liner by mysurface:

    cat list.txt | while read line; do wget $line; done
  2. Dude, your comment is not right. wget may be called more than once. A program can only receive a certain amount of arguments. 512 is the maximum number of arguments for any program, if my memory serves me right. Anyway, that’s not important. The point is xargs will invoke the command repeatedly until all the arguments are consumed.

    cat list.txt|xargs -0 wget -c

    wouldn’t work properly unless all the lines in
    list.txt terminate with a null. That’s not normally
    done in a text file because lines are usually terminated with a newline character, not null character. Have you actually tried the command before posting?

  3. Yes, you are right! Thankyou for correcting my mistake. Let me edit my post.

  4. Another example, you can use delimiter option of xargs.

    cat list.txt|xargs –delimiter=”\n” wget -c

    That way you won’t have to worry about escaping the blank spaces with %20.

  5. durian, thanks for pointing out the error. Now I learnt. :-)

    But one thing to emphesis is, xargs does not pass 10 items “one by one” to the command by launching the command 10 times.

    So, don’t ever think that xargs works as a magical loop (as I did trying to loop a script with xargs.. lol).

  6. toydi, I think you are right about that. Although, I can’t be too sure about the exact mechanism used by xargs. You’re welcome :). We learn through mistakes.

  7. [...] This is a short example, actually I think of put it in as comment of post xargs use stardard output as parameter for another command [...]

  8. To do the job, this is shorter:

    xargs -d"\n" wget -c < list.txt

    And wget -i actually did a better job

    wget -i list.txt

    Me misuse cat :P
    Read http://partmaps.org/era/unix/award.html

  9. haha :)

  10. Not to mention wget have a -i parameter to input files with links ;)
    Nevermind, your xargs example is a decent one. xargs is a very cool utility, which replaces lots of scripting work.

Leave a Reply