Categories
josephscott

Perl Oneliner: Recursive Search and Replace

If you’ve used Perl at all you are probably familiar with the simple oneliner to do a search and replace on a given string:

perl -p -i -e 's/oldstring/newstring/g' *

This will replace all occurrences of oldstring with newstring in all of the files in your current directory. Being able to do this quickly and easily is absolutely awesome, but what if you want to do this recursively across the current directory and all directories below that? I’m sure that there are plenty of ways to do this, the one below is the method that seems the easiest:

perl -p -i -e 's/oldstring/newstring/g' `find ./ -name *.html`

If you wanted to be more precise you could replace find with grep and only perform the search and replace on files that you know already contain oldstring, like this:

perl -p -i -e 's/oldstring/newstring/g' `grep -ril oldstring *`

This is one of those things that I don’t have to do very often, but when I do, this Perl oneliner is a real life saver.

25 replies on “Perl Oneliner: Recursive Search and Replace”

Couple problems. First, you have to quote ‘*.html’ or the shell will expand it and find will barf. Also, if there are spaces in any of the file names that find returns, perl will barf. So, as Klaus suggests, you should use find . -name '*.html' -print0 | xargs -0 perl -pi -e 's/oldstring/newstring/g'

Klaus is right because the find clause might produce enough output that it will reach the limit for line length (input too long). Remarkably, this happens frequently enough. I remember having to debug somebody’s perl script that was written to search through 750gb of text files. One day, the number just managed to hit the max and it errored out with “input too long” messages.

find . | xargs perl -p -i -e ‘s/something/else/g’

However, for smaller directories, I really like your second solution, grepping good matches out first. You should run some time trials to see if it’s a speed win to have grep do the detection and then only perl the ones you need. I bet there’s a break-even where less than some percentage of relevant files will give you a speed advantage.

-Josh-

Just one more way to do:

find ./ -exec perl -p -i -e ‘s/faltoo_string/aur_bhi_faltoo_string/g’ {} ;

Shell Magic: Globally Find/Replace Text Using RegExps Within Many Files…

Here’s one of my favorite Perl-based one-liners which I whip out when I need to find a bunch of files and replace text based on regular expressions at the command line.
find . -iname ‘*.js’ -exec perl -pi.bak -e ’s/2006/2007/g&#8217…

Using grep to select the files is a performance killer : It means that the files matching the regexp will be scanned twice : Once by grep and once by perl. If the file is big, it’s a problem.

Not selecting the files that will be changed is a problem: Files that do not match will change (ie time, inode). Not good for keeping timestamps of unchanged files (think for example source files: make will rebuild the objects!)

/me is wondering if perl has an option “do not replace the file by a new one if it did not change, just keep it”.

I tried all these examples to no avail using Solaris9, just trying to find all files in filesystem with *.zip and change it to *.ZIP.

# perl -v

This is perl, v5.6.1 built for sun4-solaris-64int
(with 48 registered patches, see perl -V for more detail)

# find /usr/local/tmp/aaa -name “*.zip” -exec perl -p -i -e ‘s/zip/ZIP/g’ {} ;

# ls -l /usr/local/tmp/aaa
total 6
-rw-r–r– 1 carlos staff 49 Feb 22 09:35 9_Recommended.zip
-rw-r–r– 1 carlos staff 49 Feb 22 09:35 DLG_V11_2.zip
-rw-r–r– 1 carlos staff 49 Feb 22 09:35 tzupdater-1_0_1.zip
#

Single quotes doesn’t work too

# find /usr/local/tmp/aaa -name ‘*.zip’ -exec perl -p -i -e ‘s/zip/ZIP/g’ {} ;
# ls -l /usr/local/tmp/aaa
total 6
-rw-r–r– 1 carlos staff 49 Feb 22 09:37 9_Recommended.zip
-rw-r–r– 1 carlos staff 49 Feb 22 09:37 DLG_V11_2.zip
-rw-r–r– 1 carlos staff 49 Feb 22 09:37 tzupdater-1_0_1.zip

Carlos, perl -pi -e replaces the CONTENTS of the file. You are trying to RENAME the file. Two very different things. See if you have the rename command; it is a perl script and will let you rename a file using regex substitution.

Hey There,

For “not replacing” files that already exist, I use this with that perl one-liner when I’m piping filenames to it – it’s not built into the options, but is pretty simple to type once πŸ™‚

perl -p -i -e ‘s/ThisThing/ThatThing/; rename $_, unless -e $_;’

It doesn’t work “out of context” because $_ requires a value, but you can use the rename half of my version of the one-liner to insert in the previous code to prevent against overwriting πŸ™‚

Great post πŸ™‚

, Mike

hello all

I execute this:

# perl -p -w -e ‘s/(?:n)(.*)//g’ test1.txt

And get this:

Bareword found where operator expected at -e line 1, near “s/(?:n)(.*)</collectioninfo”
Unquoted string “collectioninfo” may clash with future reserved word at -e line 1.
syntax error at -e line 1, near “s/(?:n)(.*)</collectioninfo”
Execution of -e aborted due to compilation errors.

Any help ?

Thx !!

Hey There,

That error almost always has to do with unmatched quotation marks. Probably, the easiest way to figure out what’s wrong with it would be to turn your one-liner into a simple script. You’ll get waaaaaay too much information, but the answer will probably be in there.

If not, check test1.txt for any unmatched double quotes, single quotes, backticks, etc and that may lead you to the answer, as well.

Best wishes,

Mike

Thanks a bunch! This saves me a lot of time on a daily basis as I’ve started using this in a script. Just thought I’d come let you know it helped. Although you need to add “quotes” around your find target for this to work properly.

I love perl, but for these use-cases sed would have been more than sufficient?

sed -i ‘s/oldstring/newstring/g’ *

Same w/ the other examples …

Cheers,
Andrej

sed, as shipped with many OS installs, doesn’t work with escaped characters, so it can’t be used for many cases.

echoing what everyone said above, but adding the -f switch for find will smooth things out if you happened to have a dir name which ended in dot html:
perl -p -i -e ‘s/oldstring/newstring/g’ `find ./ -type f -name “*.html” `

I like it with some env variables:

This is how I replaced the problematic with the equivalent suitable for parsing in a php environment. (Since the <? short-code invokes php on many server configs)

# assign shell environment variables
OLDSTRING='r?’
NEWSTRING=’php echo(“”); ?>’
# quoting the backslashes important here!

# invoke perl regex on the results of the find command interpreted in backquotes
perl -p -i -e “s/$OLDSTRING/$NEWSTRING/” `find ./ -type f -name “*.html”`

Which works great although I’m perplexed with the above by the fact that if I included “<?" before "php echo" in NEWSTRING, the result had "<?<?php …" which, of course causes problems.

PF

GaryB:s way is the old school way and in my opinion the best way to do it. Just a minor improvement to include just the files named *.html:

find ./ -type f -name ‘*.html’ -exec perl -i -pe ‘s/oldstring/newstring/g’ {} ;

ack -f | xargs perl -lane ‘print if s/z/Z/g’
#prints modified lines only, no changes made

ack -f | xargs perl -pi -e ‘s/z/Z/g’
#changes made, prints nothing

find . -name someFile.txt -exec perl -p -i -e ‘s/oldString/newString/g’ {} ;

The above command will search recursively for a file named `somFile.txt` and it will perform the string replace.

echo “entre MDn”
read mdn
echo “mdn is $mdn”
perl -p -ne ‘s/MDN/$mdn/g’ /cmpnt/AWCC1.0/AWCCSST3/RB/install/fid/murali/pos/mms/*
echo “changed”
i have ran the folloeing script but ithe value did not get replaced it replaced with a space please suggest

Building off of Nic’s and others’ posts, this works for file names including spaces and doesn’t update files that don’t need updating:
find . -type f -name ‘*.html’ -print0 | xargs -0 grep -lZ ‘oldstring’ | xargs -0 perl -pi -e ‘s/oldstring/newstring/g’

Leave a Reply

Your email address will not be published. Required fields are marked *