Home » Linux Commands » Linux sed Command (Find and Replace Strings in Files)
Linux sed Command (Find and Replace Strings in Files)
Last Updated on by Joshua James
The Linux sed command is a potent stream editor designed for text manipulation, including the ability to find and replace strings in files. This command-line utility is particularly useful for system administrators, programmers, and data scientists who must automate text editing tasks.
Key Features of the Linux sed Command
String Replacement: At its core, sed excels in finding and replacing strings, making it a go-to tool for editing text files.
Regular Expressions: Beyond simple string replacement, sed supports complex pattern matching using regular expressions.
In-Place Editing: Sed offers options for editing files directly, eliminating the need to create separate output files.
Case-Insensitive Matching: The command allows for case-insensitive search and replace, broadening its utility.
Global Replacement: Sed can replace all occurrences of a pattern in a file, not just the first one it encounters.
Given its robust capabilities, the Linux sed command is an indispensable tool for anyone who regularly works with text files. However, it’s crucial to exercise caution when using sed, especially for operations that modify multiple files. Always have a backup to avoid unintended data loss. In the following guide, we’ll delve into the practical applications of the Linux sed command, equipping you with the skills to make the most of this powerful utility.
Table of Contents
Understanding the sed Command
The sed command is available in several versions, with slight functional differences. For instance, macOS utilizes the BSD version, while most Linux distributions, by default, come pre-installed with GNU sed. In this guide, we will focus on the GNU version.
The general syntax for using sed to search and replace text is as follows:
sed -i 's/SEARCH_REGEX/REPLACEMENT/g' INPUTFILE
Let’s break down the components of this command:
-i: By default, sed writes its output to the standard output. The -i option instructs sed to edit files in place. If an extension is supplied (for example, -i.bak), a backup of the original file is created.
s: This is the substitute command, probably the most used command in sed.
/ / /: These are delimiter characters. While any character can be used as a delimiter, the slash (/) character is commonly used.
SEARCH_REGEX: This is the normal string or a regular expression that sed will search for.
REPLACEMENT: This is the string that will replace the SEARCH_REGEX.
g: This is the global replacement flag. By default, sed reads the file line by line and changes only the first occurrence of the SEARCH_REGEX on a line. When the g flag is provided, all occurrences are replaced.
INPUTFILE: This is the name of the file on which you want to run the command.
Enclosing the argument in quotes is a good practice to prevent the shell meta-characters from expanding.
Basic sed Command Usage for String Replacement
For the purpose of demonstration, let’s assume we have the following content in a file named file.txt:
123 Foo foo foo
foo /bin/bash Ubuntu foobar 456
If we omit the g flag, only the first instance of the search string in each line is replaced:
sed -i 's/foo/linux/' file.txt
The output will be:
123 Foo linux foo
linux /bin/bash Ubuntu foobar 456
With the global replacement flag, sed replaces all occurrences of the search pattern:
sed -i 's/foo/linux/g' file.txt
The output will be:
123 Foo linux linux
linux /bin/bash Ubuntu linuxbar 456
As you might have noticed, the substring foo inside the foobar string is also replaced in the previous example. If this is not the desired behavior, use the word-boundary expression (\b) at both ends of the search string. This ensures the partial words are not matched.
sed -i 's/\bfoo\b/linux/g' file.txt
The output will be:
123 Foo linux linux
linux /bin/bash Ubuntu foobar 456
To make the pattern match case insensitive, use the I flag. In the example below, we are using both the g and I flags:
sed -i 's/foo/linux/gI' file.txt
The output will be:
123 linux linux linux
linux /bin/bash Ubuntu linuxbar 456
If you want to find and replace a string that contains the delimiter character (/), you’ll need to use the backslash (\) to escape the slash. For example, to replace /bin/bash with /usr/bin/zsh, you would use:
sed -i 's/\/bin\/bash/\/usr\/bin\/zsh/g' file.txt
A more readable option is to use another delimiter character. Most people use the vertical bar (|) or colon (:), but you can use any other character:
Regular expressions can be used with sed to perform more complex text manipulations. For instance, if you want to search for all 3-digit numbers and replace them with the string number, you would use:
sed -i 's/\b[0-9]\{3\}\b/number/g' file.txt
The output will be:
number Foo foo foo
foo /bin/bash Ubuntu foobar number
Another powerful feature of sed is the use of the ampersand character (&), which corresponds to the matched pattern. This character can be used multiple times. For example, if you want to add curly braces {} around each 3-digit number, you would type:
Now, let’s delve into some more advanced examples.
You can specify the line number before the s command to replace a string only on a specific line. For example, to replace foo with linux only on the second line, you would use:
sed '2s/foo/linux/' file.txt
You can also replace a string on a range of lines. For example, to replace foo with linux on lines 2 to 4, you would use:
sed '2,4s/foo/linux/' file.txt
If you want to delete lines containing a specific string, you can use the d command. For example, to delete all lines containing foo, you would use:
sed '/foo/d' file.txt
To insert a line before or after a line containing a specific string, you can use the i (insert) or a (append) command. For example, to insert a line before each line containing foo, you would use:
sed '/foo/i\This is a new line' file.txt
To change (replace) a line containing a specific string, you can use the c command. For example, to replace all lines containing foo with This is a new line, you would use:
sed '/foo/c\This is a new line' file.txt
Guidelines for Using sed Effectively
When manipulating files with sed, creating a backup of the original file is always prudent. This can be easily accomplished by providing an extension for the backup file to the -i option. For instance, to modify the file file.txt and save the original file as file.txt.bak, you would use:
sed -i.bak 's/foo/linux/g' file.txt
To make sure that the backup is created, list the files with the ls command:
ls
The output will be:
file.txt file.txt.bak
In addition to creating backups, there are several other best practices to consider when using sed. One such practice is always testing your sed commands without the -i option first. This allows you to see the changes that will be made without actually modifying the file. Once satisfied with the results, you can run the command with the -i option to make the changes permanent.
Another best practice is to use the -r (or -E for macOS) option when working with extended regular expressions. This option makes sed more powerful and flexible, allowing you to use additional regular expression features.
Lastly, when working with large files, using the –unbuffered or -u option is advisable. This option tells sed to load the file into memory in smaller chunks, which can significantly improve performance.
Recursive Find and Replace with sed
There are instances where you might need to search directories recursively for files containing a specific string and replace that string in all the files. This can be achieved by using commands such as find or grep to recursively locate files in the directory and then piping the file names to sed.
The command below will recursively search for files in the current working directory and pass the file names to sed:
find . -type f -exec sed -i 's/foo/bar/g' {} +
However, files with spaces in their names can pose a problem. To circumvent this issue, use the -print0 option, which instructs find to print the file name, followed by a null character. You can then pipe the output to sed using xargs -0:
find . -type f -print0 | xargs -0 sed -i 's/foo/bar/g'
Sometimes, you might want to search and replace text only in files with a specific extension. For instance, if you want to replace text in Markdown files only, you would use:
find . -type f -name "*.md" -print0 | xargs -0 sed -i 's/foo/bar/g'
Another approach is to use the grep command to recursively find all files containing the search pattern and then pipe the filenames to sed. The -rlZ option tells grep to output the names of files with matching lines, print a null byte instead of the character that normally follows a file name, and to search directories recursively. The xargs -0 command then reads items from the standard input, delimited by null characters instead of whitespace, and executes the command (in this case, sed):
grep -rlZ 'foo' . | xargs -0 sed -i.bak 's/foo/bar/g'
This command also creates a backup of the original files before performing the replacement, which is a good practice when changing multiple files.
Final Thoughts on Using sed for String Manipulation in Files
In this guide, we’ve delved into the powerful sed command, a versatile tool for finding and replacing strings in files on Linux systems. We’ve explored its syntax, dissected its components, and demonstrated its usage through practical examples. From basic string replacement to advanced usage with regular expressions, sed is an invaluable tool for text manipulation.
Moreover, we’ve highlighted the importance of careful usage, especially when performing recursive find and replace operations. Always ensure you have a backup of your files before running such commands, as they can affect multiple files simultaneously.
