Bash - Paths and file names manipulation

2018-02-06 · bash, linux, scripting

In this post, we dive into some of the interesting ways Bash lets you work with file names and paths. This is usually helpful when you want to write a quick Bash script to copy, move, or rename files. I recently had to deal with hundreds of log files generated by dozens of containers. The issue was that each container writes logs inside its own folder, but the log files all have the same names inside each folder.

    # tree /var/log/containers
    ├── Container_01
    │   ├── java.log
    │   ├── access.log
    │   ├── security.log
    │   └── application.log
    ├── Container_02
    │   ├── java.log
    │   ├── access.log
    │   ├── security.log
    │   └── application.log

As you can see from the example above, if I want to back up the log files and move them all to a remote server, I have two options:

For each directory, I will create a mirror Directory and then move the log files there.
Rename each file so it shows which directory it belongs to — for example Container_01-java.log or java-Container_01.log

For me, the second option works best — this way I don’t have to worry about multiple directories on the remote backup host, and all the log files stay under the same location. To achieve this, I had to do some path manipulation.

I need to get the directory name. Assuming our path is /var/log/containers:

    for D in /var/log/containers/*; do
        BASE=${D##*/} # This is the same as "dirname $D"
        echo $BASE
    done

This gets the name of each directory inside the path we provided and saves the value in the variable BASE, which we can use later. The echo statement is not necessary — it is there just for debugging. Note that the loop produces a full path; ##*/ strips everything up to the last slash and gives us the directory name.

Now, for each directory gathered in the previous step, we need to get the list of files inside it. To achieve this:

    for F in $D; do
        echo $F
    done

Again, this gives us the full path. To manipulate it, we need to slice the path into two parts: the full path minus the file name, assigned to one variable, and the short file name, assigned to another.

To get the path minus the file name => %/* To get the file name => ##*/ (similar to what we used to get BASE):

    for F in $D/*; do
        FULLPATH=${F%/*}
        FILENAME=${F##*/}
    done

Rename or move the files

At this point, all we have to do is combine the variables to move the files:

    mv $F $FULLPATH/$BASE-$FILENAME
    # This is the full command if we were to run it from a terminal:
    # mv /var/log/containers/Container_01/java.log /var/log/containers/Container_01/Container_01-java.log

If we put it all together, we get the following snippet:

    for D in /var/log/containers/*; do
        BASE=${D##*/}
        for F in $D/*; do
            FULLPATH=${C%/*}
            FILENAME=${C##*/}
            mv $F $FULLPATH/$BASE-$FILENAME
        done
    done

Some other methods to manipulate files:

    FILE=/path/to/file.txt

    # To get file name without extension:
    echo ${FILE%.*}

    # To get the file's directory name:
    echo ${FILE%/*}
    # or:
    dirname $FILE

    # To get the file name and extension:
    basename $FILE

#bash #linux #scripting