Merging videos using gstreamer

My brother came to me the other day asking if I knew of a way to merge multiple small (approximately 30 seconds each) video files into a single, longer, video file. He works for a retail business and wanted to review some security footage more easily.

The GStreamer framework, as of version 1.6, now has a plugin that makes doing this operation significantly easier than ever before. The concat plugin simply takes 1 to N number of inputs, and activates those inputs one at a time, playing each entirely, filtering the EOS event from all but the last element, and then moving to the next. In this way, we can make other elements in the pipeline treat these multiple sources of data as a single, continuous, stream.

The ultra-simplified explanation is that we're simply playing each small video one after another with no pause in between, and then re-encoding the resulting video as a single file.

The problem that I was running into is two fold. First, the some aspect of the contact element, when running from the commandline using the gst-launch-1.0 debugging / prototyping program, becomes very hostile to handling more than, for example, 10 files at a time. I was able to consistently use a pipeline that looks like this

gst-launch-1.0 concat name=c \

    uridecodebin uri=file:///path/to/video1.asf ! queue ! c. \

    uridecodebin uri=file:///path/to/video2.asf ! queue ! c. \

    uridecodebin uri=file:///path/to/video3.asf ! queue ! c. \

    uridecodebin uri=file:///path/to/video4.asf ! queue ! c. \

    uridecodebin uri=file:///path/to/video5.asf ! queue ! c. \

    c. ! videoconvert ! theoraenc ! progressreport name="Encoding\ Progress" ! matroskamux ! filesink location=/path/to/output/file.mkv ;

To merge a handful of files at a time, but trying to do more than roughly 10 made gst-launch throw up errors very consistently

My second problem, which gets drastically compounded by the first problem, is that my brother handed me a thumb drive with over 5 thousand video files on it. Personally, I have no interest in sitting there typing out each file name one at a time, and to the best of my, admittedly limited) knowledge, gstreamer simply doesn't have an element that opens each file that matches some pattern and attaches them to a sink in a way that would be useful for this situation.

So, I need to write a meta program that generates a program that calls gst-launch with only a few video files at a time. I figured that if I could merge them 5 at a time, I could merge the resulting videos 5 at a time, and repeat that pattern until there were only around a dozen big video files, at which point he can search through them manually on his own.

So here's what I've come up with so far. It's a bit of a big-huge-hack, but at least it'll let me get the job done and move onto other projects.

OUTPUTSCRIPT="path_to_script_name.sh"
VIDEOFOLDER="/path/to/directory/containing/videos/"
COUNTER=0
NUMBER_PER_SWEEP=10
FIRST=""
echo -n > $OUTPUTSCRIPT
cd $VIDEOFOLDER
mkdir -p encode_results
echo "runfunction() {" >> $OUTPUTSCRIPT
echo "    until \$@" >> $OUTPUTSCRIPT
echo "    do" >> $OUTPUTSCRIPT
echo "        :" >> $OUTPUTSCRIPT
echo "    done" >> $OUTPUTSCRIPT
echo "}" >> $OUTPUTSCRIPT
echo "" >> $OUTPUTSCRIPT
for f in `ls *.ASF | sort -g` ;
do
    if [ $COUNTER -eq 0 ] ;
    then
        FIRST="$f"
        echo "runfunction gst-launch-1.0  --no-fault --gst-debug-disable concat name=c \\" >> $OUTPUTSCRIPT
    fi
    echo "    uridecodebin uri=file://$VIDEOFOLDER/$f ! videorate ! video/x-raw,framerate=[1/1,10/1] ! queue ! c. \\" >> $OUTPUTSCRIPT
    COUNTER=$(($COUNTER+1))
    if [ $COUNTER -eq $NUMBER_PER_SWEEP ] ;
    then
        COUNTER=0
        echo "    c. ! videoconvert ! theoraenc ! progressreport name=\"Encoding\ Progress\" ! matroskamux ! filesink location=$VIDEOFOLDER/encode_results/encode_result_${FIRST}_${f}.mkv &" >> $OUTPUTSCRIPT
        echo "" >> $OUTPUTSCRIPT
    fi
done
if [ $COUNTER -ne 0 ] ;
then
    COUNTER=0
    echo "    c. ! videoconvert ! theoraenc ! progressreport name=\"Encoding\ Progress\" ! matroskamux ! filesink location=$VIDEOFOLDER/encode_results/encode_result_${FIRST}_${f}.mkv &" >> $OUTPUTSCRIPT
    echo "" >> $OUTPUTSCRIPT
fi

You should be able to simply copy it into a bash prompt after changing the filenames at the top.

It'll generate a new script. That new script will all the encoding jobs in parallel. If any encoding job fails to complete successfully, then the "runfunction" will automatically start it again.