Analyzing security camera footage with AWS Rekognition

Some of this code can be obtained here:  https://bitbucket.org/lizardworks/aztec-rekog1/overview


THE PROBLEM

 After my house was broken into, I got a simple game camera and set it up to take pictures that would hopefully identify the recreants should they strike again.  And so I wound up with lots of data footage that needs to be analyzed - over 600 AVI files.  Rather than go through this like a diligent hard working detective, I decided to be lazy and automate it as much as I could using AWS's new service, Rekognition.

About Rekognition

Rekognition is an "AI" system that specializes in identifying possible objects in pictures.  It identifies such as 'labels' and confidence value pairs.  The console for this subsystem is actually tied to the demo, and is therefore not useful for general purposes.


First Part - Creating snapshots from the .AVI footage

 Pymedia seemed to be the solution, but was abandoned. It was tightly coupled to the python version, and thus no current version existed. Then I found a command line utility1 that does just about everything, if you know how to use it. I'm not going to claim that I am this competent - however scanning the docs, I found an example that pretty much does what I need:

 Ultimately, I hacked together…er, I automated a program to do what I need :

use File::Basename;

# glob all .AVIs

# globals

$datadir = "/Volumes/NO\\ NAME/DCIM/100EK113";

# can't check $datadir: glob and -d disagree on the escaping

#die "data dir ($datadir) not found !"

# if !-d $datadir;

@list = glob "$datadir/*.AVI";

# clear data directory

unlink $_ foreach ;

my $counter = 0;

foreach my $file (@list)

{

my $basename = basename($file);

print "$counter : $basename\n";

$com = "ffmpeg -i $datadir/$basename -r 0.5 -f image2 snapshots/$basename-%03d.jpeg";

# TODO: TRY system() - get rid of the program's output somehow

my $r = `$com`;

++$counter;

# uncomment to restrict processing

##die if $counter >= 3;

}

Listing - snap.pl 

Second part - Enter AWS Rekognition

 Rekog is a curious thing; references about to it having a console, but what is there is clearly for demo purposes only. If you want to work with batch files, you must use the API programmatically it seems. Given that free tier is limited to 5,000 requests / month, and I had approximately 6,500 snapshots, I decided to split the file set into two parts, do one set now and one set in a couple of days when the month would change. Since I had a list of filenames2 it was an easy matter to split them using a Unix tool called, oddly enough, "split": 

 Split -l 5000 allfiles 

 This generated two files, xaa & xab, which I promptly renamed to filelist1.txt and filelist2.txt, not that that matters. Then I hacked up another …automation…to use the AWS API and ensure victory: 

// Ref: Amazon Rekognition SDK: http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-nodejs.html

// Load the SDK

var AWS = require('aws-sdk');

var rekognition = new AWS.Rekognition();

var fs = require('fs');

//* Read in list of files to process

var array = fs.readFileSync('files2.txt').toString().split("\n");

//* Loop through list processing for each file

for(i in array)

{

var file = array[i];

console.log(i, ". processing " + file);

examine (file); // for example, "01020403.bird.AVI-001.jpeg")

}

console.log("Program End ?");

// Examine the file using AWS recognition,

// saving the results to a file in the results/ directory.

function examine (filename)

{

// parameters for call

var params = {

Image: {

S3Object: {

Bucket: "phantomtec2",

Name: filename

}

},

MaxLabels: 123

};

// make the call to AWS

rekognition.detectLabels(params, function(err, data) {

if (err)

{

const outfile = "results/" + filename + ".ERROR";

fs.writeFile(outfile,

JSON.stringify(err, undefined, 2),

(err) => { return console.log(err) });

return;

}

else

{

// write results to file

const outfile = "results/" + filename + ".result";

fs.writeFileSync(outfile,

JSON.stringify(data,undefined,2),

(err) => { return console.log(err) });

}

});

} // end - examine

Listing - scan.js

Why didn't I stick to perl ? I could have, but I like to use perl3 to code quick custom utilities on the fly. And since I'm trying to up my javascript game, that’s what I used. 

This spits out a bunch of result files, from which it is a simple matter to grep to see which ones contain 'person' or 'persons', thereby reducing the mass of files down to a few candidates.  I found several false positives, but (as far as I know) no 'false negatives'.

A result file contains the JSON returned from the detectLabels call.  Here's an excerpt:

{
  "Labels": [
    {
      "Name": "Blossom",
      "Confidence": 74.3183364868164
    },
    {
      "Name": "Flora",
      "Confidence": 74.3183364868164
    },
    {
      "Name": "Flower",
      "Confidence": 74.3183364868164
    },
    {
      "Name": "Plant",
      "Confidence": 74.3183364868164
    },

Conclusion

Amazon's Rekognition is indeed impressive.  However I found that sometimes it saw stuff (people) that wasn't there - or maybe they were ninjas, and I just can't see them.  But that's ok, I would rather have a few false postives than misses.  

Being a new technology, I expect it to have many rough edges and unfinished areas.  Indeed, it seems to think that the night pictures (shot in IR) were actually x-rays.  This did not affect processing however.



 REFERENCES & Sundry Notes 

 1. Man ffmpeg - the man listing for this amazing and complex program. 

 2. I used something like "ls /Volumes/NO\ NAME/DCIM/100EK113/*.jpeg >example.filelist" to make the filelist. 

 3. Perl doesn't have an official SDK from Amazon. But it does have at least two unofficial ones - I'm not sure what the status of Rekognition support is with them. So another reason to go javascript/node!  

EOF.