Wednesday, November 25, 2020

Monday, January 1, 2018

The Booklist #0003 : JavaScript, 9th edition (2015) by Negrino & Smith

With all the hype/excitement behind javascript technologies such as node.js, express, mocha, etc. Client-side javascript kinda got pushed to the side.

I found myself needing to know the other side of the street, but never found a book that I liked.

Until now.

Being a Visual Quickstart Guide, you can expect lucid explanations, and this volume is no exception.  I only wish I was this lucid.  You, my dear reader, will just have to continue to suffer ;)

The really great thing I like about this book is, I can pick it up, and learn a technique in 15 minutes or maybe less.  Short and neat, somewhat like 'Scala for the Impatient' - I love books like this !

I also like the fact that it continues to be diligently revised.


https://amzn.to/2A4NzPh


Friday, December 29, 2017

The Booklist: #0001 : Elements by Euclid

Perhaps one of the greatest and yet somewhat misunderstood books of all time.  Elements is a classical work on geometry, which has become a somewhat neglected lore, and poorly understood.

In the beginning, the Greeks came up with a pair of arts, one to shape the body, and one to shape the mentality.  The first, we know as gymnastics, and has become a competitive sport.  The second, has fallen to the wayside, neglected through ignorance.  Geometry was never meant to be an end to itself, but primarily a set of exercises to tone the mind.  That it often does have practical significance is the proverbial icing on the cake.  So the purpose of this art is the first lost secret.

The second lost secret, or misunderstanding perhaps, is the nature of the teaching.  True, it does teach/encourage deductive reasoning...however it takes more than even deduction to attain victory in this art.  What few enough understand is that it takes a pair of assets to trod in the footsteps of the ancient Greeks.  The other asset it takes is this:  Imagination.

Even if one never uses Euclidean geometry in life, this is still an incredibly worthwhile pursuit for exactly those two qualities.  Learning to navigate a geometry essentially IS thinking!  Everything has its rules, its principles whereby it is ordered.  A building cannot stand without an implicit order in its structure, a diamond cannot be cut, nor a bone broken, or a chemical reaction unfolded, without involving some type of geometry.

In creating a reasonably abstract form of geometry, the Ancients have given us a way to perfect true rationality.  Because we changed ourselves, our vision of the universe arose from myth and legend towards wonder and mystery.

The Green Lion Press edition, highly recommended:

https://amzn.to/2WwYRmP




1,000 Books to Greatness: A Journey through technology towards Mastery

I've decided, against my better judgement, to create an imaginary library of one thousand, or thereabouts, books, mostly technical in nature.  This is to illustrate the resources that a professional engineer must have at his disposal if he should aspire to greatness.  In my case, that lofty goal will always be a work in progress, but I have met many great books in my time, and would like to share them.

And so its all over, but for the doing.


Sunday, November 5, 2017

Analyzing security camera footage with AWS Rekognition


Some of this code can be obtained here:  https://bitbucket.org/lizardworks/aztec-rekog1/overview


THE PROBLEM

 After my house was broken into, I got a simple game camera and set it up to take pictures that would hopefully identify the recreants should they strike again.  And so I wound up with lots of data footage that needs to be analyzed - over 600 AVI files.  Rather than go through this like a diligent hard working detective, I decided to be lazy and automate it as much as I could using AWS's new service, Rekognition.

About Rekognition

Rekognition is an "AI" system that specializes in identifying possible objects in pictures.  It identifies such as 'labels' and confidence value pairs.  The console for this subsystem is actually tied to the demo, and is therefore not useful for general purposes.


First Part - Creating snapshots from the .AVI footage

 Pymedia seemed to be the solution, but was abandoned. It was tightly coupled to the python version, and thus no current version existed. Then I found a command line utility1 that does just about everything, if you know how to use it. I'm not going to claim that I am this competent - however scanning the docs, I found an example that pretty much does what I need:

 Ultimately, I hacked together…er, I automated a program to do what I need :

use File::Basename;

# glob all .AVIs

# globals
$datadir = "/Volumes/NO\\ NAME/DCIM/100EK113";
# can't check $datadir: glob and -d disagree on the escaping
#die "data dir ($datadir) not found !"
# if !-d $datadir;


@list = glob "$datadir/*.AVI";

# clear data directory
unlink $_ foreach ;

my $counter = 0;
foreach my $file (@list)
{
my $basename = basename($file);
print "$counter : $basename\n";
$com = "ffmpeg -i $datadir/$basename -r 0.5 -f image2 snapshots/$basename-%03d.jpeg";
# TODO: TRY system() - get rid of the program's output somehow
my $r = `$com`;
++$counter;

# uncomment to restrict processing
##die if $counter >= 3;
}


Listing - snap.pl 


Second part - Enter AWS Rekognition


 Rekog is a curious thing; references about to it having a console, but what is there is clearly for demo purposes only. If you want to work with batch files, you must use the API programmatically it seems. Given that free tier is limited to 5,000 requests / month, and I had approximately 6,500 snapshots, I decided to split the file set into two parts, do one set now and one set in a couple of days when the month would change. Since I had a list of filenames2 it was an easy matter to split them using a Unix tool called, oddly enough, "split": 

 Split -l 5000 allfiles 

 This generated two files, xaa & xab, which I promptly renamed to filelist1.txt and filelist2.txt, not that that matters. Then I hacked up another …automation…to use the AWS API and ensure victory: 


// Ref: Amazon Rekognition SDK: http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-nodejs.html

// Load the SDK
var AWS = require('aws-sdk');
var rekognition = new AWS.Rekognition();
var fs = require('fs');

//* Read in list of files to process
var array = fs.readFileSync('files2.txt').toString().split("\n");

//* Loop through list processing for each file
for(i in array)
{
var file = array[i];
console.log(i, ". processing " + file);
examine (file); // for example, "01020403.bird.AVI-001.jpeg")
}

console.log("Program End ?");


// Examine the file using AWS recognition,
// saving the results to a file in the results/ directory.
function examine (filename)
{
// parameters for call
var params = {
Image: {
S3Object: {
Bucket: "phantomtec2",
Name: filename
}
},
MaxLabels: 123
};

// make the call to AWS
rekognition.detectLabels(params, function(err, data) {
if (err)
{
const outfile = "results/" + filename + ".ERROR";
fs.writeFile(outfile,
JSON.stringify(err, undefined, 2),
(err) => { return console.log(err) });
return;
}
else
{
// write results to file
const outfile = "results/" + filename + ".result";
fs.writeFileSync(outfile,
JSON.stringify(data,undefined,2),
(err) => { return console.log(err) });
}

});
} // end - examine

Listing - scan.js

Why didn't I stick to perl ? I could have, but I like to use perl3 to code quick custom utilities on the fly. And since I'm trying to up my javascript game, that’s what I used. 

This spits out a bunch of result files, from which it is a simple matter to grep to see which ones contain 'person' or 'persons', thereby reducing the mass of files down to a few candidates.  I found several false positives, but (as far as I know) no 'false negatives'.

A result file contains the JSON returned from the detectLabels call.  Here's an excerpt:

{
  "Labels": [
    {
      "Name": "Blossom",
      "Confidence": 74.3183364868164
    },
    {
      "Name": "Flora",
      "Confidence": 74.3183364868164
    },
    {
      "Name": "Flower",
      "Confidence": 74.3183364868164
    },
    {
      "Name": "Plant",
      "Confidence": 74.3183364868164
    },



Conclusion

Amazon's Rekognition is indeed impressive.  However I found that sometimes it saw stuff (people) that wasn't there - or maybe they were ninjas, and I just can't see them.  But that's ok, I would rather have a few false postives than misses.  

Being a new technology, I expect it to have many rough edges and unfinished areas.  Indeed, it seems to think that the night pictures (shot in IR) were actually x-rays.  This did not affect processing however.



 REFERENCES & Sundry Notes 

 1. Man ffmpeg - the man listing for this amazing and complex program. 

 2. I used something like "ls /Volumes/NO\ NAME/DCIM/100EK113/*.jpeg >example.filelist" to make the filelist. 

 3. Perl doesn't have an official SDK from Amazon. But it does have at least two unofficial ones - I'm not sure what the status of Rekognition support is with them. So another reason to go javascript/node!  

EOF.

Just found a fantastic Back-end software engineering channel on youtube!

 Check this guy out! https://www.youtube.com/c/HusseinNasser-software-engineering/videos