Thursday, January 16, 2020

big data, big money: Snowflake Inc.


Snowflake Inc. is a cloud-based data-warehousing startup that was founded in 2012. It has raised more than $900 million in venture capital...
valuation to $3.5 billion 
Snowflake runs on Amazon S3 since 2014, and on Microsoft Azure since 2018

podcast interview

See Snowflake in 8 Minutes @YouTube

nowflake Architecture - Learn How Snowflake Stores Table data @YouTube

free ebooks

Key Concepts & Architecture — Snowflake Documentation

Architecture overview

AWS Glue

AWS Glue - Amazon Web Services

AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog. Once cataloged, your data is immediately searchable, queryable, and available for ETL.

Queries against an Amazon S3 Data Lake diagram

Saturday, January 11, 2020

dapr: runtime for building microservices: cloud native, mesh, sidecar

An event-driven, portable runtime for building microservices on cloud and edge. (GoLang, MIT license)

"Dapr injects a side-car container/process to each compute unit. 
The side-car interacts with event triggers and communicates with the compute unit via standard HTTP or gRPC protocols. This enables Dapr to support all existing and future programming languages without requiring you to import frameworks or libraries.
Dapr offers built-in state management, reliable messaging (at least once delivery), triggers and bindings through standard HTTP verbs or gRPC interfaces. This allows you to write stateless, stateful and actor-like services following the same programming paradigm. You can freely choose consistency model, threading model and message delivery patterns.

Dapr runs natively on Kubernetes, as a standalone binary on your machine, on an IoT device, or as a container that can be injected into any system, in the cloud or on-premises.

Dapr uses pluggable state stores and message buses such as Redis as well as gRPC to offer a wide range of communication methods, including direct dapr-to-dapr using gRPC and async Pub-Sub with guaranteed delivery and at-least-once semantics."

Dapr Conceptual Model

"Dapr is a distributed application runtime. Unlike a service mesh which is focused on networking concerns, Dapr is focused on providing building blocks that make it easier for developers to build microservices. Dapr is developer-centric versus service meshes being infrastructure-centric."
Learn all about Distributed Application Runtime (Dapr), Part 1 | Azure Friday @YouTube (Channel 9)
Part 2

Dapr, Rudr, OAM | Mark Russinovich presents next gen app development & deployment (Microsoft Ignite)

microservices app diagram

Sidecar pattern
Sidecar Design Pattern

A sidecar is a one-wheeled device attached to the side of a motorcycle, scooter, or bicycle,[1] making the whole a three-wheeled vehicle. A motorcycle with a sidecar is sometimes called a combination, an outfit, a rig or a hack

Sidecar on Vespa scooter

Service Mesh

"In software architecture, a service mesh is a dedicated infrastructure layer for facilitating service-to-service communications between microservices, often using a sidecar proxy.

Having such a dedicated communication layer can provide a number of benefits, such as providing observability into communications, providing secure connections, or automating retries and backoff for failed requests."

course: Sustainable Building Design | edX | MITx

Sustainable Building Design | edX

"Learn and explore key scientific principles, technologies, and analysis techniques for designing comfortable indoor environments while reducing energy use and associated climate change effects."
Sustainable Building Design

Thursday, January 09, 2020

AWS CloudFormation vs. Terraform

CloudFormation vs. Terraform | cloudonaut


CloudFormation: using JSON or YAML for templates

  "Parameters": {
    "KeyName": {
      "Description": "The EC2 Key Pair for SSH access",
      "Type": "AWS::EC2::KeyPair::KeyName"
  "Resources": {
    "CFExampleInstance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "SecurityGroups": [ { "Ref": "CFSSHAccess" } ],
        "KeyName": { "Ref": "KeyName" },
        "ImageId": "ami-0f4cfd64",
        "InstanceType": "t1.micro"
    "CFSSHAccess": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupDescription": "SSH access",
        "SecurityGroupIngress": [ {
          "IpProtocol": "tcp",
          "FromPort": "22",
          "ToPort": "22",
          "CidrIp": ""
        } ]
    Description: The EC2 Key Pair for SSH access
    Type: AWS::EC2::KeyPair::KeyName
    Type: AWS::EC2::Instance
      - Ref: CFSSHAccess
        Ref: KeyName
      ImageId: ami-0f4cfd64
      InstanceType: t1.micro
    Type: AWS::EC2::SecurityGroup
      GroupDescription: SSH access
      - IpProtocol: tcp
        FromPort: '22'
        ToPort: '22'
Teraform custom syntax in HCL (HashiCorp Configuration Language)

variable "key_name" {}

resource "aws_instance" "TFExampleInstance" {
  ami = "ami-0f4cfd64"
  instance_type = "t1.micro"
  key_name = "${var.key_name}"
  security_groups = [ "${}" ]

resource "aws_security_group" "TFSSHAccess" {
  name = "TFSSHAccess"
  description = "SSH access"

  ingress {
    protocol = "tcp"
    from_port = 22
    to_port = 22
    cidr_blocks = [""]

hashicorp/terraform @GitHub

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned. 

Wednesday, January 08, 2020

C++ smart pointers vs Rust

Smart pointers (Modern C++) | Microsoft Docs

"As shown in the example, a smart pointer is a class template that you declare on the stack, and initialize by using a raw pointer that points to a heap-allocated object. After the smart pointer is initialized, it owns the raw pointer. This means that the smart pointer is responsible for deleting the memory that the raw pointer specifies. The smart pointer destructor contains the call to delete, and because the smart pointer is declared on the stack, its destructor is invoked when the smart pointer goes out of scope, even if an exception is thrown somewhere further up the stack."

void UseRawPointer() { // Using a raw pointer -- not recommended. Song* pSong = new Song(L"Nothing on You", L"Bruno Mars"); // Use pSong... // Don't forget to delete! delete pSong; } void UseSmartPointer() { // Declare a smart pointer on stack and pass it the raw pointer. unique_ptr song2(new Song(L"Nothing on You", L"Bruno Mars")); // Use song2... wstring s = song2->duration_; //... } // song2 is deleted automatically here.

Smart Pointers in C++ - GeeksforGeeks

intro/smart pointers -

How to think about Rust ownership versus C++ unique_ptr · Franklin Chen

Learning smart pointers in C++ and Rust | Junchao's blog

Smart Pointers - The Rust Programming Language

Tuesday, January 07, 2020

Apache NiFi: NiagaraFiles, flow based programming

Apache NiFi - Wikipedia

Apache NiFi is a software project from the Apache Software Foundation designed to automate the flow of data between software systems. It is based on the "NiagaraFiles" software previously developed by the NSA, which is also the source of a part of its present name – NiFi.
Flow-based programming - Wikipedia

flow-based programming (FBP) is a programming paradigm that defines applications as networks of "black box" processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented.

FBP is a particular form of dataflow programming based on bounded buffers, information packets with defined lifetimes, named ports, and separate definition of connections.

Saturday, January 04, 2020

AWS-shell vs AWS CLI + JMESPath

AWS Command Line Interface and aws-shell Sample for AWS Cloud9 - AWS Cloud9

"The AWS CLI and the aws-shell are unified tools that provide a consistent interface for interacting with all parts of AWS. You can use the AWS CLI or the aws-shell instead of the AWS Management Console to quickly run commands to interact with AWS, and some of these commands can only be run with the AWS CLI or the aws-shell."

AWS Shell demo video @ OReilly Learning (subscription or trial needed)
AWS CLI overview code
a query language for JSON.

JMESPath Tutorial — JMESPath

locations[?state == 'WA'].name | sort(@) | {WashingtonCities: join(', ', @)}


  "WashingtonCities": "Bellevue, Olympia, Seattle"

AWS Architecture & Solutions

Architecture Resources | AWS

Cloud Solutions | Vetted, Technical Reference Architecture | AWS

AWS Well-Architected - Build secure, efficient, cloud enabled applications

AWS Well-Architected Tool - Amazon Web Services


Wednesday, January 01, 2020

book Paradigms of Artificial Intelligence Programming @ GitHub

open-source repository for the book Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp by Peter Norvig (1992), and the code contained therein.

ELIZA in Python

ELIZA - Wikipedia

"ELIZA is an early natural language processing computer program created from 1964 to 1966[1] at the MIT Artificial Intelligence Laboratory by Joseph Weizenbaum.[2] Created to demonstrate the superficiality of communication between humans and machines, Eliza simulated conversation by using a "pattern matching" and substitution methodology that gave users an illusion of understanding on the part of the program, but had no built in framework for contextualizing events."

Eliza, Computer Therapist (online chat)
This is a little version of ELIZA, a famous natural-language AI demo from the 1960s, packaged up as a Python module. It's all smoke and mirrors; the program doesn't have a clue what it is saying and it's not difficult to catch it out, but it's amusing and it means your chatbot always has something to say.

wadetb/eliza: Python implementation of the Eliza chatbot @GitHub
Loosely based on Charles Hayden's version in Java, at @GitHub
This is a Processing library based on the Java implementation of the Eliza program by Charles Hayden. Hayden's code is a complete and faithful implementation of the original program described by Joseph Weizenbaum in the Communications of the ACM in January 1966.