Saturday, July 16, 2016

Docker DevOps Around the Globe

When I completed the Docker Birthday #3 App Challenge I was presented with the following telling globe image that shows where other lab completions were occurring around the Globe:

Successful Lab Completions Per Country

Analysis

The biggest surprise, for me, were the number of successful lab completions from the Nordic countries (Norway, Sweden, Finland).

Way to go up (and over) there!

Also of note:
  • More Mid East completions than India
  • Spain includes North Africa and that count was higher that I expected
  • The Kazakhstan count also includes Turkey
  • Participation from Africa
  • No participation from Australia or Far East


Please let me know if I interpreted these numbers properly. Perhaps, part of the map was missing from my view.

My Completion Certificate

That bumped the total count for the US East up to 214.

The Voting App

This is what the running app looked like:

Components

We developed an app using Docker Toolbox that included:

Front End

  • Python webapp which lets you vote between several options
  • Node.js webapp showing the results of the voting in real time

Back End

  • Redis Queue to collect new votes
  • Java Worker which consumes votes and stores them in the database
  • Postgres database backed by a Docker Volume

Architecture Diagram

Project Details

Docker-Compose YAML File


version: "2"

services:
  voting-app:
    build: ./voting-app/.
    volumes:
     - ./voting-app:/app
    ports:
      - "5000:80"
    networks:
      - front-tier
      - back-tier

  result-app:
    build: ./result-app/.
    volumes:
      - ./result-app:/app
    ports:
      - "5001:80"
    networks:
      - front-tier
      - back-tier

  worker:
    image: manomarks/worker
    networks:
      - back-tier

  redis:
    image: redis:alpine
    container_name: redis
    ports: ["6379"]
    networks:
      - back-tier

  db:
    image: postgres:9.4
    container_name: db
    volumes:
      - "db-data:/var/lib/postgresql/data"
    networks:
      - back-tier

volumes:
  db-data:

networks:
  front-tier:
  back-tier:


This Compose file defines

  • A voting-app container based on a Python image
  • A reslit-app container based on a Node.js image
  • A redis container based on a redis image, to temporarily store the data.
  • A Java based worker app based on a Java image
  • A Postgres container based on a postgres image
Note that three of the containers are built from Dockerfiles, while the other two are images on Docker Hub. To learn more about how they're built, you can examine each of the Dockerfiles in the two directories: voting-app, result-app. We included the code for the Java worker in worker but pre-built the image to save on downloads.

The Compose file also defines two networks, front-tier and back-tier. Each container is placed on one or two networks. Once on those networks, they can access other services on that network in code just by using the name of the service. To learn more about networking check out the Networking with Compose documentation.

To launch your app navigate to the example-voting-app directory and run the following command:


$ docker-compose up -d


This tells Compose to start all the containers specified in the docker-compose.yml file. The -d tells it to run them in daemon mode, in the background.

Last you'll need to figure out the ip address of your Docker host. If you're running Linux, it's just localhost, or 127.0.0.1. If you're using Docker Machine on Mac or Windows, you'll need to run:

$ docker-machine ip default It'll return an IP address. If you only have one Docker Machine running, most likely, that's 192.168.99.100. We'll call that . Navigate to http://:5000 in your browser, and you'll see the voting app, something like this:

References



This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Tuesday, June 28, 2016

Escaping Ruby Strings

The Rules

1. If possible, use single quotes (')

2. If you have a single line that needs interpolation, without inline double quotes, do the following:

"#{name} says, 'Hey!'"

3. If you have multiple lines use %[ <MUILTI_LINE_INTERPOLATION> ]


Note that %[ ] handles interpolation.

Example


name = 'Bob'

msg = %[#{name} says, "Hey!
What's up?"]

>> puts msg
Bob says: "Hey!
What's up?"


References

Don't escape in Strings





This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Thursday, June 9, 2016

Today's Cloud Architecture

Automating deployments with Constant Integration (CI) and Constant Deployments (CD) have improved our Software Development Life Cycle (SDLC) dramatically.

Most corporations doing any software development have figured out that it is better to integrate changes early. It reduces risk and helps find issues earlier in the life cycle.

Here is a typical Cloud Architecture for software deployed to "The Cloud":
This was acceptable Cloud Architecture last year.

Features

  • Build Automation
  • Github workflow (pull requests & code review)
  • Constant Integration (to the "cloud")
  • Constant Deployment (via Bamboo)
  • Managed Releases (integrated with Jira tickets)

Q: What's wrong with that picture?


There are quite a few areas for process improvement requiring a different way to approach application development, testing, deployment and monitoring.

Answer: A lot

  • Heterogeneous Development Environment
  • Time Consuming Build Process
  • Time Consuming Deploy Process
  • Stateful Ansible/Chef/Puppet Packaging
  • Code Complexity
  • Extra Image Creation Time
  • Costly Server Startup Time
  • Manual or Roll-ur-own HA

There are a lot of inefficient, stateful, slow, not-easily-repeatable parts to the way many cloud deployments are done today.

Stateful Ansible/Chef/Puppet Packaging


Notice that output files of one software package are inputs to the other software packages, making our solutions complex and non-deterministic.

Questions For You to Ponder...

  1. How does your company manage it's Software Development Life Cycle (SLDC)?
  2. How long does it take from the time you get a feature request (or bug report) to a delivered solution?
  3. How well utilized are the CPUs of your computing resources?
  4. How resilient are your applications to failure?
  5. Can you handle rolling software deployments?
  6. How much complexity and technical debt is your company's IT infrastructure running on?

Disruptive Innovation

There are a lot of inefficient, stateful, slow, not-easily-repeatable parts to the way many cloud deployments are done today.

Arguably the most exciting area of improvement is the change to the way the developer's workstation is configured.

The same immutable server environment that has been hardened for production is leveraged for the developer (and used throughout all environments).

Declarative, portable, disposable and environment variables configured.

That's Disruptive Innovation.

References



This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Thursday, April 28, 2016

Search and Replace

Here's a quickie, but goodie...


sed-i -e "s/alice/bob/g" data.json


That will replace every instance of "alice" with "bob" in the file, data.json





This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Wednesday, April 13, 2016

Page Up & Down in VIM and ITerm2

Sometimes I just forget the easy stuff...

VIM

PAGE UPCTRL+b--OR--FN+UpArrow
PAGE DOWNCTRL+f--OR--FN+DnArrow

ITERM2

PAGE UPfn+Command+UpArrow
PAGE DOWNfn+Command+DnArrow

References



This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Wednesday, March 23, 2016

Docker Ecosystem

Docker is a containerization platform that is used to develop, distribute and deploy applications in a portable and predictable way.

It accomplishes this by packaging components and their dependencies into standardized, lightweight process environments called Docker containers.

Docker containerization is especially useful in distributed systems, where there is a need to scale automatically, based on configuration profiles and resource utilization.

After 3 years, a lot of software has been developed around this containerization platform. See diagram below.


Containers v. VMs

Containers are much lighter than VMs.

Whereas VMs require a separate, entire guest operating system for each VM, Docker containers only require a single Docker Engine per workstation and all that is packaged in the container are the applications and their direct library dependencies.

Development/Deployment Implications

Now, we're finally able to configure develop our microservices in a Docker container in our local development workstation.

We can tune our apps locally and have confidence that our deliverables that are deployed via our continuous integration and continuous deployment processes are consistent with our dev environment.

References


This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Thursday, February 18, 2016

Golang Pointers on the Heap

In this post we'll delve into the differences between the Stack and the Heap and decompile some Go code to see how the new function allocates space on the heap struct and returns its 8 byte pointer.

The Stack

The stack is the memory that is used to store local variables for a executing process's thread.

When a function is called, a block is reserved on the top of the stack for local variables and some bookkeeping data.

When that function returns, the block becomes unused and can be used the next time a function is called.

The stack is always reserved in a LIFO (last in first out) order; the most recently reserved block is always the next block to be freed.

This makes it really simple to keep track of the stack; freeing a block from the stack is nothing more than adjusting one pointer.

Note that in Golang, there is a stack per goroutine.

The Heap

The heap is storage space in RAM set aside for dynamic allocation.

Unlike the stack, there's no enforced pattern to the allocation and deallocation of blocks from the heap; you can allocate a block at any time and free it at any time.

This makes it much more complex to keep track of which parts of the heap are allocated or free at any given time; there are many custom heap allocators available to tune heap performance for different usage patterns.

Note that in Golang, the new keyword always allocates on the heap.

Escape Analysis

In some cases the compiler follows rigid rules (like "new always allocates on the heap") and in others the compiler does "escape analysis" to decide if an object can live on the stack or if it must be allocated on the heap.

Code Example

When we compile the code below...


package main

func main() {
}

type DemoStruct struct{}

func demoFunc1() (*DemoStruct, error) {
 var data *DemoStruct = new(DemoStruct)
 return data, nil
}

Generate Plan 9 Assembly Code


$ go build -gcflags "-S " heap.go


"".demoFunc1 t=1 size=96 value=0 args=0x18 locals=0x10
 0x0000 00000 (heap.go:8) TEXT "".demoFunc1+0(SB),$16-24
 0x0000 00000 (heap.go:8) MOVQ (TLS),CX
 0x0009 00009 (heap.go:8) CMPQ SP,16(CX)
 0x000d 00013 (heap.go:8) JHI ,22
 0x000f 00015 (heap.go:8) CALL ,runtime.morestack_noctxt(SB)
 0x0014 00020 (heap.go:8) JMP ,0
 0x0016 00022 (heap.go:8) SUBQ $16,SP
 0x001a 00026 (heap.go:8) FUNCDATA $0,gclocals·0528ab8f76149a707fd2f0025c2178a3+0(SB)
 0x001a 00026 (heap.go:8) FUNCDATA $1,gclocals·3280bececceccd33cb74587feedb1f9f+0(SB)
 0x001a 00026 (heap.go:8) MOVQ $0,"".~r1+32(FP)
 0x0023 00035 (heap.go:8) MOVQ $0,"".~r1+40(FP)
 0x002c 00044 (heap.go:9) MOVQ $type."".DemoStruct+0(SB),BX
 0x0033 00051 (heap.go:9) MOVQ BX,(SP)
 0x0037 00055 (heap.go:9) PCDATA $0,$0
 0x0037 00055 (heap.go:9) CALL ,runtime.newobject(SB)
 0x003c 00060 (heap.go:9) MOVQ 8(SP),BX
 0x0041 00065 (heap.go:9) NOP ,
 0x0041 00065 (heap.go:10) MOVQ BX,"".~r0+24(FP)
 0x0046 00070 (heap.go:10) MOVQ $0,"".~r1+32(FP)
 0x004f 00079 (heap.go:10) MOVQ $0,"".~r1+40(FP)
 0x0058 00088 (heap.go:10) ADDQ $16,SP
 0x005c 00092 (heap.go:10) RET ,

Note that Go linker inserts some assembly code to support segmented stacks

Analysis shows the pointer to the struct escaping; The compiler allocated memory for the struct of type DemoStruct on the heap and returns its address in the form of an 8 byte pointer.

Since this is an 8 byte pointer MOVQ 8(SP),BX, we know that we're running on a 64 bit operating system


$ uname -a
Darwin venom 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64

If we were running this in a 32 bit operating system, the pointer would get 4 bytes of storage space.

Escape Analysis of Variable Not Escaping

Because the numbers slice is only referenced inside Sum, the compiler will store the 100 integers for that slice on the stack, rather than the slower heap.

There is no need to garbage collect numbers, it is automatically freed when Sum returns.

This is just one example of how Go is intelligently designed for speed.


References



This work is licensed under the Creative Commons Attribution 3.0 Unported License.