Thursday, April 28, 2016

Search and Replace

Here's a quickie, but goodie...


sed-i -e "s/alice/bob/g" data.json


That will replace every instance of "alice" with "bob" in the file, data.json





This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Wednesday, April 13, 2016

Page Up & Down in VIM and ITerm2

Sometimes I just forget the easy stuff...

VIM

PAGE UPCTRL+b--OR--FN+UpArrow
PAGE DOWNCTRL+f--OR--FN+DnArrow

ITERM2

PAGE UPfn+Command+UpArrow
PAGE DOWNfn+Command+DnArrow

References



This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Wednesday, March 23, 2016

Docker Ecosystem

Docker is a containerization platform that is used to develop, distribute and deploy applications in a portable and predictable way.

It accomplishes this by packaging components and their dependencies into standardized, lightweight process environments called Docker containers.

Docker containerization is especially useful in distributed systems, where there is a need to scale automatically, based on configuration profiles and resource utilization.

After 3 years, a lot of software has been developed around this containerization platform. See diagram below.


Containers v. VMs

Containers are much lighter than VMs.

Whereas VMs require a separate, entire guest operating system for each VM, Docker containers only require a single Docker Engine per workstation and all that is packaged in the container are the applications and their direct library dependencies.

Development/Deployment Implications

Now, we're finally able to configure develop our microservices in a Docker container in our local development workstation.

We can tune our apps locally and have confidence that our deliverables that are deployed via our continuous integration and continuous deployment processes are consistent with our dev environment.

References


This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Thursday, February 18, 2016

Golang Pointers on the Heap

In this post we'll delve into the differences between the Stack and the Heap and decompile some Go code to see how the new function allocates space on the heap struct and returns its 8 byte pointer.

The Stack

The stack is the memory that is used to store local variables for a executing process's thread.

When a function is called, a block is reserved on the top of the stack for local variables and some bookkeeping data.

When that function returns, the block becomes unused and can be used the next time a function is called.

The stack is always reserved in a LIFO (last in first out) order; the most recently reserved block is always the next block to be freed.

This makes it really simple to keep track of the stack; freeing a block from the stack is nothing more than adjusting one pointer.

Note that in Golang, there is a stack per goroutine.

The Heap

The heap is storage space in RAM set aside for dynamic allocation.

Unlike the stack, there's no enforced pattern to the allocation and deallocation of blocks from the heap; you can allocate a block at any time and free it at any time.

This makes it much more complex to keep track of which parts of the heap are allocated or free at any given time; there are many custom heap allocators available to tune heap performance for different usage patterns.

Note that in Golang, the new keyword always allocates on the heap.

Escape Analysis

In some cases the compiler follows rigid rules (like "new always allocates on the heap") and in others the compiler does "escape analysis" to decide if an object can live on the stack or if it must be allocated on the heap.

Code Example

When we compile the code below...


package main

func main() {
}

type DemoStruct struct{}

func demoFunc1() (*DemoStruct, error) {
 var data *DemoStruct = new(DemoStruct)
 return data, nil
}

Generate Plan 9 Assembly Code


$ go build -gcflags "-S " heap.go


"".demoFunc1 t=1 size=96 value=0 args=0x18 locals=0x10
 0x0000 00000 (heap.go:8) TEXT "".demoFunc1+0(SB),$16-24
 0x0000 00000 (heap.go:8) MOVQ (TLS),CX
 0x0009 00009 (heap.go:8) CMPQ SP,16(CX)
 0x000d 00013 (heap.go:8) JHI ,22
 0x000f 00015 (heap.go:8) CALL ,runtime.morestack_noctxt(SB)
 0x0014 00020 (heap.go:8) JMP ,0
 0x0016 00022 (heap.go:8) SUBQ $16,SP
 0x001a 00026 (heap.go:8) FUNCDATA $0,gclocals·0528ab8f76149a707fd2f0025c2178a3+0(SB)
 0x001a 00026 (heap.go:8) FUNCDATA $1,gclocals·3280bececceccd33cb74587feedb1f9f+0(SB)
 0x001a 00026 (heap.go:8) MOVQ $0,"".~r1+32(FP)
 0x0023 00035 (heap.go:8) MOVQ $0,"".~r1+40(FP)
 0x002c 00044 (heap.go:9) MOVQ $type."".DemoStruct+0(SB),BX
 0x0033 00051 (heap.go:9) MOVQ BX,(SP)
 0x0037 00055 (heap.go:9) PCDATA $0,$0
 0x0037 00055 (heap.go:9) CALL ,runtime.newobject(SB)
 0x003c 00060 (heap.go:9) MOVQ 8(SP),BX
 0x0041 00065 (heap.go:9) NOP ,
 0x0041 00065 (heap.go:10) MOVQ BX,"".~r0+24(FP)
 0x0046 00070 (heap.go:10) MOVQ $0,"".~r1+32(FP)
 0x004f 00079 (heap.go:10) MOVQ $0,"".~r1+40(FP)
 0x0058 00088 (heap.go:10) ADDQ $16,SP
 0x005c 00092 (heap.go:10) RET ,

Note that Go linker inserts some assembly code to support segmented stacks

Analysis shows the pointer to the struct escaping; The compiler allocated memory for the struct of type DemoStruct on the heap and returns its address in the form of an 8 byte pointer.

Since this is an 8 byte pointer MOVQ 8(SP),BX, we know that we're running on a 64 bit operating system


$ uname -a
Darwin venom 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64

If we were running this in a 32 bit operating system, the pointer would get 4 bytes of storage space.

Escape Analysis of Variable Not Escaping

Because the numbers slice is only referenced inside Sum, the compiler will store the 100 integers for that slice on the stack, rather than the slower heap.

There is no need to garbage collect numbers, it is automatically freed when Sum returns.

This is just one example of how Go is intelligently designed for speed.


References



This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Friday, October 30, 2015

Java Applets Don't Run In Chrome Browser


Have you noticed that some web pages have previously dynamic sections that no longer work in Chrome?

Have you seen a message stating, "Chrome no longer supports NPAPI"?

That's because the Chrome Browser no longer supports java applets... and for good reason: Running java applets in your Chrome browser is a security risk.

Options for the Die Hards

If you are determined to run that web page with that non-functional java applet, regardless of it's security implications, you have options:

Run Java Applets By

  • Using a browser that still supports NPAPI (MS IE, Safari, Firefox)
  • Use the IE Tab plugin for Chrome (for Windows platform)
  • Convert Java Applet to a Web Start application (if you can influence development)

NPAPI

Google's Chrome version 45 (scheduled for release in September 2015) drops support for NPAPI, impacting plugins for Silverlight, Java, Facebook Video and other similar NPAPI based plugins.

Netscape Plugin Application Programming Interface (NPAPI) is an application programming interface (API) that allow plug-ins (more specifically, browser extensions) to be developed for web browsers.

It was first developed for Netscape browsers, starting in 1995 with Netscape Navigator 2.0, but was subsequently adopted by other browsers.

In NPAPI architecture, a plugin declares content types (e.g. "audio/mp3") it can handle. When the browser encounters a content type it cannot handle natively, it loads the appropriate plugin, sets aside space within the browser context for the plugin to render and then streams data to it. The plugin is responsible for rendering the data. The plugin runs in-place within the page, as opposed to older browsers that had to launch an external application to handle unknown content types.

NPAPI requires each plugin to implement and expose approximately 15 functions for initializing, creating, destroying and positioning plugin content. NPAPI also supports scripting, printing, full-screen plugins, windowless plugins and content streaming.

Full privileges are only granted by default to chrome scripts.

Mozilla

Mozilla is deprecating all plugins.

"Plugins are now a legacy technology. They are not available on most mobile devices. Mozilla encourages website developers to avoid using plugins wherever possible. If there are plugin features which are not available in the web platform, we encourage developers to post their use cases to mozilla.dev.platform project list, so that Mozilla can prioritize web platform work to make those use cases possible."

Note that plugins are shared libraries that users can install to display content that the application itself can't display natively. For example, the Adobe Reader plugin lets the user open PDF files directly inside the browser, and the QuickTime and RealPlayer plugins are used to play special format videos in a web page.

Furthermore...


If you developed a java applet for a web page and deployed it to production, you might want to keep fact that off your resume.

Running java in a web browser was never a good idea.

The java applet is executed within a bloated Java Virtual Machine (JVM) in a process separate from the web browser itself. The java plugin was designed to run the java applets in a "secure sandbox" in the browser. This would supposedly prevent any java applet from presenting security risks to your computer.

The reality is that there have been so many vulnerabilities that allow nefarious Java applet code to escape the sandbox and exploit your system that Oracle has basically given up.

Java will no longer run unsigned applets, unless you go to the trouble of reducing your browser's default security settings. Running unsigned applets shouldn’t be a problem if the security sandbox were trustworthy in the first place. Right?

Furthermore, the graphics generated from Java apps, IMHO, never were crisp and/or visually appealing.

Cisco’s 2014 annual security report claims that 91 percent of all web attacks in 2013 targeted Java.

Running java applets in a browser will be insecure, slow, have high resource requirements and look sub-par; So, don't do it.

References



This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Wednesday, October 28, 2015

The Hand-Crafted SQL Anti Pattern

You probably should have used an ORM if...

  • your app has a lot of SQL statements that exceed 200 lines
  • you use multiple CTE (WITH Clauses) to deal with the complexity of your looong SQL statements
  • you frequently implement helper functions that return SQL clauses
  • you frequently refactor your helper functions that return SQL clauses
  • you find it takes more time to add functionality to your SQL queries than it took to design the database DDL, indexes and write that super long SQL in the first place

Those are a few indications of deeper issues.

Before you discount the "exceed 200 lines" or "multiple CTE" items, please consider that the reason for having sql helper clauses is typically to reduce the complexity of the sql statement; If the sql statement needs simplification then odds are it's too darn long; Hence, the "exceeds 200 lines" and "multiple CTE" remarks.

Another rule of thumb of mine... If any logic (any language) exceeds 150 lines long, it's probably too long and should probably be refactored.

Technical Debt

With a lot of hard coded SQL, you'll likely encounter problems down the road like...

SQL Errors you might get with Pagination


In this case, we were using a WITH clause and wanted to enforce a new sorting order...


ERROR:  SELECT DISTINCT ON expressions must match initial ORDER BY expressions
LINE 52: select distinct on (s.id)
                             ^
********** Error **********

ERROR: SELECT DISTINCT ON expressions must match initial ORDER BY expressions
SQL state: 42P10
Character: 1213

We learn by ERROR that when we try to add the flexibility of changing the sort order, we have new, unforeseen problems to solve which are a direct result of our legacy, hard-coded SQL.

Reasons to use a lot of hard-coded SQL, instead of an ORM...

  • Your business objects are not easy to map to database tables
  • Your application needs lot of complex, hand-tuned SQL
  • You use a lot of stored procedures and complex logic in your database

If this is your case, you should probably revisit your database design and Data Manipulation Language (DML) implementation strategies and actively reduce your technical debt before you are asked to maintain all that hard-coded SQL.

Intelligent enterprise application design begins with information architecture and a solid database design.

If you frequently feel that it takes a lot longer to add functionality to your application than it should, you should closely examine your application architecture.

Disclaimer

Q: Am I saying that you should always use an ORM to solve your all of your DML needs?

A: No.

However, I am saying that if the vast majority of your DML code is hard-coded SQL, you should probably find a good ORM that fits in your technology stack and learn how to use it.

References

None. That's just my hard-knock experience speaking.

This work is licensed under the Creative Commons Attribution 3.0 Unported License.

Sunday, October 25, 2015

Handling Errors in Go


Using Golang best practices for handling errors, we handle the error ASAP if it's not nil.

This avoids nested blocks of logic and is generally a good idea in any language.

Frequently, we want to return an error message in the form of a formatted string.

First Try

This technique requires both fmt and error packages to be imported.


   err = db.Select(&customer, "SELECT * FROM customer WHERE id = ?", 999)
   if err != nil {
      err = errors.New(fmt.Sprintf("Unable to process customer: %v [Error: %+v]", id, err))
      return
   }

Better - Use fmt.Errorf

  • Fewer imports (only requires fmt)
  • Fewer function calls
  • Less code


   err = db.Select(&customer, "SELECT * FROM customer WHERE id = ?", 999)
   if err != nil {
      err = fmt.Errorf("Unable to process customer: %v [Error: %+v]", id, err)
      return
   }

Generally speaking...

  • the fewer packages you include, the less there is to break now and in the future
  • the fewer function calls you make, the faster your code is going to run
  • the less code you have, the less code you have to review, compile and maintain

Okay, But Can Do Even Better?

Granted, we've simplified our code, but if we use the New function from the errors package to create package level, exported defined error variables we can return those to a caller that can compare them.

Define Errors

Define errors in customer package:


ErrCustomerNotFound := errors.New("customer: id not found")
ErrTimeout := errors.New("customer: timeout")

Compare Error


response, err := processCustomer()
if err != nil {
    switch err {
    case customer.ErrCustomerNotFound:
        // Handle customer not found error.
        return
    case customer.ErrTimeout:
        // Handle timeout error.
        return
    default:
        // General error handler
        return
    }
}

Return Error Interface Type

It is idiomatic in Go to use the error interface type as the return type for any error that is going to be returned from a function or method.

This interface is used by all the functions and methods in the standard library that return errors.

For example, here is the declaration for the Get method from the http package:


http://golang.org/pkg/net/http/#Client.Get

func (c *Client) Get(url string) (resp *Response, err error)


Since we have defined our err return argument in our function definition, we can use a simple return command and our err value will be returned through the error interface type definition.

Even Better

Go's design encourages us to explicitly check for errors where they occur; Contrast this with other languages that encourage throwing exceptions and sometimes catching them.

However, this can lead to a lot of error handling code.

How can we reduce our code and add features, like adding an error code, that can be leveraged to support I18N requirements?

If we have one service handler which is the only place that handles processCustomer calls, we can return a service-specific error object.


type svcError struct {
    Code    int
    Error   error
}


The code can be used to lookup the appropriate, localized message and the original error can be passed along to the handler.


func processCustomer(req *svc.Request, resp svc.Response) *svcError {    
    if custID, err := req.Get("id"); err != nil {
        return &svcError{CustomerIdNotFound, err}
    }
    return nil
}


It helps to define error code constants.

const CustomerIdNotFound = 10001

Improvements


  • Allow svcError to take a slice of parameters that can be used to construct a more complicated error message
  • Include a StackTrace
  • Handle panics in svcError and report a more user friendly message to the consumer




References


This work is licensed under the Creative Commons Attribution 3.0 Unported License.