Day 20 - A go http/2 server example

I was really excited when i read that support for http2 is now on golang.org! I was thinking about checking out SPDY in go for a long time, but now HTTP/2 support moved one important step further.

I think its one of the "Next big Things" on the interweb (the last change to http was 18! years ago)

So i skip SPDY and go directly for HTTP/2 - this post is about how to get a first grip and test it serverside as a go html server example.

Test preparations clientside:

1. Get a browser that has http/2 support enabled by default: Chrome Canary
2. Download the http2 and spdy indicator addon for chrome 
3. Test your browsers http2 support at: https://http2.akamai.com/demo

 Serverside::

1. Cd into your GOPATH/src
2. Run go get golang.org/x/net/http2
3. Check if the package has been downloaded

Create, build and run:

1. Create a new go project, name it eg. testHttp2Server
2. Create a ssl certificate (localhost.cert and localhost.key)
   You can ether generate a self signed cert: selfsignedcertificate
   or use mine: testHttp2Server
   This two files must be in the same directory as the server binary.
3. Code for http2Server.go:
   
   
   package main

   import (

       "fmt"

       "golang.org/x/net/http2"

       "html"

       "log"

       "net/http"

   )

   func main() {

       var srv http.Server

       http2.VerboseLogs = true

       srv.Addr = ":8080"

       // This enables http2 support

       http2.ConfigureServer(&srv, nil)

       // Plain text test handler

       // Open https://localhost:8080/randomtest

       // in your Chrome Canary browser

       http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {

           fmt.Fprintf(w, "Hi tester %q\n", html.EscapeString(r.URL.Path))

           ShowRequestInfoHandler(w, r)

       })

       // Listen as https ssl server

       // NOTE: WITHOUT SSL IT WONT WORK!!

       // To self generate a test ssl cert/key you could go to

       // http://www.selfsignedcertificate.com/

       // or read the openssl manual

       log.Fatal(srv.ListenAndServeTLS("localhost.cert", "localhost.key"))

   }

   func ShowRequestInfoHandler(w http.ResponseWriter, r *http.Request) {

       w.Header().Set("Content-Type", "text/plain")

       fmt.Fprintf(w, "Method: %s\n", r.Method)

       fmt.Fprintf(w, "Protocol: %s\n", r.Proto)

       fmt.Fprintf(w, "Host: %s\n", r.Host)

       fmt.Fprintf(w, "RemoteAddr: %s\n", r.RemoteAddr)

       fmt.Fprintf(w, "RequestURI: %q\n", r.RequestURI)

       fmt.Fprintf(w, "URL: %#v\n", r.URL)

       fmt.Fprintf(w, "Body.ContentLength: %d (-1 means unknown)\n", r.ContentLength)

       fmt.Fprintf(w, "Close: %v (relevant for HTTP/1 only)\n", r.Close)

       fmt.Fprintf(w, "TLS: %#v\n", r.TLS)

       fmt.Fprintf(w, "\nHeaders:\n")

       r.Header.Write(w)

   }

   
   

4. Compile:  go build http2Server.go
5. Run the http2Server binary

Test it:

1. Open in Chrome Canary: https://localhost:8080/kim
2. Accept the certificate security error, continue to localhost
3. Check if Protocol is HTTP/2.0, example output should be like:
   
   

   Hello tester "/kim"
   Method: GET
   Protocol: HTTP/2.0
   Host: localhost:8080

   ... 

NOTE: http/2 in go ONLY WORKS WITH SSL/TLS! (currently)
I not sure why, but i did not got the server to support http/2 without TLS!
This could be a clientside or serverside issue - i dont know currently.

Some discussions about this:
why-wouldnt-it-be-great-if-http-2-would-only-allow-communication-via-tls
daniel.haxx.se tls-in-http2
arstechnica http2-finished-coming-to-browsers-within-weeks

As far as i have understood it, the spec says it "should be" possible to use it without the need to buy SSL certificates:
https://http2.github.io/http2-spec/
rfc7540 - html2

Day 19 - Upgrading to Go 1.5.1 on Win7 64

I upgraded to Go 1.5.1 from 1.4.2 today

0:) Deinstall old version of Go (doc/install)

1:) Download https://storage.googleapis.com/golang/go1.5.1.windows-amd64.msi

2:) Get worried by this message from chrome:

It says that there is something fishy with the download and could be harmful.

3:) Go on and accept it - its from google, so what ...

4:) Go to the download folder and run your virus scanner over the msi:

5:) UhOh - WTF is an Archive Bomb?

6:) Google for Archive Bomb, read Wiki about Zip_bomb

7:) Looks like its detected because of special compression used by bzip2, i feel brave today and go on installing the msi.

8.) Checking again the c:\go folder for viruses - all ok

9:)
D:\>go version
go version go1.5.1 windows/amd64 

Yeah, it works

10:) Recompiling and running some of my most important projects - no problems whatsoever.

Day 17 - Using reflection to write into an interface slice

How to use reflection to write into a slice of unknown items inside an unknown struct:

While working on Gorp with Indexes i had to solve the problem of how to write into a slice inside a struct passed as an interface{}. A new struct with the filled slice should be returned. Both the struct and the slice are not known at compile time, i only have the fieldname of the slice passed to me at runtime. After some serious headscratching and reading through the reflection code i luckily found a way to do it.


The basic method is:

1. Convert the incoming interface i to a reflect.Type t
2. Dereference until we have a t which is a Struct
3. Create a new reflect.Value v from the Type t
4. From v we now can ask for the FieldName of the slice, getting back another reflect.Value s
5. As we now have the slice s as a reflect.Value, we need to get the slice elements type
6. From the slice elements type we create a new instance of it (reflect.New)
7. Write to fields in this newItem using FieldByName (Hardcoded in the example)
8. Append the newItem to the slice (the set append was the hard part to find out for me)
9. Return the reflect.Value v as an interface
10. ???
11. Profit - Heureka!!

Output:
Input Type main.Post:

Slice Type []*main.Comment:

Slice Elem Type main.Comment:

Comment 0, Body XYZ 0, PostId 0

Comment 1, Body XYZ 1, PostId 2

Comment 2, Body XYZ 2, PostId 4

Comment 3, Body XYZ 3, PostId 6

Comment 4, Body XYZ 4, PostId 8

Erfolg: Prozess beendet mit Rückgabewert 0.

// This is a demo to show how to convert from a normal struct

// to a reflection type and back to a struct without knowing

// the original one. Input is passed as an Interface and the  

// output will be an interface, too.

// Bonus points for writing into an embedded slice

// (= the embedded Comment struct slice in Post)

package main

import (

    "errors"

    "fmt"

    "os"

    "reflect"

)

type Post struct {

    Id       uint64

    Title    string

    Comments []*Comment

}

// holds a single comment bound to a post

type Comment struct {

    Id     uint64

    PostId uint64

    Body   string

}

func CreateAndFillSlice(i interface{}, sliceName string) (interface{}, error) {

    // Convert the interface i to a reflect.Type t 

    t := reflect.TypeOf(i)

    // Check if the input is a pointer and dereference it if yes

    if t.Kind() == reflect.Ptr {

        t = t.Elem()

    }

    // Check if the input is a struct

    if t.Kind() != reflect.Struct {

        return nil, errors.New("Input param is not a struct")

    }

    fmt.Printf("Input Type %v:\n", t)

    // Create a new Value from the input type

    // this will be returned to the caller

    v := reflect.New(t).Elem()

    // Get the field named "sliceName" from the input struct, which should be a slice

    s := v.FieldByName(sliceName)

    if s.Kind() == reflect.Slice {

        st := s.Type()

        fmt.Printf("Slice Type %s:\n", st)

        // Get the type of a single slice element

        sliceType := st.Elem()

        // Pointer?

        if sliceType.Kind() == reflect.Ptr {

            // Then dereference it

            sliceType = sliceType.Elem()

        }

        fmt.Printf("Slice Elem Type %v:\n", sliceType)

        for i := 0; i < 5; i++ {

            // Create a new slice element

            newitem := reflect.New(sliceType)

            // Set some field in it

            newitem.Elem().FieldByName("Body").SetString(fmt.Sprintf("XYZ %d", i))

            newitem.Elem().FieldByName("PostId").SetUint(uint64(i * 2))

            // This is the important part here - append and set

            // Append the newitem to the slice in "v" which will be the output

            s.Set(reflect.Append(s, newitem))

        }

    } else {

        return nil, fmt.Errorf("Field %s is not a slice\n", sliceName)

    }

    // IMPORTANT

    // Cast back to the empty interface type

    // So the cast back to Post outside will work

    return v.Interface(), nil

}

func main() {

    var err error

    p := Post{Id: 1, Title: "Title 1"}

    result, err := CreateAndFillSlice(p, "Comments")

    if err != nil {

        fmt.Println(err.Error())

        os.Exit(1)

    }

    // Cast the returned interface to a Post

    post := result.(Post)

    for i, c := range post.Comments {

        fmt.Printf("Comment %d, Body %s, PostId %d\n", i, c.Body, c.PostId)

    }

}

Day 16 - Comparison of html parsers for a webcrawler, today GoQuery

github.com/goquery is the king of go html parsing. After trying the two other
relevant libs i can conclude:

• GoQuery: All you want, you can use it as a Ferrari or as a heavy load truck

• Scrape: Small, light and neat: Its your bicycle, it always works

• go-html-transform: For me it feels like an energy plant with all buttons in "русский этикетки"

This example to scrape posts from hackernews has been stripped down to
fit onto one page. All error handling and debug printouts have been
removed, so only the pure GoQuery logic remains.
The full source is available at github.com/hackernewsCrawlerGoQuery.

// Parse for posts in html from hackernews, input html is an io.Reader 

// and returns recognized posts in a psout slice of posts.

// Errors which affect only a single post are stored in their post.Err

func ParseHtmlHackerNews(body io.Reader, ps []*post.Post)  

                        (psout []*post.Post, err error) { 

    // Create a qoquery document to parse from an io.Reader 

    doc, err := goquery.NewDocumentFromReader(body)

    // Find hackernews posts = elements with class "athing"

    thing := doc.Find(".athing")

    for iThing := range thing.Nodes {

        // Create a new post struct - if the crawling fails 

        // the post will have its Err set, but will be added  

        // to the outgoing (psout) slice nevertheless

        post := post.NewPost()

        ps = append(ps, &post)

        // use singlearticle as a selection of one single post

        singlearticle := thing.Eq(iThing)

        // Get the next element containing additional info for this post

        scorenode := singlearticle.Next()

        // Get the post title

        htmlpost := singlearticle.Find(".title a").First()

        post.Title = htmlpost.Text()

        // Get the post url

        post.Url, exists = htmlpost.Attr("href")

        // Get the post score

        scoretag := scorenode.Find(".subtext .score").First()

        post.SetScore(strings.Split(scoretag.Text(), " ")[0])

        // Get the post id

        postid, exists := scoretag.Attr("id")

        post.PostId = strings.Split(postid, "_")[1]

        // Get the username and postdate

        hrefs := scorenode.Find(".subtext a")

        for i := range hrefs.Nodes {

            href := hrefs.Eq(i)

            t, _ := href.Html()

            s, exists := href.Attr("href")

            if exists {

                if strings.HasPrefix(s, "user?id") {

                    post.User = t

                    continue

                }

                if strings.HasPrefix(s, "item?id") {

                    if strings.Contains(t, "ago") {

                        var postDate time.Time

                        postDate, err = GetDateFromCreatedAgo(t)

                        post.PostDate = postDate

                        post.Err = nil

                    }

                }

            }

        }

    }

    return ps, err

}