Tagol, the part I and a half

Genesis

In the first part of our saga, we wrote a simple AWS Lambda, called through API Gateway, and deployed using Terraform.

I explained that the saga were incremental steps to a bigger, mysterious project.

In order to test the first part, I wrote a small a type that would wrap the response object in order to make assertions on it. It felt handy, but there still was a fair share of duplicated code in the tests.

As the next steps of the saga will imply more Lambdas, it seemed to be a good idea to have a library that allows easy testing of a API Gateway proxified Lambda.

Design

A fluent interface

I built the library around a fluent interface (a.k.a method chaining) : calling a method will return the value (object) on which it was called, to allow further calls.

This pattern is what led me to tell that the library is a small Domain Specific Language. In his Domain Specific Languages book, Martin Fowler cites method chaining and expression building as two of the patterns uses by an internal DSL.

The library can therefore be used like this :

// Act
Invoking(lambdaHandler).
    WithPathParameter("id", "a110").
    WithQuery("area", "gt42").
    WithHeader("Warning", "High speed cars ahead").
    WithBody(testingType{Name: "A", Area: 32}).

    // Assert
    Should(tester).
    ReturnNoError().
    ReturnHeader("Gopher-Name", "Goophy").
    ReturnValidatedBodyAs(new(testingType), func(_ interface{}) error { return nil }).
    ReturnStatus(http.StatusOK)

Albeit it's not a complicated library and not a fully-fledged DSL, it's more expressive in that way than it would be in a procedural way.

Working against Go ?

Having this multiple lines method chaining with the . at the end of the line strikes me as kinda ugly.

I'm used to C# or PHP where you can put the . operator anywhere you like :

// Pseudo C# code
object
    .Invoking(object => object.Method())
    .Should()
    .NotThrow()

Maybe it's just an habit, but having the . at the beginning of the line feels more readable.

Go does not requires commas at the end of the lines because the parser inserts them. So if the statement looks like finished, it will insert the comma and your next .Something() on the following line will be a syntax error.

This is why the fluent interface doesn't look like fitting into Go's idioms. I guess the idiomatic way is too use a 1-char variable name and do :

i := tagol.Invoking(fut)
i.WithBody(...)
i.WithQuery(...)
i.WithHeader(...)

I still prefer my fluent way 🙂.

How small

I wanted to keep the library small.

For example, ReturnBodyAs checks for deep and strict equality between the expected and actual response body (as JSON). This means that you're testing against well known, constant values.

This might not always suit : what if the logic handled by your Lambda depends on :

• Actual time
• Random value
• External data sources (that you can't or won't mock in your test for some reason)

I thought about implementing a more complex DSL where you could assert things such as :

• A field of the actual value must be greater than
• A field must be equal to one of these enumerated values
• ...

But it seemed complicated and long to implement.

In the end, the easy way was to provide a ReturnValidatedBodyAs that would take a function argument :

• The function argument would receive the unmarshal-ed value as a can-be-anything interface{} argument.
• It can cast it to the expected type using Go's type assertion (v, ok := i.(Type))
• Then it can validate any property of the value
• It returns an error if something's wrong.

The great news is that using the popular validator library :

• You can validate the returned value using tag-written rules :
  • In Go, struct fields can have tags that you can discover through the reflection package and use programmatically
  • These tags are less-powerful equivalents to annotations or attributes in other languages
  • You cannot apply a tag on methods or functions, only on fields (but the field can be of func type ?)
• The usage is quite easy :
  • Add validation tags to your struct
  • Pass the validate.Struct method in place of the function argument of ReturnValidatedBodyAs
• You don't need to write your custom validator with a bunch of if's
• Struct tags are also acting as self-documentation.

Implementation

Embedding structs

In the beginning, I used Go's struct embedding. It's handy because all fields and methods of the embedded structs are directly accessible (when you have a value of your type). So it was simpler to write the library, with less indirections (. operators). But it turns out that users of the library would be able to access those fields, too.

And it's not that I would not trust, nor care about a potential user's use of those fields. It's because, with IDE code completion, having more fields at clutters the completion popup with items I don't care about, since I only want to use my library's method set.

Pointers vs. values

I started using simple values, and not pointers, for passing the data around.

But, as pointed out by the Go wiki :

If the receiver is a large struct or array, a pointer receiver is more efficient. 

How large is large? 

Assume it's equivalent to passing all its elements as arguments to the method. 
If that feels too large, it's also too large for the receiver.

The receiver is Go's equivalent for this or self : it's like an argument, but changes the method set of its type. Thus declaring a receiver allows to create object-oriented methods.

A function signature like func (t Type) Something(s string) allows you to write val.Something(somedata) when val is of type Type.

If you use a struct type value (and not a pointer to that value), Go will make a copy of your value every time you call a method on that value. It's great because it allows you some degree of functional programming : if you want no side effect (calling does not update your object), declare the method on a value type, and pass a value.

But there's a runtime cost involved. To measure that cost Assume it's equivalent to passing all its elements as arguments to the method..

I'm usually not creating functions with more than 3 arguments, so if you have a struct with more than 2-3 fields, pass it as a pointer. Watch out for types with non-pointers fields or embedded structs : each of their fields will count as an argument, too.

But I did'nt pass pointers. I wanted to see first if it would have an impact on performance such big that I would need to use them. On the library's test suite or in the first usage on the part I project, it's quick enough.

Maybe in a bigger test suite it would start having an impact ?

In normal usage, the library decides for itself :

• The types returned by Invoking and Should :
  • You can't create valid values of type Invocation and Assertions yourself (the inner fields are private and thus non-writable from outside the package)
  • Consider Invoking and Should as constructors or factory functions, and Invocation and Assertions as private types
• The types of the method receivers on Invocation and Assertions

You should not make any assumption on Invoking's return type : just know that you can build the Lambda's calling context and response's assertions with it.

Thus it should be easy to change to pointers, even without using Go's interface, without breaking compatibility.

Panic

To keep Invoking and With both :

• Simple (no need for Go's t *testing.T in the act step of the test)
• Fluent (cannot return two arguments to be able to chain calls)

I had to panic (Go's almost-fatal error function) in case of JSON serialization error when describing the request body. In effective Go it says panicking should be avoided by libraries.

But I guess a test that provides abnormal inputs (that would trigger the JSON error) deserves panics. We're not in the assertion part of the test's code yet; we're not panicking because the assertions do not pass. It should not be surprising that one must pass a struct that can be serialized.

Deep equality and interfaces

This was a hard one.

The ReturnBodyAs method allows you to unmarshal the response body into an actual value, and then compare it for deep equality with a value you provide.

To implement this for any type, you must use Go's interface{} type as argument. This accepts ... any argument.

When implementing this, I effectively wanted to allow anything to be passed as arguments, allowing flexibility for the user. However, the JSON library uses a pointer to a value to unmarshal (allowing the library to know what's the type to populate from the JSON).

To achieve both flexible arguments and the pointer constraint on the JSON library, I wanted to use Go's built-in reflection. The goal was to convert anything that was not already a pointer to something, to a pointer to that thing.

Instead, reflects seems to only allow converting an interface{} (the static type of the argument) to a pointer to an interface{}. So you don't get a pointer to the interface's actual value. It makes sense, but didn't work well for me.

As I was trying to compare the actual and expected value, I was deep-comparing a pointer to an interface to a pointer to a struct. Go's reflect.DeepEqual considers that to be false.

Of course it makes sens for a statically typed language to behave this way. Not that I'm complaining, but if Go had generics (a long time requested feature by many), the function would just be type-parametrized and we'd be done with it.

Instead, I gave up and asked users to always provide two pointers as arguments. If not, there will be panicking or false negative test results.

Publishing the module

When the time came to publish the library, I read about Go's new module system. I tried to publish my module, and failed with a mysterious 410 Gone on go get commands, because the Gitlab repository I used was still private.

Once fixed, it was quite enjoyable :

• No need for registration or configuration anywhere, the system just takes your source from you VCS platform
• Automatic registration of the module in Go's registry when the first program that depends upon your module is built
• Nice Go doc and module presentation

Conclusion

I now have an enjoyable library to test my Lambdas. When I'll take the time to pursue my tale, hopefully this will be valuable.