Dr. Michiel Overeem

The Art of Computer Programming - part 2 - MMIX and fundamental programming techniques

2017-03-18T00:00:00+00:00

I set out to read The Art of Computer Programming this year. In these blog posts I will report on the progress. Mostly for me, but maybe someone finds it fun to read :)

sections 1.3’ (Volume 1 Fascicle 1) - MMIX

The section on MMIX, the language that Knuth uses throughout the book for his code samples. Although it seems a strange choice, a machine language for samples, I can appreciate the reasons. It is not that hard, and it shows a lot of how the machine works.

Nice from a theoretical perspective, but to be honest, I skipped/skimmed these parts. Although I typed some programs, I didn’t put to much effort in this part.

sections 1.4’ (Volume 1 Fascicle 1) and 1.4.4 and 1.4.5 (Volume 1 Third Edition) - Fundamental programming techniques

These sections are a bit more general, and I really liked the ideas in it.

In 1.4.1’ Knuth explains the steps for writing a program with subroutines as follows:

Start with the problem: what wil be the plan.
Write a top-down version of the solution. First the general flow, while putting details in subroutines (that are not yet implemented). Then the subroutines are filled in. In this step, any language can be used. The interesting idea is that this version is an exploration, and the result is not used except for knowledge discovery.
After that initial exploration, a second version can be written. This time in a real programming language, but bottom-up. Write the subroutines, and then the main routine that combines them.
The final step is the reexamination of the solution. Depending on the state, return to step 1 or 2 and refine.
After the reexamination, it is time to finalize the version by running and inspecting it. Or debugging as Knuth says. This is necessary to observe the behaviour at run-time.

Step 1 reminded my of Alistair Cockburn’s view: design as knowledge acquisition. Choosing something else than a programming language, or maybe a programming language other than your normal language frees you from the feeling that you are writing production code. It gives opportunity for knowledge acquisition without the burden of quality.

Step 3 is of course non-trivial: when to stop, when is enough enough. This is something that one can only learn through experience, I guess.

Step 4 might be something that we often forget to do in a controlled fashion. A run, not just to validate the functionality and correctness, but also a run to validate the performance and other aspects. Maybe automated tests can improve this: a controlled fashion that is part of source control and can be ran on every build. This prevents errors that slip with later changes.

Secion 1.4.2’ was also interesting, coroutines versus multiple passes. It reminded me of generators, iterators, reactive programming, etc. Much to think about! I might write another blog about that in the future.

The Art of Computer Programming - part 1

2017-01-05T00:00:00+00:00

Goal for 2017: read through this massive work. #knuth #art #computerprogramming #taocp pic.twitter.com/DGh5XnVg8X
— Michiel Overeem (@michielovereem) December 20, 2016

I have these books on my shelve since graduation (2007), and a couple of weeks ago I finally set the goal to actually read them. It is a challenge for myself to see if I can read all volumes, comprehend them (more or less), and get something useful out them. So in this serie of blogs I will summarize, share notes, and tell the world about my progress. I have not set a deadline or a schedule, I am just going with the flow. And in the spirit of The Pragmatic Programmer I will use this challenge as an opportunity to learn some F#. Code snippets and what more can be found in the github repo.

These books are impressive. And to be honest a little intimidating. I like reading and learning, but I’m not sure if I can follow all this stuff. But we will see.

sections 1.1 and 1.2 (Volume 1 Third Edition)

Section 1.1 is labeled Algorithms, and this first section was fun. It contained history (which I like), a little background, and some general explanation. An example of the background: the origin of the word algorithm. I sure didn’t know that its origins are in the name of the Persian author al-Khwārizmī. An algorithm is a receipe, procedure, a routine. But how to know when something is an algorithm? Knuth gives the following list:

Finiteness. An algorithm should terminate in a finite number of steps. A finite number of steps could mean a large number of steps, but it should stay reasonable.
Definiteness. Every step should be precisely defined. The definition should have rigour and no ambiguity. This is where formally defined languages like programming languages are useful.
Input. Algorithms have input: zero or more inputs.
Output. And they have one more outputs.
Effectiveness. Effective here is defined as using basic operations such that they can be done exactly and in finite time.

What is a good algorithm? We have to look at elegance, the number of steps, simplicity, and more.

The book gives an example by lookinmg at Euclid’s algorithm: find the greatest common divisor of two integers.

Section 1.2 is Mathematical Preliminaries and for this section I followed the advise of Knuth: skimming these sections and familiarizing myself with the math. I did not do any exercises. To be honest, I am more interested in the computer programming parts and less in the math part, so I will regard this as reference material. The subsections of 1.2 covered among other things numbers, powers, logarithms, sums, products, permutations, factorials, fibonacci, the analysis of algorithms, and asymptotic representations.

As an example of the analysis of an algorithm, the algorithm for finding the maximum is given.

The last subsection was most interesting as it covered the big-O notation. And it made my think about another book in my digital shelve: The Imposter’s Handbook. In this book Rob Conery dives into different computer science topics for those who never studied computer science. And to be honest, his explanations also make sense for the ones that did study computer science (yep, that’s me). In this case I read Knuth’s subsection and after that I read what Rob Conery said about it. Knuth is pure and comprehensive with all the math, while Rob gives a rather short overview, but they reinforce each other and I am good to go :)

bookreview - Learning JavaScript Design Patterns

2014-02-25T00:00:00+00:00

Learning JavaScript Design Patterns by Addy Osmani

JavaScript is everywhere. We cannot ignore it. And I even think it is foolish to ignore it. Like Scott Hanselman, I think everyone should learn JavaScript. The web is here to stay, and we should embrace it. And if you are doing web, client side stuff, you cannot ignore JavaScript. The frontend of many applications is much larger then it was a few years ago. You could build a web application, and add a little JavaScript for the effect. Nowadays, there is a lot more to be done.

And that is exactly why this book is important. Because JavaScript was seen as a script language, nobody thought about patterns, modules, organization, etc, etc. But if you have a few hundred, or a few thousand lines of JavaScript, you will need organization. And then you will need design patterns. Most software developers will be familiar with patterns. There are some classic books that will tell you everything. But those books are mostly aimed at Java, or C#. Altough JavaScript is object-oriented, its lack of classes and prototypical inheritance asks for a different description of patterns.

This book will give you everything to get started. It is split into two parts. The first part describes what patterns are, how patterns are found and described, and how to write your own patterns. I think that it will be usefull for experienced developers and larger teams that want to write up their own patterns. The second part is more of a reference. You can find many patterns and various ways to implement them. It is something that I will use to look up details of patterns. But it will also give you inspiration to solve you problems in different ways.

I would recommend this book to every one that is serious about programming JavaScript. Let us look at the language as professional developers, and force ourself to develop maintainable code.

Find the book at http://shop.oreilly.com/product/0636920025832.do

Disclaimer: I have joined the O’Reilly Blogger Review Program (http://oreilly.com/bloggers/). It means that I receive free books that I review. I am not forced to write nice things, so this review is an honest one.

Certified Rx Developer

2014-02-15T00:00:00+00:00

My current company gives every employee a training budget. This budget can be spend on every training that you like, there is no need to ask permission up front (but how you spend it is part of the evaluation). So when Erik Meijer asked on twitter if people wanted to follow a certified Rx developer training, I had a training to spend my money on.

I had already taken the coursera course Principles of Reactive Programming (which is highly recommended!), but the Rx stuff had not really sunk in yet. That is why I started a new project. But still, the opportunity to learn from Erik, the one that started the whole Rx movement (years ago)? I had to be there :-)

This blog post is a combination of a summary of Rx and my experience following the certified Rx developer training. Let me start with the last, my experience. Erik is a great teacher. Not only is he super smart, he is also funny, entertaining and a nice person to chat with. But more, he is a real hacker, not some scientist that cannot write code. He honestly ended our lunch to start hacking on some idea (with goto’s for crying out loud)!

So, what is Rx anyway? I could point you to this great introduction, or to the code. But it might be fun to see if I can explain some things. I like the path that Erik took, which started not with Rx, but with monads. Those things always sound awful. In the Haskell community, there is a quote saying that they should have called those things warm fuzzy things, because that sounds so much more fun. But monad it is. But before we get to monads, let us first look at exceptions.

Exceptions

When programming C# and JavaScript (my primarly languages), exceptions are used to abort the execution when something happends that you did not expect. Which exceptions a method might throw is not visible from the outside. The method signature does not tell you about those cases (in Java you have checked exceptions, not in C#). So when we have a method like this, the signature is lying to us!

#c#
private int DoSomething(int number)
{
  if(number < 0) {
    throw new Exception("No negative numbers allowed.");
  }

  return number * number;
}

It tells us that this method takes a integer as argument, and returns an integer. There is however, another possible outcome: the exception. So the effect of failing is hidden from us. To make this more explicit, we introduce the Try type. Using this will give us the following method (implementing the Try type is left as an exercise to the reader).

#c#
private Try&lt;int&gt; DoSomething(int number)
{
  if(number < 0) {
    return Try { Failure = new Exception("No negative numbers allowed.") };
  }
 
  return Try { Success = number * number };
}

This signature shows us that the method has two possible outputs: an integer, or an error. We have made the possibility of failure explicit.

The four essential effects in programming

The Try type is essential for programming, because we use it for synchronous, single values. If we move from single values to many values we need to use another type, the Iterable type. In C#, this is the Enumerable type. But this type is still synchronous. This means that asking values from an Iterable blocks the execution till all values are given. You are pulling values from the iterable and that blocks the execution. When you want to develop a scalable, responsive system, blocking is not done: asynchronous is the way to go.

If we move from a synchronous single value to an asynchronous single value, we need yet another type, the Future type. This type makes it explicit that we have latency. It might take a while for a value to be available. In C#, this is done using the Task type, which corresponds with Futures. In JavaScript we have the Future/Promise objects.

The last step is from a asynchronous single value to multiple values, asychronously. That is where the Observable type comes into play. And that, that is the heart of Rx. To summarise these four effects:

	Single	Many
Synchronous	Try<T>	Iterable<T>
Asynchronous	Future<T>l;	Observable<T>

But what about monads? And why is Rx so special? Untill now I only explained four effects and the corresponding types. Let’s move on to monads now.

Monads

So what is a monad? I do not pretend to understand it better then others, so I will not try to write a formal (or informal for that matter) explanation. Search google if you want the real stuff, or read this one! For now, all you need to grasp is that a monad is a datastructure that comes with a set of functions, of which I will give you three examples in C#:

#c#
public static Something&lt;U&gt; FlatMap(this Something&lt;T&gt; source, Func&lt;T, Something&lt;U&gt;&gt; func)

public static Something&lt;U&gt; Map(this Something&lt;T&gt; source, Func&lt;T, U&gt; func)   

public static Something&lt;U&gt; Filter(this Something&lt;T&gt; source, Func&lt;T, bool&gt; func)   

Now, these types are very general. But you could substitute the generic Something type with a couple of familiar types. For instance with Try. Or with Future, Iterable. You might even substite it with Observable.

A monad is a general data structure that gives you a set of common operations (others are zip and flatten). From them, you can build lots of stuff. You can filter, combine and project your data. And that is the power of monads. They give you a basic set of functions that work on all four effects: synchronous, asynchronous, single or many.

By the way, the names FlatMap, Map and Filter seem new. But just put them next to the LINQ functions SelectMany, Select and Where. They do look familiar now, don’t they?

Rx

Rx thus fills in the last quadrant, the missing piece: many asynchronous values. Now that I write it down, it all seems very simple and nothing new in general. But the Rx library gives you an implementation of the Observable type, with all the monad functions implemented. It allows you to turn any stream of data (from a database, from a webservice, from an input device, from an user) into an Observable that you can then filter, combine and project. And all of this is asynchronous by default. You are forced to think about latency, which is a good thing if you want your app to stay responsive.

In the training, Erik went on the give some example use cases. Projecting earthquakes on a map for instance. Or updating the average earthquake strength per region while earthquake data comes in. The guys at github used Rx to build a reactive UI. Netflix uses Rx. Square (could not find any source but Erik told us) and SoundCloud as well.

One thing you have to keep in mind, is that Rx is not about concurrency, or parallelism. It is about asynchronous computations. However, you could use it for concurrency and parallelism, using the schedulers that they implemented. But that is not the key point.

Take aways

So what are the take aways from this training and Rx in general? Let me offer a small list of things that I can think of:

Programming is fun, seeing Erik enjoy coding and talking about it was such an inspiration.
Rx is not hard, it makes stuff easier because you can use the same toolset that you use with synchronous or single value sources.
Asynchronousity and latency are effects that we need to make explicit.
Futures and promises are also not that hard. They are cleaner then callbacks as a pattern.

Automating JavaScript testing in a Microsoft environment

2014-02-08T00:00:00+00:00

Writing software with JavaScript can be a lot like writing software in the dark. There is no static typing and no compiler, most errors are not found untill the code is executed. Hopefully that is in a test environment, and not when a user is working with your software. I think that a lot of emphasis should be on automated testing and analysing the code. At my current project, we have a rather nice setup in place to do this and guard our quality.

The thing that caused a bit more work is that we are working in a Microsoft environment. So we have Visual Studio (2013) and TFS (2013). The rest of our project consists of C# and we like to work in Visual Studio. Any tooling that we add is preferred to integrate with our current tools. We already have gated checkins and continous integration builds for the .NET code. The CI build runs everything that is in the gated checkin build plus more (and slower) tests.

Code analysis

We started without any code analysis on our JavaScript. We had to do lots of testing to find bugs before checking in/ releasing/ deploying. And we started to get more and more different coding patterns and styles. We wanted to get rid of this time, therefore we integrated ESLint into our development process. There are more linters for JavaScript available. ESLint however was started with the promise to enable people to write their own rules, specific for their architecture. Nicholas Zakas being one of our inspirations while evolving our architecture, the decision for ESLint was made rather quick.

Running ESLint on your code is one thing, integrating and forcing it to pass on every checkin is another. What we did is building a C# Unit test that executes ESLint on our files. It fails when there is an error or warning, and passes when everything is fine. This test is run during the gated checkin, so we know that the code in our source control system is following our rules.

We started with some debt, as we needed to cleanup. What we did was enabling all rules that passed, and worked from there to meet our goals (a set of rules and limits that we decided on). This allowed us to move gradually to a more stricter rule set without slipping and adding more stuff that we needed to cleanup one day.

Rules that really help us to keep a clean code base are things like unused variables, naming conventions, scope issues, using strict mode and lots more. Things that we find less important are order (using vars that are assigned further on the file) and defining all vars on top of the function. But it sure helps us to keep things nice and clean.

Unit tests

From Leonard's Comic - Unit Testing

A test suite was also something on our todo list. We started out with QUnit and a static html file to run the tests. QUnit is a simple test framework, with few concepts. That’s why we like it so much, it is dead simple. There are many more frameworks, so go find the one that suits you.

The first approach, using one static html file for all QUnit tests, became soon unmanageable. And we wanted to execute those tests in our build. So we looked at some test runners for Visual Studio and TFS. We already use Resharper for Visual Studio and they have some support for running JavaScript tests. But is far from complete, for instance it misses basic support for RequireJS. We also found Chutzpah, an open source project with much more features and support. With Chutzpah, we could get rid of our html files and execute the tests from the build. Chutzpah can run the tests in PhantomJS or in the browser. Running them in the browser gives us much better support for debugging and solving problems.

In previous versions, Chutzpah also didn’t really have support for RequireJS, the module loader that we use. We implemented a mock of RequireJS instead, that enabled us to run our tests and make sure that we didn’t need to modify our code. It was far from beautiful. We had a double administration of dependencies (both in RequireJS and in QUnit tests). But hurray, the latest version has support for RequireJS. We deleted our mock and now we can really run our tests the way we want.

There is one thing that made our setup a little more difficult. We like to make separate Visual Studio Projects for code and test-code. With C# tests, this is no problem. For JavaScript, it is somewhat harder. We have our code in code-library projects, no web projects. We release and deploy through nuget packages, that’s way we do not need web projects. Our QUnit tests are in a separate code-library project under the folder tests (our code is under src). To make sure that RequireJS can find the files, we have a special RequireJS config for the QUnit tests, in which we use the path config option to point to the right source folders.

Integration tests

Another thing that we wanted to test was the JavaScript code that we generate. Our current architecture consists of generated code and framework code combined. But we wanted to make sure that certain generated code patterns were tested. Therefore we added C# tests that calls our generator and executes JavaScript tests on that code, all from PhantomJS. A hacky way of testing, but it works rather well in our case. The generated code is linted with ESLint (using a different set of rules) and then loaded in PhantomJS. We insert the test code and execute them. The result is a passing or failing test.

The upside of this is that we automatically check a number of code generation patterns and know for sure that they work. The downside is that those tests are hard to debug. When we need to attach a debugger, we start PhantomJS with the debug option and connect to the right port. It works, but is far from beautifull. Lucky for us that we do not need it that much (most failing tests are solvable without attaching a debugger).

Code coverage

The last step that we took was enabling code coverage. The latest and greatest version of Chutzpah also has support for code coverage, through Blanket.js. It instruments every code file and returns a report of lines that were executed.

Now of course we know the downside of code coverage and that it can be misused. But the one thing that is useful, is to show you that certain code is never executed under tests. It gives some direction and level of confidence in our code. We see that it tells us something about our test level, not using code coverage would mean that we know nothing about this level except some gut feeling.

First we enabled it for our QUnit tests. That was easy. Although you need to know that the Chutzpah config file is being cached, so changing it will not always directly work. See this bug. But we had some tests failing because of timeouts. That was odd. Then we saw that it were tests that not exactly tested a unit, they loaded most of the framework. And with the framework, most of the third party libraries (with both JQuery and Handlebars). Blanket.js does not really handle those files well. So we needed to exclude those files, not that we wanted coverage on them anyway. But we hit a minor missing feature in Chutzpah, excluding files for coverage does not play well with RequireJS. We made a local fix for our situation, but I am working on a pull request to get this fixed (and filled an issue).

After that we enabled coverage in our integration tests. We altered the C# code to insert Blanket.js in the generated code, and wrote some extra code to export the coverage results to disk. By then we had tens of coverage files, all telling us something about the code coverage from their point of view. A little C# console app fixed the last hurdle: combining all that information into a single coverage report, telling us how much code we covered.

Conclusion

And now we have a nightly build. It runs all of our tests with code coverage enabled. It gives us two reports, a coverage file for the C# code, and html file for the JavaScript code. And all was well :-)

One thing is missing: the html file is in the drop folder. But we want it to be attached to the build report like the coverage file. If someone knows how to do this, I would love to hear!

Any suggestions? I love to hear them!

Testing if the eventloop is blocked

2014-01-22T00:00:00+00:00

In the previous post, we talked about some core concepts of node.js, one of them being the event loop. In the first version of the GrapNode code, we did some asynchronous phantomjs stuff. That looked good, but I wanted to know for sure that the eventloop is not blocked during the 1.5 second that it takes to make a screenshot.

So I added a couple of other requests handlers to my code. See how we can do this with rx? We can assign the stream of requests to a variable, filter them and subscribe to the filtered stream:

#js
var server = new RxServer();
var requests = server.requests;

requests
  .filter(function (data) { return data.request.url === "/"; })
  .subscribe(function(data) {
    /* The root, returns a simple HTML page */
  });

requests
  .filter(function (data) { return data.request.url === "/capture"; })
  .subscribe(function(data) {
    /* Here we will perform our screencapture stuff */
  });

requests
  .filter(function (data) { return data.request.url.indexOf("/assets") === 0; })
  .subscribe(function(data) {
    /* Simple handler for static content */
  });

requests
  .filter(function (data) { return data.request.url === "/time"; })
  .subscribe(function(data) {
    /* Returns the server time */
  });

server.listen(8080);

By writing four different filters and four different subscribe handlers, we have created a basic routing mechanism.

Requests for the root URL will get back a basic HTML page that include a (clientside) JavaScript file with the following code:

#js
(function () {
  var requestTime = function () {
    var req = new XMLHttpRequest();
    req.open("GET", "/time");
    req.onreadystatechange = function (oEvent) {
      if (req.readyState === 4) {
        if (req.status === 200) {
          document.getElementsByTagName("span")[0].innerText = req.responseText;
        } else {
          document.getElementsByTagName("span")[0].innerText = req.statusText;
        }
        requestTime();
      }
    };
    req.send(null);
  };
  requestTime();
})();

Basically, this fetches the time of the server and shows it in an element. This fetching is done continuously, so when we block the event loop, the time will fall behind and start to lag. Along with this time polling, I will start the ab executable to do some concurrent requests for the screenshot (/capture).

As one can see in the video, the timer keeps running and is not blocked by the concurrent requests for the screencapture. This basically shows that our screencapture does not block the event loop.

There is another problem: the phantomjs bridge spawns new processes and those are limited. So we need to build a request queue and allow people to poll for their screencapture (or use socket.io to apply some realtime web).

$ ab -n 10 -c 1 http://127.0.01:8080/capture
  Time taken for tests:   15.673 seconds

$ ab -n 10 -c 2 http://127.0.01:8080/capture
  Time taken for tests:   9.555 seconds

$ ab -n 10 -c 5 http://127.0.01:8080/capture
  Time taken for tests:   7.379 seconds

The results from ab show fine numbers, I guess. I am not sure yet how to read this, but hopefully, that will come later on.

Some node concepts explained

2014-01-19T00:00:00+00:00

The project GrapNode is my attempt to get some experience with node.js and understand the key concepts and benefits. This short post will allow me to touch on these a little. By no means is this to be seen as the best explanation of these concepts. These are just my thoughts, and summaries of other explanations.

Modules

Node.js uses a simple module system, files are modules and modules are files. A module can be loaded through the function require. In the first post this function was used to require the http module:

#js
var http = require('http');

This call will look for the module (or file) http and load it. Where will it look? The documentation explains that it will do a couple of things:

Is there a core module with this name (which there is in the case of http).
If the argument starts with ‘/’, it is an absolute path and the file is searched for on disc.
If the argument starts with ‘./’, the path is relative to the current folder.
If the argument has no leading ‘./’ or ‘/’, node.js will look in the node_modules folder. This is the folder that is created by npm when a dependency is added.

This last step has some really smart rules which allow you to create a hierarchy of folders and dependencies. I suggest that you read the docs for more info!

The event loop

The code that is written in node.js, runs single threaded. You do not need any lock or synchronize constructs to prevent concurrency problems. Better yet, there are no locks! However, the runtime itself offers asynchronousity. When we do a http request, or a database call, or some IO, the node.js runtime will execute that task on another thread. You cannot interact with that thread, but you are allowed to give a callback function, that is executed when the work is done. How does this work in a single threaded runtime? When will that callback be executed? The event loop takes care of that.

Basically, the event loop is an ever running while loop that executes code. When a callback should be executed, the runtime puts it in a queue (the following code snippets are copied from this stackoverflow answer):

#js
lock (queue) {
  queue.push(event);
}

The event loop itself will take this queue and execute the functions on every tick:

#js
while (true) {
  // this is the beginning of a tick

  lock (queue) {
    // copy the current queue items into thread-local memory
    var tickEvents = copy(queue); 
    // ..and empty out the shared queue
    queue.empty(); 
  }

  for (var i = 0; i < tickEvents.length; i++) {
    InvokeJSFunction(tickEvents[i]);
  }

  // this the end of the tick
}

So on every thick, the queue is emptied and executed. All callbacks triggered while executing those functions are put in the queue, and executed on the next tick.

Scheduling stuff with setTimeout or setInterval actually means that the given callback is queued after the given timeout or interval, not executed. That is why you should never block the event loop with cpu intensive or long running tasks! This will prevent the event loop of executing queued callbacks and thus firing events.

Presenting GrapNode

2014-01-18T00:00:00+00:00

I’m like a couple of years late to the party, but finally started a node.js project. The plan is to log my progress on the blog, by writing about different aspects and maybe do some explaining of the core concepts (mostly for me, but maybe some others see benefit from it). The project will combine a couple of things. First of all node.js. I love JavaScript, and hope to learn more about using it on the server. Along with node.js, the project will use RxJS, the reactive programming library for JavaScript. After following the Principles of Reactive Programming on coursera a project to get some real life experience was needed! And of course, the project might catch up on other cool stuff as well.

In this series, I will try to take (baby)-steps in developing a system to take screenshots from given urls. PhantomJS will be used to take those screenshots, as I am in no way looking to build a browser environment myself. The website will allow you to enter an url, and maybe a screen size. The result will be a png of the website at the given url. The code will be online of course, at github.

Getting started

We start with npm, the Node Package Manager. The packages that are needed are rx and node-phantom. Installing is easy:

npm install rx
npm install node-phantom

After that we will want to create a package.json with:

npm init

This will ask you lots of questions, but will give you a package.json that can be used to restore dependencies and tell lots of things about your application. I am not sure what the other uses are yet, but I hope to find out later on. So now we got some dependencies and an application definition.

Write some code

Let’s get a node.js server up and running. We create an index.js file and put the following lines of code in it:

#js
require('http').createServer(function handleRequest(req, res) {
  res.writeHead(200, { 'content-type': 'text/plain'});
  res.end('Hello World!');
}).listen(8080);

This will give us a nice “Hello World!”, but in the plain node.js way. We want to rx-ify it. For now we will install a new dependency, rx-httpserver:

npm install rx-http-server --save

This will allow us to change the code to

#js
var RxServer = require("rx-http-server");

var server = new RxServer();

server.requests.subscribe(function(data) {
    data.response.writeHead(200, {"Content-Type": "text/plain"});
    data.response.end("Hello from Rx!");
});

server.listen(8080);

Much nicer, right? I might change that dependency later on to a custom written library, so that I can learn more about the internals.

That’s all for now. Lets hook up PhantomJS the next time. Feedback is welcome at anytime, just send a message through twitter or google+!

JavaScript architecture - some lessons learned

2013-02-22T00:00:00+00:00

The secret to building large apps is NEVER build large apps. Break up your applications into small pieces. Then, assemble those testable, bite-sized pieces into your big application. Brian Moschel

In the past year, I’ve been busy with researching architecture in JavaScript. And there are a few things that I’ve found and learned during this year. Now, I am nowhere near being an expert, I’m just someone that loves the language and the platform it offers. I happen to like the fact that it opens up so much possibilities, because JavaScript is everywhere!

I would like to list some principles and observations that follow from my learning.

Create little parts and not one big application

Now, JavaScript is a language that misses some features when you compare it with more mature and better designed language. There are no modules or namespaces, and there is only function-scope. When you do not look for a solution for this missing, you end up with spaghetti code. One of the things I’ve learned is that a module framework is one of the most important parts of your application.

I myself are only in depth familiar with RequireJS, but there are more. A module framework forces you to break up your app in little parts that have their own scope and clearly export what others can use.

#js
define("moduleNam", [/* list of dependencies*/], 
  function ( /* dependencies */ ) {
  
    /* place for private code */
  
    return {
      /* your contract with the outside */
    }
  }
);

You see, a module has clear definition of what it needs and what it delivers. This way, you are forced to let go of easy global variables.

Create abstractions for changing things that you cannot control

One of the really hard points that I discovered is working with a quickly changing environment. The third party libraries are developing fast. And it makes it hard to plan for the future. Of course no one wants to right code that we do not need. But last year, the jQuery UI library was widely used. Nowadays, it is Twitter Bootstrap everywhere. How do you deal with this? Do you choose and never look back? Or do you try to abstract it? Or do you write your own code?

Unfortunately there are no easy answers here. I found it to be possible to abstract some of these choices. But your situation might be different. But it is important to choose widely what you depend on. The JavaScript world is moving fast. Depending on something that is hard to change might be a mistake.

Be careful with accessing the DOM

The DOM is what your environment looks like in JavaScript running in the browser. You will need it. But you do not want it to be everywhere. JavaScript code is not compiled. Unit tests are more important then ever. Nobody likes script errors that make a web application become useless. So testing is required. But using the DOM everywhere makes your code untestable. Writing code that is modular and testable is a must!

There are more lessons and more paths to follow. But these three items are key to success in my opinion.

Side project - drawing signatures in HTML5

2013-02-03T00:00:00+00:00

This weekend I worked on another little side project. This time it consisted of a little HTML5 and JavaScript to use the power of a small tablet (iPad mini in this case). And it amazes me again how easy it is to put some open source projects together and get something working. Of course the proof-of-concept is no big deal, and it is not production ready. But all these little machines that we have give us so many possibilities.

The requirements are easy: enter information about work done at a customer and get a signature of someone. This will then by e-mailed to both the company and the customer, forming the basis of the bill that needs to be payed at the end. The tablets where chosen easily: iPads where already available. The code will hopefully be build in such way that we can easily expand support to other platforms, but for now we will stick with iPads. They come with a modern browser and support most of the HTML5 features. Important is the touch of course, because we will use that to get a signature.

The input of information is easy: put some input elements in the body and write some PHP script to turn it into an e-mail. We need a signature. It will be the official approval of the customer, and we do not want to put it on paper. HTML5 comes with the canvas element. That allows us to get some drawing input. But this is a side project and I do not want to write my own drawing code if it is not necessary. So I turned to Google. There I found Signature Pad, and let me tell you, using it is easy. With some configuration I managed to get a good size canvas displayed on the iPad. The PHP script transforms it to an image and with swiftmailer we build a beautiful HTML mail and send it off.

Follow progress on github!

JavaScript and the flexibility that is sometimes needed

2013-01-16T00:00:00+00:00

Integration of third part libraries. The promises that a library makes often force you to delete your own code right away and start integrating. Finally, ripping out code and replacing it with a tool, framework or library that is developed by a team instead of you. Developed over a couple of months, maybe even years. And all is good.

But is it? Because using a third party library comes at a price. What if integration is not that easy. Or what if some parts are not that stable. What if the demands from your customers are a bit different. Of course, as good developers we start with analyzing, prototyping, proofing that the library does what it needs. And of course, every manager, or stakeholder, approves and gives you the green light. Until a new requirement is found, that was not known at that time. Or maybe just not told to you. And now what to do?

There are a lot of discussions about JavaScript. About it being a language that is bad designed. About the risks in large enterprises. Microsoft started to design a better version. But there is great power in this language. At least, that is my experience. You can shoot yourself in the foot. Sure, you can also hit yourself with a hammer. In a perfect world, we might want to have static analysis that finds all the errors before we release stuff in the world. But that is not going to happen.

If you want to program for the web, there is only one option: JavaScript. And sure, you can use CoffeeScript, TypeScript or ClojureScript. But still, at one point you will hit JavaScript. I do not believe that we are already arrived at the point that JavaScript is only the assembly language underneath it all. I think that you still need to have some knowledge of the underlying core. Of course you can use tools that compile into JavaScript. But at least learn some, I think you will need it when debugging.

Back to integrating third party libraries. One thing that JavaScript stands out in, is the flexibility. The fact that you can change any object after creation makes it super easy to alter behavior of libraries. And you can mimic any object, without a type-system that will not allow it.

Sometimes it feels wrong to do this kind of stuff. And my computer science background screams for type-systems, proves of correctness and a high level of maintainability. But the maintainability of an application does not need to suffer from JavaScripts flexibility. You will need discipline. Maybe some unit tests. Even a little documentation.

But most important, we as developers need to get the job done and deliver what our customers want. Because in the end, all they care about is working applications that do the job for them in the best possible way.

Best tech books in 2012

2012-12-27T00:00:00+00:00

The year 2012 is almost done, and while enjoying my holiday I thought it would be nice to take a look back and see what books I have read and which of them I still would recommend. On looking at the list of books that I have read in 2012 (yes, I do keep a list) I saw that it weren’t that much tech books. I read more then 45 books in 2012, but only seven of them I can call tech or tech-related books.

On the tech side, the book reading was a bit quiet this year. It was my first full year at AFAS Software as a Software Architect, but that did not really drive my into more book reading. I also became a dad for the second time, so reading time is heavily cut down :)

So on to the books that I did manage to read, and after that, maybe a little bit of a wish list.

Thomas Myer - Beginning PhoneGap

A book that I reviewed for the O’Reilly blogger program. I was mildly positive about the book. Pure tech books are becoming less relevant, because stuff is moving hard. Jeff Atwood said it better:

Do highly technical books tied to a specific technology have any reason to exist in an era of ubiquitous, high speed Internet access? I wonder. I think they’re increasingly irrelevant, and almost by definition out of date by the time they manage to hit bookshelves.

But I did like to book as an entry into PhoneGap. And PhoneGap still looks very cool. I should make it a goal for 2013, to finally get something into the App Store…. I should, really…

Clay Johnson - The information diet

This book was a gift from the O’Reilly blogger program and not really technical. But it did touch on something that I recognized in my life. There is so much information coming from Twitter, Blogs and other sources that it is becoming more important to actively decide on what to read and what to ignore. Important stuff!

Jordan Mechner - The making of Prince of Persia

This book read more like a biography then a tech book, but the topic of course being very much tech. I remember playing Prince Of Persia when I was a kid, so it was really fun to read about the creative process behind it. And being a programmer in current modern times, it is really useful to read about problems like memory management and CPU limitations.

Nicholas Zakas - Maintainable Javascript

My first six months at AFAS mostly consisted of working on JavaScript and the architecture of a JavaScript application. Nicholas Zakas was a big source of inspiration, so this book was a must have. It is a nice overview and summary of knowledge.

Marty Cagan - Inspired

This book was a little disappointing because it did not talk about inspiration, but more about the job of a Product Manager. It was nice read, but it was not what I was looking for…

Donald Norman - Living with complexity

Donald Norman is well known for his books on design. And this book is no exception. This one really goes into the problems around complexity and complicated software. There is in fact a difference between complex and complicated software. Something that I was not aware of.

Sander Hoogendoorn - Dit is agile

Last but not least is a dutch book about agile software development. At my previous job I was a scrum master and implemented agile with a colleague of mine. Agile stuff is still very interesting, but often it is used more like a religion then a useful way of developing software. Sander Hoogendoorn is very pragmatic in his thinking about agile, and therefore this book is highly recommended! A German translation is already finished, and an English version is planned. Visit the website for more information!

So far for the books that I have read. What is next? What will follow next year? … I hope to continue diving into the world of JavaScript, HTML5 and front-end architecture and design. Maybe some UX. And for books? The list keeps growing, but on it are

Which books should I add? Let me know on twitter!

Suggested books

Clojure Programming by Chas Emerick, Brian Carper, Christophe Grand via @borkdude

The rise of CSS Frameworks and some thoughts on how to use them

2012-10-06T00:00:00+00:00

The advance of HTML5, CSS3 and responsive design has given rise to a whole new type of (open source) projects: CSS frameworks. The first time I saw these kind of projects I found it rather strange. I am used to frameworks in terms of reusable libraries giving some functionality. Like ASP.NET MVC, helping you serve content over HTTP. CSS however consists of rules and style definitions, there is no code, there is no functionality. These definitions can be reused of course, but it often becomes a pain to maintain large CSS files. The language itself does not have constructs to deal with reuse in a way like programming languages.

My first initial reaction was to view these frameworks not as ordinary frameworks, but rather as bootstraps: a basic starting point for your CSS design. Much like CSS resets. Such a bootstrap file gives you a certain amount of basic rules to build your layout. This approach does comes with some warnings.

Bootstrapping your project

Let us look at the relation between CSS and HTML and then how a bootstrap file can go wrong. CSS stylesheets consists of style definitions. These style definitions are annotated with selectors. These selectors link the style to your HTML file. In this way you get a clean separation: the HTML tells you what you are presenting and the CSS tells you how to present it. But they do have a stricter relationship then simple a separation of concerns. This relationship can go two ways. The selectors used in CSS can use some basic constructs: HTML tag names, CSS classes, ID attributes and some special cases (like hover, when a cursor is over an element). These constructs can be combined to match the structure of your HTML. The CSS file can use your nesting of HTML tags to link style definitions, but then the relationship becomes very fragile: a change in nesting can mess up your style. The other way around is also possible: you can annotate the HTML with class names and id attributes, but then you can wound up with to much knowledge about your CSS file in your HTML.

The two ways to use a CSS bootstrap are to use only the style definitions and not the selectors. This gives you freedom in both your HTML and CSS selector structure. However, it is lots of work and to be honest: you would only use the knowledge of the CSS in the bootstrap file and not the file itself. Another approach is to use the bootstrap file as is and change your HTML structure to match the file.

In the last approach, you get the extra benefit that it is possible to update the bootstrap file to incorporate bug fixes. But the usage of class names from the CSS file does violate the principle of separation: HTML tells what, and CSS tells how. These class names do not carry semantics for your project. They carry knowledge about what the rule definition does (for instance how much columns wide an element is presented).

Extending CSS to gain reuse

This is where two popular languages that extend CSS come into play: LESS and SASS. These languages add extra features on top of CSS to make the language more flexible; they add constructs for re-usability. They both add variables and mixins to the language.

Variables allow you to reuse certain key values, like colors or sizes. This makes it easy to change the color palette in one place, instead of throughout your CSS rules. Mixins on the other hand allow you to reuse a set of style definitions inside other definitions. This allows you to reuse for instance a set of prefix specific definitions without writing them out every time.

These language allow you to distribute a CSS framework in terms of mixins and variables. That way you can set up your CSS file and HTML file in the cleanest way, but use the already defined style definitions and gain development speed.

Reflection

So far I’ve used two CSS frameworks: Skeleton and Zurb Foundation. I like the Zurb framework because of the easy responsiveness that it adds. My approach is to add the Zurb class names throughout my HTML. Yes, this is not really clean, but the projects sizes are small and the likeliness of switching a framework or updating to a newer version are small. I am not sure of a larger project would succeed with such an approach. In such a project I would switch to just using the knowledge and style definitions, but not the actual code.

I am not sure if the LESS/SASS approach is any better. In my opinion, CSS and HTML are in a very strict relation, enforced by the language capabilities. Positioning and sizing within style definitions are depended on context of parent elements, and often you cannot view the CSS separate from the HTML. No language extension or framework will fix that problem.

But without the CSS frameworks, LESS/SASS do add better maintainability to your stylesheets. There are multiple tools to integrate it with your IDE. LESS even comes with a client-side library that transforms it on the fly to CSS. Performance would keep me from going that way, but it adds speed to the development process.

Depending on the size of a project, I would choose to learn from the CSS frameworks and write my own code or integrate them into the projects like any third party library.

Fall 2012 update on projects

2012-10-04T00:00:00+00:00

I thought it would be nice to give an update on some of the side projects that I’m working on. Reflecting on those I see that I currently am focussed on design, CSS3, responsiveness and less on core programming.

The first side project is very simple and plain informative website. I’ve started using Zurb Foundation as a CSS bootstrap and worked with Jekyll as a templating engine. There is no server-side stuff in this website but a simple mail form. I could really focus on (responsive) design on this one and I think it worked out nice.

The second project is a larger website build using ASP.NET MVC. Originally it was a website build on Joomla but I got fed up with it, because it feld limitting. Any CMS will force you in setting up pages a certain way. And styling or plugins are also limited in a way. I wanted more flexibility, so choose to build something from scratch.

I started with the design of the frontpage, again using Zurb Foundation. After the frontpage I started to add dynamic pages to move the current content. As back-end storage it used a mix of plain text files: csv, xml and markdown files. This allows me to manage the content through FTP and also to easily backup the content. There will be an administrative part, but it can wait because I am the sole maintainer.

Another reason to rebuilt this one is that I want to build a Phonegap/Cordova app that builds on stuff in the website. By designing the website in a mobile-ready way, I hope that the app will be easier.

This project is on the edge of ready for release, but I need to get approval from certain people first.

The last project is not really started yet, but I am researching the design currently. Again a simple website, but with a larger backend. Multiple roles will need to login and do their stuff. This one is constrained by the hosting platform. Therefor it will be written in PHP, and of course HTML5 and CSS3. I haven’t written PHP in years, so I am curious on the current state. Hopefully some blog posts will come from that project.

The last project is of course this blog. I hope to get to writting some posts soon. There is enough to write about, but time is limited.

Sharing that what we have received

2012-06-22T00:00:00+00:00

Last week the new MacBooks arrived. And what a gorgeous new model was introduced, with a Retina Display. I myself was waiting for the new models because the old laptop needed to be replaced. The Retina version was a little to high priced for me, but I got a beautiful 13” instead.

While reviewing and researching a new laptop, I also read some books about poverty and how we have a responsibility in al of that. We (at least my readers from The Netherlands) are living in one of the richest countries in the world. We are in a financial crisis, but still we have plenty of everything. And so I made the decision (again) to share from that what I have received.

There are a lot of possibilities if you want to share. You could give money, you could give time, etc. But in the book The hole in our Gospel, Rich Stearns makes the case for using peoples abilities and giving them a platform to work on their own poverty. Yes, we are talking about microfinance here. So that is what I have done. I’ve gone over to kiva and lend some money to people that have far less than me.

Now, I normally blog about tech stuff and software development. The whole microfinance thing has got me thinking: why are we spending money on kickstarter stuff that are run by people that have access to far more resources than those in third world countries? Would it not be cool if we as tech people could help fellow tech people all over the world to use their talents and fight poverty?

Developing, previewing and debugging on mobile devices - how to make your blog beautiful across devices

2012-03-11T00:00:00+00:00

This week I spotted Adobe Shadow: a tool for developing, previewing and debugging websites on mobile devices. The tool consists of a server component, installed on your development machine, a Google Chrome plugin and client apps (both iOs and Android). But before I tell you my experience, watch this demo video:

Pretty amazing, don’t you think? So I installed it to preview my blog. The connection was not without problems, but I managed to get it working on my iPhone and iPad. And of course, there were some problems to fix. So let me give you some things that I did to get a nice blog for mobile devices.

Start with a responsive desing

According to Wikipedia, responsive desing is:

Responsive Web Design (RWD) essentially indicates that a web site is crafted to use W3C CSS3 media queries with fluid proportion-based grids, to adapt the layout to the viewing environment, and probably also flexible images. As a result, users across a broad range of devices and browsers will have access to a single source of content, laid out so as to be easy to read and navigate with a minimum of resizing, panning and scrolling.

So the key is to use CSS3 features to get one design that scales nicely on different screen sizes. Personally, I started with a free CSS framework: skeleton. But there are many more. Pick the one that matches your style best. And of course, adapt to get your own style. I’ve choosen to let my blog be basic black and white, because I did not yet find a nice design that fits my personal style. To be honest, I like the black and white simplicity, so I might keep it longer.

Use your media queries

These things are key, because it let you define custom CSS rules, matching different screen sizes. It is far better then browser sniffing, or other conditional things, because it focusses on features and device-types and not on brands or other vendor specific things. A concrete example is the scaling of images to keep them in the main content.

!css
@media only screen and (min-width: 480px) and (max-width: 767px) {
    .container img {
        max-width: 400px;
    }
}

My blog on iPhone and iPad:

Any more tips on CSS3, mobile web and responsive design? Let me know on twitter!

Building and distributing an iOS app with PhoneGap - part 1

2012-02-19T00:00:00+00:00

I’ve started working on an iOS app. Finally. The goals for this app are multiple. First of all, I wanted to work with PhoneGap, to see what the platform looked like. Second, I found a nice case for an app that I could build and put into production. I try to write up the experience and other stuff that I encounter. This first blog post marks some sources that I found to be useful.

The application will be build with PhoneGap and jQuery mobile. Not because I think that PhoneGap is the way to go, but because I want some experience about the edge cases.

After a day, I had something working and wanted it to deploy. Being new to iOS development, I found this blogpost useful in all the steps. A tweet let me to TestFlight, a useful service for distributing test versions to a group of people.

So the first one is small, but I hope to write some more about the experience.

Javascript sources, a summary

2012-02-18T20:00:00+00:00

I decided to summarize some JavaScript sources that I use, have used and want to share with others. This is by no means complete, but it might serve as a good starting point. Please, send my anything that I have missed :-)

Books

This is a short list of all the books that are available on JavaScript. I only wanted to list the books that I have read. I cannot recommend others.

The Good Parts , by Douglas Crockford
JavaScript: the definitive guide, by David Flanagan
Pro JavaScript techniques, by John Resig

People with weblogs, twitter accounts

Douglas Crockford

http://www.crockford.com/
Inventor of JSON and JSLint

Brendan Eich

http://brendaneich.com
http://twitter.com/BrendanEich
Inventor of JavaScript, currently CTO of Mozilla

Nicholas Zakas

http://www.nczonline.net/
http://www.twitter.com/slicknet/
Front-end tech lead for Yahoo!, author of Professional JavaScript for Web Developers (Wrox, 2012), Professional Ajax (Wrox, 2007), and High Performance JavaScript (O’Reilly, 2010)

John Resig

http://ejohn.org/
http://twitter.com/jeresig
Inventor of jQuery, worked at Mozilla, currently head of JavaScript development at Khan Academy

David Flanagan

Projects and libraries

There are a number of cool projects, build in JavaScript that I try to follow.

Cloud9 IDE: a new IDE for JavaScript build in JavaScript. A very cool project, showing what can be done with JavaScript.
Google docs: what Google has build into docs is awesome.
jQuery: one of the many cool DOM manipulation libraries.

bookreview - Beginning PhoneGap

2012-02-18T16:00:00+00:00

Beginning PhoneGap by Thomas Myer

Beginning PhoneGap aims to give the reader a push in the right direction when using PhoneGap. PhoneGap is a platform for writing mobile applications that can run on iPhones, Blackberry’s, Androids and more. By using HTML5 and JavaScript inside a native app users get the benefits of native apps, while developers get the benefits of code sharing and reusing. Of course by abstracting away the native platform you also loose something.

The author starts by giving a very practical tutorial explaning how to start with PhoneGap. Screenshots and hyperlinks explain every step from downloading to installing. After that the key API’s are explained: media, camera, storage and more. The author goes into great dept to show all details. Every chapter describes the key elements for iPhone, Android and BlackBerry development. The other platforms are also mentioned, but in less detail.

The book itself was good to read, and it showed that the author did everything he could to be as precise as possible. However, I am not sure if the content justifies a book. Some chapters (like chapter 7 “Compass”) feel more like a large blog post. The web is stuffed with blog posts about these techniques. Although it is nice to have everything combined, a book suffers from the speed of development. During writing PhoneGap released a new version, the author updated everything. But what if PhoneGap releases a new version next week? Or maybe tomorrow? To justify a book, I would expect eloboration about the technique and platform, not screenshots of applications using a compass. But that would also need another title, not Beginning PhoneGap.

Find the book at http://shop.oreilly.com/product/9781118156650.do

Moved to Github Pages

2012-02-08T00:00:00+00:00

Yep, that’s right, I have moved my blog. I thought that posterous was the place to be. And the integration with google docs worked nice. But I wasn’t satisfied. I missed some control, and posterous was growing into a social media platform instead of a blogging platform. I missed some features, like good syntax highlighting and markdown support.

So, why did I go with Github Pages? Well, first I thought let’s write this myself. But there are already so many blogging platforms, and to be honest: I wouldn’t know when to build this. Now I have something that I can easily port or host myself. The posts are easily maintained in markdown.

What is Github Pages and how do my posts look like?

Github Pages is a simple way to host static html files. There is no dynamic pages support. But the pages are run through Jekyll before being hosted. Jekyll converts markdown files and … to html files and copies everything else over. The files are pulled from a github repository. This means that everything is under version control, you can create branches and you can fork someone else’s website. Go ahead, fork my blog :-)

The moving itself was just a lot of work, but that is because I did this manually. I wanted all my posts to be converted to markdown, and not simply copy the html files. There are converters, but I didn’t try that out.

I really like the concept of generated files. I allows me to maintain the templating seperate from the actual content, but doesn’t lay any requirements on the hosting so moving again is easy. Downside is that hosting comments on your own blog is impossible. There are comment-services, but I decided to redirect feedback to google+ or twitter. That way reacting for me is easier.

The autoposting to my twitter account is done through if this then that. I hope that they will add google+ support in the near future.

Let the posts start flowing!

JavaScript and the runaway timer

2012-01-26T00:00:00+00:00

One of the things that you probably will run into when creating a JavaScript application is the runaway timer. The runaway timer? Yes, the runaway timer. Ever seen this message?

or this one?

That is the runaway timer telling you that something is running out of control. It is the thing that keeps your browser from crashing or not responding because of a long running script. JavaScript in browsers is executed in a single thread. But, any DOM manipulation or event handling is also done on this single thread (yes I’m ignoring web workers in this post). When your script runs for seconds, no other thing can be done thus making your browser unresponsive. Clicks on buttons or hyperlinks will not be executed, hovers or other effects will do nothing. You should be timing your JavaScript code and split everything that takes more than 100 ms. Why 100 ms? The different runaway timers have enforce limits of multiple seconds. Usability research however shows that humans find a User Interface responsive when it reacts within 100 ms.

I myself ran into problems with the runaway timer when implementing the Mandelbrot figure using JavaScript and the Canvas element. The basis of the mandelbrot figure is an algorithm that calculates a color for every pixel on the canvas. A nested for-loop that is, visiting every pixel and doing some calculation. With a 400 by 400 pixel canvas, I was already running into problems.

#js
var paint = function () {
    for (var screenX = startpoint_x; screenX &lt; endpoint_x ; screenX++) {
        for (var screenY = startpoint_y; screenY &lt; endpoint_y ; screenY++) {
            var xynumber = getNumber(screenX, screenY, center_x, center_y);
            var color = getColor(xynumber);
            setColor(color);
            canvascontext.fillRect(screenX + canvas_half_width, screenY + canvas_half_height, 1, 1);
        }
    }
};

The solution was to split this up in parts that run within the 100 ms limit (actually I choose 50 ms blocks to stay on the safe side).

#js
var paint = function () {
    paintPart(startpoint_x);
};

var paintPart = function(start_x) {
    var start = new Date();
    for (var screenX = start_x; screenX &lt; endpoint_x &amp;&amp; (new Date() - start) &lt; 50 ; screenX++) {
        for (var screenY = startpoint_y; screenY &lt; endpoint_y ; screenY++) {
            var xynumber = number.getNumber(screenX, screenY, center_x, center_y);
            var color = number.getColor(xynumber);
            setColor(color);
            canvascontext.fillRect(screenX + canvas_half_width, screenY + canvas_half_height, 1, 1);
        }
    }    
    if(screenX &lt; endpoint_x) {
        setTimeout(function() { paintPart(screenX); }, 25);
    }
};

The effect of applying this pattern is that the browser stays responsive. Of course, the splitting also gives some overhead. But the site and the browser stay usable, and that is key in every good UI.

Want to more about creating responsive JavaScript applications? This presentation by Nicholas Zakas is a must see.

Inheritance and reuse in a prototype world

2012-01-12T00:00:00+00:00

One of the things that I struggled with writing JavaScript, is the prototype-based programming. Coming from a Java and C# background, it is hard to let go of classes, interfaces and normal inheritance. In a Class-based object-oriented language the classes define how objects behave. An object is an instance of a class. Classes can inherit from other classes and interfaces. In JavaScript (and other prototype-based languages like IO) there are no classes. There are objects, constructors and prototypes.

The prototype way in JavaScript is simply shown in the following example. Let us first define a constructor:

#js
function Vehicle() {
  this.speed = 10;
}

Now you shouldn’t think to much of this constructor. It is a normal function in JavaScript, but by convention we let it start with a capital letter. The important thing is that you use it in combination with the new keyword.

The next step is to define functions on the prototype object of this constructor:

#js
Vehicle.prototype.setMaxSpeed = function (speed) {
  this.speed = speed;
};

Vehicle.prototype.maxSpeed = function () {
  return this.speed;
};

When we construct a new object, we can use the constructor to create an instance:

#js
var myVehicle = new Vehicle();
var isVehicle = myVehicle instanceof Vehicle; 
// isVehicle is true

The constructor makes sure that the new object is based on the prototype. This makes all defined functions available in the prototype chain. There are two advantages of this approach: we can inherit from this object, and we can use the instanceof operator.

#js
function Car() {
}
Car.prototype = new Vehicle();

var myCar = new Car();
var isVehicle = myCar instanceof Vehicle;

Now we have an instance that is more specific then our Vehicle object, but can be used as a Vehicle none the less. As one might notice, the inheritance is achieved by setting the prototype of a constructor. But it is not set to another constructor, but to the instance of an object. The inheritance chain is thus build on instances and not on classes.

There are other approaches possible to share code between objects. One of them is object augmentation. Because we can alter the JavaScript objects at any moment, we can augment objects with functionality.

#js
function makeVehicle(o) {
    var speed = 10;
    o.maxSpeed = function() {
       return speed;
    };
    o.setMaxSpeed = function(s) {
       speed = s;
    };
    return o;
}

var myVehicle = makeVehicle({});

function makeCar(o){
    o = makeVehicle(o);
    // add extra functionality
    return o;
}

We cannot use instanceof anymore, but we can easily create objects with the same functionality. Of course there are some (subtle) difference with the prototype approach. If we decide that vehicles need more functions later on, we cannot alter all instances by changing the prototype. As a positive gain, we can use a Mixin pattern to get more flexibility.

The prototype approach can also be used without constructors and the new keyword. ECMAScript 5th Edition added a new function, Object.create. This function creates a new object using the given object as prototype.

Douglas Crockford has some nice articles (this and this) about inheritance and reuse in JavaScript. Because objects in JavaScript are soft and can be changed at any moment, he hardly sees use for the Classical Inheritance approach. Peter Michaux also writes about the augmentation approach, seeing the Decorator Pattern in it.

Another discussion raised by looking at the SOLID principles is the usage of inheritance. Most of the time, inheritance is used for method overloading and polymorphic methods. The Liskov Substitution Principle states that

Subtypes must be substitutable for their base types.

But JavaScript is not statically typed. This gives us the freedom to do more dynamic typing. Instead of looking at inheritance, we can look at behavior. By checking if certain functions are present we can be much more flexible in the range of objects that we can act on.

#js
function move(vehicle){
    if(!vehicle instanceof Vehicle)
        throw Error('not a vehicle');
    // ...
}

function moveDynamic(vehicle){
    if(!vehicle.move)
        throw Error('move function not present');
    // ...
}

The moveDynamic function shows that we can move any object that has a move function. The move function only works on instances of our constructor function. The last fits the dynamic model and gives more flexibility.

Things learned from the Game of Life

2011-12-24T00:00:00+00:00

I mentioned in the previous blog post that I am digging deeper into JavaScript lately. This post is the second in a series about JavaScript.

I love mathematical stuff. Things like fractals, but also Conway’s Game of Life are amazing. So I started implementing the Game of Life using JavaScript (github and a running example). While implementing, I used a number of patterns and techniques that are worth explaining. These patterns are useful because they let you work more effective with the language.

But let me first explain two differences between JavaScript and traditional object-oriented languages.

Scope

JavaScript only knows two scopes: the global scope and the function scope. This is different then for example C#. In C# block structures also start a new scope. So when you use a conditional or a loop structure, you know that you enter a new scope. This gives advantages in reuse variable names, but it also means that the garbage collector can free up memory when your done. The next example shows how this works in C#.

#java
if(condition) {
    int i = 9 + 8;
    // do something with i
}
return i; // error, because i is out of scope

If we would try the same in JavaScript, it would lead to different results.

#js
if(condition) {
    var i = 9 + 8; 
    // do something with i
}
return i; // fine, i is in scope

This can really fool you. Because these are the only two scopes we have, libraries that define global variables (like jQuery does) can conflict with each other easily. So you want and should be careful to not pollute the global scope. That is also why you should always use the var keyword to declare your variables. Declarations without this keyword cause global variables, even if you only use them inside of a function.

Access modifiers

One of the first things that we learn in object-oriented languages is the usage of private and public modifiers. With those modifiers it possible to hide information inside of object instances. It is one of the core principles of good code design: information hiding.

In JavaScript there is no such thing as access modifiers. So we are left with two choices: give away our implementation details, or come up with some clever way to hide them anyway.

These two key differences between JavaScript and other object-oriented languages make that we should come up with some clever way’s to structure and design our code. There are three things that I would like to point out.

Namespaces

The first thing that I choose to do is adding namespaces to my code. Namespaces are used to create unique names within a certain naming scope. C# has them, and so does XML. Unfortunately, JavaScript doesn’t know namespaces. But you can still simulate them by using object properties.

#js
// create the first level
var overeemm = window.overeemm || {}; 
// create a subnamespace
overeemm.conway = overeemm.conway || {}; 
// continue creating as many levels as you would like

What happens is actually very simple: we just create empty objects to simulate a namespace level. By using the OR notation I am sure not to redefine another namespace and thus clearing functions and objects created on that level. By putting this into every file, I know that the namespace that I will use is available.

With this simple technique we prevent naming conflicts with other libraries.

Hiding your private stuff

We want to prevent all of our internal implementation details to leak out of our objects. At least I would. So we need some clever way to do this. There are a couple of things that you could do, I have been using closures to achieve this.

A closure is a function stored together with his environment (the variable bindings). When you execute this function, it has access to the variable bindings from the stored environment. It is said that the function is closed over the free variables. Because the function keeps access to all those variables, it is possible to create an hidden environment. This hidden environment is only accessible by the functions defined in this environment.

#js
var f, g;
function doSomething(x, y) {
    var z = x + y;
    f = function () { return z++; };
    g = function () { return z--; };
}
doSomething(1, 4);
// f and g are free to use

After calling doSomething we can use the functions f and g. But they are defined within the scope of doSomething! Because they are defined in the same scope, they both have access to the same variable z. This variable is not available outside of doSomething, making it a private variable.

If we change the code snippet a little bit, we can create an object with his own private scope.

#js
function createObject(x, y) {
    var z = x + y;
    return {
        f : function () { return z++; },
        g : function () { return z--; }
    }
}
var o = createObject(1, 4);
// o.f and o.g are free to use

With this technique we actually achieved two things:

we created an object o with a private state.
we did not pollute the global namespace with f and g, but only with o.

Singletons through self executing functions

But what if we have a library with utilities. Or we want to create a singleton object? We can use the same technique, but place it in a different context.

#js
var singleton = (function(x, y) {
    var z = x + y;
    return {
        f : function () { return z++; },
        g : function () { return z--; }
    }
})();
// singleton.f and singleton.g are free to use

We change the create-function to an anonymous function that is directly executed. Because of the object that we return, we have created a private scope. And by executing the anonymous function, we make sure that nobody can create a second instance.

For me, these examples show a nice and elegant side of JavaScript. One could argue that they are solution around problems in a bad designed language. However, we have to deal with our inheritance even if we do not like it. And to me, these techniques show creativity and possibilities.

Singleton revisited

After a twitter discussion I wanted to revisite the singleton pattern a bit. One of the aspects of the singleton pattern is the fact that you can ask for the thing you want and get the same instance back, no matter how often you ask for it. That is something that was not captured in my code example.

On stackoverflow a nice example is given. The principle is that you redefine the creation function and let it return the same instance. I thought it was a creative solution, and again I am amazed by the things you can do :-)

#js
function singleton() {
  var instance = (function() {
    var privateVar;

    function privateMethod () {
      // ...
    }

    return { // public interface
      publicMethod1: function () {
          // private members can be accessed here
       },
      publicMethod2: function () {
        // ...
      }
    };
  })();

  singleton = function () { // re-define the function for subsequent calls
    return instance;
  };

  return singleton(); // call the new function
}

The rise of JavaScript

2011-12-22T00:00:00+00:00

I’m digging deeper into JavaScript lately, and I like what I find, see and try.

I remember that when I first saw JavaScript I was building a website for our graduating class in high school, it was around 2001. It was the period that JavaScript was used in the marketing term DHTML. It was primarily used for dynamic menu’s with overlays and for the snow effects during winter. And yes, I am guilty for doing that too.

During college, fast forward three years, we used JavaScript for client-side validation. It was still advertised as a scripting language, useful for dynamic HTML.

Then during my first full-time job, the rise of AJAX happened. Microsoft added dynamic capabilities through various WebControls (like the UpdatePanel). Prototype.js was the first major library. Everything and everybody that used AJAX was hot. People started pointing out the downsides: JavaScript could be disabled, so websites should work without it. The battle of browser compatibility began, Internet Explorer 6 was still mainstream.

And then, slowly it all evolved from sprinkling extra’s into an HTML environment to full web applications build in JavaScript. Great examples are of course gmail, google docs and Cloud 9 IDE. JavaScript has grown into a serious programming language, that people should know and care about. With Node.js it even became possible to write your web application’s server-side in JavaScript. Microsoft is enabling you to use it for desktop applications on Windows 8.

Some saw this evolution, thought about it and stated bold things ( Yes, it was written in 2007, I know that I’m late to the game. ):

Any application that can be written in JavaScript, will eventually be written in JavaScript. (Atwood’s Law).

and

JavaScript is an assembly language. The JavaScript + HTML generate is like a .NET assembly. The browser can execute it, but no human should really care what’s there.(Erik Meijer on hanselman.com)

Atwood’s Law is getting confirmed every day: real life web applications and games are written in JavaScript, not only for fun! Hanselman’s statement is seen in things like the Google Web Toolkit and the Cappuccino framework. But also in ClojureScript and CoffeeScript. They all try to lift some of the problems in JavaScript by using other languages or creating new languages that compile into JavaScript.

I am not sure if that is the way to go. I like JavaScript. I’ve read some great books by Douglas Crockford (JavaScript The Good Parts), John Resig (Pro JavaScript Techniques) and David Flanagam (JavaScript The Definitive Guid). Still I think that I scratched nothing but the surface. There are so many tools, libraries and frameworks from which I can learn. However, I can also see where those abstraction are coming from. I never learned C or C++. It is great that we have C#, Java, Ruby and so on on top of those languages to make development easier, faster and less error prone. But is JavaScript already in that stage? I am not sure… only time will tell.

I hope to digg into JavaScript and some libraries in the next couple blog posts.

First steps in a new language

2011-11-03T00:00:00+00:00

One of my all time favorite books on software development is The Pragmatic Programmer: From Journeyman To Master. I highly recommend it. You do not have it? You didn’t read it yet? Go buy it!

One of the tips given in the book is

Learn at least one new language every year. Different languages solve the same problems in different ways. By learning several different approaches, you can help broaden your thinking and avoid getting stuck in a rut. Additionally, learning many languages is far easier now, thanks to the wealth of freely available software on the Internet.

Although I read through the seven languages in seven days book and the book of Ruby this year, I was still not comfortable enough with Ruby to use it when necessary. So one of my goals was to really learn Ruby. And the best way to learn is by using it. Like Knuth already said in his classic The Art Of Computer Programming:

It is difficult, if not impossible, for anyone to learn a subject purely by reading about it, without applying the information to specific problems and thereby begin encouraged to think about what has been read. Furthermore, we all learn best the things that we have discovered for ourselves.

This write-up is a summary of my first (baby) steps in Ruby.

I started out with doing some katas from the pragmatic programmers. I quickly found out that although they are good exercises, they did not interest me. And because of that I was not really motivated in doing them.

The second exercise was a rewrite of some tool that I build in C# to encode, upload and publish mp3 files to a Joomla-website. The .NET tool is a WinForms executable, using lame to encode, using ftp to upload and using XML-RPC to publish the post. The Ruby version is one script, doing all those things in sequential steps. This exercise was more fun, because I needed it (the .NET tool didn’t work on my MacBook).

My scripts can be found on github.

So far so good. I like the language, but I do realize that these are just scripts. And by no means they are building a real life system. So the next exercise should be bigger, forcing me to use more advanced features. Maybe I should rewrite the Joomla-website in a simple RoR version?

How do you learn a new language?

Using Google APIs through OAuth 2.0

2011-09-09T00:00:00+00:00

Lately I’ve been working on a web application that connects to Google docs: gdocerous. During the development I’ve stumbled up on OAuth and other authorization protocols. This blog post is a small write-up of the experience.

As the major cloud provider, Google has lots of content available through their applications. Fortunately, they also have a nice API with lots of support and samples. But also lots of ways to connect and authenticate. They support AuthSub (their own proprietary protocol), OAuth 1.0 and OAuth 2.0 (still in beta), but also a plain user name and password connection. They do recommend that you use OAuth, and because the specs for OAuth 2.0 seem stable enough, I would recommend to use the 2.0 version. This post shares some example code on how to use it in a .NET web application.

OAuth 2.0 in short

The OAuth 2.0 protocol is an evolution from OAuth 1.0 (really? yes it is!). If you are interested in knowing why a new version was necessary, I suggest that you read the following article from Eran Hammer-Lahav: Introducing OAuth 2.0.

The global idea is that you redirect a user to the OAuth-server. There he can authenticate and grant access to the application that redirected him in the first place. The user gives access for a certain scope. In the case of Google, this scope can be Google docs, gmail or some other application. It is of course important that the scope for which access is requested is as narrow as possible. After the user has authorized the application, the OAuth-server redirects to the application, giving an authorization code along. This code can be used to request an access token. And finally, the token can be used to access APIs and resources.

The main advantage is that the user does not have to give his user name and password to the application. Instead the application requests access and receives a short-lived access token. This makes it easy to revoke the access in a later stage, without having to change any credentials.

Is all good in the Google OAuth 2.0 world? No, not really. Some things remain hard. For instance using Google’s IMAP or SMTP. Although the protocol documentation is clear, there is no easy way to use this from .NET.

Show me the code!

So how do you get started with these APIs? Let me walk you through the basic steps.

First, register your app in the Google APIs console. This gives you a clientid and clientsecret. It also allows you to give your application more trust by supplying a description and logo. This way users can recognize the application that asks for trust.

On requesting authorization, your application starts with redirecting the user to https://accounts.google.com/o/oauth2/auth, following the description of Using OAuth 2.0 to Access Google APIs. This part is not that hard. Users are shown a recognizable Google page on which they grant access for your application.

After these two steps we need to write some real code. First retrieve the code from the request en get an access token from Google.

#csharp
string queryStringFormat = @"code={0}&client_id={1}&client_secret={2}&redirect_uri={3}&grant_type=authorization_code";
string postcontents = string.Format(queryStringFormat
                                   , HttpUtility.UrlEncode(code)
                                   , HttpUtility.UrlEncode(clientid)
                                   , HttpUtility.UrlEncode(clientsecret)
                                   , HttpUtility.UrlEncode(redirect_url));
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("https://accounts.google.com/o/oauth2/token");
request.Method = "POST";
byte[] postcontentsArray = Encoding.UTF8.GetBytes(postcontents);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = postcontentsArray.Length;
using (Stream requestStream = request.GetRequestStream())
{
    requestStream.Write(postcontentsArray, 0, postcontentsArray.Length);
    requestStream.Close();
    WebResponse response = request.GetResponse();
    using (Stream responseStream = response.GetResponseStream())
    using (StreamReader reader = new StreamReader(responseStream))
    {
        string responseFromServer = reader.ReadToEnd();
        reader.Close();
        responseStream.Close();
        response.Close();
        return SerializeToken(responseFromServer);
    }
}

The result is a JSON string that, with the help of DataContractJsonSerializer, can be serialized into

#csharp
public class GoogleOAuthToken
{
    public string access_token;
    public string expires_in;
    public string token_type;
    public string refresh_token;
}

The given access token comes with an expiration time. To extend your timeperiod, you can use the refresh token to request a new access token. The code to do this is similar to the example above.

After this we can finally get to work. Let’s get a list of all folders from google docs.

#csharp
var settings = new RequestSettings("your appname", "oauth access token");
var request = new DocumentsRequest(settings);
foreach (Document doc in request.GetFolders().Entries)
{
    /* do something */
}    

This snippet uses the Google Data Protocol, using the C# library available from google code. The Data Protocol it self is REST-inspired, but the library gives a nice object-oriented wrapper around that.

The library wraps all major APIs from Google, so anything is possible.

Have fun coding!

Blogging in the cloud through gdocerous

2011-08-21T00:00:00+00:00

I’m trying to find more time and topics to blog about. While doing that, I reviewed the work flow that I have for blogging. My blog is hosted on posterous.com, a great service (although they do have some downtime-issues). It makes posting a blog easy: just send a mail to posterous@posterous.com and your done. The sender e-mail is recognized and the e-mail is converted to a blog post. They automatically convert images to galleries, embed youtube links and so on. If you are working on multiple blog posts at a time, you could store your mail as a draft or store it somewhere else. Somewhere else is no fun, because it means that you will have to have some other intermediate form. Storing it as a draft means that you do not have history on your blog post.

Lately I’ve started to use google docs more and more. And it looked like a great place to store my draft-blog posts. But about posting the final version? In that case you had to copy-paste it over to your e-mail and reformat it. Not that easy and fun anymore.

And then I got this great idea: wouldn’t it be great if you could choose your google document and post it to posterous? Of course it would be :) So that became my weekend-project: gdocerous. ( the domain is not yet active, it should be this week ).

I have a working version, but unfortunately it is not open for public yet. I still have some things to finish before opening it up:

I want to use the OAuth API instead of the AuthSub version
I still need to figure out how to use gmail’s smtp through OAuth

The whole thing is pretty simple, but I think useful. One of the things is that I do not want to store sensitive data or user accounts. That’s why the posterous API is not usable. Another thing is that I only want to select and post, no mutation of articles and such. Last is that the complete source code is open for everyone to look at.

Reading != learning

2011-08-07T00:00:00+00:00

I think that I have always been a reader. There is a box with books from my childhood, and now I have shelves full of books. A great number of them are technical. But there are also novels and history related books. I even signed up to review books for O’Reilly. I already did a number of them (including two reviews that were not for O’Reilly).

But you know what, I did not really learn anything from them yet. The problem with technical books, programming books is that you have to do something with the knowledge. Otherwise it will not stick. So I’m taking a break from the reviewing stuff. I’m going to set two goals for myself, after which I can pick up the reviewing again. What are those goals?

1. Build an iPad/iPhone app and release it to the Appstore

After the iOS book I was eager to write something, but then again another book came by and the app is left unfinished. So I’m going to finish it and release it. I have an idea and already started on it. It is time to focus and finish it. Probably will take a couple of months though.

Progress will be visible in my github repo.

2. Learn Ruby and change my default language for a certain period

I have just finished the book of Ruby and before that touched on Ruby while reading the seven languages book. But again I’m lacking some real experience. So I will change my default script/tool language for a certain period and see where that will take me. I hope to get some fluency in Ruby and try on IronRuby and MacRuby. That will give me some flexibility in developing on my MacBook or on my Windows laptop. I’m planning of working through some of the coding Katas as described here. That will give me some guidance as I’m trying stuff.

The big eye-opener here is that I have to stop reading technical books, but I have to start doing them. Let’s see how I will do.

bookreview - the book of Ruby

2011-08-07T00:00:00+00:00

The book of Ruby by Huw Collingbourne

This book covers every part of Ruby that a newbie Ruby developer should know. It starts out with the basics around classes, objects, collections and moves over to more advanced stuff like threads and marshaling. Not only the language is covered: developing, debugging and the framework Rails are also explained to the reader. The book is full of example code snippets and good explanation. For the more experienced developers the author has added ‘digging deeper’ sections, that take you to more advanced topics. The book finishes of with more reading material and guidance around development tools.

The book has a subtitle telling us that this is ‘a hands-on guide for the adventurous’. That sure does sound promising. Unfortunately, the book does not read as a ‘hands-on guide’. It reads more like a reference manual. There are no assignments to practice the language and the examples are short. It misses a real life example that is build from start to end. You would expect some real application coming from a hands-on guide. The book self is well written and reads very easily, but the ‘reference manual’ style makes it somewhat boring.

I definitely recommend this book to people starting in Ruby, but not as a first book. Start with a real hands-on book that walks you to the creation of a basic application. Then pick up this book and walk to all the different topics. That will give you a good understanding of Ruby.

Find the book at http://oreilly.com/catalog/9781593272944/

bookreview - Head First iPhone & iPad Development, 2nd Edition

2011-07-16T00:00:00+00:00

Head First iPhone & iPad Development, 2nd Edition by Dan Pilone & Tracey Pilone

The Head First series are always a great way to get started in some new technology. This book is no exception to that. This second edition is completely rewritten to match the new iOS and Xcode versions. The authors take the reader with them on a path to learn these new technologies. Every aspect of developing for the iDevices is shown and discussed. Real life examples are used to build complete applications that can be submitted to the App Store. The book starts out with a simple iPhone application which is then expanded to incorporate more advanced stuff such as iPad support, device orientation, Core Data and usage of the Camera. Every part is explained in detail with code samples and screenshots.

Overall the book is a great way to get introduced to development for iOS. The authors try to cut lose from boring introductions. Code samples are alternated with screenshots, in dept explanations and silly interviews. This way the book stays fun and interesting. However, I found the style a bit to childish from time to time. Interviews with GUI controls are not the content that I would expect. It does make this book useful for teenagers that want to get started. I think the content could be made a little more dense and less childish.

Find the book at http://oreilly.com/catalog/9781449387822/

bookreview - JavaScript, The Definitive Guide, 6th edition by David Flanagan

2011-05-15T00:00:00+00:00

Javascript, The Definitive Guide, 6th Edition by David Flanagan

This book is called ‘the definitive guide’ for a reason: it is by far the most complete book on Javascript that you can buy. Every subject, from syntax to popular libraries is covered. The 6th edition is completely updated to reflect the latest standard, ECMAscript 5. The newest features such as localstorage and geolocation are explained in this new edition. The book follows a clear outline. It starts by explaining the language and it’s syntax, followed by it’s features. Next is the usage of the language, both as a client-side language and as a server-side language. The book also covers the most popular Javascript library: jQuery.

I found the book, although it is written as a reference, easy and entertaining to read. The book is a great resource to use while developing Javascript, because every little detail is covered. A good indication of its quality is that the book survived five editions and is still going strong! What I did miss is some guidance around real world Javascript development and tooling. The book does not explain anything around unit testing, development environments and other useful tooling. That would be a welcome addition for the 7th edition!

Find the book at http://oreilly.com/catalog/9780596805531/

Two years of reading summarized

2011-05-05T00:00:00+00:00

So I like to read… a lot. And not just books about programming or computer science, but like everything. Exactly two years ago, I started to use @deboekenlijst. It is a twitter service that builds a list of the books that you have read. So every time you have read a book, you tweet the information to that account. Once a week (or less), a website is build with statistics and an overview. This week I thought, let’s see how much I read. I’m currently in the top 10 of the most active readers:

I started on May 3 2009, the last book was entered on April 28 2011. In that period I’ve read 108 books in 104 weeks.

The analysis of my reading started with getting the information from the website, parsed it to JSON and expanded the list with books that weren’t yet analysed. After that, I added information about language and type (ebook vs dead-tree). With google charts, I build some nice charts.

44 books are ebooks, 64 are paper books. The ebooks are in ePub, Kindle and PDF format. They were read on sony’s PRS-300 and on Apple’s iPad 1. And although eInkt is very nice to read, I like the iPad more. The screen is larger and it supports more formats. Downloading and reading without the need to link it to my laptop is also very nice.

I think that ebooks are the future, and I try to buy and read them as much as possible. I do not like DRM however.

I’ve read 31 English books, the rest is in Dutch. The English books are mostly technical, although I buy some non-technical books in English too. Number of books per month

On average, I read 4 books per month. Some months were more productive, reading wise. How come?

July 2009: two week holiday
May 2010: two week holiday
October 2010: the service went down (due to twitter errors), so I re-added a number of books at once
March/April 2011: became father and had two weeks holiday

On to some more reading!

bookreview - Seven Languages in Seven Weeks by Bruce Tate

2011-04-29T00:00:00+00:00

Seven Languages in Seven Weeks by Bruce Tate

I love to try out new programming languages and different to ways to develop software. But my time is limited. So it is hard to really learn new languages and keep up with what happens outside of my comfort zone (I develop mostly in C# and JavaScript). So a book that promisses to learn you seven languages in a short period makes me curious.

The concept of this book is great: show you enough of a language to give you an idea, but keeps it short enough to not make you bored. Programming language books often become lists of syntax rules and library calls. This book gives you enough to become interested and curious, but moves on before you become bored.

My reaction on every one of the seven languages:

Ruby: The language for cool web developers. What I like is the dynamic aspect: quick prototyping and developing of software. Together with the Rails framework it really blends well with the client-side aspect.

Io: Prototype based, so you recognize JavaScript aspects. Not really useful in a commercial environment, but great to learn the prototype based stuff. Hard to find stuff about this language through google :)

Prolog: I already learned this during college, but it is nice to revisit. Prolog really makes you think about programming. Solutions become more declarative, instead of imperative.

Scala: The new language for Java developers. I like how it blends functional and object-oriented. But with C# becoming more functional, it didn’t capture my interest.

Erlang: A functional programming language. It strength is in the concurrency library. Didn’t trigger my curiosity.

Clojure: Awkward syntax because of the prefix notation. Lots of parentheses. Looks a bit to academic.

Haskell: Did not really put a lot of time into this chapter. During college we did a lot of Haskell, so nothing new for me in this chapter. The language still looks to academic, but seeing a number of dutch startups using it (http://tupil.com/, http://www.silkapp.com/), it still triggers interest.

The downside of this book is that it doesn’t show you how to use these languages in real world problems. You should develop larger applications with these languages to really get the feeling. What’s next? I hope to dive into Ruby and Clojure, because these did trigger something. Not so much with Io, because I do enough JavaScript programming. Scala and Erlang do not really have my interest either. Haskell and Prolog are out, because I did that in college already.

Notes from the class Advanced Agile Development

2011-04-01T00:00:00+00:00

Last week I attended the Advanced Agile Development class, organised by Xebia and given by Alistair Cockburn. With this post, I try to make my notes persistent for future reference.

these are my notes. they are far from complete. as seen from this tweet, the class was really advanced :)

But I did survive!

I survived @TotherAlistair :)moby.to/z4s4u2
— Michiel Overeem (@michielovereem) March 30, 2011

Content

craft; deepening (life long learning) skills in a medium (for instance a progr language); shu-ha-ri
cooperative game; all about strategies (invent, decide, communicate)
flow management; manufacturing lines; palette size
self awareness/ personalities; everybody is different, look at the personalities in your team
design as a knowledge acquisition; try to get as much knowledge as you can with every step

Craft

Learning takes place in three stages:

SHU - learn a skill; copying a technique step by step
HA - collect techniques; search for other techniques to try.
RI - blend and invent techniques; knowledge and experience are combined into something new

Do not try to become RI without really archiving that level. Otherwise you are just picking techniques and doing something that you do not understand.

good trick > technique > process > dogma It starts with a ‘good trick’, that becomes a technique, which is formulated into a process and that becomes a dogma. You have to bring everything back to a ‘good trick’. There are no rules. Master agile and reach RI level.

Cooperative game

The speed of the project is the speed at which ideas move between minds

what are conditions to facilitate speed: colocation, …
attitude: ego, fear, self preservation, …

Programming is Community Theory Building Communication is touching into shared experiences

Development is a cooperative game: Invent, Decide, Communicate primary goal: deliver working software secondary goal: set up for the next game (maintenaince, new release)

Flow management

Design looks like manufacturing if we use ‘decisions’ as inventory. Let the decisions flow through the team. Find the bottleneck and fix the process.

Remove waste. Example: tracking bugs is waste, we need to fix them.

Design is knowledge acquisition

Software development is like knowledge acquisition. Eliminate risks early on.

business risk: are we building the right thing
social risk: can they/we build it
technical risk: performance, compatibility
cost schedule risk: see if features change the costs of other parts of the project.

trim the tail: first do risk, then do value, then do the fine-tuning/glossy/polishing slicing helps with trimming the tail; build complete vertical-slices of user stories.

Properties of success projects

frequent delivery; quick visible progress; stakeholders can give feedback; little steps so little errors
reflective improvement; evaluate to improve
close/osmotic communication; communication should be easy, everybody should be involved
personal safety; willing to listen to others & ability to speak without fear to be damaged; trust for everybody to be part of the team
focus; knowing what to work on, having time to work on it; distraction
easy access to expert users; quick and better feedback on decisions
technical environment; trust to try stuff out; unit tests/ CI/ version control
sunshine/visibility; transparency; be honest when failing; motivation

Some aha’s

More reading: Jeff Patton website, Book Agile Software Development, the cooperative game

bookreview - Gamestorming by Gray, Brown, Macanufo

2011-03-22T00:00:00+00:00

Gamestorming, A Playbook for Innovators, Rulebreakers, and Changemakers by Dave Gray, Sunni Brown, James Macanufo

Gamestorming is all about using games to bring up ideas and solutions in your team. The games are used to give a creativity boost. The book starts out with an explanation of the different concepts that are used in gamestorming. The essentials are explained to give a better understanding of using games in business. A game is split up in three parts: an opening, an exploration and a closing phase. During the exploration phase the team should produce nodes. This can be done by sketching and modeling. Important things during this phase are allowing randomness and improvisation. In the closing phase it is important that an artifact is produced. Through selection the best results can be taken out of the game. The book also points out some important skills; for example asking questions, visual language and improvisation. After the more theoretical part the authors give a list of games that can be used in gamestorming. These games can be used as a reference list that can be used to guide a team in producing new ideas and solutions.

The book is a fun read. The authors are clearly speaking from a lot of experience. The theoretical part gives a solid understanding of the different parts and gives body to the book. The larger part of the book is a list of games that can be used. This is somewhat disappointing, because it makes it more of a reference book. After the first part I was tempted to put the book away and leave it there until I could use the games. That is a shame, because there is a lot to learn from those games. The authors could have taken more information from the game-descriptions to make the theoretical part larger. I cannot wait to try these new ideas in real life.

Find the book at http://oreilly.com/catalog/9780596804183/

My photo library over the years

2011-01-06T00:00:00+00:00

After reading Hanselmans blog I became curious. How does my photo library look?

The journey, or better adventure, started with seeing if I could get those statistics from my library. I use iPhoto to manage all my photo’s, but it doesn’t come with these kind of statistics. I does have some .xml file, so I thought lets use XSLT. But no, the necessary information (like resolution) is not in this .xml file. Next step was to do it myself. So after copying everything to a Windows environment (yes, I do use Windows and I like Visual Studio and C#) I started to do some gathering. The opensource library jpegdata came to the rescue.

But then you also want to view charts, right? Nobody wants to look at some numbers. So I dove into google charts to generate some nice pictures. And the result is:

So what does this tell me?

I don’t invest a lot in camera’s (I often make photo’s with my iPhone), so the resolution does not increase a lot.
I probably need to clean up my library, because there was a lot of garbage in it :-)

Again, not very interesting, but a fun exercise.

bookreview - Continuous Integration in .NET by Marcin Kawalerowicz and Craig Berntson

2010-09-27T00:00:00+00:00

Disclaimer: I responded to a tweet of Manning (http://www.twitter.com/manningbooks) and received a free copy of the e-book to review it. I reviewed chapter 1 to 11 released to the MEAP.

Continuous Integration in .NET by Marcin Kawalerowicz and Craig Berntson

A bit about my background: I am a .NET developer for almost 5 years (3 of them full time). I am familiar with Visual Studio 2003 up to 2010, TFS 2005 and 2008, SVN and SourceSafe. I am very interested in using the right tools to speed up development and increase software quality.

Continuous Integration is getting more and more attention from the .NET community. With Microsoft focusing on ALM tooling and releasing new and better versions of Team Foundation Server, CI is getting a place in the spotlights. However, the barrier for starting with CI is still high. There are lots of tools, and starting with CI can be overwhelming. Many aspects and things need to be thought out. This book brings a lot of information about starting with CI to the .NET community. The book doesn’t presume any knowledge except for some basic .NET and Visual Studio things.

The book is very pragmatic and practical. The authors take you through the complete process of installing and configuring a CI server, visualizing this process even more with an example project. The strong part of this book is that the authors aren’t focusing on one technology by forcing Microsoft tools on the reader. They show different approaches, and give you the pros and cons of Microsoft tools, open source tools and commercial third party tooling. So the reader can make a solid choice based on functionality, complexity and pricing.

Although the book gives a lot of information, there are also some downsides to the book. One of those downsides is the current collection of blogs, twitter accounts and other resources that are available through the Internet. Many pieces of the information found in the book can be found online. The danger is that the book becomes outdated very fast.

By making the book very practical, the writers have made the book also less readable. The content forces you to read the tutorial steps one by one, because sometimes good information is hidden between those steps. Not all readers are interested in every technology, so it could be wise to split up the content into different parts. In that way readers can decide to skip some parts of the book. By splitting up the book, it can also become more of a timeless resource. Readers interested into the concepts and the big picture can focus on the main content part, while readers wanting to get started on implementing CI into their organization can go to the tutorial section.

A piece of information that I missed is the fact that most of the readers probably aren’t starting with a fresh project. Many developers are already maintaining pieces of software that could really benefit from CI. However, how does someone start with CI on an existing piece of software? That could really be a valuable addition to the book. A smaller piece that is missing if a little bit of information about Mono and how to make sure that you software is compatible with Mono. I would be nice to spend a chapter on that part as well.

To summarize this review: I would recommend this book to any one who is starting fresh on the CI process. It is a very good starting point with practical guides on the implementation of CI. For people already familiar with CI and using it to some extend, the book can be a next step. Experienced developers familiar with CI and the different steps will probably find little new information in this book.

ebooks uitlenen (dutch)

2009-10-08T00:00:00+00:00

Zoals de meeste die mij volgen op twitter wel weten lees ik nogal veel boeken (zie bijvoorbeeld mijn pagina op @deboekenlijst).

Wat mij betreft zijn ebooks dan ook helemaal geweldig. Ik koop mijn boeken toch al, dus waarom niet op een formaat dat ik ook makkelijk mee kan nemen. In plaats van 10 keer een boek van 3 a 4 centimeter dik heb ik nu alleen mijn reader nodig op vakantie. Maar, het moet natuurlijk wel net zo gemakkelijk zijn als een boek. Als het even kan wil ik mijn boeken kunnen uitlenen en wil ik boeken van vrienden en familie lenen. En dat laatste bleek toch iets lastiger…

Ik heb een reader van bol.com en heb daar inmiddels ook een aantal boeken gekocht. Bol.com gebruikt het epub formaat, wat betekent dat je je boeken moet registreren met Adobe Digital Editions. Je autoriseert je computer, je reader en daarna het ebook. Zolang het 1 reader betreft is er niks aan de hand. Maar nu wil ik mijn ebook uitlenen. Dus ik stuur het .acsm bestand op dat je krijgt van bol.com. Ze adverteren immers met het feit dat je boeken kan uitlenen (Het lijkt erop dat dit terug gedraaid is? Ik kan dit niet meer op de website vinden). Helaas kan het ebook bestand niet met een ander Adobe ID geregistreerd worden. Dus dan maar proberen het boek met mijn ID registreren voor een andere reader. En weer helaas: een reader kan maar aan 1 ID gelinkt zijn! Oftewel: wil je boeken (uit)lenen? Dan moet je 1 Adobe ID gebruiken. Dat is natuurlijk niet handig.

Daarom ben ik maar op zoek gegaan om de DRM van de epub files te slopen. Let op, niet met het doel om al mijn boeken te verspreiden. Ik verspreid ze alleen maar naar vrienden en familie, met daarbij natuurlijk de voorwaarde dat zij het niet weer verder verspreiden. Lijkt mij geen echt probleem, ik koop immers al mijn boeken nog steeds… (overigens zijn er via Google en Projet Gutenberg ook gratis boeken te downloaden).

Het verwijderen van DRM is eigenlijk vrij simpel (Dit is alleen voor Windows, ik heb niet verder gezocht naar een MacOSX handleiding). Even googlen levert al snel 2 links op: http://rapidfind.org/upload/showthread.php?p=1035319 http://klungvik.com/index.php/2008/how-to-remove-drm-from-ebooks/

Wat heb je nodig?

Python voor Windows (versie 2.6)
De library Pycrypto
Adobe Digital Editions
Een tweetal scripts: ineptkey.pyw en ineptepub.pyw - deze zet ik niet ter download hier, maar even googlen levert ze al snel op.

Nu begint het echte werk:

Eerst moet je een ebook bij bol.com kopen. Dit levert een .acsm file op.
Deze open je met Adobe Digital Editions, unlock hem met je Adobe ID.
Zoek nu de map ‘My Digital Editions’ op. Hierin staat het echte .epub bestand.
Kopieer het .epub bestand naar de map met python scripts.
Run eerst het script ineptkey.pyw. Dit levert een adeptkey.der bestand op.
Run het script ineptepub.pyw. Je krijgt een schermpje dat vraagt naar het key bestand (zie vorige stap) en het .epub bestand. Specifieer het output bestand en runnen maar.

Het bestand zelf kan nu met bijvoorbeeld Calibre omgezet worden naar andere formaten, of op de reader geplaatst worden.