Edge of the desert

Death and taxes

2010-06-20T13:27:00.003+02:00

It's that time of the year again to start filing taxes, and here in Belgium we are "blessed" to be able to use our Belgian eid to file those taxes electronically at our governments Tax on Web site.

Linux support

Officialy, there are some drivers available for Redhat and Ubuntu 9.10. However, remembering last years debacle of trying to follow those instructions, I decided to not stumble into that tarpit and just did a bit of googling around. After pulling together some various bits and pieces, this is what I did to make it work:

$ sudo apt-get install libacr38u libacr38ucontrol0 beid-tools pcscd libpcsclite-dev beidgui libbeid2 libbeidlibopensc2

Reboot. Yes, I know this is Linux and I know it is 2010. The problem is the pcscd service won't start properly without a reboot. If you have a better clue then I do, you can probably fix this without a reboot, but there it is.
Launch firefox and navigate to file:///usr/share/beid/beid-pkcs11-register.html Depending on your version of Firefox that may or may not properly install the pkcs11 module. For me it did not so I moved on to the next step.
In firefox, navigate to Edit > Preferences > Advanced > Security Devices Click on Load and fill out file:///usr/lib/libbeidpkcs11.so.2 as the Module filename. Then click OK.

After these steps I was able to log in to the Tax on Web site. Hope this is helpful to somebody else out there.

Animated typography is awesome

2010-04-02T14:58:00.002+02:00

Visualizing tweets over a 16 hour span, a Twitterswarm

2010-03-31T20:07:00.008+02:00

Every second thursday of the month, the Media and Design Academy organizes its puntKom event. A few work colleagues have attended these events on and off and my employer, Nascom, has been known to provide the attendees with free snacks and beer.

During one puntKom, people challenged us to do an actual presentation for their next theme, which was to be "20 lines of code". The main idea was to come up with something cool / quirky which could be done in 20 lines of code and give a short presentation about it. After the usual discussion on how worthless loc is as a measurement and how most heavy lifting would always be done by frameworks, a few colleagues and I had a short brainstorming session to see if we could think of something interesting.

The plan

It quickly became clear that to do something visually interesting, we would have to find some data, and glue it into an interesting visualization scheme. So we had to come up with 3 components; glue, data and a visualization idea.

Glue, data and visualizations

As any linux user will tell you, the best way to glue stuff together on a computer is to use a quick and dirty bash script. Finding the glue was easy, and from the start, I had wanted to do something with code_swarm. The only thing left to do was to come up with a good dataset. After considering a few different sources, we ended up with the plan to pull down all the tweets we could that showed up for a few predefined search results and feed those to code_swarm.

Approach

After the brainstorming, I was given some time to hack together ourlittle idea. I started out by breaking down the problem in the followingsubproblems:

Fetch a bunch of tweets from Twitter
Convert those tweets to a format usable by code_swarm
Run code_swarm to generate a bunch of png's
Use ffmpeg to generate a video out of those png's

The first problem wasn't too hard, a simple wget to the Twitter search api to fetch the .atom version worked just fine. Until it didn't. It turns out that Twitter auto throttles ip's that make a lot of requests to their search api, and while normal Twitter api usage is linked to a specific Twitter account, their search api usage is linked to an ip. A pretty harsh constraint when there's about 80 colleagues with various Twitter clients hitting that same api.

Luckily, the Twitter engineers were kind enough to return a HTTP 420 code and a Retry-After header whenever a request was throttled, which allowed me to just put the script to sleep for the specified time whenever a 420 occured. After this fix, fetching tweets was a breeze.

As for the second problem, converting an atom feed to a custom xml format is annoying enough without a 20 loc constraint, so I googled around a bit for some xslt tools. Pretty soon I discovered the excellent XMLStarlet tool which would allow me to generate and apply XSLT from the commandline, the perfect tool to glue into my script.

One thing that was annoying me was how to convert the dates of the tweets to the millisecond format required by code_swarm, so I ended up tweaking code_swarm's source to also accept the standard tweet date format.

From this point, it was fairly straight forward to come up with a semi-decent configuration file for code_swarm, feed my xml to it, and then let ffmpeg lose on the generated png's as per the code_swarm wiki instructions.

Results

The code ended up looking a bit like this:


#!/bin/bash
function twitter_to_swarm { xmlstarlet sel --text -N a=http://www.w3.org/2005/Atom -N twitter=http://api.twitter.com/ --template --match '//a:entry' -o '&lt;event filename=&quot;' -v 'a:author/a:name' -o 'lang:' -v 'twitter:lang' -o '&quot;' -o ' date=&quot;' -v 'a:updated' -o '&quot; ' -o 'author=&quot;' -v '/a:feed/a:title' -o '&quot;/&gt;' -n $1 | sed -e 's/ since:.*- Twitter Search//' | sed -e 's/date="\([^T.]*\)T\([^Z.]*\)Z"/date="\1 \2"/' | sed -e 's/&/&amp;/g' >> $2; }

read PAGE STATUS <<<$(echo 1 200)
for p in "$@"; do QUERY=$(perl -MURI::Escape -e 'foreach $argnum (0 .. $#ARGV) {print uri_escape($ARGV[$argnum])."+";}' $p)
    until [ $PAGE -gt 15 ]; do
        RESPONSE=$(wget --server-response -Oout.wget http://search.twitter.com/search.atom?q=$QUERY\&rpp=100\&page=$PAGE\&since=`date +%Y-%m-%d` 2>&1)
        STATUS=$(echo $RESPONSE | grep 'HTTP/' | sed -e 's/.*HTTP\/[^ ]* \([^ ]*\).*/\1/' | tail -1)
        case $STATUS in
            200) echo STATUS $STATUS for page $PAGE && twitter_to_swarm out.wget out.xml && let PAGE+=1;;
            420) SLEEP=$(echo $RESPONSE | grep 'Retry-After' | sed -e 's/.*Retry-After[^ ]* \([^ ]*\).*/\1/') && until [ $SLEEP -le 0 ]; do echo sleeping $SLEEP seconds & sleep 10 && let SLEEP-=10; done;;
            *) echo STATUS $STATUS for page $PAGE && let PAGE=17;;
        esac; done; let PAGE=1
done
echo '<file_events>' > result.xml && cat out.xml | sort -t '"' -k 3 -r >> result.xml && echo '</file_events>' >> result.xml
echo Generating images... && cd codeswarm/ && codeswarm ../mpeg.config && cd .. && echo Done.
echo Generating video... && ffmpeg -f image2 -r 24 -i images/twitter-%05d.png -sameq ./result.mov -pass 2 && echo Done. Generated result.mov

I ran the script with parameters to search for tweets containing, google buzz or buzz, haiti, snow and iPad. The resulting video ended up looking like this:

The terms that were searched for float through space, every colored dot that moves towards a search term is a tweet matching that term. It's a bit hard to see, but tweets are also colored depending on the language they're in, the following leged was used:

yellow for English
cyan for German
violet for French
red for Portuguese
blue for Dutch
purple for Spanish
brown for Japanese
pink for Italian
grey for Other

Rerun dammit

I wasn't completely satisfied with the first result, since it turned out to only cover a very short timespan. This was due to the fact that the Twitter api only returns a max of 1500 tweets, so I modified the script above a bit so that I could schedule it as a cronjob and run it every hour. I then let my cronjob run for about 16 hours and produced the video below with the results:

Social tips and a semantic wikipedia

2007-05-07T19:57:00.002+02:00

Two barcamp talks in particular interested me enough that I just had to find out more about them.

First of these was John Baeyens' talk on his soon-to-go-live social tips website "not so-so", of which you can find the current test version right here.
Basically, not so-so is kind of a mix between blogging and delicious in that you write up a short review, or tip on something (anything!) you like, which you then automatically share with the rest of the community. You can stay up to date with the tips that other people submit by following those people ala the delicious network feature. Extra features that are currently in the works are a calendar system and a rating system. The rating system in particular seems very interesting to me, as it could be tied to a recommendation system to put you in touch with people that share your interests, or just interesting tips from people not currently in your network. John's reply to the question "I want to see all events today within a radius of 20km from where I live" was along the lines of: "We have the data, we just need to implement it.".

The other cool talk was given by Will Moffat. It was a short introduction to Freebase which is basically a queryable service containing a structured version of the Wikipedia data. The database is queryable through JavaScript and returns a list of results using JSON, allowing for some interesting mash-up ideas. Better yet it's also possible to update the database using JavaScript. Currently freebase is still in alpha phase and registration is on an invite only basis. There are still some open questions though, mainly, how are the Freebase people going to deal with vandalism, as currently doesn't appear to be moderated. Also, will this project be able to gain enough interest as to gain a large enough userbase to actually fill out all the currently missing data.

Pictures by Bert Heymans, for more pictures, check out his flickr stream.

Barcamp Brussels #3

2007-05-04T17:16:00.000+02:00

Tomorrow I will be attending the third Barcamp Brussels event which will mark the end of my barcamp virginity. Looking over the barcamp wiki, it looks like there will be plenty of interesting topics to mull over. I'm particularly interested in Frank Louwers' talk on OpenID and the talk that Jan de Poorter will be having on Rails.
As for myself, I had been planning on doing something around Collaborative Filtering and Slope One, but a busy schedule has kept me from putting together something even remotely cohesive. I'm thinking of taking some pictures and if anyone is interested in the slope one stuff, I can always annoy anyone who asks :)

Tomcat UTF-8 CharsetMapper

2007-04-26T07:32:00.000+02:00

Just a quickie I thought I'd share, if you're running into encoding problems using Tomcat and JSTL's fmt:xxx tags, you might want to implement your own Tomcat CharsetMapper. The following example will fix things for UTF-8 encoding:


public class UTF8CharsetMapper extends CharsetMapper {
    public UTF8CharsetMapper() {
        super();
    }

    public UTF8CharsetMapper(String name) {
        super(name);
    }

    public String getCharset(Locale locale) {
        return "UTF-8";
    }
}

You should then edit your Tomcat's server.xml file by mapping your Context's charsetMapperClass attribute to your custom made CharsetMapper:


<Context charsetMapperClass="your.package.here.UTF8CharsetMapper" otherParam="xxx">
    ...
</Context>

UTF-8 Checklist

2006-12-05T01:50:00.000+01:00

This is getting tedious, again I wasted a good 40 minutes looking for a solution as to why my perfectly fine utf-8 text which was being submitted through a get form was being garbled up by the server. So, as a matter of checklist, I'm going to sum up a few things that can be quickly run through to see why utf-8 might not be working. Mind you, this is going to be Java oriented, since that's where the source of my utf-8 woes lies, however, I don't have much doubt that getting a website properly using utf-8 is a pain in the ass in other languages as well. I'll gladly have anyone prove me wrong on that last assumption though.

What you should know first

The first thing you should know is that There Ain't No Such Thing As Plain Text. there, that's said, now go read the article and then come back.

Checklist

The following checklist is mainly a shorter, version of stuff I stole from a Sun article called Character Conversions from Browser to Database. You will learn more if you read that article, but my list is shorter.

First of all, make sure your container understands that the de facto encoding of any GET request it will receive will be in utf-8. In Tomcat, this is done by setting the property URIEncoding="UTF-8" inside your Connector definition. For other containers, there is always Google 'cause I'm lazy like that.

For some containers, setting the equivalent of the above configuration will make both GET and POST work properly, this is not the case however with Tomcat. To let Tomcat use utf-8 as the default encoding for POST requests, set the following in your application's web.xml
<context-param> <param-name>PARAMETER_ENCODING</param-name> <param-value>UTF-8</param-value> </context-param>
Okay, so now you have the encoding available as a parameter in every request, which means you can set it before reading any parameters like so:
String paramEncoding = application.getInitParameter("PARAMETER_ENCODING"); request.setCharacterEncoding(paramEncoding);
If you are using a Servlet instead of a JSP, you can spread this code over the Servlet's init() and processRequest() methods:
private String encoding; public void init() throws ServletException { ServletConfig config = getServletConfig(); encoding = config.getInitParameter("PARAMETER_ENCODING"); } protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { if (encoding != null) { request.setCharacterEncoding(encoding); } ... }
Alternatively, you might put this code in a filter.

When using JSP's, make sure you have your page directive set up correctly, and by correctly I mean:
<%@page pageEncoding="UTF-8" contentType="text/html; charset=UTF-8"%>
If you are going to enter anything in your JSP page directive, the above should be it.

Write proper HTML. The <meta> tag in your header should define the content attribute:
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
It's also generally a good idea to set the accept-charset attribute of any forms you define:
<form name="form" action="action.do" method="post" accept-charset="UTF-8"></form>
The accept-charset attribute will tell confused clients what charsets the server will accept.

Alright, I think that will do nicely as an initial checklist. If I come across any other pointers, I'll edit this post and add them.

Slimy emacs

2006-11-17T00:02:00.000+01:00

Slimy emacs
This blog entry is being typed in emacs21. I finally decided to clear the hurdle and start learning myself emacs and try to wrestle through Paul Graham's "On lisp". Starting out with emacs has been fun, I've been working through the included tutorial, and the shortcut system already feels powerfull. Writing / programming without having to move my hand from keyboard to mouse is a really enticing prospect to me, and I'm intent on getting through the 36% of TUTORIAL remaining. This also seems like a really good time to finally trade in my azerty keyboard for a real one, which will have the added bonus of no longer having to adjust when switching from typing at home to typing at work.

Anyway, emacs and lisp. So on to lisp, I first downloaded slime for emacs, and after including the following short lisp snippet in my .emacs file tried to launch it:


(add-to-list 'load-path "/home//slime-2.0")
(require 'slime)
(setq inferior-lisp-program "clisp")
(add-hook 'lisp-mode-hook (lambda () (slime-mode t)
                                (local-set-key "\r" 'newline-and-indent)
                                (setq lisp-indent-function 'common-lisp-indent-function)
                                (setq indent-tabs-mode nil)))

Obviously, no clisp could be found since I'd been dumb enough to not install it first. So on to the next step of installing lisp. Now, my first version of my .emacs file did not refer to clisp, rather it refered to lisp as I was following the short tutorial on unmutual.info and had chosen to use CMU Common Lisp. Right, so how to install CMU Common Lisp on my Ubuntu install, I visited the CMU Common Lisp website but then decided I'd rather use the synaptic packet manager. After a search for "lisp" presented me with a huge list of packages to install, I spent a good 10 minutes just skimming through them all untill I finally settled for the clisp package. A brief few seconds later, clisp and it's dependencies were installed and all I had to do was to edit my .emacs file to update my inferior-lisp-program from "lisp" to "clisp". Restart emacs, M-x slime, a quick compile and tada, slime ! Cool, now I just have 20% of emacs TUTORIAL and Paul Graham's "On Lisp" left to digest.

An inconvenient truth

2006-10-29T09:22:00.000+01:00

I finally went to see Al Gore's "An Inconvenient Truth" last night, and it's really one of those must-see movies because it's quite an eye-opener. There aren't many things that stress the urgency of turning around our climate crisis as much as seeing a piece of ice the size of Rhode Island break away from Antarctica over the course of days. The documentary does an excellent job in breaking down the scepticism that is surrounding global warming, it emphasizes that there is scientific consensus and that global warming isn't just some theory. The idea that there is doubt over the veracity of global warming is being spoonfed to us through the political and economic propaganda machine. The main idea to take away from the movie, is to talk about global warming, help dispelling the myth of it being a theory as opposed to fact, and change. Make small changes to how you live, and encourage others to do the same.
On the way back home I had a lengthy discussion with my dad and grandfather over how nothing would really change unless everyone participated and it was hard to explain them the irony of saying: "Well this won't work unless everyone changes, so I'm not changing". In the end though I managed to explain how the upward trend in global warming can not, in fact, be turned into a downward trend overnight, but that it would be a huge step forward if only we managed to slow it down, then let it stagnate, and then finally, let it diminish. I still think Al Gore said it better, and certainly more terse, when he said that some people go from denial to despair, without considering the steps in between. Despair will promote our inaction, and it doesn't seem like we have alot of time left for that. If you want to do something right now, the climatecrisis.com website has a nice pdf with 10 easy tips to pollute less.

Ironically and sadly, the only people that walked out during this movie were a bunch of teenagers.

Reddit and subreddits

2006-10-19T12:40:00.000+02:00

There's currently a discussion going on on reddit on the concept of subreddits. A subreddit is basically a clone of the reddit concept but it is limited to a certain topic, a good example is the programming subreddit. To be completely fair, the discussion isn't really about the concept of subreddits per se, but for the purpose of this blog entry, I'm going to pretend it is.

Anyway, the discussion seems to be between largely two groups of people. One group argues that subreddits are fragmenting the community, while the other group loves subreddits for the way they (apparently) succinctly categorize submissions by their topic. Now, it's obvious that both groups have valid arguments, fragmenting a community is obviously not a good thing, while being able to group submissions by topic obviously is. To a certain extent people could argue that it is possible to group submissions by using the search, but the way it is dependant on the submission's title pretty much makes that an invalid argument in the long run.

So what could be a solution ?
One possible solution that would allow people to both organize submissions by topic while at the same time keeping the community together on a single reddit would be a tagging system of some form. If people want to view mainly programming related submissions, they could hit the programming tag.

Right now, subreddits are a way of categorizing submissions that is broken because it fragments the community for one, and secondly, it fragments discussion of a submission by allowing people to submit the same link to both a sub and the main reddit. I do personally read the programming reddit frequently, but it is not visible at all to new people and so it does a good job in preventing people who might have only a latent interest in programming from reading the better programming related submissions. On the other hand, since there are no restrictions on what link you can submit to what reddit (how could there be), it also happens that good programming related submissions are only submitted to the main reddit. This basically forces people to read both the main reddit and the subreddit of their interest if they want to have the maximum amount of coverage for their favorite topic.

But, but, tags are broken !
Ok, so there are some problems with tagging content, the largest issue being the fact that people do not use the same tag for the same thing, so there would still be a good amount of fragmentation. However, taking into account the great asset that the reddit community is, there should be no reason not to leverage that asset. It would be perfectly plausible to implement this in a way that people can only select tags from a predetermined pool. With this approach, the community could file requests for new tags, and if such a request is popular enough, it could be added to the existing pool.

Tag based filtering and modifying
Once there is a consistent way to apply tags, users could start filtering on those tags, if there's a topic people don't like (a lot of people complain about the amount of submissions on politics) they could set their preferences to block submissions with that tag outright. Or a more subtle approach could be to allow users to configure their preferences so that submissions with a certain tag are only visible after they get a certain rating by the rest of the community. Or what about attributing a default modifier to particular tags; you're into programming but not into politics ? Well then why not apply a standard +5 modifier to submissions with the programming tags, and a -5 to those with a politics tag. How exactly to implement the weight of these modifiers is up for debate, but looking at the great job that the reddit creators did with their recommendation engine, I'm sure they could figure out a kickass way to factor in personal weights with community approval.

Stack traces as coordinates

2006-10-18T23:38:00.003+02:00

I picked up on Scott Rosenberg's code reads a while ago by following a post on reddit, and have started cursively following it. I've skipped out on reading the first book, The Mythical Man Month but I do intend to read up on that at some point in the future. Anyway, the reading material for this week consists of no less then three of Edsger Dijkstra's articles / papers and I'm fully intent on keeping up with the several reads to come from now on.

So I'm reading through "Notes on structured programming" in which Dijkstra lays out the concept of a coordinate system to identify a discrete point in a computation process. Basically, this is what you want your stack trace to be, something blows up, and you want to know as much as possible about when and where exactly things went awry.

In the context of understanding programs he notes how a mathematical theory and a computer program are both structured, timeless objects, but while the mathematical theory makes sense on it's own, the program doesn't make sense until its execution. With this in mind, he describes a way to decompose programs in such a way that it becomes easier for humans to make sense out of them. For this purpose, he distinguishes three types of decomposition: concatenation, selection and repetition.

We have now seen three types of decomposition; we could call them "concatenation", "selection" and "repetition" respectively. The first two are understood by enumerative reasoning, the lost one by mathematical induction.
- Dijkstra, notes on structured programming.

Concatenation is used to decompose computations directly following each other, selection is a way of decomposing computations that are executed based on a condition and repetition is the decomposition of computations that are in a loop.
Now, with these 3 types of decomposition, it becomes easier to read and understand a program because once you decompose a number of computations, you can then mentally abstract it, and view it as a single step.

Once we have a properly decomposed program, we want to be able to make assertions about its computations, we want to know what value a certain variable has at a certain point in the program, during its execution. To be able to do this Dijkstra explains:

In short, we are looking for a co-ordinate system in terms of which the discrete points of computation progress can be identified,... We can state our problem in another way. Given a program in action and suppose that before completion of the computation the latter is stopped at one of the discrete points of progress. How can we identify the point of interruption, for instance if we want to redo the computation up to the very same point ?
- Dijkstra, notes on structured programming.

As far as the concatenation and selection decompositions are concerned, it is trivial to do this by using what Dijkstra calls the textual index of the program text, in other words, the line number. However, as we all know, due to the nature of loops, the same does not hold true for the repetition decomposition. Simply because a loop iterates over the same computations (with the same textual index) several times, the textual index is of no use to us for the purpose of indicating where we are in the computation progress. The solution ? Introduce a dynamic index, a variable independent from the computation that keeps track of where exactly in the repetition we are. OK so now we almost have a coordinate system as described above, there's just one more concept that isn't covered yet,... functions. When a language allows for functions, we need a way to represent the exact point of progress where our function is called. A textual index won't suffice, since it is lacking context, however, we can pass along the textual index of where our function was called.

Mix all the above together, and you get a perfect stack trace which would give you:

the line number of the call producing the error
if the error-producing call is part of a function, the line number of the call invoking this function
if the error-producing call is nested in any loops, the dynamic index indicating the specific iteration of those loops

Hmm, now I wonder why the loop index is never provided in any stack traces I've seen up to now. Is it a design decision to reduce overhead from tracking dynamic index variables, or am I missing something obvious?
Regardless, that's an interesting bit of history right there, and I'm only on page 29 :)