Stefan’s Blog

IDFTags

2015-11-28T11:08:33+00:00

I developed a gem called idftags. The purpose of this gem is to extract significant words of a string based on the tfidf algorithm. The algorithm checks for every word (or so called term) of a document the term frequency and computes based on a collection of documents the inverse document frequency. Based on this to values we calculate a tfidf value, which represents the significance of a term.

This sounds a little abstract hence I want to show some results based on real life scenarios.

Challenges

The idea of this gem was born with the rails rumble. We created a rails app challenge me, where you can create challenges and blog about your progress beating it. Technically spoken a challenge is represented by a model, which looks like the following:

class Challenge < ActiveRecord::Base

validates :title, :presence => true, :length => {:maximum => 140}
validates :description, :presence => true

# some other code ...

end

To make things easier in terms of searching we wanted to add tags to a challenge, so that a user can filter challenges by tags. And to make things even easier we wanted to generate the tags based on the challenge title, taking all the challenge titles into account. With idftags the code would look like the following:

require 'idftags'

document = challenge.title
documents = Challenge.all.map(&:title)

idftags = IDFTags::IDFTags.new 
tags = idftags.tags(document, documents, 3

At the time of writing this we have 16 challenges yielding the following titles

[
  "Participate in RailsRumble", 
  "LOOSE WEIGHT", 
  "TRAVELING AROUND THE WORLD", 
  "Win the soccer world championship", 
  "test", 
  "Sleep 8 hours", 
  "Fun in the kitchen", 
  "I want to save money", 
  "I want to be awesome", 
  "Demo challenge", 
  "Sport", 
  "I want to beat cancer", 
  "100 days (public) streak on Github", 
  "Daily UI", 
  "Learn to solve the Rubik's cube blindfolded", 
  "Learn Clojure and release a Application"
]

And to test idftags I created tags for the following document

'I want to create something amazing and get famous'

By evaluating the tags with different weights I get the most common tags, which were

famouts
get
amazing

Not bad if we consider that we only have 16 documents. The algorithm works better when having more and larger documents to match a term against. This leads to our next case study.

Stackoverflow

I checked out the 6 latest questions on stackoverflow for the tag "Ruby on Rails" and extracted both the title and the original comment of the author as document base. I do not post them here but I can add them as download if you insist ;)

Then I took the latest question, which was

"Can I use api controller for other controller in Ruby on Rails"

and generated tags with idftags. Again by trying out several weights I got the 5 most significant words with

api
use
controller
rails
ruby

Again not bad for only 6 larger documents. Note that I used for both scenarios an appropriate bad word lexicon to filter common words (like 'a', 'is', 'to' and so on).

Summary

idftags works like intended and you may now have a starting point when using this gem. Although sometimes there are some undesired words found as tag (like 'use' in the stackoverflow example). But still most of the time it generates useful output and usually you do not create tags from it and store them away but you use them as suggestions for the user, so that he can accept or decline them.

This brings us to the roadmap. Accepting and declining tags found by the algorithm can help idftag to produce more useful tags in the future by applying several learning algorithms. This is not implemented not even planned yet but I really want to add it in the future. So maybe there will be some updates going on rather soon ;)

For more information checkout the official repo and leave some feedback if you want to support it.

Rails rumble 2015

2015-11-14T16:01:32+00:00

Hi @all,

EDIT: We ranked 9th in the rails rumble. Not bad for first timers, I guess... I crunched some numbers and will post them soon.

those who do not follow my twitter account (you certainly should :P) maybe missed that I participated in the 2015 rails rumble contest. The rails rumble is a 48 long contest for developers to show everybody what they are capable of and who is the best. The best entry gets determined by the votes of so called judges, which are basically the contestants themselves. This can also be read here. At the point of writing this the results were not out yet, so I will edit this probably once they are open to the public.

The intention of this post is to tell about our project and to share my experience. First of all I did not participate as a solo entry but as a team entry consisting of 4 members namely Sebastian and Philip Stapefeldt, Ralph-Gordon Paul and myself. We all work together with appcom interactive and thus know each other very well. We also have a pretty disjunct skill set meaning that we complement each other pretty well. While Sebastian is a very talented designer Philip is a not less talented front end developer experienced in many javascript frameworks. Gordon was more or less the administrative guy and also writing parts of the backend. The rest of the backend was written by me, which gave us the opportunity to share only a minimum of code and thus hold the continuity of the flow high.

The idea we realized can be described in two words challenge me. Everyone in this world is confronted with challenges day in day out. For the most of us these challenges are implicit and not noticeable like "Keep my flat tidy" or "Save up money to buy [...]", while other challenges are more prominent and meaningful like "I want to beat cancer" or "I want to loose weight and eat healthier". However as recent studies showed challenges are more likely to completed if you do them together. Feedback by others generate a synergy effect helping us to hang in, when dealing with challenges get out of hand. Also viewing other people undergoing the same struggles you do keeps you sane and gives you a feel of not being all alone. This is where challenge me comes into play.

On challenge me you can create challenges and/or participate in challenges created by others. Once you do so you create a so called "story", where you can tell your progress in completing the challenge. Story entries are displayed as blog or diary entries and can be read by others. Other viewers can like your story or comment on entries giving you feedback and thus helping you to pull through. When following other stories you get notified when there is something going on (like a new story entry). Please note that the current version of challenge me is not very stable since the whole code stems from a 48 hour hackaton. So if you find a bug once in a while do not hesitate to tell us.

However we are planning to turn challenge me into an actual working version. Also we're going to add more features rather soon to add more functionality. As this project progresses I will add more blog posts to keep you updated. Maybe this is a good time to follow also railsrave. This as a twitter account purely created for this contest.

We added a raspicam, which took a snapshot every hour, which was uploaded to twitter automatically. Additionally the cam took a snapshot every 2 minutes, which was condensed into a timelapse, which also turned out pretty great. All in all the whole rails rumble experience was worth having. We are looking forward to participate again next year competing against the best of the best. Maybe also against you?

Simple, fast and powerful - PyQt

2014-09-16T14:28:16+00:00

Hi there! Over the past several months I really enjoyed the perks of dynamic programming languages like Ruby and Python, to name the most outstanding ones. They are easy to use with a REPL, encourage you to cover your software with tests, what you always should do and you get stuff done pretty quickly.

But to quote a famous (now Disney) villian - "I find your lack of GUI (builders) disturbing". I always found it hard to develop a GUI or use available GUI frameworks both for python and ruby. So when I needed a GUI for a tool I had to develop I usually fell back to C++ and Qt, which is the most advanced GUI system in my opinion (especially with the QtBuilder). So I always wondered if there is possibility to use the strength and experience of Qts GUI concept and the andvantages of dynamic programming languages such as python. And in deed there is, called PyQt.

PyQt is a binding for Qt written in python. It is available for python 3.4 and 2.7, which are the most frequently used versions. So I tried it today and I'm so stunned, that I'd like to share it with you right now.

For starters: I'm using Windows at home, which makes the installation of PyQt very easy, since they provide a one click installer. So I cannot say anything about the effort of installation on other operating systems. Anyway as I mentioned, they have a one click installer, which copies all the files you'll need to your computer. With only 30mb of size I was a bit sceptical but in the end it was all fine.

Afterwards I googled for a tutorial in order to get started. I stumbled over that one, which covers the basics. So following this tutorial was fairly easy. I started with building the GUI with the QtBuilder (the strength) and set up the signals and slots, which are used later on (experience). After that I had to run a command line tool provided by PyQt. This was the first dangerous spot. Windows and command line tools - without setting the path manuallyin beforehand? But it worked like a charm. It transformed my *.ui file into python code, which I have to import in order to use my form. Then in my main python file I used the following code:

import sys

from PyQt4 import QtCore, QtGui
from form_ui import Ui_Form # this imports my GUI from form_ui.py

class MyForm(QtGui.QMainWindow):
	def __init__(self, parent=None):
		QtGui.QWidget.__init__(self, parent)
		self.ui = Ui_Form()
		self.ui.setupUi(self)
		

if __name__ == "__main__":
	app = QtGui.QApplication(sys.argv)
	myapp = MyForm()
	myapp.show()
	sys.exit(app.exec_())

After executing this small piece of code my form popped up, which surprised my a little bit, since this works too well. And also the predefined signals and slots worked as intended. And this whole procedure took me about 10 minutes, including reading the article mentioned above.

There you go. The strength and experience of a reliable GUI framework (I don't want to reduce Qt only to its GUI, but that's what I'm mainly using) combined with the perks (easy to develop, interactive, everything at runtime (!) and many more) of a dynamic programming language. And to tap it all off: It works on windows.

For everybody who didn't give it a shot by now, I highly recommend it to you. Happy coding.

Langton’s Ant

2014-05-31T17:55:17+00:00

Hi there. It's been a while since I posted something. As far as I remember my last post is more than 2 years ago and was about game developing with C++ and Irrlicht. And that is almost the amount of time I didn't do anything regarding that topic :D

I took a deeper look into the secrets of theoretical computer science. To be more exact: graph theory (which is also the topic my master thesis), cellular automatons and some fun stuff like game of life evaluation or generic map creation. One of the more interesting issues was without a doubt Langton's Ant. So I will make this the main topic of my latest blog entry.

For those who never heard of Langton's ant I will give a short description (for a more detailed version consider this article).

Langton's ant is basicly a 2 dimensional cellular automaton, operated from an ant (as you might guess from the name). This ant follows some rules, which are defined in before hand. A cycle is a sequence of actions, which are

The ant takes a step depending on its facing direction
The ant analyzes the color of the tile on which the ant is currently standing
Depeding on the color the ant changes its facing direction and the color of the tile

This cycle may go on forever or stop at a certain point (depending on what you want to do with the ant). The rules that we mentioned earlier are a list of tuples, which describe how the facing direction of the ant and the color of the tile will change. Take a look at the example:

{ 
"start": "white", "direction": "left", "end": "red" 
}, 
{ 
"start": "red", "direction": "right", "end": "white" 
}

Whenever the ant is analyzing a white tile, it will turn 90° left and will change the tile to red. On the other side, whenever it is located on a red tile, it'll change the tile to a white one and will turn 90° right. All not visited tiles (especially the first one) are considered as white.

So depending on the rules we provide, we can achieve some interesting behaviour of the automaton. There are rules, which lead to a non repeating behaviour (at least we do not know yet if it somehow contains a repeating pattern), where as some rules lead to a quite repetitive pattern. The most common recurring pattern is a highway (consider the pictures below for an example of a highway).

You may wonder what the big idea behind this concept is. First of all it's very fun to code and to watch as well. An implementation in python takes less than 200 lines of code, where at least 30% deals only with drawing and window creation stuff. Secondly searching for patterns or determing if there are loops in a certain ruleset is also interesting as well and may have some practical use cases.

Imagine a level editor, which bases on langton's ant. Each color is identified with a specific texture, which is mapped on the tile later on. Depending on the rule set, we could create basic layouts (e.g. highways ;)) or some ambient second layer sets (dirt, grass, decoration, ...). For this we would need a proper ruleset though. However we could use a layout produced by a ruleset and tune it manually afterwards. It takes more in-depth research to decide whether langton's ant could (automatically) produce decent tilemaps.

We also can extend langton's ant in many ways. First of all we could use multiple ants running parallely. The next we could change for more fun is the probability of changing the facing and color (e.g. the ant analyzing a red tile will turn left by 20% of the cases and turn right in 80% of the cases). However this would make searching for loops more complicated, since we would have to deal with multi states. Another easy to evaluate modification would be adding more directions (e.g. stay, turn backwards). Combining all these extensions langton's ant could produce complex structures.

As you can see langton's ant is a pretty interesting thing, which is widely unterrated. I attended 3 practical courses (in my currenty university), where I learned functional and dynamically typed programming languages (namely Haskell, Clojure and Python). In all of them I had to implement Conway's game of life, which is also another interesting cellular automaton. But I wish I had to implement langton's ant in at least one of these courses instead of writing another game of life clone. So please spread the word. Let langton's ant live and be a more valuable part in our lifes!

The new german Irrlicht board

2011-07-15T09:47:57+00:00

After a few weeks of hesitating we finally did it. Since the official german Irrlicht community was raided by bots (that is actually pretty annoying...) we founded a new community, cause we noticed that there is still need for it.

I'd like to introduce: Irrlicht-engine.co.de It's still not finished after all, since there is no real ending for a community. Some things are always in progress, so if you'd like to do your part, we're glad if you'd do so ;) Create an account and be part of it. See you there!

Neue deutsche Irrlicht Community

2011-07-15T09:41:18+00:00

Nach langem hin und her haben wirs dann doch gewagt. Da das offizielle deutsche Irrlichtboard schon seit fast einem Jahr konstant durch Bots garaided wird aber dennoch der Bedarf nach einer deutschen Irrlicht-Community da zu sein scheint, geht ein neues Board ins rennen.

Wir dürfen vorstellen: Irrlicht-engine.co.de Es ist noch nicht perfekt (wirds auch nie sein), aber eine Community wird ja besser in dem sie wächst, da hier stetes Angebot-Nachfrage-Prinzip herrscht. Wer also aktiv teilnehmen und verbessern will sei herzlichst eingeladen mit zu wirken ;)

Wir sehen uns da!

Irrlicht - From Noob to Pro

2011-03-21T15:10:20+00:00

Ein Hallo an alle deutschen Irrlichtnutzer (da es leider noch keine Übersetzungs ins Englische gibt). Vor einiger Zeit gabs doch dieses ziemlich coole Wikibuch Irrlicht - From Noob to Pro, welches von OnkelTorty und Close1 geschrieben wurde. Jedenfalls hatten die ziemlich gute Artikel geschrieben und dann irgendwann, brach das ganze ab, vermutlich aufgrund von Zeitmangel.

Seit März 2011 haben sich aber zwei neue Autoren gefunden, die das Buch weiterschrieben wollen. Heißt im Klartext: neues Cover, neues Layout und neuer Inhalt. Derzeit sind 3 große Kapitel geplant. Das erste beschäftigt sich mit Einsteigersachen, z.B. wie man Irrlicht einrichtet, ein "Hello World"-Programm und solche Dinge eben.

Teil zwei beschäftigt sich mit Irrlicht in Bezug auf die Spieleprogrammierung. Wie optimiere ich meine Dateizugriffe, wenn ich einen TexturManager verwende oder wie kann ich Anwendungseinstellungen speichern und laden?

Teil drei soll Highlight werden. Es soll eine komplette Tutorialreihe über die Entwicklung eines mehr oder weniger einfachen Spiels entstehen. Vom Projektbeginn, bis hin zu abschließenden Dingen wie das Modding soll der Leser an die Hand genommen werden (mit downloadbarem Quellcode). Derzeit ist noch nicht sicher, welches Spiel umgesetzt werden soll, also (nicht nur deswegen) bleibt gespannt auf die nächsten Veröffentlichungen dieses Buches:

Irrlicht - From Noob to Pro

2011-03-20T15:02:47+00:00

Hey there GERMAN Irrlicht user (since there is no translation into english yet). Long time ago there was that cool Wikibook called Irrlicht - From Noob to Pro written by OnkelTorty and Close1. After a bunch of good and informative articles, they had less time and the book was rather closed.

But since march 2011 two new authors were found. That means new cover, new structure and new content. The book will contain 3 main chapter. The first one (First steps using Irrlicht) will deal with the starting stuff, like setting up the IDE for Irrlicht, writing a Hello World Program and stuff like that.

The second chapter (Gameprogramming with Irrlicht) will give some more advanced topics like how you can improve your application by using a Texture Manager or how you can store and load Application Settings, which might be pretty usefull.

And the third chapter will be the eye catcher. It describes the whole way of writing a game in Irrlicht. The authors aren't sure by now which game it will be, so stay tuned for any new article of this Book:

Sneak Preview TownDefender [UPDATED]

2010-12-28T13:25:52+00:00

Currently I'm working on a project. Till now nothing is post about it here. And it won't gonna change the next days, so that I can focus on working on the project. Anyway here is a little preview ...

This is a sample, where the game takes place. It was created with a map editor, which creates (at least by now) orthogonal, isometric and hexagonal maps at any dimension. The player can use the map editor too, so that anybody can create maps like this above. And this is only the peak, so stay tuned for the next sneak preview ;)

Updated 29.12.2010

This is another arkwort:

It shows a simple watchtower firing off its arrows. Besides are some information about that tower like strength or price.

Anleitung: Torpia Combat Analyzor

2010-12-21T20:38:36+00:00

Endlich wurde der Torpia Combat Analyzor (zumindest in einer ersten Version) fertig gestellt. Enthalten sind ein Editor zum Pflegen der Kampfdaten, welche als XML abgelegt werden und der Analysator, der die Daten anhand der vorher geschriebenen Formel abgleicht und grafisch aufbereitet.

Dieses Tool soll helfen hinter das Kampfsystem von Torpia zu kommen. Dazu werden eine große Anzahl an Kampfberichten gesammelt und eine Formel entwickelt. Der Analysator gleicht die berechneten Daten mit den ermitellten Daten ab und gibt einen Umriss wie nah man sich an der Lösung befindet.

Download

Die aktuelle Version ist in unserem Downloadmanager zufinden. Zum Download stehen derzeit eine statische und eine dynamische gebaute Variante bereit. Ab Windows Vista kann es zu Side-by-Side-Konfigurationsfehlern kommen aufgrund von Schwierigkeiten die richtige Dll zu laden. Sollte es zu diesen Fehlern kommen sollte man sich die statische Variante herunterladen, in der alle Dlls (statisch) in die Exe einkompiliert wurden.

Inhalt

Kampfdatenbank

Die Kampfdatenbank ist relativ selbsterklärend. Man hat eine Eingabemaske vor sich, von der wir glauben, dass diejenigen Daten für die Brechnung der Verluste eines Kampfes relevant sind. Man hat die Möglichkeit weitere Verteidiger hinzuzufügen oder diese auch wieder zu löschen. Sollten alle Einheiten eins Spielers gefallen sein, so kann man die entsprechende Checkbox anklicken und die Verluste automatisch anpassen lassen. Hat man alle Daten korrekt eingetragen, so speichert man diese ab und trägt den nächsten Kampfbericht ein.

Sind irgendwelche Daten nicht bekannt so sind diese mit "-1" zu bezeichnen. Sollten also Berichte mit unvollständigen Informationen vorliegen, so können diese gesondert behandelt werden um nicht die Daten zu verfälschen.

Aus diversen Gründen wurden die Grafiken durch Bezeichners ausgetauscht, da diese nicht unbedingt für sich selbst sprechen, folgt hier eine Legende:

Axe - Axtkämpfer
Hand - Bogenschütze
Bow - Armbrustschütze
Swor - Schwertkämpfer
Pike - Lanzenkrieger
Hob - Lanzenreiter
Knig - Ritter
Ram - Rammböcke
Man - Mangonele
Treb - Trebuchet
Wall - Mauer
Towe - Turm

Analysator

Zuerst wählt man sich die entsprechende Formel aus (Wie man eine eigene Formel erstellt sieht man hier!), woraufhin alle Daten mit der ausgewhälten Formel ermittelt werden.

Im oberen Bereich sieht man die globale Abweichung in allen Fällen. So bekommt man einen Grobüberblick, wie stark die Unterschiede zwischen bestehenden und berechneten Daten sind. Entsteht eine sehr flache Kurve mit wenigen Schwankungen sind die Abweichungen sher gering. Zeigt die Kurve ein schwankendes und sehr ausuferndes Verhalten scheint man sich wohl auf dem Holzweg zu befinden. Abbildung 2 zeigt eine Formel, die relativ ungenau berechnet.

Man wählt sich nun einen speziellen Fall aus und untersucht die Daten für den einzelenen Fall um ein genaueres Bild von der Sache zu bekommen. Entweder man hovert über der Kurve oder klickt auf den gewünschten Fall. Rechts unten sieht man eine Übersicht der ermittelten und tatsächlichen Werte. Dabei gilt folgende Notation:

Blau: Tatsächliche Verluste des Angreifers
Hellgrün: Berechnete Verluste des Angreifers
Orange: Tatsächliche Verluste der Verteidiger
Türkis: Berechnete Verluste der Verteidiger

Links daneben sieht man die Daten grafisch aufbereitet - für die, die eher der visuelle Typ sind ;) Man kann dabei noch entweder Angreifer oder Verteidiger ausblenden um nicht durcheinander zugeraten.

Feedback

Für Fehler, Kritik, Lob, Verbesserungsvorschläge etc... ist immer Platz. Gern gesehen natürlich hier im Forum, als Kommentar zum Download oder an stefan[at]rpden[dot]net. Bei Fehlern bitte eine aussagekräftige Beschreibung und Version der Anwendung (Versionsnummer und dynamisch/statisch).