Some random thoughts about XMPP spaces

Posted by pulkomandy on Tue Aug 17 17:58:46 2021  •  Comments (0)  • 

You may or may not know I am involved in XMPP in a way or another for some time. Recently I have started work on Renga, an XMPP client for Haiku, and participated in an online meeting and discussion about why Discord is so succesful and what ideas XMPP could borrow from it. A part of the discussion revolved around the way Discord organizes multiple channels in a "server" and how that fits very well with their user base.

Today someone contacted me and shared a work in progress document about XMPP "spaces" which is an attempt to see how something similar could be made in XMPP. I was surprised to see the document dive straight into discussion about protocols and stuff like that, with the UI/UX part being "TODO write this later". I am not sure this is the right way to design the thing. I was asked my input on it, so here it is. I have not a lot of experience of the XMPP protocol, but as a user, I chat on various systems with various people and there are several instances where I can see an use for such things.

So, let's tackle this from the user experience point of view. I will completely ignore the "but how do we do that?" part and focus on what I think could work well. Let's start by going back to basics and define some use cases. Why do we want to do "Spaces" in the first place?

Use cases

Let's imagine that XMPP is a very popular protocol and everyone uses it. No other chat system exists anymore. Let's see what it did to become so succesful. I will take my own usage of various IM networks to see how this could look in that alternate (or future?) world.

Communicating with my family

My parents are doing ok with phones and computers, but still, let's keep things simple for them. A single chat channel and ability to send 1:1 messages will be enough. A media library archiving all the pictures and links we sent to each other could be nice. There will be almost no movement of users joining/leaving this space (maybe a new girlfriend/boyfriend joining the family or leaving after a breakup once every few years?), and everyone can be a moderator or it could be just one person.

I think that was the easy case which is already mostly covered by existing options

In the office

I am working remotely this year and in our company this meant reviewing and improving our chat system, which we now use a lot.

I work in a company that has in total about 800 employees, of which 150 in the local branch I work at. We are software engineers and developers. We work on many projects for different customers. Each team is typically 3 to 30 people (with sub-teams in the largest teams). We also have some people who need to do things for many teams at once (for example our sysadmins are taking care of services and tools deployed globally or specifically for a given team).

In our current chat system, we have one single space for the 150 persons in the local branch. Each project has one or more private channels, which are not listed anywhere in the UI. When people join a project, their account is invited to all the corresponding channels. This works quite nicely for us and there doesn't seem to be a reason to group channels together here.

What we would like to have, however, is a way to create temporary sub-channels for discussing specific issues. Something that would be similar to an e-mail thread. Slack and Zulip are examples of chat systems which allow this. Zulip is very close to the email way, having separate threads at the core of its UI. In Slack, it is done by picking a specific message in a channel and replying to it, which creates a sub-discussion. This would be great to organize our chats and more easily keep the info we need.

Other nice to have features are a way to search for old messages in a specific set of channels (but probably this doesn't need to have them formally tied together as a "space"), and a way to pin things (mainly URLs or file attachments) to be able to find them easily. I can imagine also more advanced features such as a shared calendar to place our meetings and days off work in.

Probably, larger companies will want a more segregated system, and I can imagine companies which have non-computer-inclined people (not software engineers) may need some more centralized admin roles to oversee who has access to what. So that probably means accounts tied to some LDAP server, not being able to list MUCs unless your account is added to the appropriate space, and not allowing people to leave a space on their own because they would be unable to re-join it.

Opensource projects

I contribute to a lot of projects, some large, some small.

I will not go on for very long about the small projects because the existing solution (just a single MUC) is just fine for me. So let's see about the larger ones which have a need to split their discussion into multiple aspects.

So, in this case, the main thing to think about is onboarding. If you don't care about onboarding, you will be just fine with a dozen independant channels which have no apparent relation to each other, except they are listed together on your website, and maybe they all have your project logo or some variation of it on them. If you care about onboarding, you want to make it easy for a newcomer to click on a single button/link and immediately join your Space and discover all the channels from inside there, in their favourite IM client.

You will probably want some kind of "backstage" channel for moderators to discuss ongoing issues. This should not be visible to regular users, of course. Which means multiple channels in a space can have different access rights. On the other hand, you may want to nominate moderators and automatically allow them to be moderators on all the channels. Speaking of moderation, you also want the ability to kick/ban someone from the whole space if they misbehave.

As an opensource project, you want to be transparent and have an archive of everything that was said and shared, possibly over the course of decades. This includes channels that are currently unused because the project was reorganized. Possibly you'll want to split the history of a space because one project was split into two separate parts. You may want to copy it to create a fork of the project while retaining the past history in both branches of the fork. And you may also want to merge the history from two projects together and form a single space, but probably I'm going a little crazy here.

You also want to preserve the privacy of your users. It should not be easily possible to identify that user A in space X is in fact the same person as user B in space Y, if they decided to use different nicknames in each place. On the other hand, you want to be really sure that if you talk to someone named "user A", it is really them, and not some other person using the same nickname.

Another aspect to think about here is notifications. For high traffic channels and projects, probably I won't read everything. I will have the chat client on my computer and read it when I have time or maybe if someone pings me. But I don't want this to ring my phone everytime something happens. It should be a distraction free thing that I can have running in the background. This mean I need easy configuration for which notifications I want on each of my clients. I think both for the whole space, and for specific channels (there may be some channels I have no interest at all in following, maybe because they are in languages I don't understand, maybe because the project is large and some topics are not interesting to me).

Chat with friends

One of my uses of IM currently is organizing board games sessions with friends (but whatever your hobby is, probably some of the same applies). Here, there isn't really a notion of a fixed "space". Some of my friends don't know each other or have met once during a board game afternoon and then never met again. Currently I have one rather large channel with a lot of people but I think I will just create and delete smaller groups as needed. In my case, a "space" is probably not useful here.

Gamer communities

I am a lot less familiar with this. I think a large part of the "opensource" section will also apply. Probably channels with restricted permissions (only a few people can talk) are needed. Also, some nice to have things: custom "stickers"/emojis specific to the server, ability to define and rename roles and assign them specific permissions, ... Just read the Discord documentation.

Chat with strangers

One place where IRC is still somewhat popular. There are chat services with various thematic or so channels (by age, location, or shared interests) all thrown together into a "space". People can join and talk with complete strangers. There are a lot of trolls and people with inappropriate behavior. Users of the service need an easy way to signal such things so a moderator can quickly intervene. If the space is big enough, there will be separate moderator staff for each channel, but probably still a common backstage channel for coordination

Thinking about the user interface

So, what do we need to put in our user interface? Here is an attempt to summarize it:

  • Single-button way to join a space
  • Ability to see a list of channels in a space you joined, with a description of what their purpose is
  • Media library with all pictures/links/? or pinned messages
  • Ability to see long term logs (multiple years) of all channels, including now inactive ones
  • Possibility for space moderators to archive a channel (only past logs available, no way to post new messages)
  • Manage permissions for a single channel and for the whole space (who can talk, who is a moderator, etc)
  • Ability to configure notifications, per-client, globally on a space and more specifically on each channel
  • Know who is joined in a space, ability to reliably ban people (in a way they can't avoid just by rejoining with a different nickname)
  • No way to identify that two users in two different spaces are in fact the same person
  • Multiple levels of administration: the owner of a space can nominate moderators for different channels, control which channels are visible to all users or to users with some specific role only, etc. Moderators can adjust some, but not all, settings of the channel they are moderating
  • Ability to join a space but only join some of the channels inside and not all

In terms of user interface, channels from the same space should of course be grouped together. There will probably be a LOT of channels so you probably won't get away with a single tree view, it will never fit everything on screen. Which means you need a first level with a list of all the spaces, showing which ones have ongoing activity. Then you can select one of the spaces and see the channels inside.

In the XMPP world, one thing to think about is how to handle things that are not in a space. Maybe they can just be put into a "default" space from the UI point of view?

If you know someone's real JID, and you start a chat with them from inside a MUC, it would be super annoying if that ended up being a separate chat history than if you contact them directly. Or maybe it's a feature to have separate discussions (let's say if you have a colleague and you talk work things, but they're also a friend and at other times you talk non-work things).

You will have some kind of management menu (maybe right click on the space icon/name) to decide if you want to leave a space, configure notifications, see who is a moderator or admin.

Quick notes on building gcc

Posted by pulkomandy on Sat Apr 1 13:44:05 2017  •  Comments (0)  • 

This may not be up to date anymore. A complete GCC for AVR (and AMR) is now available as HaikuPorts recipes, which provide a more complete process, with a C library and everything. Refer to these recipes if you need to do it (even on other platforms, the recipes aren't all that hard to read and adjust)

As you may have noticed, I like to develop stuff for all kind of weird devices. For this, I usually need a C compiler, and most of the time it's gcc (not always).

gcc is a big piece of software and there are some tricks needed to build it. Also, I run the Haiku operating system, which is quite nonstandard, so additional workaround are needed.

Today I built gcc for avr. Here are notes on how to do it so I don't spent a month figuring it out next time.

# gcc compilation for gcc 4.4.5 (4.5.x needs more stuff. maybe later)
# Made by PulkoMandy in 2010
# Before you start :
# * Download gmp and mpfr from HaikuPorts ( and extract to /boot
# * Download gcc-core-4.4.5.tar.bz2 from gcc mirror and extract to work directory
mkdir obj && cd obj # This is the output folder. So you can keep the source area clean
setgcc gcc4 # We can't compile gcc4 with gcc2.
../gcc-4.4.5/configure --target=avr --enable-languages=c --prefix=/boot/common/ --with-mpfr=/boot/common/
	# Tell the target, then language we want, and where to install the result. Binaries will be called avr-* so don't care about overwriting other ones.
	# For some obscure reason mpfr isn't detected properly, so we force the prefix.
make all-gcc ; make install-gcc # This does compile only gcc, not libgcc; which failed to work for me.

There are other things to watch out : I had to remove a -lm somewhere as Haiku doesn't have a separate libmath.

Next : build a PlayStation development toolchain, including gcc MIPS target.

Projects I'm NOT coding

Posted by pulkomandy on Sat Apr 1 13:41:22 2017  •  Comments (0)  • 

SometimesI have ideas about software that could be interesting to write or useful to use; but I'm already contributing to a lot of projetcts and I'd rather not start all of the new ones.

Following a talk on #haiku irc channel, I decided to put the list online so other people can pick these projects up and start working on them.

Please let me know if you made (part of) one of them. So I can link to you here :)

PulkoMandy's ever growing TODO-list

Older items first.

  • Port the znax flash game to the Atari Lynx console.
  • Create a device that plug on the amiga clockport and can serve as a base for complicated projects. A DSP to decode OGG would be nice.
  • Build a mouse adapter, similar to AmiPS/2, but using the AMXmouse protocol for the CPC.
  • Compile the SDL version of Road Fighter for the GP2X
  • Port Rick Dangerous2, Prince of Persia 1 and 2, The Lost Vikings 1 and 2, and Jazz JackRabbit 2 to SDL or another lib and make them run on the GP2X.
  • Maplegochi : an electronic Maple tree for the Haiku desktop. You feed it with some water and you watch it grow day after day. The tree is built with random fractals so everyone gets an unique tree on its desktop. It changes over the seasons (depending on the system date and the locale). It is a replicant living on the desktop and acts as a living background,without being too disturbing.
  • Make the various usb to serial chips work on haiku with a simple terminal program.
  • Code a ROMDOS D1 filesystem add-on for Haiku to read/write floppies for Amstrad/Schneider CPC computers.
  • Network-shared whiteboard application for Haiku, allowing people to draw diagrams and see each other drawings. Likely use Playground as a base.

Some projects

Posted by pulkomandy on Fri Nov 8 12:56:42 2013  •  Comments (0)  • 

Some time ago I set up a trac install on my server to put all my running stuff. As I ended up using the provided wiki for tech documents, there is nothing visible on this website. This article list all these projects so they are linked from somewhere, maybe this way Google will index them better.

Also, there's more on my github page, where I plan to migrate most of the stuff above, someday (because git is better, github is more visible, and Trac in fastCGI mode has a very annoying memory leak making it the first source of problems on my homeserver). There's also my Google Code Prject Hosting page, which I plan on not using anymore but there are some projects to migrate, still.

Amelie, a filesystem for 8-bit computers

Posted by pulkomandy on Sat Jun 8 23:23:19 2013  •  Comments (0)  • 

This is a project I've worked on and off for the past year. It all started with the MO5SD project by Daniel Coulom (French page). The idea of this is to use an SD card plugged to the tape port of the french Thomson MO5 computer, which happens to use TTL logic levels.

I reused the ideas from MO5SD to build a similar interface for the Amstrad CPC printer port. However, the Amstrad operating system has better capabilities than the MO5 one and is friendly to expansion ROMs. This makes it possible to run a full-blown filesystem rather than a simple bootloader like the MO5 version does.

Getting the SD card read/write code working was the easy part. After spending some time (with help from SyX and Cloudstrife) optimizing the z80 assembler code for SPI bit banging, I started looking for a suitable filesystem. The most common floppy disc ROMs for the CPC are AMSDOS (the one that shipped with the machine) and Parados, which improves support for dual side, 80 tracks floppy drives from PC. None of them easily allow to reroute disk access to anything else than the floppy controller. I heard that RODOS, a less known disk ROM, has such a capability, and that it also has directories, permissions, and some other nice features. However, close inspection of the RODOS ROM showed that there is actually no way to redirect disk access to anything else than the FDC (if you know a way...)

Moreover, the RODOS filesystem design looked like it would be slow. My bit-bang access to the SD card doesn't go very fast, and jumping around between several sectors isn't a good thing. I wanted to keep files without too much fragmentation so I could load them fast. RODOS is also limited to 20MB volumes, which sounded huge in the 1980s but felt ridiculous for my 4GB SD card. Finally, RODOS requires some RAM, and any ROM that does that on the CPC reduces compatibility with some software.

So, I decided to design my own filesystems. The goals are simple:

  • Use as few RAM as possible. AMSDOS allocates a 2K buffer, that should be the only space we are allowed to use (and some stack, of course).
  • Limit access to the storage medium whenever possible. Try to not read the same sector multiple times.
  • Allow the use of big drives. This requires directories, and also some other tricks.
  • Be z80-friendly. No 32-bit math is ever done.
I worked on a C++ version first. This allowed me to keep the code readable while I experimented with various things. I did all the testing on PC and will start converting the whole stuff to z80 code only when I'm fairly sure there won't be big changes to it again.

An SD card can hold up to 4GB of data. I decided to split this into 256 volumes that map to the CPC notion of "users". On an AMSDOS floppy, each file belongs to a single user and can't be seen by others. This limits each user to 16 megabytes of data. Since the files are allocated on 512-byte sectors, these can all be addressed with a 16-bit offset. I also limited the number of files and directories to fit them on a 16-bit counter.

The filesystem uses a block map with sector granularity (the 16 first sectors are used for that), and a file/directory list based on extents. When the filesystem is not fragmented, a directory will use just 16 bytes of space (including the 11 character name and up to 256 entries), and a file will use a 16 bytes header to store up to 128K of data. These entries can be extended, so directories have unlimited number of entries, and files have unlimited size.

You can read more about the data structures in the filesystem readme, and also check the source code. They are both available at CPCSDK. The C++ code makes use of some C++ features such as vector, but this could easily be converted into plain C. You will also find a WIP ROM code for the z80 version, mostly done by SyX. I'll start filling it with actual disk access code someday, for now it's just managing the patching of AMSDOS code and forwarding the access to floppy drives.

GuideML AmigaGuide converter for Haiku

Posted by pulkomandy on Tue May 28 18:22:36 2013  •  Comments (2)  • 

I'm currently porting some Amiga software (namely, the ACE CPC Emulator). As usual with Amiga software, the user documentations are written in AmigaGuide. I wanted to convert it to a more usual format for Haiku users. I found some tools, but they all run only on Amiga systems. Fortunately, one of them is written in C and open source.

You can get GuideML for Haiku sourcecode at my GitHub account.

The interesting part (development-wise) of this is that I used a set of wrapper header that convert the Amiga API calls into Haiku ones. The Amiga API is based on BCPL, which predates C and has differences such as using FPutC instead of fputc, it also has extra stuff such as support for lists, and a different way to allocate and free memory (where you have to tell FreeVec() the size of the block you're freeing).

This set of header allowed me to make very few changes to the core code of GuideML. I'm also using these headers for the port of ACE, where I could getthe emulator core running quite easily in a short time.

I'm still refining these headers to remove warnings, make them C++ safe (as my user interface for ACE is written in C++) and make them behave as close as possible to the original system. I even reused parts of AROS, an open source rewrite of the Amiga OS, which aims for source-level compatibility. Their ReadArgs implementation was not too hard to port to Haiku, and now my software can use just the same smart argument parsing as on Amiga. I still have to get something similar to icon tooltypes working, however, but that shouldn't be too hard as ReadArgs can parse strings from a file, instead of the command line.

IUP portable user interface

Posted by pulkomandy on Fri Mar 30 22:48:11 2012  •  Comments (5)  • 

Just a quick note to say I started a project with IUP as the framework for user interface. After spending some time with Qt and wxWidgets, I've finally found an UItoolkit that just does what's expected. No need for a precompiler, no replacement of my main function by some wrapper, no rewrite of the C++ STL.

IUP is written in C, but has a nice attribute-based interface that makes it very easy and pleasant to use. I've made good progress on building my windows and the layouting system is nice to work with (still fighting with Qt one...). IUP is cross platform as it uses either comctl32, GTK or Motif. I think I'll write an Haiku/BeAPI backend for it as it's going to be rather useful.

It's quite easy to install in MinGW, just get the prebuilt binaries (either dll or static linked) and includes and drop them at the right place. No need to recompile for hours like wxWidgets.

I still have to see how integration of custom widgets is possible. This gets useful quicker than one may think, as soon as you need an hex-editor, a music-tracker like interface, or something similar. But these seem to be handled by IUP with a generic "grid" control that looks quite flexible. In wxWidgets both of these were a mess, with no easy way to do a custom control like an hex spinbox, and a button grid giving very bad performance. Having to manually call freeze() and thaw() on widgets got boring really quickly. Not to mention the complete lack of threading support...

Let's see how it goes in a few weeks!

IUP with C++

While the documenation suggests a way of using IUP in C++ and encourages it, I was not so happy with using friend functions or static methods. So I came up with my own solution involving a bit of C++11 magic (variadic templates). The result is the IUP++ class that reigsters itself as an IUP callback (with any arguments) and forwards the call to a C++ object and method. It's used this way :

Callback<Gui>::create(menu_open, "ACTION", this, &Gui::doStuff);
Callback<Gui, int, int>::create(toggle, "ACTION", this, &Gui::doMoreStuff);

Where Gui is the class you want to answer to the event, menu_open and toggle are IUP handles to UI objects, "ACTION" is the callback name, this is the object to forward the event to (an instance of Gui), and doStuff and doMoreStuff are methods called. Notice the Callback also needs the parameters to these - that's the second "int" in the second example (the first one being the return type, that defaults to int if missing, but is needed when you add parameters). I'm looking for suggestions on how to make this simpler, as there is still some repetition in it...

Smarter vim filetype detection

Posted by pulkomandy on Tue Jan 3 20:46:05 2012  •  Comments (0)  • 

Vim is, as you may know, is my favorite editor for all development purposes. The syntax highlighting is powerful and extensible easily. Most of the time, the file type detection for this is based on file extensions. Works well, unless you have files named .src or .asm for assembly language on different CPUs...

Vim documentation only shows how to set the filetype guessing from the file extension. Here's an example of doing something a bit more smart.

The idea is to put the CPU name on the first line of the file (in a comment). Then use the powerful regexp match features of vim to detect it:

; vimfiles/ftdetect/z80.vim
au BufRead,BufNewFile *.z80	set filetype=z80
	; The usual way to do it for clear file extensions

func! s:detect()
	if getline(1) =~ z80
		set filetype=z80

au BufRead *.src	call s:detect()
au BufRead *.asm	call s:detect()
	; And the smart one. Note it is useless on BufNewFile,
	; as the file will not have the header yet.

Do a similar file for each of your CPUs.

Note, it should be possible to scan fr the use of particular mnemonics to go without the header, but that requires a bit more work to identify many CPUs. Any volunteer ?

The software archive

Posted by pulkomandy on Sat Jul 16 20:50:40 2011  •  Comments (0)  • 

This is the script that runs my BeOS software archive

This is a website presenting software, similar to aminet for the amiga or others repositories. It runs without a database and is meant to be easily open to external contributions through ftp uploading.

The full script is less than 200 lines of perl and features a category hierarchy, screenshots, and some other useful infos about the software.

As usual, it is distributed under the MIT Licence.

#!/bin/perl -w
use strict;
use CGI::Carp qw(fatalsToBrowser);
use URI::Escape;
use Time::HiRes qw(tv_interval gettimeofday);

my $t0 = [gettimeofday];

my @QUERY = split(/&/, $ENV{'QUERY_STRING'});
my %query;
foreach my $i (@QUERY)
    my $mydata;
    my $varname;
    ($varname, $mydata) = split(/=/, $i);
    $query{$varname} = $mydata ;
my $line;
read(STDIN, $line, $ENV{'CONTENT_LENGTH'});
@QUERY = split(/&/, $line);
foreach my $i (@QUERY)
    my $mydata;
    my $varname;
    ($varname, $mydata) = split(/=/, $i);
    $query{$varname} = $mydata ;

print "Content-Type: text/html\n\n";
print <<ENDHTML;
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "">
<html xmlns="">
<meta name="keywords" content="Haiku Software Archive BeOS" />
<meta name="description" content="BeOS software archive" />
<meta http-equiv="content-type" content="text/html; charset=iso-8859-15" />
<title>PulkoMandy's BeOS software archive</title>
<link href="style.css" rel="stylesheet" type="text/css" media="screen" />
<h1>PulkoMandy's BeOS software archive</h1>
<p>This is an archive of BeOS software. Unlike BeBits, the files are archived locally so if their origin gets lost, they'll still here safely.</p>

<p>Most of the files in this archive are from the emupt website, gathered by Xeon3D. Thanks to him for the great work.</p>
<p>You can access the archive directly if you don't like this web interface <a href="archive/">here</a>.</p>
<p>Most of the stuff from emupt is still unsorted, have a look <a href="">here</a>.</p>

<p>The main goal of this website is to try to get some of this software open sourced so it can be improved for future use.
Tracking of the original authors is sometimes difficult, but usually gives good results. They may also be happy to know the
BeOS world is still alive. This goal is the reason for the separation in three folders.</p>
<li>Nosource : unfortunately, these apps are closed source. We need to get in touch with the authors and ask them to release the code.</li>
<li>Source: these application are distributed with their source code, but they are not updated anymore. Take over the development of some of them !</li>
<li>Adopted : these apps are living their own life somewhere else.</li>

if ($query{file}) { # User wants to show details about a file
    my $file = $query{file};
    open(F, "$file.desc");
    my %data;
    my $description;
    while (<F>) {
        my $field;
        my $value;
        ($field, $value) = split(/:/,$_,2);
        if (length($value) > 0 && $field =~ /^[a-z]+$/) {
            $data{$field} = $value;
        } else {
            $description = $description . $_;
    print "
    <h2><a href=\"$file\">Download $data{name}</a></h2>
    <tr><th>Short description</th><td>$data{shortdesc}</td></tr>
    <tr><th>Author</th><td><a href=\"$data{url}\">$data{author}</a></td></tr>
    <img src=\"$data{screenshot}\" alt=\"screenshot\"/>";
    print "<p>$description</p>";

    print "<a href=\"/~beosarchive/\">Go up</a>";
} else { #User wants the full software list
    print "<h2>Full software list</h2>";

    sub loopDir {
        my $f;
        my($dir) = @_;
        opendir(DIR, "$dir");
        while ($f=readdir(DIR)) {
            next if ($f eq "." || $f eq "..");

            my $path = "$dir/$f";

            if (-d $path) {
                # We found a directory: recurse into it
                print "<li class=\"dir\">$f</li>\n";
                print("<ul class=\"dir\">");
            } elsif($path =~ /\.desc$/) {
                # we found a .desc file
                $f = substr($f, 0, -5);
                $path = "$dir/$f";
                print "<li class=\"file\"><a href=\"?file=$path\">$f</a></li>\n";
        print "</ul>";

    print "<div style=\"float:left\"><h3>Files without source</h3><ul>";
    print "</ul></div><div style=\"float:left\"><h3>Files with source looking \
        for a maintainer</h3>";
    print "</ul></div><div style=\"float:left\"><h3>Adopted projects</h3>";
    print "</ul></div>";

my $elapsed = tv_interval ( $t0 );
print "<p style=\"clear:both; width:100%; border-top: 1px solid #ECC;\">\
    Page generated in $elapsed seconds.</p></body>";

SVN to IRC commit bot

Posted by pulkomandy on Thu Jul 14 22:26:08 2011  •  Comments (0)  • 

CommitOMatic is an SVN post-commit hook that connects to IRC and tells the logmessage.

Put this file as "post-commit" in the hooks directory of a subversion repository, and set the settings as you need.

#!/usr/bin/perl -w
# see
# see
use strict;
# We will use a raw socket to connect to the IRC server.
use IO::Socket;

# The server to connect to and our details.
my $server = "";
my $nick = "Commit-O-Matic";
my $login = "pulkobot";
my $channel = "#commits";

my $repos = $ARGV[0];
my $rev = $ARGV[1];
my $commit = `/usr/bin/svnlook log $repos -r$rev`;
my $user = `/usr/bin/svnlook author $repos -r$rev`;
chomp $user;

# Connect to the IRC server.
my $sock = new IO::Socket::INET(PeerAddr => $server,
                                PeerPort => 6667,
                                Proto => 'tcp') or
                                die "Can't connect\n";
# Log on to the server.
print $sock "NICK $nick\r\n";
print $sock "USER $login 8 * :Commit-O-Matic Robot\r\n";

# Read lines from the server until it tells us we have connected.
while (my $input = <$sock>) {
    # Check the numerical responses from the server.
    if ($input =~ /004/) {
        last; # exit the loop
    elsif ($input =~ /433/) {
        die "Nickname is already in use.";

# We are now logged in : join a channel
print $sock "JOIN $channel\r\n";

# Now wait for the "end of name list" message
while (my $input = <$sock>) {
    # Check the numerical responses from the server.
    if ($input =~ /366/) {
        last; # exit the loop
    elsif ($input =~ /433/) {
        die "Nickname is already in use.";

# We are now logged in.
my $cmd = "PRIVMSG $channel :";
$repos =~ s#/home/subversion/##;
print $sock "$cmd$repos: $user * r$rev\r\n";
chomp $commit; #svnlook add an extra newline.
chomp $commit; #svnlook add an extra newline.
my $com = $commit;
$com =~ s/\n/\n$cmd/g;
$com =~ s/^/$cmd/g;
print $sock $com;
print $sock "\n";

# Get out of it
print $sock "QUIT bye... \r\n";

Opensource your abandonware

Posted by pulkomandy on Tue Nov 30 22:27:26 2010  •  Comments (3)  • 

As you may know, back in 2007 I ressurected the development of GrafX2. this old pixelart program, made only for DOS, was left out 6 years earlier by the authors, that had moved on to more modern computers. Today, graFX2 is amongst the best tool for pixelling, particularly on Linux or other alternative operating systems. Many people are using it daily to draw really nice pictures. The newer versions added a lot of features such as layers, and there's more to come.

This was possible only because the authors decided to release the source when the project stopped. The code wasn't perfectly clean; it was tied to ms-dos with some optimized parts written directly in assembly language and accessing the video card hardware directly. Of course, getting an SDL-based version out of it was not easy. But still, it took considerably less time than rewriting everything from scratch. Also, part of the userbase for the old GrafX2 upgraded to the new one. For some of them it felt like getting back home after years of using suboptimal tools.

During the revival of GrafX2, I had to develop my web searching skills a lot. First, the original GrafX2 website was offline, and the sourcecode was gone with it. Thanks to filewatcher, an ftpsearch engine, and the web archive, I was able to locate a copy on some russian FTP. Then, I wanted to get in touch with the authors to let them know their software finally found some use.

But grafX2 isn't the main purpose of this article. Last month, I downloaded APlayer, a music player for BeOS. After some hacking to get it working on Haiku (which eventually led to uncovering and fixing a compatibility bug), I noticed that most musics from Burned Sounds, my preferred chiptune collection, didn't load. The strange thing is that most of them were in formats supposed to be recognized by APlayer. But looking closer, it turned out they are packed using the Shrink algorithm. This is a packing system from Amiga days, which can be unpacked only on amiga for lack of any sourcecode or format information. Well, that was until yesterday. Using my high web searching skills, I found the author of Shrink and kindly asked him by mail if he was willing to release the sourcecode for hissoftware, the last version being from 1996.

He was a bit surprised to see there was some files using Shrink still around, but he had a linux version of the archiver. This version is now released as GPL sourcecode at sourceforge. This is an important step for me in getting more open source software available ; but also in preserving old files packed in this format. I hope some other people will find it useful too.

I forgot to mention I also made possible the release of a whole lot of other BeOS software made by Arvid and Jonas Norberg. This include the sawteeth sound synthetizer, as well as the backslash n demo, and also some other unfinished code.

The overall message is, for developpers : think about opensourcing your old projects. Even if the source is not as clean as it could be ; even if they are of no use for you ; even if they only work on a dead since 10 years operating system : someone, somewhere, may find it useful. You can visit the Unamintained Free Software page to get some examples of how passing on a project to someone else may work. But not everything goes through this website.

For non-developpers, don't hesitate to get in touch with the devs, even for unmaintained apps, and ask for an open source release. If the software is dead, the author isn't going to get anymoney for it, so why not release it so other people can improve it ? This is the fastest way to get more open source software. And don't be shy, developpers are, above all, normal people, and they do like hearing from users.

Static linking nightmares

Posted by pulkomandy on Mon May 31 11:25:58 2010  •  Comments (0)  • 

I recently ported Reloaded, an Amstrad CPC emulator, to Windows. This turned out to be more complicated than expected, and I encoutered problems for which I can't find any proper solution on the internet, so I decided to tell you how I solved them.

Step one : SDL and wxWidgets

Reloaded is a very special project. It started its life as a fork of the Caprice32 emulator. Caprice was an emulator designed for Windows, but later ported to Linux using SDL to render the screen. SDL didn't provide support for anything else than pixels, and we wanted complex windows, so we decided to use wxWidgets. After some hacks, we got that running : SDL was used for sound and timers, and wxWidgets for display rendering.

It worked quite fine on Linux, but when trying to run it on Windows, I encountered a problem : both SDL and wxWidgets are trying to define the WinMain function to do some special inits and call the application's main. Of course, having two WinMains didn't please my compiler.

wxWidgets offers a nice way to solve that : their WinMain is defined in the macro IMPLEMENT_APP, which is easy to replace with something else if you want so (you have to do somme inits by yourself, but that's ok). SDL, however, doesn't allow you to do so : as soon as you #include SDL.h, it is not possible anymore to define WinMain yourself!

Reloaded is now using Portaudio for sound and doesn't rely on SDL anymore. I also had to disable OpenGL as our code used SDL_Surfaces, and rewrite some timer handling to use native windows functions.

Step 2 : getting it to link

Another particularity of Reloaded is the way its built : we wanted to create a platform-independant core, and a gui part that wraps around it and provide the windows and graphical stuff. This allows for really easy porting : you have very little things to alter in the core and only mess with rewriting the GUI part.

This was counting without autotools limitations : to build these files properly, we had to use different defines on each side. The core will have to look for portaudio includes while the gui will want wxWidgets. We solved that by building them into two separate .a static libraries, then linking these libs to a single executable.

Again, this was working well on Linux, but Windows strangely failed to link the thing, giving undefined references to portaudio and some other DLLs we are using. Also, I was getting an "undefined reference to main" for no apparent reason (as there was a WinMain function in my program). After some searching, I found I was supposed to add -lmingw32 _before_ my .a in g++ command line, or else the runtime loader wouldn't find WinMain. But doing so would result in undefined reference to every possible function in nwxWidgets.

After a full week of trial and error, I finally managed to get it working : you have to link -lmingw32 first, then your static libs (in the right order so they can link to each other), and finally link wxWidgets and other stuff you need.

I don't know why the gcc tools want the static libs to be in the right order while dynamic one will not care, I think that's not very practical and even annoying. But I hope this article will help you solve the same kind of problem, shouldyou ever encounter it.

Atari Lynx development under Linux

Posted by pulkomandy on Sun Jul 5 01:33:33 2009  •  Comments (0)  • 

Bon alors, j'ai une Lynx 2 qui traine dans mon bazar et j'ai décidé d'essayer de programmer un peu dessus. Le hardware a l'air plutôt sympa et assez costaud pour permettre de développer des trucs rapidement. Sprites zoomables et déplaçables ligne par ligne permettant de faire des polygones texturés, son 4 canaux à base de générateurs polynomiaux, et quelques autres gadgets sympa. Donc voilà une catégorie de mon site dédiée à mes aventures avec cette console.


cc65 est un compilateur pour le 6502 qui est le processeur de la Lynx. Il est fourni avec des bibliothèques de base permettant de faire quelques trucs. Ces bibliothèques sont portables sur plusieurs plateformes (c64, NES, ...) mais avec pas mal de limites. Elles sont loin d'exploiter les capacités de la console à fond. Je vais donc probablement écrire mon propre toolkit pour gérer tout ça. D'autre part cc65 a l'air d'être assez limité niveau optimisation du C, il faut tout faire à la main pour avoir du code efficace, aussi je pense que je vais rapidement devoir apprendre l'assembleur 6502 pour passer aux choses sérieuses. Cela dit, je parle déjà l'ARM et le z80 donc la transition devrait se faire sans trop de mal

CC65 n'est pas disponible dans les paquets Debian, pour des raisons de problèmes de licence. Le code est trop ancien pour avoir été mis sous GPL... Il faut donc faire l'installation à la main. Hereusement, ça se passe plutôt bien.

wget # téléchargement de la dernière version de cc65
tar xvf cc65-sources-2.12.0.tar.bz2 # on décompresse...
cd cc65-2.12.0                      # on va voir ce qu'il y a dans le dossier
make -f make/gcc.mak                # on lance la compilation
sudo make -f make/gcc.mak install   # on installe!

Une seule petite subtilité : il faut ajuster deux variables d'environnement pour dire à cc65 où il est installé. J'ai mis ça dans mon fichier .bashrc pour régler le problème une fois pour toutes.

# Configuration de schemins d'accès importants pour cc65
export CC65_INC=/usr/local/lib/cc65/include 
export CC65_LIB=/usr/local/lib/cc65/lib

Squelette de projet

Bon, voilà, j'ai maintenant un compilateur qui fonctionne. Pour le tester (et pour s'en servir plus tard), on a besoin du package d'exemple de Karri. Ce package contient un projet complet avec makefile et tout le bazar pour générer un fichier .lnx. C'est un programme simple qui fait une petite application de dessin. Ça servira de base à mes prochains projets mais je vais surement faire quelques modifications dans le makefile. Bref, pour le moment, je lance make, et ça me compile un fichier .lnx sans le moindre problème. Cool.

Émulateur: Handy SDL

Pour l'instant je n'ai pas de cartouche ni de câble BLL pour ma Lynx. J'ai donc besoin d'un émulateur pour tester mon bazar. J'ai essayé mednafen qui est dans les paquets Debian, mais il s'est lamentablement planté avec une segfault au premier lancement. Poubelle, donc. J'ai téléchargé le code source de Handy SDL. Petit problème avec cette archive, les dossiers n'ont pas les droits en exécution ce qui empêche la compilation de fonctionner. Un chmod +x -R sur le dossier extrait de l'archive règle le problème. Ensuite, ça compile tout seul. Pour utiliser l'émulateur il faut aussi un dump de la ROM interne de la Lynx. Celui qu'on trouve chez planetemu a l'air très bien. Il semblerait que vous n'ayez pas le droit d'avoir ce fichier si vous ne disposez pas d'une vraie Lynx. Je suis pas spécialiste en droit, renseignez vous.

Pour que Handy marche bien chez moi, je dois mettre le bpp à 16, sinon l'image ne s'affiche pas correctement. À part ça tout se passe bien. J'ai aussi l'impression que la musique de ma cartouche de test (faite avec ABC) ne fonctionne pas. Mais de toutes façons, ABC n'a pas l'air très simple à utiliser donc mon premier objectif pour la Lynx sera de coder un vrai tracker capable d'exploiter à fond les capacités de la machine, qui ont l'air plutôt sympathiques.

La suite...

Prochaines étapes pour quand j'aurais le temps :

  • Faire une cartouche et un cable BLL pour tester des trucs en vrai sur la Lynx,
  • Coder un tracker capable d'exploiter les capacités de la console à fond,
  • Coder un jeu ou une démo sur la console. Commencer simple et puis faire des trucs plus compliqués après,
  • Vendre les jeux programmés (bon là on rêve hein, pas avant 2020 au moins...),
  • Tenter de conquérir le monde !