Hello Oslo, Hello Squarehead Technology

Posted on6 Comments

Two years after moving to Berlin and joining Openismus, it is time for another big change. I learned a lot while at Openismus, and had a lot of fun both in and outside of working hours. Those of you who have been to the barbecue parties know what a great bunch of people they are. I’m very glad to see that they are now going strong again.

In December I will join Squarehead Technology in Oslo as a Software Developer. There I will work on their advanced microphone array systems for acoustic cameras and acoustical zoom. The role includes both real-time programming, digital signal processing of audio/video and embedded Linux, which is pretty much exactly what I was looking for.

Here is a very quick demo of the technology, used for noise analysis: Nor 848 video

The array microphone system records 200+ channels of audio simultaneously. By exploiting the time difference between channels, digital signal processing can extract and/or visualize audio content at different positions in the recording. This can be done both in real time, and in retrospect (unlike parabolic microphones).

Fun times ahead!

Piksels and Lines – Libre Graphics Research Unit seminar in Bergen

Posted on2 Comments

Last week I was so lucky as to attend the 3rd Libre Graphics Research Unit (LGRU) meeting in beautiful Bergen, Norway.

The meeting was titled Piksels and Lines and had “a particular focus on improvements, interoperability between and fringe use of F/LOSS graphic bitmap and vector software, as well as generative software used in performative contexts.”

The meeting was structured into three different areas: Seminar, Workshop and Performance.

Seminar

The attendants that were invited for the meeting each did a presentation of their choosing. They were recorded and are available in the online archive of the seminar. The video quality leaves something to be desired, but the audio is generally good.

The presentations I found particularly interesting were:

I gave a presentation titled MyPaint and cross-application workflows. It was an introduction to MyPaint as a creative tool, how it combines raster and vector (piksels and lines) concepts, and my perspective on interoperability between libre graphics applications.

 

 

Workshop

I had hoped to hack some code for one of my existing ideas during the workshops. That did not happen. Instead I ended up hacking specifications. Maybe that is just as good. Hacking one can always do later, hashing out and documenting ideas has to be done while it is fresh.

First the results of some discussions with Øyvind Kolås, the GEGL maintainer:

A journal for GEGL: transaction log over changes made to a GEGL graph. Specification. Discussion. This feature would allow for applications based on GEGL to:

  • Implement non-linear histories (undo/redo), and a timeline of the changes
  • Store the history in a document like OpenRaster
  • Share the history between different applications
  • Let multiple applications to work on the same document at the same time

A strategy for improved file format support in GEGL, and using this to improve  file support and interoperability in libre graphics applications. Proposed plan.

Executing this plan would move a lot of the existing file format support from GIMP (PSD, XCF, OpenRaster) down into GEGL so that it can be reused across applications. And would then let GEGL provide image support plugins for GdkPixbuf and QImage – so that at the very least – previews will work everywhere.

 

Chatting with Egil Möller, creator of Sketchspace, also resulted in:

A web based system supporting a continuous work-flow from free-hand
sketch to finished product. Concept and mockups

“Imagine starting from a freehand drawing or imported raster image and gradually refining this into a technical document with illustrations, UML-diagrams or even running code or a 3d model.”

Refining here means that the user guides the tool to transform freehand sketch into vector paths, then into vector shapes, then into something domain-specific and formal like UML – by adding additional data like annotations, strengthening of lines to “disambiguating” the transformation.

Needless to say, this is more a visionary thing. Realizing this would involve finding good solutions to a fair amount of computer vision problems.

Making GEGL available for use in web-based applications. Proposal.

More on the concrete side: allow GEGL to be used in interactive or batch-oriented web applications, or in native applications based on web technologies (Javascript, HTML5 user interfaces).

Some of the discussions also resulted in me writing down the strategy for GEGL integration in MyPaint and the related ideas/plans for how to improve the performance of the MyPaint brush engine.

Now we just need to implement all the stuff… Contributions welcomed!

 

Performance

Since Piksel, with a long history in generative performance arts, was the hosting organization it was not surprising a project in that area materialized.

A workshop session hosted by media artist Brendan Howell called Demonstrating the Unexpected came up with the idea of the Piksels & Lines Orchestra (PLO): think of the collaborative use of our traditional libre graphics software as an orchestra. The applications, from MyPaint to Scribus, are instruments; the people using them players; a performance the use of these instruments. Can we create an experience for an audience based on this framework? How would it sound? How would it look?

Architecture diagram - no need for them to be dull looking.

Having plenty of code-crafting people available, the next afternoon it was decided to spend a couple of hours realizing a prototype. The LGRU blog has the details. We recorded video of our initial performances with this prototype as well, but that has sadly not made it online yet…

 

Thanks!

Thanks a lot to Piksel and LGRU for sponsoring my attendance, and the EU Culture Programme and Bergen municipality for funding activities that support libre graphics and free culture!

MyPaint and goats at LGM2012

Posted on6 Comments

Already covered in the news from LGM was the release of GIMP 2.8, and that GIMP 2.10 will be fully GEGLified. The goat-invasion branch which has most of that work, the result of 3 weeks of pippin and mitch on a couch hacking together, has already landed in master. This means that GIMP now has support for high bit-depth workflows for most operations. Finally.

Putting the goat in MyPaint

During LGM I started working on using GEGL in MyPaint. I have already mentioned this idea several times, so it was time to stop talking and get hacking.

As a first step in making use of GEGL I wanted to replace the current surface implementation with one based on GeglBuffer. Since GeglBuffer already provides tiling, and can store any buffer data supported by Babl this turned out to be easy. Øyvind (pippin) added the semi-quirky pixel format we currently use* in MyPaint to Babl, and I was able to get a rough working GEGL based Surface implementation the first evening.

The MyPaint brush engine working on top of GeglBuffer

* RGBA premultiplied alpha, in 16 bit unsigned integers with  2^15 being the maximum value.

The next couple of days went to moving to the GeglBufferIterator API instead of gegl_buffer_{get,set} to have zero-copy access to improve performance, and improving GEGL and GEGL-GTK so that some of the hacks in the initial implementation could be removed.

Most of the work is in the gegl branch of MyPaint. A simple test application, mypaint-gegl.py, is included, and you can read README.gegl for how to try it out. Warning: only intended for curious developers at this stage.

A lots of work remains to be done for MyPaint to be able to fully use GEGL. The progress is tracked in two bugs, one for MyPaint work and one for GEGL issues. Because one cannot combine PyGObject with PyGTK, it will likely not be possible to fully integrate GEGL in MyPaint before porting to PyGI and GTK+ 3.

Oh, in case the goat references are lost on you – check the GEGL page on wikipedia.

Application-hosted and compositor-hosted Maliit

Posted on7 Comments

The standard way of deploying Maliit is to have a single maliit-server instance (per user session), hosting the actual input method (virtual keyboard, handwriting). Applications then communicate with the server (and by extension, the IM) through an IPC.

This allows for a single instance of Maliit to serve all applications, which is memory efficient and robust. A crash in a Maliit IM plugin cannot take down the application and risk loss of significant user data. The disadvantage is the increased system complexity (a separate server process needs to be running at all times*) and requiring compositing of the application and input method windows. The latter can be quite challenging to do in a well-performing way on low-powered mobile/embedded devices. See Jans blogpost for how we handled that on the Nokia N9.

* By default we make use of DBus autostarting, of course.

Application-hosted Maliit

To make Maliit more suitable for systems where only a single application runs (embedded) or compositing performance is not good enough, we now also allow Maliit to be “application-hosted”: the Maliit server and input method plugins lives in the application process, not a separate server process. Enabling this feature has been a long running task of mine: All the code in input-context and server was made transport independent, a direct transport (no IPC) was introduced, and setting up the server for a given configuration (X11, QPA, app-hosted) was simplified. Other motivations for this work include being able to run the server and IM plugin easily for automated end-to-end system or acceptance testing, or just to easily start the server with a given IM plugin loaded for quick manual testing during development (see Michaels merge request).

An example application exists as part of the Maliit SDK that demonstrates the feature: maliit-exampleapp-embedded

Maliit running in application-hosted mode: The Maliit Server and input method plugin is embedded in the application instead of running in a standalone server process.

This works by having a special input-context “MaliitDirect” which instead of connecting to the server over DBus, creates the server and a direct connection. As when running standalone the server will instantiate and manage the necessary input method plug-ins.

Because the IM does not have its own window in this configuration, the application is responsible for retrieving the IM widget from the server, and re-parenting it into the appropriate place in the widget hierarchy. For all other purposes the application uses the same interface as if the IM was hosted remotely, making sure the abstraction is not broken and that one can easily use the application with Maliit deployed in different configuration.

This feature currently works with Qt4 applications, and is in Maliit since the latest release (0.90.0). One issue is that with the current input method API, the plugin assumes a fullscreen window; overlays extending the base area of the IM will be clipped and size needs to be overridden. This is something we are fixing in the new improved API.

Compositor-hosted Maliit

Another approach to make rendering perform better is to host the input method in the process responsible for the compositing. This also reduces the number or processes involved in rendering/compositing, and the associated overhead. This could be a X11 compositing window manager (like KWin or mcompositor), but a more realistic use-case is a Wayland compositor (for instance based on QtCompositor).

The API allows the consumer to inject an class instance for the configuration dependent logic, allowing to integrate the Maliit server with the logic in the rest of the compositor. Applications will use the normal “Maliit” inputcontext and communicate to the server through an IPC like DBus.

After the work with application-hosted Maliit, this feature was completed by making the server and connection libraries available as public API. The API is available in the latest  Maliit release (0.90.0), but is considered unstable until Maliit hits the 1.0 mark.

 

 

 

 

Maliit on Windows: Basic build working

Posted on4 Comments

Enabling third-party developers of input methods is one of the primary goals in the Maliit project. In an attempt to improve this story I spent some time on getting Maliit to work on Windows.

Since we use Qt there were few changes needed to the code, but since we use qmake, quite many to the build system. One of the bigger changes was making glib-dbus and qdbus  optional, which is also useful for Maliit on embedded systems.

With the Windows build fixes merge requests for maliit-framework and for maliit-plugins, one can now build Maliit on Windows and run the provided example applications. This feature is currently being reviewed and should be in the next Maliit release.

Thanks to the standalone viewer application for Maliit Keyboard this allows one to develop new features, theming and language layouts for it on Windows.

Sadly loading an input method plugin in the maliit server crashes for an unknown reason. With my limited Windows software development experience I was not able to solve this within the couple of days I had available. This is necessary for application-hosted Maliit to work and to enable general development of Maliit input method plugins (not just maliit-keyboard). Help would be much appreciated, even just someone checking  if it is reproducible on another Windows system.

Also left on the todo-list due to lack of time is to set up a Windows build slave for the Maliit buildbot, to test that the build continues to work on Windows and to produce executables.

Gitorious Merge Request Monitor

Posted on1 Comment

In Maliit, all changes have to be reviewed by two people in order to be merged to mainline. This helps us catch issues early and keep code quality high. Since the code is hosted on Gitorious, we use their merge requests feature for that purpose. Up until now we have periodically checked the website for changes (potentially going through each and every one of the repositories), and manually mentioned updates in the IRC channel. This is both tedious and inefficient, so I wrote a simple tool to help the issue: Gitorious Merge Request Monitor

It provides an IRC Bot which gives status updates on merge requests in an IRC channel:

16:01 < mrqbot-AfFa1> desertconsulting requested merge of ~desertconsulting/maliit/desertconsultings-maliit-framework with maliit-framework in http://gitorious.org/maliit/maliit-framework/merge_requests/126
16:01 < mrqbot-AfFa1> mikhas updated http://gitorious.org/maliit/maliit-framework/merge_requests/125  State changed from Go ahead and merge to Merged

One can also query the current status from it:

15:02 < jonnor> mrqbot-7ACeB: list
15:02 < mrqbot-7ACeB> maliit-framework/127: - New - Allow QML plugins to add custom import paths for QML files and QML modules
15:02 < mrqbot-7ACeB> maliit-framework/126: - Need info - configurable importPath for qml
15:02 < mrqbot-7ACeB> maliit-plugins/26: - New - Add PluginClose from main view and add key repetition support
15:02 < mrqbot-7ACeB> maliit-plugins/25: - New - Clear active keys and magnifier on keyboard change
15:02 < mrqbot-7ACeB> maliit-plugins/24: - New - Remove QtGui dependency from libmaliit-keyboard
15:02 < mrqbot-7ACeB> maliit-plugins/23: - New - Get rid of Qt keywords
15:02 < mrqbot-7ACeB> maliit-plugins/22: - New - Add phone number and number layout getters.

Status changes are retrieved by periodically checking the Gitorious project activity feed (Atom)*, and the status itself is scraped from the website. There is no other API right now, unfortunately. Implemented in Python with Twisted, feedparser and BeautifulSoup doing all of the heavy lifting.

Get it from PyPi, using easy_install or pip:
pip install gitorious-mrq-monitor
gitorious-mrq-monitor --help # For usage information

For now this solves the immediate need for the development work-flow we have in the Maliit project. Several ideas for extending the tool are mentioned in the TODO. Contributions welcomed!

The Maliit buildbot

Posted on6 Comments

After having set up the typical things open source projects needs like a website/wiki, mailing-list and bug-tracker, Maliit now also has something not so common: a build-bot.

As Maliit consists of several components that can be built in several different ways (and for several different platforms), we wanted to automate the build and tests of the different variations to ensure that we do not break any of them. This is especially important for variations which are not easy to test for the individual developers, like for instance Maliit on Qt 5.

The software chosen to help with this task was Buildbot. Getting an initial instance it up and running was very quick and pain-free, especially thanks to packages being easily available and the excellent documentation. The current setup now builds, tests and installs the two major components we have: Maliit Framework and Maliit (Reference) Plugins, in the most important build/config variations we have. A total of 12 individual build jobs, plus 2 meta-builds. The configuration for the instance can be found in the maliit-buildbot-configuration repository.

For security reasons the build-bot is not directly exposed to the Internet. Instead a script runs every 5 minutes to generate a static HTML website and publish on the public web-server: Maliit build-bot

Buildbot says: All green!

This gives us a minimal continuous integration system for Maliit, which for now will hopefully helps us avoid breakage. In the future, the usage of the build-bot might extend to include:

  • Automating the release process
  • Testing of merge-requests/patches before merging to master
  • Automated integration/system testing, complementing the unit-tests
  • Triggering external builders for packaging. OpenSUSE OBS, Maemo 5 Garage, etc.
  • Automating certain aspects of bug-lifetime. Resolving when fix is committed, closing on release if pre-verified, etc

 

GTK+ application support integrated into Maliit mainline

Posted on3 Comments

GTK+ application support for Maliit input methods has existed for a long time, but up until now it has lived in separate repositories. This has been inconvenient for users and for developers, and was the major cause for it to not be on the same level as the Qt support. This has changed as the GTK+ support has now been merged into the maliit-framework repository, and along side the Qt support. Maliit 0.80.8, which was released yesterday, contains these changes.

Maliit running on WeTab with Fedora 15, showing QML reference plugin and GTK+ application

Two implementations existed for Maliit GTK+ support. One was written by Javis Pedro as part of a Google Summer of Code project for MeeGo in 2010. His blog has several posts on the topic. The other implementation was maintained by Raymond Liu (Intel). This is the implementation shipped in Meego Netbook, and the one improved by Claudio Saavedra (Igalia) as part of the GTK+ on MeeGo project. It was also the only one that was updated to work with the DBus connection changes that was done quite some time ago, and supporting both GTK 2 and 3. For these reasons this was the implementation integrated into mainline Maliit.

Once the code was integrated, improvements soon followed. The application now correctly reconnects to server, and make install will automatically update the GTK+ input module cache on Ubuntu, thanks to Łukasz Zemczak (Canonical), and on Fedora. This means GTK+ application support will work out of the box, no twiddling needed.

While this is a huge step in the right direction, the GTK+ support is not as good as for Qt yet. Javis Pedros implementation has features that does not exist in mainline, so code/principles can hopefully be reused from there to implement these. This includes custom toolbars and attribute extensions, and content type hints for text entries. Other features looks hard to implement due to limitations/differences in the input context plugin architecture found in GTK+, and will probably need work in GTK+ itself to solve.

Silent Stream Of Godless Elegy – Návaz

Posted onLeave a comment

I have a strong preference for music in a language I can understand, if nothing else than for the opportunity to mumble along. But “Návaz”, the latest album from Czech band Silent Stream Of Godless Elegy, is a little gem, even if I can’t understand a word of the Czech lyrics.
Návaz, like the previous albums is a doom metal album with strong influences from Moravian folk music and lore. What I like most about this album are the heavy contrasts: Quick, playful violins against slow, drudging guitar riffs; soft female voice against harsh male voices; traditional folk instruments and musical themes against modern metal.

Here is the opening track, Mokoš, which I think is quite representative:

 

Those who don’t have the stomach for the above, thinking it is too noisy, might still enjoy the songs Sudice or Samodiva. More samples are available on their Myspace page. You can find the album at Amazon.

If you liked this music you should also consider checking out In Via (2010) by Illuminandi. Style wise it leans closer to gothic metal, more gloomy and bleak, but overall very much the same type of feel.