Tag: C

Live programming IoT systems with MsgFlo+Flowhub

Posted on4 Comments

Last weekend at FOSDEM I presented in the Internet of Things (IoT) devroom,
showing how one can use MsgFlo with Flowhub to visually live-program devices that talk MQTT.

If the video does not work, try the alternatives here. See also full presentation notes, incl example code.

Background

Since announcing MsgFlo in 2015, it has mostly been used to build scalable backend systems (“cloud”), using AMQP and RabbitMQ. At The Grid we’ve been processing hundred thousands of jobs each week, so that usecase is pretty well tested by now.

However, MsgFlo was designed from the beginning to support multiple messaging systems (including MQTT), as well as other kinds of distributed systems – like a networks of embedded devices working together (one aspect of “IoT”). And in MsgFlo 0.10 this is starting to work pretty nicely.

Visual system architecture

Typical MQTT devices have the topic names hidden in code. Any documentation is typically kept in sync (or not…) by hand.
MsgFlo lets you represent your devices and services as FBP/dataflow “components”, and a system as a connected graph of component instances. Each device periodically sends a discovery message to the broker. This message describing the role name, as well as what ports exists (including the MQTT topic name). This leads to a system architecture which can be visualized easily:

Imaginary solution to a typically Norwegian problem: Avoiding your waterpipes freezing in the winter.

Rewire live system

In most MQTT devices, output is sent directly to the input of another device, by using the same MQTT topic name. This hardcodes the system functionality, reducing encapsulation and reusability.
MsgFlo each device *should* send output and receive inports on topic namespaced to the device.
Connections between devices are handled on the layer above, by the broker/router binding different topics together. With Flowhub, one can change these connections while the system is running.

Change program values on the fly

Changing a parameter or configuration of an embedded device usually requires changing the code and flashing it. This means recompiling and usually being connected to the device over USB. This makes the iteration cycle pretty slow and tedious.
In MsgFlo, devices can (and should!) expose their parameters on MQTT and declare them as inports.
Then they can be changed in Flowhub, the device instantly reflecting the new values.

Great for exploratory coding; quickly trying out different values to find the right one.
Examples are when tweaking animations or game mechanics, as it is near impossible to know up front what feels right.

Add component as adapters

MsgFlo encourages devices to be fairly stupid, focused on a single generally-useful task like providing sensor data, or a way to cause actions in the real world. This lets us define “applications” without touching the individual devices, and adapt the behavior of the system over time.

Imagine we have a device which periodically sends current temperature, as a floating-point number in Celcius unit. And a display device which can display text (for instance a small OLED). To show current temperature, we could wire them directly:

Our display would show something like “22.3333333”. Not very friendly, how does one know what this number means?

Better add a component to do some formatting.

Adding a Python component

Component formatting incoming temperature number to a friendly text string

And then insert it before the display. This will create a new process, and route the data through it.

Our display would now show “Temperature: 22.3 C”

Over time maybe the system grows further

Added another sensor, output now like “Inside 22.2 C Outside: -5.5 C”.

Getting started with MsgFlo

If you have existing “things” that support MQTT, you can start using MsgFlo by either:
1) Modifying the code to also send the discovery message.
2) Use the msgflo-foreign-participant tool to provide discovery without code changes

If you have new things, using one of the MsgFlo libraries is a quick way to support MQTT and MsgFlo. Right now there are libraries for Python, C++11, Node.js, NoFlo and Arduino.

imgflo 0.2, The Grid launched

Posted on1 Comment

When I announced the first release of the imgflo project in April, it was perhaps difficult to see what exactly it was useful for and why we are developing it. This has changed now as 3 weeks ago we launched The Grid, our AI-based web publishing platform. We are on a bold mission to have “websites build themselves”; because until posting to personal websites becomes easier and more rewarding than posting to social media, content on the web will continue to pile up in closed silos.

thegrid-5k-join

To help solve this problem we built several open source technologies:

NoFlo: for creating highly testable, component-based, distributed software.
Flowhub: for visually and interactively building programs and extensions.
GSS: for building constraint-based, responsive layouts
And of course imgflo: for on-demand server-side image processing.

In total over 100k lines of code, and around 5000 commits over the last 12 months. Some of the stack is expained in more detail in a recent interview with Libre Graphics World.

imgflo on The Grid

thegrid.io launch site is of course built with The Grid. In the particular layout filter used, the look & feel is driven largely by the content. Colors for text captions are extracted from tweets and social media posts, and the featured images are largely unfiltered. Other Grid layout filters may style all provided content, including images, towards a uniform look specified by a color scheme. Or a layout filter may mix-and-match content- versus style-driven design.

The background texture on this section was created with imgflo, by passing the featured image through a blur graph:

thegridio-imgflo-bg-texture
https://imgflo.herokuapp.com/graph/vahj1ThiexotieMo/1ff47cef6f354fe0fbdefb…Fimages%2Fgrid-chrome.jpg&width=1300&height=768&std-dev-x=25&std-dev-y=25

It is important to note that no-one chose this exact image to be used in the particular layout section (and thus have the given image filter applied), which is why processing happens on-demand. The layout section with image inside a computer screen is available for content which has images of type “screenshot”. This property may be automatically detected by our image analytics pipeline, or manually annotated by user. The system allows describing many other such constraints, which are all taken into account when it works to create the appropriate layout for given content.

Even without considering styling, imgflo has a couple of important roles on a Grid site. Important is the ability create multiple scaled down versions to optimize download size. For this we also created a helper library called RIG, which is used to generate a set of CSS media-queries with imgflo request urls.


> rig = require 'rig-up'
> css = rig content, serverconfig, 'passthrough', parameters, ... 
 # passthrough is name of the graph to process through
 
@media (max-width: 503px) {
  .media, .background {
    background-image: url('https://imgflo.herokuapp.com/graph/apikey/6bb56129dc707894baa88d10a02a12b9/passthrough?input=https%3A%2F%2Fa.com%2Fb.png&width=400&height=225');
  }
}
@media (min-width: 504px) and (max-width: 1007px) {
  .media, .background {
    background-image: url('https://imgflo.herokuapp.com/graph/apikey/d099f7222293d335a6192d742f523bfa/passthrough?input=https%3A%2F%2Fa.com%2Fb.png&width=800&height=450');
  }
}

Processing images through imgflo also means that they are cached. So if the original image becomes unavailable, the website still has versions it can use. This can happen for instance on Twitter when people change their profile picture.
Note that while we optimize images when presented on site, we don’t touch the original image (non-destructive). This means image uploaded to The Grid has the full data & metadata preserved, unlike on some other social/web services. However, we are currently not preserving metadata in processed images.

 

imgflo 0.2

imgflo is now split into three repositories, the GEGL-based Flowhub runtime, the HTTP API server and the native dependencies. The runtime itself is plain C with glib, and could be used in non-web applications for desktop, mobile or embedded.

A major feature is that processing requests can now be authenticated, so that non-legitimate users cannot disrupt legitimate ones by overloading the server. We also use Amazon S3 for caching processed images, offloading a large portion of the work. Servicing 10k++ visits a day with a 2-dyno Heroko app has been no problem with this setup.

In imgflo-server we’ve also added support for using different processors than imgflo (which uses GEGL), in particular NoFlo with noflo-canvas. One can now build and deploy image processing pipelines using JavaScript, including all the libraries that work with the <canvas> element.

Building NoFlo image processing graph in Flowhub, then requesting from imgflo-server

Building NoFlo image processing graph in Flowhub, then requesting from imgflo-server

Full details about the changes can be found in the changelogs: server, runtime.

Scale

Flowhub provides imgflo a node-based visual & interactive IDE for developing new image filters for The Grid. It is similar to etablished tools like FilterForge, the Blender compositor,  vvvv and nuke – which many designers and visual artists are familiar with. However there are still many snags in the workflow for non-technical people. Smoothing out these is major part of the next imgflo milestone.
After that the focus will be on horizontal scalability, to handle the load as The Grid enters beta and opens to founding members in spring.

imgflo 0.1: An image processing server and Flowhub runtime

Posted on2 Comments

At TheGrid we are in need for a flexible service for doing server-side image processing. So after some discussion at LGM in Leipzig, I started writing one based on GEGL, the image processing library that will power the upcoming GIMP 2.10 release. The library provides a demand-driven graph-based API , with a ton of operations included and support for GPU processing using OpenCL.

Runtime

For creating image processing pipelines for the server, imgflo acts as a runtime for Flowhub, our open source visual programming IDE. It adds to the existing NoFlo browser, Node.js and MicroFlo microcontroller runtime targets, all possible thanks to the runtime-agnostic protocol.

The preview is live and changes whenever changes are made to the graph. This allows to quickly experiment and develop new graphs.

Server

As a server, imgflo provides a simple HTTP API where you specify the input image as an URL, the graph to process it through and any parameters exposed on that graph.

Requests being made to the imgflo server HTTP API on demo page

Processed images are cached, so that subsequent request on the same url just returns the image out of the cache.
The git repository includes configuration and build setup for Heroku, so deploying an instance is a 5 minute job.

Next

imgflo 0.1 is now minimally useful as image processing server, but there are many more enhancements on the todo list. Scalability and integration with NoFlo are two big topics, as as expanding the pool of available operations and graphs. Porting filters from GIMP to GEGL is a way of helping with the latter.

Additionally it would be interesting to provide a way of using imgflo with GIMP. First of all one could use graphs made with Flowhub via GEGL meta-operations. A more crazy idea is to integrate directly,  using Flowhub as a companion node-based editor.

 

 

MicroFlo 0.2.0, visual Arduino programming

Posted on2 Comments

Two months after MicroFlo 0.1.0, another important milestone has been reached. This release brings a basic visual programming environment and initial support for all major desktop platforms (Win/OSX/Linux). The project is still very much experimental, but it is now starting to demonstrate potential advantages over traditional Arduino programming.

Official release notes and announcement here.

The start of something visual

The “Hello World” adopted from Arduino, a program that blinks the built-in LED a couple of times per second. Pressing Play (>) uploads the program to the Arduino using MicroFlo.

The IDE shown is NoFlo UI, a visual programming environment which can also be used to program JavaScript for the browser and Node.js using the NoFlo runtime. This project is developed by Henri Bergius and rest of the NoFlo team. For more details about the NoFlo IDE project, check their latest update and follow their Kickstarter project.

Talk

At Piksel 2013 in Bergen, I also presented MicroFlo for the first time, to an audience of mostly new media and experimental sound artists. The talk goes into detail about the motivations behind the project, from the quite practical to the more philosophical considerations. Not my most coherent talk, but it gives some insight.

Link

Next

For the next milestone, MicroFlo 0.3, several things are already planned. Focus is mostly on practical improvements to the system, but I also hope to complete prototype support for “heterogeneous FBP”: Allowing to program systems consisting of both host computer and microcontroller programs in a unified manner using NoFlo+MicroFlo.

I am also planning a MicroFlo workshop at Bitraf some time in December and to demo the project at Maker Faire Oslo.

In the meantime, you can get started with MicroFlo for Arduino by following this tutorial. Feedback and contributions welcomed!

MicroFlo 0.1.0, and an Arduino powered fridge

Posted on7 Comments

Lately I’ve been playing with microcontrollers again; Atmel AVRs with and without Arduino boards. I’ve make a couple of tiny projects myself, helped an artist friend do interactive works and helped to integrated a microcontroller it in an embedded product at work. With Arduino, one does not have to worry about interrupts, registers and custom hardware programmers to get things done using a microcontroller. This has opened the door for many more people that pre-Arduino. But the Arduino language is just a collection of C++ classes and functions, users are still left with telling the microcontroller how to do things; “first do this, then this, then this…”.

I think always having to work on such a a low level limits what people make with Arduino, both in who’s able to use it and what current users are able to achive. So, I created a new experimental project: MicroFlo. It has a couple of goals, the two first being the most important:

People should not need to understand text-based, C style programming to be able to program microcontrollers. But those that do know it should be able to use that knowledge, and be able to mix-and-match it with higher-level paradims within a single program.

It should be possible to verify correctness of a microcontroller program in an automated way, and ideally in a hardware-independent manner.

Inspired by NoFlo, and designed for integration with it, MicroFlo implements Flow-based programming (FBP). In FBP, a program is constructed by connecting a set of independent components. Each component has in-ports and out-ports, and can only communicate with eachother through these. The connections can be defined using programatically, using a declarative text language,  or using a visual editor. 2D/3D artists will recognise this the concept from node compositors like in Blender, sound artists from applications like Reaktor.

Current status: A fridge

I have an old used fridge, by the looks of it made in the GDR some time before I was born. Not long after I got it, the thermostat broke and the cooler would not turn off. Instead of throwing it away and getting a new one, which would be the cool and practical* thing to do, I decided to fix it. Using an Arduino and MicroFlo.
* especially considering that it is several months since it broke…

A fridge is a simple system, something that should be simple for hobbyists to create. So it was a decent first usecase to test the framework on. Principially, such a system looks something like this:

 

The thermostat decides whether to turn the cooler on or off, and the cooler switch realizes this decision. There are many alternative methods of implemening each of these two components. I used a DS1820 digital thermometer IC to read temperature, and a hacked NEXA remote controlled relay for the switch.
All the logic, including temperature threshold is done in software on an Arduino Uno.

The code below for the cooler switch would have been simpler (a oneliner, left as excersise for the reader) if I instead had used a active high relay directly on the mains (illegal if not a certified electrician). Or alternatively reverse-engineered the 433Mhz protocol used.

 

MicroFlo code for the fridge, in the .FBP domain specific language (examples/fridge.fbp)
# Thermostat
timer(Timer) OUT -> TRIGGER thermometer(ReadDallasTemperature)
thermometer() OUT -> IN hysteresis(HysteresisLatch)

# On/Off switch
hysteresis() OUT -> IN switch(BreakBeforeMake)
switch() OUT1 -> IN ia(InvertBoolean) OUT -> IN turnOn(DigitalWrite)
switch() OUT2 -> IN ic(InvertBoolean) OUT -> IN turnOff(DigitalWrite)
# Feedback cycle to switch required for syncronizing break-before-make logic
turnOn() OUT -> IN ib(InvertBoolean) OUT -> MONITOR1 switch()
turnOff() OUT -> IN id(InvertBoolean) OUT -> MONITOR2 switch()

# Config
‘5000’ -> INTERVAL timer() # milliseconds
‘2’ -> LOWTHRESHOLD hysteresis() # Celcius
‘5’ -> HIGHTHRESHOLD hysteresis() # Celcius
‘[“0x28”, “0xAF”, “0x1C”, “0xB2”, “0x04”, “0x00”, “0x00”, “0x33”]’ -> ADDRESS thermometer()
board(ArduinoUno) PIN9 -> PIN thermometer()
board() PIN12 -> PIN turnOff()
board() PIN11 -> PIN turnOn()

Is the above solution nicer than using the Arduino IDE and writing in C++? At the moment maybe not significantly so. But it does prove that this kind of high-level dynamic programming model is feasible to implement also on devices with 2kB RAM and 32kB program memory. And it is a starting point for more interesting exploration.

Next steps

I will continue to experiment with using MicroFlo for new projects, to develop more components and test/validate the architecture and programming model. I also need to read through all of the canonical book on FBP by J. Paul Morrison.

Some bigger things that I want to add include:

  • Ability to introspect the graph running on the device, in particular the packets moving between components.
  • Automated testing (of the framework, individual components and application graphs)  using  JavaScript BDD test frameworks like Mocha or Vows.
  • Ability to change graphs at runtime,  and then persist it to EEPROM so it will be loaded on next reset.

And eventually: Allowing to manipulate and monitor running graphs visually, using the NoFlo development environment. See bug #1.

Curious still? Check out the code, and ask on the FBP mailing list if you have any questions!

Improved drawing performance in MyPaint brush engine

Posted on7 Comments

A first set of performance improvements for the brush engine has just landed in MyPaint master. The goals for this work for me were, in priority: a) Making sure that moving to a GEGL backend in MyPaint does not reduce performance, b) Improve performance when integrating the MyPaint brush engine in other applications, and lastly c) Improving the performance in MyPaint itself.

TL;DR: * Users of the soon-to-be-released MyPaint 1.1 should experience about 15% faster drawing of strokes for medium to big brushes. * Switching to the GEGL based backed for MyPaint 1.2 is now both feasible and highly desirable from a performance perspective.


Optimizations

The optimizations are implemented through three complimentary strategies:

1. Deferred data access to minimize fetching and updating of tiles

All dab drawing operations that happen as a result of a motion update event are queued up. When the brush engine has calculated where all dabs should go, tiles are fetched, all dabs drawn and the tiles updated. This in contrast to before where each dab drawing operation would fetch and update tiles.

2. Coarse grained parallelism using multi-threading via OpenMP directives

The tiles to be processed are divided evenly between processing threads (one per core). Each tile is processed completely independent of other tiles, so there is no locking or synchronization in the drawing code. The tile backing store must naturally be thread-safe and may ensure this using locks.

3. Fine grained parallelism using SSE via GCC auto-vectorization

Within each tile we attempt to make use of auto-vectorization to create the brush dab mask and do the composition of the dab onto the tile. Currently this is only implemented for a part of the mask calculation.

Results

Details of the results and how they can be reproduced is found in the original email thread.

Gains for MyPaint 1.1

Starting with the lowest priority goal, but the most relevant to users; performance impact on MyPaint right now.

Surface drawing results for existing Python-based backend

In terms of raw speed of drawing brushes to onto the underlying surface, speedups range from 20% to 50% for larger brushes (16 px+). This sets an upper boundary for the speedup perceived by the user.

Looking at the UI-enabled benchmarks of MyPaint, which is doing everything a normal application instance does, including layer compositing and rendering to screen, around 15% speedup was observed.  As the UI benchmark only tests a single brush at size=8.0px, it is possible that larger brushes will a higher speedup.

Users of the soon-to-be-released MyPaint 1.1 should experience about 15% faster drawing of strokes for medium to big brushes.

Note that the backend in use does not make use of the multi-threading introduced by (2) due to the tile store not being thread-safe and that it already had a cache to mitigate the problem fixed by (1).
Note

GEGL-based backend results, outlook for MyPaint 1.2

Surface drawing results, GEGL versus Python-based surface.
Test results by Till Hartmann on his Phenom II X6.

So in terms of raw surface rendering speed, the GEGL based backend is now significantly faster than the Python-based one. With 1 and 2 threads it is respectively up to 25% and 100% faster for big brush sizes. With 6 threads, it can be up to 4 times faster.

Switching to the GEGL based backed for MyPaint 1.2 is now both feasible and highly desirable from a performance perspective.

Note that to see UI performance increases approaching the raw surface drawing performance increase we may also need to do the layer compositing multi-threaded.

Gains in other applications

I’m trying to convince the Krita guys to update to the new version and to provide some feedback on the impact. Other consumers of the MyPaint brush engine do not tend to communicate much with us (some are proprietary).
I have strong hopes that (1) should increase their performance radically as their tile get/set cost is significantly higher than in the MyPaint case: They need to convert between the internal Krita and the MyPaint brush engine working colorspace each time. They may also be able to enable multi-threading and see speedups similar to the GEGL-based backend as a result.

Future Work

This is only lays the groundwork of better optimized MyPaint brush engine, many areas have room for improvement. For one only a small subset of the heavy code is vectorized. There may be inner loops that can be tweaked. It may be that, with a different tile access pattern compared to before, a different tile size would be more ideal. Perhaps doing the expensive calculation of the brush dab could be avoided some times by caching them… Thinking bigger, one could move all the drawing (and rendering) to the GPU.

More details on these ideas can be found here. If you are interested in working on any of it, get in touch and start hacking!