When I announced the first release of the imgflo project in April, it was perhaps difficult to see what exactly it was useful for and why we are developing it. This has changed now as 3 weeks ago we launched The Grid, our AI-based web publishing platform. We are on a bold mission to have “websites build themselves”; because until posting to personal websites becomes easier and more rewarding than posting to social media, content on the web will continue to pile up in closed silos.

To help solve this problem we built several open source technologies:

NoFlo: for creating highly testable, component-based, distributed software.
Flowhub: for visually and interactively building programs and extensions.
GSS: for building constraint-based, responsive layouts
And of course imgflo: for on-demand server-side image processing.

In total over 100k lines of code, and around 5000 commits over the last 12 months. Some of the stack is expained in more detail in a recent interview with Libre Graphics World.

imgflo on The Grid

thegrid.io launch site is of course built with The Grid. In the particular layout filter used, the look & feel is driven largely by the content. Colors for text captions are extracted from tweets and social media posts, and the featured images are largely unfiltered. Other Grid layout filters may style all provided content, including images, towards a uniform look specified by a color scheme. Or a layout filter may mix-and-match content- versus style-driven design.

The background texture on this section was created with imgflo, by passing the featured image through a blur graph:

https://imgflo.herokuapp.com/graph/vahj1ThiexotieMo/1ff47cef6f354fe0fbdefb…Fimages%2Fgrid-chrome.jpg&width=1300&height=768&std-dev-x=25&std-dev-y=25

It is important to note that no-one chose this exact image to be used in the particular layout section (and thus have the given image filter applied), which is why processing happens on-demand. The layout section with image inside a computer screen is available for content which has images of type “screenshot”. This property may be automatically detected by our image analytics pipeline, or manually annotated by user. The system allows describing many other such constraints, which are all taken into account when it works to create the appropriate layout for given content.

Even without considering styling, imgflo has a couple of important roles on a Grid site. Important is the ability create multiple scaled down versions to optimize download size. For this we also created a helper library called RIG, which is used to generate a set of CSS media-queries with imgflo request urls.

> rig = require 'rig-up'
> css = rig content, serverconfig, 'passthrough', parameters, ... 
 # passthrough is name of the graph to process through
 
@media (max-width: 503px) {
  .media, .background {
    background-image: url('https://imgflo.herokuapp.com/graph/apikey/6bb56129dc707894baa88d10a02a12b9/passthrough?input=https%3A%2F%2Fa.com%2Fb.png&width=400&height=225');
  }
}
@media (min-width: 504px) and (max-width: 1007px) {
  .media, .background {
    background-image: url('https://imgflo.herokuapp.com/graph/apikey/d099f7222293d335a6192d742f523bfa/passthrough?input=https%3A%2F%2Fa.com%2Fb.png&width=800&height=450');
  }
}

Processing images through imgflo also means that they are cached. So if the original image becomes unavailable, the website still has versions it can use. This can happen for instance on Twitter when people change their profile picture.
Note that while we optimize images when presented on site, we don’t touch the original image (non-destructive). This means image uploaded to The Grid has the full data & metadata preserved, unlike on some other social/web services. However, we are currently not preserving metadata in processed images.

imgflo 0.2

imgflo is now split into three repositories, the GEGL-based Flowhub runtime, the HTTP API server and the native dependencies. The runtime itself is plain C with glib, and could be used in non-web applications for desktop, mobile or embedded.

A major feature is that processing requests can now be authenticated, so that non-legitimate users cannot disrupt legitimate ones by overloading the server. We also use Amazon S3 for caching processed images, offloading a large portion of the work. Servicing 10k++ visits a day with a 2-dyno Heroko app has been no problem with this setup.

In imgflo-server we’ve also added support for using different processors than imgflo (which uses GEGL), in particular NoFlo with noflo-canvas. One can now build and deploy image processing pipelines using JavaScript, including all the libraries that work with the <canvas> element.

Building NoFlo image processing graph in Flowhub, then requesting from imgflo-server

Full details about the changes can be found in the changelogs: server, runtime.

Scale

Flowhub provides imgflo a node-based visual & interactive IDE for developing new image filters for The Grid. It is similar to etablished tools like FilterForge, the Blender compositor,  vvvv and nuke – which many designers and visual artists are familiar with. However there are still many snags in the workflow for non-technical people. Smoothing out these is major part of the next imgflo milestone.
After that the focus will be on horizontal scalability, to handle the load as The Grid enters beta and opens to founding members in spring.