libvips/whatsnew-8.5.md

This branch has a new implementation of sequential mode:

https://github.com/jcupitt/libvips/tree/remove-seq-stalling

It'd be great to get this merged to master for 8.5, but it needs some testing.

How seq used to work:

• The vips sink functions create a set of N threads and scan images top-to-bottom in tiles, allocating tiles to workers as they finish.

• They have extra logic to keep workers together. They track the position of the most-delayed worker and if the lead thread gets more than M scanlines ahead, it stalls until the stragglers catch up.

• There is more logic in the loaders: they keep track of the current Y position, and if the lead request thread gets ahead of the current read point, it stalls with a 60s timeout until the intermediate tiles are requested. This logic is implemented in the vips_sequential() operation.

The point of trying to keep thread locality and ordering is that we want to limit the number of scanlines that loaders have to keep behind the read point. We want to stream images through memory, not be forced into a load / process / save model.

This works reasonably well for simple cases, like thumbnailing single images, but can fail in more complex cases, such as repeated vips_insert(). Consider this Python program:

#!/usr/bin/python

import sys import random

import gi gi.require_version('Vips', '8.0') from gi.repository import Vips

composite = Vips.Image.black(100000, 100000)

for filename in sys.argv[2:]:
    tile = Vips.Image.new_from_file(filename, access = Vips.Access.SEQUENTIAL)
    x = random.randint(0, composite.width - tile.width) y = random.randint(0,
    composite.height - tile.height) composite = composite.insert(tile, x, y)

composite.write_to_file(sys.argv[1]) 

This makes a 100,000 x 100,000 pixel black image, then inserts a lot of other files into it and writes the result.

With vips8.4, this could very easily fail. Imagine this situation:

• image1 is very tall and thin

• image2 is short and fat, and by chance covers the centre of image1

• we'll write the top part of image1, then write the body of image2

• after image2 has been written, we need to write the bottom of image1, so a thread will ask for a set of pixels near the end of image1

• image1 knows that the previous request was for some scanlines near the top, so it thinks this request must be from a thread that has run way ahead of the pack and stalls it

And we have a deadlock. In fact, vips wouldn't deadlock, it would just pause on a 60s timeout on each thread. Sad!

Here's how the new seq works:

• Sinks work as before.

• Loaders use a revised vips_sequential() with the stalling logic removed. All it does now is track the read position, cache a few 100 lines behind the read point, and makes sure that lines are evaluated in order with no missing areas.

• Operations like vips_shrinkv() which can cause large non-local references have an extra bit of code which, if the input comes from a sequential source, adds a an extra vips_sequential() operator on the output. This forces vips_shrinkv() input to be sequential.

The old one constrained thread location on output, and on input as well. The new idea is to get rid of input constraints, and instead add extra code to the operations which could trigger large non-local references. Rather than tying threads down to stop them drifting apart, it makes sure they can never get too far apart in the first place.

Running the test program with git master gives this result:

real    1m2.317s 
user    2m58.472s 
sys     0m7.568s 
peak mem: 10gb 

Not bad!