* reimplement threadpool
just a set of threads that get recycled
"ninja test" passes and dzsave seems to work, though pytest fails for
some reason
* clean up threading code a bit
move base stuff into thread.c, so we have a simple
thread -> threadset -> threadpool
layering
* start trying to revert g_threadpool
based on the original commit
* working!
nice low systime again
still need to repply cnages to threadpool.c since 80e0cc3d1
* reapply fixes from master
so threadpool.c is now up to date
* rename VipsThread as VipsWorker
a bit less confusing
* use a semaphore to count workers in a pool
* tidy up
* formatting
* dynamic threadpool sizing
based on counting the number of blocked threads in each pool
it works, but the improvement is not great :(
* add "concurrency" metadata item
so operators can hint threadpool size (dzsave especially)
* don't use thinstrip for small images
* add RGB mode to openslide
since flatten was taking 20% of CPU time for dzsave
* fix up rgb mode
now actually works
* make the tile buffer per thread
in the new openslideload rgb mode
* fix dynamic pool downsize
* mild refactoring
* fix the buffer system
oops, turned it off by mistake
* all done!
* revise changelog
* Update libvips/iofuncs/threadset.c
Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl>
* Update libvips/iofuncs/threadset.c
Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl>
* LSan: add libMagickCore to suppression file
* Revert "Remove mutex lock for VipsThreadStartFn"
This reverts commit 41440491.
* add VIPS_MAX_THREADS
to set a hard limit on the threadset size
* Revert "Revert "Remove mutex lock for VipsThreadStartFn""
This reverts commit 77e8520966ba79194fff3b4e648bbd295cd5c260.
* remove sslock from sink.c
* move fixed threadpool build to init
not first use
* add some doc comments
* revert test suite threshold change
* add a test for MAX_THREADS
and move the test tmp/ area into the builddir
* limit VIPS_MAX_THREADS to sane values
* use tabs rather than spaces
Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl>
Use GLib's i18n support instead of copying and pasting that
logic into its own header. This deprecates the vips/intl.h
header in favour of glib/gi18n.h.
Since threads can be reused, we need to free the
calculated pixel buffer cache early (i.e. during
vips_thread_shutdown).
This (partially) reverts commit 1a915db.
We were freeing pixel buffers on thread exit. This was convenient, but
meant that main thread buffers were not freed until program exit. As a
result, long-running programs which created main thread buffers would
slowly fill the operation cache with useless junk, forcing everything else out.
This change also frees pixel buffers on image close. This makes the
cache work much better in long-running programs, and can substantially
reduce memory use.
See https://github.com/jcupitt/libvips/issues/466
threads keep pixel buffers in thread-private storage, and free these
buffers on thread exit ... this means buffers created by the main thread will
only be freed on program exit!
if your program creates any main-thread buffers, these buffers will
eventually fill the operation cache and force everything else out,
making the cache useless
this patch explicitly frees main-thread pixel buffers on image close
You now need to call vips_thread_shutdown() from all threads you create
yourself just before they exit. This gives vips a chance to free any
thread-private memory and to write out profiling data. libvips will
complain noisily if you fail to do this. You can harmlessly call
vips_thread_shutdown() more than once if you wish.
libvips will call vips_thread_shutdown for you automatically for the main
thread (the one that calls vips_shutdown()) and for threads it creates
directly itself.
so now recycle lists are short, scale with pipeline complexity, and
buffers are always appropriately sized for the image instead of being
slowly sized up to the max size for the pipeline
before:
$ vips sharpen k2.jpg x.jpg --radius 20
memory: high-water mark 38.99 MB
after:
$ vips sharpen k2.jpg x.jpg --radius 20
memory: high-water mark 29.46 MB
we were unreffing windows and buffers attached to regions on
vips_region_region() etc.
when these regions were reinitialised later, fresh buffers were
allocated causing a lot of churn
glib-2.32 deprecated g_mutex_new() ... use a compat function to hide
this change
also remove the --without-threads configure flag, gthread is now
compulsory, so there was no longer any point
John Cupitt
Kleis Auke Wolthuizen
Lovell Fuller
Benjamin Gilbert