chickadee » wodehaus

wodehaus

Description

Wodehaus is a lightweight forking job server. It dispatches workers created by a user-provided "producer" procedure to independent processes, then sends the results to a "consumer" procedure for finalization.

The source for this extension is available here.

API

[type] job any

A job may be any object for which read/write invariance holds.

[type] result any

A result, like a job, may be any read/write-invariant object.

[type] worker (job -> result)

A worker may be any procedure that accepts a job and returns result.

[type] producer (-> worker job)

A producer is called repeatedly by a job server to produce pairs of workers and jobs to be dispatched.

[type] consumer (job result -> any)

A consumer is provided with the results of each job.

[type] logger (symbol string #!rest any -> void)

A logger is used to print information at various points during a job server's operation. It receives a symbolic log level (one of error or info) and a printf-style format string and arguments.

(process-name) procedure

Returns a value that's unique to the worker process from which it's called. Currently, this is simply the worker's process ID.

When called from within the server process, some descriptive object is returned. Exactly what type of object is unspecified.

This value can be changed with set!.

(start producer #!optional consumer logger fixnum) procedure

Starts a job server.

When started, the server loops indefinitely, calling producer to generate workers and jobs to be dispatched. If there are no jobs to run, it is the producer's responsibility to block until one becomes available. To stop the server, producer should raise the #:stop keyword.

The final argument to start sets the maximum number of concurrent workers the job server will use. The default value is 4.

Each worker is executed in a dedicated OS-level process. A worker receives as a single argument the job with which it was produced, and its result is passed back to the consumer in the server process. Worker arguments and results must be fully read/write-invariant.

The consumer procedure is invoked with each completed job and its result. Its own result is discarded.

Each job server uses its thread's "specific" data slot for bookkeeping and installs a SIGINT handler that will cause the server to shut down cleanly on interrupt. It's recommended that each job server be run in a dedicated thread.

Author

Evan Hanson

License

Copyright (c) 2016, 3-Clause BSD.

Contents »