Commit 2894d20e authored by Paul McCarthy's avatar Paul McCarthy 🚵
Browse files

DOC: Document macOS fork issue

parent 67b9ab9b
......@@ -285,6 +285,49 @@ Then you can run the test suite using ``pytest``::
pytest
macOS issues
------------
FUNPACK makes extensive use of the Python `multiprocessing
<https://docs.python.org/3/library/multiprocessing.html>`_ module to speed up
certain steps in its processing pipeline. FUNPACK relies on the POSIX `fork()
<https://www.man7.org/linux/man-pages/man2/fork.2.html>`_ mechanism, so that
worker processes may inexpensively inherit the memory space of the main
process (often referred to as *copy-on-write*). This is to avoid having to
serialise the data set being processed (stored internally as a
``pandas.DataFrame``).
In python 3.8 on macOS, the default method used by the ``multiprocessing``
module was changed from ``fork`` to ``spawn``, due to changes in macOS 10.13
restricting the use of ``fork()`` for safety reasons. Some background
information on this change can be found at https://bugs.python.org/issue33725,
and at `this blog post
<https://wefearchange.org/2018/11/forkmacos.rst.html>`_.
FUNPACK therefore explicitly sets the method used by the ``multiprocessing``
to ``fork``, to take advantage of copy-on-write semantics. Using ``fork()``
on macOS *should* be safe for single-threaded parent processes, but as FUNPACK
calls ``fork()`` numerous times (by creating and discarding
``multiprocessing.Pool()`` objects on an as-needed basis), this assumption may
not be valid, and FUNPACK may crash with an error message resembling the
following::
+[SomeClass initialize] may have been in progress in another thread
when fork() was called. We cannot safely call it or ignore it in the
fork() child process. Crashing instead.
You might be able to work around this error by setting an environment variable
before calling FUNPACK, like so::
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
Citing
------
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment