Towards Jupyter Book 2 with MyST-MD#

Jupyter Book and MyST - A Success Story#

Over the past four years, the Executable Book(s) (EB) project has been working to improve workflows for writing and publishing with Jupyter Notebooks, along with the broader open source science ecosystem.

Within this collaboration, much of the development effort has been spent on building Jupyter Book, which has been a tremendous success; there are at least 13,209 GitHub repositories using the tool according to the dependency graph for the project.[1] Furthermore, the reach of EB has grown to encompass documentation writers and package authors who use EB tools to provide a richer experience for their audiences. For example, six of the flagship scientific python projects[3] like NumPy, Pandas, and Dask make use of at least one of the Sphinx-based tools we’ve developed such as the PyData Sphinx Theme, the Sphinx Book Theme, MyST Markdown parser, or the Jupyter Notebook parser.[4]

During this time, it has become clear that the MyST Markdown language has been at the heart of the project’s growth. In 2023, this was recognized by designating MyST a top-level project in Executable Books and launching the MyST-MD (previously MyST-JS) in collaboration with Curvenote. The JavaScript tools for parsing and rendering MyST Markdown on the web are now used by JupyterLab MyST to bring MyST Markdown to JupyterLab, and even power the proceedings rendering at SciPy 2024.

MyST-MD is able to replace Sphinx in Jupyter Book […].

With MyST-MD, the project added a critical new capability: a document structure and engine in addition to the MyST markdown syntax. Rather than parsing MyST to Sphinx, MyST-MD provides its own standard document format and structure (as JSON), and an engine that exposes MyST documents in a flexible and reusable way. As such, MyST-MD is able to replace Sphinx in a growing number of scenarios; with the mystmd command line interface (CLI), it is now possible to use MyST-MD to produce scientific PDFs, documents, and websites.

Next steps for Jupyter Book#

The 1.x release cycle across the Jupyter Book packages conveys our belief that many of the Sphinx-based tools that power Jupyter Book, such as myst-nb and myst-parser have matured to a point of stability, and are feature-complete for Jupyter Book’s goals.[2]

Looking forward, there are still clear areas of improvements for Jupyter Book, such as simplifying the PDF export process, lowering the barrier to extensibility, improving the UX for authoring with Jupyter interfaces, and publishing to a wider range of platforms (such as scientific journals). For the next phase of Jupyter Book, we believe that the new MyST-MD document engine is the right foundation to build on.

Jupyter Book 2 will be an opinionated distribution of MyST-MD.

To continue to deliver on these aims and principles, we are building a new Jupyter Book experience on top of MyST-MD. Unlike Jupyter Book 1, which provides its own configuration and interface on top of Sphinx, Jupyter Book 2 will be an opinionated distribution of MyST-MD.

By building on top of MyST-MD, we hope that:

  • Users will face lower barriers in producing high-quality PDF outputs (e.g. via Typst PDF rendering) and interactive websites.

  • Users will more easily discover, connect, reuse, and remix books published using MyST-MD / Jupyter Book, for richer and more reproducible communication. For example, via cross-book embedding.

  • Users will benefit from the latest developments in web tooling for interactive, rich, computational experiences across all of the MyST-tools e.g., JupyterLab, and the CLI.

  • Users will have a clear separation between the single source of truth for configuration (https://mystmd.org), and the book-focused guide for the Jupyter Book distribution (https://jupyterbook.org).

  • Developers will face a lower barrier to entry to contribute through the use of standards such as unist (see this example of contributing a new feature to help you get started).

  • Developers will spend more time on enhancements and less time on maintenance, due to a more modern and streamlined infrastructure stack and lower technical debt.

  • Users and developers alike will have a tighter feedback loop through the centralized development of MyST and Jupyter Book.

Incorporation under the Jupyter project#

In order to more effectively focus our development efforts and ensure long-term sustainability, we intend to find a new organizational home for the MyST document engine, and focus Executable Books on the Sphinx-based ecosystem of tools for software developers. Our intention is to form a new sub-community within the Jupyter Project to steward the MyST document engine. To follow along, see this Jupyter Enhancement Proposal for creating a jupyter-book community within Jupyter.

What this means for Jupyter Book v1 users#

Our first priority is to make a transition to Jupyter Book v2 as frictionless as possible, by supporting v1 workflows and configuration where we can, and providing migration pathways for users (see the principles we’re following for migration for more information). This will be a one-off, guided process that will produce a book that is easier to maintain and fully-compatible with MyST-MD.

Below are the major areas where we anticipate users needing to perform steps to migrate.

Configuration. Existing authors of Jupyter Books will need to migrate their books to the new myst.yml configuration format that replaces config.yaml and _toc.yml. This will be made easier using an interactive CLI tool that assists in performing the conversion.

Directives and roles. We aim to support most Sphinx-based directives and roles, but some may be deprecated or change their configuration. Jupyter Book 2 will provide an interactive CLI tool that upgrades legacy syntax to the latest MyST standard, such as glossary directives. This should be a mostly automatic process; although the presence of Sphinx was generally invisible in Jupyter Book 1, there are some places where it was not possible to hide this from the user. Such cases will require some human intervention to upgrade.

Functionality. Broad functionality of Jupyter Book v2 should be nearly the same as v1, with additional capabilities as well. There may be some short-term regressions in functionality for less-used features, and we aim to grow these back over time as users request.

To ease this process, the Executable Books team will author a migration guide and develop tooling to minimize the number of manual actions required to be taken by the user.

What this means for Sphinx extension users#

The Executable Books organization will remain the stewards of the Sphinx-based stack, and we anticipate continuing support and development of the Sphinx ecosystem of tools, likely with a focus on supporting the open source developer community rather than the broader “scientific communication” community. These projects are strongly supported by open-source contributors, and we hope that this continues alongside work from the Executable Books team. Some Executable Books tools will likely slow their development (e.g., MyST-NB), while more heavily-used tools will likely continue to evolve and improve (e.g., myst-parser, which now powers a large part of the markdown experience in Sphinx). We recognize that there is an on-going need for these tools, and will continue working to ensure their long-lived success for the developer community.

How can I follow along?#

We’re excited about these new directions for the MyST project, and welcome others to join the conversation or get involved improving these tools.

For synchronous chat about the MyST engine: See our Discord chat room.

For asynchronous discussion about the MyST engine: See the Discussions forum in the mystmd repository.

For high-level information about the MyST engine: See mystmd.org.