Using Quarto with Python

Lucas A. Meyer

2022-07-07

Why Quarto

The content value chain

Content stuck in my computer is nearly worthless.

Goal

  • Move good content out of my computer as fast as possible
  • Reproducible
  • Git-based collaboration
  • Write once, generate:
    • Code
    • Paper
    • PowerPoint
    • Site/documentation

The content value chain by David Robinson (@drrob) captured by Amelia McNamara

Literate Programming

Literate programming can help create high-quality reproducible, documented code.

Donald E. Knuth proposed literate programming in a 1984 article.

Jupyter implements the literate programming paradigm, but generating high-quality mass-consumable output (articles, websites) requires additional tools.

Donald Knuth’s Literate Programming

I tried many tools for Literate Programming

LaTeX

  • Needs extra tools like CWeb
  • Great for PDFs… Beamer for slides
  • Not great for websites
  • Dynamic content -> code in Latex
    • \usepackage{ifthen}
    • @for, @while

Python notebooks

  • Great, with Pandoc
  • Config separated from files

Word/PPT

  • Hard to collaborate before O365
  • Hard to reproduce / auto-generate
    • code and writing separate

RMarkdown

  • Great for everything
  • Heavily dependent on R
  • New features in Quarto

Quarto

Quarto® is an open-source scientific and technical publishing system built on Pandoc.

The name “quarto” comes from the format of a book or pamphlet printed with eight pages of text, four to a side, then folded twice to produce four leaves.

The earliest known European printed book, the Sibyllenbuch (Gutemberg, c.1452), was done in the quarto format. Shakespeare’s plays, too!

With Quarto, you can:

  • Write Markdown and Python (or R, Julia, OJS)
  • Use equations, diagrams, citations, figures, etc.
  • Output articles, presentations, interactive websites…
  • Work with Jupyter Lab or VSCode

You can install Quarto on Linux, Windows and Mac.

Quarto workflow

The content pipeline for .ipynb

About 75% of data scientists use Python through Jupyter notebooks.

With some scripting, you can use Pandoc on .ipynb files to generate papers, HTML, PowerPoint, etc.

You just need to learn Pandoc and shell scripting.

Refresh the page to see the diagram on the right.

graph TD
    A[.ipynb] --> B(("Pandoc"))
    B ----> E[.doc]
    B ----> H[.pptx]
    B --> C[.md]
    B --> D[.tex] 
    D --> F((Xetex))
    C --> I((Hugo))
    F --> G[.pdf]
    I --> J[.html]
    style B fill:#FF6655AA
    style F fill:#88ffFF
    style I fill:#88ffFF

Quarto in Python, in a nutshell

All you need to use Quarto is to add some YAML (mostly simplified Pandoc configurations) to a .qmd file.

ipynb + YAML = .qmd.

This keeps the configuration and content in the same file. You can then render the outputs using quarto render <file.qmd> in the command line.

Refresh the page to see the diagram on the right.

graph TD
Q[.qmd] --> A
subgraph Quarto
    A[.ipynb] --> B(("Pandoc"))
    B --> C[.md]
    B --> D[.tex] 
    D --> F((Xetex))
    C --> I((Hugo))
    style B fill:#FF6655AA
    style F fill:#88ffFF
    style I fill:#88ffFF
end
    B ----> E[.doc]
    B ----> H[.pptx]
    F --> G[.pdf]
    I --> J[.html]

Using Quarto

The YAML front-matter

Quarto .qmd files always start with a YAML front-matter.

The YAML configuration determines what’s the output format of your document. A few popular output options are html, pptx, docx, and pdf.

You can use a single source file to generate multiple output types.

For example, the YAML on the right will generate a PowerPoint file and a Revealjs presentation.

---
title: "Quarto with Python"
format: 
  pptx:
    reference-doc: templates/template.pptx
  revealjs:
    incremental: false
    theme: pulse

author: Lucas A. Meyer
date: 2022-07-07
---

Main content

Writing content

Write content in Markdown.

Quarto’s Markdown supports figures, tables, bibliography, etc.

It also supports lots of extra features, like diagrams with mermaid and GraphViz, and even LaTeX equations:

\[ E = mc^2 \]

### Writing content

Write content in [Markdown].

Quarto's Markdown supports
figures, tables, bibliography, etc.       

It also supports lots of extra features, 
like diagrams with `mermaid` and 
`GraphViz`, and even LaTeX equations: 

$$
E = mc^2
$$

What if I want to add code?

The best thing about Quarto is that you can use it to run any code that you would be able to run in a Python notebook.


import numpy as np
import matplotlib.pyplot as plt

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(subplot_kw=\
                {'projection': 'polar'})
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()

Diagrams

You can use mermaid to create diagrams.

The diagram in this and in previous sections were created with mermaid.

Refresh the page to see the diagram on the right.

flowchart TD

A[Hard] -->|Text| B(Round)
B --> C{Decision}
C -->|One| D[Result 1]
C -->|Two| E[Result 2]
flowchart TD

A[Hard] -->|Text| B(Round)
B --> C{Decision}
C -->|One| D[Result 1]
C -->|Two| E[Result 2]

Regression and results

This code runs the first simple regression in Wooldridge’s Econometrics book.

\(\text{wage} = \alpha + \beta_1 \text{educ} + \epsilon\)

var coef s.e. t p-val
int -0.9 0.68 -1.32 0.19
educ 0.54 0.05 10.17 0 
# Load the data
df_wage = pd.read_csv("data/wage1.csv")

# Create an OLS model using R syntax
mod = smf.ols(formula="wage ~ educ",
              data=df_wage)

# Fit the model
res = mod.fit()

# Show the results
reg_table = pd.read_html(res.summary().
  tables[1].as_html(), header=0)[0]
display(Markdown(reg_table.
  to_markdown(index=False)))

Presentations in Quarto

Basic slide syntax

To create slides, you create sections with #, titles with ##, and bullets with -.

Content types

  • You can add several types of content
    • code (use backticks)
    • images
    • diagrams
    • tables
    • etc.
## Basic slide syntax

To create slides, you create sections 
with `#`, titles with `##`, and bullets 
with `-`.

### Content types

- You can add several types of content
    - code (use backticks)
    - images
    - diagrams
    - tables
    - etc.

Creating PowerPoint slides

To generate a presentation from a .qmd file, add format: pptx to the YAML front-matter.

Quarto will use the pandoc PowerPoint rules to render the content from the .qmd into .pptx.

The “pandoc rules” limit the flexibility to create PowerPoint presentations. Quarto has better presentation support for revealjs and beamer.

  • PowerPoint can use a template with only these layouts:
    • Title Slide
    • Title and Content
    • Section Header
    • Two Content
    • Comparison
    • Content with Caption
    • Blank

PowerPoint layout rules

The rules are available at:
https://pandoc.org/MANUAL.html#powerpoint-layout-choice

  • Title Slide: created from metadata fields like title and author
  • Section Header: created from the top-level headings (#)
  • Two Content: used when .qmd source contains :::: {.columns} and only text content. Previous slide is an example.
  • Comparison: same as “Two Content”, but content of columns is not text
  • Content with Caption: used when slide has non-text content, but no columns
  • Title and Content: whatever doesn’t fit the rules above.

PowerPoint templates

By adding a reference-doc entry to your YAML, you can tell Quarto (and pandoc) to use a file as a template for the format of your presentation.

The “Slide Master” needs to contain layouts named as per the previous slide (e.g. “Comparison”).

This allows you a lot of flexibility in the design of your slide deck, even if it is for just the small number of layouts that were listed in the previous slide.

You can control fonts, add background images, page numbering, etc.

---
title: "Using Quarto for everything"
format: pptx
reference-doc: templates/template.pptx
author: Lucas A. Meyer
date: 2022-07-14
---

Best feature: generate content dynamically

Let’s say you’re presenting a project about population dynamics but you don’t know which world leaders are coming to the conference.

On the presentation day, you learn that Italy, China, Brazil, India, Japan and Nigeria are attending.

You can use Python or R to automatically generate slides.

Generating slides with Python

The next slides/sections were generated using the code below:


df_dr = pd.read_csv("data/dr.csv.gz", compression="gzip")
df_pop = pd.read_csv("data/pop_brackets.csv.gz", compression="gzip")
years = [2000, 2025, 2050, 2075, 2100]
regions = ["Italy", "China", "Brazil", "India", "Japan", "Nigeria"]

for name in regions:
    display(Markdown(f"## Age and Population Pyramids for {name}"))
    display(Markdown(f'<div class="columns">'))
    display(Markdown(f'<div class="column">'))
    plot_dependency_ratio(df_dr[df_dr.Location == name])
    display(Markdown(f'</div>'))
    display(Markdown(f'<div class="column">'))
    plot_population_pyramid_series(df_pop[df_pop["Location"]==name], years)
    display(Markdown(f'</div>'))
    display(Markdown(f'</div>'))

Age and Population Pyramids for Italy

Age and Population Pyramids for China

Age and Population Pyramids for Brazil

Age and Population Pyramids for India

Age and Population Pyramids for Japan

Age and Population Pyramids for Nigeria

Generating a website

Changing one line creates a website

---
title: "Quarto with Python"
format: html
    # revealjs:
    #     incremental: false
    #     theme: [simple, revealjs-customizations.scss]
    #     title-slide-attributes:
    #         data-background-image: images/data-viz-bg.jpg
    #         data-background-size: contain
    #         data-background-position: right

author: Lucas A. Meyer
date: 2022-07-14
---


Adding or changing the format to html will create a website.

Screenshot of website

Scholarly articles

Writing scholarly articles

I reused some of the content of this presentation to create two scholarly-looking articles.

The purpose of the articles is just to show how easy it is to generate them with Quarto, they don’t contain original research.

Quarto adds cross-reference, citations and bibliography support to Markdown.

The relevant files are:

Scholarly article screenshots

PDF version of the article

Microsoft Word version of the article

Citations and Footnotes

Citations don’t work on presentations, but are easy to add to articles.

You need to reference a BibTex file in the YAML front-matter bibliography: references.bib. Quarto supports any of the 8000+ Citation Style Languages and will generate the “References” section automatically.

You can cite by using [@citation-name] in your text. Please check the article .qmd source and the PDF and DOCX outputs.

Generating footnotes is also easy. Using [^ref] links to a footnote, and [^ref: content of the footnote] generates its content1.

Cross references

The Quarto guide has a great section on cross-references. I cover only the main points.

To create a cross-referenceable figure, section or equation, you need to tag it with its corresponding prefix, respectively “fig”, “sec” and “eq”.

To tag it, use the following syntax: #prefix-name.

For example, when declaring a figure, you can use:

![Elephant](elephant.png){#fig-elephant}

Later, you can refer to it using:

See @fig-elephant for an illustration.

Books in Quarto

Books in Quarto

You can also write books with quarto. From the same collection of .qmd files, Quarto can generate:

  • ePub
  • PDF
  • Online book

Two recent examples are:

Hands-on Programming in R

This is a free book, and you can see the Quarto (source) that generated it.

Python for Data Analysis, 3E

This is another free book, and you can see the Quarto (source) that generated it.

Should I use Quarto?

Where Quarto excels (July 2022)

Articles: maybe

I think Quarto® is more helpful for a team that already uses Git with Python notebooks or LaTeX to write articles.

Microsoft Word collaboration through SharePoint and Teams is easier than Git and Quarto… but it’s not reproducible.

Python notebook: excellent

Quarto adds features to Python notebooks without detracting anything. You just need a few YAML lines.

Blog: excellent

Quarto allowed me to have a scriptable, Python-based blog. I wrote code to post new articles to Twitter and LinkedIn.

Presentations: maybe

Great for RevealJS. For PPT, render process -> long edit cycle. Useful for:

  • a lot of dynamic content
  • Reproducibility needs
  • Collaborators used to Git/Beamer

Where to go next

Next Steps

  • The Quarto website has great tutorials:
  • If you have multiple input files (e.g., blog or book), you can create a project. This allows rendering multiple files that link to each other.
    • This also allows combining Python and R natively for the same project
    • Some writers can use R, some can use Python, their content can link to each other as long as they’re in different .qmd files.
  • Projects can have pre-render and post-render steps in Python, R, Lua and shell script. I tend to use shell scripts that call other scripts.

Hacks

Quarto is under active development, and quickly reaching v1.0. While creating this content, I had to do some workarounds.

  1. I created a script to quickly install the latest version
  2. Mermaid diagrams blank in PPTX. Bug quickly fixed.
  3. Mermaid diagrams blank in RevealJS. Workaround: refresh.
  4. HTML website and RevealJS from same source: images disappear
    • Workaround: add self-contained: true to YAML front-matter
  5. Pandoc large monospaced font size in PowerPoint. Workaround: script.
  6. The decorations on this slide only appear on RevealJS, by design.
  7. Slide with article screenshots renders weird in HTML, no workaround.

THANK YOU