Merge pull request #1597 from getzola/next

Next

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+
+## 0.15.0 (2021-12-05)
+
+- Fix config file watching
+- Support custom syntax highlighting themes
+- Add a `required` argument to taxonomy template functions to allow them to return empty taxonomies
+- Support colocating subfolders
+- Shortcodes and `anchor-link.html` can now access the `lang` context
+- Add prompt before replacing the output directory with `zola build` if the `output-dir` flag is given
+- Shortcode handling has been completely rewritten, solving many issues
+- Also add internal links starting with `#` without any internal Zola link
+
 ## 0.14.1 (2021-08-24)
 
 - HTML minification now respects HTML spec (it still worked before because browsers can handle invalid HTML well and minifiers take advantage of it)

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "zola"
-version = "0.14.1"
+version = "0.15.0"
 authors = ["Vincent Prouillet <hello@vincentprouillet.com>"]
 edition = "2018"
 license = "MIT"
@@ -28,7 +28,7 @@ termcolor = "1.0.4"
 url = "2"
 # Below is for the serve cmd
 hyper = { version = "0.14.1", default-features = false, features = ["runtime", "server", "http2", "http1"] }
-tokio = { version = "1.0.1", default-features = false, features = ["rt", "fs"] }
+tokio = { version = "1.0.1", default-features = false, features = ["rt", "fs", "time"] }
 percent-encoding = "2"
 notify = "4"
 ws = "0.9"
@@ -36,6 +36,7 @@ ctrlc = "3"
 open = "2"
 globset = "0.4"
 relative-path = "1"
+pathdiff = "0.2"
 serde_json = "1.0"
 # For mimetype detection in serve mode
 mime_guess = "2.0"

Dockerfile

@@ -1,8 +1,7 @@
 FROM rust:slim AS builder
 
 RUN apt-get update -y && \
-  apt-get install -y python-pip make g++ python-setuptools libssl-dev pkg-config rsync && \
+  apt-get install -y make g++ libssl-dev && \
-  pip install dockerize && \
   rustup target add x86_64-unknown-linux-gnu
 
 WORKDIR /app
@@ -10,12 +9,7 @@ COPY . .
 
 RUN cargo build --release --target x86_64-unknown-linux-gnu
 
-RUN mv target/x86_64-unknown-linux-gnu/release/zola /usr/bin
-RUN mkdir -p /workdir
-WORKDIR /workdir
-RUN dockerize -n  -o /workdir  /usr/bin/zola
 
-
+FROM gcr.io/distroless/cc
-FROM scratch
+COPY --from=builder /app/target/x86_64-unknown-linux-gnu/release/zola /bin/zola
-COPY --from=builder /workdir .
+ENTRYPOINT [ "/bin/zola" ]
-ENTRYPOINT [ "/usr/bin/zola" ]

README.md

@@ -7,53 +7,27 @@ A fast static site generator in a single binary with everything built-in.
 Documentation is available on [its site](https://www.getzola.org/documentation/getting-started/installation/) or
 in the `docs/content` folder of the repository and the community can use [its forum](https://zola.discourse.group).
 
-## Comparisons with other static site generators
+This tool and the template engine it is using were born from an intense dislike of the (insane) Golang template engine and therefore of 
+Hugo that I was using before for 6+ sites.
 
-|                                 | Zola   | Cobalt | Hugo   | Pelican |
+# List of features
-|:--------------------------------|:------:|:------:|:------:|:-------:|
-| Single binary                   | ![yes] | ![yes] | ![yes] | ![no]   |
-| Language                        | Rust   | Rust   | Go     | Python  |
-| Syntax highlighting             | ![yes] | ![yes] | ![yes] | ![yes]  |
-| Sass compilation                | ![yes] | ![yes] | ![yes] | ![yes]  |
-| Assets co-location              | ![yes] | ![yes] | ![yes] | ![yes]  |
-| Multilingual site               | ![ehh] | ![no]  | ![yes] | ![yes]  |
-| Image processing                | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Sane & powerful template engine | ![yes] | ![yes] | ![ehh] | ![yes]  |
-| Themes                          | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Shortcodes                      | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Internal links                  | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Link checker                    | ![yes] | ![no]  | ![no]  | ![yes]  |
-| Table of contents               | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Automatic header anchors        | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Aliases                         | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Pagination                      | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Custom taxonomies               | ![yes] | ![no]  | ![yes] | ![no]   |
-| Search                          | ![yes] | ![no]  | ![no]  | ![yes]  |
-| Data files                      | ![yes] | ![yes] | ![yes] | ![no]   |
-| LiveReload                      | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Netlify support                 | ![yes] | ![no]  | ![yes] | ![no]   |
-| Vercel support                  | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Cloudflare Pages support        | ![yes] | ![no]  | ![yes] | ![yes]  |
-| Breadcrumbs                     | ![yes] | ![no]  | ![no]  | ![yes]  |
-| Custom output formats           | ![no]  | ![no]  | ![yes] | ![no]   |
 
-### Supported content formats
+- Single binary
-
+- Syntax highlighting 
-- Zola: markdown
+- Sass compilation
-- Cobalt: markdown
+- Assets co-location
-- Hugo: markdown, asciidoc, org-mode
+- (Basic currently) multilingual site suport
-- Pelican: reStructuredText, markdown, asciidoc, org-mode, whatever-you-want
+- Image processing
-
+- Themes
-### ![ehh] explanations
+- Shortcodes
-
+- Internal links
-Hugo gets ![ehh] for the template engine because while it is probably the most powerful template engine in the list (after Jinja2) it personally drives me insane, to the point of writing my own template engine and static site generator. Yes, this is a bit biased.
+- External link checker
-
+- Table of contents automatic generation
-Zola gets ![ehh] for multi-language support as it only has a basic support and does not (yet) offer things like i18n in templates.
+- Automatic header anchors
-
+- Aliases
-### Pelican notes
+- Pagination
-
+- Custom taxonomies
-Many features of Pelican come from plugins, which might be tricky to use because of a version mismatch or inadequate documentation. Netlify supports Python and Pipenv but you still need to install your dependencies manually.
+- Search with no servers or any third parties involved
-
+- Live reload
-[yes]: ./is-yes.svg
+- Deploy on many platforms easily: Netlify, Vercel, Cloudflare
-[ehh]: ./is-ehh.svg
+- Breadcrumbs
-[no]:  ./is-no.svg

azure-pipelines.yml

@@ -21,7 +21,7 @@ stages:
               rustup_toolchain: stable
             linux-pinned:
               imageName: 'ubuntu-20.04'
-              rustup_toolchain: 1.49.0
+              rustup_toolchain: 1.52.0
         pool:
           vmImage: $(imageName)
         steps:

components/config/src/config/markup.rs

@@ -1,9 +1,15 @@
-use std::path::Path;
+use std::{path::Path, sync::Arc};
 
 use serde_derive::{Deserialize, Serialize};
-use syntect::parsing::{SyntaxSet, SyntaxSetBuilder};
+use syntect::{
+    highlighting::{Theme, ThemeSet},
+    html::css_for_theme_with_class_style,
+    parsing::{SyntaxSet, SyntaxSetBuilder},
+};
 
-use errors::Result;
+use errors::{bail, Result};
+
+use crate::highlighting::{CLASS_STYLE, THEME_SET};
 
 pub const DEFAULT_HIGHLIGHT_THEME: &str = "base16-ocean-dark";
 
@@ -43,26 +49,106 @@ pub struct Markdown {
     pub external_links_no_referrer: bool,
     /// Whether smart punctuation is enabled (changing quotes, dashes, dots etc in their typographic form)
     pub smart_punctuation: bool,
-
+    /// A list of directories to search for additional `.sublime-syntax` and `.tmTheme` files in.
-    /// A list of directories to search for additional `.sublime-syntax` files in.
+    pub extra_syntaxes_and_themes: Vec<String>,
-    pub extra_syntaxes: Vec<String>,
     /// The compiled extra syntaxes into a syntax set
     #[serde(skip_serializing, skip_deserializing)] // not a typo, 2 are need
     pub extra_syntax_set: Option<SyntaxSet>,
+    /// The compiled extra themes into a theme set
+    #[serde(skip_serializing, skip_deserializing)] // not a typo, 2 are need
+    pub extra_theme_set: Arc<Option<ThemeSet>>,
 }
 
 impl Markdown {
-    /// Attempt to load any extra syntax found in the extra syntaxes of the config
+    /// Gets the configured highlight theme from the THEME_SET or the config's extra_theme_set
-    pub fn load_extra_syntaxes(&mut self, base_path: &Path) -> Result<()> {
+    /// Returns None if the configured highlighting theme is set to use css
-        if self.extra_syntaxes.is_empty() {
+    pub fn get_highlight_theme(&self) -> Option<&Theme> {
-            return Ok(());
+        if self.highlight_theme == "css" {
+            None
+        } else {
+            Some(self.get_highlight_theme_by_name(&self.highlight_theme))
+        }
+    }
+
+    /// Gets an arbitrary theme from the THEME_SET or the extra_theme_set
+    pub fn get_highlight_theme_by_name(&self, theme_name: &str) -> &Theme {
+        (*self.extra_theme_set)
+            .as_ref()
+            .and_then(|ts| ts.themes.get(theme_name))
+            .unwrap_or_else(|| &THEME_SET.themes[theme_name])
+    }
+
+    /// Attempt to load any extra syntaxes and themes found in the extra_syntaxes_and_themes folders
+    pub fn load_extra_syntaxes_and_highlight_themes(
+        &self,
+        base_path: &Path,
+    ) -> Result<(Option<SyntaxSet>, Option<ThemeSet>)> {
+        if self.extra_syntaxes_and_themes.is_empty() {
+            return Ok((None, None));
         }
 
         let mut ss = SyntaxSetBuilder::new();
-        for dir in &self.extra_syntaxes {
+        let mut ts = ThemeSet::new();
+        for dir in &self.extra_syntaxes_and_themes {
             ss.add_from_folder(base_path.join(dir), true)?;
+            ts.add_from_folder(base_path.join(dir))?;
+        }
+        let ss = ss.build();
+
+        Ok((
+            if ss.syntaxes().is_empty() { None } else { Some(ss) },
+            if ts.themes.is_empty() { None } else { Some(ts) },
+        ))
+    }
+
+    pub fn export_theme_css(&self, theme_name: &str) -> String {
+        let theme = self.get_highlight_theme_by_name(theme_name);
+        css_for_theme_with_class_style(theme, CLASS_STYLE)
+    }
+
+    pub fn init_extra_syntaxes_and_highlight_themes(&mut self, path: &Path) -> Result<()> {
+        if self.highlight_theme == "css" {
+            return Ok(());
+        }
+
+        let (loaded_extra_syntaxes, loaded_extra_highlight_themes) =
+            self.load_extra_syntaxes_and_highlight_themes(path)?;
+
+        if let Some(extra_syntax_set) = loaded_extra_syntaxes {
+            self.extra_syntax_set = Some(extra_syntax_set);
+        }
+        if let Some(extra_theme_set) = loaded_extra_highlight_themes {
+            self.extra_theme_set = Arc::new(Some(extra_theme_set));
+        }
+
+        // Validate that the chosen highlight_theme exists in the loaded highlight theme sets
+        if !THEME_SET.themes.contains_key(&self.highlight_theme) {
+            if let Some(extra) = &*self.extra_theme_set {
+                if !extra.themes.contains_key(&self.highlight_theme) {
+                    bail!(
+                        "Highlight theme {} not found in the extra theme set",
+                        self.highlight_theme
+                    )
+                }
+            } else {
+                bail!("Highlight theme {} not available.\n\
+                You can load custom themes by configuring `extra_syntaxes_and_themes` to include a list of folders containing '.tmTheme' files", self.highlight_theme)
+            }
+        }
+
+        // Validate that all exported highlight themes exist as well
+        for theme in self.highlight_themes_css.iter() {
+            let theme_name = &theme.theme;
+            if !THEME_SET.themes.contains_key(theme_name) {
+                // Check extra themes
+                if let Some(extra) = &*self.extra_theme_set {
+                    if !extra.themes.contains_key(theme_name) {
+                        bail!("Can't export highlight theme {}, as it does not exist.\n\
+                        Make sure it's spelled correctly, or your custom .tmTheme' is defined properly.", theme_name)
+                    }
+                }
+            }
         }
-        self.extra_syntax_set = Some(ss.build());
 
         Ok(())
     }
@@ -110,8 +196,9 @@ impl Default for Markdown {
             external_links_no_follow: false,
             external_links_no_referrer: false,
             smart_punctuation: false,
-            extra_syntaxes: Vec::new(),
+            extra_syntaxes_and_themes: vec![],
             extra_syntax_set: None,
+            extra_theme_set: Arc::new(None),
         }
     }
 }

components/config/src/config/mod.rs

@@ -12,7 +12,6 @@ use globset::{Glob, GlobSet, GlobSetBuilder};
 use serde_derive::{Deserialize, Serialize};
 use toml::Value as Toml;
 
-use crate::highlighting::THEME_SET;
 use crate::theme::Theme;
 use errors::{bail, Error, Result};
 use utils::fs::read_file;
@@ -97,6 +96,7 @@ pub struct SerializedConfig<'a> {
     title: &'a Option<String>,
     description: &'a Option<String>,
     languages: HashMap<&'a String, &'a languages::LanguageOptions>,
+    default_language: &'a str,
     generate_feed: bool,
     feed_filename: &'a str,
     taxonomies: &'a [taxonomies::Taxonomy],
@@ -105,6 +105,7 @@ pub struct SerializedConfig<'a> {
 }
 
 impl Config {
+    // any extra syntax and highlight themes have been loaded and validated already by the from_file method before parsing the config
     /// Parses a string containing TOML to our Config struct
     /// Any extra parameter will end up in the extra field
     pub fn parse(content: &str) -> Result<Config> {
@@ -117,15 +118,6 @@ impl Config {
             bail!("A base URL is required in config.toml with key `base_url`");
         }
 
-        if config.markdown.highlight_theme != "css"
-            && !THEME_SET.themes.contains_key(&config.markdown.highlight_theme)
-        {
-            bail!(
-                "Highlight theme {} defined in config does not exist.",
-                config.markdown.highlight_theme
-            );
-        }
-
         languages::validate_code(&config.default_language)?;
         for code in config.languages.keys() {
             languages::validate_code(code)?;
@@ -165,7 +157,16 @@ impl Config {
         let path = path.as_ref();
         let content =
             read_file(path).map_err(|e| errors::Error::chain("Failed to load config", e))?;
-        Config::parse(&content)
+
+        let mut config = Config::parse(&content)?;
+        let config_dir = path
+            .parent()
+            .ok_or_else(|| Error::msg("Failed to find directory containing the config file."))?;
+
+        // this is the step at which missing extra syntax and highlighting themes are raised as errors
+        config.markdown.init_extra_syntaxes_and_highlight_themes(config_dir)?;
+
+        Ok(config)
     }
 
     /// Makes a url, taking into account that the base url might have a trailing slash
@@ -291,6 +292,7 @@ impl Config {
             title: &options.title,
             description: &options.description,
             languages: self.languages.iter().filter(|(k, _)| k.as_str() != lang).collect(),
+            default_language: &self.default_language,
             generate_feed: options.generate_feed,
             feed_filename: &options.feed_filename,
             taxonomies: &options.taxonomies,
@@ -330,7 +332,7 @@ pub fn merge(into: &mut Toml, from: &Toml) -> Result<()> {
 
 impl Default for Config {
     fn default() -> Config {
-        Config {
+        let mut conf = Config {
             base_url: DEFAULT_BASE_URL.to_string(),
             title: None,
             description: None,
@@ -355,7 +357,9 @@ impl Default for Config {
             search: search::Search::default(),
             markdown: markup::Markdown::default(),
             extra: HashMap::new(),
-        }
+        };
+        conf.add_default_language();
+        conf
     }
 }
 
@@ -675,4 +679,32 @@ output_dir = "docs"
         let config = Config::parse(config).unwrap();
         assert_eq!(config.output_dir, "docs".to_string());
     }
+
+    // TODO: Tests for valid themes; need extra scaffolding (test site) for custom themes.
+
+    #[test]
+    fn invalid_highlight_theme() {
+        let config = r#"
+[markup]
+highlight_code = true
+highlight_theme = "asdf"
+    "#;
+
+        let config = Config::parse(config);
+        assert_eq!(config.is_err(), true);
+    }
+
+    #[test]
+    fn invalid_highlight_theme_css_export() {
+        let config = r#"
+[markup]
+highlight_code = true
+highlight_themes_css = [
+  { theme = "asdf", filename = "asdf.css" },
+]
+    "#;
+
+        let config = Config::parse(config);
+        assert_eq!(config.is_err(), true);
+    }
 }

components/config/src/highlighting.rs

@@ -1,10 +1,12 @@
 use lazy_static::lazy_static;
 use syntect::dumps::from_binary;
 use syntect::highlighting::{Theme, ThemeSet};
+use syntect::html::ClassStyle;
 use syntect::parsing::{SyntaxReference, SyntaxSet};
 
 use crate::config::Config;
-use syntect::html::{css_for_theme_with_class_style, ClassStyle};
+
+pub const CLASS_STYLE: ClassStyle = ClassStyle::SpacedPrefixed { prefix: "z-" };
 
 lazy_static! {
     pub static ref SYNTAX_SET: SyntaxSet = {
@@ -16,8 +18,6 @@ lazy_static! {
         from_binary(include_bytes!("../../../sublime/themes/all.themedump"));
 }
 
-pub const CLASS_STYLE: ClassStyle = ClassStyle::SpacedPrefixed { prefix: "z-" };
-
 #[derive(Clone, Debug, PartialEq, Eq)]
 pub enum HighlightSource {
     /// One of the built-in Zola syntaxes
@@ -42,11 +42,7 @@ pub fn resolve_syntax_and_theme<'config>(
     language: Option<&'_ str>,
     config: &'config Config,
 ) -> SyntaxAndTheme<'config> {
-    let theme = if config.markdown.highlight_theme != "css" {
+    let theme = config.markdown.get_highlight_theme();
-        Some(&THEME_SET.themes[&config.markdown.highlight_theme])
-    } else {
-        None
-    };
 
     if let Some(ref lang) = language {
         if let Some(ref extra_syntaxes) = config.markdown.extra_syntax_set {
@@ -88,8 +84,3 @@ pub fn resolve_syntax_and_theme<'config>(
         }
     }
 }
-
-pub fn export_theme_css(theme_name: &str) -> String {
-    let theme = &THEME_SET.themes[theme_name];
-    css_for_theme_with_class_style(theme, CLASS_STYLE)
-}

components/library/Cargo.toml

@@ -14,6 +14,7 @@ serde_derive = "1"
 regex = "1"
 lazy_static = "1"
 lexical-sort = "0.3"
+walkdir = "2"
 
 front_matter = { path = "../front_matter" }
 config = { path = "../config" }

components/library/src/content/mod.rs

@@ -3,9 +3,10 @@ mod page;
 mod section;
 mod ser;
 
-use std::fs::read_dir;
 use std::path::{Path, PathBuf};
 
+use walkdir::WalkDir;
+
 pub use self::file_info::FileInfo;
 pub use self::page::Page;
 pub use self::section::Section;
@@ -29,10 +30,17 @@ pub fn has_anchor(headings: &[Heading], anchor: &str) -> bool {
 
 /// Looks into the current folder for the path and see if there's anything that is not a .md
 /// file. Those will be copied next to the rendered .html file
-pub fn find_related_assets(path: &Path, config: &Config) -> Vec<PathBuf> {
+/// If `recursive` is set to `true`, it will add all subdirectories assets as well. This should
+/// only be set when finding page assets currently.
+/// TODO: remove this flag once sections with assets behave the same as pages with assets
+pub fn find_related_assets(path: &Path, config: &Config, recursive: bool) -> Vec<PathBuf> {
     let mut assets = vec![];
 
-    for entry in read_dir(path).unwrap().filter_map(std::result::Result::ok) {
+    let mut builder = WalkDir::new(path);
+    if !recursive {
+        builder = builder.max_depth(1);
+    }
+    for entry in builder.into_iter().filter_map(std::result::Result::ok) {
         let entry_path = entry.path();
         if entry_path.is_file() {
             match entry_path.extension() {
@@ -46,18 +54,11 @@ pub fn find_related_assets(path: &Path, config: &Config) -> Vec<PathBuf> {
     }
 
     if let Some(ref globset) = config.ignored_content_globset {
-        // `find_related_assets` only scans the immediate directory (it is not recursive) so our
-        // filtering only needs to work against the file_name component, not the full suffix. If
-        // `find_related_assets` was changed to also return files in subdirectories, we could
-        // use `PathBuf.strip_prefix` to remove the parent directory and then glob-filter
-        // against the remaining path. Note that the current behaviour effectively means that
-        // the `ignored_content` setting in the config file is limited to single-file glob
-        // patterns (no "**" patterns).
         assets = assets
             .into_iter()
-            .filter(|path| match path.file_name() {
+            .filter(|p| match p.strip_prefix(path) {
-                None => false,
+                Err(_) => false,
-                Some(file) => !globset.is_match(file),
+                Ok(file) => !globset.is_match(file),
             })
             .collect();
     }
@@ -68,27 +69,58 @@ pub fn find_related_assets(path: &Path, config: &Config) -> Vec<PathBuf> {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use std::fs::File;
+    use std::fs::{create_dir, File};
 
     use config::Config;
     use tempfile::tempdir;
 
     #[test]
-    fn can_find_related_assets() {
+    fn can_find_related_assets_recursive() {
         let tmp_dir = tempdir().expect("create temp dir");
-        File::create(tmp_dir.path().join("index.md")).unwrap();
+        let path = tmp_dir.path();
-        File::create(tmp_dir.path().join("example.js")).unwrap();
+        File::create(path.join("index.md")).unwrap();
-        File::create(tmp_dir.path().join("graph.jpg")).unwrap();
+        File::create(path.join("example.js")).unwrap();
-        File::create(tmp_dir.path().join("fail.png")).unwrap();
+        File::create(path.join("graph.jpg")).unwrap();
+        File::create(path.join("fail.png")).unwrap();
+        create_dir(path.join("subdir")).expect("create subdir temp dir");
+        File::create(path.join("subdir").join("index.md")).unwrap();
+        File::create(path.join("subdir").join("example.js")).unwrap();
 
-        let assets = find_related_assets(tmp_dir.path(), &Config::default());
+        let assets = find_related_assets(path, &Config::default(), true);
-        assert_eq!(assets.len(), 3);
+        assert_eq!(assets.len(), 4);
-        assert_eq!(assets.iter().filter(|p| p.extension().unwrap() != "md").count(), 3);
+        assert_eq!(assets.iter().filter(|p| p.extension().unwrap() != "md").count(), 4);
-        assert_eq!(assets.iter().filter(|p| p.file_name().unwrap() == "example.js").count(), 1);
+
-        assert_eq!(assets.iter().filter(|p| p.file_name().unwrap() == "graph.jpg").count(), 1);
+        for asset in vec!["example.js", "graph.jpg", "fail.png", "subdir/example.js"] {
-        assert_eq!(assets.iter().filter(|p| p.file_name().unwrap() == "fail.png").count(), 1);
+            assert!(assets
+                .iter()
+                .find(|p| p.strip_prefix(path).unwrap() == Path::new(asset))
+                .is_some())
+        }
     }
 
+    #[test]
+    fn can_find_related_assets_non_recursive() {
+        let tmp_dir = tempdir().expect("create temp dir");
+        let path = tmp_dir.path();
+        File::create(path.join("index.md")).unwrap();
+        File::create(path.join("example.js")).unwrap();
+        File::create(path.join("graph.jpg")).unwrap();
+        File::create(path.join("fail.png")).unwrap();
+        create_dir(path.join("subdir")).expect("create subdir temp dir");
+        File::create(path.join("subdir").join("index.md")).unwrap();
+        File::create(path.join("subdir").join("example.js")).unwrap();
+
+        let assets = find_related_assets(path, &Config::default(), false);
+        assert_eq!(assets.len(), 3);
+        assert_eq!(assets.iter().filter(|p| p.extension().unwrap() != "md").count(), 3);
+
+        for asset in vec!["example.js", "graph.jpg", "fail.png"] {
+            assert!(assets
+                .iter()
+                .find(|p| p.strip_prefix(path).unwrap() == Path::new(asset))
+                .is_some())
+        }
+    }
     #[test]
     fn can_find_anchor_at_root() {
         let input = vec![

components/library/src/content/page.rs

@@ -14,7 +14,7 @@ use front_matter::{split_page_content, InsertAnchor, PageFrontMatter};
 use rendering::{render_content, Heading, RenderContext};
 use utils::site::get_reading_analytics;
 use utils::slugs::slugify_paths;
-use utils::templates::render_template;
+use utils::templates::{render_template, ShortcodeDefinition};
 
 use crate::content::file_info::FileInfo;
 use crate::content::ser::SerializingPage;
@@ -208,7 +208,7 @@ impl Page {
 
         if page.file.name == "index" {
             let parent_dir = path.parent().unwrap();
-            page.assets = find_related_assets(parent_dir, config);
+            page.assets = find_related_assets(parent_dir, config, true);
             page.serialized_assets = page.serialize_assets(base_path);
         } else {
             page.assets = vec![];
@@ -225,6 +225,7 @@ impl Page {
         tera: &Tera,
         config: &Config,
         anchor_insert: InsertAnchor,
+        shortcode_definitions: &HashMap<String, ShortcodeDefinition>,
     ) -> Result<()> {
         let mut context = RenderContext::new(
             tera,
@@ -234,7 +235,8 @@ impl Page {
             permalinks,
             anchor_insert,
         );
-
+        context.set_shortcode_definitions(shortcode_definitions);
+        context.set_current_page_path(&self.file.relative);
         context.tera_context.insert("page", &SerializingPage::from_page_basic(self, None));
 
         let res = render_content(&self.raw_content, &context).map_err(|e| {
@@ -276,7 +278,7 @@ impl Page {
     fn serialize_assets(&self, base_path: &Path) -> Vec<String> {
         self.assets
             .iter()
-            .filter_map(|asset| asset.file_name())
+            .filter_map(|asset| asset.strip_prefix(&self.file.path.parent().unwrap()).ok())
             .filter_map(|filename| filename.to_str())
             .map(|filename| {
                 let mut path = self.file.path.clone();
@@ -336,8 +338,14 @@ Hello world"#;
         let res = Page::parse(Path::new("post.md"), content, &config, &PathBuf::new());
         assert!(res.is_ok());
         let mut page = res.unwrap();
-        page.render_markdown(&HashMap::default(), &Tera::default(), &config, InsertAnchor::None)
+        page.render_markdown(
-            .unwrap();
+            &HashMap::default(),
+            &Tera::default(),
+            &config,
+            InsertAnchor::None,
+            &HashMap::new(),
+        )
+        .unwrap();
 
         assert_eq!(page.meta.title.unwrap(), "Hello".to_string());
         assert_eq!(page.meta.slug.unwrap(), "hello-world".to_string());
@@ -502,8 +510,14 @@ Hello world
         let res = Page::parse(Path::new("hello.md"), &content, &config, &PathBuf::new());
         assert!(res.is_ok());
         let mut page = res.unwrap();
-        page.render_markdown(&HashMap::default(), &Tera::default(), &config, InsertAnchor::None)
+        page.render_markdown(
-            .unwrap();
+            &HashMap::default(),
+            &Tera::default(),
+            &config,
+            InsertAnchor::None,
+            &HashMap::new(),
+        )
+        .unwrap();
         assert_eq!(page.summary, Some("<p>Hello world</p>\n".to_string()));
     }
 
@@ -526,8 +540,14 @@ And here's another. [^2]
         let res = Page::parse(Path::new("hello.md"), &content, &config, &PathBuf::new());
         assert!(res.is_ok());
         let mut page = res.unwrap();
-        page.render_markdown(&HashMap::default(), &Tera::default(), &config, InsertAnchor::None)
+        page.render_markdown(
-            .unwrap();
+            &HashMap::default(),
+            &Tera::default(),
+            &config,
+            InsertAnchor::None,
+            &HashMap::new(),
+        )
+        .unwrap();
         assert_eq!(
             page.summary,
             Some("<p>This page has footnotes, here\'s one. </p>\n".to_string())

components/library/src/content/section.rs

@@ -10,7 +10,7 @@ use front_matter::{split_section_content, SectionFrontMatter};
 use rendering::{render_content, Heading, RenderContext};
 use utils::fs::read_file;
 use utils::site::get_reading_analytics;
-use utils::templates::render_template;
+use utils::templates::{render_template, ShortcodeDefinition};
 
 use crate::content::file_info::FileInfo;
 use crate::content::ser::SerializingSection;
@@ -122,7 +122,7 @@ impl Section {
         let mut section = Section::parse(path, &content, config, base_path)?;
 
         let parent_dir = path.parent().unwrap();
-        section.assets = find_related_assets(parent_dir, config);
+        section.assets = find_related_assets(parent_dir, config, false);
         section.serialized_assets = section.serialize_assets();
 
         Ok(section)
@@ -147,6 +147,7 @@ impl Section {
         permalinks: &HashMap<String, String>,
         tera: &Tera,
         config: &Config,
+        shortcode_definitions: &HashMap<String, ShortcodeDefinition>,
     ) -> Result<()> {
         let mut context = RenderContext::new(
             tera,
@@ -156,7 +157,8 @@ impl Section {
             permalinks,
             self.meta.insert_anchor_links,
         );
-
+        context.set_shortcode_definitions(shortcode_definitions);
+        context.set_current_page_path(&self.file.relative);
         context.tera_context.insert("section", &SerializingSection::from_section_basic(self, None));
 
         let res = render_content(&self.raw_content, &context).map_err(|e| {
@@ -195,7 +197,7 @@ impl Section {
     fn serialize_assets(&self) -> Vec<String> {
         self.assets
             .iter()
-            .filter_map(|asset| asset.file_name())
+            .filter_map(|asset| asset.strip_prefix(&self.file.path.parent().unwrap()).ok())
             .filter_map(|filename| filename.to_str())
             .map(|filename| format!("{}{}", self.path, filename))
             .collect()

components/library/src/pagination/mod.rs

@@ -6,7 +6,7 @@ use tera::{to_value, Context, Tera, Value};
 
 use config::Config;
 use errors::{Error, Result};
-use utils::templates::render_template;
+use utils::templates::{check_template_fallbacks, render_template};
 
 use crate::content::{Section, SerializingPage, SerializingSection};
 use crate::library::Library;
@@ -94,8 +94,16 @@ impl<'a> Paginator<'a> {
         taxonomy: &'a Taxonomy,
         item: &'a TaxonomyItem,
         library: &'a Library,
+        tera: &Tera,
+        theme: &Option<String>,
     ) -> Paginator<'a> {
         let paginate_by = taxonomy.kind.paginate_by.unwrap();
+        // Check for taxon-specific template, or use generic as fallback.
+        let specific_template = format!("{}/single.html", taxonomy.kind.name);
+        let template = match check_template_fallbacks(&specific_template, tera, theme) {
+            Some(template) => template,
+            None => "taxonomy_single.html",
+        };
         let mut paginator = Paginator {
             all_pages: Cow::Borrowed(&item.pages),
             pagers: Vec::with_capacity(item.pages.len() / paginate_by),
@@ -110,7 +118,7 @@ impl<'a> Paginator<'a> {
                 .clone()
                 .unwrap_or_else(|| "page".to_string()),
             is_index: false,
-            template: format!("{}/single.html", taxonomy.kind.name),
+            template: template.to_string(),
         };
 
         // taxonomy paginators have no sorting so we won't have to reverse
@@ -249,7 +257,7 @@ impl<'a> Paginator<'a> {
 #[cfg(test)]
 mod tests {
     use std::path::PathBuf;
-    use tera::to_value;
+    use tera::{to_value, Tera};
 
     use crate::content::{Page, Section};
     use crate::library::Library;
@@ -408,6 +416,7 @@ mod tests {
     #[test]
     fn test_can_create_paginator_for_taxonomy() {
         let (_, library) = create_library(false, 3, false);
+        let tera = Tera::default();
         let taxonomy_def = TaxonomyConfig {
             name: "tags".to_string(),
             paginate_by: Some(2),
@@ -427,7 +436,7 @@ mod tests {
             permalink: "/tags/".to_string(),
             items: vec![taxonomy_item.clone()],
         };
-        let paginator = Paginator::from_taxonomy(&taxonomy, &taxonomy_item, &library);
+        let paginator = Paginator::from_taxonomy(&taxonomy, &taxonomy_item, &library, &tera, &None);
         assert_eq!(paginator.pagers.len(), 2);
 
         assert_eq!(paginator.pagers[0].index, 1);
@@ -444,6 +453,7 @@ mod tests {
     #[test]
     fn test_can_create_paginator_for_slugified_taxonomy() {
         let (_, library) = create_library(false, 3, false);
+        let tera = Tera::default();
         let taxonomy_def = TaxonomyConfig {
             name: "some tags".to_string(),
             paginate_by: Some(2),
@@ -463,7 +473,7 @@ mod tests {
             permalink: "/some-tags/".to_string(),
             items: vec![taxonomy_item.clone()],
         };
-        let paginator = Paginator::from_taxonomy(&taxonomy, &taxonomy_item, &library);
+        let paginator = Paginator::from_taxonomy(&taxonomy, &taxonomy_item, &library, &tera, &None);
         assert_eq!(paginator.pagers.len(), 2);
 
         assert_eq!(paginator.pagers[0].index, 1);

components/library/src/taxonomies/mod.rs

@@ -7,7 +7,7 @@ use tera::{Context, Tera};
 
 use config::{Config, Taxonomy as TaxonomyConfig};
 use errors::{bail, Error, Result};
-use utils::templates::render_template;
+use utils::templates::{check_template_fallbacks, render_template};
 
 use crate::content::SerializingPage;
 use crate::library::Library;
@@ -202,10 +202,16 @@ impl Taxonomy {
         );
         context.insert("current_path", &format!("/{}/{}/", self.kind.name, item.slug));
 
-        render_template(&format!("{}/single.html", self.kind.name), tera, context, &config.theme)
+        // Check for taxon-specific template, or use generic as fallback.
-            .map_err(|e| {
+        let specific_template = format!("{}/single.html", self.kind.name);
-                Error::chain(format!("Failed to render single term {} page.", self.kind.name), e)
+        let template = match check_template_fallbacks(&specific_template, tera, &config.theme) {
-            })
+            Some(template) => template,
+            None => "taxonomy_single.html",
+        };
+
+        render_template(&template, tera, context, &config.theme).map_err(|e| {
+            Error::chain(format!("Failed to render single term {} page.", self.kind.name), e)
+        })
     }
 
     pub fn render_all_terms(
@@ -224,10 +230,16 @@ impl Taxonomy {
         context.insert("current_url", &config.make_permalink(&self.kind.name));
         context.insert("current_path", &format!("/{}/", self.kind.name));
 
-        render_template(&format!("{}/list.html", self.kind.name), tera, context, &config.theme)
+        // Check for taxon-specific template, or use generic as fallback.
-            .map_err(|e| {
+        let specific_template = format!("{}/list.html", self.kind.name);
-                Error::chain(format!("Failed to render a list of {} page.", self.kind.name), e)
+        let template = match check_template_fallbacks(&specific_template, tera, &config.theme) {
-            })
+            Some(template) => template,
+            None => "taxonomy_list.html",
+        };
+
+        render_template(&template, tera, context, &config.theme).map_err(|e| {
+            Error::chain(format!("Failed to render a list of {} page.", self.kind.name), e)
+        })
     }
 
     pub fn to_serialized<'a>(&'a self, library: &'a Library) -> SerializedTaxonomy<'a> {

components/rendering/benches/all.rs

@@ -5,7 +5,7 @@ use std::collections::HashMap;
 
 use config::Config;
 use front_matter::InsertAnchor;
-use rendering::{render_content, render_shortcodes, RenderContext};
+use rendering::{render_content, RenderContext};
 use tera::Tera;
 
 static CONTENT: &str = r#"
@@ -85,12 +85,13 @@ fn bench_render_content_with_highlighting(b: &mut test::Bencher) {
     let mut tera = Tera::default();
     tera.add_raw_template("shortcodes/youtube.html", "{{id}}").unwrap();
     let permalinks_ctx = HashMap::new();
-    let config = Config::default();
+    let mut config = Config::default();
+    config.markdown.highlight_code = true;
     let current_page_permalink = "";
     let context = RenderContext::new(
         &tera,
         &config,
-        "",
+        &config.default_language,
         current_page_permalink,
         &permalinks_ctx,
         InsertAnchor::None,
@@ -109,7 +110,7 @@ fn bench_render_content_without_highlighting(b: &mut test::Bencher) {
     let context = RenderContext::new(
         &tera,
         &config,
-        "",
+        &config.default_language,
         current_page_permalink,
         &permalinks_ctx,
         InsertAnchor::None,
@@ -117,7 +118,6 @@ fn bench_render_content_without_highlighting(b: &mut test::Bencher) {
     b.iter(|| render_content(CONTENT, &context).unwrap());
 }
 
-#[bench]
 fn bench_render_content_no_shortcode(b: &mut test::Bencher) {
     let tera = Tera::default();
     let content2 = CONTENT.replace(r#"{{ youtube(id="my_youtube_id") }}"#, "");
@@ -128,7 +128,7 @@ fn bench_render_content_no_shortcode(b: &mut test::Bencher) {
     let context = RenderContext::new(
         &tera,
         &config,
-        "",
+        &config.default_language,
         current_page_permalink,
         &permalinks_ctx,
         InsertAnchor::None,
@@ -138,26 +138,7 @@ fn bench_render_content_no_shortcode(b: &mut test::Bencher) {
 }
 
 #[bench]
-fn bench_render_shortcodes_one_present(b: &mut test::Bencher) {
+fn bench_render_content_with_emoji(b: &mut test::Bencher) {
-    let mut tera = Tera::default();
-    tera.add_raw_template("shortcodes/youtube.html", "{{id}}").unwrap();
-    let config = Config::default();
-    let permalinks_ctx = HashMap::new();
-    let current_page_permalink = "";
-    let context = RenderContext::new(
-        &tera,
-        &config,
-        "",
-        current_page_permalink,
-        &permalinks_ctx,
-        InsertAnchor::None,
-    );
-
-    b.iter(|| render_shortcodes(CONTENT, &context));
-}
-
-#[bench]
-fn bench_render_content_no_shortcode_with_emoji(b: &mut test::Bencher) {
     let tera = Tera::default();
     let content2 = CONTENT.replace(r#"{{ youtube(id="my_youtube_id") }}"#, "");
     let mut config = Config::default();
@@ -168,7 +149,7 @@ fn bench_render_content_no_shortcode_with_emoji(b: &mut test::Bencher) {
     let context = RenderContext::new(
         &tera,
         &config,
-        "",
+        &config.default_language,
         current_page_permalink,
         &permalinks_ctx,
         InsertAnchor::None,

components/rendering/src/codeblock/highlight.rs

@@ -26,7 +26,7 @@ pub(crate) struct ClassHighlighter<'config> {
 }
 
 impl<'config> ClassHighlighter<'config> {
-    pub fn new(syntax: &'config SyntaxReference, syntax_set: &'config SyntaxSet) -> Self {
+    pub fn new(syntax: &SyntaxReference, syntax_set: &'config SyntaxSet) -> Self {
         let parse_state = ParseState::new(syntax);
         Self { syntax_set, open_spans: 0, parse_state, scope_stack: ScopeStack::new() }
     }
@@ -236,7 +236,7 @@ mod tests {
         let syntax_and_theme = resolve_syntax_and_theme(Some("py"), &config);
         let mut highlighter = SyntaxHighlighter::new(false, syntax_and_theme);
         let mut out = String::new();
-        for line in LinesWithEndings::from(&code) {
+        for line in LinesWithEndings::from(code) {
             out.push_str(&highlighter.highlight_line(line));
         }
         assert!(!out.contains("<script>"));

components/rendering/src/content.pest

@@ -25,7 +25,7 @@ string = @{
 
 boolean = { "true" | "false" }
 
-literal = { boolean | string | float | int }
+literal = { boolean | string | float | int | array }
 array = { "[" ~ (literal  ~ ",")* ~ literal? ~ "]"}
 
 /// Idents
@@ -40,7 +40,7 @@ ident = @{
 
 // shortcode is abbreviated to sc to keep things short
 
-kwarg   = { ident ~ "=" ~ (literal | array) }
+kwarg   = { ident ~ "=" ~ literal }
 kwargs  = _{ kwarg ~ ("," ~ kwarg )* }
 sc_def  = _{ ident ~ "(" ~ kwargs* ~ ")" }
 

components/rendering/src/context.rs

@@ -4,6 +4,7 @@ use std::collections::HashMap;
 use config::Config;
 use front_matter::InsertAnchor;
 use tera::{Context, Tera};
+use utils::templates::ShortcodeDefinition;
 
 /// All the information from the zola site that is needed to render HTML from markdown
 #[derive(Debug)]
@@ -11,9 +12,12 @@ pub struct RenderContext<'a> {
     pub tera: Cow<'a, Tera>,
     pub config: &'a Config,
     pub tera_context: Context,
+    pub current_page_path: Option<&'a str>,
     pub current_page_permalink: &'a str,
     pub permalinks: Cow<'a, HashMap<String, String>>,
     pub insert_anchor: InsertAnchor,
+    pub lang: &'a str,
+    pub shortcode_definitions: Cow<'a, HashMap<String, ShortcodeDefinition>>,
 }
 
 impl<'a> RenderContext<'a> {
@@ -27,25 +31,46 @@ impl<'a> RenderContext<'a> {
     ) -> RenderContext<'a> {
         let mut tera_context = Context::new();
         tera_context.insert("config", &config.serialize(lang));
+        tera_context.insert("lang", lang);
+
         Self {
             tera: Cow::Borrowed(tera),
             tera_context,
+            current_page_path: None,
             current_page_permalink,
             permalinks: Cow::Borrowed(permalinks),
             insert_anchor,
             config,
+            lang,
+            shortcode_definitions: Cow::Owned(HashMap::new()),
         }
     }
 
+    /// Set in another step so we don't add one more arg to new.
+    /// And it's only used when rendering pages/section anyway
+    pub fn set_shortcode_definitions(&mut self, def: &'a HashMap<String, ShortcodeDefinition>) {
+        self.shortcode_definitions = Cow::Borrowed(def);
+    }
+
+    /// Same as above
+    pub fn set_current_page_path(&mut self, path: &'a str) {
+        self.current_page_path = Some(path);
+    }
+
     // In use in the markdown filter
+    // NOTE: This RenderContext is not i18n-aware, see MarkdownFilter::filter for details
+    // If this function is ever used outside of MarkdownFilter, take this into consideration
     pub fn from_config(config: &'a Config) -> RenderContext<'a> {
         Self {
             tera: Cow::Owned(Tera::default()),
             tera_context: Context::new(),
+            current_page_path: None,
             current_page_permalink: "",
             permalinks: Cow::Owned(HashMap::new()),
             insert_anchor: InsertAnchor::None,
             config,
+            lang: &config.default_language,
+            shortcode_definitions: Cow::Owned(HashMap::new()),
         }
     }
 }

components/rendering/src/lib.rs

@@ -4,20 +4,36 @@ mod markdown;
 mod shortcode;
 mod table_of_contents;
 
+use shortcode::{extract_shortcodes, insert_md_shortcodes};
+
 use errors::Result;
 
 pub use context::RenderContext;
 use markdown::markdown_to_html;
-pub use shortcode::render_shortcodes;
+pub use markdown::Rendered;
 pub use table_of_contents::Heading;
 
 pub fn render_content(content: &str, context: &RenderContext) -> Result<markdown::Rendered> {
-    // Don't do shortcodes if there is nothing like a shortcode in the content
+    // avoid parsing the content if needed
-    if content.contains("{{") || content.contains("{%") {
+    if !content.contains("{{") && !content.contains("{%") {
-        let rendered = render_shortcodes(content, context)?;
+        return markdown_to_html(content, context, Vec::new());
-        let html = markdown_to_html(&rendered, context)?;
-        return Ok(html);
     }
 
-    markdown_to_html(content, context)
+    let definitions = context.shortcode_definitions.as_ref();
+    // Extract all the defined shortcodes
+    let (content, shortcodes) = extract_shortcodes(content, definitions)?;
+
+    // Step 1: we render the MD shortcodes before rendering the markdown so they can get processed
+    let (content, html_shortcodes) =
+        insert_md_shortcodes(content, shortcodes, &context.tera_context, &context.tera)?;
+
+    // Step 2: we render the markdown and the HTML markdown at the same time
+    let html_context = markdown_to_html(&content, context, html_shortcodes)?;
+
+    // TODO: Here issue #1418 could be implemented
+    // if do_warn_about_unprocessed_md {
+    //     warn_about_unprocessed_md(unprocessed_md);
+    // }
+
+    Ok(html_context)
 }

components/rendering/src/markdown.rs

@@ -1,6 +1,5 @@
 use lazy_static::lazy_static;
 use pulldown_cmark as cmark;
-use regex::Regex;
 
 use crate::context::RenderContext;
 use crate::table_of_contents::{make_table_of_contents, Heading};
@@ -12,6 +11,7 @@ use utils::vec::InsertMany;
 
 use self::cmark::{Event, LinkType, Options, Parser, Tag};
 use crate::codeblock::{CodeBlock, FenceSettings};
+use crate::shortcode::{Shortcode, SHORTCODE_PLACEHOLDER};
 
 const CONTINUE_READING: &str = "<span id=\"continue-reading\"></span>";
 const ANCHOR_LINK_TEMPLATE: &str = "anchor-link.html";
@@ -59,28 +59,6 @@ fn find_anchor(anchors: &[String], name: String, level: u16) -> String {
     find_anchor(anchors, name, level + 1)
 }
 
-/// Returns whether the given string starts with a schema.
-///
-/// Although there exists [a list of registered URI schemes][uri-schemes], a link may use arbitrary,
-/// private schemes. This function checks if the given string starts with something that just looks
-/// like a scheme, i.e., a case-insensitive identifier followed by a colon.
-///
-/// [uri-schemes]: https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml
-fn starts_with_schema(s: &str) -> bool {
-    lazy_static! {
-        static ref PATTERN: Regex = Regex::new(r"^[0-9A-Za-z\-]+:").unwrap();
-    }
-
-    PATTERN.is_match(s)
-}
-
-/// Colocated asset links refers to the files in the same directory,
-/// there it should be a filename only
-fn is_colocated_asset_link(link: &str) -> bool {
-    !link.contains('/')  // http://, ftp://, ../ etc
-        && !starts_with_schema(link)
-}
-
 /// Returns whether a link starts with an HTTP(s) scheme.
 fn is_external_link(link: &str) -> bool {
     link.starts_with("http:") || link.starts_with("https:")
@@ -111,14 +89,23 @@ fn fix_link(
                 return Err(format!("Relative link {} not found.", link).into());
             }
         }
-    } else if is_colocated_asset_link(link) {
-        format!("{}{}", context.current_page_permalink, link)
     } else {
         if is_external_link(link) {
             external_links.push(link.to_owned());
+            link.to_owned()
+        } else if link.starts_with("#") {
+            // local anchor without the internal zola path
+            if let Some(current_path) = context.current_page_path {
+                internal_links.push((current_path.to_owned(), Some(link[1..].to_owned())));
+                format!("{}{}", context.current_page_permalink, &link)
+            } else {
+                link.to_string()
+            }
+        } else {
+            link.to_string()
         }
-        link.to_string()
     };
+
     Ok(result)
 }
 
@@ -145,8 +132,7 @@ fn get_heading_refs(events: &[Event]) -> Vec<HeadingRef> {
                 heading_refs.push(HeadingRef::new(i, *level));
             }
             Event::End(Tag::Heading(_)) => {
-                let msg = "Heading end before start?";
+                heading_refs.last_mut().expect("Heading end before start?").end_idx = i;
-                heading_refs.last_mut().expect(msg).end_idx = i;
             }
             _ => (),
         }
@@ -155,7 +141,11 @@ fn get_heading_refs(events: &[Event]) -> Vec<HeadingRef> {
     heading_refs
 }
 
-pub fn markdown_to_html(content: &str, context: &RenderContext) -> Result<Rendered> {
+pub fn markdown_to_html(
+    content: &str,
+    context: &RenderContext,
+    html_shortcodes: Vec<Shortcode>,
+) -> Result<Rendered> {
     lazy_static! {
         static ref EMOJI_REPLACER: gh_emoji::Replacer = gh_emoji::Replacer::new();
     }
@@ -177,9 +167,10 @@ pub fn markdown_to_html(content: &str, context: &RenderContext) -> Result<Render
     let mut internal_links = Vec::new();
     let mut external_links = Vec::new();
 
+    let mut stop_next_end_p = false;
+
     let mut opts = Options::empty();
     let mut has_summary = false;
-    let mut in_html_block = false;
     opts.insert(Options::ENABLE_TABLES);
     opts.insert(Options::ENABLE_FOOTNOTES);
     opts.insert(Options::ENABLE_STRIKETHROUGH);
@@ -189,65 +180,108 @@ pub fn markdown_to_html(content: &str, context: &RenderContext) -> Result<Render
         opts.insert(Options::ENABLE_SMART_PUNCTUATION);
     }
 
+    // we reverse their order so we can pop them easily in order
+    let mut html_shortcodes: Vec<_> = html_shortcodes.into_iter().rev().collect();
+    let mut next_shortcode = html_shortcodes.pop();
+    let contains_shortcode = |txt: &str| -> bool { txt.contains(SHORTCODE_PLACEHOLDER) };
+
     {
-        let mut events = Parser::new_ext(content, opts)
+        let mut events = Vec::new();
-            .map(|event| {
+
-                match event {
+        for (event, mut range) in Parser::new_ext(content, opts).into_offset_iter() {
-                    Event::Text(text) => {
+            match event {
-                        // if we are in the middle of a highlighted code block
+                Event::Text(text) => {
-                        if let Some(ref mut code_block) = code_block {
+                    if let Some(ref mut code_block) = code_block {
-                            let html = code_block.highlight(&text);
+                        let html = code_block.highlight(&text);
-                            Event::Html(html.into())
+                        events.push(Event::Html(html.into()));
-                        } else if context.config.markdown.render_emoji {
+                    } else {
-                            let processed_text = EMOJI_REPLACER.replace_all(&text);
+                        let text = if context.config.markdown.render_emoji {
-                            Event::Text(processed_text.to_string().into())
+                            EMOJI_REPLACER.replace_all(&text).to_string().into()
                         } else {
-                            // Business as usual
+                            text
-                            Event::Text(text)
-                        }
-                    }
-                    Event::Start(Tag::CodeBlock(ref kind)) => {
-                        let fence = match kind {
-                            cmark::CodeBlockKind::Fenced(fence_info) => {
-                                FenceSettings::new(fence_info)
-                            }
-                            _ => FenceSettings::new(""),
                         };
-                        let (block, begin) = CodeBlock::new(fence, context.config, path);
+
-                        code_block = Some(block);
+                        if !contains_shortcode(text.as_ref()) {
-                        Event::Html(begin.into())
+                            events.push(Event::Text(text));
-                    }
+                            continue;
-                    Event::End(Tag::CodeBlock(_)) => {
-                        // reset highlight and close the code block
-                        code_block = None;
-                        Event::Html("</code></pre>\n".into())
-                    }
-                    Event::Start(Tag::Image(link_type, src, title)) => {
-                        if is_colocated_asset_link(&src) {
-                            let link = format!("{}{}", context.current_page_permalink, &*src);
-                            return Event::Start(Tag::Image(link_type, link.into(), title));
                         }
 
-                        Event::Start(Tag::Image(link_type, src, title))
+                        // TODO: find a way to share that code with the HTML handler
-                    }
+                        let mut new_text = text.clone();
-                    Event::Start(Tag::Link(link_type, link, title)) if link.is_empty() => {
+                        loop {
-                        error = Some(Error::msg("There is a link that is missing a URL"));
+                            if let Some(ref shortcode) = next_shortcode {
-                        Event::Start(Tag::Link(link_type, "#".into(), title))
+                                let sc_span = shortcode.span.clone();
-                    }
+                                if range.contains(&sc_span.start) {
-                    Event::Start(Tag::Link(link_type, link, title)) => {
+                                    if range.start != sc_span.start {
-                        let fixed_link = match fix_link(
+                                        events.push(Event::Text(
-                            link_type,
+                                            new_text[..(sc_span.start - range.start)]
-                            &link,
+                                                .to_string()
-                            context,
+                                                .into(),
-                            &mut internal_links,
+                                        ));
-                            &mut external_links,
+                                    }
-                        ) {
+
-                            Ok(fixed_link) => fixed_link,
+                                    let shortcode = next_shortcode.take().unwrap();
-                            Err(err) => {
+
-                                error = Some(err);
+                                    match shortcode.render(&context.tera, &context.tera_context) {
-                                return Event::Html("".into());
+                                        Ok(s) => {
+                                            events.push(Event::Html(s.into()));
+                                            new_text = new_text[(sc_span.end - range.start)..]
+                                                .to_owned()
+                                                .into();
+                                            range.start = sc_span.end - range.start;
+                                        }
+                                        Err(e) => {
+                                            error = Some(e);
+                                            break;
+                                        }
+                                    }
+
+                                    next_shortcode = html_shortcodes.pop();
+                                    continue;
+                                }
                             }
-                        };
+
+                            break;
+                        }
+
+                        events.push(Event::Text(new_text[..].to_string().into()));
+                    }
+                }
+                Event::Start(Tag::CodeBlock(ref kind)) => {
+                    let fence = match kind {
+                        cmark::CodeBlockKind::Fenced(fence_info) => FenceSettings::new(fence_info),
+                        _ => FenceSettings::new(""),
+                    };
+                    let (block, begin) = CodeBlock::new(fence, context.config, path);
+                    code_block = Some(block);
+                    events.push(Event::Html(begin.into()));
+                }
+                Event::End(Tag::CodeBlock(_)) => {
+                    // reset highlight and close the code block
+                    code_block = None;
+                    events.push(Event::Html("</code></pre>\n".into()));
+                }
+                Event::Start(Tag::Link(link_type, link, title)) if link.is_empty() => {
+                    error = Some(Error::msg("There is a link that is missing a URL"));
+                    events.push(Event::Start(Tag::Link(link_type, "#".into(), title)));
+                }
+                Event::Start(Tag::Link(link_type, link, title)) => {
+                    let fixed_link = match fix_link(
+                        link_type,
+                        &link,
+                        context,
+                        &mut internal_links,
+                        &mut external_links,
+                    ) {
+                        Ok(fixed_link) => fixed_link,
+                        Err(err) => {
+                            error = Some(err);
+                            events.push(Event::Html("".into()));
+                            continue;
+                        }
+                    };
+
+                    events.push(
                         if is_external_link(&link)
                             && context.config.markdown.has_external_link_tweaks()
                         {
@@ -264,32 +298,93 @@ pub fn markdown_to_html(content: &str, context: &RenderContext) -> Result<Render
                             )
                         } else {
                             Event::Start(Tag::Link(link_type, fixed_link.into(), title))
-                        }
+                        },
-                    }
+                    )
-                    Event::Html(ref markup) => {
-                        if markup.contains("<!-- more -->") {
-                            has_summary = true;
-                            Event::Html(CONTINUE_READING.into())
-                        } else if in_html_block && markup.contains("</pre>") {
-                            in_html_block = false;
-                            Event::Html(markup.replacen("</pre>", "", 1).into())
-                        } else if markup.contains("pre data-shortcode") {
-                            in_html_block = true;
-                            let m = markup.replacen("<pre data-shortcode>", "", 1);
-                            if m.contains("</pre>") {
-                                in_html_block = false;
-                                Event::Html(m.replacen("</pre>", "", 1).into())
-                            } else {
-                                Event::Html(m.into())
-                            }
-                        } else {
-                            event
-                        }
-                    }
-                    _ => event,
                 }
+                Event::Start(Tag::Paragraph) => {
+                    // We have to compare the start and the trimmed length because the content
+                    // will sometimes contain '\n' at the end which we want to avoid.
+                    //
+                    // NOTE: It could be more efficient to remove this search and just keep
+                    // track of the shortcodes to come and compare it to that.
+                    if let Some(ref next_shortcode) = next_shortcode {
+                        if next_shortcode.span.start == range.start
+                            && next_shortcode.span.len() == content[range].trim().len()
+                        {
+                            stop_next_end_p = true;
+                            events.push(Event::Html("".into()));
+                            continue;
+                        }
+                    }
+
+                    events.push(event);
+                }
+                Event::End(Tag::Paragraph) => {
+                    events.push(if stop_next_end_p {
+                        stop_next_end_p = false;
+                        Event::Html("".into())
+                    } else {
+                        event
+                    });
+                }
+                Event::Html(text) => {
+                    if text.contains("<!-- more -->") {
+                        has_summary = true;
+                        events.push(Event::Html(CONTINUE_READING.into()));
+                        continue;
+                    }
+                    if !contains_shortcode(text.as_ref()) {
+                        events.push(Event::Html(text));
+                        continue;
+                    }
+
+                    let mut new_text = text.clone();
+                    loop {
+                        if let Some(ref shortcode) = next_shortcode {
+                            let sc_span = shortcode.span.clone();
+                            if range.contains(&sc_span.start) {
+                                if range.start != sc_span.start {
+                                    events.push(Event::Html(
+                                        new_text[..(sc_span.start - range.start)].to_owned().into(),
+                                    ));
+                                }
+
+                                let shortcode = next_shortcode.take().unwrap();
+                                match shortcode.render(&context.tera, &context.tera_context) {
+                                    Ok(s) => {
+                                        events.push(Event::Html(s.into()));
+                                        new_text = new_text[(sc_span.end - range.start)..]
+                                            .to_owned()
+                                            .into();
+                                        range.start = sc_span.end - range.start;
+                                    }
+                                    Err(e) => {
+                                        error = Some(e);
+                                        break;
+                                    }
+                                }
+
+                                next_shortcode = html_shortcodes.pop();
+                                continue;
+                            }
+                        }
+
+                        break;
+                    }
+                    events.push(Event::Html(new_text[..].to_string().into()));
+                }
+                _ => events.push(event),
+            }
+        }
+
+        // We remove all the empty things we might have pushed before so we don't get some random \n
+        events = events
+            .into_iter()
+            .filter(|e| match e {
+                Event::Text(text) | Event::Html(text) => !text.is_empty(),
+                _ => true,
             })
-            .collect::<Vec<_>>(); // We need to collect the events to make a second pass
+            .collect();
 
         let mut heading_refs = get_heading_refs(&events);
 
@@ -342,6 +437,7 @@ pub fn markdown_to_html(content: &str, context: &RenderContext) -> Result<Render
                 let mut c = tera::Context::new();
                 c.insert("id", &id);
                 c.insert("level", &heading_ref.level);
+                c.insert("lang", &context.lang);
 
                 let anchor_link = utils::templates::render_template(
                     ANCHOR_LINK_TEMPLATE,
@@ -384,25 +480,6 @@ pub fn markdown_to_html(content: &str, context: &RenderContext) -> Result<Render
 mod tests {
     use super::*;
 
-    #[test]
-    fn test_starts_with_schema() {
-        // registered
-        assert!(starts_with_schema("https://example.com/"));
-        assert!(starts_with_schema("ftp://example.com/"));
-        assert!(starts_with_schema("mailto:user@example.com"));
-        assert!(starts_with_schema("xmpp:node@example.com"));
-        assert!(starts_with_schema("tel:18008675309"));
-        assert!(starts_with_schema("sms:18008675309"));
-        assert!(starts_with_schema("h323:user@example.com"));
-
-        // arbitrary
-        assert!(starts_with_schema("zola:post?content=hi"));
-
-        // case-insensitive
-        assert!(starts_with_schema("MailTo:user@example.com"));
-        assert!(starts_with_schema("MAILTO:user@example.com"));
-    }
-
     #[test]
     fn test_is_external_link() {
         assert!(is_external_link("http://example.com/"));

components/rendering/src/shortcode/mod.rs

@@ -0,0 +1,120 @@
+use std::collections::HashMap;
+
+use errors::{Error, Result};
+use utils::templates::{ShortcodeDefinition, ShortcodeFileType};
+
+mod parser;
+
+pub(crate) use parser::{parse_for_shortcodes, Shortcode, SHORTCODE_PLACEHOLDER};
+
+/// Extracts the shortcodes present in the source, check if we know them and errors otherwise
+pub fn extract_shortcodes(
+    source: &str,
+    definitions: &HashMap<String, ShortcodeDefinition>,
+) -> Result<(String, Vec<Shortcode>)> {
+    let (out, mut shortcodes) = parse_for_shortcodes(source)?;
+
+    for sc in &mut shortcodes {
+        if let Some(def) = definitions.get(&sc.name) {
+            sc.tera_name = def.tera_name.clone();
+        } else {
+            return Err(Error::msg(format!("Found usage of a shortcode named `{}` but we do not know about. Make sure it's not a typo and that a field name `{}.{{html,md}} exists in the `templates/shortcodes` directory.", sc.name, sc.name)));
+        }
+    }
+
+    Ok((out, shortcodes))
+}
+
+pub fn insert_md_shortcodes(
+    mut content: String,
+    shortcodes: Vec<Shortcode>,
+    tera_context: &tera::Context,
+    tera: &tera::Tera,
+) -> Result<(String, Vec<Shortcode>)> {
+    // (span, len transformed)
+    let mut transforms = Vec::new();
+    let mut html_shortcodes = Vec::with_capacity(shortcodes.len());
+
+    for mut sc in shortcodes.into_iter() {
+        for (md_sc_span, rendered_length) in &transforms {
+            sc.update_range(md_sc_span, *rendered_length);
+        }
+        // It has been checked before that this exist
+        if sc.file_type() == ShortcodeFileType::Html {
+            html_shortcodes.push(sc);
+            continue;
+        }
+
+        let span = sc.span.clone();
+        let res = sc.render(tera, tera_context)?;
+        transforms.push((span.clone(), res.len()));
+        content.replace_range(span, &res);
+    }
+
+    Ok((content, html_shortcodes))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::shortcode::SHORTCODE_PLACEHOLDER;
+    use tera::to_value;
+
+    #[test]
+    fn can_insert_md_shortcodes() {
+        let mut tera = templates::ZOLA_TERA.clone();
+
+        tera.add_raw_template("shortcodes/a.md", "{{ nth }}").unwrap();
+        tera.add_raw_template("shortcodes/bodied.md", "{{ body }}").unwrap();
+
+        let tera_context = tera::Context::new();
+        assert_eq!(
+            insert_md_shortcodes(
+                format!("{}{}", SHORTCODE_PLACEHOLDER, SHORTCODE_PLACEHOLDER),
+                vec![
+                    Shortcode {
+                        name: "a".to_string(),
+                        args: to_value(&HashMap::<u8, u8>::new()).unwrap(),
+                        span: 0..SHORTCODE_PLACEHOLDER.len(),
+                        body: None,
+                        nth: 1,
+                        tera_name: "shortcodes/a.md".to_owned(),
+                    },
+                    Shortcode {
+                        name: "a".to_string(),
+                        args: to_value(&HashMap::<u8, u8>::new()).unwrap(),
+                        span: SHORTCODE_PLACEHOLDER.len()..(2 * SHORTCODE_PLACEHOLDER.len()),
+                        body: None,
+                        nth: 2,
+                        tera_name: "shortcodes/a.md".to_owned(),
+                    }
+                ],
+                &tera_context,
+                &tera
+            )
+            .unwrap()
+            .0,
+            "12".to_string()
+        );
+
+        assert_eq!(
+            insert_md_shortcodes(
+                format!("Much wow {}", SHORTCODE_PLACEHOLDER),
+                vec![Shortcode {
+                    name: "bodied".to_string(),
+                    args: to_value(&HashMap::<u8, u8>::new()).unwrap(),
+                    span: 9..(9 + SHORTCODE_PLACEHOLDER.len()),
+                    body: Some("Content of the body".to_owned()),
+                    nth: 1,
+
+                    tera_name: "shortcodes/bodied.md".to_owned(),
+                },],
+                &tera_context,
+                &tera
+            )
+            .unwrap()
+            .0,
+            "Much wow Content of the body".to_string()
+        );
+    }
+}

components/rendering/src/shortcode/parser.rs

@@ -1,26 +1,79 @@
-use lazy_static::lazy_static;
+use std::ops::Range;
+
+use errors::{bail, Result};
 use pest::iterators::Pair;
 use pest::Parser;
 use pest_derive::Parser;
-use regex::Regex;
 use std::collections::HashMap;
-use tera::{to_value, Context, Map, Value};
+use tera::{to_value, Context, Map, Tera, Value};
+use utils::templates::ShortcodeFileType;
 
-use crate::context::RenderContext;
+pub const SHORTCODE_PLACEHOLDER: &str = "||ZOLA_SC_PLACEHOLDER||";
-use errors::{bail, Error, Result};
+
+#[derive(PartialEq, Debug)]
+pub struct Shortcode {
+    pub(crate) name: String,
+    pub(crate) args: Value,
+    pub(crate) span: Range<usize>,
+    pub(crate) body: Option<String>,
+    pub(crate) nth: usize,
+    // set later down the line, for quick access without needing the definitions
+    pub(crate) tera_name: String,
+}
+
+impl Shortcode {
+    pub fn file_type(&self) -> ShortcodeFileType {
+        if self.tera_name.ends_with("md") {
+            ShortcodeFileType::Markdown
+        } else {
+            ShortcodeFileType::Html
+        }
+    }
+
+    pub fn render(self, tera: &Tera, context: &Context) -> Result<String> {
+        let name = self.name;
+        let tpl_name = self.tera_name;
+        let mut new_context = Context::from_value(self.args)?;
+
+        if let Some(body_content) = self.body {
+            // Trimming right to avoid most shortcodes with bodies ending up with a HTML new line
+            new_context.insert("body", body_content.trim_end());
+        }
+        new_context.insert("nth", &self.nth);
+        new_context.extend(context.clone());
+
+        let res = utils::templates::render_template(&tpl_name, tera, new_context, &None)
+            .map_err(|e| errors::Error::chain(format!("Failed to render {} shortcode", name), e))?
+            .replace("\r\n", "\n");
+
+        Ok(res)
+    }
+
+    pub fn update_range(&mut self, sc_span: &Range<usize>, rendered_length: usize) {
+        if self.span.start > sc_span.start {
+            let delta = if sc_span.end < rendered_length {
+                rendered_length - sc_span.end
+            } else {
+                sc_span.end - rendered_length
+            };
+
+            if sc_span.end < rendered_length {
+                self.span = (self.span.start + delta)..(self.span.end + delta);
+            } else {
+                self.span = (self.span.start - delta)..(self.span.end - delta);
+            }
+        }
+    }
+}
 
 // This include forces recompiling this source file if the grammar file changes.
 // Uncomment it when doing changes to the .pest file
-const _GRAMMAR: &str = include_str!("content.pest");
+const _GRAMMAR: &str = include_str!("../content.pest");
 
 #[derive(Parser)]
 #[grammar = "content.pest"]
 pub struct ContentParser;
 
-lazy_static! {
-    static ref OUTER_NEWLINE_RE: Regex = Regex::new(r"^\s*\n|\n\s*$").unwrap();
-}
-
 fn replace_string_markers(input: &str) -> String {
     match input.chars().next().unwrap() {
         '"' => input.replace('"', ""),
@@ -30,7 +83,7 @@ fn replace_string_markers(input: &str) -> String {
     }
 }
 
-fn parse_literal(pair: Pair<Rule>) -> Value {
+fn parse_kwarg_value(pair: Pair<Rule>) -> Value {
     let mut val = None;
     for p in pair.into_inner() {
         match p.as_rule() {
@@ -46,6 +99,16 @@ fn parse_literal(pair: Pair<Rule>) -> Value {
             Rule::int => {
                 val = Some(to_value(p.as_str().parse::<i64>().unwrap()).unwrap());
             }
+            Rule::array => {
+                let mut vals = vec![];
+                for p2 in p.into_inner() {
+                    match p2.as_rule() {
+                        Rule::literal => vals.push(parse_kwarg_value(p2)),
+                        _ => unreachable!("Got something other than literal in an array: {:?}", p2),
+                    }
+                }
+                val = Some(Value::Array(vals));
+            }
             _ => unreachable!("Unknown literal: {:?}", p),
         };
     }
@@ -54,7 +117,7 @@ fn parse_literal(pair: Pair<Rule>) -> Value {
 }
 
 /// Returns (shortcode_name, kwargs)
-fn parse_shortcode_call(pair: Pair<Rule>) -> (String, Map<String, Value>) {
+fn parse_shortcode_call(pair: Pair<Rule>) -> (String, Value) {
     let mut name = None;
     let mut args = Map::new();
 
@@ -72,20 +135,7 @@ fn parse_shortcode_call(pair: Pair<Rule>) -> (String, Map<String, Value>) {
                             arg_name = Some(p2.as_span().as_str().to_string());
                         }
                         Rule::literal => {
-                            arg_val = Some(parse_literal(p2));
+                            arg_val = Some(parse_kwarg_value(p2));
-                        }
-                        Rule::array => {
-                            let mut vals = vec![];
-                            for p3 in p2.into_inner() {
-                                match p3.as_rule() {
-                                    Rule::literal => vals.push(parse_literal(p3)),
-                                    _ => unreachable!(
-                                        "Got something other than literal in an array: {:?}",
-                                        p3
-                                    ),
-                                }
-                            }
-                            arg_val = Some(Value::Array(vals));
                         }
                         _ => unreachable!("Got something unexpected in a kwarg: {:?}", p2),
                     }
@@ -96,58 +146,18 @@ fn parse_shortcode_call(pair: Pair<Rule>) -> (String, Map<String, Value>) {
             _ => unreachable!("Got something unexpected in a shortcode: {:?}", p),
         }
     }
-    (name.unwrap(), args)
+    (name.unwrap(), Value::Object(args))
 }
 
-fn render_shortcode(
+pub fn parse_for_shortcodes(content: &str) -> Result<(String, Vec<Shortcode>)> {
-    name: &str,
+    let mut shortcodes = Vec::new();
-    args: &Map<String, Value>,
+    let mut nths = HashMap::new();
-    context: &RenderContext,
-    invocation_count: u32,
-    body: Option<&str>,
-) -> Result<String> {
-    let mut tera_context = Context::new();
-    for (key, value) in args.iter() {
-        tera_context.insert(key, value);
-    }
-    if let Some(b) = body {
-        // Trimming right to avoid most shortcodes with bodies ending up with a HTML new line
-        tera_context.insert("body", b.trim_end());
-    }
-    tera_context.insert("nth", &invocation_count);
-    tera_context.extend(context.tera_context.clone());
-
-    let mut template_name = format!("shortcodes/{}.md", name);
-    if !context.tera.templates.contains_key(&template_name) {
-        template_name = format!("shortcodes/{}.html", name);
-    }
-
-    let res = utils::templates::render_template(&template_name, &context.tera, tera_context, &None)
-        .map_err(|e| Error::chain(format!("Failed to render {} shortcode", name), e))?;
-
-    let res = OUTER_NEWLINE_RE.replace_all(&res, "");
-
-    // A blank line will cause the markdown parser to think we're out of HTML and start looking
-    // at indentation, making the output a code block. To avoid this, newlines are replaced with
-    // "<!--\n-->" at this stage, which will be undone after markdown rendering in lib.rs. Since
-    // that is an HTML comment, it shouldn't be rendered anyway. and not cause problems unless
-    // someone wants to include that comment in their content. This behaviour is unwanted in when
-    // rendering markdown shortcodes.
-    if template_name.ends_with(".html") {
-        Ok(format!("<pre data-shortcode>{}</pre>", res))
-    } else {
-        Ok(res.to_string())
-    }
-}
-
-pub fn render_shortcodes(content: &str, context: &RenderContext) -> Result<String> {
-    let mut res = String::with_capacity(content.len());
-    let mut invocation_map: HashMap<String, u32> = HashMap::new();
     let mut get_invocation_count = |name: &str| {
-        let invocation_number = invocation_map.entry(String::from(name)).or_insert(0);
+        let nth = nths.entry(String::from(name)).or_insert(0);
-        *invocation_number += 1;
+        *nth += 1;
-        *invocation_number
+        *nth
     };
+    let mut output = String::with_capacity(content.len());
 
     let mut pairs = match ContentParser::parse(Rule::page, content) {
         Ok(p) => p,
@@ -190,33 +200,41 @@ pub fn render_shortcodes(content: &str, context: &RenderContext) -> Result<Strin
     // We have at least a `page` pair
     for p in pairs.next().unwrap().into_inner() {
         match p.as_rule() {
-            Rule::text => res.push_str(p.as_span().as_str()),
+            Rule::text => output.push_str(p.as_span().as_str()),
             Rule::inline_shortcode => {
+                let start = output.len();
                 let (name, args) = parse_shortcode_call(p);
-                res.push_str(&render_shortcode(
+                let nth = get_invocation_count(&name);
-                    &name,
+                shortcodes.push(Shortcode {
-                    &args,
+                    name,
-                    context,
+                    args,
-                    get_invocation_count(&name),
+                    span: start..(start + SHORTCODE_PLACEHOLDER.len()),
-                    None,
+                    body: None,
-                )?);
+                    nth,
+                    tera_name: String::new(),
+                });
+                output.push_str(SHORTCODE_PLACEHOLDER);
             }
             Rule::shortcode_with_body => {
+                let start = output.len();
                 let mut inner = p.into_inner();
                 // 3 items in inner: call, body, end
                 // we don't care about the closing tag
                 let (name, args) = parse_shortcode_call(inner.next().unwrap());
-                let body = inner.next().unwrap().as_span().as_str();
+                let body = inner.next().unwrap().as_span().as_str().trim();
-                res.push_str(&render_shortcode(
+                let nth = get_invocation_count(&name);
-                    &name,
+                shortcodes.push(Shortcode {
-                    &args,
+                    name,
-                    context,
+                    args,
-                    get_invocation_count(&name),
+                    span: start..(start + SHORTCODE_PLACEHOLDER.len()),
-                    Some(body),
+                    body: Some(body.to_string()),
-                )?);
+                    nth,
+                    tera_name: String::new(),
+                });
+                output.push_str(SHORTCODE_PLACEHOLDER)
             }
             Rule::ignored_inline_shortcode => {
-                res.push_str(
+                output.push_str(
                     &p.as_span().as_str().replacen("{{/*", "{{", 1).replacen("*/}}", "}}", 1),
                 );
             }
@@ -224,14 +242,14 @@ pub fn render_shortcodes(content: &str, context: &RenderContext) -> Result<Strin
                 for p2 in p.into_inner() {
                     match p2.as_rule() {
                         Rule::ignored_sc_body_start | Rule::ignored_sc_body_end => {
-                            res.push_str(
+                            output.push_str(
                                 &p2.as_span()
                                     .as_str()
                                     .replacen("{%/*", "{%", 1)
                                     .replacen("*/%}", "%}", 1),
                             );
                         }
-                        Rule::text_in_ignored_body_sc => res.push_str(p2.as_span().as_str()),
+                        Rule::text_in_ignored_body_sc => output.push_str(p2.as_span().as_str()),
                         _ => unreachable!("Got something weird in an ignored shortcode: {:?}", p2),
                     }
                 }
@@ -241,17 +259,12 @@ pub fn render_shortcodes(content: &str, context: &RenderContext) -> Result<Strin
         }
     }
 
-    Ok(res)
+    Ok((output, shortcodes))
 }
 
 #[cfg(test)]
 mod tests {
-    use std::collections::HashMap;
-
     use super::*;
-    use config::Config;
-    use front_matter::InsertAnchor;
-    use tera::Tera;
 
     macro_rules! assert_lex_rule {
         ($rule: expr, $input: expr) => {
@@ -267,20 +280,6 @@ mod tests {
         };
     }
 
-    fn render_shortcodes(code: &str, tera: &Tera) -> String {
-        let config = Config::default_for_test();
-        let permalinks = HashMap::new();
-        let context = RenderContext::new(
-            tera,
-            &config,
-            &config.default_language,
-            "",
-            &permalinks,
-            InsertAnchor::None,
-        );
-        super::render_shortcodes(code, &context).unwrap()
-    }
-
     #[test]
     fn lex_text() {
         let inputs = vec!["Hello world", "HEllo \n world", "Hello 1 2 true false 'hey'"];
@@ -294,7 +293,8 @@ mod tests {
         let inputs = vec![
             "{{ youtube() }}",
             "{{ youtube(id=1, autoplay=true, url='hey') }}",
-            "{{ youtube(id=1, \nautoplay=true, url='hey') }}",
+            "{{ youtube(id=1, \nautoplay=true, url='hey', array=[]) }}",
+            "{{ youtube(id=1, \nautoplay=true, url='hey', multi_aray=[[]]) }}",
         ];
         for i in inputs {
             assert_lex_rule!(Rule::inline_shortcode, i);
@@ -364,119 +364,112 @@ mod tests {
     }
 
     #[test]
-    fn does_nothing_with_no_shortcodes() {
+    fn can_update_ranges() {
-        let res = render_shortcodes("Hello World", &Tera::default());
+        let mut sc = Shortcode {
-        assert_eq!(res, "Hello World");
+            name: "a".to_string(),
+            args: Value::Null,
+            span: 10..20,
+            body: None,
+            nth: 0,
+            tera_name: String::new(),
+        };
+        sc.update_range(&(2..8), 10);
+        assert_eq!(sc.span, 12..22);
+        sc.update_range(&(24..30), 30);
+        assert_eq!(sc.span, 12..22);
+        sc.update_range(&(5..11), 6);
+        assert_eq!(sc.span, 7..17);
     }
 
     #[test]
-    fn can_unignore_inline_shortcode() {
+    fn can_extract_basic_inline_shortcode_with_args() {
-        let res = render_shortcodes("Hello World {{/* youtube() */}}", &Tera::default());
+        let (out, shortcodes) = parse_for_shortcodes(
-        assert_eq!(res, "Hello World {{ youtube() }}");
+            "Inline shortcode: {{ hello(string='hey', int=1, float=2.1, bool=true, array=[true, false]) }} hey",
-    }
-
-    #[test]
-    fn can_unignore_shortcode_with_body() {
-        let res = render_shortcodes(
-            r#"
-Hello World
-{%/* youtube() */%}Some body {{ hello() }}{%/* end */%}"#,
-            &Tera::default(),
-        );
-        assert_eq!(res, "\nHello World\n{% youtube() %}Some body {{ hello() }}{% end %}");
-    }
-
-    // https://github.com/Keats/gutenberg/issues/383
-    #[test]
-    fn unignore_shortcode_with_body_does_not_swallow_initial_whitespace() {
-        let res = render_shortcodes(
-            r#"
-Hello World
-{%/* youtube() */%}
-Some body {{ hello() }}{%/* end */%}"#,
-            &Tera::default(),
-        );
-        assert_eq!(res, "\nHello World\n{% youtube() %}\nSome body {{ hello() }}{% end %}");
-    }
-
-    #[test]
-    fn can_parse_shortcode_arguments() {
-        let inputs = vec![
-            ("{{ youtube() }}", "youtube", Map::new()),
-            ("{{ youtube(id=1, autoplay=true, hello='salut', float=1.2) }}", "youtube", {
-                let mut m = Map::new();
-                m.insert("id".to_string(), to_value(1).unwrap());
-                m.insert("autoplay".to_string(), to_value(true).unwrap());
-                m.insert("hello".to_string(), to_value("salut").unwrap());
-                m.insert("float".to_string(), to_value(1.2).unwrap());
-                m
-            }),
-            ("{{ gallery(photos=['something', 'else'], fullscreen=true) }}", "gallery", {
-                let mut m = Map::new();
-                m.insert("photos".to_string(), to_value(["something", "else"]).unwrap());
-                m.insert("fullscreen".to_string(), to_value(true).unwrap());
-                m
-            }),
-        ];
-
-        for (i, n, a) in inputs {
-            let mut res = ContentParser::parse(Rule::inline_shortcode, i).unwrap();
-            let (name, args) = parse_shortcode_call(res.next().unwrap());
-            assert_eq!(name, n);
-            assert_eq!(args, a);
-        }
-    }
-
-    #[test]
-    fn can_render_inline_shortcodes() {
-        let mut tera = Tera::default();
-        tera.add_raw_template("shortcodes/youtube.html", "Hello {{id}}").unwrap();
-        let res = render_shortcodes("Inline {{ youtube(id=1) }}.", &tera);
-        assert_eq!(res, "Inline <pre data-shortcode>Hello 1</pre>.");
-    }
-
-    #[test]
-    fn can_render_shortcodes_with_body() {
-        let mut tera = Tera::default();
-        tera.add_raw_template("shortcodes/youtube.html", "{{body}}").unwrap();
-        let res = render_shortcodes("Body\n {% youtube() %}Hey!{% end %}", &tera);
-        assert_eq!(res, "Body\n <pre data-shortcode>Hey!</pre>");
-    }
-
-    // https://github.com/Keats/gutenberg/issues/462
-    #[test]
-    fn shortcodes_with_body_do_not_eat_newlines() {
-        let mut tera = Tera::default();
-        tera.add_raw_template("shortcodes/youtube.html", "{{body | safe}}").unwrap();
-        let res = render_shortcodes("Body\n {% youtube() %}\nHello \n \n\n World{% end %}", &tera);
-        assert_eq!(res, "Body\n <pre data-shortcode>Hello \n \n\n World</pre>");
-    }
-
-    #[test]
-    fn outer_newlines_removed_from_shortcodes_with_body() {
-        let mut tera = Tera::default();
-        tera.add_raw_template("shortcodes/youtube.html", "  \n  {{body}}  \n  ").unwrap();
-        let res = render_shortcodes("\n{% youtube() %}  \n  content  \n  {% end %}\n", &tera);
-        assert_eq!(res, "\n<pre data-shortcode>  content  </pre>\n");
-    }
-
-    #[test]
-    fn outer_newlines_removed_from_inline_shortcodes() {
-        let mut tera = Tera::default();
-        tera.add_raw_template("shortcodes/youtube.html", "  \n  Hello, Zola.  \n  ").unwrap();
-        let res = render_shortcodes("\n{{ youtube() }}\n", &tera);
-        assert_eq!(res, "\n<pre data-shortcode>  Hello, Zola.  </pre>\n");
-    }
-
-    #[test]
-    fn shortcodes_that_emit_markdown() {
-        let mut tera = Tera::default();
-        tera.add_raw_template(
-            "shortcodes/youtube.md",
-            "{% for i in [1,2,3] %}\n* {{ i }}\n{%- endfor %}",
         )
         .unwrap();
-        let res = render_shortcodes("{{ youtube() }}", &tera);
+        assert_eq!(out, format!("Inline shortcode: {} hey", SHORTCODE_PLACEHOLDER));
-        assert_eq!(res, "* 1\n* 2\n* 3");
+        assert_eq!(shortcodes.len(), 1);
+        assert_eq!(shortcodes[0].name, "hello");
+        assert_eq!(shortcodes[0].args.as_object().unwrap().len(), 5);
+        assert_eq!(shortcodes[0].args["string"], Value::String("hey".to_string()));
+        assert_eq!(shortcodes[0].args["bool"], Value::Bool(true));
+        assert_eq!(shortcodes[0].args["int"], to_value(1).unwrap());
+        assert_eq!(shortcodes[0].args["float"], to_value(2.1).unwrap());
+        assert_eq!(
+            shortcodes[0].args["array"],
+            Value::Array(vec![Value::Bool(true), Value::Bool(false)])
+        );
+        assert_eq!(shortcodes[0].span, 18..(18 + SHORTCODE_PLACEHOLDER.len()));
+        assert_eq!(shortcodes[0].nth, 1);
+    }
+
+    #[test]
+    fn can_unignore_ignored_inline_shortcode() {
+        let (out, shortcodes) =
+            parse_for_shortcodes("Hello World {{/* youtube() */}} hey").unwrap();
+        assert_eq!(out, "Hello World {{ youtube() }} hey");
+        assert_eq!(shortcodes.len(), 0);
+    }
+
+    #[test]
+    fn can_extract_shortcode_with_body() {
+        let (out, shortcodes) = parse_for_shortcodes(
+            "Body shortcode\n {% quote(author='Bobby', array=[[true]]) %}DROP TABLES;{% end %} \n hey",
+        )
+        .unwrap();
+        assert_eq!(out, format!("Body shortcode\n {} \n hey", SHORTCODE_PLACEHOLDER));
+        assert_eq!(shortcodes.len(), 1);
+        assert_eq!(shortcodes[0].name, "quote");
+        assert_eq!(shortcodes[0].args.as_object().unwrap().len(), 2);
+        assert_eq!(shortcodes[0].args["author"], Value::String("Bobby".to_string()));
+        assert_eq!(
+            shortcodes[0].args["array"],
+            Value::Array(vec![Value::Array(vec![Value::Bool(true)])])
+        );
+        assert_eq!(shortcodes[0].body, Some("DROP TABLES;".to_owned()));
+        assert_eq!(shortcodes[0].span, 16..(16 + SHORTCODE_PLACEHOLDER.len()));
+        assert_eq!(shortcodes[0].nth, 1);
+    }
+
+    #[test]
+    fn can_unignore_ignored_shortcode_with_body() {
+        let (out, shortcodes) =
+            parse_for_shortcodes("Hello World {%/* youtube() */%} Somebody {%/* end */%} hey")
+                .unwrap();
+        assert_eq!(out, "Hello World {% youtube() %} Somebody {% end %} hey");
+        assert_eq!(shortcodes.len(), 0);
+    }
+
+    #[test]
+    fn can_extract_multiple_shortcodes_and_increment_nth() {
+        let (out, shortcodes) = parse_for_shortcodes(
+            "Hello World {% youtube() %} Somebody {% end %} {{ hello() }}\n {{hello()}}",
+        )
+        .unwrap();
+        assert_eq!(
+            out,
+            format!(
+                "Hello World {} {}\n {}",
+                SHORTCODE_PLACEHOLDER, SHORTCODE_PLACEHOLDER, SHORTCODE_PLACEHOLDER
+            )
+        );
+        assert_eq!(shortcodes.len(), 3);
+        assert_eq!(shortcodes[0].nth, 1);
+        assert_eq!(shortcodes[1].nth, 1);
+        assert_eq!(shortcodes[2].nth, 2);
+    }
+
+    #[test]
+    fn can_handle_multiple_shortcodes() {
+        let (_, shortcodes) = parse_for_shortcodes(
+            r#"
+        {{ youtube(id="ub36ffWAqgQ") }}
+        {{ youtube(id="ub36ffWAqgQ", autoplay=true) }}
+        {{ vimeo(id="210073083") }}
+        {{ streamable(id="c0ic") }}
+        {{ gist(url="https://gist.github.com/Keats/32d26f699dcc13ebd41b") }}"#,
+        )
+        .unwrap();
+        assert_eq!(shortcodes.len(), 5);
     }
 }

components/rendering/tests/common/mod.rs

@@ -0,0 +1,16 @@
+pub struct ShortCode {
+    pub name: &'static str,
+    pub output: &'static str,
+    pub is_md: bool,
+}
+
+impl ShortCode {
+    pub const fn new(name: &'static str, output: &'static str, is_md: bool) -> ShortCode {
+        ShortCode { name, output, is_md }
+    }
+
+    /// Return filename for shortcode
+    pub fn filename(&self) -> String {
+        format!("{}.{}", self.name, if self.is_md { "md" } else { "html" })
+    }
+}

components/rendering/tests/integration.rs

@@ -0,0 +1,342 @@
+mod common;
+
+use common::ShortCode;
+
+const COMPLETE_PAGE: &str = r#"
+<!-- Adapted from https://markdown-it.github.io/ -->
+
+# h1 Heading
+
+## h2 Heading
+
+### h3 Heading
+
+#### h4 Heading
+
+##### h5 Heading
+
+###### h6 Heading
+
+## Horizontal Rules
+
+___
+
+---
+
+***
+
+## Emphasis
+
+**This is bold text**
+
+__This is bold text__
+
+*This is italic text*
+
+_This is italic text_
+
+~~Strikethrough~~
+
+
+## Blockquotes
+
+
+> Blockquotes can also be nested...
+>> ...by using additional greater-than signs right next to each other...
+> > > ...or with spaces between arrows.
+
+
+## Lists
+
+Unordered
+
++ Create a list by starting a line with `+`, `-`, or `*`
++ Sub-lists are made by indenting 2 spaces:
+  - Marker character change forces new list start:
+    * Ac tristique libero volutpat at
+    + Facilisis in pretium nisl aliquet
+    - Nulla volutpat aliquam velit
++ Very easy!
+
+Ordered
+
+1. Lorem ipsum dolor sit amet
+2. Consectetur adipiscing elit
+3. Integer molestie lorem at massa
+
+
+1. You can use sequential numbers...
+1. ...or keep all the numbers as `1.`
+
+Start numbering with offset:
+
+57. foo
+1. bar
+
+
+## Code
+
+Inline `code`
+
+Indented code
+
+    // Some comments
+    line 1 of code
+    line 2 of code
+    line 3 of code
+
+
+Block code "fences"
+
+```
+Sample text here...
+```
+
+Syntax highlighting
+
+``` js
+var foo = function (bar) {
+  return bar++;
+};
+
+console.log(foo(5));
+```
+
+## Shortcodes
+
+{% quote(author="John Doe") %}
+This is a test quote!
+1900-01-01
+{% end %}
+
+## Tables
+
+| Option | Description |
+| ------ | ----------- |
+| data   | path to data files to supply the data that will be passed into templates. |
+| engine | engine to be used for processing templates. Handlebars is the default. |
+| ext    | extension to be used for dest files. |
+
+Right aligned columns
+
+| Option | Description |
+| ------:| -----------:|
+| data   | path to data files to supply the data that will be passed into templates. |
+| engine | engine to be used for processing templates. Handlebars is the default. |
+| ext    | extension to be used for dest files. |
+
+
+## Links
+
+[link text](http://duckduckgo.com)
+
+[link with title](http://duckduckgo.com/)
+
+## Images
+
+![Minion](https://octodex.github.com/images/minion.png)
+![Stormtroopocat](https://octodex.github.com/images/stormtroopocat.jpg "The Stormtroopocat")
+
+Like links, Images also have a footnote style syntax
+
+![Alt text][id]
+
+With a reference later in the document defining the URL location:
+
+[id]: https://octodex.github.com/images/dojocat.jpg  "The Dojocat"
+
+### Footnotes
+
+Footnote 1 link[^first].
+
+Footnote 2 link[^second].
+
+Duplicated footnote reference[^second].
+
+[^first]: Footnote **can have markup**
+and multiple paragraphs.
+
+[^second]: Footnote text."#;
+
+#[test]
+fn complete_page() {
+    let config = config::Config::default_for_test();
+
+    let mut tera = tera::Tera::default();
+
+    let shortcodes: Vec<ShortCode> = vec![ShortCode::new(
+        "quote",
+        r"<blockquote>
+{{ body }} <br>
+-- {{ author}}
+</blockquote>",
+        false,
+    )];
+
+    let mut permalinks = std::collections::HashMap::new();
+
+    permalinks.insert("".to_string(), "".to_string());
+
+    // Add all shortcodes
+    for ShortCode { name, is_md, output } in shortcodes.into_iter() {
+        tera.add_raw_template(
+            &format!("shortcodes/{}.{}", name, if is_md { "md" } else { "html" }),
+            &output,
+        )
+        .unwrap();
+    }
+
+    let mut context = rendering::RenderContext::new(
+        &tera,
+        &config,
+        &config.default_language,
+        "",
+        &permalinks,
+        front_matter::InsertAnchor::None,
+    );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
+
+    let rendered = rendering::render_content(COMPLETE_PAGE, &context);
+    assert!(rendered.is_ok(), "Rendering failed");
+
+    let rendered = rendered.unwrap();
+
+    let asserted_internal_links: Vec<(String, Option<String>)> = vec![];
+    let asserted_external_links: Vec<String> =
+        vec!["http://duckduckgo.com".to_string(), "http://duckduckgo.com/".to_string()];
+
+    assert_eq!(rendered.internal_links, asserted_internal_links, "Internal links unequal");
+    assert_eq!(rendered.external_links, asserted_external_links, "External links unequal");
+
+    assert_eq!(
+        rendered.body,
+        r##"<!-- Adapted from https://markdown-it.github.io/ -->
+<h1 id="h1-heading">h1 Heading</h1>
+<h2 id="h2-heading">h2 Heading</h2>
+<h3 id="h3-heading">h3 Heading</h3>
+<h4 id="h4-heading">h4 Heading</h4>
+<h5 id="h5-heading">h5 Heading</h5>
+<h6 id="h6-heading">h6 Heading</h6>
+<h2 id="horizontal-rules">Horizontal Rules</h2>
+<hr />
+<hr />
+<hr />
+<h2 id="emphasis">Emphasis</h2>
+<p><strong>This is bold text</strong></p>
+<p><strong>This is bold text</strong></p>
+<p><em>This is italic text</em></p>
+<p><em>This is italic text</em></p>
+<p><del>Strikethrough</del></p>
+<h2 id="blockquotes">Blockquotes</h2>
+<blockquote>
+<p>Blockquotes can also be nested...</p>
+<blockquote>
+<p>...by using additional greater-than signs right next to each other...</p>
+<blockquote>
+<p>...or with spaces between arrows.</p>
+</blockquote>
+</blockquote>
+</blockquote>
+<h2 id="lists">Lists</h2>
+<p>Unordered</p>
+<ul>
+<li>Create a list by starting a line with <code>+</code>, <code>-</code>, or <code>*</code></li>
+<li>Sub-lists are made by indenting 2 spaces:
+<ul>
+<li>Marker character change forces new list start:
+<ul>
+<li>Ac tristique libero volutpat at</li>
+</ul>
+<ul>
+<li>Facilisis in pretium nisl aliquet</li>
+</ul>
+<ul>
+<li>Nulla volutpat aliquam velit</li>
+</ul>
+</li>
+</ul>
+</li>
+<li>Very easy!</li>
+</ul>
+<p>Ordered</p>
+<ol>
+<li>
+<p>Lorem ipsum dolor sit amet</p>
+</li>
+<li>
+<p>Consectetur adipiscing elit</p>
+</li>
+<li>
+<p>Integer molestie lorem at massa</p>
+</li>
+<li>
+<p>You can use sequential numbers...</p>
+</li>
+<li>
+<p>...or keep all the numbers as <code>1.</code></p>
+</li>
+</ol>
+<p>Start numbering with offset:</p>
+<ol start="57">
+<li>foo</li>
+<li>bar</li>
+</ol>
+<h2 id="code">Code</h2>
+<p>Inline <code>code</code></p>
+<p>Indented code</p>
+<pre><code>&#x2F;&#x2F; Some comments
+line 1 of code
+line 2 of code
+line 3 of code
+</code></pre>
+<p>Block code &quot;fences&quot;</p>
+<pre><code>Sample text here...
+</code></pre>
+<p>Syntax highlighting</p>
+<pre data-lang="js" class="language-js "><code class="language-js" data-lang="js">var foo = function (bar) {
+  return bar++;
+};
+
+console.log(foo(5));
+</code></pre>
+<h2 id="shortcodes">Shortcodes</h2>
+<blockquote>
+This is a test quote!
+1900-01-01 <br>
+-- John Doe
+</blockquote><h2 id="tables">Tables</h2>
+<table><thead><tr><th>Option</th><th>Description</th></tr></thead><tbody>
+<tr><td>data</td><td>path to data files to supply the data that will be passed into templates.</td></tr>
+<tr><td>engine</td><td>engine to be used for processing templates. Handlebars is the default.</td></tr>
+<tr><td>ext</td><td>extension to be used for dest files.</td></tr>
+</tbody></table>
+<p>Right aligned columns</p>
+<table><thead><tr><th align="right">Option</th><th align="right">Description</th></tr></thead><tbody>
+<tr><td align="right">data</td><td align="right">path to data files to supply the data that will be passed into templates.</td></tr>
+<tr><td align="right">engine</td><td align="right">engine to be used for processing templates. Handlebars is the default.</td></tr>
+<tr><td align="right">ext</td><td align="right">extension to be used for dest files.</td></tr>
+</tbody></table>
+<h2 id="links">Links</h2>
+<p><a href="http://duckduckgo.com">link text</a></p>
+<p><a href="http://duckduckgo.com/">link with title</a></p>
+<h2 id="images">Images</h2>
+<p><img src="https://octodex.github.com/images/minion.png" alt="Minion" />
+<img src="https://octodex.github.com/images/stormtroopocat.jpg" alt="Stormtroopocat" title="The Stormtroopocat" /></p>
+<p>Like links, Images also have a footnote style syntax</p>
+<p><img src="https://octodex.github.com/images/dojocat.jpg" alt="Alt text" title="The Dojocat" /></p>
+<p>With a reference later in the document defining the URL location:</p>
+<h3 id="footnotes">Footnotes</h3>
+<p>Footnote 1 link<sup class="footnote-reference"><a href="#first">1</a></sup>.</p>
+<p>Footnote 2 link<sup class="footnote-reference"><a href="#second">2</a></sup>.</p>
+<p>Duplicated footnote reference<sup class="footnote-reference"><a href="#second">2</a></sup>.</p>
+<div class="footnote-definition" id="first"><sup class="footnote-definition-label">1</sup>
+<p>Footnote <strong>can have markup</strong>
+and multiple paragraphs.</p>
+</div>
+<div class="footnote-definition" id="second"><sup class="footnote-definition-label">2</sup>
+<p>Footnote text.</p>
+</div>
+"##
+    );
+}

components/rendering/tests/links.rs

@@ -0,0 +1,62 @@
+use std::collections::HashMap;
+
+use errors::Result;
+use rendering::Rendered;
+
+mod common;
+
+fn render_content(content: &str, permalinks: HashMap<String, String>) -> Result<Rendered> {
+    let config = config::Config::default_for_test();
+    let tera = tera::Tera::default();
+    let mut context = rendering::RenderContext::new(
+        &tera,
+        &config,
+        &config.default_language,
+        "http://mypage.com",
+        &permalinks,
+        front_matter::InsertAnchor::None,
+    );
+    context.set_current_page_path("mine.md");
+
+    rendering::render_content(content, &context)
+}
+
+#[test]
+fn can_detect_links() {
+    // no links
+    let rendered = render_content("Hello World!", HashMap::new()).unwrap();
+    assert_eq!(rendered.internal_links.len(), 0);
+    assert_eq!(rendered.external_links.len(), 0);
+
+    // external
+    let rendered = render_content("[abc](https://google.com/)", HashMap::new()).unwrap();
+    assert_eq!(rendered.internal_links.len(), 0);
+    assert_eq!(rendered.external_links.len(), 1);
+    assert_eq!(rendered.external_links[0], "https://google.com/");
+
+    // internal
+    let mut permalinks = HashMap::new();
+    permalinks.insert("def/123.md".to_owned(), "https://xyz.com/def/123".to_owned());
+    let rendered = render_content("[abc](@/def/123.md)", permalinks).unwrap();
+    assert_eq!(rendered.internal_links.len(), 1);
+    assert_eq!(rendered.internal_links[0], ("def/123.md".to_owned(), None));
+    assert_eq!(rendered.external_links.len(), 0);
+
+    // internal with anchors
+    let mut permalinks = HashMap::new();
+    permalinks.insert("def/123.md".to_owned(), "https://xyz.com/def/123".to_owned());
+    let rendered = render_content("[abc](@/def/123.md#hello)", permalinks).unwrap();
+    assert_eq!(rendered.internal_links.len(), 1);
+    assert_eq!(rendered.internal_links[0], ("def/123.md".to_owned(), Some("hello".to_owned())));
+    assert_eq!(rendered.external_links.len(), 0);
+
+    // internal link referring to self
+    let rendered = render_content("[abc](#hello)", HashMap::new()).unwrap();
+    assert_eq!(rendered.internal_links.len(), 1);
+    assert_eq!(rendered.internal_links[0], ("mine.md".to_owned(), Some("hello".to_owned())));
+    assert_eq!(rendered.external_links.len(), 0);
+
+    // Not pointing to anything so that's an error
+    let res = render_content("[abc](@/def/123.md)", HashMap::new());
+    assert!(res.is_err());
+}

components/rendering/tests/markdown.rs

@@ -111,7 +111,7 @@ fn can_higlight_code_block_with_unknown_lang() {
 fn can_render_shortcode() {
     let permalinks_ctx = HashMap::new();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &ZOLA_TERA,
         &config,
         &config.default_language,
@@ -119,6 +119,8 @@ fn can_render_shortcode() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&ZOLA_TERA);
+    context.set_shortcode_definitions(&shortcode_def);
     let res = render_content(
         r#"
 Hello
@@ -128,6 +130,7 @@ Hello
         &context,
     )
     .unwrap();
+    println!("{:?}", res.body);
     assert!(res.body.contains("<p>Hello</p>\n<div >"));
     assert!(res
         .body
@@ -138,7 +141,7 @@ Hello
 fn can_render_shortcode_with_markdown_char_in_args_name() {
     let permalinks_ctx = HashMap::new();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &ZOLA_TERA,
         &config,
         &config.default_language,
@@ -146,6 +149,8 @@ fn can_render_shortcode_with_markdown_char_in_args_name() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&ZOLA_TERA);
+    context.set_shortcode_definitions(&shortcode_def);
     let input = vec!["name", "na_me", "n_a_me", "n1"];
     for i in input {
         let res =
@@ -158,7 +163,7 @@ fn can_render_shortcode_with_markdown_char_in_args_name() {
 fn can_render_shortcode_with_markdown_char_in_args_value() {
     let permalinks_ctx = HashMap::new();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &ZOLA_TERA,
         &config,
         &config.default_language,
@@ -166,6 +171,8 @@ fn can_render_shortcode_with_markdown_char_in_args_value() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&ZOLA_TERA);
+    context.set_shortcode_definitions(&shortcode_def);
     let input = vec![
         "ub36ffWAqgQ-hey",
         "ub36ffWAqgQ_hey",
@@ -181,6 +188,54 @@ fn can_render_shortcode_with_markdown_char_in_args_value() {
     }
 }
 
+#[test]
+fn can_render_html_shortcode_with_lang() {
+    let permalinks_ctx = HashMap::new();
+    let config = Config::default_for_test();
+    let mut tera = Tera::default();
+    tera.extend(&ZOLA_TERA).unwrap();
+    tera.add_raw_template("shortcodes/i18nshortcode.html", "{{ lang }}").unwrap();
+    let mut context = RenderContext::new(
+        &tera,
+        &config,
+        &config.default_language,
+        "",
+        &permalinks_ctx,
+        InsertAnchor::None,
+    );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
+
+    let res = render_content("a{{ i18nshortcode() }}a", &context).unwrap();
+    assert_eq!(res.body, "<p>aena</p>\n");
+}
+
+#[test]
+fn can_render_md_shortcode_with_lang() {
+    let permalinks_ctx = HashMap::new();
+    let config = Config::default_for_test();
+    let mut tera = Tera::default();
+    tera.extend(&ZOLA_TERA).unwrap();
+    tera.add_raw_template(
+        "shortcodes/i18nshortcode.md",
+        "![Book cover in {{ lang }}](cover.{{ lang }}.png)",
+    )
+    .unwrap();
+    let mut context = RenderContext::new(
+        &tera,
+        &config,
+        &config.default_language,
+        "",
+        &permalinks_ctx,
+        InsertAnchor::None,
+    );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
+
+    let res = render_content("{{ i18nshortcode() }}", &context).unwrap();
+    assert_eq!(res.body, "<p><img src=\"cover.en.png\" alt=\"Book cover in en\" /></p>\n");
+}
+
 #[test]
 fn can_render_body_shortcode_with_markdown_char_in_name() {
     let permalinks_ctx = HashMap::new();
@@ -195,7 +250,7 @@ fn can_render_body_shortcode_with_markdown_char_in_name() {
             "<blockquote>{{ body }} - {{ author}}</blockquote>",
         )
         .unwrap();
-        let context = RenderContext::new(
+        let mut context = RenderContext::new(
             &tera,
             &config,
             &config.default_language,
@@ -203,6 +258,8 @@ fn can_render_body_shortcode_with_markdown_char_in_name() {
             &permalinks_ctx,
             InsertAnchor::None,
         );
+        let shortcode_def = utils::templates::get_shortcodes(&tera);
+        context.set_shortcode_definitions(&shortcode_def);
 
         let res =
             render_content(&format!("{{% {}(author=\"Bob\") %}}\nhey\n{{% end %}}", i), &context)
@@ -233,7 +290,7 @@ Here is another paragraph.
 
     tera.add_raw_template("shortcodes/figure.html", shortcode).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -241,6 +298,8 @@ Here is another paragraph.
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
 
@@ -266,14 +325,11 @@ This is a figure caption.
 Here is another paragraph.
 "#;
 
-    let expected = "<p>This is a figure caption.</p>
+    let expected = "<p>This is a figure caption.</p><p>This is a figure caption.</p>\n<p>Here is another paragraph.</p>\n";
-<p>This is a figure caption.</p>
-<p>Here is another paragraph.</p>
-";
 
     tera.add_raw_template("shortcodes/figure.html", shortcode).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -281,6 +337,8 @@ Here is another paragraph.
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
 
@@ -291,7 +349,7 @@ Here is another paragraph.
 fn can_render_several_shortcode_in_row() {
     let permalinks_ctx = HashMap::new();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &ZOLA_TERA,
         &config,
         &config.default_language,
@@ -299,6 +357,8 @@ fn can_render_several_shortcode_in_row() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&ZOLA_TERA);
+    context.set_shortcode_definitions(&shortcode_def);
     let res = render_content(
         r#"
 Hello
@@ -317,6 +377,7 @@ Hello
         &context,
     )
     .unwrap();
+    println!("{:?}", res);
     assert!(res.body.contains("<p>Hello</p>\n<div >"));
     assert!(res
         .body
@@ -356,7 +417,7 @@ fn can_render_shortcode_with_body() {
     .unwrap();
     let permalinks_ctx = HashMap::new();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -364,6 +425,8 @@ fn can_render_shortcode_with_body() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(
         r#"
@@ -375,7 +438,7 @@ A quote
         &context,
     )
     .unwrap();
-    assert_eq!(res.body, "<p>Hello</p>\n<blockquote>A quote - Keats</blockquote>\n");
+    assert_eq!(res.body, "<p>Hello\n<blockquote>A quote - Keats</blockquote></p>\n");
 }
 
 #[test]
@@ -689,6 +752,25 @@ fn can_insert_anchor_with_other_special_chars() {
     );
 }
 
+#[test]
+fn can_insert_anchor_with_lang() {
+    let mut tera = Tera::default();
+    tera.extend(&ZOLA_TERA).unwrap();
+    tera.add_raw_template("anchor-link.html", "({{ lang }})").unwrap();
+    let permalinks_ctx = HashMap::new();
+    let config = Config::default_for_test();
+    let context = RenderContext::new(
+        &tera,
+        &config,
+        &config.default_language,
+        "",
+        &permalinks_ctx,
+        InsertAnchor::Right,
+    );
+    let res = render_content("# Hello", &context).unwrap();
+    assert_eq!(res.body, "<h1 id=\"hello\">Hello(en)</h1>\n");
+}
+
 #[test]
 fn can_make_toc() {
     let permalinks_ctx = HashMap::new();
@@ -935,7 +1017,7 @@ fn can_make_permalinks_with_colocated_assets_for_link() {
         InsertAnchor::None,
     );
     let res = render_content("[an image](image.jpg)", &context).unwrap();
-    assert_eq!(res.body, "<p><a href=\"https://vincent.is/about/image.jpg\">an image</a></p>\n");
+    assert_eq!(res.body, "<p><a href=\"image.jpg\">an image</a></p>\n");
 }
 
 #[test]
@@ -951,10 +1033,7 @@ fn can_make_permalinks_with_colocated_assets_for_image() {
         InsertAnchor::None,
     );
     let res = render_content("![alt text](image.jpg)", &context).unwrap();
-    assert_eq!(
+    assert_eq!(res.body, "<p><img src=\"image.jpg\" alt=\"alt text\" /></p>\n");
-        res.body,
-        "<p><img src=\"https://vincent.is/about/image.jpg\" alt=\"alt text\" /></p>\n"
-    );
 }
 
 #[test]
@@ -1058,8 +1137,7 @@ fn doesnt_try_to_highlight_content_from_shortcode() {
     let mut tera = Tera::default();
     tera.extend(&ZOLA_TERA).unwrap();
 
-    let shortcode = r#"
+    let shortcode = r#"<figure>
-<figure>
      {% if width %}
      <img src="/images/{{ src }}" alt="{{ caption }}" width="{{ width }}" />
      {% else %}
@@ -1075,7 +1153,7 @@ fn doesnt_try_to_highlight_content_from_shortcode() {
 
     tera.add_raw_template("shortcodes/figure.html", shortcode).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -1083,6 +1161,8 @@ fn doesnt_try_to_highlight_content_from_shortcode() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
     assert_eq!(res.body, expected);
@@ -1104,7 +1184,7 @@ fn can_emit_newlines_and_whitespace_with_shortcode() {
 
     tera.add_raw_template(&format!("shortcodes/{}.html", "preformatted"), shortcode).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -1112,6 +1192,8 @@ fn can_emit_newlines_and_whitespace_with_shortcode() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
     assert_eq!(res.body, expected);
@@ -1238,7 +1320,7 @@ Bla bla"#;
 
     tera.add_raw_template("shortcodes/quote.md", shortcode).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -1246,6 +1328,8 @@ Bla bla"#;
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
 
@@ -1259,8 +1343,7 @@ fn can_render_shortcode_body_with_no_invalid_escaping() {
     let mut tera = Tera::default();
     tera.extend(&ZOLA_TERA).unwrap();
 
-    let shortcode = r#"
+    let shortcode = r#"<a class="resize-image" href="/tlera-corp-gnat/gnat-with-picoblade-cable.jpg">
-<a class="resize-image" href="/tlera-corp-gnat/gnat-with-picoblade-cable.jpg">
     <img
         src="https://placekitten.com/200/300"
         alt="{{ alt }}">
@@ -1271,11 +1354,11 @@ fn can_render_shortcode_body_with_no_invalid_escaping() {
 
     let markdown_string = r#"{{ resize_image(path="tlera-corp-gnat/gnat-with-picoblade-cable.jpg", width=600, alt="Some alt") }}"#;
 
-    let expected = "<a class=\"resize-image\" href=\"/tlera-corp-gnat/gnat-with-picoblade-cable.jpg\">\n    <img\n        src=\"https://placekitten.com/200/300\"\n        alt=\"Some alt\">\n    </img>\n    <p>(click for full size)</p>\n</a>";
+    let expected = "<a class=\"resize-image\" href=\"/tlera-corp-gnat/gnat-with-picoblade-cable.jpg\">\n    <img\n        src=\"https://placekitten.com/200/300\"\n        alt=\"Some alt\">\n    </img>\n    <p>(click for full size)</p>\n</a>\n";
 
     tera.add_raw_template("shortcodes/resize_image.html", shortcode).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -1283,11 +1366,14 @@ fn can_render_shortcode_body_with_no_invalid_escaping() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
     assert_eq!(res.body, expected);
 }
 
+// TODO: handle it in the html part of the md renderer
 // https://github.com/getzola/zola/issues/1172
 #[test]
 fn can_render_commented_out_shortcodes_fine() {
@@ -1295,23 +1381,14 @@ fn can_render_commented_out_shortcodes_fine() {
     let mut tera = Tera::default();
     tera.extend(&ZOLA_TERA).unwrap();
 
-    let shortcode = r#"
+    let shortcode = r#"<a width={{width}} class="resize-image" href="/tlera-corp-gnat/gnat-with-picoblade-cable.jpg">{{alt}}</a>"#;
-<a class="resize-image" href="/tlera-corp-gnat/gnat-with-picoblade-cable.jpg">
-    <img
-        src="https://placekitten.com/200/300"
-        alt="{{ alt }}">
-    </img>
-    <p>(click for full size)</p>
-</a>
-"#;
 
-    let markdown_string = r#"<!--{{ resize_image(path="tlera-corp-gnat/gnat-with-picoblade-cable.jpg", width=600, alt="Some alt") }}-->"#;
+    let markdown_string = r#"<!--{{ resize_image(path="gnat-with-picoblade-cable.jpg", width=600, alt="Alt1") }}{{ resize_image(path="gnat-with-picoblade-cable.jpg", width=610, alt="Alt2") }}-->"#;
-
+    let expected = "<!--<a width=600 class=\"resize-image\" href=\"/tlera-corp-gnat/gnat-with-picoblade-cable.jpg\">Alt1</a><a width=610 class=\"resize-image\" href=\"/tlera-corp-gnat/gnat-with-picoblade-cable.jpg\">Alt2</a>-->";
-    let expected = "<!--<a class=\"resize-image\" href=\"/tlera-corp-gnat/gnat-with-picoblade-cable.jpg\">\n    <img\n        src=\"https://placekitten.com/200/300\"\n        alt=\"Some alt\">\n    </img>\n    <p>(click for full size)</p>\n</a>-->";
 
     tera.add_raw_template("shortcodes/resize_image.html", shortcode).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -1319,6 +1396,8 @@ fn can_render_commented_out_shortcodes_fine() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
     assert_eq!(res.body, expected);
@@ -1332,6 +1411,7 @@ fn can_render_read_more_after_shortcode() {
     tera.extend(&ZOLA_TERA).unwrap();
 
     let shortcode = r#"<p>Quote: {{body}}</p>"#;
+    tera.add_raw_template("shortcodes/quote.md", shortcode).unwrap();
     let markdown_string = r#"
 # Title
 
@@ -1348,9 +1428,8 @@ Again more text"#;
 <p>Again more text</p>
 "#;
 
-    tera.add_raw_template("shortcodes/quote.md", shortcode).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -1358,6 +1437,8 @@ Again more text"#;
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
     assert_eq!(res.body, expected);
@@ -1411,16 +1492,16 @@ fn invocation_count_increments_in_shortcode() {
 {{ b() }}
 "#;
 
-    let expected = r#"<p>a: 1</p>
+    let expected = r#"<p><p>a: 1</p>
 <p>b: 1</p>
 <p>a: 2</p>
-<p>b: 2</p>
+<p>b: 2</p></p>
 "#;
 
     tera.add_raw_template("shortcodes/a.html", shortcode_template_a).unwrap();
     tera.add_raw_template("shortcodes/b.html", shortcode_template_b).unwrap();
     let config = Config::default_for_test();
-    let context = RenderContext::new(
+    let mut context = RenderContext::new(
         &tera,
         &config,
         &config.default_language,
@@ -1428,6 +1509,8 @@ fn invocation_count_increments_in_shortcode() {
         &permalinks_ctx,
         InsertAnchor::None,
     );
+    let shortcode_def = utils::templates::get_shortcodes(&tera);
+    context.set_shortcode_definitions(&shortcode_def);
 
     let res = render_content(markdown_string, &context).unwrap();
     assert_eq!(res.body, expected);

components/rendering/tests/spec.rs

@@ -0,0 +1,501 @@
+mod common;
+
+use common::ShortCode;
+use std::path::PathBuf;
+use templates::ZOLA_TERA;
+
+macro_rules! test_scenario {
+    ($in_str:literal, $out_str:literal, [$($shortcodes:ident),*]) => {
+        let config = config::Config::default_for_test();
+
+        #[allow(unused_mut)]
+        let mut tera = tera::Tera::default();
+        tera.extend(&ZOLA_TERA).unwrap();
+
+        $(
+            let ShortCode { name, is_md, output } = $shortcodes;
+            tera.add_raw_template(
+                &format!("shortcodes/{}.{}", name, if is_md { "md" } else { "html" }),
+                &output,
+            ).unwrap();
+        )*
+
+        let mut permalinks = std::collections::HashMap::new();
+        permalinks.insert("".to_string(), "".to_string());
+
+        tera.register_filter(
+            "markdown",
+            templates::filters::MarkdownFilter::new(
+                PathBuf::new(),
+                config.clone(),
+                permalinks.clone(),
+            ).unwrap()
+        );
+
+        let mut context = rendering::RenderContext::new(
+            &tera,
+            &config,
+            &config.default_language,
+            "",
+            &permalinks,
+            front_matter::InsertAnchor::None,
+        );
+        let shortcode_def = utils::templates::get_shortcodes(&tera);
+        context.set_shortcode_definitions(&shortcode_def);
+
+        let rendered = rendering::render_content($in_str, &context);
+        println!("{:?}", rendered);
+        assert!(rendered.is_ok());
+
+        let rendered = rendered.unwrap();
+        assert_eq!(rendered.body, $out_str.to_string());
+    }
+}
+
+#[test]
+fn plain_text() {
+    // Test basic formation of text and paragraphs tags
+    // - Plain sentences (Long and broken up)
+    // - Multiple paragraphs
+
+    test_scenario!("Hello World!", "<p>Hello World!</p>\n", []);
+
+    test_scenario!("Hello\nWorld!", "<p>Hello\nWorld!</p>\n", []);
+
+    test_scenario!("Hello\n\nWorld!", "<p>Hello</p>\n<p>World!</p>\n", []);
+
+    test_scenario!("Hello\n\nWorld\n\n!", "<p>Hello</p>\n<p>World</p>\n<p>!</p>\n", []);
+}
+
+#[test]
+fn header() {
+    // Test basic header ids
+    // - Plain headers
+    // - Headers with text
+
+    test_scenario!("# Header1", "<h1 id=\"header1\">Header1</h1>\n", []);
+
+    test_scenario!(
+        "## A longer string of text!",
+        "<h2 id=\"a-longer-string-of-text\">A longer string of text!</h2>\n",
+        []
+    );
+
+    test_scenario!(
+        "# Header1\nHello World!",
+        "<h1 id=\"header1\">Header1</h1>\n<p>Hello World!</p>\n",
+        []
+    );
+
+    test_scenario!(
+        "# Header1\n\nHello World!",
+        "<h1 id=\"header1\">Header1</h1>\n<p>Hello World!</p>\n",
+        []
+    );
+}
+
+#[test]
+fn code_block() {
+    test_scenario!("```\nWow Code!\n```", "<pre><code>Wow Code!\n</code></pre>\n", []);
+
+    test_scenario!("    Wow Code!", "<pre><code>Wow Code!</code></pre>\n", []);
+}
+
+const MD_SIMPLE: ShortCode = ShortCode::new("simple", "Hello World!", true);
+const HTML_SIMPLE: ShortCode = ShortCode::new("simple", "Hello World!", false);
+
+#[test]
+fn simple_shortcodes() {
+    // Test both MD & HTML plain text shortcodes
+
+    test_scenario!("{{ simple() }}", "<p>Hello World!</p>\n", [MD_SIMPLE]);
+
+    test_scenario!("hey {{ simple() }}", "<p>hey Hello World!</p>\n", [HTML_SIMPLE]);
+}
+
+const MD_LINK: ShortCode = ShortCode::new("link", "[Link...](/)", true);
+const HTML_LINK: ShortCode = ShortCode::new("link", "<a href=\"/\">Link...</a>", false);
+
+#[test]
+fn md_inline_shortcodes() {
+    // Test both MD & HTML inline shortcodes
+
+    test_scenario!(
+        "A read more link: {{ link() }}",
+        "<p>A read more link: <a href=\"/\">Link...</a></p>\n",
+        [MD_LINK]
+    );
+
+    test_scenario!(
+        "A read more link: {{ link() }}",
+        "<p>A read more link: <a href=\"/\">Link...</a></p>\n",
+        [HTML_LINK]
+    );
+}
+
+const HTML_DIV: ShortCode = ShortCode::new("dived", "<div>Hello World!</div>", false);
+
+#[test]
+fn html_div_shortcodes() {
+    // Test the behaviour of HTML div-ed shortcodes
+
+    test_scenario!("{{ dived() }}", "<div>Hello World!</div>", [HTML_DIV]);
+
+    test_scenario!(
+        "{{ dived() }} {{ dived() }}",
+        "<p><div>Hello World!</div> <div>Hello World!</div></p>\n",
+        [HTML_DIV]
+    );
+    test_scenario!(
+        "{{ dived() }}\n{{ dived() }}",
+        "<p><div>Hello World!</div>\n<div>Hello World!</div></p>\n",
+        [HTML_DIV]
+    );
+    test_scenario!(
+        "{{ dived() }}\n\n{{ dived() }}",
+        "<div>Hello World!</div><div>Hello World!</div>",
+        [HTML_DIV]
+    );
+}
+
+const HTML_TABS_MULTILINE: ShortCode =
+    ShortCode::new("multiline", "<div>\n\tHello World!\n</div>", false);
+
+const HTML_SPACES_MULTILINE: ShortCode =
+    ShortCode::new("multiline", "<div>\n    Hello World!\n</div>", false);
+
+#[test]
+fn html_tabs_multiline_shortcodes() {
+    // Test the behaviour multiline HTML shortcodes
+    // This can cause problems mostly because the 4 spaces sometimes used for tabs also are used
+    // to indicate code-blocks
+
+    test_scenario!("{{ multiline() }}", "<div>\n\tHello World!\n</div>", [HTML_TABS_MULTILINE]);
+
+    test_scenario!(
+        "{{ multiline() }} {{ multiline() }}",
+        "<p><div>\n\tHello World!\n</div> <div>\n\tHello World!\n</div></p>\n",
+        [HTML_TABS_MULTILINE]
+    );
+
+    test_scenario!(
+        "{{ multiline() }}\n{{ multiline() }}",
+        "<p><div>\n\tHello World!\n</div>\n<div>\n\tHello World!\n</div></p>\n",
+        [HTML_TABS_MULTILINE]
+    );
+
+    test_scenario!(
+        "{{ multiline() }}\n\n{{ multiline() }}",
+        "<div>\n\tHello World!\n</div><div>\n\tHello World!\n</div>",
+        [HTML_TABS_MULTILINE]
+    );
+}
+
+#[test]
+fn html_spaces_multiline_shortcodes() {
+    // Test the behaviour multiline HTML shortcodes
+    // This can cause problems mostly because the 4 spaces sometimes used for tabs also are used
+    // to indicate code-blocks
+
+    test_scenario!("{{ multiline() }}", "<div>\n    Hello World!\n</div>", [HTML_SPACES_MULTILINE]);
+
+    test_scenario!(
+        "{{ multiline() }} {{ multiline() }}",
+        "<p><div>\n    Hello World!\n</div> <div>\n    Hello World!\n</div></p>\n",
+        [HTML_SPACES_MULTILINE]
+    );
+
+    test_scenario!(
+        "{{ multiline() }}\n{{ multiline() }}",
+        "<p><div>\n    Hello World!\n</div>\n<div>\n    Hello World!\n</div></p>\n",
+        [HTML_SPACES_MULTILINE]
+    );
+
+    // a single \n would keep it in the same paragraph as above
+    // 2 \n would result in different paragraphs and basically ignoring the 2 \n
+    test_scenario!(
+        "{{ multiline() }}\n\n{{ multiline() }}",
+        "<div>\n    Hello World!\n</div><div>\n    Hello World!\n</div>",
+        [HTML_SPACES_MULTILINE]
+    );
+}
+
+// Related to Issue of recursive shortcodes
+// const MD_OUTER: ShortCode = ShortCode::new("outer", "Hello {{ inner() }}!", true);
+// const MD_INNER: ShortCode = ShortCode::new("inner", "World", true);
+//
+// const HTML_OUTER: ShortCode = ShortCode::new("outer", "Hello {{ inner() }}!", false);
+// const HTML_INNER: ShortCode = ShortCode::new("inner", "World", false);
+//
+// const MD_REUSER: ShortCode = ShortCode::new("reuser", "{{ reuser() }}", true);
+// const MD_REFBACK: ShortCode = ShortCode::new("refback", "{{ leveledreuser() }}", true);
+// const MD_LEVELED_REUSER: ShortCode = ShortCode::new("leveledreuser", "{{ refback() }}", true);
+//
+// const HTML_REUSER: ShortCode = ShortCode::new("reuser", "{{ reuser() }}", false);
+// const HTML_REFBACK: ShortCode = ShortCode::new("refback", "{{ leveledreuser() }}", false);
+// const HTML_LEVELED_REUSER: ShortCode = ShortCode::new("leveledreuser", "{{ refback() }}", false);
+//
+// #[test]
+// fn md_recursive_shortcodes() {
+//     // Test recursive shortcodes in a MD context.
+//     // This should always work, unless a shortcode is reused
+//
+//     test_scenario!("{{ outer() }}", "<p>Hello World!</p>\n", [MD_OUTER, MD_INNER]);
+//     test_scenario!("{{ outer() }}", "<p>Hello World!</p>\n", [MD_INNER, MD_OUTER]);
+// }
+//
+// #[test]
+// fn html_recursive_shortcodes() {
+//     // Test recursive shortcodes in a HTML context.
+//     // One can add HTML shortcodes within html shortcodes, unless a shortcode is reused
+//
+//     test_scenario!("{{ outer() }}", "<p>Hello {{ inner() }}!</p>", [HTML_OUTER, HTML_INNER]);
+//     test_scenario!("{{ outer() }}", "<p>Hello {{ inner() }}!</p>", [HTML_INNER, HTML_OUTER]);
+// }
+//
+// #[test]
+// fn shortcodes_recursion_stop() {
+//     // Test whether recursion stops if a shortcode is reused.
+//
+//     test_scenario_fail!("{{ reuser() }}", [MD_REUSER]);
+//     test_scenario_fail!("{{ leveledreuser() }}", [MD_LEVELED_REUSER, MD_REFBACK]);
+//
+//     test_scenario_fail!("{{ reuser() }}", [HTML_REUSER]);
+//     test_scenario_fail!("{{ leveledreuser() }}", [HTML_LEVELED_REUSER, HTML_REFBACK]);
+//
+//     test_scenario_fail!("{{ leveledreuser() }}", [HTML_LEVELED_REUSER, MD_REFBACK]);
+//     test_scenario_fail!("{{ leveledreuser() }}", [MD_LEVELED_REUSER, HTML_REFBACK]);
+// }
+//
+// #[test]
+// fn html_in_md_recursive_shortcodes() {
+//     // Test whether we can properly add HTML shortcodes in MD shortcodes
+//
+//     test_scenario!("{{ outer() }}", "<p>Hello World!</p>\n", [HTML_INNER, MD_OUTER]);
+//     test_scenario!("{{ outer() }}", "<p>Hello World!</p>\n", [MD_OUTER, HTML_INNER]);
+// }
+//
+// #[test]
+// fn md_in_html_recursive_shortcodes() {
+//     // Test whether we can not add MD shortcodes in HTML shortcodes
+//
+//     test_scenario!("{{ outer() }}", "<p>Hello {{ inner() }}!</p>", [HTML_OUTER, MD_INNER]);
+//     test_scenario!("{{ outer() }}", "<p>Hello {{ inner() }}!</p>", [MD_INNER, HTML_OUTER]);
+// }
+
+const MD_BODY_SHORTCODE: ShortCode = ShortCode::new("bdy", "*{{ body }}*", true);
+const HTML_BODY_SHORTCODE: ShortCode = ShortCode::new("bdy", "<span>{{ body }}</span>", false);
+
+#[test]
+fn md_body_shortcodes() {
+    // Test basic MD body shortcodes
+
+    test_scenario!("abc {% bdy() %}def{% end %}", "<p>abc <em>def</em></p>\n", [MD_BODY_SHORTCODE]);
+
+    test_scenario!(
+        "abc\n\n{% bdy() %}def{% end %}",
+        "<p>abc</p>\n<p><em>def</em></p>\n",
+        [MD_BODY_SHORTCODE]
+    );
+}
+
+#[test]
+fn html_body_shortcodes() {
+    // Test basic HTML body shortcodes
+
+    test_scenario!(
+        "abc {% bdy() %}def{% end %}",
+        "<p>abc <span>def</span></p>\n",
+        [HTML_BODY_SHORTCODE]
+    );
+
+    // Should it wrap the shortcode in a `<p>`?
+    test_scenario!(
+        "abc\n\n{% bdy() %}def{% end %}",
+        "<p>abc</p>\n<span>def</span>",
+        [HTML_BODY_SHORTCODE]
+    );
+}
+
+// Related to issue #515
+// #[test]
+// fn shortcode_in_md_body() {
+//     // Test whether we can properly insert a shortcode in a MD shortcode body
+//
+//     test_scenario!(
+//         "{% bdy() %}Wow! {{ simple() }}{% end %}",
+//         "<p><em>Wow! Hello World!</em></p>\n",
+//         [MD_BODY_SHORTCODE, MD_SIMPLE]
+//     );
+//
+//     test_scenario!(
+//         "{% bdy() %}Wow! {{ simple() }}{% end %}",
+//         "<p><em>Wow! Hello World!</em></p>\n",
+//         [MD_SIMPLE, MD_BODY_SHORTCODE]
+//     );
+//
+//     test_scenario!(
+//         "{% bdy() %}Wow! {{ simple() }}{% end %}",
+//         "<p><em>Wow! Hello World!</em></p>\n",
+//         [MD_BODY_SHORTCODE, HTML_SIMPLE]
+//     );
+//
+//     test_scenario!(
+//         "{% bdy() %}Wow! {{ simple() }}{% end %}",
+//         "<p><em>Wow! Hello World!</em></p>\n",
+//         [HTML_SIMPLE, MD_BODY_SHORTCODE]
+//     );
+// }
+//
+// #[test]
+// fn shortcode_in_html_body() {
+//     // Test whether we can properly insert a shortcode in a HTML shortcode body
+//
+//     test_scenario!(
+//         "{% bdy() %}Wow! {{ simple() }}{% end %}",
+//         "<p><span>Wow! Hello World!</span></p>\n",
+//         [HTML_BODY_SHORTCODE, MD_SIMPLE]
+//     );
+//
+//     test_scenario!(
+//         "{% bdy() %}Wow! {{ simple() }}{% end %}",
+//         "<p><span>Wow! Hello World!</span></p>\n",
+//         [MD_SIMPLE, HTML_BODY_SHORTCODE]
+//     );
+//
+//     test_scenario!(
+//         "{% bdy() %}Wow! {{ simple() }}{% end %}",
+//         "<p><span>Wow! Hello World!</span></p>\n",
+//         [HTML_BODY_SHORTCODE, HTML_SIMPLE]
+//     );
+//
+//     test_scenario!(
+//         "{% bdy() %}Wow! {{ simple() }}{% end %}",
+//         "<p><span>Wow! Hello World!</span></p>\n",
+//         [HTML_SIMPLE, HTML_BODY_SHORTCODE]
+//     );
+// }
+
+const MD_ARG_SHORTCODE: ShortCode = ShortCode::new(
+    "argeater",
+    "{{ s }}\n{{ b }}\n{{ f }}\n{{ i }}\n{{ array | join(sep=' // ') | safe }}",
+    true,
+);
+const HTML_ARG_SHORTCODE: ShortCode = ShortCode::new(
+    "argeater",
+    "{{ s }}\n{{ b }}\n{{ f }}\n{{ i }}\n{{ array | join(sep=' // ') | safe }}",
+    false,
+);
+
+#[test]
+fn shortcode_arguments() {
+    // Test for properly inserting all shortcodes
+
+    test_scenario!(
+        "{{ argeater(s='Hello World!', b=true, f=3.1415, i=42, array=[1, 3, 3, 7]) }}",
+        "<p>Hello World!\ntrue\n3.1415\n42\n1 // 3 // 3 // 7</p>\n",
+        [MD_ARG_SHORTCODE]
+    );
+
+    test_scenario!(
+        "{{ argeater(s='Hello World!', b=true, f=3.1415, i=42, array=[1, 3, 3, 7]) }}",
+        "Hello World!\ntrue\n3.1415\n42\n1 // 3 // 3 // 7",
+        [HTML_ARG_SHORTCODE]
+    );
+}
+
+// const MD_BDY_OUTER: ShortCode = ShortCode::new("outer", "*{{ body }}*", true);
+// const MD_BDY_INNER: ShortCode = ShortCode::new("inner", "**{{ body }}**", true);
+
+// Originally from PR #1475
+// #[test]
+// fn body_in_body() {
+//     test_scenario!(
+//         "{% outer() %}\n\tTest text\n\t{% inner() %}\n\t\tHello World!\n\t{% end %}\n{% end %}",
+//         "<p><em>Test text <b>Hello World!</b></em></p>\n",
+//         [MD_BDY_OUTER, MD_BDY_INNER]
+//     );
+// }
+
+// https://github.com/getzola/zola/issues/1355
+#[test]
+fn list_with_shortcode() {
+    test_scenario!(
+        "* a\n* b\n\t{{ multiline() }}\n*c\n\t{{ multiline() }}\n",
+        "<ul>\n<li>a</li>\n<li>b\n<div>\n\tHello World!\n</div>\n*c\n<div>\n\tHello World!\n</div></li>\n</ul>\n",
+        [HTML_TABS_MULTILINE]
+    );
+}
+
+const WEB_COMPONENT_SHORTCODE: ShortCode = ShortCode::new(
+    "examplecode",
+    "<bc-authorizer-example>
+  <code>{{ body | safe}}</code>
+</bc-authorizer-example>",
+    false,
+);
+// https://github.com/getzola/zola/issues/1655
+#[test]
+fn shortcodes_do_not_generate_paragraphs() {
+    test_scenario!(
+        r#"{% examplecode() %}
+some code;
+more code;
+
+other code here;
+{% end %}"#,
+        "<bc-authorizer-example>\n  <code>some code;\nmore code;\n\nother code here;</code>\n</bc-authorizer-example>",
+        [WEB_COMPONENT_SHORTCODE]
+    );
+}
+
+const CODE_BLOCK_SHORTCODE: ShortCode = ShortCode::new(
+    "details",
+    r#"<details>
+<summary>{{summary | markdown(inline=true) | safe}}</summary>
+<div class="details-content">
+{{ body | markdown | safe}}
+</div>
+</details>"#,
+    false,
+);
+// https://github.com/getzola/zola/issues/1601
+#[test]
+fn works_with_code_block_in_shortcode() {
+    test_scenario!(
+        r#"{% details(summary="hey") %}
+```
+some code
+```
+{% end %}"#,
+        "<details>\n<summary>hey</summary>\n<div class=\"details-content\">\n<pre><code>some code\n</code></pre>\n\n</div>\n</details>",
+        [CODE_BLOCK_SHORTCODE]
+    );
+}
+
+// https://github.com/getzola/zola/issues/1600
+#[test]
+fn shortcodes_work_in_quotes() {
+    test_scenario!(
+        "> test quote\n> {{ vimeo(id=\"124313553\") }}\n> test quote",
+        "<blockquote>\n<p>test quote\n<div >\n    <iframe src=\"//player.vimeo.com/video/124313553\" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>\n</div>\n\ntest quote</p>\n</blockquote>\n",
+        []
+    );
+}
+
+const GOOGLE_SHORTCODE: ShortCode = ShortCode::new(
+    "google",
+    r#"<div>
+<a href="https://google.com/search?q={{query}}">Google Search</a>
+</div>"#,
+    false,
+);
+// https://github.com/getzola/zola/issues/1500
+#[test]
+fn can_handle_issue_1500() {
+    test_scenario!(
+        r#"foo {{ google(query="apple") }} bar."#,
+        "<p>foo <div>\n<a href=\"https://google.com/search?q=apple\">Google Search</a>\n</div> bar.</p>\n",
+        [GOOGLE_SHORTCODE]
+    );
+}

components/rendering/tests/summary.rs

@@ -0,0 +1,118 @@
+mod common;
+
+use common::ShortCode;
+
+macro_rules! test_scenario_summary {
+    ($in_str:literal, $summary:literal, [$($shortcodes:ident),*]) => {
+        let config = config::Config::default_for_test();
+
+        #[allow(unused_mut)]
+        let mut tera = tera::Tera::default();
+
+        // Add all shortcodes
+        $(
+            tera.add_raw_template(
+                &format!("shortcodes/{}", $shortcodes.filename()),
+                $shortcodes.output
+            ).expect("Failed to add raw template");
+        )*
+
+        let permalinks = std::collections::HashMap::new();
+        let mut context = rendering::RenderContext::new(
+            &tera,
+            &config,
+            &config.default_language,
+            "",
+            &permalinks,
+            front_matter::InsertAnchor::None,
+        );
+        let shortcode_def = utils::templates::get_shortcodes(&tera);
+        context.set_shortcode_definitions(&shortcode_def);
+
+        let rendered = rendering::render_content($in_str, &context);
+        assert!(rendered.is_ok());
+        let rendered = rendered.unwrap();
+
+        assert!(rendered.summary_len.is_some());
+
+        let summary_len = rendered.summary_len.unwrap();
+        assert_eq!(&rendered.body[..summary_len], $summary);
+    }
+}
+
+#[test]
+fn basic_summary() {
+    test_scenario_summary!("Hello World!\n<!-- more -->\nAnd others!", "<p>Hello World!</p>\n", []);
+    test_scenario_summary!(
+        "Hello World!\n\nWow!\n<!-- more -->\nAnd others!",
+        "<p>Hello World!</p>\n<p>Wow!</p>\n",
+        []
+    );
+}
+
+#[test]
+fn summary_with_headers() {
+    test_scenario_summary!(
+        "# Hello World!\n<!-- more -->\nAnd others!",
+        "<h1 id=\"hello-world\">Hello World!</h1>\n",
+        []
+    );
+    test_scenario_summary!(
+        "# Hello World!\n\nWow!\n<!-- more -->\nAnd others!",
+        "<h1 id=\"hello-world\">Hello World!</h1>\n<p>Wow!</p>\n",
+        []
+    );
+}
+
+const MD_SIMPLE: ShortCode =
+    ShortCode::new("simple", "A lot of text to insert into the document", true);
+const HTML_SIMPLE: ShortCode =
+    ShortCode::new("simple", "A lot of text to insert into the document", true);
+
+#[test]
+fn summary_with_md_shortcodes() {
+    test_scenario_summary!(
+        "{{ simple() }}\n<!-- more -->\nAnd others!",
+        "<p>A lot of text to insert into the document</p>\n",
+        [MD_SIMPLE]
+    );
+    test_scenario_summary!(
+        "{{ simple() }}\n\nWow!\n<!-- more -->\nAnd others!",
+        "<p>A lot of text to insert into the document</p>\n<p>Wow!</p>\n",
+        [MD_SIMPLE]
+    );
+}
+
+#[test]
+fn summary_with_html_shortcodes() {
+    test_scenario_summary!(
+        "{{ simple() }}\n<!-- more -->\nAnd others!",
+        "<p>A lot of text to insert into the document</p>\n",
+        [HTML_SIMPLE]
+    );
+    test_scenario_summary!(
+        "{{ simple() }}\n\nWow!\n<!-- more -->\nAnd others!",
+        "<p>A lot of text to insert into the document</p>\n<p>Wow!</p>\n",
+        [HTML_SIMPLE]
+    );
+}
+
+// const INNER: ShortCode = ShortCode::new("inner", "World", false);
+//
+// const MD_RECURSIVE: ShortCode = ShortCode::new("outer", "Hello {{ inner() }}!", true);
+// const HTML_RECURSIVE: ShortCode = ShortCode::new("outer", "Hello {{ inner() }}!", false);
+//
+// #[test]
+// fn summary_with_recursive_shortcodes() {
+//     test_scenario_summary!(
+//         "{{ outer() }}\n<!-- more -->\nAnd others!",
+//         "<p>Hello World!</p>\n",
+//         [MD_RECURSIVE, INNER]
+//     );
+//
+//     test_scenario_summary!(
+//         "{{ outer() }}\n<!-- more -->\nAnd others!",
+//         "<p>Hello World!</p>\n",
+//         [HTML_RECURSIVE, INNER]
+//     );
+// }

components/rendering/tests/toc.rs

@@ -0,0 +1,112 @@
+mod common;
+
+use common::ShortCode;
+use rendering::Heading;
+
+#[derive(PartialEq, Debug)]
+struct HelperHeader {
+    title: String,
+    children: Vec<HelperHeader>,
+}
+
+impl PartialEq<Heading> for HelperHeader {
+    fn eq(&self, other: &Heading) -> bool {
+        self.title == other.title && self.children == other.children
+    }
+}
+
+macro_rules! hh {
+    ($title:literal, [$($children:expr),*]) => {{
+        HelperHeader {
+            title: $title.to_string(),
+            children: vec![$($children),*],
+        }
+    }}
+}
+
+macro_rules! test_toc {
+    ($in_str:literal, $toc:expr, [$($shortcodes:ident),*]) => {
+        let config = config::Config::default_for_test();
+
+        #[allow(unused_mut)]
+        let mut tera = tera::Tera::default();
+
+        // Add all shortcodes
+        $(
+            tera.add_raw_template(
+                &format!("shortcodes/{}", $shortcodes.filename()),
+                $shortcodes.output
+            ).expect("Failed to add raw template");
+        )*
+
+        let permalinks = std::collections::HashMap::new();
+        let mut context = rendering::RenderContext::new(
+            &tera,
+            &config,
+            &config.default_language,
+            "",
+            &permalinks,
+            front_matter::InsertAnchor::None,
+        );
+        let shortcode_def = utils::templates::get_shortcodes(&tera);
+        context.set_shortcode_definitions(&shortcode_def);
+
+        let rendered = rendering::render_content($in_str, &context);
+        assert!(rendered.is_ok());
+
+        let rendered = rendered.unwrap();
+        let toc = rendered.toc.clone();
+
+        assert!($toc == toc);
+    }
+}
+
+#[test]
+fn basic_toc() {
+    test_toc!("Hello World!", <Vec<HelperHeader>>::new(), []);
+    test_toc!("# ABC\n## DEF", vec![hh!("ABC", [hh!("DEF", [])])], []);
+}
+
+#[test]
+fn all_layers() {
+    test_toc!(
+        "# A\n## B\n### C\n#### D\n##### E\n###### F\n",
+        vec![hh!("A", [hh!("B", [hh!("C", [hh!("D", [hh!("E", [hh!("F", [])])])])])])],
+        []
+    );
+}
+
+#[test]
+fn multiple_on_layer() {
+    test_toc!(
+        "# A\n## B\n## C\n### D\n## E\n### F\n",
+        vec![hh!("A", [hh!("B", []), hh!("C", [hh!("D", [])]), hh!("E", [hh!("F", [])])])],
+        []
+    );
+}
+
+// const MD_SIMPLE1: ShortCode = ShortCode::new("simple", "Hello World!", true);
+// const MD_SIMPLE2: ShortCode = ShortCode::new("simple2", "Wow, much cool!", true);
+//
+// #[test]
+// fn with_shortcode_titles() {
+//     test_toc!(
+//         "# {{ simple() }}\n## {{ simple2() }}\n### ABC\n#### {{ simple() }}\n",
+//         vec![hh!(
+//             "Hello World!",
+//             [hh!("Wow, much cool!", [hh!("ABC", [hh!("Hello World!", [])])])]
+//         )],
+//         [MD_SIMPLE1, MD_SIMPLE2]
+//     );
+// }
+//
+// const MD_MULTILINE: ShortCode = ShortCode::new("multiline", "<div>\n    Wow!\n</div>", false);
+//
+// #[test]
+// fn with_multiline_shortcodes() {
+//     test_toc!(
+//         "# {{ multiline() }}\n{{ multiline() }}\n## {{ multiline()() }}\n",
+//         vec![hh!("Wow!", [hh!("Wow!", [])])],
+//         [MD_MULTILINE]
+//     );
+// }

components/site/src/lib.rs

@@ -14,7 +14,6 @@ use rayon::prelude::*;
 use tera::{Context, Tera};
 use walkdir::{DirEntry, WalkDir};
 
-use config::highlighting::export_theme_css;
 use config::{get_config, Config};
 use errors::{bail, Error, Result};
 use front_matter::InsertAnchor;
@@ -27,7 +26,7 @@ use utils::fs::{
 };
 use utils::minify;
 use utils::net::get_available_port;
-use utils::templates::render_template;
+use utils::templates::{render_template, ShortcodeDefinition};
 
 lazy_static! {
     /// The in-memory rendered map content
@@ -65,6 +64,7 @@ pub struct Site {
     /// Whether to load draft pages
     include_drafts: bool,
     build_mode: BuildMode,
+    shortcode_definitions: HashMap<String, ShortcodeDefinition>,
 }
 
 impl Site {
@@ -74,7 +74,6 @@ impl Site {
         let path = path.as_ref();
         let config_file = config_file.as_ref();
         let mut config = get_config(config_file)?;
-        config.markdown.load_extra_syntaxes(path)?;
 
         if let Some(theme) = config.theme.clone() {
             // Grab data from the extra section of the theme
@@ -82,6 +81,7 @@ impl Site {
         }
 
         let tera = load_tera(path, &config)?;
+        let shortcode_definitions = utils::templates::get_shortcodes(&tera);
 
         let content_path = path.join("content");
         let static_path = path.join("static");
@@ -103,6 +103,7 @@ impl Site {
             // We will allocate it properly later on
             library: Arc::new(RwLock::new(Library::new(0, 0, false))),
             build_mode: BuildMode::Disk,
+            shortcode_definitions,
         };
 
         Ok(site)
@@ -364,7 +365,13 @@ impl Site {
             .par_iter_mut()
             .map(|page| {
                 let insert_anchor = pages_insert_anchors[&page.file.path];
-                page.render_markdown(permalinks, tera, config, insert_anchor)
+                page.render_markdown(
+                    permalinks,
+                    tera,
+                    config,
+                    insert_anchor,
+                    &self.shortcode_definitions,
+                )
             })
             .collect::<Result<()>>()?;
 
@@ -373,7 +380,9 @@ impl Site {
             .values_mut()
             .collect::<Vec<_>>()
             .par_iter_mut()
-            .map(|section| section.render_markdown(permalinks, tera, config))
+            .map(|section| {
+                section.render_markdown(permalinks, tera, config, &self.shortcode_definitions)
+            })
             .collect::<Result<()>>()?;
 
         Ok(())
@@ -386,7 +395,13 @@ impl Site {
         if render_md {
             let insert_anchor =
                 self.find_parent_section_insert_anchor(&page.file.parent, &page.lang);
-            page.render_markdown(&self.permalinks, &self.tera, &self.config, insert_anchor)?;
+            page.render_markdown(
+                &self.permalinks,
+                &self.tera,
+                &self.config,
+                insert_anchor,
+                &self.shortcode_definitions,
+            )?;
         }
 
         let mut library = self.library.write().expect("Get lock for add_page");
@@ -413,7 +428,12 @@ impl Site {
     pub fn add_section(&mut self, mut section: Section, render_md: bool) -> Result<()> {
         self.permalinks.insert(section.file.relative.clone(), section.permalink.clone());
         if render_md {
-            section.render_markdown(&self.permalinks, &self.tera, &self.config)?;
+            section.render_markdown(
+                &self.permalinks,
+                &self.tera,
+                &self.config,
+                &self.shortcode_definitions,
+            )?;
         }
         let mut library = self.library.write().expect("Get lock for add_section");
         library.remove_section(&section.file.path);
@@ -594,8 +614,11 @@ impl Site {
             let asset_path = asset.as_path();
             self.copy_asset(
                 asset_path,
-                &current_path
+                &current_path.join(
-                    .join(asset_path.file_name().expect("Couldn't get filename from page asset")),
+                    asset_path
+                        .strip_prefix(&page.file.path.parent().unwrap())
+                        .expect("Couldn't get filename from page asset"),
+                ),
             )?;
         }
 
@@ -691,7 +714,7 @@ impl Site {
         for t in &self.config.markdown.highlight_themes_css {
             let p = self.static_path.join(&t.filename);
             if !p.exists() {
-                let content = export_theme_css(&t.theme);
+                let content = &self.config.markdown.export_theme_css(&t.theme);
                 create_file(&p, &content)?;
             }
         }
@@ -831,7 +854,13 @@ impl Site {
                 if taxonomy.kind.is_paginated() {
                     self.render_paginated(
                         comp.clone(),
-                        &Paginator::from_taxonomy(taxonomy, item, &library),
+                        &Paginator::from_taxonomy(
+                            taxonomy,
+                            item,
+                            &library,
+                            &self.tera,
+                            &self.config.theme,
+                        ),
                     )?;
                 } else {
                     let single_output =
@@ -988,7 +1017,9 @@ impl Site {
             self.copy_asset(
                 asset_path,
                 &output_path.join(
-                    asset_path.file_name().expect("Failed to get asset filename for section"),
+                    asset_path
+                        .strip_prefix(&section.file.path.parent().unwrap())
+                        .expect("Failed to get asset filename for section"),
                 ),
             )?;
         }

components/templates/src/filters.rs

@@ -33,9 +33,15 @@ impl MarkdownFilter {
 
 impl TeraFilter for MarkdownFilter {
     fn filter(&self, value: &Value, args: &HashMap<String, Value>) -> TeraResult<Value> {
+        // NOTE: RenderContext below is not aware of the current language
+        // However, it should not be a problem because the surrounding tera
+        // template has language context, and will most likely call a piece of
+        // markdown respecting language preferences.
         let mut context = RenderContext::from_config(&self.config);
         context.permalinks = Cow::Borrowed(&self.permalinks);
         context.tera = Cow::Borrowed(&self.tera);
+        let def = utils::templates::get_shortcodes(&self.tera);
+        context.set_shortcode_definitions(&def);
 
         let s = try_get_value!("markdown", "value", String, value);
         let inline = match args.get("inline") {
@@ -123,6 +129,24 @@ mod tests {
         assert_eq!(result.unwrap(), to_value(&"<h1 id=\"hey\">Hey</h1>\n").unwrap());
     }
 
+    #[test]
+    fn markdown_filter_override_lang() {
+        // We're checking that we can use a workaround to explicitly provide `lang` in markdown filter from tera,
+        // because otherwise markdown filter shortcodes are not aware of the current language
+        // NOTE: This should also work for `nth` although i don't see a reason to do that
+        let args = HashMap::new();
+        let config = Config::default();
+        let permalinks = HashMap::new();
+        let mut tera =
+            super::load_tera(&PathBuf::new(), &config).map_err(tera::Error::msg).unwrap();
+        tera.add_raw_template("shortcodes/explicitlang.html", "a{{ lang }}a").unwrap();
+        let filter = MarkdownFilter { config, permalinks, tera };
+        let result = filter.filter(&to_value(&"{{ explicitlang(lang='jp') }}").unwrap(), &args);
+        println!("{:?}", result);
+        assert!(result.is_ok());
+        assert_eq!(result.unwrap(), to_value(&"ajpa").unwrap());
+    }
+
     #[test]
     fn markdown_filter_inline() {
         let mut args = HashMap::new();

components/templates/src/global_fns/content.rs

@@ -40,10 +40,17 @@ impl TeraFn for GetTaxonomyUrl {
         let lang =
             optional_arg!(String, args.get("lang"), "`get_taxonomy`: `lang` must be a string")
                 .unwrap_or_else(|| self.default_lang.clone());
+        let required = optional_arg!(
+            bool,
+            args.get("required"),
+            "`get_taxonomy_url`: `required` must be a boolean (true or false)"
+        )
+        .unwrap_or(true);
 
-        let container = match self.taxonomies.get(&format!("{}-{}", kind, lang)) {
+        let container = match (self.taxonomies.get(&format!("{}-{}", kind, lang)), required) {
-            Some(c) => c,
+            (Some(c), _) => c,
-            None => {
+            (None, false) => return Ok(Value::Null),
+            (None, true) => {
                 return Err(format!(
                     "`get_taxonomy_url` received an unknown taxonomy as kind: {}",
                     kind
@@ -154,14 +161,21 @@ impl TeraFn for GetTaxonomy {
             args.get("kind"),
             "`get_taxonomy` requires a `kind` argument with a string value"
         );
+        let required = optional_arg!(
+            bool,
+            args.get("required"),
+            "`get_taxonomy`: `required` must be a boolean (true or false)"
+        )
+        .unwrap_or(true);
 
         let lang =
             optional_arg!(String, args.get("lang"), "`get_taxonomy`: `lang` must be a string")
                 .unwrap_or_else(|| self.default_lang.clone());
 
-        match self.taxonomies.get(&format!("{}-{}", kind, lang)) {
+        match (self.taxonomies.get(&format!("{}-{}", kind, lang)), required) {
-            Some(t) => Ok(to_value(t.to_serialized(&self.library.read().unwrap())).unwrap()),
+            (Some(t), _) => Ok(to_value(t.to_serialized(&self.library.read().unwrap())).unwrap()),
-            None => {
+            (None, false) => Ok(Value::Null),
+            (None, true) => {
                 Err(format!("`get_taxonomy` received an unknown taxonomy as kind: {}", kind).into())
             }
         }

components/templates/src/global_fns/files.rs

@@ -117,10 +117,15 @@ impl TeraFn for GetUrl {
             }
 
             if cachebust {
-                match search_for_file(&self.base_path, &path_with_lang, &self.config.theme, &self.output_path)
+                match search_for_file(
-                    .map_err(|e| format!("`get_url`: {}", e))?
+                    &self.base_path,
-                    .and_then(|(p, _)| fs::File::open(&p).ok())
+                    &path_with_lang,
-                    .and_then(|f| compute_file_hash::<Sha256>(f, false).ok())
+                    &self.config.theme,
+                    &self.output_path,
+                )
+                .map_err(|e| format!("`get_url`: {}", e))?
+                .and_then(|(p, _)| fs::File::open(&p).ok())
+                .and_then(|f| compute_file_hash::<Sha256>(f, false).ok())
                 {
                     Some(hash) => {
                         permalink = format!("{}?h={}", permalink, hash);
@@ -135,6 +140,19 @@ impl TeraFn for GetUrl {
                 };
             }
 
+            if cfg!(target_os = "windows") {
+                permalink = match url::Url::parse(&permalink) {
+                    Ok(parsed) => parsed.into(),
+                    Err(_) => {
+                        return Err(format!(
+                            "`get_url`: Could not parse link `{}` as a valid URL",
+                            permalink
+                        )
+                        .into())
+                    }
+                };
+            }
+
             Ok(to_value(permalink).unwrap())
         }
     }
@@ -176,14 +194,15 @@ impl TeraFn for GetFileHash {
         )
         .unwrap_or(true);
 
-        let file_path = match search_for_file(&self.base_path, &path, &self.theme, &self.output_path)
+        let file_path =
-            .map_err(|e| format!("`get_file_hash`: {}", e))?
+            match search_for_file(&self.base_path, &path, &self.theme, &self.output_path)
-        {
+                .map_err(|e| format!("`get_file_hash`: {}", e))?
-            Some((f, _)) => f,
+            {
-            None => {
+                Some((f, _)) => f,
-                return Err(format!("`get_file_hash`: Cannot find file: {}", path).into());
+                None => {
-            }
+                    return Err(format!("`get_file_hash`: Cannot find file: {}", path).into());
-        };
+                }
+            };
 
         let f = match std::fs::File::open(file_path) {
             Ok(f) => f,
@@ -241,7 +260,12 @@ title = "A title"
     #[test]
     fn can_add_cachebust_to_url() {
         let dir = create_temp_dir();
-        let static_fn = GetUrl::new(dir.path().to_path_buf(), Config::default(), HashMap::new(), PathBuf::new());
+        let static_fn = GetUrl::new(
+            dir.path().to_path_buf(),
+            Config::default(),
+            HashMap::new(),
+            PathBuf::new(),
+        );
         let mut args = HashMap::new();
         args.insert("path".to_string(), to_value("app.css").unwrap());
         args.insert("cachebust".to_string(), to_value(true).unwrap());
@@ -251,7 +275,12 @@ title = "A title"
     #[test]
     fn can_add_trailing_slashes() {
         let dir = create_temp_dir();
-        let static_fn = GetUrl::new(dir.path().to_path_buf(), Config::default(), HashMap::new(), PathBuf::new());
+        let static_fn = GetUrl::new(
+            dir.path().to_path_buf(),
+            Config::default(),
+            HashMap::new(),
+            PathBuf::new(),
+        );
         let mut args = HashMap::new();
         args.insert("path".to_string(), to_value("app.css").unwrap());
         args.insert("trailing_slash".to_string(), to_value(true).unwrap());
@@ -261,7 +290,12 @@ title = "A title"
     #[test]
     fn can_add_slashes_and_cachebust() {
         let dir = create_temp_dir();
-        let static_fn = GetUrl::new(dir.path().to_path_buf(), Config::default(), HashMap::new(), PathBuf::new());
+        let static_fn = GetUrl::new(
+            dir.path().to_path_buf(),
+            Config::default(),
+            HashMap::new(),
+            PathBuf::new(),
+        );
         let mut args = HashMap::new();
         args.insert("path".to_string(), to_value("app.css").unwrap());
         args.insert("trailing_slash".to_string(), to_value(true).unwrap());
@@ -272,7 +306,12 @@ title = "A title"
     #[test]
     fn can_link_to_some_static_file() {
         let dir = create_temp_dir();
-        let static_fn = GetUrl::new(dir.path().to_path_buf(), Config::default(), HashMap::new(), PathBuf::new());
+        let static_fn = GetUrl::new(
+            dir.path().to_path_buf(),
+            Config::default(),
+            HashMap::new(),
+            PathBuf::new(),
+        );
         let mut args = HashMap::new();
         args.insert("path".to_string(), to_value("app.css").unwrap());
         assert_eq!(static_fn.call(&args).unwrap(), "http://a-website.com/app.css");
@@ -290,7 +329,8 @@ title = "A title"
         create_file(&public.join("style.css"), "// Hello world")
             .expect("Failed to create file in output directory");
 
-        let static_fn = GetUrl::new(dir.path().to_path_buf(), Config::default(), HashMap::new(), public);
+        let static_fn =
+            GetUrl::new(dir.path().to_path_buf(), Config::default(), HashMap::new(), public);
         let mut args = HashMap::new();
         args.insert("path".to_string(), to_value("style.css").unwrap());
         assert_eq!(static_fn.call(&args).unwrap(), "http://a-website.com/style.css");
@@ -300,7 +340,8 @@ title = "A title"
     fn error_when_language_not_available() {
         let config = Config::parse(CONFIG_DATA).unwrap();
         let dir = create_temp_dir();
-        let static_fn = GetUrl::new(dir.path().to_path_buf(), config, HashMap::new(), PathBuf::new());
+        let static_fn =
+            GetUrl::new(dir.path().to_path_buf(), config, HashMap::new(), PathBuf::new());
         let mut args = HashMap::new();
         args.insert("path".to_string(), to_value("@/a_section/a_page.md").unwrap());
         args.insert("lang".to_string(), to_value("it").unwrap());
@@ -361,7 +402,8 @@ title = "A title"
     fn can_get_feed_url_with_default_language() {
         let config = Config::parse(CONFIG_DATA).unwrap();
         let dir = create_temp_dir();
-        let static_fn = GetUrl::new(dir.path().to_path_buf(), config.clone(), HashMap::new(), PathBuf::new());
+        let static_fn =
+            GetUrl::new(dir.path().to_path_buf(), config.clone(), HashMap::new(), PathBuf::new());
         let mut args = HashMap::new();
         args.insert("path".to_string(), to_value(config.feed_filename).unwrap());
         args.insert("lang".to_string(), to_value("fr").unwrap());
@@ -372,7 +414,8 @@ title = "A title"
     fn can_get_feed_url_with_other_language() {
         let config = Config::parse(CONFIG_DATA).unwrap();
         let dir = create_temp_dir();
-        let static_fn = GetUrl::new(dir.path().to_path_buf(), config.clone(), HashMap::new(), PathBuf::new());
+        let static_fn =
+            GetUrl::new(dir.path().to_path_buf(), config.clone(), HashMap::new(), PathBuf::new());
         let mut args = HashMap::new();
         args.insert("path".to_string(), to_value(config.feed_filename).unwrap());
         args.insert("lang".to_string(), to_value("en").unwrap());
@@ -456,6 +499,27 @@ title = "A title"
         );
     }
 
+    #[test]
+    fn can_resolve_asset_path_to_valid_url() {
+        let config = Config::parse(CONFIG_DATA).unwrap();
+        let dir = create_temp_dir();
+        let static_fn =
+            GetUrl::new(dir.path().to_path_buf(), config, HashMap::new(), PathBuf::new());
+        let mut args = HashMap::new();
+        args.insert(
+            "path".to_string(),
+            to_value(dir.path().join("app.css").strip_prefix(std::env::temp_dir()).unwrap())
+                .unwrap(),
+        );
+        assert_eq!(
+            static_fn.call(&args).unwrap(),
+            format!(
+                "https://remplace-par-ton-url.fr/{}/app.css",
+                dir.path().file_stem().unwrap().to_string_lossy()
+            )
+        )
+    }
+
     #[test]
     fn error_when_file_not_found_for_hash() {
         let dir = create_temp_dir();

components/templates/src/global_fns/helpers.rs

@@ -21,7 +21,8 @@ pub fn search_for_file(
     theme: &Option<String>,
     output_path: &Path,
 ) -> Result<Option<(PathBuf, String)>> {
-    let mut search_paths = vec![base_path.join("static"), base_path.join("content"), base_path.join(output_path)];
+    let mut search_paths =
+        vec![base_path.join("static"), base_path.join("content"), base_path.join(output_path)];
     if let Some(t) = theme {
         search_paths.push(base_path.join("themes").join(t).join("static"));
     }

components/templates/src/global_fns/images.rs

@@ -62,14 +62,15 @@ impl TeraFn for ResizeImage {
         }
 
         let mut imageproc = self.imageproc.lock().unwrap();
-        let (file_path, unified_path) = match search_for_file(&self.base_path, &path, &self.theme, &self.output_path)
+        let (file_path, unified_path) =
-            .map_err(|e| format!("`resize_image`: {}", e))?
+            match search_for_file(&self.base_path, &path, &self.theme, &self.output_path)
-        {
+                .map_err(|e| format!("`resize_image`: {}", e))?
-            Some(f) => f,
+            {
-            None => {
+                Some(f) => f,
-                return Err(format!("`resize_image`: Cannot find file: {}", path).into());
+                None => {
-            }
+                    return Err(format!("`resize_image`: Cannot find file: {}", path).into());
-        };
+                }
+            };
 
         let response = imageproc
             .enqueue(unified_path, file_path, &op, width, height, &format, quality)
@@ -108,17 +109,18 @@ impl TeraFn for GetImageMetadata {
         )
         .unwrap_or(false);
 
-        let (src_path, unified_path) = match search_for_file(&self.base_path, &path, &self.theme, &self.output_path)
+        let (src_path, unified_path) =
-            .map_err(|e| format!("`get_image_metadata`: {}", e))?
+            match search_for_file(&self.base_path, &path, &self.theme, &self.output_path)
-        {
+                .map_err(|e| format!("`get_image_metadata`: {}", e))?
-            Some((f, p)) => (f, p),
+            {
-            None => {
+                Some((f, p)) => (f, p),
-                if allow_missing {
+                None => {
-                    return Ok(Value::Null);
+                    if allow_missing {
+                        return Ok(Value::Null);
+                    }
+                    return Err(format!("`get_image_metadata`: Cannot find path: {}", path).into());
                 }
-                return Err(format!("`get_image_metadata`: Cannot find path: {}", path).into());
+            };
-            }
-        };
 
         let mut cache = self.result_cache.lock().expect("result cache lock");
         if let Some(cached_result) = cache.get(&unified_path) {

components/templates/src/global_fns/load_data.rs

@@ -96,7 +96,7 @@ impl DataSource {
         }
 
         if let Some(path) = path_arg {
-            return match search_for_file(&base_path, &path, &theme, &output_path)
+            return match search_for_file(base_path, &path, theme, output_path)
                 .map_err(|e| format!("`load_data`: {}", e))?
             {
                 Some((f, _)) => Ok(Some(DataSource::Path(f))),
@@ -226,7 +226,13 @@ impl TeraFn for LoadData {
 
         // If the file doesn't exist, source is None
         let data_source = match (
-            DataSource::from_args(path_arg.clone(), url_arg, &self.base_path, &self.theme, &self.output_path),
+            DataSource::from_args(
+                path_arg.clone(),
+                url_arg,
+                &self.base_path,
+                &self.theme,
+                &self.output_path,
+            ),
             required,
         ) {
             // If the file was not required, return a Null value to the template

components/utils/src/templates.rs

@@ -15,6 +15,58 @@ macro_rules! render_default_tpl {
     }};
 }
 
+#[derive(Debug, Clone, PartialEq)]
+pub enum ShortcodeFileType {
+    Markdown,
+    Html,
+}
+
+#[derive(Debug, Clone)]
+pub struct ShortcodeDefinition {
+    pub file_type: ShortcodeFileType,
+    pub tera_name: String,
+}
+impl ShortcodeDefinition {
+    pub fn new(file_type: ShortcodeFileType, tera_name: &str) -> ShortcodeDefinition {
+        let tera_name = tera_name.to_string();
+
+        ShortcodeDefinition { file_type, tera_name }
+    }
+}
+
+/// Fetches all the shortcodes from the Tera instances
+pub fn get_shortcodes(tera: &Tera) -> HashMap<String, ShortcodeDefinition> {
+    let mut shortcode_definitions = HashMap::new();
+
+    for (identifier, template) in tera.templates.iter() {
+        let (file_type, ext_len) = if template.name.ends_with(".md") {
+            (ShortcodeFileType::Markdown, "md".len())
+        } else {
+            (ShortcodeFileType::Html, "html".len())
+        };
+
+        if template.name.starts_with("shortcodes/") {
+            let head_len = "shortcodes/".len();
+            shortcode_definitions.insert(
+                identifier[head_len..(identifier.len() - ext_len - 1)].to_string(),
+                ShortcodeDefinition::new(file_type, &template.name),
+            );
+            continue;
+        }
+
+        if template.name.starts_with("__zola_builtins/shortcodes/") {
+            let head_len = "__zola_builtins/shortcodes/".len();
+            shortcode_definitions.insert(
+                identifier[head_len..(identifier.len() - ext_len - 1)].to_string(),
+                ShortcodeDefinition::new(file_type, &template.name),
+            );
+            continue;
+        }
+    }
+
+    shortcode_definitions
+}
+
 /// Renders the given template with the given context, but also ensures that, if the default file
 /// is not found, it will look up for the equivalent template for the current theme if there is one.
 /// Lastly, if it's a default template (index, section or page), it will just return an empty string
@@ -25,23 +77,8 @@ pub fn render_template(
     context: Context,
     theme: &Option<String>,
 ) -> Result<String> {
-    // check if it is in the templates
+    if let Some(template) = check_template_fallbacks(name, tera, theme) {
-    if tera.templates.contains_key(name) {
+        return tera.render(&template, &context).map_err(std::convert::Into::into);
-        return tera.render(name, &context).map_err(std::convert::Into::into);
-    }
-
-    // check if it is part of a theme
-    if let Some(ref t) = *theme {
-        let theme_template_name = format!("{}/templates/{}", t, name);
-        if tera.templates.contains_key(&theme_template_name) {
-            return tera.render(&theme_template_name, &context).map_err(std::convert::Into::into);
-        }
-    }
-
-    // check if it is part of ZOLA_TERA defaults
-    let default_name = format!("__zola_builtins/{}", name);
-    if tera.templates.contains_key(&default_name) {
-        return tera.render(&default_name, &context).map_err(std::convert::Into::into);
     }
 
     // maybe it's a default one?
@@ -78,8 +115,40 @@ pub fn rewrite_theme_paths(tera_theme: &mut Tera, theme: &str) {
     tera_theme.templates.extend(new_templates);
 }
 
+/// Checks for the presence of a given template. If none is found, also looks for a
+/// fallback in theme and default templates. Returns the path of the most specific
+/// template found, or none if none are present.
+pub fn check_template_fallbacks<'a>(
+    name: &'a str,
+    tera: &'a Tera,
+    theme: &Option<String>,
+) -> Option<&'a str> {
+    // check if it is in the templates
+    if tera.templates.contains_key(name) {
+        return Some(name);
+    }
+
+    // check if it is part of a theme
+    if let Some(ref t) = *theme {
+        let theme_template_name = format!("{}/templates/{}", t, name);
+        if let Some((key, _)) = tera.templates.get_key_value(&theme_template_name) {
+            return Some(key);
+        }
+    }
+
+    // check if it is part of ZOLA_TERA defaults
+    let default_name = format!("__zola_builtins/{}", name);
+    if let Some((key, _)) = tera.templates.get_key_value(&default_name) {
+        return Some(key);
+    }
+
+    None
+}
+
 #[cfg(test)]
 mod tests {
+    use crate::templates::check_template_fallbacks;
+
     use super::rewrite_theme_paths;
     use tera::Tera;
 
@@ -105,4 +174,23 @@ mod tests {
             Some("index.html".to_string())
         );
     }
+
+    #[test]
+    fn template_fallback_is_successful() {
+        let mut tera = Tera::parse("test-templates/*.html").unwrap();
+        tera.add_raw_template(&"hyde/templates/index.html", "Hello").unwrap();
+        tera.add_raw_template(&"hyde/templates/theme-only.html", "Hello").unwrap();
+
+        // Check finding existing template
+        assert_eq!(check_template_fallbacks("index.html", &tera, &None), Some("index.html"));
+
+        // Check trying to find non-existant template
+        assert_eq!(check_template_fallbacks("not-here.html", &tera, &None), None);
+
+        // Check theme fallback
+        assert_eq!(
+            check_template_fallbacks("theme-only.html", &tera, &Some("hyde".to_string())),
+            Some("hyde/templates/theme-only.html")
+        );
+    }
 }

docs/content/documentation/content/linking.md

@@ -39,11 +39,12 @@ This option is set at the section level: the `insert_anchor_links` variable on t
 
 The default template is very basic and will need CSS tweaks in your project to look decent.
 If you want to change the anchor template, it can be easily overwritten by
-creating an `anchor-link.html` file in the `templates` directory.
+creating an `anchor-link.html` file in the `templates` directory. [Here](https://github.com/getzola/zola/blob/master/components/templates/src/builtins/anchor-link.html) you can find the default template.
 
 The anchor link template has the following variables:
 
 - `id`: the heading's id after applying the rules defined by `slugify.anchors`
+- `lang`: the current language, unless called from the `markdown` template filter, in which case it will always be `en`
 - `level`: the heading level (between 1 and 6)
 
 ## Internal links

docs/content/documentation/content/shortcodes.md

@@ -134,10 +134,19 @@ If you want to have some content that looks like a shortcode but not have Zola t
 you will need to escape it by using `{%/*` and `*/%}` instead of `{%` and `%}`. You won't need to escape
 anything else until the closing tag.
 
+## Shortcode context
+
+Every shortcode can access some variables, beyond what you explicitly passed as parameter. These variables are explained in the following subsections:
+
+- invocation count (`nth`)
+- current language (`lang`), unless called from the `markdown` template filter (in which case it will always be the same value as `default_language` in configuration, or `en` when it is unset)
+
+When one of these variables conflict with a variable passed as argument, the argument value will be used.
+
 ### Invocation Count
 
 Every shortcode context is passed in a variable named `nth` that tracks how many times a particular shortcode has
-been invoked in a Markdown file. Given a shortcode `true_statement.html` template:
+been invoked in the current Markdown file. Given a shortcode `true_statement.html` template:
 
 ```jinja2
 <p id="number{{ nth }}">{{ value }} is equal to {{ nth }}.</p>
@@ -152,6 +161,18 @@ It could be used in our Markdown as follows:
 
 This is useful when implementing custom markup for features such as sidenotes or end notes.
 
+### Current language
+
+**NOTE:** When calling a shortcode from within the `markdown` template filter, the `lang` variable will always be `en`. If you feel like you need that, please consider using template macros instead. If you really need that, you can rewrite your Markdown content to pass `lang` as argument to the shortcode.
+
+Every shortcode can access the current language in the `lang` variable in the context. This is useful for presenting/filtering information in a shortcode depending in a per-language manner. For example, to display a per-language book cover for the current page in a shortcode called `bookcover.md`:
+
+```jinja2
+![Book cover in {{ lang }}](cover.{{ lang }}.png)
+```
+
+You can then use it in your Markdown like so: `{{/* bookcover() */}}`
+
 ## Built-in shortcodes
 
 Zola comes with a few built-in shortcodes. If you want to override a default shortcode template,
@@ -164,7 +185,7 @@ Embed a responsive player for a YouTube video.
 The arguments are:
 
 - `id`: the video id (mandatory)
-- `playlist: the playlist id (optional)
+- `playlist`: the playlist id (optional)
 - `class`: a class to add to the `<div>` surrounding the iframe
 - `autoplay`: when set to "true", the video autoplays on load
 

docs/content/documentation/content/syntax-highlighting.md

@@ -150,7 +150,7 @@ Here is a full list of supported languages and their short names:
 Note: due to some issues with the JavaScript syntax, the TypeScript syntax will be used instead.
 
 If you want to highlight a language not on this list, please open an issue or a pull request on the [Zola repo](https://github.com/getzola/zola).
-Alternatively, the `extra_syntaxes` configuration option can be used to add additional syntax files.
+Alternatively, the `extra_syntaxes_and_themes` configuration option can be used to add additional syntax (and theme) files.
 
 If your site source is laid out as follows:
 
@@ -169,7 +169,7 @@ If your site source is laid out as follows:
     └── ...
 ```
 
-you would set your `extra_syntaxes` to `["syntaxes", "syntaxes/Sublime-Language1"]` to load `lang1.sublime-syntax` and `lang2.sublime-syntax`.
+you would set your `extra_syntaxes_and_themes` to `["syntaxes", "syntaxes/Sublime-Language1"]` to load `lang1.sublime-syntax` and `lang2.sublime-syntax`.
 
 ## Inline VS classed highlighting
 
@@ -347,3 +347,40 @@ Line 2 and 7 are comments that are not shown in the final output.
 
 When line numbers are active, the code block is turned into a table with one row and two cells. The first cell contains the line number and the second cell contains the code.
 Highlights are done via the `<mark>` HTML tag. When a line with line number is highlighted two `<mark>` tags are created: one around the line number(s) and one around the code.
+
+## Custom Highlighting Themes
+
+The default *theme* for syntax highlighting is called `base16-ocean-dark`, you can choose another theme from the built in set of highlight themes using the `highlight_theme` configuration option.
+For example, this documentation site currently uses the `kronuz` theme, which is built in.
+
+```
+[markdown]
+highlight_code = true
+highlight_theme = "kronuz"
+```
+
+Alternatively, the `extra_syntaxes_and_themes` configuration option can be used to add additional theme files.
+You can load your own highlight theme from a TextMate `.tmTheme` file.
+
+It works the same way as adding extra syntaxes. It should contain a list of paths to folders containing the .tmTheme files you want to include.
+You would then set `highlight_theme` to the name of one of these files, without the `.tmTheme` extension.
+
+If your site source is laid out as follows:
+
+```
+.
+├── config.toml
+├── content/
+│   └── ...
+├── static/
+│   └── ...
+├── highlight_themes/
+│   ├── MyGroovyTheme/
+│   │   └── theme1.tmTheme
+│   ├── theme2.tmTheme
+└── templates/
+    └── ...
+```
+
+you would set your `extra_highlight_themes` to `["highlight_themes", "highlight_themes/MyGroovyTheme"]` to load `theme1.tmTheme` and `theme2.tmTheme`.
+Then choose one of them to use, say theme1, by setting `highlight_theme = theme1`.

docs/content/documentation/deployment/github-pages.md

@@ -18,7 +18,7 @@ We can use any continuous integration (CI) server to build and deploy our site.
 
 In either case, it seems to work best if you use `git submodule` to include your theme, e.g.:
 
-```sh
+```bash
 git submodule add https://github.com/getzola/after-dark.git themes/after-dark
 ```
 
@@ -110,7 +110,7 @@ Depending on how you added your theme, Travis may not know how to access
 it. The best way to ensure that it will have full access to the theme is to use git
 submodules. When doing this, ensure that you are using the `https` version of the URL.
 
-```sh
+```bash
 $ git submodule add {THEME_URL} themes/{THEME_NAME}
 ```
 

docs/content/documentation/deployment/gitlab-pages.md

@@ -21,13 +21,13 @@ This guide assumes that your Zola project is located in the root of your reposit
 Depending on how you added your theme, your repository may not contain it. The best way to ensure that the theme will
 be added is to use submodules. When doing this, ensure that you are using the `https` version of the URL.
 
-```sh
+```bash
 $ git submodule add {THEME_URL} themes/{THEME_NAME}
 ```
 
 For example, this could look like:
 
-```sh
+```bash
 $ git submodule add https://github.com/getzola/hyde.git themes/hyde
 ```
 

docs/content/documentation/getting-started/cli-usage.md

@@ -46,7 +46,7 @@ $ zola build --base-url $DEPLOY_URL
 This is useful for example when you want to deploy previews of a site to a dynamic URL, such as Netlify
 deploy previews.
 
-You can override the default output directory `public` by passing another value to the `output-dir` flag (if this directory already exists, it is deleted).
+You can override the default output directory `public` by passing another value to the `output-dir` flag (if this directory already exists, the user will be prompted whether to replace the folder).
 
 ```bash
 $ zola build --output-dir $DOCUMENT_ROOT

docs/content/documentation/getting-started/configuration.md

@@ -236,6 +236,9 @@ Zola currently has the following highlight themes available:
 Zola uses the Sublime Text themes, making it very easy to add more.
 If you want a theme not listed above, please open an issue or a pull request on the [Zola repo](https://github.com/getzola/zola).
 
+Alternatively you can use the `extra_syntaxes_and_themes` configuration option to load your own custom themes from a .tmTheme file.
+See [Syntax Highlighting](@/documentation/content/syntax-highlighting.md) for more details.
+
 ## Slugification strategies
 
 By default, Zola will turn every path, taxonomies and anchors to a slug, an ASCII representation with no special characters.

docs/content/documentation/getting-started/installation.md

@@ -130,7 +130,7 @@ $ choco install zola
 Zola does not work in PowerShell ISE.
 
 ## From source
-To build Zola from source, you will need to have Git, [Rust (at least 1.49) and Cargo](https://www.rust-lang.org/)
+To build Zola from source, you will need to have Git, [Rust and Cargo](https://www.rust-lang.org/)
 installed. You will also need to meet additional dependencies to compile [libsass](https://github.com/sass/libsass):
 
 - OSX, Linux and other Unix-like operating systems: `make` (`gmake` on BSDs), `g++`, `libssl-dev`

docs/content/documentation/templates/overview.md

@@ -23,6 +23,8 @@ A few variables are available on all templates except feeds and the sitemap:
 Config variables can be accessed like `config.variable`, in HTML for example with `{{ config.base_url }}`.
 The 404 template does not get `current_path` and `current_url` (this information cannot be determined).
 
+On top of the `config` attributes mentioned above, it also gets `config.mode` which is whether it's run in `build`, `serve` or `check`.
+
 ## Standard templates
 By default, Zola will look for three templates: `index.html`, which is applied
 to the site homepage; `section.html`, which is applied to all sections (any HTML
@@ -64,7 +66,10 @@ Zola adds a few filters in addition to [those](https://tera.netlify.com/docs/#fi
 in Tera.
 
 ### markdown
-Converts the given variable to HTML using Markdown. Please note that shortcodes evaluated by this filter cannot access the current rendering context. `config` will be available, but accessing `section` or `page` (among others) from a shortcode called within the `markdown` filter will prevent your site from building. See [this discussion](https://github.com/getzola/zola/pull/1358).
+Converts the given variable to HTML using Markdown. There are a few differences compared to page/section Markdown rendering:
+
+- shortcodes evaluated by this filter cannot access the current rendering context: `config` will be available, but accessing `section` or `page` (among others) from a shortcode called within the `markdown` filter will prevent your site from building (see [this discussion](https://github.com/getzola/zola/pull/1358))
+- `lang` in shortcodes will always be equal to the site's `default_lang` (or `en` otherwise) ; it should not be a problem, but if it is in most cases, but if you need to use language-aware shortcodes in this filter, please refer to the [Shortcode context](@/documentation/content/shortcodes.md#shortcode-context) section of the docs.
 
 By default, the filter will wrap all text in a paragraph. To disable this behaviour, you can
 pass `true` to the inline argument:
@@ -153,6 +158,8 @@ the value should be the same as the one in the front matter, not the slugified v
 
 `lang` (optional) default to `config.default_language` in config.toml
 
+`required` (optional) if a taxonomy is defined but there isn't any content that uses it then throw an error. Defaults to true.
+
 ### `get_taxonomy`
 Gets the whole taxonomy of a specific kind.
 
@@ -169,6 +176,8 @@ items: Array<TaxonomyTerm>;
 
 `lang` (optional) default to `config.default_language` in config.toml
 
+`required` (optional) if a taxonomy is defined but there isn't any content that uses it then throw an error. Defaults to true.
+
 See the [Taxonomies documentation](@/documentation/templates/taxonomies.md) for a full documentation of those types.
 
 ### `get_url`
@@ -194,14 +203,14 @@ This can also be used to get the permalinks for static assets, for example if
 we want to link to the file that is located at `static/css/app.css`:
 
 ```jinja2
-{{/* get_url(path="static/css/app.css") */}}
+{{/* get_url(path="css/app.css") */}}
 ```
 
 By default, assets will not have a trailing slash. You can force one by passing `trailing_slash=true` to the `get_url` function.
 An example is:
 
 ```jinja2
-{{/* get_url(path="static/css/app.css", trailing_slash=true) */}}
+{{/* get_url(path="css/app.css", trailing_slash=true) */}}
 ```
 
 In the case of non-internal links, you can also add a cachebust of the format `?h=<sha256>` at the end of a URL

docs/content/documentation/templates/taxonomies.md

@@ -3,11 +3,15 @@ title = "Taxonomies"
 weight = 40
 +++
 
-Zola will look up the following files in the `templates` directory:
+Zola will look up the following, taxon-specific files in the `templates` directory:
 
 - `$TAXONOMY_NAME/single.html`
 - `$TAXONOMY_NAME/list.html`
 
+if they are not found, it will attempt to fall back on the following generic template files:
+- `taxonomy_single.html`
+- `taxonomy_list.html`
+
 First, `TaxonomyTerm` has the following fields:
 
 ```ts

docs/content/documentation/themes/installing-and-using-themes.md

@@ -35,7 +35,7 @@ in the configuration file is `simple-blog`. Also make sure to place the variable
 Any file from the theme can be overridden by creating a file with the same path and name in your `templates` or `static`
 directory. Here are a few examples of that, assuming that the theme name is `simple-blog`:
 
-```txt
+```
 templates/pages/post.html -> replace themes/simple-blog/templates/pages/post.html
 templates/macros.html -> replace themes/simple-blog/templates/macros.html
 static/js/site.js -> replace themes/simple-blog/static/js/site.js

is-ehh.svg

@@ -1,3 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" style="fill:#dbab09" width="16" height="16" viewBox="0 0 24 24">
-  <circle cx="12" cy="12" r="8"/>
-</svg>

is-no.svg

@@ -1,3 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" style="fill:#cb2431" width="16" height="16" viewBox="0 0 24 24">
-  <path d="M23 20.168l-8.185-8.187 8.185-8.174-2.832-2.807-8.182 8.179-8.176-8.179-2.81 2.81 8.186 8.196-8.186 8.184 2.81 2.81 8.203-8.192 8.18 8.192z"/>
-</svg>

is-yes.svg

@@ -1,3 +0,0 @@
-<svg xmlns="http://www.w3.org/2000/svg" style="fill:#28a745" width="16" height="16" viewBox="0 0 24 24">
-  <path d="M9 21.035l-9-8.638 2.791-2.87 6.156 5.874 12.21-12.436 2.843 2.817z"/>
-</svg>

src/cmd/build.rs

@@ -1,9 +1,12 @@
 use std::path::Path;
 
-use errors::Result;
+use errors::{Error, Result};
 use site::Site;
 
 use crate::console;
+use crate::prompt::ask_bool_timeout;
+
+const BUILD_PROMPT_TIMEOUT_MILLIS: u64 = 10_000;
 
 pub fn build(
     root_dir: &Path,
@@ -14,6 +17,27 @@ pub fn build(
 ) -> Result<()> {
     let mut site = Site::new(root_dir, config_file)?;
     if let Some(output_dir) = output_dir {
+        // Check whether output directory exists or not
+        // This way we don't replace already existing files.
+        if output_dir.exists() {
+            console::warn(&format!("The directory '{}' already exists. Building to this directory will delete files contained within this directory.", output_dir.display()));
+
+            // Prompt the user to ask whether they want to continue.
+            let clear_dir = tokio::runtime::Runtime::new()
+                .expect("Tokio runtime failed to instantiate")
+                .block_on(ask_bool_timeout(
+                    "Are you sure you want to continue?",
+                    false,
+                    std::time::Duration::from_millis(BUILD_PROMPT_TIMEOUT_MILLIS),
+                ))?;
+
+            if !clear_dir {
+                return Err(Error::msg(
+                    "Cancelled build process because output directory already exists.",
+                ));
+            }
+        }
+
         site.set_output_path(output_dir);
     }
     if let Some(b) = base_url {

src/cmd/serve.rs

@@ -40,6 +40,7 @@ use ws::{Message, Sender, WebSocket};
 
 use errors::{Error as ZolaError, Result};
 use globset::GlobSet;
+use pathdiff::diff_paths;
 use relative_path::{RelativePath, RelativePathBuf};
 use site::sass::compile_sass;
 use site::{Site, SITE_CONTENT};
@@ -72,6 +73,7 @@ static NOT_FOUND_TEXT: &[u8] = b"Not Found";
 const LIVE_RELOAD: &str = include_str!("livereload.js");
 
 async fn handle_request(req: Request<Body>, mut root: PathBuf) -> Result<Response<Body>> {
+    let original_root = root.clone();
     let mut path = RelativePathBuf::new();
     // https://zola.discourse.group/t/percent-encoding-for-slugs/736
     let decoded = match percent_encoding::percent_decode_str(req.uri().path()).decode_utf8() {
@@ -111,6 +113,11 @@ async fn handle_request(req: Request<Body>, mut root: PathBuf) -> Result<Respons
     // otherwise `PathBuf` will interpret it as an absolute path
     root.push(&decoded[1..]);
 
+    // Ensure we are only looking for things in our public folder
+    if !root.starts_with(original_root) {
+        return Ok(not_found());
+    }
+
     let metadata = match tokio::fs::metadata(root.as_path()).await {
         Err(err) => return Ok(io_error(err)),
         Ok(metadata) => metadata,
@@ -300,12 +307,14 @@ pub fn serve(
         return Err(format!("Cannot start server on address {}.", address).into());
     }
 
-    let config_path = config_file.to_str().unwrap_or("config.toml");
+    let config_path = PathBuf::from(config_file);
+    let config_path_rel = diff_paths(&config_path, &root_dir).unwrap_or(config_path.clone());
 
-    // An array of (path, bool, bool) where the path should be watched for changes, and the boolean value
+    // An array of (path, WatchMode) where the path should be watched for changes,
-    // indicates whether this file/folder must exist for zola serve to operate
+    // and the WatchMode value indicates whether this file/folder must exist for
+    // zola serve to operate
     let watch_this = vec![
-        (config_path, WatchMode::Required),
+        (config_path_rel.to_str().unwrap_or("config.toml"), WatchMode::Required),
         ("content", WatchMode::Required),
         ("sass", WatchMode::Condition(site.config.compile_sass)),
         ("static", WatchMode::Optional),
@@ -518,7 +527,7 @@ pub fn serve(
                         );
 
                         let start = Instant::now();
-                        match detect_change_kind(root_dir, &path, config_path) {
+                        match detect_change_kind(root_dir, &path, &config_path) {
                             (ChangeKind::Content, _) => {
                                 console::info(&format!("-> Content changed {}", path.display()));
 
@@ -644,13 +653,10 @@ fn is_temp_file(path: &Path) -> bool {
 
 /// Detect what changed from the given path so we have an idea what needs
 /// to be reloaded
-fn detect_change_kind(pwd: &Path, path: &Path, config_filename: &str) -> (ChangeKind, PathBuf) {
+fn detect_change_kind(pwd: &Path, path: &Path, config_path: &Path) -> (ChangeKind, PathBuf) {
     let mut partial_path = PathBuf::from("/");
     partial_path.push(path.strip_prefix(pwd).unwrap_or(path));
 
-    let mut partial_config_path = PathBuf::from("/");
-    partial_config_path.push(config_filename);
-
     let change_kind = if partial_path.starts_with("/templates") {
         ChangeKind::Templates
     } else if partial_path.starts_with("/themes") {
@@ -661,7 +667,7 @@ fn detect_change_kind(pwd: &Path, path: &Path, config_filename: &str) -> (Change
         ChangeKind::StaticFiles
     } else if partial_path.starts_with("/sass") {
         ChangeKind::Sass
-    } else if partial_path == partial_config_path {
+    } else if path == config_path {
         ChangeKind::Config
     } else {
         unreachable!("Got a change in an unexpected path: {}", partial_path.display());
@@ -709,43 +715,43 @@ mod tests {
                 (ChangeKind::Templates, PathBuf::from("/templates/hello.html")),
                 Path::new("/home/vincent/site"),
                 Path::new("/home/vincent/site/templates/hello.html"),
-                "config.toml",
+                Path::new("/home/vincent/site/config.toml"),
             ),
             (
                 (ChangeKind::Themes, PathBuf::from("/themes/hello.html")),
                 Path::new("/home/vincent/site"),
                 Path::new("/home/vincent/site/themes/hello.html"),
-                "config.toml",
+                Path::new("/home/vincent/site/config.toml"),
             ),
             (
                 (ChangeKind::StaticFiles, PathBuf::from("/static/site.css")),
                 Path::new("/home/vincent/site"),
                 Path::new("/home/vincent/site/static/site.css"),
-                "config.toml",
+                Path::new("/home/vincent/site/config.toml"),
             ),
             (
                 (ChangeKind::Content, PathBuf::from("/content/posts/hello.md")),
                 Path::new("/home/vincent/site"),
                 Path::new("/home/vincent/site/content/posts/hello.md"),
-                "config.toml",
+                Path::new("/home/vincent/site/config.toml"),
             ),
             (
                 (ChangeKind::Sass, PathBuf::from("/sass/print.scss")),
                 Path::new("/home/vincent/site"),
                 Path::new("/home/vincent/site/sass/print.scss"),
-                "config.toml",
+                Path::new("/home/vincent/site/config.toml"),
             ),
             (
                 (ChangeKind::Config, PathBuf::from("/config.toml")),
                 Path::new("/home/vincent/site"),
                 Path::new("/home/vincent/site/config.toml"),
-                "config.toml",
+                Path::new("/home/vincent/site/config.toml"),
             ),
             (
                 (ChangeKind::Config, PathBuf::from("/config.staging.toml")),
                 Path::new("/home/vincent/site"),
                 Path::new("/home/vincent/site/config.staging.toml"),
-                "config.staging.toml",
+                Path::new("/home/vincent/site/config.staging.toml"),
             ),
         ];
 
@@ -760,7 +766,7 @@ mod tests {
         let expected = (ChangeKind::Templates, PathBuf::from("/templates/hello.html"));
         let pwd = Path::new(r#"C:\\Users\johan\site"#);
         let path = Path::new(r#"C:\\Users\johan\site\templates\hello.html"#);
-        let config_filename = "config.toml";
+        let config_filename = Path::new(r#"C:\\Users\johan\site\config.toml"#);
         assert_eq!(expected, detect_change_kind(pwd, path, config_filename));
     }
 
@@ -769,7 +775,7 @@ mod tests {
         let expected = (ChangeKind::Templates, PathBuf::from("/templates/hello.html"));
         let pwd = Path::new("/home/johan/site");
         let path = Path::new("templates/hello.html");
-        let config_filename = "config.toml";
+        let config_filename = Path::new("config.toml");
         assert_eq!(expected, detect_change_kind(pwd, path, config_filename));
     }
 }

src/console.rs

@@ -48,15 +48,14 @@ fn colorize(message: &str, color: &ColorSpec) {
     writeln!(&mut stdout).unwrap();
 }
 
-/// Display in the console the number of pages/sections in the site, and number of images to process
+/// Display in the console the number of pages/sections in the site
 pub fn notify_site_size(site: &Site) {
     let library = site.library.read().unwrap();
     println!(
-        "-> Creating {} pages ({} orphan), {} sections, and processing {} images",
+        "-> Creating {} pages ({} orphan) and {} sections",
         library.pages().len(),
         library.get_all_orphan_pages().len(),
         library.sections().len() - 1, // -1 since we do not count the index as a section there
-        site.num_img_ops(),
     );
 }
 

src/main.rs

@@ -19,7 +19,9 @@ fn main() {
             .unwrap_or_else(|_| panic!("Cannot find root directory: {}", path)),
     };
     let config_file = match matches.value_of("config") {
-        Some(path) => PathBuf::from(path),
+        Some(path) => PathBuf::from(path)
+            .canonicalize()
+            .unwrap_or_else(|_| panic!("Cannot find config file: {}", path)),
         None => root_dir.join("config.toml"),
     };
 

src/prompt.rs

@@ -1,7 +1,9 @@
 use std::io::{self, BufRead, Write};
+use std::time::Duration;
 
 use url::Url;
 
+use crate::console;
 use errors::Result;
 
 /// Wait for user input and return what they typed
@@ -32,6 +34,24 @@ pub fn ask_bool(question: &str, default: bool) -> Result<bool> {
     }
 }
 
+/// Ask a yes/no question to the user with a timeout
+pub async fn ask_bool_timeout(question: &str, default: bool, timeout: Duration) -> Result<bool> {
+    let (tx, rx) = tokio::sync::oneshot::channel();
+
+    let q = question.to_string();
+    std::thread::spawn(move || {
+        tx.send(ask_bool(&q, default)).unwrap();
+    });
+
+    match tokio::time::timeout(timeout, rx).await {
+        Err(_) => {
+            console::warn("\nWaited too long for response.");
+            Ok(default)
+        }
+        Ok(val) => val.expect("Tokio failed to properly execute"),
+    }
+}
+
 /// Ask a question to the user where they can write a URL
 pub fn ask_url(question: &str, default: &str) -> Result<String> {
     print!("{} ({}): ", question, default);

test_site/config.toml

@@ -13,7 +13,8 @@ ignored_content = ["*/ignored.md"]
 
 [markdown]
 highlight_code = true
-extra_syntaxes = ["syntaxes"]
+highlight_theme = "custom_gruvbox"
+extra_syntaxes_and_themes = ["syntaxes", "highlight_themes"]
 
 [slugify]
 paths = "on"

test_site/content/posts/extra_syntax.md

@@ -10,6 +10,12 @@ for (int i = 0; ; i++ ) {
 }
 ```
 
+```
+for (int i = 0; ; i++ ) {
+    if (i < 10) 
+}
+```
+
 ```c
 for (int i = 0; ; i++ ) {
     if (i < 10) 

test_site/highlight_themes/custom_gruvbox.tmTheme

@@ -0,0 +1,394 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>name</key>
+  <string>Gruvbox-N</string>
+  <key>settings</key>
+  <array>
+    <dict>
+      <key>settings</key>
+      <dict>
+        <key>background</key>
+        <string>#1a1a1a</string>
+        <key>caret</key>
+        <string>#908476</string>
+        <key>foreground</key>
+        <string>#EAD4AF</string>
+        <key>invisibles</key>
+        <string>#3B3836</string>
+        <key>lineHighlight</key>
+        <string>#3B3836</string>
+        <key>selection</key>
+        <string>#3B3836</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Comment</string>
+      <key>scope</key>
+      <string>comment</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#908476</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>String</string>
+      <key>scope</key>
+      <string>string</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#AAB11E</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Separator</string>
+      <key>scope</key>
+      <string>punctuation.separator.key-value</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#CF8498</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Constant</string>
+      <key>scope</key>
+      <string>constant</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#CC869B</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Variable</string>
+      <key>scope</key>
+      <string>variable</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#EAD4AF</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Other variable objct</string>
+      <key>scope</key>
+      <string>variable.other.object</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#CAB990</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Other variable class</string>
+      <key>scope</key>
+      <string>variable.other.class, variable.other.constant</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#F1C050</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Object property</string>
+      <key>scope</key>
+      <string>meta.property.object, entity.name.tag</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#EAD4AF</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Arrows</string>
+      <key>scope</key>
+      <string>meta.function, meta.function.static.arrow, meta.function.arrow</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#EAD4AF</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Keyword</string>
+      <key>scope</key>
+      <string>keyword, string.regexp punctuation.definition</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#FB4938</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Storage</string>
+      <key>scope</key>
+      <string>storage, storage.type</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#FB4938</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Inline link</string>
+      <key>scope</key>
+      <string>markup.underline.link</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#FB4938</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Class name</string>
+      <key>scope</key>
+      <string>entity.name.class, entity.name.type.class</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#BABC52</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Inherited class</string>
+      <key>scope</key>
+      <string>entity.other.inherited-class, tag.decorator, tag.decorator entity.name.tag</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#7BA093</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Function name</string>
+      <key>scope</key>
+      <string>entity.name.function, meta.function entity.name.function</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#8AB572</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Function argument</string>
+      <key>scope</key>
+      <string>variable.parameter, meta.function storage.type</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#FD971F</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Tag name</string>
+      <key>scope</key>
+      <string>entity.name.tag</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#FB4938</string>
+        <key>fontStyle</key>
+        <string> italic  </string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Tag attribute</string>
+      <key>scope</key>
+      <string>entity.other.attribute-name</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#8AB572</string>
+        <key>fontStyle</key>
+        <string> italic</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Library class/type</string>
+      <key>scope</key>
+      <string>support.type, support.class, support.function, variable.language, support.constant, string.regexp keyword.control</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#F1C050</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Template string element</string>
+      <key>scope</key>
+      <string>punctuation.template-string.element, string.regexp punctuation.definition.group, constant.character.escape</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#8AB572</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Invalid</string>
+      <key>scope</key>
+      <string>invalid</string>
+      <key>settings</key>
+      <dict>
+        <key>background</key>
+        <string>#FB4938</string>
+        <key>fontStyle</key>
+        <string />
+        <key>foreground</key>
+        <string>#F8F8F0</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Invalid deprecated</string>
+      <key>scope</key>
+      <string>invalid.deprecated</string>
+      <key>settings</key>
+      <dict>
+        <key>background</key>
+        <string>#FD971F</string>
+        <key>foreground</key>
+        <string>#F8F8F0</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Operator</string>
+      <key>scope</key>
+      <string>keyword.operator, keyword.operator.logical, meta.property-name, meta.brace, punctuation.definition.parameters.begin, punctuation.definition.parameters.end, keyword.other.parenthesis</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#CAB990</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Special operator</string>
+      <key>scope</key>
+      <string>keyword.operator.ternary</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#7BA093</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Separator</string>
+      <key>scope</key>
+      <string>punctuation.separator.parameter</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#EAD4AF</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Module</string>
+      <key>scope</key>
+      <string>keyword.operator.module</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#FB4938</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>SublimeLinter Error</string>
+      <key>scope</key>
+      <string>sublimelinter.mark.error</string>
+      <key>settings</key>
+      <dict>
+          <key>foreground</key>
+          <string>#D02000</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>SublimeLinter Warning</string>
+      <key>scope</key>
+      <string>sublimelinter.mark.warning</string>
+      <key>settings</key>
+      <dict>
+          <key>foreground</key>
+          <string>#DDB700</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>SublimeLinter Gutter Mark</string>
+      <key>scope</key>
+      <string>sublimelinter.gutter-mark</string>
+      <key>settings</key>
+      <dict>
+          <key>foreground</key>
+          <string>#FFFFFF</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Diff inserted</string>
+      <key>scope</key>
+      <string>markup.inserted</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#70c060</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Diff changed</string>
+      <key>scope</key>
+      <string>markup.changed</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#DDB700</string>
+      </dict>
+    </dict>
+    <dict>
+      <key>name</key>
+      <string>Diff deleted</string>
+      <key>scope</key>
+      <string>markup.deleted</string>
+      <key>settings</key>
+      <dict>
+        <key>foreground</key>
+        <string>#FB4938</string>
+      </dict>
+    </dict>
+  </array>
+  <key>uuid</key>
+  <string>D8D5E82E-3D5B-46B5-B38E-8C841C21347D</string>
+  <key>colorSpaceName</key>
+  <string>sRGB</string>
+</dict>
+</plist>