Markdown Instructions
pymdtools.instruction contains the comment-based directives used to assemble
Markdown documents from reusable fragments, variables, titles, and external
files.
Directive Families
Reference blocks
Reference blocks define reusable content:
<!-- begin-ref(header) -->
# Shared Header
<!-- end-ref -->
They can be inserted into include blocks:
<!-- begin-include(header) -->
<!-- end-include -->
Variable directives
Variables are stored as one-line declarations and inserted into variable blocks:
<!-- var(project/name)="pymdtools" -->
<!-- begin-var(project/name) -->
<!-- end-var -->
File includes
External text files can be resolved and inserted with include-file:
<!-- include-file(snippets/example.md) -->
Referenced paths are intentionally constrained to relative paths without parent traversal.
For file-backed operations, explicitly configure trusted search folders. The
resolver confines results to those roots and MarkdownContent automatically
adds the source document’s parent directory. Directive-looking text inside
fenced or inline code is treated as an example, not as an instruction.
Title helpers
The module can read and update the first level-1 Markdown title, preserving or forcing Setext / ATX style.
Common Usage
Collect refs around a Markdown file and apply include blocks in-place:
from pymdtools.instruction import search_include_refs_to_md_file
search_include_refs_to_md_file("docs/page.md", backup_option=True)
Replace variables in an in-memory string:
from pymdtools.instruction import search_include_vars_to_md_text
updated = search_include_vars_to_md_text(markdown_text)
Public API
Markdown instruction helpers.
This module implements the comment-based directives used by pymdtools to
assemble Markdown documents:
reference blocks, declared with
begin-ref/end-refand inserted withbegin-include/end-include;variable declarations, declared with
var(NAME)="value"and inserted withbegin-var/end-var;include-filedirectives that inline external text files;level-1 Markdown title extraction and rewriting.
The functions operate either on text strings or on Markdown files. File-oriented
helpers delegate path validation, encoding detection, backup creation, and
content writes to pymdtools.common.
- pymdtools.instruction.del_include_file_to_md_text(text: str, filename: str, *, include_file_re: str | Pattern[str] = re.compile('\n <!--\\s*\n include-file\\(\n (?P<name>(?:\\.\\.?/)?[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*)\n \\)\n (?P<content>[\\s\\S]*?)\n -->\n ', re.VERBOSE), first_only: bool = False) str
Remove include-file directives referencing filename from markdown text.
- Parameters:
text – Markdown text.
filename – Target referenced filename to remove.
include_file_re – Regex matching include-file directives; must capture group ‘name’.
first_only – If True, remove only the first matching directive.
- Returns:
Updated markdown text.
- pymdtools.instruction.del_var_to_md_text(text: str, var_name: str) str
Remove all var(…) directives with the given name from a markdown text.
- Parameters:
text – Markdown text.
var_name – Variable name to remove.
- Returns:
Updated markdown text.
- Raises:
TypeError – If inputs are not strings.
ValueError – If var_name is invalid.
- pymdtools.instruction.ensure_include_file_in_md_text(text: str, filename: str, *, include_file_re: str | Pattern[str] = re.compile('\n <!--\\s*\n include-file\\(\n (?P<name>(?:\\.\\.?/)?[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*)\n \\)\n (?P<content>[\\s\\S]*?)\n -->\n ', re.VERBOSE)) str
Ensure that an include-file(filename) directive exists in the markdown text.
The directive is appended after the last existing include-file directive. If no include-file directives exist, it is inserted at the beginning of the text.
- Parameters:
text – Markdown text.
filename – Referenced file name as used in include-file(…).
include_file_re – Regex matching include-file directives; must capture group ‘name’.
- Returns:
Updated markdown text.
- pymdtools.instruction.escape_var_value(value: str) str
Escape a value so it can be safely embedded inside <!– var(…)=”…” –>.
- pymdtools.instruction.get_file_content_to_include(filename: str | PathLike[str] | Path, *, search_folders: Iterable[str | PathLike[str] | Path] | None = None, include_cwd: bool = False, relative_paths: Sequence[str] = ('.', 'referenced_files'), nb_up_path: int = 0, encoding: str | None = None) str
Retrieve the content of a referenced file to include.
- The function searches filename using common.find_file starting from:
the directory containing this module,
optionally the current working directory,
and any additional folders provided via search_folders.
Search is performed within relative_paths under each start point, and can traverse up to nb_up_path parent levels.
- Parameters:
filename – Referenced filename (typically relative, e.g. “snippet.md”).
search_folders – Additional start points for the search.
include_cwd – Whether to include the current working directory as a start point.
relative_paths – Relative subpaths to probe under each start point.
nb_up_path – Number of parent levels to traverse during the search.
encoding – Encoding for reading.
Nonetriggers auto-detection.
- Returns:
File content as text.
- Raises:
Exception – Propagated if the file cannot be found or read.
- pymdtools.instruction.get_include_file_list(text: str, *, include_file_re: str | Pattern[str] = re.compile('\n <!--\\s*\n include-file\\(\n (?P<name>(?:\\.\\.?/)?[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*)\n \\)\n (?P<content>[\\s\\S]*?)\n -->\n ', re.VERBOSE), unique: bool = False) list[str]
Return the list of filenames referenced by include-file(…) directives.
- Parameters:
text – Markdown text.
include_file_re – Regex matching include-file directives; must capture group ‘name’.
unique – If True, remove duplicates while preserving first-seen order.
- Returns:
A list of referenced filenames, in appearance order.
- pymdtools.instruction.get_refs_around_md_file(filename: str | PathLike[str] | Path, filename_ext: str = '.md', previous_refs: Dict[str, str] | None = None, depth_up: int = 1, depth_down: int = -1) Dict[str, str]
Discover refs around a markdown file by scanning parent folders.
The search root is computed by moving depth_up times to the parent directory of filename (or stopping at filesystem root). Then refs are collected by scanning that root directory with get_refs_from_md_directory.
- Depth semantics:
depth_down == -1: unlimited recursion from the chosen root
depth_down == 0: current directory only
depth_down > 0 : limited recursion levels
Note
When depth_up > 0 and depth_down > 0, the effective downward depth is increased by the number of levels actually moved up, so that the scan still covers the original file directory.
- Parameters:
filename – Path to a markdown file (used only to locate directories).
filename_ext – Extension to scan in directories (e.g. “.md”).
previous_refs – Optional dict to extend.
depth_up – Number of parent levels to move up (>= 0).
depth_down – Recursion depth from the computed root (-1 unlimited, >= 0 limited).
- Returns:
A dict mapping ref names to extracted content.
- Raises:
ValueError – If depth_up < 0 or depth_down < -1.
- pymdtools.instruction.get_refs_from_md_directory(folder: str | PathLike[str] | Path, filename_ext: str = '.md', previous_refs: Dict[str, str] | None = None, depth: int = -1) Dict[str, str]
Extract refs from markdown files in a directory tree.
- Depth semantics:
depth == -1: recurse into all subdirectories (unlimited)
depth == 0: only current directory
depth > 0 : recurse up to depth levels
- Parameters:
folder – Root directory to scan.
filename_ext – File extension to include (e.g. “.md”).
previous_refs – Optional dict to merge with extracted refs (copied).
depth – Recursion depth.
- Returns:
A dict mapping ref names to extracted content.
- pymdtools.instruction.get_refs_from_md_file(filename: str | PathLike[str] | Path, filename_ext: str = '.md', previous_refs: Dict[str, str] | None = None) Dict[str, str]
Extract reference blocks from a markdown file.
The file is validated as an existing file and (optionally) checked for the expected extension, then read as text (encoding auto-detected if supported), and analyzed by get_refs_from_md_text.
- Parameters:
filename – Path to the markdown file.
filename_ext – Expected file extension (including dot), e.g. “.md”.
previous_refs – Optional dict to merge with extracted refs.
- Returns:
A dict mapping ref names to extracted content.
- Raises:
RuntimeError / Exception – propagated from common.check_file.
IOError / UnicodeDecodeError – propagated from file reading helpers.
ValueError – propagated from get_refs_from_md_text for malformed refs.
- pymdtools.instruction.get_refs_from_md_text(text: str, previous_refs: Dict[str, str] | None = None) Dict[str, str]
Extract reference blocks from a markdown text.
- Reference blocks are delimited by:
<!– begin-ref(name) –>
<!– end-ref –>
The function extracts blocks sequentially (not nested). Each name must be unique.
- Parameters:
text – Input markdown text.
previous_refs – Optional dict to update (copied to avoid side effects).
- Returns:
A dict mapping ref names to their raw extracted content.
- Raises:
ValueError – If a ref name is duplicated or an end marker is missing.
- pymdtools.instruction.get_refs_from_search_folders(search_folders: Iterable[str | PathLike[str] | Path], *, refs: Dict[str, str] | None = None, filename_ext: str = '.md', depth: int = -1) Dict[str, str]
Extend/collect refs by scanning one or more folders recursively.
- Parameters:
search_folders – Folders to scan.
refs – Existing refs mapping to extend (copied to avoid side effects).
filename_ext – File extension to scan (e.g. “.md”).
depth – Recursion depth (-1 unlimited, 0 current dir only, >0 limited).
- Returns:
A dict mapping ref names to extracted content.
- pymdtools.instruction.get_title_from_md_text(text: str, return_match: bool = False) None | str | Match[str]
Extract the first level-1 Markdown title from text.
Supported syntaxes are Setext H1 (
Titlefollowed by=====) and ATX H1 (# Title).Comments and Markdown code regions are ignored while searching.
- Parameters:
text – Markdown text.
return_match – If True, return the re.Match object on the source text.
- Returns:
The title string (stripped), or None if not found. If return_match is True, returns the match object instead.
- Raises:
TypeError – If text is not a string.
- pymdtools.instruction.get_vars_around_md_file(filename: str | PathLike[str] | Path, *, filename_ext: str = '.md', previous_vars: Dict[str, str] | None = None, depth_up: int = 1, depth_down: int = -1, encoding: str | None = None) Dict[str, str]
Discover var(…) declarations around a markdown file by scanning nearby directories.
Starting from the directory of filename, this function moves up depth_up levels (stopping at filesystem root), then scans downward with depth depth_down.
- Parameters:
filename – Markdown file path.
filename_ext – Extension for markdown files.
previous_vars – Existing mapping to extend.
depth_up – Number of parent levels to move up (>= 0).
depth_down – Downward recursion depth (-1 unlimited, 0 current dir only, >0 limited).
encoding – Encoding used to read markdown files.
Nonetriggers auto-detection.
- Returns:
A dict of var name -> interpreted value.
- Raises:
ValueError – If depth_up < 0 or depth_down < -1.
RuntimeError/Exception – Propagated by filesystem helpers.
- pymdtools.instruction.get_vars_from_md_directory(folder: str | PathLike[str] | Path, *, filename_ext: str = '.md', previous_vars: Dict[str, str] | None = None, depth: int = -1, encoding: str | None = None) Dict[str, str]
Find var(…) declarations in markdown files in folder and (optionally) its subfolders.
- Depth:
-1: recurse into all subfolders
0: current folder only
n>0: recurse n levels
- Parameters:
folder – Directory to scan.
filename_ext – Markdown file extension to consider.
previous_vars – Existing mapping to extend.
depth – Recursion depth.
encoding – Encoding for reading files.
Nonetriggers auto-detection.
- Returns:
A dict of var name -> interpreted value.
- Raises:
RuntimeError – If folder is not a directory.
ValueError – If duplicate var names are found across scanned files.
- pymdtools.instruction.get_vars_from_md_file(filename: str | PathLike[str] | Path, *, filename_ext: str = '.md', previous_vars: Dict[str, str] | None = None, encoding: str | None = None) Dict[str, str]
Extract var(…) directives from a markdown file.
The returned values are interpreted (escapes are processed) as defined by get_vars_from_md_text.
- Parameters:
filename – Path to the markdown file.
filename_ext – Expected file extension.
previous_vars – Optional existing mapping to extend.
encoding – Encoding to use for reading.
Nonetriggers auto-detection.
- Returns:
A dict mapping variable names to interpreted values.
- Raises:
RuntimeError/Exception – Propagated from path checks and file reading helpers.
ValueError – If duplicate var names are found.
- pymdtools.instruction.get_vars_from_md_text(text: str, previous_vars: Dict[str, str] | None = None) Dict[str, str]
Extract variable declarations from markdown text and return interpreted values.
- Variables are declared using:
<!– var(NAME) = “value” –> <!– var(NAME) = ‘value’ –>
Escape sequences inside the quoted value are interpreted (n, t, r, \, ", ').
- Parameters:
text – Markdown text to scan.
previous_vars – Optional dict to extend (copied to avoid side effects).
- Returns:
A dict mapping variable names to interpreted string values.
- Raises:
TypeError – If text is not a string.
ValueError – If a variable name is declared twice.
- pymdtools.instruction.include_files_to_md_file(filename: str | PathLike[str] | Path, *, backup_option: bool = True, backup_ext: str = '.bak', filename_ext: str = '.md', read_encoding: str | None = None, write_encoding: str = 'utf-8', error_if_no_file: bool = True, render_mode: Literal['box', 'raw'] = 'box', **kwargs: Any) str
Apply include-file substitutions to a markdown file in-place.
- Parameters:
filename – Markdown file to process.
backup_option – Create a backup before writing.
backup_ext – Backup extension.
filename_ext – Expected markdown extension.
read_encoding – Encoding to read.
Nonetriggers auto-detection.write_encoding – Encoding used to write.
error_if_no_file – If False, keep unresolved directives unchanged.
render_mode – Forwarded to include_files_to_md_text (e.g. “box” or “raw”).
**kwargs – Forwarded to get_file_content_to_include (e.g. search_folders).
- Returns:
Normalized filename.
- pymdtools.instruction.include_files_to_md_text(text: str, *, include_file_re: str | Pattern[str] = re.compile('\n <!--\\s*\n include-file\\(\n (?P<name>(?:\\.\\.?/)?[A-Za-z0-9._-]+(?:/[A-Za-z0-9._-]+)*)\n \\)\n (?P<content>[\\s\\S]*?)\n -->\n ', re.VERBOSE), error_if_no_file: bool = True, render_mode: Literal['box', 'raw'] = 'box', **kwargs: Any) str
Replace include-file directives with the content of referenced files.
The directive must match include_file_re and provide a group ‘name’ containing the referenced filename.
- Parameters:
text – Markdown text.
include_file_re – Regex to match include-file directives (must capture ‘name’).
error_if_no_file – If False, keep the directive unchanged when the file is not found/readable.
render_mode – “box” to wrap content in an ASCII box, “raw” to insert content as-is.
**kwargs – Forwarded to get_file_content_to_include (e.g. search_folders).
- Returns:
Updated markdown text.
- pymdtools.instruction.include_refs_to_md_file(filename: str | PathLike[str] | Path, refs: Mapping[str, str], *, backup_option: bool = True, backup_ext: str = '.bak', filename_ext: str = '.md', begin_include_re: str | Pattern[str] = re.compile('\n <!--\\s*\n begin-include\n \\(\\s*(?P<name>[A-Za-z0-9_-]+)\\s*\\)\n \\s*-->\n ', re.VERBOSE), end_include_re: str | Pattern[str] = re.compile('\n <!--\\s*\n end-include\n \\s*-->\n ', re.VERBOSE), error_if_no_key: bool = True, read_encoding: str | None = None, write_encoding: str = 'utf-8') str
Apply include references to a markdown file in-place.
The file is read, include markers are processed via include_refs_to_md_text, then the file is overwritten with the resulting content.
- Parameters:
filename – Markdown file path.
refs – Mapping of include names to replacement content.
backup_option – If True, create a backup before overwriting.
backup_ext – Backup extension (e.g. “.bak”).
filename_ext – Expected extension for filename.
begin_include_re – Regex for begin marker (string or compiled, must capture ‘name’).
end_include_re – Regex for end marker (string or compiled).
error_if_no_key – Raise if an include name is unknown.
read_encoding – Encoding used for reading.
Nonetriggers auto-detection.write_encoding – Encoding used for writing.
- Returns:
Normalized filename (string).
- Raises:
ValueError/KeyError – Propagated from include processing.
RuntimeError/Exception – Propagated from common.check_file / IO helpers.
- pymdtools.instruction.include_refs_to_md_text(text: str, refs_include: Mapping[str, str], begin_include_re: str | Pattern[str] = re.compile('\n <!--\\s*\n begin-include\n \\(\\s*(?P<name>[A-Za-z0-9_-]+)\\s*\\)\n \\s*-->\n ', re.VERBOSE), end_include_re: str | Pattern[str] = re.compile('\n <!--\\s*\n end-include\n \\s*-->\n ', re.VERBOSE), error_if_no_key: bool = True) str
Insert include references into a markdown text.
- Markers:
<!– begin-include(NAME) –> … <!– end-include –>
If NAME exists in refs_include, its content is inserted immediately after the begin marker, and the original inner content (between markers) is skipped. Begin/end markers are preserved in the output.
- If NAME is missing:
if error_if_no_key is True: raise KeyError
else: keep the include block unchanged and continue
Notes
Processing is sequential (not nesting-aware): the first end marker after a begin marker is used.
- Parameters:
text – Markdown text to process.
refs_include – Mapping of include names to replacement content.
begin_include_re – Regex (compiled or string) matching the begin marker and capturing group ‘name’.
end_include_re – Regex (compiled or string) matching the end marker.
error_if_no_key – Whether to raise if the key is missing.
- Returns:
The processed markdown text.
- Raises:
TypeError – If text is not a str.
KeyError – If an include key is missing and error_if_no_key=True.
ValueError – If an end marker is missing.
- pymdtools.instruction.include_vars_to_md_file(filename: str | PathLike[str] | Path, vars_include: Mapping[str, str], *, backup_option: bool = True, backup_ext: str = '.bak', filename_ext: str = '.md', begin_var_re: str | Pattern[str] = re.compile('<!--\\s*begin-var\\(\\s*(?P<name>[A-Za-z0-9:_-]+(?:/[A-Za-z0-9:_-]+)*)\\s*\\)\\s*-->'), end_var_re: str | Pattern[str] = re.compile('<!--\\s*end-var\\s*-->'), error_if_var_not_found: bool = True, read_encoding: str | None = None, write_encoding: str = 'utf-8') str
Apply begin-var/end-var substitutions to a markdown file in-place.
This is a wrapper around include_refs_to_md_file, using the var markers.
- Parameters:
filename – Markdown file to process.
vars_include – Mapping var name -> replacement content.
backup_option – Whether to create a backup before overwriting.
backup_ext – Backup extension.
filename_ext – Expected file extension.
begin_var_re – Regex for begin-var marker (must capture group ‘name’).
end_var_re – Regex for end-var marker.
error_if_var_not_found – Raise if a referenced var is missing.
read_encoding – Encoding used to read the file.
Nonetriggers auto-detection.write_encoding – Encoding used to write the file.
- Returns:
Normalized filename (string).
- pymdtools.instruction.include_vars_to_md_text(text: str, vars_include: Mapping[str, str], *, begin_var_re: str | Pattern[str] = re.compile('<!--\\s*begin-var\\(\\s*(?P<name>[A-Za-z0-9:_-]+(?:/[A-Za-z0-9:_-]+)*)\\s*\\)\\s*-->'), end_var_re: str | Pattern[str] = re.compile('<!--\\s*end-var\\s*-->'), error_if_var_not_found: bool = True) str
Insert variable values into begin-var/end-var blocks in markdown text.
This is a thin wrapper around
include_refs_to_md_text()configured with the variable block markers:<!-- begin-var(NAME) --><!-- end-var -->
- Parameters:
text – Markdown text to process.
vars_include – Mapping of variable names to replacement content.
begin_var_re – Regex matching the opening variable marker. It must expose a named group
name.end_var_re – Regex matching the closing variable marker.
error_if_var_not_found – If True, raise
KeyErrorwhen a block refers to a missing variable. If False, leave that block unchanged.
- Returns:
Markdown text with matching variable blocks updated.
- pymdtools.instruction.refs_in_md_text(text: str) List[str]
Extract include reference names from markdown text.
Matches patterns of the form:
<!– begin-include(NAME) –>
where NAME contains only alphanumeric characters, underscores or hyphens.
- Parameters:
text – Markdown text to analyze.
- Returns:
A list of include reference names (strings). Example: [“header”, “footer”]
- pymdtools.instruction.search_include_refs_to_md_file(filename: str | PathLike[str] | Path, *, backup_option: bool = True, backup_ext: str = '.bak', filename_ext: str = '.md', depth_up: int = 1, depth_down: int = -1) str
Discover refs around a markdown file and apply include substitutions in-place.
- This is a convenience function:
Collect refs by scanning folders around filename (see get_refs_around_md_file)
Apply includes into filename (see include_refs_to_md_file)
- Parameters:
filename – Markdown file to process.
backup_option – Whether to create a backup before overwriting.
backup_ext – Backup extension (e.g. “.bak”).
filename_ext – Expected extension for filename.
depth_up – Number of parent directory levels to move up for the search root (>= 0).
depth_down – Depth for scanning downward (-1 unlimited, 0 current dir only, >0 limited).
- Returns:
Normalized filename (string).
- Raises:
ValueError – If depth_up < 0 or depth_down < -1.
KeyError/ValueError – Propagated from include resolution if refs are missing/malformed.
RuntimeError/Exception – Propagated from filesystem helpers.
- pymdtools.instruction.search_include_vars_to_md_file(filename: str | PathLike[str] | Path, *, backup_option: bool = True, backup_ext: str = '.bak', filename_ext: str = '.md', depth_up: int = 1, depth_down: int = -1, encoding: str | None = None) str
Search vars around filename and apply begin-var/end-var substitutions in-place.
- pymdtools.instruction.search_include_vars_to_md_text(text: str, *, error_if_var_not_found: bool = True, begin_var_re: str | Pattern[str] = re.compile('<!--\\s*begin-var\\(\\s*(?P<name>[A-Za-z0-9:_-]+(?:/[A-Za-z0-9:_-]+)*)\\s*\\)\\s*-->'), end_var_re: str | Pattern[str] = re.compile('<!--\\s*end-var\\s*-->')) str
Extract var(…) declarations from text and apply begin-var/end-var substitutions.
- Parameters:
text – Markdown text.
error_if_var_not_found – Raise if a referenced var is missing.
begin_var_re – Regex for begin-var marker (must capture group ‘name’).
end_var_re – Regex for end-var marker.
- Returns:
Updated markdown text.
- Raises:
KeyError/ValueError – If a referenced var is missing (depending on implementation).
- pymdtools.instruction.set_title_in_md_text(text: str, new_title: str, *, style: Literal['preserve', 'setext', 'atx'] = 'preserve') str
Set or insert the first level-1 Markdown title in text.
Supported syntaxes are Setext H1 (
Titlefollowed by=====) and ATX H1 (# Title).- Behavior:
- If an H1 title exists, it is replaced according to style:
“preserve”: keep the existing style (Setext stays Setext, ATX stays ATX)
“setext”: force Setext output
“atx”: force ATX output
- If no H1 title exists, a new title is inserted at the beginning using:
“preserve” -> Setext (default insertion format)
“setext” -> Setext
“atx” -> ATX
Notes
Titles inside XML comments are ignored (comments are stripped before detection).
Replacement is applied to the first detected H1 only.
- Parameters:
text – Markdown text.
new_title – New title (must be non-empty after stripping).
style – “preserve” | “setext” | “atx”.
- Returns:
Updated markdown text.
- Raises:
TypeError – If inputs are not strings.
ValueError – If new_title is blank or style is invalid.
- pymdtools.instruction.set_var_to_md_text(text: str, var_name: str, value: str) str
Set or add a var(…) directive in markdown text.
If the variable is already present, its declaration is replaced. If absent, the declaration is inserted after the existing var(…) block, and after any include-file directives that immediately follow.
- Parameters:
text – Markdown text.
var_name – Variable name (allowed: [A-Za-z0-9:_-]+).
value – Interpreted value (will be escaped for storage).
- Returns:
Updated markdown text.
- Raises:
TypeError – If inputs are not strings.
ValueError – If var_name is invalid.
- pymdtools.instruction.strip_xml_comment(text: str) str
Remove XML / HTML comments from a text.
XML comments are defined as any content between
<!--and-->, including multi-line comments.- Parameters:
text – Input markdown (or text) content.
- Returns:
The input text with all XML comments removed.
- pymdtools.instruction.unescape_var_value(value: str) str
Interpret backslash escapes inside a var(…) value.
- Supported escapes:
n, t, r, \, ", '
Unknown escapes keep the escaped character as-is (e.g. “x” -> “x”).
- Parameters:
value – Raw captured value (without surrounding quotes).
- Returns:
Interpreted value.