GO-YQ

A lightweight and portable command-line YAML, JSON and XML processor. yq uses jq like syntax but works with yaml files as well as json, xml, properties, csv and tsv.

URL: https://github.com/mikefarah/yq

Example

This wrapper can be used in the following way:

rule test_yq_concat:
    input:
        ["bazquux.yaml", "foobar.yaml"],
    output:
        "concat.yaml",
    threads: 1
    log:
        "test_yq_concat.log",
    params:
        extra="",
        subcommand="",
        expression="",
    wrapper:
        "v9.9.0/utils/go-yq"


rule test_yq_simple_yaml_update:
    input:
        "foobar.yaml",
    output:
        "updated.yaml",
    threads: 1
    log:
        "test_yq_simple_yaml_update.log",
    params:
        extra="",
        subcommand="",
        expression='.[].foo = "foobar"',
    wrapper:
        "v9.9.0/utils/go-yq"


rule test_yq_evaluate_yaml_content:
    input:
        "eval.yaml",
    output:
        "evaluated.yaml",
    threads: 1
    log:
        "test_yq_evaluate_yaml_content.log",
    params:
        extra="",
        subcommand="",
        expression="eval(.pathExp)",
    wrapper:
        "v9.9.0/utils/go-yq"


rule test_yq_split_yaml:
    input:
        "concat.yaml",
    output:
        "foo_bar.yml",
        "foo_bar2.yml",
    threads: 1
    log:
        "test_yq_split_yaml.log",
    params:
        extra="-s '\"foo_\" + .foo'",
        subcommand="",
        expression=".[]",
    wrapper:
        "v9.9.0/utils/go-yq"


rule test_yq_format_conversion:
    input:
        "table.csv",
    output:
        "table.json",
    threads: 1
    log:
        "test_yq_format_conversion.log",
    params:
        extra="",
        subcommand="",
        expression="",
    wrapper:
        "v9.9.0/utils/go-yq"

Note that input, output and log file paths can be chosen freely.

When running with

snakemake --use-conda

the software dependencies will be automatically deployed into an isolated environment before execution.

Notes

This tool does not support multithreading.

Format are mostly auto-detected using –output-format and –input-format. However, you may want to provide formats through params.extra in case of ambiguity (e.g. uri, base64).

Through file splitting, files name expression are not automatically inferred. User should provide the proper -s/–split-exp parameters in params.extra.

Software dependencies

• go-yq=4.53.3

• snakemake-wrapper-utils=0.8.0

Input/Output

Input:

• Optional path to one or multiple file(s) to process

Output:

• Path to one or multiple output file(s)

Params

• extra: Optional arguments for yq command, besides –null-input.

• expression: Optional expression to evaluate.

• subcommand: Optional subcommand for yq

Authors

• Thibault Dayris

Code

# coding: utf-8

"""Snakemake wrapper for go-yq"""

from snakemake_wrapper_utils.snakemake import is_arg, get_format
from snakemake.shell import shell


def detect_fmt(files, long_option, short_option, extra) -> str:
    """Try to detect input/output formats"""
    formats = [get_format(f) for f in files]
    accepted_formats = {
        "yml": "yaml",
        "yaml": "yaml",
        "json": "json",
        "properties": "props",
        "csv": "csv",
        "tsv": "tsv",
        "toml": "toml",
        "sh": "shell",
        "lua": "lua",
        "ini": "ini",
        "xml": "xml",
    }

    # Ensure user did not provide any format information
    if not (is_arg(long_option, extra) or is_arg(short_option, extra)):
        # Check unique format in input and output separately
        if len(set(formats)) > 1:
            raise ValueError(
                f"The following files: {files}, should have the same extension. "
                f"Got {set(formats)} instead."
            )

        # Trying to detect output format from file extension
        # since yq defaults to yaml output and writes information
        # in standard output.
        fmt = accepted_formats.get(formats[0])
        if fmt:
            return f" {long_option} {fmt}"

    return ""


# Enhance extra parameters if needed
extra = snakemake.params.get("extra", "")
extra += detect_fmt(snakemake.input, "--input-format", "-p", extra)
extra += detect_fmt(snakemake.output, "--output-format", "-o", extra)

subcommand = snakemake.params.get("subcommand", "")
expression = snakemake.params.get("expression", "")

# Handle the case user creates a file from command line
infile = snakemake.input
if len(infile) == 0:
    infile = " --null-input "

# Handling file splitting
outfile = snakemake.output
if len(snakemake.output) > 1:
    if not (is_arg("--split-exp", extra) or is_arg("-s", extra)):
        raise ValueError(
            "File splitting is not automatically handled. Please provide "
            "adequate '-s/--split-exp' in 'params.extra'."
        )
    log = snakemake.log_fmt_shell(stdout=True, stderr=True)
    outfile = ""
elif len(snakemake.output) == 1:
    log = snakemake.log_fmt_shell(stdout=False, stderr=True)
    outfile = f"> {snakemake.output[0]}"
else:
    raise ValueError("Inplace modifications are not handled by this wrapper.")

shell("yq {subcommand} {extra} {expression:q} {infile} {outfile} {log}")