Stages

Listing of stages implemented in YMP

stage Import

Imports raw read files into YMP.

>>> ymp make toy
>>> ymp make mpic

rule export_qiime_map_file

rule symlink_raw_reads

Normalize FQ names by creating symlinks to original files

rule symlink_raw_reads_SE

Normalize FQ names by creating symlinks to original files

stage annotate_blast

Annotate sequences with BLAST

Searches a reference database for hits with blastn. Use E flag to specify exponent to required E-value. Use N or Mega to specify default. Use Best to add -subject_besthit flag.

This stage produces blast7.gz files as output.

>>> ymp make toy.ref_genome.index_blast.annotate_blast

rule blast_db_size

Determines size of BLAST database (for splitting)

rule blast_db_size_SPLIT

Variant of blast_db_size for multi-file blast indices

rule blast_db_size_V4

Variant of blast_db_size for V4 blast indices

rule blastn_join_result

Merges BLAST results

rule blastn_query

Runs BLAST

rule blastn_query_SPLIT

Variant of blastn_query for multi-file blast indices

rule blastn_query_V4

Variant of blastn_query for V4 blast indices

rule blastn_split_query_fasta

Split FASTA query file into chunks for individual BLAST runs

rule blastn_split_query_fasta_hack

Workaround for a problem with snakemake checkpoints and run: statements

stage annotate_diamond

FIXME

rule diamond_blastx_fasta

rule diamond_view

Convert Diamond binary output (daa) to BLAST6 format

stage annotate_prodigal

Call genes using prodigal

>>> ymp make toy.ref_genome.annotate_prodigal

rule prodigal

Predict genes using prodigal

stage annotate_tblastn

Runs tblastn

rule blast7_to_gtf

Convert from Blast Format 7 to GFF/GTF format

rule tblastn_query

Runs a TBLASTN search against an assembly.

stage assemble_megahit

Assemble metagenome using MegaHit.

>>> ymp make toy.assemble_megahit.map_bbmap
>>> ymp make toy.group_ALL.assemble_megahit.map_bbmap
>>> ymp make toy.group_Subject.assemble_megahit.map_bbmap

rule megahit

Runs MegaHit.

stage assemble_spades

Assemble reads using spades

>>> ymp make toy.assemble_spades
>>> ymp make toy.group_ALL.assemble_spades
>>> ymp make toy.group_Subject.assemble_spades
>>> ymp make toy.assemble_spades
>>> ymp make toy.assemble_spadesMeta
>>> ymp make toy.assemble_spadesSc
>>> ymp make toy.assemble_spadesRna
>>> ymp make toy.assemble_spadesIsolate
>>> ymp make toy.assemble_spadesNC
>>> ymp make toy.assemble_spadesMetaNC

rule spades

Runs Spades. Supports reads.by_COLUMN.sp/complete as target for by group co-assembly.

rule spades_input_yaml

Prepares a dataset config for spades. Spades commandline is limited to at most 9 pairs of fq files, so to allow arbitrary numbers we need to use the dataset config option.

Preparing in a separate rule so that the main spades rule can use the shell: rule and not run:, which would preclude it from using conda environments.

stage assemble_trinity

rule trinity

rule trinity_stats

stage assemble_unicycler

Assemble reads using unicycler

>>> ymp make toy.assemble_unicycler

rule unicycler

Runs unicycler

stage basecov_bedtools

Creates BLAST index running makeblastdb on input fasta.gz files.

>>> ymp make toy.ref_genome.index_blast

rule bedtools_genomecov

stage bin_metabat2

Bin metagenome assembly into MAGs

>>> ymp make mock.assemble_megahit.map_bbmap.sort_bam.bin_metabat2
>>> ymp make mock.group_ALL.assemble_megahit.map_bbmap.sort_bam.group_ALL.bin_metabat2

rule metabat2_bin

Bin metagenome with MetaBat2

rule metabat2_depth

Generates a depth file from BAM

stage check

Verify file availability

This stage provides rules for checking the file availability at a given point in the stage stack.

Mainly useful for testing and debugging.

rule check_fasta

Verify availability of FastA type reference

rule check_fastp

Verify availability of FastP type reference

stage cluster_cdhit

Clusters protein sequences using CD-HIT

>>> ymp make toy.ref_query.cluster_cdhit

rule cdhit_clstr_to_csv

rule cdhit_faa_single

Clustering predicted genes using cdhit

rule cdhit_prepare_input

Prepares input data for CD-HIT

• rewrites ‘*’ to ‘X’ as stop-codon not understood by CD-HIT

• prefixes lost ID to Fasta ID

stage correct_bbmap

Correct read errors by overlapping inside tails

Applies BBMap's “bbmerge.sh ecco” mode. This will overlap the inside of read pairs and choose the base with the higher quality where the alignment contains mismatches and increase the quality score as indicated by the double observation where the alignment contains matches.

>>> ymp make toy.correct_bbmap
>>> ymp make mpic.correct_bbmap

rule bbmap_error_correction

Error correction with BBMerge overlapping

rule bbmap_error_correction_all

rule bbmap_error_correction_se

Error correction with BBMerge overlapping

stage count_diamond

rule diamond_count

stage count_stringtie

rule stringtie

rule stringtie_abundance

rule stringtie_all

rule stringtie_all_target

rule stringtie_gather_ballgown

rule stringtie_merge

stage coverage_samtools

Computes coverage from a sorted bam file using samtools coverage

rule samtools_coverage

stage dedup_bbmap

Remove duplicate reads

Applies BBMap's “dedupe.sh”

>>> ymp make toy.dedup_bbmap
>>> ymp make mpic.dedup_bbmap

rule bbmap_dedupe

Deduplicate reads using BBMap’s dedupe.sh

rule bbmap_dedupe_all

rule bbmap_dedupe_se

Deduplicate reads using BBMap's dedupe.sh

stage dust_bbmap

Perform entropy filtering on reads using BBMap's bbduk.sh

The parameter Enn gives the entropy cutoff. Higher values filter more sequences.

>>> ymp make toy.dust_bbmap
>>> ymp make toy.dust_bbmapE60

rule bbmap_dust

stage extract_reads

Extract reads from BAM file using samtools fastq.

Parameters fn, Fn and Gn are passed through to samtools view. Reads are output only if all bits in f are set, none of the bits in F are set, and any of the bits in G is unset.

1: paired 2: proper pair (both aligned in right orientation) 4: unmapped 8: other read unmapped

Some options include:

• f2: correctly mapped (only proper pairs)

• F12: both ends mapped (but potentially “improper”)

• G12: either end mapped

• F2: not correctly mapped (not proper pair, could also be unmapped)

• f12: not mapped (neither read mapped)

rule samtools_fastq

stage extract_seqs

Extract sequences from .fasta.gz file using samtools faidx

Currently requires a .blast7 file as input.

Use parameter Nomatch to instead keep unmatched sequences.

rule samtools_faidx

rule samtools_select_blast

stage filter_bmtagger

Filter(-out) contaminant reads using BMTagger

>>> ymp make toy.ref_phiX.index_bmtagger.remove_bmtagger
>>> ymp make toy.ref_phiX.index_bmtagger.remove_bmtagger.assemble_megahit
>>> ymp make toy.ref_phiX.index_bmtagger.filter_bmtagger
>>> ymp make mpic.ref_phiX.index_bmtagger.remove_bmtagger

rule bmtagger_filter

Filter reads using reference

rule bmtagger_filter_all

rule bmtagger_filter_out

Filter-out reads using reference

rule bmtagger_filter_revread

Filter reads using reference

rule bmtagger_find

Match paired end reads against reference

rule bmtagger_find_se

Match single end reads against reference

rule bmtagger_remove_all

stage format_bbmap

Process sequences with BBMap's format.sh

Parameter Ln filters sequences at a minimum length.

>>> ymp make toy.assemble_spades.format_bbmapL200

rule bbmap_reformat

stage humann2

Compute functional profiles using HUMAnN2

rule humann2

Runs HUMAnN2 with separately processed Metaphlan2 output.

Note

HUMAnN2 has no special support for paired end reads. As per manual, we just feed it the concatenated forward and reverse reads.

rule humann2_all

rule humann2_join_tables

Joins HUMAnN2 per sample output tables

rule humann2_renorm_table

Renormalizes humann2 output tables

stage index_bbmap

Creates BBMap index

>>> ymp make toy.ref_genome.index_bbmap

rule bbmap_makedb

Precomputes BBMap index

stage index_blast

rule blast_makedb

Build Blast index

stage index_bmtagger

rule bmtagger_bitmask

rule bmtagger_index

stage index_bowtie2

>>> ymp make toy.ref_genome.index_bowtie2

rule bowtie2_index

stage index_diamond

rule diamond_makedb

Build Diamond index file

stage map_bbmap

Map reads using BBMap

>>> ymp make toy.assemble_megahit.map_bbmap
>>> ymp make toy.ref_genome.map_bbmap
>>> ymp make mpic.ref_ssu.map_bbmap

rule bbmap_map

Map read from each (co-)assembly read file to the assembly

rule bbmap_map_SE

Map read from each (co-)assembly read file to the assembly

stage map_bowtie2

Map reads using Bowtie2

>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2
>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2VF
>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2F
>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2S
>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2VS
>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2X800
>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2I5
>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2L
>>> ymp make toy.assemble_megahit.index_bowtie2.map_bowtie2
>>> ymp make toy.group_Subject.assemble_megahit.index_bowtie2.map_bowtie2
>>> ymp make mpic.ref_ssu.index_bowtie2.map_bowtie2

rule bowtie2_map

rule bowtie2_map_SE

stage map_diamond

rule diamond_blastx_fastq

rule diamond_blastx_fastq2

rule diamond_view_2

Convert Diamond binary output (daa) to BLAST6 format

stage map_hisat2

Map reads using Hisat2

rule hisat2_map

For hisat we always assume a pre-build index as providing SNPs and haplotypes etc is beyond this pipelines scope.

stage map_star

Map RNA-Seq reads with STAR

rule star_map

stage markdup_sambamba

rule sambamba_markdup

stage metaphlan2

Assess metagenome community composition using Metaphlan 2

rule metaphlan2

Computes community profile from mapped reads and Metaphlan’s custom reference database.

rule metaphlan2_map

Align reads to Metaphlan’s custom reference database.

rule metaphlan2_merge

Merges Metaphlan community profiles.

stage polish_pilon

Polish genomes with Pilon

Requires fasta.gz and sorted.bam files as input.

rule pilon_polish

stage primermatch_bbmap

Filters reads by matching reference primer using BBMap's “bbduk.sh”.

>>> ymp make mpic.ref_primers.primermatch_bbmap

rule bbduk_primer

Splits reads based on primer matching into “primermatch” and “primerfail”.

rule bbduk_primer_all

rule bbduk_primer_se

Splits reads based on primer matching into “primermatch” and “primerfail”.

stage profile_centrifuge

Classify reads using centrifuge

rule centrifuge

stage qc_fastqc

Quality screen reads using FastQC

>>> ymp make toy.qc_fastqc

rule qc_fastqc

Run FastQC on read files

stage qc_multiqc

Aggregate QC reports using MultiQC

rule multiqc_fastqc

Assemble report on all FQ files in a directory

stage qc_quast

Estimate assemly quality using Quast

rule metaquast_all_at_once

Run quast on all assemblies in the previous stage at once.

rule metaquast_by_sample

Run quast on each assembly

rule metaquast_multiq_summary

Aggregate Quast per assembly reports

stage quant_rsem

Quantify transcripts using RSEM

rule rsem_all

rule rsem_all_for_target

rule rsem_quant

stage references

This is a “virtual” stage. It does not process read data, but comprises rules used for reference provisioning.

rule human_db_download

Download HUMAnN2 reference databases

rule prepare_reference

Provisions files in <reference_dir>/<reference_name>

• Creates symlinks to downloaded references

• Compresses references provided uncompressed upstream

• Connects files requested by stages with downloaded files and unpacked archives

rule unpack_archive

Template rule for unpacking references provisioned upstream as archive.

rule unpack_ref_GRCh38_eaa4c10f

Unpacks ref_GRCh38 archive:

URL: ftp://ftp.ccb.jhu.edu/pub/infphilo/hisat2/data/grch38_snp_tran.tar.gz

Files:

• ALL.1.ht2

• ALL.2.ht2

• ALL.3.ht2

• ALL.4.ht2

• ALL.5.ht2

• ALL.6.ht2

• ALL.7.ht2

• ALL.8.ht2

rule unpack_ref_centrifuge_0d910a96

Unpacks ref_centrifuge archive:

URL: ftp://ftp.ccb.jhu.edu/pub/infphilo/centrifuge/data/p+h+v.tar.gz

Files:

• p+h+v.1.cf

• p+h+v.2.cf

• p+h+v.3.cf

rule unpack_ref_centrifuge_1ee7c028

Unpacks ref_centrifuge archive:

URL: ftp://ftp.ccb.jhu.edu/pub/infphilo/centrifuge/data/nt.tar.gz

Files:

• nt.1.cf

• nt.2.cf

• nt.3.cf

rule unpack_ref_centrifuge_43ba6165

Unpacks ref_centrifuge archive:

URL: ftp://ftp.ccb.jhu.edu/pub/infphilo/centrifuge/data/p_compressed.tar.gz

Files:

• p_compressed.1.cf

• p_compressed.2.cf

• p_compressed.3.cf

rule unpack_ref_centrifuge_a9964521

Unpacks ref_centrifuge archive:

URL: ftp://ftp.ccb.jhu.edu/pub/infphilo/centrifuge/data/p_compressed+h+v.tar.gz

Files:

• p_compressed+h+v.1.cf

• p_compressed+h+v.2.cf

• p_compressed+h+v.3.cf

rule unpack_ref_greengenes_305aa905

Unpacks ref_greengenes archive:

URL: ftp://greengenes.microbio.me/greengenes_release/gg_13_5/gg_13_8_otus.tar.gz

Files:

• rep_set/99_otus.fasta

• rep_set/97_otus.fasta

• rep_set/94_otus.fasta

rule unpack_ref_metaphlan2_a6545140

Unpacks ref_metaphlan2 archive:

URL: https://depot.galaxyproject.org/software/metaphlan2/metaphlan2_2.6.0_src_all.tar.gz

Files:

• db_v20/mpa_v20_m200.1.bt2

• db_v20/mpa_v20_m200.2.bt2

• db_v20/mpa_v20_m200.3.bt2

• db_v20/mpa_v20_m200.4.bt2

• db_v20/mpa_v20_m200.rev.1.bt2

• db_v20/mpa_v20_m200.rev.2.bt2

• db_v20/mpa_v20_m200.pkl

rule unpack_ref_mothur_SEED_39c9f686

Unpacks ref_mothur_SEED archive:

URL: https://www.mothur.org/w/images/a/a4/Silva.seed_v128.tgz

Files:

• silva.seed_v128.tax

• silva.seed_v128.align

stage remove_bbmap

Filter reads by reference

This stage aligns the reads with a given reference using BBMap in fast mode. Matching reads are collected in the stage filter_bbmap and remaining reads are collectec in the stage remove_bbmap.

>>> ymp make toy.ref_phiX.index_bbmap.remove_bbmap
>>> ymp make toy.ref_phiX.index_bbmap.filter_bbmap
>>> ymp make mpic.ref_phiX.index_bbmap.remove_bbmap

rule bbmap_split

rule bbmap_split_all

rule bbmap_split_all_remove

rule bbmap_split_se

stage sort_bam

rule sambamba_sort

stage split_library

Demultiplexes amplicon sequencing files

This rule is treated specially. If a configured project specifies a barcode_col, reads from the file (or files) are used in combination with

rule fastq_multix

rule split_library_compress_sample

stage trim_bbmap

Trim adapters and low quality bases from reads

Applies BBMap's “bbduk.sh”.

Parameters:

A: append to enable adapter trimming Q20: append to select phred score cutoff (default 20) L20: append to select minimum read length (default 20)

>>> ymp make toy.trim_bbmap
>>> ymp make toy.trim_bbmapA
>>> ymp make toy.trim_bbmapAQ10L10
>>> ymp make mpic.trim_bbmap

rule bbmap_trim

Trimming and Adapter Removal using BBTools BBDuk

rule bbmap_trim_all

rule bbmap_trim_se

Trimming and Adapter Removal using BBTools BBDuk

stage trim_sickle

Perform read trimming using Sickle

>>> ymp make toy.trim_sickle
>>> ymp make toy.trim_sickleQ10L10
>>> ymp make mpic.trim_sickleL20

rule sicke_all

rule sickle

rule sickle_se

stage trim_trimmomatic

Adapter trim reads using trimmomatic

>>> ymp make toy.trim_trimmomaticT32
>>> ymp make mpic.trim_trimmomatic

rule trimmomatic_adapter

Trimming with Trimmomatic

rule trimmomatic_adapter_all

rule trimmomatic_adapter_se

Trimming with Trimmomatic

rule download_file_ftp

Downloads remote file using wget

rule download_file_http

Downloads remote file using internal downloader

rule mkdir

Auto-create directories listed in ymp config.

Use these as input: >>> input: tmpdir = ancient(ymp.get_config().dir.tmp) Or as param: >>> param: tmpdir = “/home/docs/checkouts/readthedocs.org/user_builds/ymp/checkouts/latest/doc/tmp”

rule prefetch

Downloads SRA files into NCBI SRA folder (ncbi/public/sra).

rule fastq_dump

Extracts FQ from SRA files

rule cdhit_fna_single

Clustering predicted genes (nuc) using cdhit-est

rule normalize_16S

Normalize 16S by copy number using picrust, must be run with closed reference OTU table

rule predict_metagenome

Predict metagenome using picrust

rule categorize_by_function

Categorize PICRUSt KOs into pathways

rule rsem_index

Build Genome Index for RSEM

rule star_index

Build Genome Index for Star