Stages

Listing of stages implemented in YMP

System Message: ERROR/3 (/home/docs/checkouts/readthedocs.org/user_builds/ymp/checkouts/stable/doc/stages.rst, line 1)

Error in “sm:stage” directive: 1 argument(s) required, 0 supplied.

.. sm:stage:: 
   :source: ymp/rules/00_import.rules:64

   Imports raw read files into YMP.

   >>> ymp make toy
   >>> ymp make mpic

stage annotate_blast

Annotate sequences with BLAST

Searches a reference database for hits with blastn. Use E flag to specify exponent to required E-value. Use N or Mega to specify default. Use Best to add -subject_besthit flag.

stage annotate_diamond

FIXME

stage annotate_prodigal

Call genes using prodigal

>>> ymp make toy.ref_genome.annotate_prodigal

stage annotate_tblastn

Runs tblastn

stage assemble_megahit

Assemble metagenome using MegaHit.

>>> ymp make toy.assemble_megahit.map_bbmap
>>> ymp make toy.group_ALL.assemble_megahit.map_bbmap
>>> ymp make toy.group_Subject.assemble_megahit.map_bbmap

stage assemble_metaspades

Assemble reads using metaspades

>>> ymp make toy.assemble_metaspades
>>> ymp make toy.group_ALL.assemble_metaspades
>>> ymp make toy.group_Subject.assemble_metaspades

stage assemble_trinity

stage bin_metabat2

Bin metagenome assembly into MAGs

stage check

Verify file availability

This stage provides rules for checking the file availability at a given point in the stage stack.

Mainly useful for testing and debugging.

stage cluster_cdhit

Clusters protein sequences using CD-HIT

>>> ymp make toy.ref_query.cluster_cdhit

stage correct_bbmap

Correct read errors by overlapping inside tails

Applies BBMap's “bbmerge.sh ecco” mode. This will overlap the inside of read pairs and choose the base with the higher quality where the alignment contains mismatches and increase the quality score as indicated by the double observation where the alignment contains matches.

>>> ymp make toy.correct_bbmap
>>> ymp make mpic.correct_bbmap

stage count_diamond

stage count_stringtie

stage coverage_samtools

Computes coverage from a sorted bam file using samtools coverage

stage dedup_bbmap

Remove duplicate reads

Applies BBMap’s “dedupe.sh”

>>> ymp make toy.dedup_bbmap
>>> ymp make mpic.dedup_bbmap

stage dust_bbmap

Perform entropy filtering on reads using BBMap’s bbduk.sh

The parameter Enn gives the entropy cutoff. Higher values filter more sequences.

>>> ymp make toy.dust_bbmap
>>> ymp make toy.dust_bbmapE60

stage extract_reads

Extract reads from BAM file using samtools fastq.

Parameters fn, Fn and Gn are passed through. Some options include:

• f2: fully mapped (only proper pairs)

• F2: not fully mapped (unmapped at least one read)

• f12: not mapped (neither read mapped)

stage extract_seqs

Extract sequences from .fasta.gz file using samtools faidx

Currently requires a .blast7 file as input.

Use parameter Nomatch to instead keep unmatched sequences.

stage filter_bmtagger

Filter(-out) contaminant reads using BMTagger

>>> ymp make toy.ref_phiX.index_bmtagger.remove_bmtagger
>>> ymp make toy.ref_phiX.index_bmtagger.remove_bmtagger.assemble_megahit
>>> ymp make toy.ref_phiX.index_bmtagger.filter_bmtagger
>>> ymp make mpic.ref_phiX.index_bmtagger.remove_bmtagger

stage format_bbmap

Process sequences with BBMap’s format.sh

Parameter Ln filters sequences at a minimum length.

>>> ymp make toy.assemble_metaspades.format_bbmapL200

stage humann2

Compute functional profiles using HUMAnN2

stage index_bbmap

>>> ymp make toy.ref_genome.index_bbmap

stage index_blast

stage index_bmtagger

stage index_bowtie2

>>> ymp make toy.ref_genome.index_bowtie2

stage index_diamond

stage map_bbmap

Map reads using BBMap

>>> ymp make toy.assemble_megahit.map_bbmap
>>> ymp make toy.ref_genome.map_bbmap
>>> ymp make mpic.ref_ssu.map_bbmap

stage map_bowtie2

Map reads using Bowtie2

>>> ymp make toy.ref_genome.index_bowtie2.map_bowtie2
>>> ymp make toy.assemble_megahit.index_bowtie2.map_bowtie2
>>> ymp make toy.group_Subject.assemble_megahit.index_bowtie2.map_bowtie2
>>> ymp make mpic.ref_ssu.index_bowtie2.map_bowtie2

stage map_diamond

stage map_hisat2

Map reads using Hisat2

stage map_star

Map RNA-Seq reads with STAR

stage metaphlan2

Assess metagenome community composition using Metaphlan 2

stage primermatch_bbmap

Filters reads by matching reference primer

>>> ymp make mpic.ref_primers.primermatch_bbmap

stage profile_centrifuge

Classify reads using centrifuge

stage qc_fastqc

Quality screen reads using FastQC

>>> ymp make toy.qc_fastqc

stage qc_multiqc

Aggregate QC reports using MultiQC

stage qc_quast

Estimate assemly quality using Quast

stage quant_rsem

Quantify transcripts using RSEM

stage references

This is a “virtual” stage. It does not process read data, but comprises rules used for reference provisioning.

stage remove_bbmap

Filter reads by reference

This stage aligns the reads with a given reference using BBMap in fast mode. Matching reads are collected in the stage filter_bbmap and remaining reads are collectec in the stage remove_bbmap.

>>> ymp make toy.ref_phiX.index_bbmap.remove_bbmap
>>> ymp make toy.ref_phiX.index_bbmap.filter_bbmap
>>> ymp make mpic.ref_phiX.index_bbmap.remove_bbmap

stage sort_bam

stage split_library

Demultiplexes amplicon sequencing files

This rule is treated specially. If a configured project specifies a barcode_col, reads from the file (or files) are used in combination with

stage trim_bbmap

Trim adapters and low quality bases from reads

Applies BBMap’s “bbduk.sh”.

Parameters:

A: append to enable adapter trimming Q20: append to select phred score cutoff (default 20) L20: append to select minimum read length (default 20)

>>> ymp make toy.trim_bbmap
>>> ymp make toy.trim_bbmapA
>>> ymp make toy.trim_bbmapAQ10L10
>>> ymp make mpic.trim_bbmap

stage trim_sickle

Perform read trimming using Sickle

>>> ymp make toy.trim_sickle
>>> ymp make toy.trim_sickleQ10L10
>>> ymp make mpic.trim_sickleL20

stage trim_trimmomatic

Adapter trim reads using trimmomatic

>>> ymp make toy.trim_trimmomaticT32
>>> ymp make mpic.trim_trimmomatic

rule download_file_ftp

Downloads remote file using wget

rule download_file_http

Downloads remote file using internal downloader

rule prefetch

Downloads SRA files into NCBI SRA folder (ncbi/public/sra).

rule fastq_dump

Extracts FQ from SRA files

rule combine_with_ref

rule align_mafft

rule blast7_merge

Merges blast results from all samples into single file

rule blast7_extract

Generates meta-data csv and sequence fasta pair from blast7 file for one gene.

rule blast7_extract_merge

Merges extracted csv/fasta pairs over all samples.

rule blast7_all

rule blast7_reports

rule blast7_eval_hist

rule blast7_eval_plot

rule cdhit_fna_single

Clustering predicted genes (nuc) using cdhit-est

rule 87

rule 88

rule 89

rule 90

rule 91

rule 92

rule faa_fastp

rule fasta_to_fastp_gz

rule gunzip

Generic temporary gunzip

Use ruleorder: gunzip > myrule to prefer gunzipping over re-running a rule. E.g.

>>> ruleorder: gunzip > myrule
>>> rule myrule:
>>>   output: temp("some.txt"), "some.txt.gzip"

rule mkdir

Auto-create directories listed in ymp config.

Use these as input: >>> input: tmpdir = ancient(icfg.dir.tmp)

rule fq2fa

Unzip and convert fastq to fasta

rule make_otu_table

rule otu_to_qiime_txt

rule otu_to_biom

rule blast7_coverage_per_otu

rule pick_open_otus

Pick open reference OTUs

rule pick_closed_otus

Pick closed reference OTUs

rule rarefy_table

rule convert_to_closed_ref

Convert open reference otu table to closed reference

rule env_wait

rule ticktock

rule noop

rule normalize_16S

Normalize 16S by copy number using picrust, must be run with closed reference OTU table

rule predict_metagenome

Predict metagenome using picrust

rule categorize_by_function

Categorize PICRUSt KOs into pathways

rule raxml_tree

rule rsem_index

Build Genome Index for RSEM

rule scnic_within_minsamp

rule scnic_within_sparcc_filter

rule star_index

Build Genome Index for Star