Matrix Exchange Format to ESet | Creating a single-cell RNA-seq reference dataset for deconvolution

Author(s) orcid logoWendi Bacon avatar Wendi BaconMehmet Tekman avatar Mehmet Tekman
Tester(s) orcid logoMarisa Loach avatar Marisa Loach
Reviewers Mehmet Tekman avatarSaskia Hiltemann avatarWendi Bacon avatarHelena Rasche avatarPablo Moreno avatarJulia Jakiela avatarPavankumar Videm avatar
Overview
Creative Commons License: CC-BY Questions:
  • Where can I find good quality scRNA-seq reference datasets?

  • How can I reformat and manipulate these downloads to create the right format for MuSiC?

Objectives:
  • You will retrieve raw data from the EMBL-EBI Single cell expression atlas.

  • You will manipulate the metadata and matrix files.

  • You will combine the metadata and matrix files into an ESet object for MuSiC deconvolution.

  • You will create multiple ESet objects - both combined and separated out by disease phenotype for your single cell reference.

Requirements:
Time estimation: 1 hour
Supporting Materials:
Published: Jan 20, 2023
Last modification: Feb 13, 2024
License: Tutorial Content is licensed under Creative Commons Attribution 4.0 International License. The GTN Framework is licensed under MIT
purl PURL: https://gxy.io/GTN:T00241
version Revision: 8

After completing the MuSiC Wang et al. 2019 deconvolution tutorial, you are hopefully excited to apply this analysis to data of your choice. Annoyingly, getting data in the right format is often what prevents us from being able to successfully apply analyses. This tutorial is all about reformatting a raw scRNA-seq dataset pulled from a public resource (the EMBL-EBI single cell expression atlas Moreno et al. 2021. Let’s get started!

Agenda

In this tutorial, we will cover:

  1. Metadata Manipulation
    1. Find the data
    2. Prepare the experimental design file
    3. Prepare the barcodes file
    4. Use the barcodes list to filter out cells in the experimental design file
  2. Manipulate the expression matrix
    1. Reformat the matrix
    2. Collapse EnsemblIDs
  3. Construct Expression Set Objects
  4. Conclusion

Metadata Manipulation

First, we will tackle the metadata. We are roughly following the same concept as in the previous bulk deconvolution tutorial, by comparing human pancreas data across a disease variable (type II diabetes vs healthy), but using public datasets to do it.

Find the data

We explored the single cell expression atlas, browsing experiments in order to find a pancreas dataset (Segerstolpe et al. 2016). You can explore this dataset using their browser. These cells come from 6 healthy individuals and 4 individuals with Type II diabetes, so we will create reference Expression Set objects for the total as well as separating out by phenotype, as you may have reason to do this in your analysis (or you may not!).

Tools are frequently updated to new versions. Your Galaxy may have multiple versions of the same tool available. By default, you will be shown the latest version of the tool. This may NOT be the same tool used in the tutorial you are accessing. Furthermore, if you use a newer tool in one step, and try using an older tool in the next step… this may fail! To ensure you use the same tool versions of a given tutorial, use the Tutorial mode feature.

  • Open your Galaxy server
  • Click on the curriculum icon on the top menu, this will open the GTN inside Galaxy.
  • Navigate to your tutorial
  • Tool names in tutorials will be blue buttons that open the correct tool for you
  • Note: this does not work for all tutorials (yet) gif showing how GTN-in-Galaxy works
  • You can click anywhere in the grey-ed out area outside of the tutorial box to return back to the Galaxy analytical interface
Warning: Not all browsers work!
  • We’ve had some issues with Tutorial mode on Safari for Mac users.
  • Try a different browser if you aren’t seeing the button.

Galaxy has a specific tool for ingesting data from the Single cell expression atlas, so there are no uploads for this tutorial.

Hands-on: Data retrieval
  1. EBI SCXA Data Retrieval ( Galaxy version v0.0.2+galaxy2) with the following parameters:
    • “SC-Atlas experiment accession”: E-MTAB-5061

Data management is going to be key in this analysis, so trust me now to start adding tags.

  1. Add to the EBI SCXA Data Retrieval on E-MTAB-5061 exp_design.tsv file the following tags: #ebi #metadata #singlecell

    Datasets can be tagged. This simplifies the tracking of datasets across the Galaxy interface. Tags can contain any combination of letters or numbers but cannot contain spaces.

    To tag a dataset:

    1. Click on the dataset to expand it
    2. Click on Add Tags galaxy-tags
    3. Add tag text. Tags starting with # will be automatically propagated to the outputs of tools using this dataset (see below).
    4. Press Enter
    5. Check that the tag appears below the dataset name

    Tags beginning with # are special!

    They are called Name tags. The unique feature of these tags is that they propagate: if a dataset is labelled with a name tag, all derivatives (children) of this dataset will automatically inherit this tag (see below). The figure below explains why this is so useful. Consider the following analysis (numbers in parenthesis correspond to dataset numbers in the figure below):

    1. a set of forward and reverse reads (datasets 1 and 2) is mapped against a reference using Bowtie2 generating dataset 3;
    2. dataset 3 is used to calculate read coverage using BedTools Genome Coverage separately for + and - strands. This generates two datasets (4 and 5 for plus and minus, respectively);
    3. datasets 4 and 5 are used as inputs to Macs2 broadCall datasets generating datasets 6 and 8;
    4. datasets 6 and 8 are intersected with coordinates of genes (dataset 9) using BedTools Intersect generating datasets 10 and 11.

    A history without name tags versus history with name tags

    Now consider that this analysis is done without name tags. This is shown on the left side of the figure. It is hard to trace which datasets contain “plus” data versus “minus” data. For example, does dataset 10 contain “plus” data or “minus” data? Probably “minus” but are you sure? In the case of a small history like the one shown here, it is possible to trace this manually but as the size of a history grows it will become very challenging.

    The right side of the figure shows exactly the same analysis, but using name tags. When the analysis was conducted datasets 4 and 5 were tagged with #plus and #minus, respectively. When they were used as inputs to Macs2 resulting datasets 6 and 8 automatically inherited them and so on… As a result it is straightforward to trace both branches (plus and minus) of this analysis.

    More information is in a dedicated #nametag tutorial.

This tool will retrieve four files: a barcodes list, a genes list, an experimental design file, and a matrix market format (where columns refer to genes, cells, and quantities). We (mostly) only need the experimental design file, but keep in mind this will have data on all the cells reported by the authors.

Question
  1. How many cells are in the sample?
  2. How many cells were submitted by the authors?
  1. If you select the param-file barcodes.tsv file, you’ll find that it contains 2914 lines - this corresponds to the 2914 cells, because each cell is given a barcode.
  2. The nature of public repositories is that they ingest data from many places, which means they usually apply a uniform analysis to samples. This rarely means they yield the same cell numbers as the original authors. If you check the param-file exp_design.tsv file, which refers to the data submitted by the authors, you’ll find it contains 3514 lines - referring to 3514 cells submitted by authors. It’s important to know that (currently) these files differ.

Prepare the experimental design file

Let’s get rid of a bunch of repetitive columns in the metadata we don’t need. You can find out what each column is by inspecting the dataset galaxy-eye in the history window.

Hands-on: Cutting necessary metadata columns
  1. Cut with the following parameters:
    • “Cut columns”: c1,c4,c6,c8,c10,c14,c20,c24,c26,c30,c32,c34
    • param-file “From”: design_tsv (output of EBI SCXA Data Retrieval tool)

You can inspect the dataset galaxy-eye to see that it’s full of annoying “” everywhere, and overly long descriptions of each columns.

Columns in Galaxy history window contain "Assay" and "" around every word or ID. Open image in new tab

Figure 1: Annoying metadata

Now, there might be a better way to do this in Galaxy (or you might consider downloading the file locally and changing it in a spreadsheet application or something), but this is what will work to reformat all that annoying text.

Hands-on: Reformatting the metadata
  1. Regex Find And Replace ( Galaxy version 1.0.2) with the following parameters:
    • param-file “Select lines from”: out_file1 (output of Cut tool)
    • In “Check”:
      • param-repeat “Insert Check”
        • “Find Regex”: "Sample Characteristic\[individual\]"
        • “Replacement”: Individual
      • param-repeat “Insert Check”
        • “Find Regex”: "Sample Characteristic\[sex\]"
        • “Replacement”: Sex
      • param-repeat “Insert Check”
        • “Find Regex”: "Sample Characteristic\[age\]"
        • “Replacement”: Age
      • param-repeat “Insert Check”
        • “Find Regex”: "Sample Characteristic\[body mass index\]"
        • “Replacement”: BMI
      • param-repeat “Insert Check”
        • “Find Regex”: kilogram per square meter
      • param-repeat “Insert Check”
        • “Find Regex”: HbA1c
      • param-repeat “Insert Check”
        • “Find Regex”: "Sample Characteristic\[clinical information\]"
        • “Replacement”: HbA1c
      • param-repeat “Insert Check”
        • “Find Regex”: %
      • param-repeat “Insert Check”
        • “Find Regex”: "Sample Characteristic\[disease\]"
        • “Replacement”: Disease
      • param-repeat “Insert Check”
        • “Find Regex”: "Sample Characteristic\[single cell quality\]"
        • “Replacement”: Single cell quality
      • param-repeat “Insert Check”
        • “Find Regex”: "Sample Characteristic\[submitted single cell quality\]"
        • “Replacement”: "Submitted single cell quality"
      • param-repeat “Insert Check”
        • “Find Regex”: "Factor Value\[inferred cell type - ontology labels\]"
        • “Replacement”: Inferred cell type - ontology label
      • param-repeat “Insert Check”
        • “Find Regex”: "Factor Value\[inferred cell type - authors labels\]"
        • “Replacement”: Inferred cell type - author labels
      • param-repeat “Insert Check”
        • “Find Regex”: ""
        • “Replacement”:
      • param-repeat “Insert Check”
        • “Find Regex”: "
        • “Replacement”:
    Comment

    What’s with the \ everywhere? That’s because the [] symbols usually call the code to do something, rather than just read it as a normal character. the \ prevents this.

  2. Change the datatype to tabular.

    • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
    • In the central panel, click galaxy-chart-select-data Datatypes tab on the top
    • In the galaxy-chart-select-data Assign Datatype, select tabular from “New type” dropdown
      • Tip: you can start typing the datatype into the field to filter the dropdown menu
    • Click the Save button

Great, this file is now ready to go! But, it contains all those extra cells that didn’t pass filtration with the EBI pipeline and therefore won’t exist in the matrix. We need to remove them for future steps to work. We can use our barcodes list to remove the extra cells.

Prepare the barcodes file

Hands-on: Adding a header
  1. Add line to file ( Galaxy version 0.1.0) with the following parameters:
    • “text to add”: Cell
    • param-file “input file”: barcode_tsv (output of EBI SCXA Data Retrieval tool)
  2. Change the datatype to tabular.

    • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
    • In the central panel, click galaxy-chart-select-data Datatypes tab on the top
    • In the galaxy-chart-select-data Assign Datatype, select tabular from “New type” dropdown
      • Tip: you can start typing the datatype into the field to filter the dropdown menu
    • Click the Save button

    Comment

    This is an annoying step we have to do to get the right format, otherwise future steps won’t work.

Use the barcodes list to filter out cells in the experimental design file

Hands-on: Joining datasets
  1. Join two Datasets with the following parameters:
    • param-file “Join”: outfile (output of Add line to file tool)
    • “using column”: c1
    • param-file “with”: out_file1 (output of Regex Find And Replace tool)
    • “and column”: c1
    • “Fill empty columns”: No
    • “Keep the header lines”: Yes
      Comment

      Make sure that you join the files in the same order as above - put the output of Add line to file in first - otherwise your columns will be in a different order for the next step. Everything will still work, but you would need to change the number of the column you remove using Advanced Cut.

Question
  1. How many cells are now in your table?
  2. Is your table ready to go?
  1. If you select the output dataset in your history, you will find 2915 lines, corresponding to 2914 cells and a header. Success!
  2. Not quite - notice how you have two identical columns Cell and Assay? Let’s get rid of one.
Hands-on: Remove duplicate columns
  1. Advanced Cut ( Galaxy version 1.1.0) with the following parameters:
    • param-file “File to cut”: out_file1 (output of Join two Datasets tool)
    • “Operation”: Discard
    • “Cut by”: fields
      • “List of Fields”: c1
    Comment

    Advanced cut works slightly differently in a workflow versus running the tool independently. Independently, there is a list and you can click through the list to note your columns, while in a workflow it appears as a text option and you put each column on a different line. The point is, each number above represents a column, so remove them!

Fantastic! You’ve completed part 1 - making the single cell metadata file. It should now look like this:

Columns in the history window of a dataset contain words without any extra symbols or "". Open image in new tab

Figure 2: Pretty scRNA metadata

You can use the workflow for this portion of the tutorial, and access an example history.

Manipulate the expression matrix

Currently, the matrix data is in a 3-column format common in 10x outputs, where you need the barcodes and the genes files to interpret the matrix. What you actually need is an expression matrix with cells on one axis and genes on another. While we aren’t running a Scanpy analysis, we can still use our Scanpy tools to get this format.

Reformat the matrix

Hands-on: Task description
  1. Scanpy Read10x ( Galaxy version 1.8.1+galaxy0) with the following parameters:
    • param-file “Expression matrix in sparse matrix format (.mtx)”: matrix_mtx (output of EBI SCXA Data Retrieval tool)
    • param-file “Gene table”: genes_tsv (output of EBI SCXA Data Retrieval tool)
    • param-file “Barcode/cell table”: barcode_tsv (output of EBI SCXA Data Retrieval tool)
    • “Format of output object”: AnnData format (h5 for older versions)
  2. Change the datatype to h5ad

    • Click on the galaxy-pencil pencil icon for the dataset to edit its attributes
    • In the central panel, click galaxy-chart-select-data Datatypes tab on the top
    • In the galaxy-chart-select-data Assign Datatype, select h5ad from “New type” dropdown
      • Tip: you can start typing the datatype into the field to filter the dropdown menu
    • Click the Save button

Now your precious matrix is stored in the 10x AnnData object. Let’s retrieve it!

Hands-on: Inspect the matrix
  1. Inspect AnnData ( Galaxy version 0.7.5+galaxy1) with the following parameters:
    • param-file “Annotated data matrix”: output_h5 (output of Scanpy Read10x tool)
    • “What to inspect?”: The full data matrix
Question
  1. Which are currently the rows in your matrix, cells or genes?
  1. You may remember from earlier that the sample should have 2914 cells in it. If you inspect the dataset in your history, you will find that it contains 2915 lines (1 for the header), which means that rows correspond to cells. Unfortunately… that’s not what you need.
Hands-on: Transpose the matrix
  1. Transpose ( Galaxy version 1.1.0+galaxy2) with the following parameters:
    • param-file “Input tabular dataset”: X (output of Inspect AnnData tool)
Question
  1. How many genes are in your sample?
  1. You should have 30,416 lines in it, meaning your sample has 30,415 genes.

Collapse EnsemblIDs

Ok, real talk here. Technically, the best way of analysing anything is by using the EnsemblIDs for any given RNA transcript, because they are more specific than gene names and also cover more of the transcriptome than our gene names… But… As biologists, it’s very difficult to interpret ENSIDs. And it’s an awful shame to get to the end of the MuSiC deconvolution and have all our plots show sad ENS IDs. So, courtesy of the excellent @mtekman, we steal his workflow to collapse the ENS IDs into gene names.

First table shows 4 rows with different ENS IDs and a second column with geney symbols and some overlap. Arrow pointing to second table shows 3 rows having collapsed the overlap.Open image in new tab

Figure 3: Collapsing ENS IDs
Hands-on: Convert from Ensembl to GeneSymbol using workflow
  1. Import this workflow.

    • Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
    • Click on galaxy-upload Import at the top-right of the screen
    • Provide your workflow
      • Option 1: Paste the URL of the workflow into the box labelled “Archived Workflow URL”
      • Option 2: Upload the workflow file in the box labelled “Archived Workflow File”
    • Click the Import workflow button

    Below is a short video demonstrating how to import a workflow from GitHub using this procedure:

    Video: Importing a workflow from URL

  2. Run the workflow on your sample with the following parameters:

    • “Organism”: Human
    • param-file “Expression Matrix (Gene Rows)”: output_h5 (output of Transpose tool)
    • Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
    • Click on the workflow-run (Run workflow) button next to your workflow
    • Configure the workflow as needed
    • Click the Run Workflow button at the top-right of the screen
    • You may have to refresh your history to see the queued jobs

The output will likely be called Text transformation and will look like this:

Alphabetised gene symbols appear in column one with decimal pointed integers in the following columns corresponding to cells. Open image in new tab

Figure 4: Output of the ENS ID collapsing workflow

Construct Expression Set Objects

We’re nearly there! We have three more tasks to do: first, we need to create the expression set object with all the phenotypes combined. Then, we also want to create two separate objects - one for healthy and one for diseased as references.

Hands-on: Creating the combined object
  1. Construct Expression Set Object ( Galaxy version 0.1.1+galaxy4) with the following parameters:
    • param-file “Assay Data”: out_file #matrix (output of Text transformation tool)
    • param-file “Phenotype Data”: output (output of Advanced Cut tool)
  2. Remove the #metadata #matrix tags from the output RData ESet Object

  3. Add the tag #combined to the output RData ESet Object
Question
  1. How many genes are in your sample now?
  1. If you select the galaxy-eye of the output General Info dataset in the history, you will find it contains 21671 features and 2914 samples, or rather, 21671 genes and 2914 cells. That’s a huge reduction in genes thanks to the ENS ID collapsing!
Hands-on: Creating the disease-only object
  1. Manipulate Expression Set Object ( Galaxy version 0.1.1+galaxy4) with the following parameters:
    • param-file “Expression Set Dataset”: out_rds (output of Construct Expression Set Object tool)
    • “Concatenate other Expression Set objects?”: No
    • “Subset the dataset?”: Yes
      • “By”: Filter Samples and Genes by Phenotype Values
        • In “Filter Samples by Condition”:
          • param-repeat “Insert Filter Samples by Condition”
            • “Name of phenotype column”: Disease
            • “List of values in this column to filter for, comma-delimited”: type II diabetes mellitus
  2. Remove the #combined tag from the output RData ESet Object

  3. Add the tag #T2D to the output RData ESet Object

You can either re-run this tool or set it up again to create the healthy-only object.

Hands-on: Creating the healthy-only object
  1. Manipulate Expression Set Object ( Galaxy version 0.1.1+galaxy4) with the following parameters:
    • param-file “Expression Set Dataset”: out_rds (output of Construct Expression Set Object tool)
    • “Concatenate other Expression Set objects?”: No
    • “Subset the dataset?”: Yes
      • “By”: Filter Samples and Genes by Phenotype Values
        • In “Filter Samples by Condition”:
          • param-repeat “Insert Filter Samples by Condition”
            • “Name of phenotype column”: Disease
            • “List of values in this column to filter for, comma-delimited”: normal
  2. Remove the #combined tag from the output RData ESet Object

  3. Add the tag #healthy to the output RData ESet Object
Question
  1. Why are you making a healthy-only and diseased-only reference objects?
  1. We could imagine that the cells will express different transcript levels, but that the deconvolution tools will have to take some sort of average. Perhaps it might be more accurate to infer like from like, i.e. healthy from healthy? Or perhaps that is skewing the data through a more ‘supervised’ approach. We’re not quite sure, and it likely depends on the biology, so we’re covering all our bases by making sure you can do this every way. (We’ve tested it on our dataset in all the ways and got the same results, so it doesn’t make much of a difference as far as we can tell!)

Conclusion

You have successfully performed, essentially, three workflows. You can find the workflows for generating the ESet object and the answer key history for this entire tutorial.

6 boxes in the workflow editor. Open image in new tab

Figure 5: Workflow: Manipulating metadata
9 boxes including a subworkflow in the workflow editor. Open image in new tab

Figure 6: Workflow: Creating the ESet single-cell object
8 boxes in the workflow editor. Open image in new tab

Figure 7: Subworkflow: Collapsing the Ensembl IDs into gene names

With these workflows, you’ve created three Expression Set objects, capable of running in the MuSiC Compare tutorial. Now you just need the bulk RNA-seq Expression Set objects!

This tutorial is part of the https://singlecell.usegalaxy.eu portal (Tekman et al. 2020).

feedback To discuss with like-minded scientists, join our Galaxy Training Network chatspace in Slack and discuss with fellow users of Galaxy single cell analysis tools on #single-cell-users

We also post new tutorials / workflows there from time to time, as well as any other news.

point-right If you’d like to contribute ideas, requests or feedback as part of the wider community building single-cell and spatial resources within Galaxy, you can also join our Single cell & sPatial Omics Community of Practice.

tool You can request tools here on our Single Cell and Spatial Omics Community Tool Request Spreadsheet