Galaxy from an administrator's point of view

</span></div>

</span></div>

---

# Galaxy from an administrator's point of view

<div class="contributors-line">
		
	
<ul class="text-list">
			
			<li>
				<a href="/training-material/hall-of-fame/valentinmarcon/" class="contributor-badge contributor-valentinmarcon"><img src="https://avatars.githubusercontent.com/valentinmarcon?s=36" alt="Valentin Marcon avatar" width="36" class="avatar" />
    Valentin Marcon</a>
			<li>
				<a href="/training-material/hall-of-fame/natefoo/" class="contributor-badge contributor-natefoo"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/natefoo?s=36" alt="Nate Coraor avatar" width="36" class="avatar" />
    Nate Coraor</a>
			<li>
				<a href="/training-material/hall-of-fame/bgruening/" class="contributor-badge contributor-bgruening"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/bgruening?s=36" alt="Björn Grüning avatar" width="36" class="avatar" />
    Björn Grüning</a>
			<li>
				<a href="/training-material/hall-of-fame/hexylena/" class="contributor-badge contributor-hexylena"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/hexylena?s=36" alt="Helena Rasche avatar" width="36" class="avatar" />
    Helena Rasche</a></li>
</ul>

</div>

<div class="footnote" style="bottom: 8em;">
  <i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated:   
  <i class="fas fa-fingerprint" aria-hidden="true"></i><span class="visually-hidden">purl</span><abbr title="Persistent URL">PURL</abbr>: <a href="https://gxy.io/GTN:S00015">gxy.io/GTN:S00015</a>
</div>

<div class="footnote" style="bottom: 5em;">

<i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a> |

</div>

<div class="footnote" style="bottom: 2em;">
    <strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes
    | <i class="fa fa-arrows" aria-hidden="true"></i><span class="visually-hidden">arrow-keys</span> Use arrow keys to move between slides

</div>

???
Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press `P` again to switch presenter notes off

Press `C` to create a new window where the same presentation will be displayed.
This window is linked to the main window. Changing slides on one will cause the
slide to change on the other.

Useful when presenting.

---

![Picture with 125+ platforms for using galaxy written and a number of screenshots of those galaxies.](../../images/use-resource-banner.png)

---

# Where can Galaxy run?

* Cloud (SaaS)
  - A usegalaxy.* site: [Galaxy Main](https://usegalaxy.org), [Galaxy France](https://usegalaxy.fr), [Galaxy Australia](https://usegalaxy.org.au), [Galaxy Europe](https://usegalaxy.eu)
  - [Public Galaxy Servers](https://galaxyproject.org/use)
  - [Amazon EC2 or MS Azure](https://launch.usegalaxy.org/)
  - Semi-private cloud (e.g.: [NeCTAR](https://nectar.org.au/), [Jetstream](http://jetstream-cloud.org/))
* Private cloud (build your own Galaxy SaaS)
* Cloud (IaaS)
  - Any cloud
* Scalable Local Server
  - Dedicated or shared compute cluster(s)
  - Cloud compute resources
* Standalone Local Server

---

# Choosing where to run

[Resource Directory](https://galaxyproject.org/use/)

Supported                                         | UseGalaxy.* | Local     | Cloud
---------                                         | ----        | -----     | -----
Moderate Data                                     | ✅          | ✅        | ✅
Big Data                                          | ❌          | 🤷‍♀️ | ✅
Moderate Computations                             | ✅          | ✅        | ✅
Long/Expensive Computations                       | ❌          | 🤷‍  | ✅
You want to share your Galaxy objects with others | ✅          | ✅        | ✅
All needed Tools are pre-installed                | ✅          | 🤷‍♂️ | ✅
Human Data / Proprietary Data                     | ❌          | ✅        | ✅
No network transfer of data                       | ❌          | ✅        | ❌

---

## Reasons to Install Your Own Galaxy

- You want to run a local [production Galaxy](https://docs.galaxyproject.org/en/latest/admin/production.html)
- You want to develop [Galaxy tools](https://galaxyproject.org/admin/tools/add-tool-tutorial)
- You want to [Develop Galaxy](https://galaxyproject.org/develop/) itself
- [Install](https://galaxyproject.org/admin/tools/add-tool-tutorial) and use tools unavailable on [public Galaxies](https://galaxyproject.org/public-galaxy-servers)
- Use sensitive data (e.g. clinical)
- Process large datasets that are too big for public Galaxies
- [plug-in](https://galaxyproject.org/admin/internals/data-sources) new datasources

---

# Software Requirements

Required:
  - Galaxy is written in Python and depends on **Python 3.8** or newer

Minimal [production](https://docs.galaxyproject.org/en/master/admin/production.html) requirements:
  - [PostgreSQL](/training-material/topics/admin/tutorials/database/slides.html)
  - Reverse proxy server (NGINX, Apache)

---

# Hardware Requirements

This depends:

- What do you intend to run?
- Where do you intend to run it?

If possible, run the Galaxy server **separate** from Galaxy jobs

**Storage** will usually be the biggest concern

???

- Depends on your available infrastructure
- If you are storage limited, can be addressed by policy of deleting old/unused histories
- If you are compute limited, can be addressed with queue limits

---
# Server Hardware Requirements

Based on concurrent user count and assuming separate compute for jobs:

Users     | Resource estimate
----------|-------------------
1 - 5     | 1 core, 1GB, 10 TB
5 - 20    | 2 cores, 2 GB, 40 TB
20 - 40   | 8 cores, 8 GB, 200 TB
40+       | multiple hosts, 16 GB/host, 500 TB, dedicated DB host

Storage is more difficult to estimate since it is, like compute, **analysis** and **policy** dependent

---
template: left-aligned

# Galaxy Storage Philosophy

- Foster transparency and reproducibility
- Data is always created, *never overwritten*
- Copying history or library datasets associate them with the original file on disk without an actual copy
- By default, data is never really deleted unless *explicitly instructed*
- Even deleted data can be undeleted unless *forcibly purged*

---

# Storage Requirements

An "average" 2018 NGS analysis (by Anton Nekrutenko): **66 GB**

10 users, 10 histories: **> 6 TB**

Solutions:

- Quotas
- Set job limits in the job conf
- Clean up deleted data (with a cron job)
- Forced removal based on age
- Users can configure their workflows to delete intermediate tool outputs
- Data libraries for common data
- Public servers: require email verification (and watch for duplicates)
- Plug in more/heterogeneous storage using Object Store configuration

---

# Compute Requirements

This depends:

- What tools will your users be using?
  - What are their requirements?
- In general, the most commonly used tools use a single core
  - But can use lots of memory!
- Some compute-intensive tools use multiple cores

usegalaxy.org allocates from **8 GB/core** to **16 GB/core**

Connecting Galaxy to clusters/HPC is covered in the advanced section.

---

# Making plans

**Before** deploying your first Galaxy server:

- Figure out where Galaxy will be stored
  - Make sure it will be accessible to any eventual compute
- Figure out where data will be stored
  - Make sure it will be accessible to any eventual compute

---

# Galaxy deployment options

- As developer: `git clone https://github.com/galaxyproject/galaxy.git`
- Ready-to-use locally: [Docker](https://github.com/bgruening/docker-galaxy-stable/)
- Production server: configuration management (e.g. Ansible)

In future:

- Alternative to git clone: Galaxy wheel in PyPI

---

# Deployment Best Practices

- **Use configuration management**
  - [Ansible](http://docs.ansible.com) for which Galaxy Project maintains roles ([tutorial](https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible/slides.html))
  - Other systems are possible (Chef, Puppet, SaltStack, CFEngine) but do not have project-maintained roles.

- Whether you use configuration management or not, *record every change you make on a version control system* (e.g. git):
  - Large, complex deployments grow organically
  - If you don't know what you did, you can't do it again

---

# System Administration Best Practices

- Take security seriously
- **Update Galaxy when security updates are released**
- Follow OS security best practices
- Privilege separate code/job/data ownership
- Write protect Galaxy and data if you can
- Read-only cluster mounts

Back up everything (except that which is managed by configuration management)

---

## Related tutorials

---

## Thank You!

This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors!

<div class="contributors-line">
		
<table class="contributions">
	
	<tr>
		<td><abbr title="These people wrote the bulk of the tutorial, they may have done the analysis, built the workflow, and wrote the text themselves.">Author(s)</abbr></td>
		<td>
			<a href="/training-material/hall-of-fame/valentinmarcon/" class="contributor-badge contributor-valentinmarcon"><img src="https://avatars.githubusercontent.com/valentinmarcon?s=36" alt="Valentin Marcon avatar" width="36" class="avatar" />
    Valentin Marcon</a><a href="/training-material/hall-of-fame/natefoo/" class="contributor-badge contributor-natefoo"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/natefoo?s=36" alt="Nate Coraor avatar" width="36" class="avatar" />
    Nate Coraor</a><a href="/training-material/hall-of-fame/bgruening/" class="contributor-badge contributor-bgruening"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/bgruening?s=36" alt="Björn Grüning avatar" width="36" class="avatar" />
    Björn Grüning</a><a href="/training-material/hall-of-fame/hexylena/" class="contributor-badge contributor-hexylena"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/hexylena?s=36" alt="Helena Rasche avatar" width="36" class="avatar" />
    Helena Rasche</a>
		</td>
	</tr>

<tr class="reviewers">
		<td><abbr title="These people reviewed this material for accuracy and correctness">Reviewers</abbr></td>
		<td>
			<a href="/training-material/hall-of-fame/slugger70/" class="contributor-badge contributor-badge-small contributor-slugger70"><img src="https://avatars.githubusercontent.com/slugger70?s=36" alt="Simon Gladman avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/bebatut/" class="contributor-badge contributor-badge-small contributor-bebatut"><img src="https://avatars.githubusercontent.com/bebatut?s=36" alt="Bérénice Batut avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/natefoo/" class="contributor-badge contributor-badge-small contributor-natefoo"><img src="https://avatars.githubusercontent.com/natefoo?s=36" alt="Nate Coraor avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/hexylena/" class="contributor-badge contributor-badge-small contributor-hexylena"><img src="https://avatars.githubusercontent.com/hexylena?s=36" alt="Helena Rasche avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/mtekman/" class="contributor-badge contributor-badge-small contributor-mtekman"><img src="https://avatars.githubusercontent.com/mtekman?s=36" alt="Mehmet Tekman avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/shiltemann/" class="contributor-badge contributor-badge-small contributor-shiltemann"><img src="https://avatars.githubusercontent.com/shiltemann?s=36" alt="Saskia Hiltemann avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/nsoranzo/" class="contributor-badge contributor-badge-small contributor-nsoranzo"><img src="https://avatars.githubusercontent.com/nsoranzo?s=36" alt="Nicola Soranzo avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/bgruening/" class="contributor-badge contributor-badge-small contributor-bgruening"><img src="https://avatars.githubusercontent.com/bgruening?s=36" alt="Björn Grüning avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/gallardoalba/" class="contributor-badge contributor-badge-small contributor-gallardoalba"><img src="https://avatars.githubusercontent.com/gallardoalba?s=36" alt="Cristóbal Gallardo avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/willdurand/" class="contributor-badge contributor-badge-small contributor-willdurand"><img src="https://avatars.githubusercontent.com/willdurand?s=36" alt="William Durand avatar" width="36" class="avatar" /></a></td>
	</tr>

</table>

</div>

</div>

Tutorial Content is licensed under <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.<br/>