Deploying a Beacon v1 in Galaxy
Author(s) | Helena Rasche |
Editor(s) | Saskia Hiltemann |
Reviewers |
OverviewQuestions:Objectives:
What is a Beacon?
How do I deploy it?
Is v1 the same as v2?
Requirements:
Deploy a Beacon
- slides Slides: Ansible
- tutorial Hands-on: Ansible
- slides Slides: Galaxy Installation with Ansible
- tutorial Hands-on: Galaxy Installation with Ansible
Time estimation: 30 minutesSupporting Materials:
Published: Apr 16, 2023Last modification: Dec 9, 2024License: Tutorial Content is licensed under Creative Commons Attribution 4.0 International License. The GTN Framework is licensed under MITpurl PURL: https://gxy.io/GTN:T00325version Revision: 4
This tutorial will guide you through setting up a GA4GH Beacon!
The Beacon Project is developed under a Global Alliance for Genomics and Health (GA4GH) Iniciative for the federated discovery of genomic data in biomedical research and clinical applications.
Warning: Beacon v1This deploys an older Beacon v1 which was a simpler system. The Beacon v1 is more or less deprecated, with users being pushed to Beacon v2 which gives much richer answers, and offers better querying syntax.
Agenda
Installing and Configuring
Hands-on: Setting up a Beacon with Ansible
Setup the hosts
--- a/hosts +++ b/hosts @@ -9,3 +9,12 @@ gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu [sentryservers] gat-0.eu.training.galaxyproject.eu ansible_connection=local ansible_user=ubuntu + +[beacon] +[beacon:children] +beacon_import +beacon_server +[beacon_server] +gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu +[beacon_import] +gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
Here we use some of the more advanced features of Ansible’s Inventory system. We declare a host group called ‘beacon’ with no hosts of its own.
Then we declare that this beacon group has two children: beacon_import, and beacon_server. We can then define host groups for those two entries with as many different hosts as we need. This makes it very easy to scale up configuration.
Here we will be using that feature to declare some ‘beacon variables’, which will be shared between the beacon_server and beacon_importer. Because they’re children of ‘beacon’, they’ll inherit any group variables defined for
group_vars/beacon.yml
.Setup the requirements
--- a/requirements.yml +++ b/requirements.yml @@ -60,3 +60,8 @@ # Our FTP Server - src: galaxyproject.proftpd version: 0.3.1 +# Beacon support +- name: paprikant.beacon + src: https://github.com/Paprikant/ansible-role-beacon +- name: paprikant.beacon-importer + src: https://github.com/Paprikant/ansible-role-beacon_importer
Install the role with:
Input: Bashansible-galaxy install -p roles -r requirements.yml
Install the docker role as well.
Input: Bashansible-galaxy collection install community.docker
Ansible over time added the feature of ‘collections’ in addition to roles. It’s a nice and convenient method for grouping roles together. Unfortunately this means we need to update a significant portion of the GAT learning path to use this, which we have not done yet, as it will take a bit of time to do. Collections come with the unpleasant additional aspect that they will be installed into a different path than roles, so we’ll need to add a
collections
folder as well, or maybedeps/collections
anddeps/roles
folders. When this happens we’ll have to additionally modify the ansible.cfg to look for collections in this additional path.Create the vars file
--- /dev/null +++ b/group_vars/beacon.yml @@ -0,0 +1,31 @@ +--- +postgres_data_dir: /data/beacon/postgresql/data +postgres_init_dir: /data/beacon/postgresql/init +bp_external_binding: 5050 # The default +postgres_user: "{{ beacon_db_user }}" +postgres_pass: "{{ beacon_db_password }}" +postgres_external_binding: "{{ beacon_db_port }}" +# Database Configuration +beacon_db_user: beacon +beacon_db_host: "{{ groups['beacon_server'][0] }}" +beacon_db_password: "{{ vault_beacon_db_password }}" +beacon_db_port: 9001 +#galaxy_api_key: This we will set in secrets. +# Information about your beacon (consider filling this out. +beacon_info_title: GA4GH Beacon +beacon_info_beacon_id: your.galaxy.beacon +beacon_info_description: Beacon service hosting datasets from all over the Galaxy +beacon_info_url: https://{{ groups['beacon_server'][0] }}/beacon/ +beacon_info_service_group: galaxy-eu +beacon_info_org_id: usegalaxy.aq +beacon_info_org_name: Some Galaxy +beacon_info_org_description: Galaxy community +beacon_info_org_address: 123 Main Street, ZA +beacon_info_org_welcome_url: https://galaxyproject.org/ +beacon_info_org_contact_url: https://galaxyproject.org/ +beacon_info_org_logo_url: https://galaxyproject.org/images/galaxy-logos/galaxy_project_logo_square.png +beacon_info_org_info: More information about the organisation than just the description can go here. +# Script Configuration +galaxy_api_url: "https://{{ groups['galaxyservers'][0] }}" +script_user: beacon +script_dir: /home/beacon/script
Here we again use some advanced features of Ansible’s inventory system. Ansible knows the name of every hostname in the inventory. Now that we want to point the beacon configuration, either at the database which should be on
beacon_server
, or at Galaxy ingalaxyservers
, we ask thegroups
variable for what the inventory looks like. We use[0]
to pull out the first hostname we find for both of those groups.Edit your
group_vars/secret.yml
and define some random passwords:
- The API key for your account, which must be an admin
- A password for the beacon database
Galaxy admin accounts are specified as a comma-separated email list in the
admin_users
directive ofgalaxy.yml
. If you have set up your Galaxy server using the Galaxy Installation with Ansible tutorial, this is set toadmin@example.org
.
- In your browser, open your Galaxy homepage
- Log in, or register a new account, if it’s the first time you’re logging in
- Go to
User -> Preferences
in the top menu bar, then click onManage API key
- If there is no current API key available, click on
Create a new key
to generate it- Copy your API key to somewhere convenient, you will need it throughout this tutorial
Input: Bashansible-vault edit group_vars/secret.yml
--- a/group_vars/secret.yml +++ b/group_vars/secret.yml @@ -1,13 +1,15 @@ +galaxy_api_key: your-galaxy-api-key +vault_beacon_db_password: some-super-secret-password
Add the beacon-server playbook
--- /dev/null +++ b/beacon-server.yml @@ -0,0 +1,9 @@ +--- +- name: Beacon Server + hosts: beacon_server + become: true + become_user: root + vars_files: + - group_vars/secret.yml + roles: + - paprikant.beacon
Run the playbook
Input: Bashansible-playbook beacon-server.yml
TODO: Check that it works
Setting up the Importer
Now that our beacon is running, we need to get data from Galaxy to the Beacon
Hands-on: Setting up the Beacon Importer
Add the beacon-import playbook
--- /dev/null +++ b/beacon-import.yml @@ -0,0 +1,9 @@ +--- +- name: Beacon Importer + hosts: beacon_import + become: true + become_user: root + vars_files: + - group_vars/secret.yml + roles: + - paprikant.beacon-importer
Run the playbook
Input: Bashansible-playbook beacon-import.yml
Add the nginx routes
--- a/templates/nginx/galaxy.j2 +++ b/templates/nginx/galaxy.j2 @@ -115,4 +115,14 @@ server { {{ tiaas_nginx_routes }} + location /beacon { + proxy_pass http://{{ groups['beacon_server'][0] }}:5050; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + } + }
Run the playbook
Input: Bashansible-playbook galaxy.yml
Congratulations, you’ve set up a Beacon v1 for Galaxy! Go check it out at /beacon/
on your server.
Comment: Got lost along the way?If you missed any steps, you can compare against the reference files, or see what changed since the previous tutorial.
If you’re using
git
to track your progress, remember to add your changes and commit with a good commit message!
Comment: Galaxy Admin Training PathThe yearly Galaxy Admin Training follows a specific ordering of tutorials. Use this timeline to help keep track of where you are in Galaxy Admin Training.
Step 1ansible-galaxy Step 2backup-cleanup Step 3customization Step 4tus Step 5cvmfs Step 6apptainer Step 7tool-management Step 8reference-genomes Step 9data-library Step 10dev/bioblend-api Step 11connect-to-compute-cluster Step 12job-destinations Step 13pulsar Step 14celery Step 15gxadmin Step 16reports Step 17monitoring Step 18tiaas Step 19sentry Step 20ftp Step 21beacon