Contributing

Marvin is an open-source project and the Marvin team welcomes and encourages code contributions. This document describes the general guidelines for contributing to Marvin.

Reporting issues

The easiest (but critical) way to contribute to Marvin is by reporting issues, unexpected behaviours, and desired new features to the development team. To do so, go to the Marvin GitHub page and fill out a new issue. As much as possible, please follow the new issue template. We will try to at least acknowledge the issue in 24-48 hours but if that is not the case feel free to ping us again!

If you have questions about how to use Marvin that are not symptomatic of a problem with the code, please use the SDSS Help Desk instead.

Contributing code

Coding standards

We adhere to the SDSS coding standards. Before contributing any code please make sure you have read the relevant sections and that you set your linter accordingly.

Development process

The Marvin team uses the well-established git procedure of forking, branching, and pull requests. If you are not familiar with it, consider reading some introductory tutorial such as this one.

To develop code for Marvin you will first need a GitHub account. Then go to the Marvin GitHub page and fork it. Develop your changes in a new branch and, when ready, open a pull request. Pleade, make sure the pull request describes all the changes the code introduces. Before it can be merged into master, yoir pull request needs to be approved by at least one of the repository owners.

Testing

All new code must be properly tested and contain enough unittests. No new code will be accepted that does not meet this requirement. Tests must run in, at least, Python 2.7 and 3.6.

All new pull requests will trigger a Travis build. Unfortunaly, Marvin’s CI system is broken at the time and most Travis runs will fail (most likely due to timeouts). We hope to fix this soon but in the meantime it is still useful to check the logs of the build, since your tests probably did run.

For local testing, you will need to set up a PostgreSQL server with a test database called manga, restored from this dump file. Then run the command run_marvin --debug, which will create a local flask HTTP server. You can now go to the python/marvin/tests directory and run pytest, which will run all tests (fair warning, it may take a while!) or pytests <your-file>.

Documentation

If your changes introduce a new feature or change the behaviour of user-facing commands, you will need to provide Sphinx documentation on how to use it. Marvin’s documentation can be found in the docs/ directory. Add your documentation where you think it is appropriate. As part of the pull request review, you may be asked to restructure or move the documentation.

Contributing documentation

Code is only useful if it is well documented! An excellent way of contributing to Marvin is by writing documentation. There are different forms of documentation that you can submit:

  • Sphinx documentation of current features, as described in marvin-controbuting-code-documentation.
  • Plain-text documentation. If you do not want to write Sphinx documentation, you can still send us the text and we will format it appropriately. Send your text in the form of a new issue.
  • Open a new issue pointing out parts of the documentation that are unclear, misleading, or deprecated.
  • Write marvin-tutorials! Tutorials can be in Sphinx or ipyhton notebook format.
  • If you have used Marvin in the classroom and developed activities based on it we would love to hear about it!

Contributing a VAC

Value Added Catalogues (VAC) are an important part of how SDSS and MaNGA data is analysed and distributed. Following SDSS policy, Marvin does not directly support VACs, but it does supply a framework for VAC owners to implement access to their catalogues from Marvin.

At this time, only file-based access is supported (i.e., no querying or remote access is available) but the VACMixIn class provides a convenient way of matching targets with their VAC information and returning it to the user. Very little knowledge of how Marvin internally works is required! The directory marvin/contrib/vacs contains the base code and a list of already implemented VACs that you can use as a template.

Example: HI Follow-up for MaNGA

The MaNGA-HI project is a program to perform follow-up observations of all MaNGA targets to look for HI 21cm using the Greenbank Radio Telescope. This project provides HI spectral data and derived HI properties of MaNGA galaxies. For each observed MaNGA galaxy, there is an HI spectrum FITS file, as well as a summary file of all HI observations. You can find all the files here with the format <version>/mangaHIall.fits for the summary file, and format <version>/spectra/<programid>/fits/mangaHI-<plateifu>.fits. <version> is the version of the HI VAC release, <programid> is an internal identifier for the observing program, and <plateifu> is the plate-IFU of the galaxy. The source code for this VAC’s implementation can be found at marvin/contrib/vacs/mangahi.py:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
# !usr/bin/env python
# -*- coding: utf-8 -*-
#
# Licensed under a 3-clause BSD license.
#
# @Author: Brian Cherinka
# @Date:   2018-10-11 17:51:43
# @Last modified by:   Brian Cherinka
# @Last Modified time: 2018-11-29 17:23:15

from __future__ import print_function, division, absolute_import

import numpy as np
import astropy
import astropy.units as u
import marvin.tools
from marvin.tools.quantities.spectrum import Spectrum
from marvin.utils.general.general import get_drpall_table
from marvin.utils.plot.scatter import plot as scatplot

from .base import VACMixIn


class HIVAC(VACMixIn):
    """Provides access to the MaNGA-HI VAC.

    VAC name: HI

    URL: link

    Description: Returns HI summary data and spectra

    Authors: David Stark and Karen Masters

    """

    # Required parameters
    name = 'mangahi'
    description = 'Returns HI summary data and spectra'
    version = {'MPL-7': 'v1_0_1',
               'DR15': 'v1_0_1'}

    # optional Marvin Tools to attach your vac to
    include = (marvin.tools.cube.Cube,
               marvin.tools.maps.Maps,
               marvin.tools.modelcube.ModelCube)

    # Required method
    def get_data(self, parent_object):

        # get any parameters you need from the parent object
        plateifu = parent_object.plateifu
        release = parent_object.release

        # define the variables to build a unique path to your VAC file
        path_params = {'ver': self.version[release], 'type': 'all',
                       'plateifu': plateifu, 'program': 'GBT16A_095'}

        # get_path returns False if the files do not exist locally
        allfile = self.get_path('mangahisum', path_params=path_params)
        specfile = self.get_path('mangahispectra', path_params=path_params)

        # download the vac from the SAS if it does not already exist locally
        if not allfile:
            allfile = self.download_vac('mangahisum', path_params=path_params)

        # create container for more complex return data
        hidata = HIData(plateifu, allfile=allfile, specfile=specfile)

        # get the spectral data for that row if it exists
        if hidata._indata and not specfile:
            hidata._specfile = self.download_vac('mangahispectra', path_params=path_params)

        return hidata


class HIData(object):
    ''' A customized class to handle more complex data

    This class handles data from both the HI summary file and the
    individual HI spectral files.  Row data from the summary file
    is returned via the `data` property.  Spectral data can be plotted via
    the `plot_spectrum` method.

    '''

    def __init__(self, plateifu, allfile=None, specfile=None):
        self._allfile = allfile
        self._specfile = specfile
        self._plateifu = plateifu
        self._hi_data = self._open_file(allfile)
        self._indata = plateifu in self._hi_data['plateifu']
        self._specdata = None

    def __repr__(self):
        return 'HI({0})'.format(self._plateifu)

    @staticmethod
    def _open_file(hifile):
        return astropy.io.fits.getdata(hifile, 1)

    @property
    def data(self):
        ''' Returns the FITS row data from the mangaHIall summary file '''

        if not self._indata:
            return "No HI data exists for {0}".format(self._plateifu)

        idx = self._hi_data['plateifu'] == self._plateifu
        return self._hi_data[idx]

    def plot_spectrum(self):
        ''' Plot the HI spectrum '''

        if self._specfile:
            if not self._specdata:
                self._specdata = self._open_file(self._specfile)

            vel = self._specdata['VHI'][0]
            flux = self._specdata['FHI'][0]
            spec = Spectrum(flux, unit=u.Jy, wavelength=vel, wavelength_unit=u.km/u.s)
            ax = spec.plot(ylabel='HI\ Flux', xlabel='Velocity', title=self._plateifu, ytrim='minmax')
            return ax
        return None

    def plot_massfraction(self):
        ''' Plot the HI mass fraction '''

        drpall = get_drpall_table()
        drpall.add_index('plateifu')
        subset = drpall.loc[self._hi_data['plateifu']]
        log_stmass = np.log10(subset['nsa_elpetro_mass'])
        diff = self._hi_data['logMHI'] - log_stmass
        fig, axes = scatplot(log_stmass, diff, with_hist=False, ylim=[-5, 5],
                             xlabel=r'log $M_*$', ylabel=r'log $M_{HI}/M_*$')
        return axes[0]


The file itself contains just a subclass of VACMixIn. In the docstring, we make sure to include the name of the VAC, a URL with a description of its contents, and a short description of what the VAC provides and what the class returns.

The global section of the HIVAC class defines :

  • The name of the VAC (this is the name that users will enter to access the VAC from Marvin). This is the only required attribute that we need to override from the parent class.
  • A version dictionary that defines the relationship between Marvin releases (e.g., MPL-7, DR15) and internal MaNGA-HI versions.
  • An include attribute that contains a list of Marvin Tools classes to which this VAC must be added. In this particular case we only want the HI VAC to show in Cube, Maps, and ModelCube. If include is not defined the VAC will be added to all the Marvin Tools classes with the exception of Plate.

get_data is the only method that you need to override from VACMixIn. You will have noted that get_data receives a single, parent_object argument, which is the object (e.g., a Maps instance) that is trying to access the VAC information. You can use it and its attributes to do the matching with the VAC information.

We use sdss_access to download the necessary files, so we need to be sure that the paths to the VAC files are included in the tree. For these particular filetypes the entry in tree looks like:

mangahisum = $MANGA_HI/{ver}/mangaHI{type}.fits
mangahispectra = $MANGA_HI/{ver}/spectra/{program}/fits/mangaHI-{plateifu}.fits

In addition to the tree path names (mangahi, mangahispectra) we need to define a dictionary of path parameters. We use the parent_object to determine the release (and thus the HI version) and the plateifu. Because for a given program and type are fixed, they are manually specified in the VAC.

First, we use get_path to determine whether the file is already present in the local SAS. If that is not the case, we use download_vac to retrieve it.

Once you have the file(s), all that is left is to return the data you want from the FITS files. In most cases you may simply want to return the entire FITS file or a specific row of an extension. In other cases you may want to return something a bit more complex. Since the HI VAC includes for a given galaxy both data from an HI summary file and an HI flux spectrum, we want to return both the summary and spectral data, plus provide a method for plotting the HI spectrum. The best way to do that is by creating a new custom class (HIData) that will handle all the HI data for us, along with any other complexity we wish to add. We can simply return an instance of our custom class in the get_data method.

Now that we have implemented the VAC, let’s make sure it works:

>>> from marvin.tools.maps import Maps
>>> my_map = Maps('7443-12701')
>>> hi_data = my_map.vacs.mangahi
>>> # access the data from the summary file
>>> print(hi_data.data)

    FITS_rec([('7443-12701', '12-98126', 230.5074624, 43.53234133, 6139, '16A-14', 767.4, 1.76, 8.82, -999., -999., -999., -999., -999, -999., -999, -999, -999, -999, -999, -999., -999., -999., -999., -999., -999.)],
             dtype=(numpy.record, [('plateifu', 'S10'), ('mangaid', 'S9'), ('objra', '>f8'), ('objdec', '>f8'), ('vopt', '>i2'), ('session', 'S12'), ('Exp', '>f4'), ('rms', '>f4'), ('logHIlim200kms', '>f4'), ('peak', '>f4'), ('snr', '>f4'), ('FHI', '>f4'), ('logMHI', '>f4'), ('VHI', '>i2'), ('eV', '>f4'), ('WM50', '>i2'), ('WP50', '>i2'), ('WP20', '>i2'), ('W2P50', '>i2'), ('WF50', '>i2'), ('Pr', '>f4'), ('Pl', '>f4'), ('ar', '>f4'), ('br', '>f4'), ('al', '>f4'), ('bl', '>f4')]))

>>> # plot the HI spectrum
>>> hi_data.plot_spectrum()

Writing your own VAC

If you are a VAC owner by now you will have hopefully decided that you want to provide access to it with Marvin. How do you do that?

  • First, make sure you read the Coding standards and Development process sections.
  • For the Marvin repository and clone your fork. From the python/marvin/contrib/vacs directory select an example that looks similar to what you need. Within the vacs/ directory. Duplicate that file and rename it to the name of your VAC.
  • Modify the name of the class that inherits from VACMixIn (important!) and change the name attribute to the name of your VAC. Update the docstring with information about your VAC. Most importantly, make sure the description clearly states what your VAC is returning.
  • To be able to use download_vac the archetype of your VAC’s file path must have been added to sdss_paths.ini. Refer to the tree documentation to learn how to do that.
  • Test you implementation by accessing your VAC as object.vacs.<your-vac-name>.
  • As the VAC owner, you are responsible for making sure your class returns the correct information. We strongly suggest you implement unittests to make sure that is the case. You can see some examples here.
  • Once you are happy with the result, open a new pull request to integrate your changes.
  • If you have any doubt, contact the Marvin team, we will be happy to help you throughout the process.

Classes

marvin.contrib.vacs.base.VACMixIn() MixIn that allows VAC integration in Marvin.

Available VACs

marvin.contrib.vacs.mangahi.HIVAC() Provides access to the MaNGA-HI VAC.