Marvin’s Change Log
[2.3.0] - 2018/?/?
Breaking changes
- Removed
Bin
class. Bin information is now available on a per-quantity basis (#109). See What’s new? and documentation for details.
- Syntax on the inputs to the
Query
and Results
tools has been changed.
- DAP spaxel queries have been disabled due to performance issues. We expect to bring them back soon. Metadata queries (those querying the NSA or DAPall tables) are still available.
getSpaxel
now only loads the quantities from the parent object (that means that, for example, Maps.getSpaxel
only loads Maps
properties by default). Additional quantities can be loaded using load
.
getSpaxel
accepted arguments have been changed to cube
, maps
, and modelcube
. The formerly accepted arguments (drp
, properties
, model(s)
) now raise a deprecation error.
Added
- Added cheatsheet to docs.
- New Web authentication using Flask-Login
- New API authentication using Flask-JWT-Extended
- Adds MPL-7 / DR15 datamodel
- New config.access attribute indicating public or collab access
- New config.login method to handle token-based login
- New marvin.yml config file for customization of configuration options
- Adds User table into the history schema of mangadb. Tracks user logins.
Map
has a new method getSpaxel
to retrieve an Spaxel
using the parent Maps
.
- New configuration option in
marvin.yml
, default_release
, to set the release to be used when Marvin gets imported (#463).
- Applying a NumPy ufunc (except
np.log10()
) raises NotImplementedError
because ivar propagation is not implemented yet.
- New
Marvin Image
Tool to load optical images using the MMA (#22)
- New
Bundle
and Cutout
utility classes
- New
MMAMixIn
for providing multi-modal data access
qual_flag
and pixmask_flag
are now stored in the datamodel (#479, #482).
Query
tool now accepts a new targets
and quality
keyword argument which enables querying on target or quality maskbit labels. (#485)
- Added a new API route for streaming large query results. This uses a generator to stream large results back to the client to minimize memory usage and bulk responses.
Changed
- Integrated datamodel plotting params into actual datamodel structures
- Moved netrc checks into the Brain
- Expanded sasurl into public and collab urls
- Changes personal emails to sdss helpdesk email in web
- Added rawsql and return_params columns to history.query table in mangadb
- Extra keyword arguments passed to
Spectrum.plot
are now forwarded to Axes.plot
.
- Tools (e.g.,
Cube
, Maps
) can now be accessed from the marvin
namespace (e.g., marvin.tools.Cube
or marvin.tools.cube.Cube
).
- Map plotting
ax_setup()
function is now hidden.
- Moved
yanny.py
to extern/
and added a readme file for the external packages (#468).
plot
now only masks part of the spectrum that have the DONOTUSE
maskbit set (#455).
pixmask
is now available for all quantities (except AnalysisProprty
). The property masked
now uses the bit DONOTUSE
to determine what values must be masked out (#462).
- Raises error when applying
inst_sigma_correction
on stellar_sigma
MPL-6 maps. Applies correction to stellar_sigma and emline_sigma for web maps with added ‘Corrected’ title (#478)
- Removes targeting bits from
Spaxel
and Bin
(#465).
- The name of the channel is now shown in the
Property
description (#424).
- Replaced inconsistent parameter
model
in getSpaxel
. Use models
instead.
- MarvinError now accepts an optional
ignore_git
keyword to locally turn off the git addition to the message
- Using the
return_all
keyword in Query
or getAll
in Results
now calls the streaming API route instead.
- When
Cube
or ModelCube
are instantiated from file, gunzip the file to a temporary location to speed up subsequent access (#525).
- Convert MMA warnings to debug messages (:issues:`580`).
Fixed
- Issue #421 - query returning spaxel props returns wrong total count
- Bugfix - Python 3 xrange syntax bug in buildImageDict
Bin._create_spaxels
instantiating spaxels with the wrong (i,j)
values for the bin. The (i, j)
values from the binid
map were being swapped twice before sending them to SpaxelBase
(#457).
- A bug in the calculation of the deredden inverse variance in a
DataCube
.
- Issue with setting drpall path on initial import/set of release before tree has been planted
- Issue #456 - spectrum web display shows incorrect RA, Dec
- Issue #422 - ensuring config auto checks access to netrc
- Issue #423 - adding marvin.yml documentation
- Issue #431 - adding login documentation
- Issue #151 - adding web spectrum tooltips
- Issue #548 - failed to retrieve ModelCube extension in remote mode
- Fixed typo by in method name
Spectrum.derredden -> Spectrum.deredden
.
- Fixed #305 - adding ivar propogation for np.log10(Map)
- A bug when explicitly returning default parameters in a query (#484)
- Fixed #510 - fixes incorrect conversion to sky coordinates in map plotting.
- Fixed #563 - fail retrieving Query datamodels in Python 3.6+.
- Fixes bug with sasurl not properly being set to api.sdss.org on initial import
- Incorrect setting of the default bintype to download from web (#531).
- Fixed #536, #537, #538. Added modelcube to downloadList.
- Incorrect mismatch warning between MPL-7 and DR15 (#495).
- Incorrect handling of maskbits when the mask does not contain any of the bits in the list (#507).
- Fixed #534 - flipped axes in NSA scatterplot when plotting absmag colors
- Fixed #559 - bug in check_marvin when marvindb is None
- Fixed #579 - bug in MMA with marvindb preventing files from opening
- Fixed #543, #552, #553 - bugs with various Query handlings
- Fixed #575 - cannot access maps due to bug in login and authentication in Interaction class
- Fixed #539 - print downloadList target directory
- Fixed #566 - made error message for web query with non-unique parameters name more specific
Refactored
- Moved
marvin.core.core
to marvin.tools.core
and split the mixins into marvin.tools.mixins
.
- Reimplemented
getAperture
as a mixin using photutils apertures (#3, #315).
- Reimplemented
RSS
as a list of RSSFiber
objects (#27, #504).
- Moved pieces of MarvinToolsClass into
marvin.tools.mixins
.
- Reimplemented
Query
to remove local query dependencies from remote mode usage.
[2.2.5] - 2018/04/26
Changed
- Galaxy Web page spaxel loading to be robust when no modelspaxels are present in the database.
[2.2.4] - 2018/04/04
Fixed
- Issue #400: SII in BPT diagram should use sum of 6717 and 6732.
[2.2.3] - 2018/03/20
Added
- Added tests for
emline_gflux_ha_6564
and fixed values in galaxy_test_data.
Removed
- The banner that showed up in Safari has been removed since most versions should now work properly.
[2.2.2] - 2018/02/25
Fixed
- MPL-6 issue with all H-alpha extensions mapped to NII instead. Indexing issue in MPL-6 datamodel.
- MPL-6 issue with elliptical coordinate extensions; missing R/Reff channel in MPL-6 datamodel.
- Issue #324
- Issue #325
- Issue #326
- Issue #327
- Issue #330
- Issue #333
- Issue #335
- Issue #336
- Issue #343
- Issue #351
- Issue #353
- Issue #357
- Issue #358
- Issue #360
- Issue #363
- Issue #373
[2.2.1] - 2018/01/12
Fixed
- bugfix in MPL-6 datamodel for gew OII lines
[2.2.0] - 2018/01/12
Added
- Added
Maskbit
class for easy conversion between mask values, bits, and
labels.
- Better BPT documentation, in particular in the
Modifying the plot
section.
- A hack function
marvin.utils.plot.utils.bind_to_figure()
that
replicate the contents of a matplotlib axes in another figure.
- New scatter and histogram plotting utility functions
- Integrated scatter and histogram plotting into query Results
- New methods for easier query Results handling
- New Pythonic DRP, DAP, and Query DataModels
- Access to DAPall data
Changed
- Issue #190:
Maps.get_bpt()
and
marvin.utils.dap.bpt.bpt_kewley06()
now also return a list of
axes. Each axes contains a method pointing to the
marvin.utils.plot.utils.bind_to_figure()
function, for easily
transfer the axes to a new figure.
- All Cubes, Maps, and Modelcubes use Astropy Quantities
- Refactored to the Bin class
- Bin and Spaxel are now subclassed from SpaxelBase
[2.1.4] - 2017/08/02
Added
- Added new query_params object, for easier navigation of available
query parameters. Added new tests.
- Added a new guided query builder using Jquery Query Builder to the
Search page
- Added a View Galaxies link on the web results to view postage stamps
of the galaxies in the results
- Added Route Rate Limiting. Adopts a limit of 200/min for all api
routes and 60/minute for query api calls and web searches
Changed
- Changed call signature for
:meth:
marvin.utils.plot.map.no_coverage_mask
(removed value
arg because unused, added None
as default value ivar
(None
), and re-ordered args and kwargs).
- Changed call signature for
:meth:
marvin.utils.plot.map.bad_data_mask
(removed value
arg
because unused).
- Changed the Marvin web search page to use the new query_params and
parameter grouping. Removed the autocomplete input box.
- Updated the documentation on query and query_params.
- Modified Guided Search operator options to remove options that could
not be parsed by SQLA boolean_search
- Refactored the web settings, route registration, extensions to enable
extensibility
- Issue #282: Improvements to “Go to CAS” link. Changed to Go To
SkyServer and updated link to public up-to-date link
Fixed
- Issue #102: problem with urllib package when attempting to retrieve
the Marvin URLMap
- Issue #93: safari browser does not play well with marvin
- Issue #155: Contrails in Web Map
- Issue #174: sdss_access may not be completely python 3 compatible
- Issue #196: Bin not loading from local sas
- Issue #207: Get Maps in MapSpecView of Galaxy page sometimes fails to
return selected maps
- Issue #210: pip upgrade may not install new things as fresh install
- Issue #209: marvin version from pip install is incorrect
- Issue #268: Cube flux from file error
- Issue #85: Python does not start in Python 3
- Issue #273: ha.value bug
- Issue #277: Ticks for log normalized colorbar
- Issue #275: logger crashes on warning when other loggers try to log
- Issue #258: 422 Invalid Parameters
- Issue #271: Problem in dowloading image.
- Issue #97: sqlalchemy-boolean-search not found when installed from
pip source
- Issue #227: Marvin installation in python 3.6 (update setuptools to
36)
- Issue #262: problem with marvin update
- Issue #270: BPT array sizing not compatible
- Issue #88: Deployment at Utah requires automatisation
- Issue #234: Add (and use) functions to the datamodel to determine
plotting parameters
- Issue #278: marvin_test_if decorator breaks in python 2.7
- Issue #274: cube slicing to get a spaxel fails with maps error
- Issue #39: implement more complete testing framework
- Issue #242: Result object representation error with 0 query results
- Issue #159: Marvin issues multiple warnings in PY3
- Issue #149: Improve integrated flux maps display in web
[2.1.3] - 2017/05/18
Added
- Issue #204: added elpetro_absmag colours to mangaSampleDB models.
- Issue #253: Plotting tutorial.
- Issue #223: Easy multi-panel map plotting (with correctly placed
colorbars).
- Issue #232 and Issue #251: Uses matplotlib style sheets context
managers for plotting (map, spectrum, and BPT) and restores previous
defaults before methods finish.
- Issue #189: Map plotting accepts user-defined value, ivar, and/or
mask (including BPT masks).
- Issue #252: Quantile clipping for properties other than velocity,
sigma, or flux in web.
- Added
utils.plot.map
doc page.
- Added
tools.map
doc page.
Changed
- Issue #243: inverted
__getitem__
behaviour for
Cube/Maps/ModelCube and fixed tests.
- Modified Flask Profiler File to always point to
$MARVIN_DIR/flask_profiler.sql
- Issue #241: Moved map plotting methods from tools/map to
utils/plot/map
- Issue #229 and Issue #231: Switch to new gray/hatching scheme (in
tools and web):
- gray: spaxels with NOCOV.
- hatched: spaxels with bad data (UNRELIABLE and DONOTUSE) or S/N
below some minimum value.
- colored: good data.
- Issue #238: Move plot defaults to datamodel (i.e., bitmasks,
colormaps, percentile clips, symmetric, minimum SNR).
- Issue #206: SNR minimum to None (effectively 0) for velocity maps so
that they aren’t hatched near the zero velocity contour.
- Simplified default colormap name to “linearlab.”
- Decreased map plot title font size in web so that it does not run
onto second line and overlap plot.
Fixed
- Interactive prompt for username in sdss_access now works for Python
3.
- Fixed #195: The data file for the default colormap for
Map.plot()
(“linear_Lab”) is now included in pip version of Marvin and does not
throw invalid FileNotFoundError
if the data file is missing.
- Fixed #143: prevents access mode to go in to remote if filename is
present.
- Fixed #213: shortcuts are now only applied on full words, to avoid
blind replacements.
- Fixed #206: no longer masks spaxels close to zero velocity contour in
web and tools map plots
- Fixed #229: corrects web bitmask parsing for map plots
- Fixed #231: hatch regions within IFU but without data in map plots
- Fixed #255: Lean tutorial code cells did not work with the ipython
directive, so they now use the python directive.
- Highcharts draggable legend cdn.
Removed
- Issue #232 and Issue #251: Automatic setting of matplotlib style
sheets via seaborn import or
plt.style.use()
.
[2.1.2] - 2017/03/17
Added
- API and Web argument validation using webargs and marshmallow. If
parameters invalid, returns 422 status.
Changed
- Per Issue #186: Switched to using the elpetro version of stellar
mass, absolute magnitude i-band, and i-band mass-to-light ratio for
NSA web display, from sersic values. (elpetro_logmass,
elpetro_absmag_i, elpetro_mtol_i)
- Issue #188: deprecated snr in favour of snr_min for get_bpt. snr can
still be used.
- Issue #187: Renamed NSA Display tab in web to Galaxy Properties.
Added a link to the NASA-Sloan Atlas catalogue to the table title.
- Moved our documentation to readthedocs for version control. Updated
all Marvin web documenation links to point to readthedocs.
Fixed
- A bug in the calculation of the composite mask for BPT.
- Issue #179: Fixed a python 2/3 exception error compatibility with the
2.1 release.
[2.1.1] - 2017/02/18
Added
- Added query runtime output in search page html. And a warning if
query is larger than 20 seconds.
Changed
- Removed the python 3 raise Exception in the check_marvin bin
- Reverted the api/query return output from jsonify back to json.dumps
- This is an issue with python 2.7.3 namedtuple vs 2.7.11+
Fixed
- Issue #181: web display of maps were inverted; changed to xyz[jj, ii,
val] in heatmap.js
- Added more code to handle MarvinSentry exceptions to fix #179.
[2.1.0] - 2017/02/16
Added
- Restructured documentation index page.
- Improved installation documentation:
- Removed old installation text
- Added section on marvin SDSS dependencies and SAS_BASE_DIR
- Added section for FAQ about installation
- Added web browser cache issue into FAQ
- Added traceback info in the API calls
- Added traceback attribute in Brain config
- Added hidden _traceback attribute in Marvin config
- Only implemented in two Query API calls at the moment
- Added a few tests for traceback
- see usage in cube_query in marvin/api/query.py
- Added the Ha_to_Hb ratio the DAP ModelClasses for querying
- Added new script to perform somce basic system, os, and Marvin
checks: bin/check_marvin
- Added an alert banner when the user is using Safari. See #94.
- Issue #122: added ra/dec to spaxel
- Issue #145: Limited the number of query parameters in the web
- Added more tests to Results for sorting, paging, and getting subsets
- Added kwargs input for Spaxel when using Result.convertToTool
- Added automatic Sentry error logging #147 into MarvinError, and
Sentry in Flask for production mode
- Added custom error handlers for the web, with potential user feedback
form
- Added Sentry tool for grabbing and displaying Sentry statistics
- Added text to MarvinError with a Github Issues link and description
of how to submit and issue
- Added Results option to save to CSV
- Added new parameters in Marvin Config to turn off Sentry error
handling and Github Issue message
- Added Python example code for getting a spectrum in galaxy page of
web.
- Added new test for image utilities getRandomImages, getImagesByPlate,
getImagesByList
- Added new documentation on Image Utilities
- Added new image utility function showImage, which displays images
from your local SAS
- Added the Kewley+06 implementation of the BPT classification as
Maps.get_bpt()
- Added quick access to the NSA information for a Cube/Maps either from
mangaSampleDB or drpall.
Changed
- When marvin is running from source (not dist),
marvin.__version__
is dev
.
- Removed the cleanUpQueries method to assess db stability
- Switched dogpile.cache from using a file to python-memcached
- Syntax changes and bug fixes to get Marvin Web working when Marvin
run on 3.5
- Got Queries and Results working in 3.5
- Changed all convertToTool options in Results from mangaid to plateifu
- Added release explicitly into api query routes
- Modified the decision tree in query to throw an error in local mode
- Modified convertToTool to accept a mode keyword
- Modifed the MarvinError for optional Sentry exception catching, and
github issue inclusion
- Updated all Marvin tests to turn off Sentry exception catching and
the github message
- Updated some of the Tools Snippets on the web
- Overhauled Map plotting
- uses DAP bitmasks (NOVALUE, BADVALUE, MATHERROR, BADFIT, and
DONOTUSE)
- adds percentile and sigma clipping
- adds hatching for regions with data (i.e., a spectrum) but no
measurement by the DAP
- adds Linear Lab color map
- adds option for logarithmic colorbar
- adds option to use sky coordinates
- adds map property name as title
- makes plot square
- sets plotting defaults:
- cmap is linear_Lab (sequential)
- cmap is RdBu_r (diverging) for velocity plots (Note: this is
reversed from the sense of the default coolwarm colormap in
v2.0—red for positive velocities and blue for negative
velocities)
- cmap is inferno (sequential) for sigma plots
- clips at 5th and 95th percentiles
- clips at 10th and 90th percentiles for velocity and sigma plots
- velocity plots are symmetric about 0
- uses DAP bitmasks NOVALUE, BADVALUE, MATHERROR, BADFIT, and
DONOTUSE
- also masks spaxels with ivar=0
- minimum SNR is 1
- Changed Marvin Plate path back to the standard MarvinToolsClass use
- Made sdss_access somewhat more Python 3 compatible
- Modified the image utilities to return local paths in local/remote
modes and url paths when as_url is True
- downloadList utility function now downloads images
- updated the limit-as parameter in the uwsgi ini file to 4096 mb from
1024 mb for production environment
Fixed
- Issue #115: drpall does not get updated when a tool sets a custom
release.
- Issue #107: missing os library under save function of Map class
- Issue #117: hybrid colours were incorrect as they were being derived
from petroth50_el.
- Issue #119: test_get_spaxel_no_db fails
- Issue #121: bugfix with misspelled word in downloadList utility
function
- Issue #105: query results convertToTool not robust when null/default
parameters not present
- Issue #136: BinTest errors when nose2 run in py3.5 and marvin server
in 3.5
- Issue #137: PIL should work in py2.7 and py3.5
- Issue #172: broken mode=auto in image utilities
- Issue #158: version discrepancy in setup.py
[2.0.9] - 2016/11/19
Added
- Docs now use
marvin.__version__
.
Fixed
- Fixed #100, #103: problem with getMap for properties without ivar.
- Fixed #101: problem with marvin query.
[2.0.6] - 2016/11/17
Fixed
- Bug in Queries with dap query check running in remote mode. Param
form is empty.
[2.0.5] - 2016/11/17
Added
- Added netrc configuration to installation documentation.
- Added netrc check on init.
Fixed
- Added mask to model spaxel.
- Bug in Cube tool when a galaxy loaded from db does not have NSA info;
no failure with redshift
- Two bugs in index.py on KeyErrors: Sentry issues 181369719,181012809
- Bug on plate web page preventing meta-data from rendering
- Fixed installation in Python 3.
- Fixed long_description in setup.py to work with PyPI.
- Fixed a problem that made marvin always use the modules in extern
[The dark ages] - multiple versions not logged.
[1.90.0]
Changed
- Full refactoring of Marvin 1.0
- Refactored web
Added
- Marvin Tools
- Queries (only global properties, for now)
- Point-and-click for marvin-web
- RESTful API
- Many more changes
Fixed
- Issue albireox/marvin#2: Change how matplotlib gets imported.