Output mimetype for processes not set correctly

[tomLandry]

When running a job through Pyramid, the output mimetype is fetched from the WPS complex output and displayed correctly. This is the case for most NetCDF outputs. Querying the mimetype for the file through its URL does not produce the same result. For instance, for this job:
https://mouflon.dkrz.de/monitor/details/e7977e440f854d34826ef6f1957bfe2a/outputs

When querying the mimetype for the output with CURL, we obtain "application/octet-stream" instead of "application/x-netcdf":

H:>curl -I -L http://mouflon.dkrz.de:8090/wpsoutputs/flyingpigeon/ncout-241b8e4
4-9a16-11e6-ae00-1f4eae89de9f.nc
HTTP/1.1 200 OK
Server: nginx/1.10.1
Date: Fri, 25 Nov 2016 18:21:10 GMT
Content-Type: application/octet-stream
Content-Length: 7628716
Last-Modified: Mon, 24 Oct 2016 18:18:19 GMT
Connection: keep-alive
ETag: "580e506b-7467ac"
Accept-Ranges: bytes

[tomLandry]

Adding application/x-netcdf to mime.types file allows the file to be correctly queried for its mimetype. Cannot confirm that this is the source of the problem we experience at the moment, where an input NetCDF file coming from wpsoutput cannot be fed to another WPS process in input.

[cehbrecht]

i've looked into the error message in the log:

PyWPS [2016-11-25 19:39:57,599] INFO: Status [processpaused]: Getting input resource of process subset_continents
PyWPS [2016-11-25 19:41:00,953] INFO: Status [processfailed]: 'exceptions.IOError' object has no attribute 'replace'
PyWPS [2016-11-25 19:41:01,053] INFO: Status [processfailed]: 'mimetype'
PyWPS [2016-11-25 19:41:01,097] DEBUG: [Errno 2] No such file or directory: '/opt/birdhouse/var/lib/pywps/tmp/flyingpigeon/state-pywps-8df01c5e-b33e-11e6-95ca-868dacf6ed58'
Traceback (most recent call last):
  File "lib/python2.7/site-packages/pywps-3.2.6-py2.7.egg/pywps/Wps/Execute/__init__.py", line 386, in __init__
  File "lib/python2.7/site-packages/pywps-3.2.6-py2.7.egg/pywps/Wps/Execute/__init__.py", line 537, in consolidateInputs
  File "/opt/anaconda/envs/flyingpigeon/lib/python2.7/site-packages/pywps/Wps/Execute/../../../pywps/Process/InAndOutputs.py", line 405, in setValue
    self.downloadData2(input["value"])
  File "/opt/anaconda/envs/flyingpigeon/lib/python2.7/site-packages/pywps/Wps/Execute/../../../pywps/Process/InAndOutputs.py", line 525, in downloadData2
    return self.downloadData(url)
  File "/opt/anaconda/envs/flyingpigeon/lib/python2.7/site-packages/pywps/Wps/Execute/../../../pywps/Process/InAndOutputs.py", line 555, in downloadData
    self.onProblem("NoApplicableCode",e)
  File "lib/python2.7/site-packages/pywps-3.2.6-py2.7.egg/pywps/Wps/Execute/__init__.py", line 648, in onInputProblem
  File "/opt/anaconda/envs/flyingpigeon/lib/python2.7/site-packages/pywps/Wps/Execute/../../../pywps/Exceptions.py", line 108, in __init__
    self.value = xml_text_escape(value)
  File "/opt/anaconda/envs/flyingpigeon/lib/python2.7/xml/sax/saxutils.py", line 32, in escape
    data = data.replace("&", "&")
AttributeError: 'exceptions.IOError' object has no attribute 'replace'
Traceback (most recent call last):
  File "lib/python2.7/site-packages/pywps-3.2.6-py2.7.egg/pywps/Wps/Execute/__init__.py", line 415, in __init__
  File "lib/python2.7/site-packages/pywps-3.2.6-py2.7.egg/pywps/Wps/Execute/__init__.py", line 917, in outputDefinitions
  File "lib/python2.7/site-packages/pywps-3.2.6-py2.7.egg/pywps/Wps/Execute/__init__.py", line 934, in _lineageComplexOutput
KeyError: 'mimetype'

I have enabled lineage info (listing inputs and outputs in wps result document) and it failed to get the mimetype keyword when listing the complex outputs.

I'm not aware that pywps 3.x does any validation of the input ... this is introduced with pywps 4.x. In pywps 3.x when you provide a url it will just download it ... no matter what :)

The wps output files are living on an nginx file server.

I haven't got deeper into this ... but two things we can do:

• update the mimetypes of nginx (could be made configurable in the nginx recipe)
• fix the wps lineage info when mimetype is "missing"

[cehbrecht]

by accident i've entered a broken url (http missing) and i got the "mimetype" exception. Maybe it is a follow up exception when a url is not reachable during the download?