How to concatenate different datasets?

[dhohn]

Hi all,

I have the following special case:

  - {dataset: GISS-E2-1-G,    project: CMIP6, exp: 1pctCO2,               ensemble: r1i1p1f1, grid: gn, timerange: 1850/1999}
  - {dataset: GISS-E2-1-G-CC, project: CMIP6, exp: esm-1pct-brch-1000PgC, ensemble: r1i1p1f1, grid: gn, timerange: 1920/2099}
  # the following doesnt work!
  - {dataset: [GISS-E2-1-G, GISS-E2-1-G-CC], project: CMIP6, exp: [1pctCO2, esm-1pct-brch-1000PgC], ensemble: r1i1p1f1, grid: gn, timerange: 1850/2099}

where there is an experiment (esm-1pct-brch-1000PgC) that branched off from a parent (1pctCO2) at some point. Other modelling groups have kept the model name the same. But here GISS decided to change it (add the "-CC") at the end. They really belong together. So how can I concatenate them in a way that lets me reuse the preprocessing that I've already set up.

Maybe some aliasing for compatible model names?

[dhohn]

One workaround is symlinking the data. But you need to create links for both the directory structure as well as the data files themselves as they contain the model name. So works in a pinch but not really sustainable.

[bouweandela]

The easiest solution would probably be to concatenate in the diagnostic script.

[valeriupredoi]

you can try using the esmvalcore.preprocessor._io.concatenate concatenator (just made up this word) - that's pretty bullet-proof, and if you run it ahead of the actual esmvaltool run itself you can get a composite dataset that you can then write to disk and use; it's a bit bulky but might be a bit better than concatenating in the diag if you want to reduce a lot of data through preprocessing first