GladTeX should not nest math environments in "\[" "\]"

[belonesox]

Consider following case — math environment in markdown math mode $$:

…
$$
\begin{align*}
	\sum_{j=1}^{n} a_{ij} x_j = b_i, \ i = 1, \ldots, m\  (m < n) \leftarrow Ax = b,\\
	x_j \geqslant 0, \ j = 1, \ldots, n.
\end{align*}
$$
…

VSCode preview / KaTeX works OK with it (see screenshot).
In pandoc JSON we have

…
        {
            "t": "Para",
            "c": [
                {
                    "t": "Math",
                    "c": [
                        {
                            "t": "DisplayMath"
                        },
                        "\n\\begin{align*}\n\\sum_{j=1}^{n} a_{ij} x_j = b_i, \\ i = 1, \\ldots, m\\  (m < n) \\leftarrow Ax = b,\\\\\nx_j \\geqslant 0, \\ j = 1, \\ldots, n.\n\\end{align*}\n"
                    ]
                }
            ]
        },
…

Unfortunately, GladTeX nest all this in "[" "]" (and latex compilation failed, because "Erroneous nesting of equation structures"`)

    def _format_document(self, preamble):
        """Return a formatted LaTeX document with the specified formula
        embedded."""
        opening, closing = None, None
        if self.__maths_env:
            opening = '\\begin{%s}' % self.__maths_env
            closing = '\\end{%s}' % self.__maths_env
        else:
            # determine characters with which to surround the formula
            opening = '\\[' if self.__displaymath else '\\('
            closing = '\\]' if self.__displaymath else '\\)'

Probably there is planned some detection of mathenvs (set_latex_environment), but the function is not used still.

May be for displaymath node just check that the node text starting with \begin{ and then not wrap such nodes with anything? (without detecting name of the environment)

If this OK, I can patch it...

[humenda]

Pardon, I would need the actual code, as I cannot see images. Please paste your examples here.

[belonesox]

OK, I updated description (the only screenshot I left to show, that VSCode/KaTeX works with it…).

[belonesox]

OK with description of the issue?

[humenda]

OK with description of the issue?

Very helpful indeed. I don't have the time ATM to dig deeper in the code, but it looks indeed unfortunate. If you would have the time to look into it, I would appreciate it. Thanks!