Better compile error output when running CodeQL unit tests

[aeisenberg]

When running a code QL test that has a compile error, the results will look like this:

Executing 1 tests in 1 directories.
Extracting test database in /Users/andrew.eisenberg/Eclipse/git-repos/github/vscode-codeql-starter/ql/python/ql/test/experimental/dataflow/basic.
Compiling queries in /Users/andrew.eisenberg/Eclipse/git-repos/github/vscode-codeql-starter/ql/python/ql/test/experimental/dataflow/basic.
[1/1 comp 2s] FAILED(COMPILATION) /Users/andrew.eisenberg/Eclipse/git-repos/github/vscode-codeql-starter/ql/python/ql/test/experimental/dataflow/basic/callGraph.ql
0 tests passed; 1 tests failed:
  FAILED: /Users/andrew.eisenberg/Eclipse/git-repos/github/vscode-codeql-starter/ql/python/ql/test/experimental/dataflow/basic/callGraph.ql

However, there is no output indicating where the compile error occurred, what line number, etc. It would be nice if we got a message like this:

[1/1 comp 2s] FAILED(COMPILATION) /Users/andrew.eisenberg/Eclipse/git-repos/github/vscode-codeql-starter/ql/python/ql/test/experimental/dataflow/basic/callGraph.ql:7:19

Notice the :7:19 at the end. It indicates that the file should be opened on the 7th line, 19th character. This will make it easier to work with errors in unit test code.

It's likely that this can only be implemented in the CLI, so I'll also raise an issue there.

[hmakholm]

Compilation errors already appear in the codeql test run output as a difference between "expected" and "actual" output -- the errors count as "output" for the purpose of this comparison. This is used in practice in several of the nolang tests, which produce compilation errors purposefully and check that they are as expected.

Here's a test run (after mutilating the .ql file by removing the last line):

$ codeql test run ql/python/ql/test/experimental/dataflow/basic/callGra*.ql
Executing 3 tests in 1 directories.
Extracting test database in /home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic.
Compiling queries in /home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic.
--- expected
+++ actual
@@ -1,3 +1,1 @@
-| test.py:4:10:4:10 | ControlFlowNode for z | test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() |
-| test.py:7:19:7:19 | ControlFlowNode for a | test.py:1:19:1:19 | SSA variable x |
-| test.py:7:19:7:19 | ControlFlowNode for a | test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() |
+ERROR: unexpected input '<EOF>' expecting one of: 'and', 'implies', 'in', 'instanceof', 'or', '<', '<=', '=', '>', '>=', '-', ',', '!=', '/', '.', ')', '*', '%', '+' (callGraph.ql:7,1-2)
[1/3 comp 1.5s] FAILED(COMPILATION) /home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraph.ql
Executing tests in /home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic.
[2/3 eval 1.5s] PASSED /home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraphSinks.ql
[3/3 eval 197ms] PASSED /home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraphSources.ql
2 tests passed; 1 tests failed:
  FAILED: /home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraph.ql
$ 

With JSON output (which I hope the extension is using), you get the error location both as as structured location and as part of the comparison failure:

$ codeql test run ql/python/ql/test/experimental/dataflow/basic/callGra*.ql --format json -q
[
{
  "test" : "/home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraph.ql",
  "pass" : false,
  "failureStage" : "COMPILATION",
  "messages" : [
    {
      "severity" : "ERROR",
      "message" : "unexpected input '<EOF>' expecting one of: 'and', 'implies', 'in', 'instanceof', 'or', '<', '<=', '=', '>', '>=', '-', ',', '!=', '/', '.', ')', '*', '%', '+'",
      "position" : {
        "fileName" : "/home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraph.ql",
        "line" : 7,
        "column" : 1,
        "endLine" : 7,
        "endColumn" : 2
      }
    }
  ],
  "compilationMs" : 1460,
  "expected" : "/home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraph.expected",
  "actual" : "/home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraph.actual",
  "diff" : [
    "--- expected",
    "+++ actual",
    "@@ -1,3 +1,1 @@",
    "-| test.py:4:10:4:10 | ControlFlowNode for z | test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() |",
    "-| test.py:7:19:7:19 | ControlFlowNode for a | test.py:1:19:1:19 | SSA variable x |",
    "-| test.py:7:19:7:19 | ControlFlowNode for a | test.py:7:5:7:20 | ControlFlowNode for obfuscated_id() |",
    "+ERROR: unexpected input '<EOF>' expecting one of: 'and', 'implies', 'in', 'instanceof', 'or', '<', '<=', '=', '>', '>=', '-', ',', '!=', '/', '.', ')', '*', '%', '+' (callGraph.ql:7,1-2)"
  ]
},
{
  "test" : "/home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraphSinks.ql",
  "pass" : true,
  "messages" : [ ],
  "evaluationMs" : 1460,
  "expected" : "/home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraphSinks.expected"
},
{
  "test" : "/home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraphSources.ql",
  "pass" : true,
  "messages" : [ ],
  "evaluationMs" : 211,
  "expected" : "/home/henning/work/code/ql/python/ql/test/experimental/dataflow/basic/callGraphSources.expected"
}]
$ 

[aeisenberg]

Looks like the TypeScript types are not matching with what the CLI is sending back. I think that's just an extension issue and I can easily fix.

[aeisenberg]

This is already fixed.