Shared: Post-processing query for inline test expectations

[hvitved]

This PR introduces support for writing (and verifying) inline test expectations for .qlref files, for all languages. The implementation makes use of a @kind test-postprocess query, which must be referenced in the relevant .qlref files.

Syntax

The postprocessing query works for queries of kind problem and path-problem, and each query result must have a matching $ Alert comment. It is possible to augment the comment with a query ID, in order to support cases where multiple .qlref tests share the same test code:

var x = ""; // $ Alert[rust/unused-value]
return;
foo();      // $ Alert[rust/unreachable-code]

In the example above, the $ Alert[rust/unused-value] commment is only taken into account in the test for the query with ID rust/unused-value, and vice versa for the $ Alert[rust/unreachable-code] comment.

For path-problem queries, each source and sink must additionally be annotated ($ Source and $ Sink, respectively), except when their location coincides with the location of the alert itself, in which case only $ Alert is needed.

Example:

var queryParam = Request.QueryString["param"]; // $ Source
Write(Html.Raw(queryParam));                   // $ Alert

Morover, it is possible to tag sources with a unique identifier:

var queryParam = Request.QueryString["param"]; // $ Source=source1
Write(Html.Raw(queryParam));                   // $ Alert=source1

In this case, the source and sink must have the same tag in order to be matched.

Output

After enabling the @kind test-postprocess query in a .qlref test, the .expected file will contain failures in a testFailures output relation, similar to when inline test expectations are used in library tests. Unlike library tests, however, the testFailures predicate will only be present in the .expected file when it is non-empty.

Actual query output, such as #select and edges, will still be present in the .expected file, which is deliberate. For data flow queries in particular, having the edges relation explicit in the output has proven very useful in the past for identifying unintended data flow changes (this is why the shared InlineFlowTest library also includes the data flow graph, see for example this library test).

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[owen-mc]

The Go parts LGTM. If I have time before this is merged I will try to start converting more tests to use it, which would be a better test.

[geoffw0]

Minor corrections for Swift. It's really good to have this and we would benefit from using it more widely!

[jketema]

Some small comments (C++ and Swift)

[geoffw0]

Minor comments for Rust as well.

Do we have any way to know for sure that [query] inline expectations will fail when an annotation is missing or incorrect? In other words, is there a test for inline expectations tests themselves?

[hvitved]

Do we have any way to know for sure that [query] inline expectations will fail when an annotation is missing or incorrect? In other words, is there a test for inline expectations tests themselves?

No; I'll add that.

[hvitved]

@geoffw0 : Thanks for the suggestion to add actual tests of the library, which I have now added (in C#). This revealed a bug when using source/sink tags in conjunction with query ID tags, which has been fixed.

[michaelnebel]

C# LGTM 👍

[geoffw0]

LGTM too. 👍

[tausbn]

Python looks good to me. 👍