Internals of JET.jl

Abstract interpretation

In order to perform type-level program analysis, JET.jl uses the Compiler.AbstractInterpreter interface and customizes its abstract interpretation by overloading a subset of the Compiler functions, which were originally developed for the Julia compiler's type inference and for optimizations that aim at generating efficient native code for CPU execution.

JET.AbstractAnalyzer overloads a subset of Compiler methods to implement JET's core functionality, including interprocedural propagation of error reports and caching of analysis results. Each plugin analyzer, such as JET.JETAnalyzer, overloads additional Compiler methods to implement its own analysis on top of the AbstractAnalyzer infrastructure.

Most of these overloads use invoke to call the corresponding methods for AbstractInterpreter. The actual AbstractAnalyzer instance is still passed as the interpreter argument, so calls made from within those original methods can dispatch back to analyzer-specific overloads.

How AbstractAnalyzer manages caches

JET.AnalysisResult — Type
AnalysisResult

Container for error reports collected during analysis of a specific InferenceResult.

AbstractAnalyzer manages InferenceErrorReport instances by associating them with their corresponding InferenceResult. Reports found during the analysis of result::InferenceResult can be accessed via get_reports(analyzer, result).

source
JET.CachedAnalysisResult — Type
CachedAnalysisResult

A cached collection of reports associated with an AnalysisResult.

JET copies reports into this container and stacks it directly on the corresponding InferenceResult with CC.stack_analysis_result!. When that inference result is cached as a CodeInstance, the metadata is retained. Global cache lookup recovers the reports by traversing the CodeInstance with CC.traverse_analysis_results; local cache lookup can traverse the InferenceResult directly.

source
JET.AnalysisToken — Type
mutable struct AnalysisToken
    AnalysisToken() = new()
end

An identity token used as the compiler cache owner for an AbstractAnalyzer.

The token's object identity determines which analyzer instances can reuse cached inference and report results. Reuse the same token object only for analyzers with cache-compatible behavior; use distinct objects otherwise.

source

Top-level analysis

JET.virtual_process — Function
virtual_process(interp::ConcreteInterpreter,
                x::Union{AbstractString,JS.SyntaxNode},
                filename::AbstractString,
                config::ToplevelConfig;
                overrideex::Union{Nothing,Expr}=nothing) -> res::VirtualProcessResult

Simulates Julia's top-level execution and returns a VirtualProcessResult. The first fatal top-level error aborts processing, including enclosing modules and files, and is returned as res.toplevel_error_report. Ordinary exceptions caught by interpreted user code do not abort analysis; JET's own interruption signals cannot be caught by user code. Internal invariant and extension contract violations are thrown as exceptions rather than returned as reports.

If x is an AbstractString, this function first parses it into a JS.SyntaxNode. The internal overrideex keyword may be used only when x is a JS.SyntaxNode, and its value must be an Expr with head :toplevel. The expression is analyzed in place of the syntax represented by x. AbstractString input does not accept this override.

The function processes each top-level code block (blk) in the resulting or supplied syntax tree as follows:

  1. If blk is or expands to a :module expression, define the module and recursively analyze its body in that new module.
  2. Otherwise, expand macros and lower blk. Skip literal results; a :thunk result supplies the lowered CodeInfo.
  3. If the context module was virtualized, rewrite self-references from the original module to the virtual module; see fix_self_references!.
  4. Use ConcreteInterpreter to concretely interpret statements that must not be abstracted, such as :method definitions; see partially_interpret!.
  5. Use ToplevelAbstractAnalyzer to analyze the remaining statements abstractly.
Warning

To process top-level code sequentially, as the Julia runtime does, virtual_process splits the input into code blocks and simulates them one at a time. This approach does not track concrete-value dependencies between separately processed blocks. Consequently, partial interpretation of a top-level definition can fail when it needs a global value defined in another block that was abstractly rather than concretely interpreted. Use concretization_patterns to force the relevant blocks to be concretely interpreted. See ToplevelConfig for details.

Cleanup on interruption

When processing aborts, whether because of a fatal error, a concretization timeout, a missing concretization, or an internal error, the interpreter unwinds without running active user catch handlers or finally blocks. Side effects of concrete execution are not rolled back: for example, cd(...) do can leave the working directory changed, and lock(...) do can leave a lock held after analysis exits. Run analysis in a separate Julia process when the state of the current process must be preserved.

source
JET.VirtualProcessResult — Type
res::VirtualProcessResult
  • res.analyzed_files::Dict{String,AnalyzedFileInfo}: analyzed files and the source ranges associated with each module in those files.
  • res.toplevel_error_report::Union{Nothing,ToplevelErrorReport}: the first fatal error during parsing, macro expansion, lowering, or partial concrete interpretation, or nothing on success. A fatal error stops the entire top-level analysis and takes precedence over warning and inference reports.
  • res.toplevel_warning_reports::Vector{ToplevelWarningReport}: nonfatal diagnostics produced by JET, including parser warnings and unsupported features. These are returned alongside inference reports when processing succeeds, rather than also being logged with @warn.
  • res.inference_error_reports::Vector{InferenceErrorReport}: reports of potential errors found by ToplevelAbstractAnalyzer.
  • res.signature_infos::Vector{SignatureInfo}: method signatures collected for analysis from top-level definitions.
  • res.actual2virtual::Union{Actual2Virtual,Nothing}: maps the actual root module to its virtual counterpart, or is nothing when module virtualization is disabled.

The legacy res.toplevel_error_reports property is deprecated and will be removed in a future release. It emits a deprecation warning and returns a fresh vector containing the error, or an empty vector on success. Mutating that vector does not change res. Use res.toplevel_error_report instead.

source
JET.virtualize_module_context — Function
virtualize_module_context(actual::Module)

Return a fresh virtual module that provides access to the bindings of actual.

Virtualization proceeds in two steps:

  1. Use using to make the defined names of actual available in a sandbox module, then export those names from the sandbox.
  2. Use using in the virtual module to make the sandbox's exported names available.

This allows JET to define names in the virtual module even when the same names already exist in actual, without triggering errors such as cannot assign a value to variable ... from module .... It also allows JET to analyze an existing module other than Main without defining analyzed code directly in that module.

TODO

Because this function relies on Base.names, it cannot reproduce names made available through using.

source
JET.ConcreteInterpreter — Type
abstract type ConcreteInterpreter <: JuliaInterpreter.Interpreter end

An interface for concretely interpreting selected top-level statements during virtual_process using JuliaInterpreter.

Subtypes must implement:

  • InterpretationState(interp::T) -> InterpretationState: return the interpreter state.
  • ConcreteInterpreter(interp::T, state::InterpretationState) -> T: return an interpreter of type T associated with state.
  • ToplevelAbstractAnalyzer(interp::T) -> ToplevelAbstractAnalyzer: return the top-level analyzer associated with the interpreter.
source
JET.partially_interpret! — Function
partially_interpret!(interp::ConcreteInterpreter, plan::ConcretizationPlan,
                     mod::Module, src::CodeInfo) -> plan::ConcretizationPlan

Fill plan with one entry per statement of src (see ConcretizationPlan), evaluate plan.concretized using JuliaInterpreter.jl, and return plan.

The selection includes:

  • Top-level definitions, including :method, :struct_type, :abstract_type, and :primitive_type expressions, together with their dependencies.
  • Module-usage expressions, which are directly evaluated so that invalid usages can be reported. Modules loaded by import or using are not recursively analyzed.
  • include calls, which cause top-level analysis to recursively enter the included file.
  • Base.deprecate calls on the module being analyzed, such as those emitted by Base.@deprecate_binding, so that the deprecated bindings are flagged, and the _dep_message_* constants that hold the rest of their deprecation warnings.
  • Untyped global declarations emitted by assignments; ones under control flow are materialized rather than selected.
source

Analysis result

JET.JETToplevelResult — Type
res::JETToplevelResult

Represents the result of analyzing top-level code, including files, packages, and text.

  • res.analyzer::AbstractAnalyzer: the AbstractAnalyzer used for the analysis
  • res.res::VirtualProcessResult: the VirtualProcessResult produced by the analysis
  • res.source::AbstractString: a description of the analysis target that also serves as the identity key of the analysis; e.g. the VS Code integration uses it to replace superseded diagnostics
  • res.jetconfigs: the configurations associated with the analysis

JETToplevelResult implements Base.show methods for JET's supported front ends. Julia's display system selects the appropriate method when rendering the analysis result.

When top-level processing stops at a fatal error, the displayed result shows only that error, under a Top-level analysis failed heading that asks to fix the error and rerun the analysis. Otherwise, top-level warnings are shown before inference reports, and the No errors detected message is shown only when there are neither warnings nor inference reports.

source
JET.JETCallResult — Type
res::JETCallResult

Represents the result of analyzing a function call.

  • res.result::InferenceResult: the InferenceResult produced by the analysis
  • res.analyzer::AbstractAnalyzer: the AbstractAnalyzer used for the analysis
  • res.source::AbstractString: a description of the analysis target that also serves as the identity key of the analysis; e.g. the VS Code integration uses it to replace superseded diagnostics
  • res.jetconfigs: the configurations associated with the analysis

JETCallResult implements Base.show methods for JET's supported front ends. Julia's display system selects the appropriate method when rendering the analysis result.

source

Splitting and filtering reports

Both JETToplevelResult and JETCallResult can be split into individual reports for integration with tools like Cthulhu:

JET.get_reports — Function
reports = JET.get_reports(result::JETCallResult)
reports = JET.get_reports(result::JETToplevelResult)

Return the reports represented by result, one per detected issue.

For a JETCallResult, this returns the inference reports after applying report configuration. For a JETToplevelResult, a fatal top-level error stops analysis and takes precedence: this returns a vector containing only that error, without the warnings or inference reports collected before it. Otherwise, this returns top-level warning reports followed by the configured inference reports, so consumers must not assume that every element is an InferenceErrorReport. Report filtering applies only to inference reports; warnings are retained. Warnings are not also emitted as log messages.

Because the returned reports can include warnings that do not indicate problems in the analyzed code, use has_problems(result) rather than checking whether this vector is empty to decide whether result has problems.

source
JET.reportkey — Function
reportkey(report::InferenceErrorReport)

Return (report.sig.tt, last(report.vst).linfo). The first component is the reported call tuple type, or nothing when no call tuple type is available. The second component is the MethodInstance of the final virtual frame.

For a collection of reports, unique(reportkey, reports) deduplicates reports with the same two components, even when they were reached from different analysis entry points.

source

Report interface

JET.VirtualFrame — Type
VirtualFrame

Stack information representing a virtual execution context:

  • file::Symbol: the source file containing the execution context
  • line::Int: the source line containing the execution context
  • linfo::MethodInstance: the MethodInstance containing the context
  • pc::Int: the index of the statement containing the context in the code analyzed for linfo, or 0 if the context is not a statement, e.g. the definition of linfo

This type is similar to Base.StackTraces.StackFrame, but its context is collected during abstract interpretation rather than runtime execution.

source
JET.VirtualStackTrace — Type
VirtualStackTrace

A vector of VirtualFrames ordered from the analysis entry point to the error point. The first element represents the entry frame, and the last element represents the frame where the error was detected.

source
JET.Signature — Type
Signature

Represents the expression signature associated with an error point.

  • _sig::Vector{Any}: components used to render the expression
  • tt::Union{Type,Nothing}: the call tuple type, when available

Equality compares only the elements of _sig, using ===, and ignores tt. Hashing likewise uses only _sig. print_signature renders a Signature for display.

source
JET.InferenceErrorReport — Type
abstract type InferenceErrorReport end

An interface type for error reports collected during JET's abstract interpretation.

Every concrete subtype provides the following fields:

A subtype may provide additional fields to explain why the error was reported.

source
JET.ToplevelErrorReport — Type
abstract type ToplevelErrorReport end

An interface type for reports that JET collects during top-level processing, including parsing and partial concrete interpretation. All concrete subtypes of ToplevelErrorReport must have the following fields:

  • file::String: the path to the source file associated with the report
  • line::Int: the source line associated with the report

See also: virtual_process, ConcreteInterpreter

source
JET.ParseWarningReport — Type
ParseWarningReport <: ToplevelWarningReport

A nonfatal parser diagnostic, with its source location and source text.

source
JET.UnsupportedFeatureReport — Type
UnsupportedFeatureReport <: ToplevelWarningReport

A limitation for which JET continues analysis with an approximation. The message explains the unsupported feature and how the analyzed code differs from it, and is printed below a header common to all unsupported features. For example, JET reports include(mapexpr, filename) with this report and analyzes the included file without applying mapexpr.

source