VarNames and optics

VarName{sym}(optic=identity)

A variable identifier for a symbol sym and optic optic. sym refers to the name of the top-level Julia variable, while optic allows one to specify a particular property or index inside that variable.

VarNames can be manually constructed using the VarName{sym}(optic) constructor, or from an optic expression through the @varname convenience macro.

@varname(expr, concretize=false)

Create a VarName given an expression expr representing a variable or part of it.

Basic examples

In general, VarNames must have a top-level symbol representing the identifier itself, and can then have any number of property accesses or indexing operations chained to it.

julia> @varname(x)
x

julia> @varname(x.a.b.c)
x.a.b.c

julia> @varname(x[1][2][3])
x[1][2][3]

julia> @varname(x.a[1:3].b[2])
x.a[1:3].b[2]

Dynamic indices

Some expressions may involve dynamic indices, specifically, begin, end. These indices cannot be resolved, or 'concretized', until the value being indexed into is known. By default, @varname(...) will not automatically concretize these expressions, and thus the resulting VarName will contain markers for these.

Note that colons are not considered dynamic.

julia> vn = @varname(x[end])
x[DynamicIndex(end)]

julia> vn = @varname(x[1, end-1])
x[1, DynamicIndex(end - 1)]

You can detect whether a VarName contains any dynamic indices using is_dynamic:

julia> vn = @varname(x[1, end-1]); AbstractPPL.is_dynamic(vn)
true

To concretize such expressions, you can call concretize on the resulting VarName. After concretization, the resulting VarName will no longer be dynamic.

julia> x = randn(2, 3);

julia> vn = @varname(x[1, end-1]); vn2 = AbstractPPL.concretize(vn, x)
x[1, 2]

julia> getoptic(vn2).ix  # Just an ordinary tuple.
(1, 2)

julia> AbstractPPL.is_dynamic(vn2)
false

Alternatively, you can pass true as the second positional argument to the @varname macro (note that it is not a keyword argument!). This will automatically call concretize for you, using the top-level symbol to look up the value used for concretization.

julia> x = randn(2, 3);

julia> @varname(x[1:end, end][:], true)
x[1:2, 3][:]

Interpolation

Property names, as well as top-level symbols, can also be constructed from interpolated symbols:

julia> name = :hello; @varname(x.$name)
x.hello

julia> @varname($name)
hello

julia> @varname($name.a.$name[1])
hello.a.hello[1]

For indices, you do not need to use $ to interpolate, just use the variable directly:

julia> ix = 2; @varname(x[ix])
x[2]

Note that if the top-level symbol is interpolated, automatic concretization is not possible:

julia> name = :x; @varname($name[1:end], true)
ERROR: LoadError: cannot automatically concretize VarName with interpolated top-level symbol; call `concretize(vn, val)` manually instead
[...]

varname(expr, concretize::Bool)

Implementation of the @varname macro. See the documentation for @varname for details. This function is exported to allow other macros (e.g. in DynamicPPL) to reuse the same logic.

getsym(vn::VarName)

Return the symbol of the Julia variable used to generate vn.

Examples

julia> getsym(@varname(x[1][2:3]))
:x

julia> getsym(@varname(y))
:y

getoptic(vn::VarName)

Return the optic of the Julia variable used to generate vn.

Examples

julia> getoptic(@varname(x[1][2:3]))
Optic([1][2:3])

julia> getoptic(@varname(y))
Optic()

is_dynamic(vn::VarName)

Return true if vn contains any dynamic indices (i.e., begin, end, or :). If a VarName has been concretized, this will always return false.

concretize(vn::VarName, x)

Return vn concretized on x, i.e. any information related to the runtime shape of x is evaluated. This will convert any begin and end indices in vn to concrete indices with information about the length of the dimension being indexed into.

AbstractOptic

An abstract type that represents the non-symbol part of a VarName, i.e., the section of the variable that is of interest. For example, in x.a[1][2], the AbstractOptic represents the .a[1][2] part.

Property{sym}(child=Iden())

A property access optic representing access to property sym. A VarName{:x} with this optic represents access to x.sym. The child optic represents any further indexing or property access after this property access operation.

Index(ix, kw, child=Iden())

An indexing optic representing access to indices ix, which may also take the form of keyword arguments kw. A VarName{:x} with this optic represents access to x[ix..., kw...]. The child optic represents any further indexing or property access after this indexing operation.

Iden()

The identity optic. This is the optic used when we are referring to the entire variable. It is also the base case for composing optics.

@opticof(expr, concretize=false)

Extract the optic from @varname(expr, concretize). This is a thin wrapper around getoptic(@varname(...)).

If you don't need to concretize, you should use _ as the top-level symbol to indicate that it is not relevant:

julia> @opticof(_.a.b)
Optic(.a.b)

If you need to concretize, then you can provide a real variable name (which is then used to look up the value for concretization):

julia> x = randn(3, 4); @opticof(x[1:end, end], true)
Optic([1:3, 4])

Note that concretization with @opticof has the same limitations as with @varname, specifically, if the top-level symbol is interpolated, automatic concretization is not possible.

with_mutation(o::AbstractOptic)

Create a version of the optic o which attempts to mutate its input where possible.

On their own, AbstractOptics are non-mutating:

julia> optic = @opticof(_[1])
Optic([1])

julia> x = [0.0, 0.0];

julia> set(x, optic, 1.0); x
2-element Vector{Float64}:
 0.0
 0.0

With this function, we can create a mutating version of the optic:

julia> optic_mut = with_mutation(@opticof(_[1]))
Optic!!([1])

julia> x = [0.0, 0.0];

julia> set(x, optic_mut, 1.0); x
2-element Vector{Float64}:
 1.0
 0.0

Thanks to the BangBang.jl package, this optic will gracefully fall back to non-mutating behaviour if mutation is not possible. For example, if we try to use it on a tuple:

julia> optic_mut = with_mutation(@opticof(_[1]))
Optic!!([1])

julia> x = (0.0, 0.0);

julia> set(x, optic_mut, 1.0); x
(0.0, 0.0)

∘(outer::AbstractOptic, inner::AbstractOptic)

Compose two AbstractOptics together.

julia> p1 = @opticof(_.a[1])
Optic(.a[1])

julia> p2 = @opticof(_.b[2, 3])
Optic(.b[2, 3])

julia> p1 ∘ p2
Optic(.b[2, 3].a[1])

cat(optics::AbstractOptic...)

Compose multiple AbstractOptics together. The optics should be provided from innermost to outermost, i.e., cat(o1, o2, o3) corresponds to o3 ∘ o2 ∘ o1.

ohead(optic::AbstractOptic)

Get the innermost layer of an optic. For all optics, we have that otail(optic) ∘ ohead(optic) == optic.

julia> ohead(@opticof _.a[1][2])
Optic(.a)

julia> ohead(@opticof _)
Optic()

otail(optic::AbstractOptic)

Get everything but the innermost layer of an optic. For all optics, we have that otail(optic) ∘ ohead(optic) == optic.

julia> otail(@opticof _.a[1][2])
Optic([1][2])

julia> otail(@opticof _)
Optic()

oinit(optic::AbstractOptic)

Get everything but the outermost layer of an optic. For all optics, we have that olast(optic) ∘ oinit(optic) == optic.

julia> oinit(@opticof _.a[1][2])
Optic(.a[1])

julia> oinit(@opticof _)
Optic()

olast(optic::AbstractOptic)

Get the outermost layer of an optic. For all optics, we have that olast(optic) ∘ oinit(optic) == optic.

julia> olast(@opticof _.a[1][2])
Optic([2])

julia> olast(@opticof _)
Optic()

varname_to_optic(vn::VarName)

Convert a VarName to an optic, by converting the top-level symbol to a Property optic.

optic_to_varname(optic::Property{sym}) where {sym}

Convert a Property optic to a VarName, by converting the top-level property to a symbol. This fails for all other optics.

subsumes(parent::VarName, child::VarName)

Check whether the variable name child describes a sub-range of the variable parent, i.e., is contained within it.

julia> subsumes(@varname(x), @varname(x[1, 2]))
true

julia> subsumes(@varname(x[1, 2]), @varname(x[1, 2][3]))
true

This is done by recursively comparing each layer of the VarNames' optics.

Note that often this is not possible to determine statically, and so the results should not be over-interpreted. In particular, Index optics pose a problem. An i::Index will only subsume j::Index if:

1. They have the same number of positional indices (i.ix and j.ix);
2. Each positional index in i can be determined to comprise the corresponding positional index in j; and
3. The keyword indices of i (i.kw) are a superset of those in j.kw).

In all other cases, subsumes will conservatively return false, even though in practice it might well be that i does subsume j. Some examples where subsumption cannot be determined statically are:

• Subsumption between different forms of indexing is not supported, e.g. x[4] and x[2, 2] are not considered to subsume each other, even though they might in practice (e.g. if x is a 2x2 matrix).
• When dynamic indices (that are not equal) are present. (Dynamic indices that are equal do subsume each other.)
• Non-standard indices, e.g. Not(4), 2..3, etc. Again, these only subsume each other when they are equal.

prefix(vn::VarName, prefix::VarName)

Add a prefix to a VarName.

julia> prefix(@varname(x), @varname(y))
y.x

julia> prefix(@varname(x.a), @varname(y))
y.x.a

julia> prefix(@varname(x.a), @varname(y[1]))
y[1].x.a

unprefix(vn::VarName, prefix::VarName)

Remove a prefix from a VarName.

julia> unprefix(@varname(y.x), @varname(y))
x

julia> unprefix(@varname(y.x.a), @varname(y))
x.a

julia> unprefix(@varname(y[1].x), @varname(y[1]))
x

julia> unprefix(@varname(y), @varname(n))
ERROR: ArgumentError: cannot remove prefix n from VarName y
[...]

julia> unprefix(@varname(y[1]), @varname(y))
ERROR: ArgumentError: optic_to_varname: can only convert Property optics to VarName
[...]

varname_leaves(vn::VarName, val)

Return an iterator over all varnames that are represented by vn on val.

Examples

julia> using AbstractPPL: varname_leaves

julia> foreach(println, varname_leaves(@varname(x), rand(2)))
x[1]
x[2]

julia> foreach(println, varname_leaves(@varname(x[1:2]), rand(2)))
x[1:2][1]
x[1:2][2]

julia> x = (y = 1, z = [[2.0], [3.0]]);

julia> foreach(println, varname_leaves(@varname(x), x))
x.y
x.z[1][1]
x.z[2][1]

varname_and_value_leaves(vn::VarName, val)

Return an iterator over all varname-value pairs that are represented by vn on val.

Examples

julia> using AbstractPPL: varname_and_value_leaves

julia> foreach(println, varname_and_value_leaves(@varname(x), 1:2))
(x[1], 1)
(x[2], 2)

julia> foreach(println, varname_and_value_leaves(@varname(x[1:2]), 1:2))
(x[1:2][1], 1)
(x[1:2][2], 2)

julia> x = (y = 1, z = [[2.0], [3.0]]);

julia> foreach(println, varname_and_value_leaves(@varname(x), x))
(x.y, 1)
(x.z[1][1], 2.0)
(x.z[2][1], 3.0)

There is also some special handling for certain types:

julia> using LinearAlgebra

julia> x = reshape(1:4, 2, 2);

julia> # `LowerTriangular`
       foreach(println, varname_and_value_leaves(@varname(x), LowerTriangular(x)))
(x[1, 1], 1)
(x[2, 1], 2)
(x[2, 2], 4)

julia> # `UpperTriangular`
       foreach(println, varname_and_value_leaves(@varname(x), UpperTriangular(x)))
(x[1, 1], 1)
(x[1, 2], 3)
(x[2, 2], 4)

julia> # `Cholesky` with lower-triangular
       foreach(println, varname_and_value_leaves(@varname(x), Cholesky([1.0 0.0; 0.0 1.0], 'L', 0)))
(x.L[1, 1], 1.0)
(x.L[2, 1], 0.0)
(x.L[2, 2], 1.0)

julia> # `Cholesky` with upper-triangular
       foreach(println, varname_and_value_leaves(@varname(x), Cholesky([1.0 0.0; 0.0 1.0], 'U', 0)))
(x.U[1, 1], 1.0)
(x.U[1, 2], 0.0)
(x.U[2, 2], 1.0)

varname_and_value_leaves(container)

Return an iterator over all varname-value pairs that are represented by container.

This is the same as varname_and_value_leaves(vn::VarName, x) but over a container containing multiple varnames.

See also: varname_and_value_leaves(vn::VarName, x).

Examples

julia> using AbstractPPL: varname_and_value_leaves

julia> using OrderedCollections: OrderedDict

julia> # With an `AbstractDict` (we use `OrderedDict` here
       # to ensure consistent ordering in doctests)
       dict = OrderedDict(@varname(y) => 1, @varname(z) => [[2.0], [3.0]]);

julia> foreach(println, varname_and_value_leaves(dict))
(y, 1)
(z[1][1], 2.0)
(z[2][1], 3.0)

julia> # With a `NamedTuple`
       nt = (y = 1, z = [[2.0], [3.0]]);

julia> foreach(println, varname_and_value_leaves(nt))
(y, 1)
(z[1][1], 2.0)
(z[2][1], 3.0)

canview(optic, container)

Return true if optic can be used to view container, and false otherwise.

Examples

julia> AbstractPPL.canview(@opticof(_.a), (a = 1.0, ))
true

julia> AbstractPPL.canview(@opticof(_.a), (b = 1.0, )) # property `a` does not exist
false

julia> AbstractPPL.canview(@opticof(_.a[1]), (a = [1.0, 2.0], ))
true

julia> AbstractPPL.canview(@opticof(_.a[3]), (a = [1.0, 2.0], )) # out of bounds
false

hasvalue(
    vals::Union{AbstractDict,NamedTuple},
    vn::VarName,
    dist::Distribution;
    error_on_incomplete::Bool=false
)

Check if vals contains values for vn that is compatible with the distribution dist.

This is a more general version of hasvalue(vals, vn), in that even if vn itself is not inside vals, it further checks if vals contains sub-values of vn that can be used to reconstruct vn given dist.

The error_on_incomplete flag can be used to detect cases where some of the values needed for vn are present, but others are not. This may help to detect invalid cases where the user has provided e.g. data of the wrong shape.

Note that this check is only possible if a Dict is passed, because the key type of a NamedTuple (i.e., Symbol) is not rich enough to carry indexing information. If this method is called with a NamedTuple, it will just defer to hasvalue(vals, vn).

For example:

julia> d = Dict(@varname(x[1]) => 1.0, @varname(x[2]) => 2.0);

julia> hasvalue(d, @varname(x), MvNormal(zeros(2), I))
true

julia> hasvalue(d, @varname(x), MvNormal(zeros(3), I))
false

julia> hasvalue(d, @varname(x), MvNormal(zeros(3), I); error_on_incomplete=true)
ERROR: only partial values for `x` found in the dictionary provided
[...]

hasvalue(vals::NamedTuple, vn::VarName)
hasvalue(vals::AbstractDict{<:VarName}, vn::VarName)

Determine whether vals contains a value for a given vn.

Examples

With x as a NamedTuple:

julia> hasvalue((x = 1.0, ), @varname(x))
true

julia> hasvalue((x = 1.0, ), @varname(x[1]))
false

julia> hasvalue((x = [1.0],), @varname(x))
true

julia> hasvalue((x = [1.0],), @varname(x[1]))
true

julia> hasvalue((x = [1.0],), @varname(x[2]))
false

With x as a AbstractDict:

julia> hasvalue(Dict(@varname(x) => 1.0, ), @varname(x))
true

julia> hasvalue(Dict(@varname(x) => 1.0, ), @varname(x[1]))
false

julia> hasvalue(Dict(@varname(x) => [1.0]), @varname(x))
true

julia> hasvalue(Dict(@varname(x) => [1.0]), @varname(x[1]))
true

julia> hasvalue(Dict(@varname(x) => [1.0]), @varname(x[2]))
false

In the AbstractDict case we can also have keys such as v[1]:

julia> vals = Dict(@varname(x[1]) => [1.0,]);

julia> hasvalue(vals, @varname(x[1])) # same as `haskey`
true

julia> hasvalue(vals, @varname(x[1][1])) # different from `haskey`
true

julia> hasvalue(vals, @varname(x[1][2]))
false

julia> hasvalue(vals, @varname(x[2][1]))
false

getvalue(
    vals::Union{AbstractDict,NamedTuple},
    vn::VarName,
    dist::Distribution
)

Retrieve the value of vn from vals, using the distribution dist to reconstruct the value if necessary.

This is a more general version of getvalue(vals, vn), in that even if vn itself is not inside vals, it can still reconstruct the value of vn from sub-values of vn that are present in vals.

Note that this reconstruction is only possible if a Dict is passed, because the key type of a NamedTuple (i.e., Symbol) is not rich enough to carry indexing information. If this method is called with a NamedTuple, it will just defer to getvalue(vals, vn).

For example:

julia> d = Dict(@varname(x[1]) => 1.0, @varname(x[2]) => 2.0);

julia> getvalue(d, @varname(x), MvNormal(zeros(2), I))
2-element Vector{Float64}:
 1.0
 2.0

julia> # Use `hasvalue` to check for this case before calling `getvalue`.
       getvalue(d, @varname(x), MvNormal(zeros(3), I))
ERROR: `x` was not found in the dictionary provided
[...]

getvalue(vals::NamedTuple, vn::VarName)
getvalue(vals::AbstractDict{<:VarName}, vn::VarName)

Return the value(s) in vals represented by vn.

Examples

For NamedTuple:

julia> vals = (x = [1.0],);

julia> getvalue(vals, @varname(x)) # same as `getindex`
1-element Vector{Float64}:
 1.0

julia> getvalue(vals, @varname(x[1])) # different from `getindex`
1.0

julia> getvalue(vals, @varname(x[2]))
ERROR: x[2] was not found in the NamedTuple provided
[...]

For AbstractDict:

julia> vals = Dict(@varname(x) => [1.0]);

julia> getvalue(vals, @varname(x)) # same as `getindex`
1-element Vector{Float64}:
 1.0

julia> getvalue(vals, @varname(x[1])) # different from `getindex`
1.0

julia> getvalue(vals, @varname(x[2]))
ERROR: x[2] was not found in the dictionary provided
[...]

In the AbstractDict case we can also have keys such as v[1]:

julia> vals = Dict(@varname(x[1]) => [1.0,]);

julia> getvalue(vals, @varname(x[1])) # same as `getindex`
1-element Vector{Float64}:
 1.0

julia> getvalue(vals, @varname(x[1][1])) # different from `getindex`
1.0

julia> getvalue(vals, @varname(x[1][2]))
ERROR: x[1][2] was not found in the dictionary provided
[...]

julia> getvalue(vals, @varname(x[2][1]))
ERROR: x[2][1] was not found in the dictionary provided
[...]

julia> getvalue(vals, @varname(x))
ERROR: x was not found in the dictionary provided
[...]

Dictionaries can present ambiguous cases where the same variable is specified twice at different levels. In such a situation, getvalue attempts to find an exact match, and if that fails it returns the value with the most specific key.

julia> vals = Dict(@varname(x) => [[1.0]], @varname(x[1]) => [2.0]);

julia> # Here, the `x[1]` key is not used because `x` is an exact match.
       getvalue(vals, @varname(x))
1-element Vector{Vector{Float64}}:
 [1.0]

julia> # Likewise, the `x` key is not used because `x[1]` is an exact match.
       getvalue(vals, @varname(x[1]))
1-element Vector{Float64}:
 2.0

julia> # No exact match, so the most specific key, i.e. `x[1]`, is used.
       getvalue(vals, @varname(x[1][1]))
2.0

index_to_dict(::Integer)
index_to_dict(::AbstractVector{Int})
index_to_dict(::UnitRange)
index_to_dict(::StepRange)
index_to_dict(::Colon)
index_to_dict(::ConcretizedSlice{T, Base.OneTo{I}}) where {T, I}
index_to_dict(::Tuple)

Convert an index i to a dictionary representation.

dict_to_index(dict)
dict_to_index(symbol_val, dict)

Convert a dictionary representation of an index dict to an index.

Users can extend the functionality of dict_to_index (and hence VarName de/serialisation) by extending this method along with index_to_dict. Specifically, suppose you have a custom index type MyIndexType and you want to be able to de/serialise a VarName containing this index type. You should then implement the following two methods:

1. AbstractPPL.index_to_dict(i::MyModule.MyIndexType) should return a dictionary representation of the index i. This dictionary must contain the key "type", and the corresponding value must be a string that uniquely identifies the index type. Generally, it makes sense to use the name of the type (perhaps prefixed with module qualifiers) as this value to avoid clashes. The remainder of the dictionary can have any structure you like.

2. Suppose the value of index_to_dict(i)["type"] is "MyModule.MyIndexType". You should then implement the corresponding method AbstractPPL.dict_to_index(::Val{Symbol("MyModule.MyIndexType")}, dict), which should take the dictionary representation as the second argument and return the original MyIndexType object.

To see an example of this in action, you can look in the the AbstractPPL test suite, which contains a test for serialising OffsetArrays.

varname_to_string(vn::VarName)

Convert a VarName as a string, via an intermediate dictionary. This differs from string(vn) in that concretised slices are faithfully represented (rather than being pretty-printed as colons).

For VarNames which index into an array, this function will only work if the indices can be serialised. This is true for all standard Julia index types, but if you are using custom index types, you will need to implement the index_to_dict and dict_to_index methods for those types. See the documentation of dict_to_index for instructions on how to do this.

julia> varname_to_string(@varname(x))
"{\"optic\":{\"type\":\"Iden\"},\"sym\":\"x\"}"

julia> varname_to_string(@varname(x.a))
"{\"optic\":{\"child\":{\"type\":\"Iden\"},\"field\":\"a\",\"type\":\"Property\"},\"sym\":\"x\"}"

string_to_varname(str::AbstractString)

Convert a string representation of a VarName back to a VarName. The string should have been generated by varname_to_string.