parameterize decorator should add values to default_parameter_values

[vograno]

Is your feature request related to a problem? Please describe.
When a vanilla node is created the default values of its parameters, if any, are available via node.default_parameter_values attribute. However, when a node is created by @parameterize, the values passed to the decorator are not available via the attribute. In fact they are not available from the node object at all.

The point I am making is that those values are essentially default values for the expanded node and should be surfaced in the same fashion. This feature would make expanded nodes equivalent to vanilla nodes in this regards.

Describe the solution you'd like
Define a node

x_value = {}
@parameterize(output_node=dict(x=value(x_value))
def fun(x):
    return x

The following should hold

node = driver.graph.nodes['output_node']
# the passed in x_value and the stored value are the same objects
assert 'x' in node.default_parameter_values
assert node.default_parameter_values['x'] is x_value

# the passed in x_value and the object returned by the node are the same objects
result = driver.execute(final_vars=['output_node'])
assert result['output_node'] is x_value

[skrawcz]

To clarify, you wouldn't expect the following to work normally though right?

# this should not work
result = driver.execute(final_vars=['output_node'], inputs={"x": {"some": "value"}})

But I think I see where you're coming from now I think. The tricky part is how would you handle this:

x_value = {}
y_value = {}
@parameterize(
  output_node=dict(x=value(x_value), 
  output_node2=dict(x=value(y_value))
)
def fun(x: dict) -> dict:
    return x

Or is the case that all of them have a the same default value?

---

In the interim you could do something functionally equivalent like this:

@tag(marker="default") # could identify this as such
def x() -> dict:
   return {} # or reference some variable like you have above in the module

@tag(marker="parameterized")
@parameterize(output_node=dict(x=source("x")))
def fun(x: dict) -> dict:
    return x

then:

1. ask the graph for all parameterized nodes via dr.list_available_variables({"TAG_NAME": "TAG_VALUE"})
2. ask the graph for all default nodes via dr.list_available_variables({"TAG_NAME": "TAG_VALUE"})
3. figure out which default values
4. default nodes provide the originating function, so you can exercise it to get the value.
5. if you wanted to change that value, you could just use overrides= via .execute()...

[vograno]

To clarify, you wouldn't expect the following to work normally though right?

# this should not work
result = driver.execute(final_vars=['output_node'], inputs={"x": {"some": "value"}})

This shouldn't work indeed.

But I think I see where you're coming from now I think. The tricky part is how would you handle this:

x_value = {}
y_value = {}
@parameterize(
  output_node=dict(x=value(x_value), 
  output_node2=dict(x=value(y_value))
)
def fun(x: dict) -> dict:
    return x

Or is the case that all of them have a the same default value?

How do I handle it or how to implement this? In terms of handling, this is precisely what I need: I have two nodes, output_node and output_node2, they return distinct values, and I can access those values via respective node objects.

Anyway, the default values are always different, i.e. x_value and y_value are distinct objects. This is in fact the whole point of it.

If it helps, one can add node.bound_parameter_values property and store, or compute on the fly, parameterize-bound values there instead of on default_parameter_values.

By compute on the fly I, for example, mean the case where the callable, which is already stored on the node object, is a partial object and you can pull the info from there.

In fact, if this is indeed the case and the callable is surfaced, I should be able to pull this info myself, at least in the interim.

[skrawcz]

So if you use driver.graph.nodes you're using internal APIs... but yes this works:

from hamilton.function_modifiers import parameterize, value


x_value = {"b": 2}
y_value = {"a": 1}

@parameterize(
  output_node=dict(x=value(x_value)),
  output_node2=dict(x=value(y_value))
)
def fun(x: dict, i: int) -> dict:
    return x


if __name__ == "__main__":

    from hamilton import driver
    import __main__ as main

    dr = driver.Builder().with_modules(main).build()
 
    # this is using internal APIs -- but yes you have access to the bound variables
    list(dr.graph.nodes.values())[0].callable.keywords
    list(dr.graph.nodes.values())[1].callable.keywords

However, the more "blessed path", would be to go via:

    vars = dr.list_available_variables()

However it seems that there's a 🐛 because the partial function isn't exposed in originating_functions -- looking at the code there's some slight inconsistencies with how expander decorators set the originating functions field.
So four paths to get what you need:

1. use internal Node and inspect the callable.
2. we add func field to HamiltonNode, or figure out the right way to add the partial function to originating_functions field -- then you'd inspect that callable. CC @elijahbenizzy .
3. we wire through a "bound variable" construct in Node through to HamiltonNode.
4. you model the default value via the output of a function and wire it with source.

[skrawcz]

With #1125 then this would work:

vars[1].originating_functions[1].keywords

should have the partial function and what is is bound to.

[elijahbenizzy]

OK, so it seems like the requireent is that we want to know what the right way to expose this value. I think this clarifies that how we're doing it is not perfect IMO -- the concept of a "bound" variable is nice, and should be wired through.

We often do a ton of function-algebra (callables with other callables), but really in the case of parameterize/inject (and mabye even default vals), having a bound kwarg concept would actually simplify code.

I'm uncomfortable with layering the keywords onto the function and inspecting that -- it's too much internals. Unless we make everything a partial. But the partial application could be done within the node's callable with the originating function (in most cases). That said, if each node had:

(1) a function
(2) a concept of bound variables

We could pretty easily expose this all with a bit of refactoring -- can take a look soon. My take from slack is that this is not an immediate blocker, more ergonomics, but let us know how it fits into your workflow!

[vograno]

Thank you guys. I appreciate your receptiveness and responsiveness. Take your time.

A couple of notes just in case

1. For the function, consider leveraging the inspect module approach, the function signature in particular.
2. For the bound variables, I assume both sources and values bindings will be provided.

[elijahbenizzy]

Thank you guys. I appreciate your receptiveness and responsiveness. Take your time.

A couple of notes just in case

1. For the function, consider leveraging the inspect module approach, the function signature in particular.
2. For the bound variables, I assume both sources and values bindings will be provided.

Yep! We use inspect heavily in implementing Hamilton :) Quite a useful module (and Hamilton would be impossible without it). Re: bound variables -- I think it's fundamentally different. The sources bindings will actually be edges in the graph -- thus we wouldn't need to expose it (it's available now if you just ask for the dependencies...).

[vograno]

But I wanted to know the name of the function parameter which the source binds to. This doesn't affect DAG execution, but it does affect the function-algebra stuff you mentioned earlier.

[vograno]

Here is a specific example

@parameterize(
  o1=dict(x=source('fun_x_input')),
)
def fun(x: dict, i: int) -> dict:
    return x

Assume that 'fun_x_input' is an input node and it feeds into fun only. However, the value of fun_x_input is not free, rather it depends on the exact function and exact argument of the function it feeds into. I wanted to have the ability to examine the node, figure out the function and the argument and compute the implied input value for fun_x_input. Then I'll execute the dag with fun_x_input set to the computed value. In general, the implied value for fun_x_input depends on the DAG topology, including the node function names and signatures, and some user-provide input.

[elijahbenizzy]

Here is a specific example

@parameterize(
  o1=dict(x=source('fun_x_input')),
)
def fun(x: dict, i: int) -> dict:
    return x

Assume that 'fun_x_input' is an input node and it feeds into fun only. However, the value of fun_x_input is not free, rather it depends on the exact function and exact argument of the function it feeds into. I wanted to have the ability to examine the node, figure out the function and the argument and compute the implied input value for fun_x_input. Then I'll execute the dag with fun_x_input set to the computed value. In general, the implied value for fun_x_input depends on the DAG topology, including the node function names and signatures, and some user-provide input.

Hmm, interesting... So the algebra of what happens under the hood is:
(1) fun is created with input fun_x_input -- this is wired through to the original parameter of x
(2) as fun_x_input does not exist in the DAG, a node named fun_x_input gets created

So in the current framework, it's a bit hidden how exactly it wires to the underlying function. In fact, the only contract is that it will get passed in some way, not how it will get passed/that you can access it.

But to get back to your point -- if you know that fun_x_input is a dependency of fun`, you should be able to do what you want?

all_vars = dr.list_available_variables()
fun_var, = [var for var in all_vars if var.name == "fun"]
deps = var.required_dependencies | var.optional_dependencies
user_inputs = [var for var in all_vars if var.is_external_input and var.name in deps]
# this should be the information you need, in the `source` case.

Note this doesn't get the value case, but it does allow you to inspect the source. The fact that it is bound with @parameterize is more an implementation detail (as it currently stands).

[vograno]

Are you saying there will be a var named 'fun' even though the generated node name is 'o1'?

[elijahbenizzy]

un' even though the generate

Ahh, yes, sorry, didn't see that it was called o1, good point!

It should look like this:

all_vars = dr.list_available_variables()
o1_var, = [var for var in all_vars if var.name == "o1"]
deps = var.required_dependencies | var.optional_dependencies
user_inputs = [var for var in all_vars if var.is_external_input and var.name in deps]
# this should be the information you need, in the `source` case.

The fact that the function is named fun is lost in translation here -- it's not part of the contract.

Curious -- does this solve your problem? Or do you need further inspection into the way it was defined...