Headline:Post-Mortem on the Yggdrasil/Practcl debacle
Date:Saturday, June 29, 2019
Posted By:Sean Woods

I was super-excited about a great coming together of ideas a few days ago in my post Adventures in Coding June 27, 2019. And Then I discovered not all was well in the universe in a followup post: Postscript: Genius of June 27, 2019. In this post I want to examine exactly what is going on inside of the Practcl module that made it incompatible with the mechanisms of Yggdrasil.

Practcl is a module I wrote for Tcllib to take the best of all of the Tcl build system wrappers and combine them into one framework and notation. As a distribution maintainer (albeit for a very small, niche distribution), I regularly have to perform maintenance work on existing packages to either fix bugs in software nobody else is maintaining or to make the software work in ways the original designer never intended.

I had a long winded explanation of what Practcl is, and why its important, but it's probably better to just steer you to my Conference Paper from Tcl 2016 or the PDF of the slides from that presentation

The intent with Practcl was that we start with a master project object, and then connect the various pieces of the library or application to the master project as objects subordinate to the master package. Complex objects can, themselves, contain sub-objects. So much like Yggdrasil, we end up with a tree structure.

However... unlike Yggdrasil, each type of relationship has different rules about how the objects interact. Rather than simple connections, Practcl's metaclass system has a way of grouping objects by which relationship they share. It was also possible for objects to be related in several different ways. And I have to admit, one of the frustrations I've had in communicating the value of Practcl is that the relationships are arbitrary and built more for matching the existing build process and intuitive assumptions of a developer working in that field.

To lighten the load of the developer, Practcl does a lot of "filling in the gaps". For instance there is a method "add", which is comparable to the "tag" in Yggdrasil. However, if the argument is a file, the parser knows to either dig into the file for details, or simply infer information based on the file's extension, location, etc.

The add method in Practcl normally just accepts a stream of options. And then an inference engine reads those options and translates them into class mixins, and parameters for the methods of those mixins to act on.

When add didn't suffice, I have a lot of add_type type methods which take extra arguments to lead the witness about the relationships, implement methods in the object, and so on. One approach would be to create one, "add", and have add have a mandatory argument "type", as well as standard configuration option for "and here is a set of new methods for that object."

Having domain specific verbs with expected behaviors and a bespoke method for invoking those rules was a) easy to write, b) easy to understand and c) easy to debug. But it's not extendable. Every time I want a new type, I have to decide if that type is something that the entire meta class needs to support, if that's something I need to add to one type of container. And if its for one type of container, I need to have some sort of notification to the developer that the princess is in another castle if they try to add that same object type somewhere else.

But... when I pick apart some of the add_ methods, I see some sins that have to be cleansed.

For instance add_object is just a redirect to link object. Which is stupid. We have a method for that:

  method add_object object {
    my link object $object
  }

add_project and add_tool seem to have a lot of code that expresses special relationships between an object and a progeny of these types. But when you pick the implementations apart:


method add_project {pkg info {oodefine {}}} {
  ::practcl::debug [self] add_project $pkg $info
  set os [my define get TEACUP_OS]
  if {$os eq {}} {
    set os [::practcl::os]
    my define set os $os
  }
  set fossilinfo [list download [my define get download] tag trunk sandbox [my define get sandbox]]
  if {[dict exists $info os] && ($os ni [dict get $info os])} return
  # Select which tag to use here.
  # For production builds: tag-release
  set profile [my define get profile release]:
  if {[dict exists $info profile $profile]} {
    dict set info tag [dict get $info profile $profile]
  }
  dict set info USEMSVC [my define get USEMSVC 0]
  dict set info debug [my define get debug 0]
  set obj [namespace current]::PROJECT.$pkg
  if {[info command $obj] eq {}} {
    set obj [::practcl::subproject create $obj [self] [dict merge $fossilinfo [list name $pkg pkg_name $pkg static 0 class subproject.binary] $info]]
  }
  my link object $obj
  oo::objdefine $obj $oodefine
  $obj define set masterpath $::CWD
  $obj go
  return $obj
}

method add_tool {pkg info {oodefine {}}} {
  ::practcl::debug [self] add_tool $pkg $info
  set info [dict merge [::practcl::local_os] $info]

  set os [dict get $info TEACUP_OS]
  set fossilinfo [list download [my define get download] tag trunk sandbox [my define get sandbox]]
  if {[dict exists $info os] && ($os ni [dict get $info os])} return
  # Select which tag to use here.
  # For production builds: tag-release
  set profile [my define get profile release]:
  if {[dict exists $info profile $profile]} {
    dict set info tag [dict get $info profile $profile]
  }
  set obj ::practcl::OBJECT::TOOL.$pkg
  if {[info command $obj] eq {}} {
    set obj [::practcl::subproject create $obj [self] [dict merge $fossilinfo [list name $pkg pkg_name $pkg static 0] $info]]
  }
  my link add tool $obj
  oo::objdefine $obj $oodefine
  $obj define set masterpath $::CWD
  $obj go
  return $obj
}


We can see that the two method are virtually identical. However:

And that last point is where my efforts on the 26th ran smack into a wall. No matter what I tried to do, I couldn't get projects to inherit certain behaviors despite adding them to the project's core metaclass.

While bespoke object creation methods are convenient for the developer of the library, they hide important details about the objects and their relationships that should really be explained in a way that is comparable to the other relationships in the system. They also create the possibility for copy-and-sorta-paste to leave bug fixes out of one lesser used method that were fixed in the more utilized version.

My strategy for the rest of the day is properly pick apart what I was trying to do in practcl and see if some of those great ideas still make sense. And yes, I was the original author, and thusfar sole contributor, to Practcl. It's just a project I wrote several years ago, so I'm having to learn it just like a new maintainer.

... and yes... I really wish I had commented things better.

On my hitlist for today:

Another ghost from development approaches past is Practcl's tendency to headswap an object into another class:

::clay::define ::practcl::metaclass {

# Called by the object itself of the container to get it to evolve into its
# final form
method select {} {
  my variable define
  if {[info exists define(class)]} {
    my morph $define(class)                     ; # Call the morph method
  } else {
    if {[::info exists define(oodefine)]} {
      ::oo::objdefine [self] $define(oodefine)  ; # Monkey patch the object
    }
  }
}

# Called by the select method and in the case of dynamically produced products
# my the constructor
method morph classname {
  my variable define
  if {$classname ne {}} {
    set map [list @name@ $classname]
    foreach pattern [string map $map [my _MorphPatterns]] {
      set pattern [string trim $pattern]
      set matches [info commands $pattern]
      if {![llength $matches]} continue
      set class [lindex $matches 0]
      break
    }
    set mixinslot {}
    foreach {slot pattern} {
      distribution ::practcl::distribution*
      product      ::practcl::product*
      toolset      ::practcl::toolset*
    } {
      if {[string match $pattern $class]} {           ; # <- Guess the slot from context
         set mixinslot $slot
         break
      }
    }
    if {$mixinslot ne {}} {
      my clay mixinmap $mixinslot $class              ; # <- Use a mixin if we specify a slot
    } elseif {[info command $class] ne {}} {
      if {[info object class [self]] ne $class} {
        ::oo::objdefine [self] class $class           ; # <- THEREIN LIES MADNESS
        ::practcl::debug [self] morph $class
         my define set class $class
      }
    } else {
      error "[self] Could not detect class for $classname"
    }
  }
  if {[::info exists define(oodefine)]} {
    ::oo::objdefine [self] $define(oodefine)         ; # Monkey patch the object
  }
}
}

The only other place we call morph is a conditional in a conditional in a guess in a constructor:

::clay::define ::practcl::product.dynamic {
  superclass ::practcl::dynamic ::practcl::product

  method initialize {} {
    set filename [my define get filename]
    if {$filename eq {}} {
      return
    }
    if {[my define get name] eq {}} {
      my define set name [file tail [file rootname $filename]]
    }
    if {[my define get localpath] eq {}} {
      my define set localpath [my <module> define get localpath]_[my define get name]
    }
    # Future Development:
    # Scan source file to see if it is encoded in criticl or practcl notation
    #set thisline {}
    #foreach line [split [::practcl::cat $filename] \n] {
    #
    #}
    ::source $filename
    if {[my define get output_c] ne {}} {
      # Turn into a module if we have an output_c file
      my morph ::practcl::module                     ; # Make this behave... like a module?
    }
  }
}

What is clearly going on here is a developer the couldn't figure out if he was writing some sort of imperative language with fixed options, or some sort of mamby-pambi "Guess what the user wanted" sort of system. And if it sounds like I know what was in the developer's mind, I should. I'm that schmuck.

I have to admit the Practcl was cobbled together over a few years, a few beers, and I went back and forth a bit on the notation and structure. One of the difficult issues I ran across was how to provide custom methods for objects that needed a little extra customization. With the add method, I allowed the developer to tuck a script into the configuration data. With the add_project and add_tool methods, it was fed in as an argument proper. Oh, and if you are curious, the reason I had to continually re-apply the script was that if I head-swapped the class of the object, the class's methods had the potential to overwrite custom object methods. I'm not sure if that was intended or a bug, but at the time it was just easier to work around. I'm not even sure that is the case anymore. I do know with mixins that never happens. And I'm also really convinced I need to just stop morphing objects, and instead to all of my evil through mixins.

To be fair to myself, I really didn't properly undertand mixins when I started writing Practcl. And honestly, until you've seen them in action they are really hard to explain. I had to largely scrap the Tool framework (Clay's predecessor) largely because the two approaches to property inheritance and method ensembles were wholly irreconcilable.

I've developed some technologies for other projects that use clay since Practcl was written that can help straighten out this mess.

Object Definition Scripts

The first technology is passing a Tcl script as the configuration script for an object instead of a dictionary. Dictionaries are nice and easy to parse... but handling anything configuration-wise that isn't a dictionary is limiting. One of the lesser used elements of TclOO is the fact that objects are a proper tcl namespace that can have procs inserted into them. So, I add verbs (as procs) to form a domain specific language. That language can handle far more complex vocabulary, including modifying methods.

clay::define myclass {
  method dsl {} {
    proc config {args} {
      my config set {*}$args
    }
    proc method args {
      oo::objdefine [self] method {*}$args
    }
    proc clay args {
      my clay {*}$args
    }
  }
  constructor script {
    my dsl
    eval $script
  }
}

An object being instantiated would look like:

CONTAINER add_dynamic {
  config output_c foo.c

  method custom {a b} {
    set c [my frob $a]
    set d [my frob $b]
    return [grok $c $d]
  }
}

Mixin Options Slots

Another nifty too that has evolved is a standardized form of option handling. And one of the options I particularly enjoy handling are those which define a mixin for a slot:

::clay::tk::megawidget clay set option hull {
  class mixin
  default frame
  pattern ::clay::tk::hull
}

This defines an option for class ::clay::tk::megawidget, which when set looks for a class that matches the pattern ::clay::tk::hull.*. So if the object is passed:

$OBJ config set hull frame

The object automatically knows to mixin ::clay::tk::hull.frame to the hull slot. Even better, we can define a default! The magic of what actions have to occur when a class are mixed in are already handled by the framework. While in an ideal world each mixin slot should be an independant swim line, we still have some control over the mixin order by controlling the order in which the mixin options are defined. Deep, Deep down in Clay's internals, all of that meta data is still represented as a dictionary. And dictionaries preserve order.

Now I'm not saying that I'm going to start re-coding Practcl just this minute. But I can be assured that there is no magic secret in Practcl that I'm going to be slapping my head to say "Why didn't I remember that?!?" when I go off and develop Yggdrasil and Ziggurat further. I can also sleep soundly knowing that the next version of Practcl will have all of this design stuff hammered out and work in a way that developers of Clay will be able to follow right along with.

I am on the fence about whether partitioning links by type or just "all god's children" with a filter are the best approach. When Yggdrasil is implemented on top of a relational database, I don't have to chose. I just make sure one column is the type, and I can tune my search with select. (Long story short... that's something I can leave to the squishy parts of individual applications.)