<aside> 💡 The goal of this feature is for users to be able to annotate relationships between objects.

</aside>

Main design decisions

• A link is defined between two objects. A link has a direction, pointing from a parent to a child.
  • Example: a link from a vehicle path (parent) to a lane line (child)
• A link can have user-defined link attributes of type select, checkbox, text, number. This is similar to how object attributes are defined.
  • Example: a link between a vehicle path and a lane line can have a link attribute lane_line_side of type select, with values left or right.
• Link attributes can be defined as being sequence-level (they stay constant across the sequence) or frame-level (they can change from frame to frame). This is similar to how object attributes are defined.
  • Example: The link between a vehicle path and a lane line should stay constant across the sequence. But the link between a pedestrian and a stroller could change from frame to frame, e.g. the pedestrian is pushing/holding/not touching the stroller.
  • Will be a distinction in a v2

UX

Category editor

• For each category, there's a new section "Link attributes" below the existing "Object attributes" section, where you can define the possible link attributes for links from this category (parent) to other categories (child)
• In a second iteration we’ll add an additional column valid_child_categories for each defined link attribute, which is a multi-select dropdown where you can select a whitelist of child categories for which this link attribute should be available.

Labeling interface

• In the objects sidebar, a new section “Links” will appear below the existing Object Attributes section. It shows all links in which the active object is involved, either as parent or child. If you select a link:
  • It expands to show the link attributes
  • The link is highlighted on the canvas somehow, probably with an arrow from parent to child and/or some color encoding.
  • There should also be a way to show all links in the frame (e.g. add a button to the image|both|label widget) or a setting or so
• You’ll be able to create a new link between two objects by clicking a “+” button in the Links section, and then clicking on the parent and child objects. Maybe later also via a context menu when right-clicking on an object.

Label format

• We add an extra field links to the label attributes JSON, at the same level as the existing annotations field:

"frames": [
    {
      "annotations": [...],
      "links": [
        {
          "from_id": 2,
          "to_id": 5,
          "attributes": {...}
        }
      ]
    },
    {...},
    {...}
  ]

• Alternatively we store the links with each parent:

"frames": [
    {
      "annotations": [
	      {
		      "track_id": 2,
		      "category_id": 4,
		      "attributes": {...}, // The *object* attributes
		      "links": [ // List of links where this object is the parent
			      {
				      "to_id": 5,
				      "attributes": {...} // The *link* attributes
				    },
				    ...
		      ]
	      }
      ],
    },
    {...},
    {...}
  ]

Implementation for Torc BeV project

<aside> 💡 Below are some examples of how the linking feature would be used for the BeV project

</aside>