Taxonomy of deformable object shape control

From the control system's point of view, what is the available object information?

Mechanical Information

• Does the control system make use of mechanical information? (yes/no; if yes, which type: density, elasticity constants, stress, etc.)
  • The control system might receive and use data about the mechanical properties or behaviour of the object. Although some of the mechanical properties may have been specified in section A, it is now important to specify which of them are actually being exploited by the control system.

Geometric Information

• Does the control system make use of geometric information? (yes/no; if yes, which type: intrinsic and extrinsic dimensions, scale, etc.)
  • This requires specifying both intrinsic and extrinsic dimensions. Note that the received geometric information does not necessarily correspond directly to the actual object's geometry.
  • For example: A contour extraction of a balloon observed in a 2D image provides closed 1D intrinsic, 2D extrinsic information (apparent contour), whereas the actual object (the balloon) has a closed 2D intrinsic, 3D extrinsic geometry.

Spatial Resolution

• What is the spatial resolution or density of the input information? (low, medium or high)
  • Low-density: Sparse/Discrete geometric information. Example: the position of 5 visual features extracted from fiducial markers attached to the object.
  • Medium-density: More detailed geometric and structured representation. Example: A 3D mesh with 50 vertices.
  • High-density: Dense/continuous data points. Example: A 3D point cloud with 500 points capturing fine surface details.
• Note: Specifying the number of extracted discrete features or the pixel density (e.g., pixels per meter [ppm]) can provide a valuable measure of the information's geometric resolution.

Temporal Resolution

• What is the temporal resolution of the input information? (low, medium or high, quantify in Hz)
  • Low Temporal Resolution: Sparse data captured over longer intervals. Example: Sensor data sampled at 1 Hz.
  • Medium Temporal Resolution: Data captured at moderate intervals. Example: Sensor data sampled at 15 Hz.
  • High Temporal Resolution: Continuous and high-frequency data capture. Example: Sensor data sampled at 120 Hz.

Information Source

• What is the information source? (Specify whether and how the employed information is assumed, estimated and/or measured)
  • Assumed: Information based on assumptions or theoretical models. Example: assuming the elastic modulus of an object based on its material (from tables).
  • Estimated: Data derived from indirect measurements or visual parameters. Example: Estimating the weight distribution of an object based on its geometry and materials.
  • Measured: Directly obtained from sensors or empirical measurements. Example: Using strain gauges to measure deformation during a bending operation.

Acquisition Phase

• When and how often can you obtain such information? (a priori, offline or online)
  • A Priori: Information obtained before any interaction with the object. Example: Knowing object dimensions from a CAD model.
  • Offline: Acquired before the active control phase but involving direct observation. Example: Conducting a laser scan of a prototype before the control process.
  • Online: Real-time information gathered during the shape control process. Example: Using computer vision to track object surface position in real-time.
• Note: The acquisition phase is critical for ensuring timely and relevant data availability. A priori and offline information helps in planning, while online information enables adaptive control strategies.

What does the control system care about in its shape error computation? (specify one of the 3 options)

• Shape, Scale, and Transport: Control involving shape, size, and rigid body transformations (translation and rotation).
• Shape and Scale: Control involving shape and size, but not rigid body transformations.
• Shape Only: Control involving only the shape, without considering size or rigid body transformations.

• What dimensionality does the shape error consider? (Specify intrinsic and extrinsic dimensionalities: 1D, 2D, or 3D)
  • This refers to how the dimensions of the object's information correspond to the dimensions used for calculating the shape error.
  • For example: you might capture detailed 3D information of a pizza using an RGB-D camera, yet only utilise the 1D intrinsic contour (the flat circular shape) to compute the control error signal. If the pizza were to deform into a cone-like shape that maintains its circular contour, the previously described error signal would fail to detect the shape change.

Does the reference shape information vary through time? How does it vary? (specify one)

• Fixed Target Shape (Constant Reference): The control reference remains constant, sometimes facilitating stability analysis as the derivative of the shape reference is zero.
• Fixed Target Shape with Variable Reference: While the target shape remains fixed, new intermediate control references are computed between the target shape and the current shape (e.g., interpolating intermediate states or Procrustes analysis), facilitating shape control convergence.
• Variable Target Shape (Shape Trajectory Control): The reference shape itself varies over time, presenting a challenge in maintaining proper shape control over the target shape evolution.

What method does the shape control system use to compare the current and target shape representations? (specify one)

Discrete Feature-Based Errors

• Compares sparse visual or geometric features between the current and target shapes, such as specific keypoints or distinctive visual elements.
• Example: Detecting, matching, and comparing the position of ORB features between two different object configurations.
• Note: this method relies on the presence and consistent placement of visual features in the target shape.

Point Alignment Errors

• Focuses on aligning specific points on the object to points on the target shape, without directly computing point correspondences.
• Example: Aligning pixels from a homogeneously textured object, aiming to maximise the overlap between the current and target pixel positions.

Shape Point Matching

• Matches sets of points sampled from the shape's geometric domains:
  • Homogeneous Matching: Points are uniformly distributed across current and target shapes. Effective for isometric deformations. Example: Sampling 10 evenly spaced points along the medial axis of a DLO.
  • Elastic Matching: Point matches are variably distributed to represent material points during strain deformations. Example: Sampling points uniformly along cloth contours during stretching, then matching with non-uniform density.

Parametric Errors

• Based on comparing mathematical parameters that define curves (e.g., Bézier curves) or series (e.g., Fourier series), quantifying differences based on mathematical representations.
• Example: comparing the value of the complex Fourier coefficients between two closed contours.

Continuous Map Errors

• Compares continuous representations of shape domains. The most complete ways of comparing two shapes:
  • Homogeneous Continuous Maps: Points are uniformly distributed in the mapped space, such as in functional maps for surfaces.
  • Elastic Continuous Maps: Points are variably distributed, adapting to deformations like stretching or compressing, such as in FMM continuous contour mapping.

Physical Meaning Scale

• Is your model physically meaningful or abstract? (use the 1–8 scale below)

Physically Meaningful (1–3):

• 1: Fully constitutive model with detailed physical accuracy (e.g., FEM, detailed thermodynamic models).
• 2: Mostly constitutive with some abstract elements (e.g., simplified FEM with empirical adjustments).
• 3: Balanced between constitutive and abstract (e.g., reduced-order models with significant physical basis).

Mixed (4–5):

• 4: Predominantly abstract but includes key physical parameters.
• 5: Equally abstract and physical (e.g., mixed methods combining physical laws and statistical approaches).

Abstract (6–8):

• 6: Mostly abstract with minimal physical parameters.
• 7: Highly abstract with some physical parameters used for calibration.
• 8: Fully abstract control-oriented model (e.g., neural networks, pure statistical models).

Additional Characterisation

• Versatility: Can the model apply to various scenarios or is it specific to certain objects or materials?
• Complexity: Specify algorithmic complexity (Big O notation) and time-cost in [Hz] if possible.
• Accuracy: Specify the model's accuracy as the MSE between predicted and observed behaviour.

The ability of an object to undergo large deformations does not necessarily mean that a control system can handle such deformations effectively.

• What types of deformations can the shape control system handle?
  • Strain Deformations: Handling changes in length or stretching.
  • Bending Deformations: Handling changes in shape such as folding or curving.
  • Note: some shape control systems may excel at isometric deformations but fail to converge when strain deformations occur.
• Can the control system handle large deformations? What are the limitations?
  • Local Deformations: Ability to handle small-scale changes (< 5%).
  • Global Deformations: Ability to handle both small and large-scale bending or stretching (> 50% in bending and > 20% in strain).

• Does the shape control approach provide formal stability analysis? (yes/no; if yes, what type: local stability, local asymptotic stability, global asymptotic stability, etc.)
• Does the approach provide controllability/reachability analysis? (yes/no; if yes, specify the domain: full-state, local, in a set or sub-set?)
• Impact of number and placement of actuators on stability, controllability, and/or reachability: (high, moderate, or low)
  • High impact: Requires high number of actuators (>10 grippers) optimally located.
  • Moderate impact: Requires sufficient actuators (5–10 grippers) in strategic locations.
  • Low impact: Achievable with few actuators (1–4 grippers) or inconvenient placement.
• Is reachability considered in the definition of target shapes?
  • Highly considered: Explicit definitions of target shapes from reachable region analysis.
  • Moderately considered: Some discussion of how target shapes consider reachability.
  • Non-specified: No explicit specifications.
  • Note: Researchers often use robots to obtain shapes with specific gripper placements, then use these as target shapes.

Reproducibility (1–5 scale)

Implementation Complexity:

• 1: Very easy, no special requirements (e.g., PID controller).
• 2: Simple, minimal expertise (e.g., Jacobian-based controller).
• 3: Requires some expertise (e.g., Model Predictive Control).
• 4: Complex, advanced knowledge (e.g., Reinforcement Learning).
• 5: Highly complex, deep expertise (e.g., Koopman-based non-linear control).

Transferability:

• 1: Easily transferable, no modifications.
• 2: Mostly transferable, minor adjustments.
• 3: Requires parameter tuning.
• 4: Hard to transfer, significant modifications.
• 5: Very difficult, extensive changes required.

Resource Requirements:

• 1: Minimal (<10 MB, basic laptop).
• 2: Low to moderate (10–100 MB, standard desktop).
• 3: Moderate (100 MB–1 GB, moderate computational power).
• 4: High (>1 GB, high computational power).
• 5: Extremely high (specialized hardware, large-scale computing).

Training and Setup Time:

• 1: Very short (<1 hour).
• 2: Short (1–4 hours).
• 3: Moderate (4–12 hours).
• 4: Long (12–24 hours).
• 5: Very long (>24 hours, extensive training).

Examples:

• Jacobian-based online updated strategy: implementation=2, transferability=1, resources=2, setup time=1.
• Deep reinforcement learning controller: implementation=4, transferability=4, resources=4, setup time=5.

Scalability (1–5)

• 1: Maintains high accuracy and fast convergence across wide range.
• 2: Generally consistent with minor variations.
• 3: Moderate consistency, noticeable changes.
• 4: Significant variation with object size or actuator numbers.
• 5: Poor scalability, drastic performance changes.

Additional Constraints

• Real-time performance: Specify the controller's processing time [ms] or frequency [Hz].
• Practical constraints: Does the method incorporate collision avoidance, object strain limits, and other feasibility considerations? Clearly specify.

Action Type

• Describe the types of control outputs: end-effector position control, velocity control, or other types.
  • For example, while position-based control is common, it may not be suitable for tasks involving highly inertial deformations (e.g., cloth hanging from a gripper). Force-based actions are suitable for considering stress limits in delicate objects.

Dimensions and Degrees of Freedom (DoF)

• Specify the dimensions (1D, 2D or 3D) and number of DoF involved in the control actions.
  • Note: In the existing literature, shape control usually takes place in 2D planes (2D translation + 1D rotation = 3 DoF). The extension to 6 DoF introduces complexities due to multiple solutions and singularities.

Object Contact Type

• Clearly specify the type of actuator-object contact: how does it actually occur in reality, and how is it taken into account in the control system?
  • Examples: A common assumption is to consider gripper-object contact as a single point, although alternative approaches consider larger contact areas. Some methods incorporate contact points with the environment (tables, obstacles) as collaborative actuation elements. It is essential to explicitly describe how contact information is processed by the control system.