Nonprehensile Underactuated Self-Assembly
by Designing Stable Limit Sets
1. Motivation and Background
Robotic manipulators currently play a major role in manufacturing systems by assembling parts. However, with the trend of parts becoming smaller and smaller, it becomes economically infeasible to have robots individually manipulate and assemble parts that may be on the micron, even nano scales [3]. By using nonprehensile (without force closure grasping) manipulation, we do not have to pick and place each individual part meaning that the part can be manipulated in more degrees of freedom than the robot has.
One advantage of this type of system is cost. Hardware and sensing mechanisms become simple and inexpensive. By utilizing friction and the natural dynamics of the part, the robot can be as simple as a 1 DOF arm that manipulates parts by throwing them. Each additional part to assemble increases the number of degrees-of-freedom that are being controlled while keeping the control laws simple. In addition minimal sensing and feedback can dramatically improve assembly performance. Nonprehensile, underactuated self assembly is a paradigm shift from traditional robotic prehensile manipulation that can allow for cost effective miniaturization and simplification of assembly.
In our research, we have used a 1 DOF arm to point stabilize and assemble parts by throwing and catching them on an air hockey table (Fig. 1). In both cases, the parts were manipulated with feedback and without feedback. The existence of stable limit sets [2] provided the background to the experiments conducted in this research project. These experiments have provided a base from which more analytical design of stable limit sets can be built upon.
Figure 1: Top view of Flatland
2. Designing Stable Limit Sets
Control laws have traditionally been designed by linearizing smooth functions and solving ordinary differential equations. However in dealing with throws and impacts, the dynamics are non-smooth and nonlinear, thereby making it impossible to control in the classical sense. The major tool that we used to approach this problem was previous work done by Lynch, Northrop, and Pan [2] in finding stable limit sets in a dynamic parts feeder. They found that repeated throws and catches of parts with an arm caused the parts to demonstrate a unique cyclic pattern from a large set of initial conditions. This cycle consists of jogs (small forward motions while the part remains in the same orientation) and flips (counterclockwise rotation that also moves the part backwards toward the arm joint. Figure 2 shows a simulation of 300 throws demonstrating a stable limit set for a right triangle (30,60,90) [2]. In this example the cyclic pattern is as follows: the part first jogs forward on edge 1, flips counterclockwise onto side 0, flips counterclockwise onto edge 2, may or may not jog forward, flips counterclockwise once more onto edge 1 and repeats its cycle. Thus a stable limit set is a generalized equilibrium where we are guaranteed to find the part. Also, many initial positions and orientations of the part converge onto the same limit set.
For a part, a critical point for a particular side is defined as the distance on the arm away from the motor joint before the part flips. A part on a particular side jogs forward on the same side until it hits the critical point, and then flips on to another side. In fig. 2, the critical point for edge 1 is at 23 cm, 16 cm for edge 0, and 14 cm for edge 2.
Figure 2: Stable limit set of triangle [2]
The next step after showing the existence of these stable limit sets is to develop analytical ways to design a stable limit set. For a given part geometry, we would like to know where to put the center of mass and how to shape the part to achieve critical points at a desired distance on the arm for each side. In the experiments, the parts were not designed analytically, but rather used a combination of simulation and a trial and error approach to discover the basic effects of center of mass position and shape of part. This method was reasonable because it was fairly easy to adjust the center of mass on our parts by attaching weights onto them. The experimental work gave us a better intuitive sense as to where to begin the analytical design of the parts.
3. Experimental Setup
Figure 3: Isometric sketch of Flatland setup [1]
The Flatland setup (Fig. 3) is composed of a large air hockey table that can support objects with a nearly frictionless air bearing. The angle of the table is adjustable providing a control on the effect of gravity. There is an extruded 80/20 aluminum superstructure which the lights and camera are mounted on. The camera sends frames to the Cognachrome vision board that sends information about the objects that the camera is tracking.
The experimental test bed used for these experiments was also used in Craig Black's Master's Thesis [1] entitled: Planning and Control of Planar Batting using Nonlinear Optimization. More detail on the setup can be found there in Appendix A on pages 65-84.
In our specific setup, we are only using the 1.1 amp HarmonicDrive RH-8 3006 motor and a single arm. The motor amplifier used is Titech robot driver [5]. The encoder's resolution is 400000 counts per revolution with quadrature. I/O is done with a Servo-to-go card (hardware manual). The triangular parts are 1/8" thick 30-60-90 triangles with the length of the hypotenuse being 6 5/8" long. All the parts are Lexan and were cut using a bandsaw. Modifications to the arm were made by cutting and taping 1/4" thick Lexan parts to the existing arm.
3.1 Vision System
Up to three different colors may be tracked at the same time using the Newton Lab’s Cognachrome system. The performance for tracking one object is 60 Hz. After a trial and error process, we decided to use light blue and dark blue as the colors that we wished to track. Training the colors is done by placing the desired color in the center of the camera’s view and taking a sample.
The Cognachrome system communicates with the computer via the RS-232 serial line. Various information about the part can be transmitted including position of the centroid, aspect ratio, angle, and size (a full list is in the Cognachrome manual [4] on page 32, Section 5.3.1). For our purposes, we gather the row and column coordinates of the centroid, the frame number, and the angle of the object. We use the angle information to determine the side on which the part lands.
There are two methods of tracking the part. For single part experiments, the positions of two circular blobs of different colors are used to calculate the angle of the part. In the experiments with two parts, there is one long color blob with a high aspect ratio on each part. The angle is determined in the Cognachrome hardware and is used to calculate the side that the part is on. However, the limitation to using the angle data from the Cognachrome hardware is that the range of the data is limited to 180 degrees. This means that the angle data is sometimes ambiguous. This can be corrected by tracking more than one object of the same color at a time and using two circular blobs to receive full orientation data (see notes in section 3.5). The correction was not implemented for these experiments in the interest of time.
The protocol string allows the user to specify the format of the vision data. As a compromise between ease of debugging and efficiency, we chose to use a hexadecimal format. The ‘$’ character signifies the start of a frame. The capital letters are the variables that the camera is tracking. The numbers specify how many places of the data are displayed. Lowercase x, y, s, u, v, and t specify a marker before the corresponding data.
Protocol String:
h: protocol header = “$%02xG”
a: protocol chan A = “x%04xXy%04xYs%02xA”
b: protocol chan B = “u%04xXv%04xYt%02xA”
c: protocol chan C = “ ”
For more information on the vision system, see Section 5, page 25 of the Cognachrome user manual [4].
3.2 Software
The software for the system was written in C and implemented in the QNX real time operating system. The program uses drivers in QNX for the Servo-to-go card (hardware manual) that were ported by Tom Worsnopp. Each program is timed using a semaphore and runs at a 10 millisecond sampling rate. The throw is broken up into several stages labeled: hold, lower, and throw. Each stage runs for an amount of time that can be adjusted. The hold stage allows the parts to come to a rest on the arm and then records what the current state of the parts are. The vision data is received from the serial port with the “getc(ser1)” command. Once the header character ‘$’is seen, all the data for that frame is dumped into an array and converted into decimal numbers.
Point Stabilization without Feedback (Pswof.c)
Point Stabilization with Feedback (Pswf.c)
Other necessary files: servotogo.c, servotogo.h, defines.h, prototypes.h, globals.h
Note: Download all of these files to the same directory. The code for the four experiments are nearly identical. See pswf.c first.
This code implements a simple PD controller in a thread timed with a semaphore. It is meant to be a base from which more complicated code can be built up from. The program demonstrates the most basic way to do timed control of motors with a Servo-to-go card in QNX.
3.3 Color Training and calibration:
Cut out a circular blob
Train the Cognachrome system. This is done by pressing 's' in the “tpuvis” window while placing the desired color in the middle of the middle of the camera viewing area. Tweaking is done by using '<' and '>' to shrink or grow the tracking color threshold. Save by typing 'S'. (see Cognachrome manual [4] section 5 for more information).
Run code that outputs coordinates (in R and C) of blob
Record the values for 3 or more points on the table (I used 4 points and utilized the air holes as a global coordinate system as they were placed at 1" centers)
Use matrices to describe relationship between rows and columns and x and y coordinates.
where
and
With 4 data points, the matrix becomes:
6. Solve for A and B (I used MATLAB using the pseudo inverse function).
A different description of the calibration process can be found on page 82 in the appendix section A.4.5 of Craig Black’s thesis [1].
3.4 10 steps to run an experiment:
Plug in power to lights, air hockey table, amplifiers, television (for debugging), Cognachrome vision system fan, and CCD camera. Leave Cognachrome power unplugged (white circular plug).
Turn on the Motor Power, Amp Power, and DAC 1 on the front face of the amp box.
Boot the computer in Windows 2000 (boot option 1).
Start the program C:/Program Files/arc/win32bin/arc.exe. Download ARC here locally (When enabled with a Mini-ARC license key, this software allows you to download new software to your processor board, and interact with it over a serial port.) or from the Newton Labs website at http://www.newtonlabs.com/arc/arc_download.html.
Plug in power to the Cognachrome (will boot up vision software)
In the “tpuvis” window, type ‘q’ to stop recording.
Check prototype strings by typing ‘p’. This is the format in which the data will be sent to your computer and only needs to be checked once. (see section 5 of Cognachrome manual [4] for more information and options). If you are using the code in this write-up, the prototype strings should match those of section 3.1. To change the strings, type the letter of the channel that you wish to modify and type in the new prototype strings. If you don't need to change anything, just press enter. After making any changes, remember to save by typing 'S'.
Type ‘r’ to begin recording. A telnet window should start streaming vision data.
Close the main ARC terminal and reboot the computer in QNX (boot option 2, default)
In terminal, set the correct baud rate by typing the command "stty baud=38400 </dev/ser1"
Make sure that the part(s) are in the view of the camera.
Compile and run code. Include curses and math libraries in the compile line (these are standard and no additional files are needed). Here is an example of compiling and running awf2.c:
Compile: “qcc awf2.c servotogo.c –lncurses –lm –o awf”
Run: “./awf”
The screen will display that the Servotogo card has been initialized and that the different threads have started. Then tracking data (centroid locations, angle of part, side of part, throw number) will be displayed on the screen and the arm will start throwing.
3.5 Known bugs (as of 6/6/03) and other notes:
Currently the programs do not clear the buffer of the camera and thus there are sometime memory effects from experiment to experiment. Running the program a few times seems to eliminate the problem.
Code does not yet account for tracking more than one object of the same color, but is possible. This will allow for more accurate part orientation data. Perhaps “size of blob” can be used to differentiate between two parts of same color.
Tracking data is very sensitive to the amount the camera iris is open.
Once in a while, retraining of the colors is necessary to give good data.
Some occasional glitches in data that give far off numbers. Perhaps due to how the data is read from the serial port.
Sometimes the amplifier offset settings drift and need to be readjusted (input offset VT2). This is done by adjusting the potentiometer on the driver with a small Phillip's screwdriver until the arm doesn't drift anymore. (See Titech robot driver user manual for more information [5].)
Camera needs to have object in view for code to work.
Would like to add limit switches and emergency stop.
Covered some of the air holes to produce more air from the table.
For Cognachrome support, contact Newton labs: 425-251-9600, support@newtonlabs.com
4. Experiments
The following four experiments represent a progressive path where self assembly is the final and most difficult experiment. In each case, the arm approximates a step input throw by controlling the position of the arm (using a PD controller) such that the maximum overshoot is when the arm is approximately horizontal.
4.1 Point Stabilization with feedback (data, Quicktime, MPEG4)
The part dimensions are shown below:
Where:
| AB = 5.75" |
| BC = 6.75" |
| AC = 3.625" |
| angle CAB = 90 |
| angle ABC = 30 |
| angle BCA = 60 |
| CM = (1.92,1.21) |
Using the general knowledge of limit sets, we know that after a finite amount of time, the part lands on its long side. By setting the energy of the first set of throws high enough, we can ensure that the part will flip onto its long side before the end of the arm. Once the part lands on its long side, the vision system recognizes this fact and the controller switches to lower energy throws until the part reaches a desired position on the arm. During this second controller, the energy of the throws is only sufficient to jog the part forward and cannot flip the part. In this way, the final position and orientation of the part can be controlled. Figure 4 shows the stopped arm after the part has reached the desired position and orientation.
Figure 4: Point stabilization with feedback
Below is a plot showing the side of the part versus distance on arm (Fig. 5). We can see the part start on side 2 and jog forward until its critical point at about 14.5 inches. At this point, the part flips counterclockwise and onto side 1 where the controller switches to low energy throws. The part jogs forward until it reaches the end of the arm at about 16 inches.
Figure 5: Data showing side vs. distance on arm (with feedback)
This experiment was repeated 20 times with the same initial conditions and was successful 100% of those times. The margin of error for the end position of the part on the arm was 0.5".
4.2 Point Stabilization without feedback (data, Quicktime, MPEG4)
Part dimensions shown below (note change in CM location):
Where:
| AB = 5.75" |
| BC = 6.75" |
| AC = 3.625" |
| angle CAB = 90 |
| angle ABC = 30 |
| angle BCA = 60 |
| CM = (2.13,1.13) |
Once again, we use our previous knowledge of limit sets to stabilize the position and orientation of the part. However, this time, there is no feedback involved. Instead, we utilize the fact that the critical point of the long side is farther out on the arm than the critical points for the other sides. If we have the critical point of the long side off the end of the arm, then the part will continue to jog and flip as long as it is not on its long side. If the part is on its long side, then the part will fall off the arm into a cavity that is deep enough that the throws do not have enough energy to eject the part (Fig. 6). To achieve better results, we shift the center of mass by adding a weight to the part. In essence, this allows us to maximize the spread between the critical point of the long side and the other critical points.
Figure 6: Point stabilization without feedback using cavity
Figure 7 shows a plot of the side of the part versus the distance on the arm. We can see that the part starts off on the short side (side 3) at about 12.5 inches on the arm. The first throw flips the part onto side 2 and moves the part backwards on the arm. Subsequent throws jog the part forward until the critical point at 18 inches is reached. The part flips counterclockwise again and then jogs forward again until the part falls into the cavity at 24.5 inches on configuration 0 (stabilized in the cavity).
Figure 7: Data showing side vs. distance on arm (without feedback)
This experiment was repeated 20 times with the same initial conditions and was successful 95% of those times (19/20). The one experiment failed because the part hit the wall of the cavity. There is a tendency for this to occur when the center of mass of the part is very close to the end of the arm before the part falls into the cavity.
4.3 Assembly with feedback (data, Quicktime, MPEG4)
Part dimensions are the same as those for point stabilization with feedback.
In this experiment, the first controller throws two parts randomly on the arm until both are in the right orientation and order for assembly (Fig. 8a). Once the camera senses this configuration, the controller switches to lower energy throws as well as a lower angle of release. Once the parts reach the end of the arm, one part climbs the other (Fig. 8b) until the two right triangles are assembled into a rectangle (Fig. 8c). This experiment assumes that both parts will land in the “correct” configuration after a finite amount of time. This is related to notions of ergodicity from dynamical systems theory. Also, this case involves many impact interactions between the two parts leading to several configurations where the part did not land flat on one side. We ignored these interactions in our control law and only switched the controller when the parts were in the correct configuration.
Figure 8a: Assembly with feedback: correct orientation
Figure 8b: Assembly with feedback: climbing
Figure 8c: Assembly with feedback: assembled
To make the most sense of the data, it has been plotted in two different ways. First, the side of the parts is plotted against the throw number (Fig. 9). The dotted lines with the square data points show the path of the first part, and the solid lines with the star data points show the second part. From the figure, we can see that both part positions and orientations were randomized until about the 33rd throw. Here, the controller changes and parts remain on the same side until assembly at throw 50.
Figure 9: Data of side vs. throw in assembly with feedback
The second way the data can be plotted is with distance on arm against throw (Fig. 10). Once again, the dotted lines show the first part and the solid lines show the second part. We can see that the parts are at random distances on the arm until the 33rd throw where the two parts converge into their assembled state.
Figure 10: Data of distance on arm vs. throw for assembly with feedback
This experiment was repeated 20 times with the same initial conditions and was successful 70% of those times (14/20). The two reasons for failure were that the controller switch occurred before the parts were in the correct orientation, and because the arm was lowered too rapidly during the controller switch. Possible solutions may include tweaking the motor motions, waiting longer before capturing data, or making the data more consistent.
4.4 Assembly without feedback (self assembly) (data, Quicktime, MPEG4)
Part dimensions shown below:
|
Where:
|
 Where:
|
The approach to solving this problem is to first point stabilize one part using the same methodology as the single part point stabilization without feedback. As the first part is guaranteed to be stabilized after a long time, the second part is placed on the arm after the first. If the limit set of the second part is designed correctly, the two parts will fall into the cavity in the arm in an assembled state (fig. 11). The difficulty comes in dealing with multiple part interactions and balancing between the extremes of very low energy throws that do not provide stable limit cycle behavior and high energy throws that eject the parts from the cavity.
Figure 11: Assembly without feedback
This experiment was repeated 20 times with the same initial conditions and was successful only 55% of those times (11/20). Two of the most common failure modes included the second part being ejected from the assembled state on the next throw and the second part becoming stuck in the cavity in an incorrect orientation. Once the parts were in their assembled state, the parts remained assembled after the next throw 85% (17/20) of the time.
5. Conclusion
Our knowledge of stable limit sets for a triangular part being thrown on a 1 DOF arm was used successfully to control parts both open loop and closed loop. Also, it was found that stable limit sets can be designed such that the critical points of the part can be manipulated to produce desired behaviors. The experiments have provided a starting point for leading the direction of more analytical design of parts for self assembly. It is interesting that such a simple throwing arm (and in the cases with feedback, simple control laws) could control the position and orientation of parts. Moreover, getting two parts to assemble with open loop control has been exciting considering the difficulty of the task, even though it was not yet robust.
6. Future Work
The next stages in this work may include developing parameters that can determine part’s fitness for self assembly. This will be useful for the design engineers who will then be able to design parts that lend themselves to easy self assembly. Another long term goal of this research is to expand the control laws to be able to control multiple parts, ideally hundreds. In addition, the complexity of the parts being assembled should be increased in future experiments. In the future, one can imagine a single arm that is able to assemble hundreds of parts together with almost no feedback. For the case for assembly with feedback, we assumed ergodic behavior. Upcoming research should attempt to show that such behavior exists for these parts and discover their rate of convergence.
7. Bibliography
[1] C. K. Black. Planning and Control of Planar Batting using Nonlinear Optimization. Master’s Thesis, Northwestern University, 2000.
[2] K. M. Lynch, M. Northrop, and P. Pan. Stable Limit Sets in a Dynamic Parts Feeder. IEEE Transactions on Robotics and Automation, Dec. 2001.
[3] G. M. Whitesides, and B. Grzybowski. Self-Assembly at All Scales. Science, Vol. 295, Mar. 2002.
[4] A. Wright, R. Sargent, C. Witty. Cognachrome Vision System User’s Guide. Newton Research Labs, www.newtonlabs.com/cognachrome, Jun. 1996.
[5] Titech Robot Driver User Manual. http://www.okatech.com/en/index_en.html. Mar. 1996.