1、Microscope Self-calibration Technique for Tele-micromanipulation System Jianhua Wu and Jiaru Chu Department of Precision Machinery and Precision Instrumentation University of Science and Technology of China Hefei, Anhui 230027, P.R.China Email: Abstract - Based on a modified Tsais two-stage algorit
2、hm, this paper presents a self-calibration method for tele-micromanipulation system. Instead of using a standard calibration plane, the coplanar points are generated by the motor-driven working stage (or micromanipulator). The calibration algorithm recovers parameters by considering not only the pla
3、nar position errors of the tracked points but also the errors of yaw and pitch angles. A scale factor optimization process is applied because the scale factor has a significant effect on the recovered yaw and pitch angles while the system is near-parallel configured. Our calibration algorithm is ava
4、ilable even if we have no already known parameter about the vision system. A submicron average calibration error can be achieved with the configuration of a long working distance monocular microscope (using 4X optical magnification) and a common 1/3” CCD camera. Index Terms - camera calibration, opt
5、ical microscope calibration, tele-manipulation system, scale factor. I. INTRODUCTION In order to resolve the relationship between the 3-D global coordinates and their corresponding image coordinates in machine vision, camera calibration is performed to recover the imaging model parameters. The calib
6、rated explicit parameters can also be used to measure the relative position of the camera in global coordinate. The presented calibration methods, such as the well-known Tsais two-stage algorithm 1 and Zhangs multi-plane method 2, show their excellent performance in normal camera calibration situati
7、on. Unfortunately, those calibration methods cant be directly applied to calibrate a microscope system because the object plane is nearly parallel to the image plane, and we cant capture the work scene in two or more different positions. Several new methods 3-6 based on either Tsais method or Zhangs
8、 method have been developed to solve this problem and submicron calibration accuracy has been obtained. However, all those methods need part of the parameters are already known or have been previously calibrated. Furthermore, the precision of yaw and pitch angles calibrated by some of these methods
9、are too bad 3 to be applied to micromanipulation system while the camera is near-parallel configured. This is because the distance between the micromanipulator and the substrate is as small as ten microns or below hence their distance must be precisely forecasted to avoid crash. So accurate yaw and
10、pitch angles are necessary for an automated tele-micromanipulation system. Moreover, due to failing to obtain data from the manufactures, or the obtained data is not precise enough, it is a common scene that we have no exact value of the objective focal length, the scale factor, the dimension of dis
11、crete CCD sensor cells, and so on. For a tele-micromanipulation system, the hardware is often placed in a remote place or a closed chamber, it is not convenient to put a standard calibration plane under the microscope. How to keep the standard calibration plane and the real object plane in the same
12、position is also a big problem. Therefore, a self-calibration method independent of any known parameters is expected. Based on a modified Tsais two-stage algorithm, this paper presents a new microscope calibration method for tele-micromanipulation system. It gives five extrinsic parameters (three ro
13、tation angles and two translation parameters) and three intrinsic parameters (magnification ratios in two directions and the first order radial lens distortion factor) within several minutes even if we have no previously known parameters. Instead of using a standard calibration plane, the working st
14、age is controlled to move along x and y directions to generate coplanar points. An arbitrarily selected feature on the substrate is tracked with a subpixel accuracy. An initial scale factor is calculated by a new constraint based on near-parallel assumption. Then the optimum scale factor is found by
15、 considering not only the planar position errors of the image points but also the errors of yaw and pitch angles. Finally, five extrinsic parameters are calculated by radial alignment constraint (RAC). Three intrinsic parameters are recovered by solving over-determined linear equations. A submicron
16、average calibration error can be achieved with the configuration of a long working distance monocular microscope (using 4X optical magnification) and a common 1/3” CCD camera. II. MICROSCOPE VISION SYSTEM MODEL In order to measure the 3D position of the observed objects, many micromanipulation syste
17、ms use multi-view configuration 7. Two or more microscopes are utilized to observe the same work scene in different directions. If all the cameras are tilted, the system can be calibrated by classical Tsais method or Zhangs method. If there is a near-parallel configured main camera, while all other
18、accessorial cameras are tilted, the main difficulty is how to precisely calibrate the main camera. It is similar to the single camera configuration (the single camera is often near-parallel configured.). So only the near-parallel configured single camera situation is considered in this article. Figu
19、re 1 shows the perspective model of a microscope vision system. It is similar to Tsais model: The transformation of a global coordinate point (xg,yg, zg) to objective coordinate point (x, y, z) includes a rotation matrix R and a translation matrix T (note that zgis zero for coplanar points), =+=1323
20、122211211ggzyxgggyxTrrTrrTrrTzyxRzyx, (1) where, Tx, Tyand Tzare the position of the origin point of the global coordinate in the objective coordinate. z also can be written to zzTz +=, (2) where, ggzyrxr3231+= . For a perspective model, the coordinate transformation from the focal plane (x, y) to t
21、he ideal image plane (u, v) is zzzozzzoTTMyvTTMxu+=+=, (3a) where, Mois defined to be the optical magnification ratio. Since the optical magnification ratio cant be distinguished from the total system magnification ratio even if we know the parameters in image capture process, Mois appointed arbitra
22、rily. Ifzis much small than Tz, a weak perspective model is a good approximation for perspective model, yMvxMuoo=. (3b) Usually, modern microscope lens are precisely machined and only the first radial distortion needs to be considered. The relationship between real image plane (u, v) and ideal image
23、 plane is ( )()212111rkvvrkuu+=+=, (4) where, k1is the first order radial distortion coefficients and r2=u2+v2. The final digital image is grabbed by the image acquiring system from CCD sensor array. Since the proportion of the image width and image height maybe not equal to that of the sensor colum
24、ns and sensor rows, two proportional coefficients need to be introduced: Mxand My. Their units are pixel per micron. The relationship between the digital image coordinate (X, Y) and the real image point (u, v) is written as vMYuMXyx=. (5) Fig. 1. Microscope geometric model Note that the origin point
25、 of (X, Y) is selected to the image center, which is the pixel locating on the microscope optical axis. Its usually set to the center of the grabbed image. The proportion of Mxand Myis defined to be the scale factor yxxMMS =. (6) So we have eight parameters (, , , Tx, Ty, k1, Mx, My) to be recovered
26、 for a weak perspective model or nine parameters (Tzis included) to be recovered for a perspective model. III. PARAMETERS SOLUTION A. Compute the initial scale factor. Substituting (1), (3a), (4), (5) into (6) gives, xyggxggSTyrxrTyrxrYX+=22211211. (7) Equation (7) is also called radial alignment co
27、nstraint (RAC). By separating the known parameters and unknown parameters, we have XTrTrSTTSTrSTrXyXxYYyYxyyxyxxyxygggg=22211211. (8) Let vector c1 c2 c3 c4 c5Tbe the solution vector of (8). Five combined unknowns can be solved from n (n5) points. There are six independent parameters in the five com
28、bined unknowns. An extra equation is needed to recover these individual parameters. Fig. 2. A pair of intersection lines in object plane, image plane and grabbed digital image. If k1and k2are slopes of a pair of intersection lines in object plane (Figure 2). The slopes of their images in image plane
29、 are K1and K2. 1is the angle between k1and K1. 2is the angle between k2and K2. 1and2are very close to the tilt angle for a near-parallel configuration. By ignoring lens distortion, the sum of their differences to nearly independent to their slopes while the two lines are perpendicular (k1k2= -1) in
30、object plane (the proof is omitted because it is easy and lengthy to be proved), ()() 0sinsintantan21+ . (9) K1and K2are calculated from the fitted line slopes in captured image: 2211KSKKSKxx=, (10) where, K1and K2are fitted line slopes in captured digital image. Another relationship between and Sxc
31、an be obtained from the solution vector of (8), xScc tan14=. (11) So an estimation of scale factor can be obtained from the fitted line slopes by substituting (10) and (11) into (9). Once the scale factor is determined, five extrinsic parameters ( , , , Tx, Ty) can be computed from the solution vect
32、or of (8) 1. B. Search for the best scale factor An inaccurate scale factor trends to seriously enlarge or reduce and for a near-parallel configuration, their errors can be up to ten degrees 3. Since the true values of and are usually smaller than a few degrees and z variation (z) is small, a more a
33、ccurate scale factor which nears the initial value can be found by minimizing the z variation. The z variation is defines as zzz =. (12) By ignoring the lens distortion and substituting (4), (5) into (3a), we have ()()zzzyooxYXTTMYXyMxMS +=+. (13) Myand Tzcan be extracted from the least square solut
34、ion of (13). Then Mxis obtained from (6). For a long working distance objective lens, the difference between image points calculated by (3a) and (3b) are a few microns or less because the yaw and pitch angles are usually as small as a few degrees or below. So its nearly impossible to recover an accu
35、rate extrinsic parameter Tz. Nevertheless z has a strong constraint on the yaw and pitch angles. Hence Tzis needed in order to improve and . Once we have all parameters for each scale factors, the optimum scale factor can be found by minimizing the average calibration error. For each Sxnearing the f
36、irst refined scale factor, we can obtain all the extrinsic parameters (including Tz) and intrinsic parameters (k1is assumed to be zero). Then their calibration errors are calculated (by using perspective image model). The best Sxis selected while the minimized calibration error is obtained. C. Recov
37、er all parameters by substituting the optimum scale factor After the scale factor is determined, the five extrinsic parameters ( , , , Tx, Ty) are recovered by substituting the final scale factor into (8). We have two options about Tz. If the working distance of the objective lens is at hand, Tzis a
38、ppointed to be the working distance and the ideal image points are obtained by a perspective model (3a). Or a weak perspective model (3b) is used because the computed Tzis unauthentic (it is usually seriously reduced according to our experiments.). The computed Tzis only used to improve the scale fa
39、ctor. By substituting (4) into (5), and adding two equations, we have YXMMkvurYXyx+=+12)(. (14) The values of k1, Mxand Myare computed by (14). Their values are affected by the arbitrarily appointed optical magnification ratio. IV. CALIBRATION EXPERIMENTS AND RESULTS The calibration experiments were
40、 based on our micromanipulation system. The working stage is consisted of three close loop controlled motor-driven positioning stages (two PI M-126 stages for horizontal motions, one PI M-110 stage for vertical motion). Their positioning accuracies are sub micron and their positioning feedback accur
41、acies are about 10 nm. The microscope is a monocular long working distance (about 50 mm) microscope. Which can offer magnification ratios ranging from 2.8X to 18X. The camera is a color CCD camera (Panasonic WV-CP410) with a 1/3” sensor array. The captured image size is 720 by 576 pixel. After the v
42、ision system was auto focused, an arbitrarily selected feature on the substrate was selected to track. The working stage was controlled to move along x and y directions for four hundred times to generate coplanar points. The feedback positions of the stages were recorded as global coordinates. The s
43、elected feature was tracked with a sub-pixel accuracy and they were recorded as the image coordinate (Figure 3). Though the parameter recover process will reduce the calibration errors to a low level whether a good estimated scale factor or a bad estimated scale factor is used, the recovered paramet
44、ers will seriously deviate from their true values if the scale factor is bad estimated. In order to reduce the errors of calibrated angles, high precision pattern matching technology is essential for our calibration algorithm. Because the tracked feature is arbitrarily selected, regular lines and re
45、ctangles cant be obtained at all time. We just use a pattern matching algorithm base on gray difference. A small bright area surrounded by a dark area (or a small dark area surrounded by a bright area) is a good feature. The gray centroid coordinates 8 of the selected area are considered to be the f
46、eature positions in our software. This method is very simple and its precision is good since only translation motions are applied on the tracked feature. Because our coplanar points are regular lattices, the perpendicular lines can be found in these points directly. If a pair of perpendicular lines
47、cant be found or the slopes of these lines are too small or too large, the perpendicular lines should be generated purposely. Table 1 shows the scale factors during the different calculation steps. Figure 4 shows how the scale factor influences the yaw and pitch angles and how z varies. The calibrat
48、ion error in image plane is defined as the distance between the calculated points and the tracked points ( ) ( )22YYXXErrimage+=, (15) where, (X, Y) are the calculated image points by using recovered parameters. Similarly, the calibration error in object plane is defined as ( )( )22ggggobjectyyxxErr +=, (16) where, (xg, yg) are the calculated points on working stage by using recovered parameters. Table 2 shows the recovered parameters and calibration errors
