Research ArticleENGINEERING

Scalable tactile sensor arrays on flexible substrates with high spatiotemporal resolution enabling slip and grip for closed-loop robotics

See allHide authors and affiliations

Science Advances  13 Nov 2020:
Vol. 6, no. 46, eabd7795
DOI: 10.1126/sciadv.abd7795

Abstract

We report large-scale and multiplexed tactile sensors with submillimeter-scale shear sensation and autonomous and real-time closed-loop grip adjustment. We leveraged dual-gate piezoelectric zinc oxide (ZnO) thin-film transistors (TFTs) fabricated on flexible substrates to record normal and shear forces with high sensitivity over a broad range of forces. An individual ZnO TFT can intrinsically sense, amplify, and multiplex force signals, allowing ease of scalability for multiplexing from hundreds of elements with 100-μm spatial and sub–10-ms temporal resolutions. Notably, exclusive feedback from the tactile sensor array enabled rapid adjustment of grip force to slip, enabling the direct autonomous robotic tactile perception with a single modality. For biomedical and implantable device applications, pulse sensing and underwater flow detection were demonstrated. This robust technology, with its reproducible and reliable performance, can be immediately translated for use in industrial and surgical robotics, neuroprosthetics, implantables, and beyond.

INTRODUCTION

Versatile manipulation of objects has been achievable only through high spatiotemporal tactile sensing (1, 2). From manipulating sewing needles, to flipping a book page, to grasping and cracking an egg, and to opening a door, the high resolution and dynamic range of sensing at our fingertips have been critical in closing the loop to avoid dropping and slipping items (3, 4). For robots to autonomously operate in our human world, they would need similar tactile perception at their “fingertips” to interact with our everyday objects and fixtures (5, 6). A broad range of applications requires tactile sensitivity on flexible substrates that can adhere to arbitrary geometries (7). Although multiple modalities have been used in the past to indirectly infer tactile information (8, 9), direct tactile sensation is needed to increase fidelity and feasibility for haptic feedback on small tools to interact with delicate objects or manipulations (10). For direct tactile sensors, existing approaches can provide force, but shear sensitivity has been more challenging (11, 12). Tactile sensors that can sense both tactile forces and multiplex signals without additional driving elements can overcome trade-offs for density and performance. They can be engineered to sense shear at the required high spatiotemporal resolution, thereby enabling new levels of performance and applications for robotics (1316). However, the realization of tactile sensors on flexible substrates with shear sensitivity poses substantial manufacturing challenges particularly when high spatial resolution and compact integration for a scalable manufacturing are necessary (1719).

Zinc oxide (ZnO) is a semiconductor whose conductivity can be tuned to switch or amplify electronic perturbations and additionally has a high piezoelectric coefficient of 9.93 pmV−1 for force sensitivity. These features enable a single ZnO thin-film transistor (TFT) to sense, amplify, and multiplex force (16, 2024). In addition, various low-temperature deposition technologies allow ZnO TFTs to be readily integrated with flexible substrates (2527). As a result, ZnO-based materials are deployed in emerging display technologies, such as indium gallium ZnO TFTs (28, 29). Here, we implement advanced ZnO field-effect transistors (TFTs) on flexible substrates for simultaneous high-speed switching elements and tactile sensing to demonstrate slip and grip with robotic fingers across various scales. Pulse sensing and underwater flow detection were also demonstrated for its application in biomedical and potentially in implantable devices.

This work is disruptive because (i) it is the first to close the loop in robotics using shear sensors, (ii) it has much higher spatial resolution and low cross-talk than previous state-of-the-art technologies, (iii) it has much higher frame refresh rate, and (iv) it leverages in a unique way a scalable material and process that is developed on flexible substrates while preserving merits (i) to (iii) above and (iv) can be translated for manufacture lines and readily surpasses the performance of previous technologies when all considerations (resolution, speed, power, cross-talk, and sensitivity) are taken together. This is the first work that measures both force and object displacement using the single modality of pressure sensing with a piezoelectric material to close the loop in robotics and to enable grip due to slip, a demonstration that has not been accomplished before with force sensors reported in the literature. The compilation of sensory feedback in a high density, a small form factor, and a single package and with high performance provides the sensory feedback necessary to regulate the stability of objects in robotic hands. This critical capability, with necessary and sufficient conditions well defined by force/torque-closure theory in robot grasping literature (30), will ensure stable control of objects in the hand within a packaged form and will accelerate the adoption of our or similar sensors in industrial robotics.

RESULTS AND DISCUSSION

Dual-gate ZnO piezoelectric TFT for simultaneous signal multiplexing and force sensing

The mechanism of operation for force sensing in ZnO dual-gate TFT is schematically illustrated in Fig. 1A. Force applied to the surface of the TFT leads to the formation of a sheet of piezoelectric charges at the surface of the channel. This sheet affects the energy band-edge diagram and changes the flat-band voltage condition where electron accumulation begins. This effectively plays the role of an incremental increase/decrease of the gate voltage, VGS, and a correspondent amplified increase/decrease in the source-drain current, IDS, can be measured and interpreted as a force sensation. The direction of the current change is determined by the polarity of the ZnO thin film and the direction of the strain field generated by external forces. With its dual-gate structure, biasing gate electrodes enabled reliable switching (turn on/off) of the current in the TFT and therefore the ability to multiplex across arrays of elements. Prototype TFTs with a detailed structure of a single element exhibited in Fig. 1B were fabricated on ultrathin (5 μm) polyimide layers prepared on a Si carrier wafer. The step-by-step fabrication process is discussed in the Supplementary Materials and in fig. S1. Briefly, a bottom gate (Cr) was defined before the 50-nm-thick Al2O3 atomic layer deposition for the bottom-gate dielectric layer, Cr/Au metallization for leads, and indium tin oxide (ITO) for ohmic contacts. The ZnO thin film was deposited by low-pressure radio frequency magnetron sputtering at 200°C, and the upper portion of the device is nearly a mirror of the lower portion, but its surface is passivated with parylene C because of parylene C’s low-temperature process compatibility with ZnO TFTs. All layers were patterned by photolithography and wet etching. Last, an additional protection layer of parylene C layer was deposited, and polydimethylsiloxane (PDMS) was applied to further enhance the robustness of the arrays during the characterization.

Fig. 1 Active matrix ZnO piezoelectric TFT array for force sensing.

(A) Schematic illustration of force sensing using a ZnO TFT. (B) Structured layers of the ZnO TFT. (C and D) Representative transfer (C) and output (D) characteristics of the fabricated TFT. (E) Temporal response of the TFT upon different external forces.

The dual functionality of the TFT as a switching element and as a force sensor is illustrated in Fig. 1 (C to E). The transfer curves in Fig. 1C indicate a high Imax/Imin ratio of 106 to 107, a normally off-device (desired for low element-to-element leakage) with a threshold voltage of 2.2 V, and a subthreshold slope of 490 mV/dec, all of which are superior to the control TFTs made with single gates on the same channel material. Figure 1D shows the output curves of the ZnO TFTs showing linear characteristics at low VDS (ohmic behavior; Fig. 2, G to J) and excellent pinch-off characteristics in saturation at high VDS, which are important for linear operation and force responsivity. Similarly, the device exhibited excellent force sensing ability investigated by measuring the current-time curves under the repeated application and release of force, as shown in Fig. 1E. A controlled force (50 to 250 mN) is applied and released for the period of 60 s by a linear actuator equipped with a force sensor. The responses were immediate with sharp rise and fall times, while the amplitude of the response increased as the applied force increased. Detailed discussion about the temporal resolution and sensitivity will be followed in Fig. 3.

Fig. 2 DC characteristics of ZnO TFT with different O2/Ar flow rate ratios.

(A to C) Transfer characteristics and (D to F) output characteristics of the fabricated TFT with oxygen flow rates of 20, 10, and 5% during the growth. (G to J) Gate bias–dependent physical properties of ZnO TFT, (G) sheet resistance, (H) contact resistance, (I) transfer length, and (J) contact resistivity curves of ZnO TFT with oxygen flow rates of 10% (black) and 5% (red).

Fig. 3 Force sensing characteristics of the fabricated TFT arrays for finger mounting.

(A) Photograph of the fabricated active matrix TFT array on a 4-inch wafer. Inset is a magnified image of a single TFT. (B) Schematic diagram of the active matrix structured TFT array. amp, amplification; mux, multiplexing. (C) Element-wise response map of the 8 × 16 TFT array. (D) Cross-talk correlation matrix of the array. (E) Average current change curve under different applied force measured from 10 random areas in the array. Error bars are corresponding SDs. (F) Two-point discrimination study of the TFT array. Top: Illustrations of differently spaced tips used in the experiment. Middle: Response heatmaps under the force applied by the tips. Bottom: Plots of the current change along the dashed line in the heatmaps in the middle row. (G and H) Study on the temporal resolution of the TFT array. (G) Photograph of the experimental setup. (H) Corresponding response heatmap of the TFT when force is applied. (I) (Black dots) Overlapped average responses of the elements indicated by the red box in (H) and (blue curve) time versus z-position profile of the tip. Photo credit: Hongseok Oh, UC San Diego.

To investigate the origin of the force response, we measured force-dependent capacitance versus voltage (C-V) curves. Figure S2 shows the C-V curves of the TFT under 0-, 100-, and 200-mN forces. The curves shifted toward positive direction under applied force, indicating that the piezoelectric charge changed the voltage required to form electron channel, effectively working as additional gate bias. In addition, this indicates that the piezoelectric effect is most dominant compared to other effects such as piezoresistive effect or modulation of contact barrier height.

Optimization of the dual-gate ZnO TFT

We recently demonstrated that the transistor and piezoelectric properties of ZnO TFTs on glass substrates can be tailored by the O2/Ar gas flow rate ratio during deposition (16). This is primarily accomplished by manipulating the number of oxygen vacancies in the film that, when passivated with hydrogen essentially, act as donors for n-type semiconductive film property (31). On flexible VDS substrates, we fabricated ZnO TFTs for different O2 gas flow percentage simultaneously and that otherwise underwent the same fabrication steps. Figure 2 (A to C) shows the typical transfer curves of the TFTs with varying O2 flow percentage of 20, 10, and 5% in an O2/Ar gas mixture. For all cases, fabricated TFTs exhibited a high Imax/Imin ratio of 106 to 107 with negligible gate leakage current (few picoamperes), indicating excellent electrostatic control over the channels. On the other hand, threshold voltage (Vth) was shifted toward negative values as the oxygen ratio decreased, resulting in higher Imax values (Table 1).

Table 1 DC characteristics of the ZnO TFTs fabricated with different O2 flow rates.

View this table:

Figure 2 (D to F) illustrates the output characteristics of the different TFTs showing excellent transistor behavior and decreased currents with O2 percentage due to the positive Vth shift and consequent reduced gate overdrive (VGS −Vth). We elected to use the ZnO TFTs with 10% O2/Ar due to their superior subthreshold slope, which leads to smaller operational voltage range, and due to their lower threshold voltage in the normally off state compared to 20%, which is desired for low-power biasing. We applied the gated transmission line method (TLM) (32) to delineate the contact and channel resistances (see Supplementary Materials and fig. S3 for more details). Figure 2 (G to J) shows the sheet resistance of the channel, contact resistance, transfer length, and contact resistivity of the gated TLM structure, with ZnO thin films deposited under 5 and 10% of the O2 flow rate. Both sheet resistance and contact resistance decreased with increasing gate bias (6 to 10 V), as shown in Fig. 2 (G and H). The sheet resistance was in the range of 0.2 to 1.2 megohms/sq for 5% O2, and 1 to 6.5 megohms/sq for 10% O2. Contact resistance for both types of films ranged from 1 to 12 kilohms for both cases. The increased gate bias accumulates more electrons in the channel and lowers the sheet resistance. On the other hand, the small energy barrier width at the ZnO/ITO interface is also reduced with higher gate bias, resulting in smaller contact resistance. As shown in Fig. 2I, the transfer lengths of each ZnO channel were calculated to be around 1 and 4 μm, which is less than 10% of the total length of ZnO/ITO contact (50 μm). Hence, full extraction of injected carriers is expected at the drain side without current crowding at the ZnO/ITO interface. The specific contact resistance of the interface, shown in Fig. 2J, ranged from 0.05 to 0.15 ohm∙cm2 for the film with 5% O2, and 0.01 to 0.05 ohm∙cm2 for the case of 10% O2. It is worth noting that the inverse subthreshold slope initially decreases with 10% O2 when compared to 20% O2 and then increases at 5% O2. Detailed work on the influence of the O2 partial pressure on the interface trap distribution in the ZnO bandgap and its impact on the transistor characteristics can be found in the work of Kimura et al. (33).

Force sensing characteristics of active matrix ZnO TFT array

To prepare force sensor arrays for mounting on two opposite fingers, we fabricated two of 8 × 16 active matrix force sensor arrays with a pitch of 2 mm on a 100 mm-sized wafer, as depicted in Fig. 3A. Here, each row (drain) and column (gate) correspond to the data line and address line of the active matrix display counterpart (Fig. 3B). By the virtue of its working principle, it offers better performance in a wider dynamic range and higher spatial resolutions and has been extensively pursued (14, 3437). The detailed operation principle of the sensor array and associated hardware and software development are described in the Supplementary Materials and figs. S4 and S5. We mapped the force sensitivity across the entire array by pressing a constant force to every TFT element in the array using an acrylic block with a diameter of 2 mm controlled by a linear actuator with integrated force sensor (see Supplementary Materials and fig. S6 for more details). The heatmap of the current response in Fig. 3C shows uniform sensitivity with an average of 4% difference observed over the entire array except for the 15th column, which presumably had a leaky element. The correlation matrix of Fig. 3D shows strong and distinguishable response along the diagonal yet with additional lines indicative of the response of the neighboring elements due to force spreading by the PDMS layer. Hence, the cross-talk mostly arises from the mechanically distributed stress, not the electrical cross-talk in the TFT array. In terms of sensitivity and linearity, the average force responses from 10 different areas were plotted in Fig. 3E, showing monotonic increase as a function of applied force. [The measurement was done with three-dimensional (3D) PDMS pillar array placed on top for shear force sensing, which will be discussed later in Fig. 4 and fig. S7.] Here, sensitivity was given by the slope of the terminal line, and the linearity error was given by the maximum deviation of the current change as a fraction of full-scale Δymax/ymax, where Δymax is the maximum deviation of the current change from the terminal line and ymax is the maximum current change at the highest force in the given range (38). The sensitivity of the TFT is calculated to be 0.0635 (±0.0148) %/mN with linearity error of 4.35%. The linearity error was minimized at the low force range of 0 to 50 mN, and linear responses were continuously observed in the force range up to 250 mN, which can be found in the Supplementary Materials and fig. S12.

Fig. 4 Shear force measurement.

(A) Strategy of measuring shear force applied on the TFT array using the 3D PDMS bump array. (B) Photograph of the 3D PDMS array placed on the TFT sensor array. Insets: Top view (top) and side view (bottom) of a single bump. (C) Calculation of shear force from the force applied to each sensor in the unit cell. (D) Photograph of the shear force measurement setup. Inset: A close-up view of the tip applying in-plane force to the bump. (E) Average shear force versus current change curve from six random unit cells. (F to H) Element-wise characterization of shear force response for the upward shear forces. (F) Schematic illustration of applying shear force along the y axis. (G) Quiver plot of the calculated force upon the uniformly applied upward force. (H) Correlation matrix of the y-axis component for the cross-talk analysis. (I to K) Same analysis for the shear force along the x axis. Photo credit: Hongseok Oh, UC San Diego.

The spatial resolution of the TFT array was investigated by a two-point discrimination study in Fig. 3F using 3D-printed dual-tip fixture, each with a diameter of 1 mm, and intertip distances varied from 1 to 9 mm, with steps of 2 mm (top row of Fig. 3F). The force was applied along the central horizontal line of the TFT array crossing between fourth and fifth rows and was indistinguishable when the spacings were 1 and 3 mm and becomes resolved when the spacing increased to 5 mm or larger (middle and bottom row of Fig. 3F). The threshold distance of 5 mm is of the same order of magnitude to that of the human fingertip and smaller than that of the palm (39). The smallest resolvable pitch was further decreased to less than 500 μm in this work—limited by the testing instruments—as will be discussed in Fig. 7.

The response speed is characterized with a moving tip, where the z height over time follows the 2.5-Hz square wave with traveling duration of 10 ms in the experimental setup of Fig. 3G. The average response of the elements in the red box in Fig. 3H, where the tip applied forces are scatter plotted in Fig. 3I (black dots) together with the z position of the tip as a function of time in the same graph (blue curve). The full peak was developed immediately upon the applied force and rapidly returned to the baseline, demonstrating the ability of real-time tracking of the applied force with a response time that is dominated by that of the measurement apparatus. The capability of force sensing in real time indicates that the device, together with the control interface system, can be readily used in custom application–driven devices as further demonstrated in this work.

In terms of frame refresh rate, it is customary in previous works to report rates of response time of single sensors to be in the order of millisecond or smaller, but frame refresh rates are usually not reported. The frame rate of higher than 30 Hz has been reported from the work by Park et al. (40), where the mechanically transferred silicon was used as multiplexing transistors and reported a frame refresh time of 10 ms, which corresponds to a 100-Hz frame rate. In this work, we confirmed a maximal frame refresh rate of 100 Hz, and typical refresh rate ranged from 30 to 50 Hz when real-time plotting is provided using our homebuilt processing electronics. The relation between refresh rate and noise level is provided in the Supplementary Materials and fig. S5C.

Shear force sensing using 3D PDMS pillar array

Measuring the shear force requires novel strategies because force sensors are usually not sensitive to measure the in-plane forces. The piezoelectric property of the ZnO film is directional and arises from the polar dipoles oriented along the c-axis direction that is normal to the substrate surface in our films. These piezoelectric coefficients are negligible for any other orientation. The most straightforward strategy is to transfer shear stress to normal stress by using 3D structures (pillar array) on multiforce sensors as illustrated in Fig. 4A (41, 42). Lateral force applied to the pillar will interact with a subset of multiple sensors depending on the direction of the force. By calculating the difference of the sensor response for the same pillar, the direction of the force can be extracted. PDMS pillars with a base width of 2 mm, a height of 1 mm, and an overall 25% smaller width at the top surface were used to convert lateral to vertical force, as shown in Fig. 4B. Each corner of the pillar was centered at each ZnO TFT sensor. The analysis of the pillar structural design and its influence on the shear force sensing can be found in section S8 and fig. S9. The detailed fabrication process of the PDMS pillar array can be found in section S6 and fig. S7 (A to C) (4347). Figure 4C shows the calculation of shear force from the measured force values in a single unit cell (shear sensing unit). For the force measured by each sensor element, Fij, with the corresponding position vector, uij, the shear force can be calculated as Σ(Fijuij). The detailed analysis for the shear force calculation can be found in section S7 and fig. S8. We used a 3D-printed L-shape probe (see fig. S7D for more details) that is connected to a linear actuator to determine the relationship between the applied shear force and response of the TFT array, as shown in Fig. 4D. The shear force responses averaged from six randomly selected elements in the array are shown in Fig. 4E. The response (net current change) for applied shear force increased faster than the linear manner, presumably due to PDMS pillar deformation and stress redistribution that are nonlinear processes. Nevertheless, shear response along the x axis (Fig. 4F) and the y axis (Fig. 4I) was fairly uniform from the quiver plots shown in Fig. 4 (G to J). We found negligible cross-talk of shear force measurement as deduced from correlation matrices in Fig. 4 (H and K).

In case of simultaneous application of normal and shear forces, the absolute amplitude of the signals from each TFT indicates the applied normal force. The relative differences of signals in each shear sensing unit indicate the applied shear force, separated from the normal force. It should be noted that the force image would be affected by adding the 3D pillar array. The low moduli of PDMS add up minimal effect, but the very fine detail of the force image could be changed. The effect of PDMS pillar array on the spatial resolution is discussed in the Supplementary Materials and fig. S12 (E to H).

Multidimensional touch input

After quantifying the normal and shear responses, we next focused on demonstrating the application of our tactile sensors for real-time recording of multitouch and slide input using the TFT array (see movie S1). Single and repeated multitouch inputs did not show any difference in response time, sensitivity, or other properties (Fig. 5A). Shear force can also be visualized in real time with sliding a finger as above the array (Fig. 5B), where red arrows on still images indicate slide direction and quiver plots show the corresponding shear force distribution. Only the arrows related to the touch interface in the quiver plots indicate direction of the moving finger, successfully capturing the multidimensional information of the touch input. Please note that defective pixel correction algorithm was used to generate more natural response, as described in the Supplementary Materials and fig. S13. The device yield used for this demonstration was 93%.

Fig. 5 Demonstration of sensing multidimensional touch input using the TFT array.

(A) Response heatmaps of TFT arrays with single- and double-touch input. (B) Quiver plots of shear force when a finger slides over the TFT array. The direction of sliding finger is indicated by red arrows. The entire demonstration can be found in movie S1.

Closed-loop operation of robotic gripper

To demonstrate the utility of real-time spatial information of applied force, we demonstrated a closed-loop control of robotic gripper, where each finger was equipped with a tactile sensor array. Figure 6A depicts the configuration for closed-loop gripping and the operation principle of the control program. Detailed information about the gripper configuration and the control software can be found in the Supplementary Materials and figs. S14 and S15. The gripper safely holds fragile or heavy objects without human intervention in the gripping process using feedback from the tactile array, as shown in the series of photos in Fig. 6B. When a raw egg was placed at the center of range of the gripper with mounted TFT arrays and a grip command was issued, each finger approached the egg until a certain force is recorded from the TFT array. If the average normal force of the TFT array becomes stronger than the previously set value, the fingers stopped the movement. Once the gripper becomes stationary, the supporting floor for the raw egg was moved down to simulate the lift-up scenario. The distributions of normal and shear force, recorded from TFTs on each finger, are shown in Fig. 6 (C and D). The plots clearly tell the position and strength of the grip interface. The egg was stably held by the gripper for more than 10 s, and no damage was observed on the egg after multiple grip processes. The entire video showing the grip process can be found in movie S2.

Fig. 6 Application to the robotic gripper.

(A) Schematic illustration of the robotic gripper experiment with feedback from the force sensor array. (B) Snapshots of video (movie S2) showing each stage for closed-loop controlled grip experiment. (C) Normal and (D) shear force distribution of each finger when holding an egg. (E) Overlapped snapshots of the gripper holding a baseball from multiple incidents, which is slipping at grip interfaces. (F) Schematic illustration of slip action at the interface. (G) Illustration of additional feedback loop to provide secure grip. (H) Snapshot of the gripper controlled by shear force feedback closed-loop showing secure grip of the baseball. (I) Algorithm diagrams of the shear force feedback closed-loop control system. (J) Stacked plot of the gripper under shear force feedback control. Gripper angle (top), calculated threshold (middle top), and current changes corresponding to the net normal force (middle bottom) and net shear force (bottom). <1> to <5> Important moments in the grip process. <1> Grip command issued. <2> Initial grip made. <3> Shear feedback turned on. <4> Lift-off started. <5> Grip secured. The entire demonstration of the slip-adjusted grip control can be found in movie S3. a.u., arbitrary unit. Photo credit: Hongseok Oh, UC San Diego.

Shear force sensing can be used to detect object slipping at grip interfaces and can be further used in the closed-loop algorithm to implement a system with self-adjusting grip force. When a gripper is trying to grasp a relatively heavy and slippery object such as a baseball, normal grip cannot sense slip, and as a result, the object will slide on top of the sensor array. Figure 6E shows the overlapped images of the gripper holding a baseball from multiple incidents, indicating the sliding object due to the insufficient grip force. The sliding action will generate a constant downward shear force to the sensor in combination with 3D pillar array as illustrated in Fig. 6F. When the measured shear force values were fed into the feedback loop (Fig. 6G), the gripper automatically adjusted the force, without any visual or human assistance, to achieve secure grip. Figure 6H shows the gripper holding the same object securely when this algorithm is activated. To implement this, a normalized threshold force value was included in the closed-loop control system and is calculated on the basis of the normalized net shear force of the sensor array. Figure 6I shows the block diagram for this algorithm. Once shear feedback is turned on, this threshold is calculated at every data acquisition point with the formula, Threshold = a + b × net shear force, and compared with the net normal force, where a and b are constants that were determined experimentally in the baseball grip experiment. If the normal force does not exceed the threshold force, the gripper fingers apply a larger grip force. With threshold value being refreshed in real time, this process repeats until the normal force exceeds the threshold force value. Figure 6I shows the real-time recording of the gripper angle, calculated threshold, net normal force, and net shear force according to the algorithm described above. When grip command is issued, the gripper increases angle (i.e., closes the fingers) to approach and hold the object. Once the net normal force exceeds the preprogramed threshold force, the gripper does not increment the normal force anymore (does not adjust the gripper angle) and a stable grip is achieved. The detection of slip by shear force and the entire video of the shear force feedback closed-loop control can be found in movie S3.

Scaling to a 100-μm spatial resolution

By virtue of the scalable nature of the fabrication process, we were able to further scale down the force sensor to a 100-μm spatial resolution. Figure 7A shows the photo of the miniaturized 16 × 16 TFT array with different pitches of 500, 250, and 100 μm. The corresponding magnified microscope images are shown in Fig. 7 (B to D). The area coverage for each array was 8 × 8, 4 × 4, and 1.6 × 1.6 mm2, respectively, as shown in Fig. 7A. Here, the devices were released from the original Si substrates and laminated with thin PDMS sheets. The corresponding TFT channel dimensions in each array were (width/length) 200/20, 100/10, and 40/4 μm, demonstrating ease of scaling to sub–5-μm channel lengths as shown in the inset of Fig. 7D.

Fig. 7 Scaling to a 100-μm spatial resolution.

(A) Photo of the devices having 16 × 16 TFTs, with different pitch of (top) 500 μm, (bottom left) 250 μm, and (bottom right) 100 μm. For size comparison, a one-cent coin is shown together. (B to D) Microscope images of TFT array having pitch of (B) 500 μm, (C) 250 μm, and (D) 100 μm. Scale bars, 200 μm. (D) Inset: High-magnification image of a single TFT. (E and F) Normal force distribution of the 100-μm-pitch TFT array during the discrimination test. Response heatmap when pressed by a stamp having two extruded circles with the center-to-center distance of (E) 750 μm and (F) 500 μm. (G and H) Response heatmap plots of the 100-μm-pitch TFT array squeezed by a glass slide together with single hair, oriented toward (G) the diagonal direction and (H) the vertical direction. (I) Photograph of the 250-μm-pitch TFT array. Yellow circles indicate the location where controlled force is applied to investigate sensitivity distribution. (J) Response heatmaps of the TFT array when the same 25 mN is applied to the locations indicated by the yellow circles in (I). (K) Plot of average current change from nine areas as a function of the applied force. Error bars are corresponding SDs. (L) Photo of the 500-μm-pitch TFT array with 3D PDMS pillar array placed on top. Inset: Side-view photo of the 3D PDMS pillar array. Scale bar, 2 mm. (M) Snapshot of the video (movie S4) showing the transparent plastic rod rubbing the TFT array to the rightward direction. (N) Corresponding quiver plot showing the shear force distribution of the array. Photo credit: Hongseok Oh, UC San Diego.

The reduced dimensions lead to the ultrahigh spatial resolution. To confirm this, discrimination tests were carried out on the smallest TFT array with a 100-μm pitch. Acrylic stamps having two extruded pillars with center-to-center spacing of 750 and 500 μm were prepared by a laser engraving machine and used for the test. Figure 7 (E and F) shows the resulting normal force distributions from the stamps. Two peaks originated from the extruded pillars were clearly distinguishable for both cases, confirming that the array delineates the difference of objects spaced by 500 μm. Finer resolution was demonstrated by placement and pressing of a hair on top of the sensor array by a glass slide. The diameter of the hair is known to be ranging around 80 μm (48). Figure 7 (G and H) shows the photo of the hair on the sensor array and corresponding response heatmaps. Linear patterns with dual line appeared at the same location and direction of the hair placement. The appearance of dual lines rather than a single line presumably originated from the local deflection of the sensor array because the sensor array was not bonded to the underlying surface. The real-time operation of the TFT array with ultrahigh spatial resolution (100-μm pitch) can be found in movie S4. The sensor density of our smallest 100-μm-pitch array corresponds to a resolution of 254 pixels per inch (ppi), which is even higher than the pixel density of retina display used in MacBook Pro (226 ppi).

The sensitivity distribution of the downscaled sensor array was investigated by applying same forces on different locations over the array. The 250-μm-pitch TFT array was used for this study. Figure 7I shows a photo of the TFT array and locations where forces were applied. Using a 2-mm-diameter–sized acrylic block, normal forces ranging from 25 to 150 mN were applied to the area marked by the yellow circles in Fig. 7I. Figure 7J shows the responses of one trial when 25-mN forces were applied to different locations. Nearly identical responses were observed over the entire array, indicating the uniform sensitivity of the device. Figure 7K shows the averaged current change and calculated sensitivity and linearity error of this TFT array over different applied forces from 0 to 150 mN. The current change values and errors were taken from nine trials over different locations, with miniaturized 3D PDMS pillar array placed on top. At each location, we used the average current change of 4 × 4 TFTs at the center of the force. The response was highly linear over this force range. The sensitivity and linearity error of the TFT array for the force range of 0 to 150 mN were calculated to be 0.207 (±0.0172) %/mN and 7.07%, respectively. Similar to the case of the TFT array on rigid substrate, the linearity error was minimized at the low force range of 0 to 50 mN, which can be found in the Supplementary Materials and fig. S12. The released device was able to capture 300-mN force without damage. Thicker protection layers or stiffer materials can increase this critical force for damage at a compromised sensitivity and spatial resolution. The effect of the protection layers on the spatial resolution and sensitivity range can be found in section S9 and figs. S10 and S11 (43, 44, 49, 50).

The shear force sensing on the downscaled TFT array was demonstrated using miniaturized 3D PDMS pillar arrays on top of the 250- and 500-μm-pitch TFT arrays. To prepare 3D PDMS pillar arrays with sub–0.5-mm resolution, we used a laser engraving machine for creating the mold. Figure 7L shows a photo of the 500-μm-pitch sensor array, with 3D PDMS pillar array placed on top, where each pillar covered four TFTs to form a shear sensing unit. The morphology of the 3D PDMS pillars is shown in the inset of Fig. 7L. Each pillar exhibited a pyramidal shape with an approximate height and width of 450 and 500 μm, respectively. To demonstrate the shear force sensation, we slid a transparent object upon the sensor array, to visually inspect the pillars during interaction with the object. Figure 7M shows a snapshot of the video taken during this procedure in which the transparent plastic rod is sliding to the right and tilting the PDMS pillars. The corresponding shear response, shown in the quiver plot in Fig. 7N, indicates that the shear response appears only from the displaced pillars, with the same direction of the sliding motion. The full demonstration of the shear sensing from 250- and 500-μm-pitch sensor arrays can be found in movie S4.

Device bending

The device was fully operational as intended under highly bent conditions. Figure 8A shows the 500-μm-pitch 16 × 8 TFT array rolled over a plastic rod with a bending radius of 5 mm. We tested normal and shear sensing of the bent device by pressing the array with wooden stick or sliding a transparent rod on the array, as depicted in Fig. 8 (B and C). Locations of the applied normal force were well captured as shown in Fig. 8B, and locations and directions of applied shear forces are well resolved as shown in Fig. 8C. The reliable operation under highly bent conditions is attributed to the isolation of oxide materials for each TFT (ZnO channel, ITO contact, and Al2O3 dielectric layer), where these layers are patterned in mesa structures (more details can be found in the Supplementary Materials and fig. S16). Accordingly, the strain applied to the oxide materials can be minimized under bending condition and more strain could be distributed to the polymer substrates. This design strategy minimizes the damage in oxide materials and improves the durability of the TFT array. The performance of the piezoelectric TFT under different bent conditions was further quantified by measuring transfer curves of the TFT under different curvatures of 15, 10, and 5 mm and under applied forces of 25 mN, as shown in Fig. 8D. Transfer curves taken under different bending radii are plotted in Fig. 8E. In general, the transfer characteristics of the device were maintained for similar conditions. On the other hand, as shown in the inset of Fig. 8E, stronger device bending resulted in a decrease of the on-current. At a bending radius of 5 mm, the on-current decreased by 3.5%.

Fig. 8 Device performance on curved surfaces and device durability under multiple bending cycles.

(A) Photo of a 16 × 8 TFT array rolled conformally over a plastic rod having a bending radius of 5 mm. (B and C) Demonstration of normal and shear force sensing under highly bent conditions from the TFT array in (A). (B) Photos of the same TFT array on which small wooden stick applies normal force at the center left position of the array (top left) and the center right position of the array (top right), and the corresponding response heatmaps in the bottom left and right panels. (C) Photos of the TFT array on which transparent plastic rod applies a downward shear force (top left) and upward shear force (top right), and the corresponding shear force response plots in the bottom left and right panels. (D) Photos of the setup for measuring electrical characteristics of the device under different bending radii and applied forces. (E) Transfer curves of the single TFT under different bending radii. (F) Current change of the TFT with applied force, measured under different bending radii. (G) Photos of the bending cycle test, showing the device when released (top) and folded (bottom). (H) Response heatmap plots of the device under an applied force of 25 mN, measured before bending cycle test (top) and after 10,000 bending cycles (bottom). (I) Average current change of the force applied TFTs under different force values, measured before the bending cycle test and after 1000, 2000, 5000, and 10,000 bending cycles. Photo credit: Hongseok Oh, UC San Diego.

Figure 8F shows the change of the on-current of the TFT measured at VGS = 5 V and VDS = 1 V, when forces of 25 mN were applied using the 2-mm-diameter acrylic tip. The response strength increased as the bending radii decreased, meaning the smaller bending radii resulted in stronger responses. This leads us to speculate that the smaller bending radii introduce additional strain by readjusting the effective channel position exerting additional influence of the positive piezoelectric charge on the mobile charge carriers. A comprehensive understanding of these effects needs to account for modulation of defect and trap charge character under bending, for which theoretical work invoking first principles might be able to reveal.

The stability of the device under repeated bending cycles was investigated using automated bending cycle tests. Here, the device was folded and released by the linear actuator, which pushes back and forth on one side of the TFT array with the other side fixed. This motion repeatedly changed the bending radii from 2 to 7 mm. Figure 8G shows the device being released and folded during the bending cycle test. Intermediate evaluation of the response of the array was examined by applying a force of 25 to 50 mN using the 2-mm-sized acrylic tip and measuring the responses, before the bending cycle test and after 1000, 2000, 5000, and 10,000 bending cycles. Figure 8H shows the response heatmap plots before the test and after 10000 bending cycles, when 25-mN force is applied.

Clearly, the performance of the TFT array did not degrade. The average current change of TFTs pressed by the tip, before and after 1000, 2000, 5000, and 10,000 bending cycles, was plotted in Fig. 8I for force values of 30, 40, and 50 mN. The response was consistent, suggesting stability of the device sensitivity regardless of the number of bending cycles.

It is important to note that the release of the device changes the polarity of the force response of the TFT from the negative to positive direction. Figure S17 shows the transfer curves of the TFT under different applied forces, measured before and after release. As shown in fig. S17 (A and B), before release of the flexible arrays, the applied force led to a decrease in the TFT current. After release, the applied force led to an increase in the TFT current. The amount of change also increased when released. We suspect that as TFTs got released, the strain field in the ZnO channel by external force changed. The external force applied strain over the entire structure. Before release, the neutral mechanical plane (NMP) is located at the middle of the Si wafer and the device layer is way above the NMP. Once the device is released, the location of NMP is moved to near the device layer. This large shift of NMP is expected to have changed the direction and magnitude of the strain inside the ZnO channel under external force. Figure S17 (C and D) supports this hypothesis. The C-V curves were almost identical before and after release, without external forces. It implies that the release process itself did not affect the channel from an electronic point of view. The external forces shifted the C-V curves to negative direction for unreleased TFT and positive direction for the released TFTs, implying that the applied strain direction is reversed for both cases, and stronger strain is generated for the released TFT.

Blood pulse, water flow sensitivity, and closed-loop gripping with ultrahigh-density array

The superior spatiotemporal resolution, strong sensitivity, and reliable operation under bending conditions of these flexible TFT arrays substantiate their potential for future applications in health care, biomedical devices, or robotics. Figure 9 (A to D) shows the applications of the 250-μm-pitch TFT array for blood pulse sensation. The array was attached on the adult wrist where the radial artery is located, as shown Fig. 9A. Figure 9B shows the corresponding response. To capture the weak mechanical signal from the radial artery, a digital bandpass filter with a passband frequency range of 0.5 to 20 Hz was applied after the recording, to eliminate the slow shift in the baseline and high-frequency noises. The temporal response of the TFT with the strongest response (marked by the red box in Fig. 9B) is plotted in Fig. 9C, exhibiting the typical pattern of the blood pressure of a human adult. The averaged response was obtained by averaging 23 pulses of the temporal response curve as shown in Fig. 9D (51). Because of strong sensitivity and high temporal resolution (acquisition frame rate of 100 Hz) of the TFT array, the averaged response resolved different features of pulse signals such as systolic pressure (p1), late systolic pressure (p2), and early diastolic blood pressure (p3) (52). The radial augmentation index, which is defined as p2/p1 (%), and radial diastolic augmentation, which is defined as p3/p1 (%), were calculated to be 63.0 and 50.7%, respectively, which is in the range of the healthy human adults (52). The stiffness index (SI) was calculated as well, which is widely used to determine the stiffness of the radial artery. SI was obtained from the height of the human subject divided by the time between the systolic (p1) and diastolic (p3) peaks. The calculated SI was 6.17 m/s, which also lies in the range of mid-30s human adult (53, 54). The real-time demonstration of the pulse recording can be found in movie S5.

Fig. 9 Application of the TFT array.

(A to D) Demonstration of the pulse sensing. (A) Photo of the wrist showing the placement of the 250-μm-pitch TFT array. (B) Video snapshot (movie S5) of the response heatmap of the sensor array. The element marked by the red box showed the most dynamic changes during the measurement. (C) Temporal response plot obtained from the TFT marked in (B). (D) Averaged current change obtained from 23 pulse signals in (C). (E to G) Demonstration of the waterproof operation. (E) Video snapshot (movie S5) of the sensor array submerged in the saline solution. Pipet applies water flow onto the TFT array without making physical contact with it. (F) Response heatmap plots of the array capturing the moment of water flow reaching the array, taken at four different frames. (G) Corresponding shear force distribution plot of (F) at 0.023 s [top right in (F)]. (H) Video snapshot (movie S6) of the robotic gripper holding an egg, with the 2-mm-pitch TFT array mounted on the curved surface. (I) Corresponding normal force distribution map. (J) Video snapshot of the robotic gripper holding a heavy ball and detecting shear force. (K) Corresponding distribution of normal force, shear force, and derivative of shear force on the TFT array. (L) Plot of average shear force of the array as a function of time. Photo credit: Hongseok Oh, UC San Diego.

Waterproof operation is important for a variety of applications including those in biomedical devices. We tested the operation of the TFT array with pitch of 500 μm being submerged in the saline solution after encapsulating the device with a 2-μm-thin parylene C layer. Figure 9E shows the array being submerged in the saline solution. The array was able to resolve saline flow direction toward one of its edges without any physical contact with the array. As shown in Fig. 9F, the pressure from the water flow peaked for a very short period, smaller than 80 ms, and then dissipated quickly. Figure 9G shows the distribution of the shear forces on the array, indicating that the water flow was directed toward the top-right corner. The real-time demonstration of the waterproof operation and flow detection can be found in movie S5.

The large TFT array with 2-mm pitch was mounted on the robotic gripper and successfully aided in gripping an egg through a closed-loop algorithm. Figure 9H shows the gripper holding a raw egg aided by the feedback from the curved sensor array, and Fig. 9I shows the corresponding normal force distribution. For heavier objects where slip can occur, the array detected slip while gripping a lacrosse ball, as the supporting table was lowered and the ball slipped down due to its weight (Fig. 9J). The slip motion was captured as an increase of the downward slip, which is shown in the derivative of shear force map in Fig. 9K and the average shear force plot in Fig. 9L. The entire process of gripper experiment can be found in movie S6. The detailed performance of the TFT array used in this experiment can be found in the Supplementary Materials and fig. S18.

CONCLUSION

In conclusion, we demonstrated tactile sensing using active matrix of multiplexing ZnO piezoelectric TFTs. The fabricated array exhibited excellent spatial and temporal resolution with strong sensitivity to normal force and with 3D PDMS pillar structure strong sensitivity to magnitude and direction of shear force. The new TFT tactile array successfully captured the haptic input in real time with 3D force information. Moreover, the tactile array was integrated with robotic gripper for demonstration of a closed-loop control system. The gripper system successfully gripped and lifted fragile objects. The system also detected slip and adjusted grip force in lifting heavy objects using the shear force feedback. Notably, the downscaling of the device resulted in the force sensor array with extremely high spatiotemporal resolution. The devices retained their performance when released from the original substrate and under highly bent conditions or after repeated bending cycles. These capabilities were also demonstrated in multiple applications such as pulse sensor, waterproof operation, or integration with robotic arms. This research offers a general route to construct flexible normal and shear force sensing components that can be readily integrated in robotic applications for closed-loop operated delicate manipulations. In addition, the TFT tactile arrays can provide 3D information on the applied force when integrated to current consumer electronics that operate on the touch principle in mobile devices, gaming gears, or musical instruments, even under wet environments. Last, flexible form factor and free dimension scaling promise its applications in future technologies such as biomedical equipment, surgical robots, microflow meters, and implantable devices.

MATERIALS AND METHODS

Fabrication of ZnO piezoelectric TFTs

All devices were fabricated in the Nano3 cleanroom facilities at the Qualcomm Institute at the University of California (UC) San Diego. Detailed process parameters are described in the Supplementary Materials.

Measurement of DC characteristics

DC characteristics such as output, transfer, and C-V curves of the fabricated TFTs were measured using a B1500 semiconductor parameter analyzer. External force was applied by a voice coil–powered linear actuator system with internally integrated force sensor (V-275 PIMag Voice Coil Linear Actuator). At the tip of the actuator, a custom-made apparatus with specific surface was attached to apply force on only one element.

Operation of the TFT arrays

Python was used to program the main control software, which records, saves, and plots the data recorded with the custom control circuits. Teensy USB Board, version 3.5 from the electronic projects company PJRC, was used to control the integrated circuits on the custom control circuits. Arduino IDE with Teensyduino add-on was used to program the Teensy 3.5.

Robotics

ReFlex 1 robotic hand from Righthand Robotics was used in all closed-loop robotic experiment. Robot Operating System was installed on Ubuntu 14.04 system to control the ReFlex 1 robotic hand. To make the 3D structures for mounting the TFT array, Raise3D Pro2 from Dynamism was used to 3D-print objects.

SUPPLEMENTARY MATERIALS

Supplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/6/46/eabd7795/DC1

https://creativecommons.org/licenses/by-nc/4.0/

This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial license, which permits use, distribution, and reproduction in any medium, so long as the resultant use is not for commercial advantage and provided the original work is properly cited.

REFERENCES AND NOTES

Acknowledgments: We acknowledge support of Nano3 fabrication facilities and staff at UC San Diego. The Nano3 facility is part of the San Diego Nanotechnology Infrastructure (SDNI) of UCSD, a member of the National Nanotechnology Coordinated Infrastructure, which is supported by the NSF (grant ECCS-1542148). We would also like to thank D. R. Cleary for providing access to his 3D printer and J. Johnson for assisting H.O. in interfacing with the android hand. Funding: This work was supported by NIH Director’s New Innovator Award DP2-EB029757 and NSF grant CMMI-1728497. G.-C.Y.’s work was supported by the Global Research Laboratory Program (2015K1A1A2033332) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT in Korea. Author contributions: S.A.D. and H.O. conceived the project and designed the experiments. H.O. performed the fabrication and hardware and software development and analyzed all results. M.Y., S.A.D., and H.O. conceptualized the design of experiment for closed-loop robotic gripper, and G.-C.Y. provided input on the device fabrication. All authors contributed to the manuscript writing. Competing interests: S.A.D. and H.O. are inventors on a U.S. patent application related to this work filed by UC San Diego (no. 63/065,075, filed on 13 August 2020). The authors declare no other competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. Additional data related to this paper may be requested from the authors.
View Abstract

Stay Connected to Science Advances

Navigate This Article