Research ArticleAPPLIED PHYSICS

Controlling colloidal crystals via morphing energy landscapes and reinforcement learning

See allHide authors and affiliations

Science Advances  25 Nov 2020:
Vol. 6, no. 48, eabd6716
DOI: 10.1126/sciadv.abd6716

Abstract

We report a feedback control method to remove grain boundaries and produce circular shaped colloidal crystals using morphing energy landscapes and reinforcement learning–based policies. We demonstrate this approach in optical microscopy and computer simulation experiments for colloidal particles in ac electric fields. First, we discover how tunable energy landscape shapes and orientations enhance grain boundary motion and crystal morphology relaxation. Next, reinforcement learning is used to develop an optimized control policy to actuate morphing energy landscapes to produce defect-free crystals orders of magnitude faster than natural relaxation times. Morphing energy landscapes mechanistically enable rapid crystal repair via anisotropic stresses to control defect and shape relaxation without melting. This method is scalable for up to at least N = 103 particles with mean process times scaling as N0.5. Further scalability is possible by controlling parallel local energy landscapes (e.g., periodic landscapes) to generate large-scale global defect-free hierarchical structures.

INTRODUCTION

Controlling assembly of colloidal particles into different microstructures and morphologies could provide a basis to fabricate hierarchically structured materials with nontrivial emergent properties (e.g., optical, mechanical, thermal, and acoustic) important to numerous technologies (1). Many naturally occurring biological materials provide inspiration for creating periodic microstructures with unique multifunctional properties, where these microstructures often have repeat units that could be realized by assembling colloidal components (2, 3). However, despite a common recognition that colloidal assembly could provide a route to synthetic materials that mimic biological material structures and properties, many approaches are limited by defects, as well as scalability and structural diversity. In addition, changing interactions between initially stable colloids frequently results in amorphous aggregates with minimal functionality. In short, the state of the art in colloidal assembly often falls short of the grand challenge of directing matter into highly ordered nano- and microstructured materials.

Using biological materials as inspiration for target structures naturally leads to questions about how processes in biological systems produce complex structures. Biological systems are clearly complex in terms of the number of components, interactions, and parallel dynamic processes. A central feature of biological processes is exquisite control over many processes orchestrated in space and time. Although the obvious complexity of biological systems is somewhat overwhelming, it may be a necessary requirement to replicate some minimal aspects of such control in synthetic material systems to achieve comparable structures. Practically, it may be necessary to control colloidal assembly in space and time with a minimal level of sophistication to achieve many desirable microstructures and morphologies. This intervention perhaps seems at odds with a popular conception of self-assembly as simply mixing components and waiting for an autonomous process to yield complex defect-free structures. However, formally controlling assembly processes, in a manner similar to biological systems, may be the critical piece missing from many current approaches to assembling defect-free nano- and microstructured materials.

On the basis of current limits of colloidal assembly, and with inspiration from biological systems, it seems necessary to control colloidal assembly in space and time, including both microstructures and morphologies. External fields provide mechanisms to control position and orientation of colloidal assemblies and microstructure relaxation. For example, gravitational fields cause colloidal assembly to occur at different spatial coordinates, with crystalline microstructures assembling at potential energy minima (46). Although effective gravitational fields can be controlled macroscopically in centrifuges and microscopically on topographically patterned landscapes (7), they are not easily controlled locally and dynamically to manipulate colloidal assembly kinetics and defects. Macroscopic magnetic fields have been used to anneal colloidal microstructures (8), and local magnetic fields have also been used to assemble small colloidal ensembles (911) but require specific material properties. Shear fields can induce crystallization (12) but are difficult to control on length scales comparable to particle microstructures and produce numerous defects (13). Optical fields (14, 15) can produce induced dipolar interactions in highly localized fields to manipulate many single colloidal particles (e.g., holographic optical tweezers). However, optical fields have not been used to control colloidal assembly and local microstructures beyond proof-of-principle demonstrations (16, 17). These examples of colloidal assembly in external fields illustrate both desirable features and challenges with controlling assembly in space and time.

Electric fields can be designed to manipulate colloidal assembly in space and time in a manner that overcomes many limitations of other external fields. Electric fields can be shaped across length scales with different electrode designs (18) including arrays (19, 20) and patterns (21, 22). This spatial control of electric field shape and amplitude along with fast transient responses provides capabilities for rapid control over colloidal assembly and reconfiguration. In general, interactions and transport of colloids in electric fields at different frequencies (23) and in media with varying properties (e.g., polar, nonpolar, and salt) (24, 25) are sufficiently well understood to manipulate colloidal assembly for diverse materials. More recently, direct measurements have quantitatively connected dipole-field and dipole-dipole interaction potentials to local phase behavior (26) and morphology including different field shapes (27). The ability to tune electric field shape, amplitude, and frequency in space and time has a number of promising features for achieving high fidelity control over colloidal assembly processes.

Although much is known about colloidal interactions, dynamics, and microstructures in electric fields (28, 29), notable limitations remain in understanding and developing elements to implement formal control of assembly processes. These elements, in the terminology of colloid science (and control science), include capabilities to (i) quantify microstructures (sense states), (ii) tune interactions (actuate state changes), (iii) model nonequilibrium microstructure evolution (dynamic models), and (iv) determine how to choose colloidal interactions (control policy) based on current and desired states (objective). Some aspects have been previously developed, such as quantifying microstructures and their dynamic evolution between states for tunable depletion- (30), magnetic field- (31), and electric field- (32) mediated assembly processes. In addition, field-mediated colloidal assembly dynamic models have been used to implement feedback control in experiments (33) and simulations (34, 35) to assemble defect-free colloidal crystals. These approaches have yielded the unprecedented capability to assemble small defect-free colloidal crystals on relatively short time scales with complete efficacy (33). In these previous studies, the control policy, which closes the loop between sensing microstructures and deciding how to actuate fields, was based on a Markov decision process framework using coarse-grained Markov state models. This previous work not only provides a foundation to pursue control of more complex processes and objectives but also indicates limitations and directions for development.

In this work, we report morphing electric field shapes to control crystal defect removal rates and morphology. Results are obtained for varying system sizes in microscopy and computer experiments. We first investigate how field shape influences coupled grain boundary (GB) motion and crystal morphology relaxation. This approach is enabled by the capability to morph electric field–mediated energy landscapes within octupolar electrodes to control quasi–two-dimensional (2D) spatially varying colloidal phase behavior (Fig. 1) (27). We then develop coarse-grained variables, or reaction coordinates, for microstructural and morphological features (i.e., sensors) to quantify dynamic responses to field shape changes (i.e., actuators). To close the loop between sensing and actuation, we use reinforcement learning (RL) to discover control policies to determine what field shapes and orientations should be actuated to (i) rapidly remove GBs to obtain single domain crystals and (ii) restore circular crystal morphologies from anisotropic states. Last, using this control scheme, the time to obtain target defect-free circular crystals is characterized for different system sizes. Findings from this work inform the design, control, and optimization of colloidal assembly in space and time to realize complex hierarchically structured materials.

Fig. 1 Feedback-controlled morphing energy landscape for colloidal assembly.

(A) Top: Feedback control configuration consisting of octupole electrode, microscope, CCD camera, computer, and function generator. Bottom: Octupole electrode with two applied voltages, V1 and V2, and ground poles, G, with zoom view of electrode center region. (B) Feedback control scheme based on morphing energy landscape actuator, reaction coordinate sensors, and RL control policy. (C) Top row: Applied voltages determine electric field amplitude and anisotropy as shown for three illustrative cases, including isotropic fields (V1:V2 = 1:1; left), anisotropic field with 112° orientation (V1:V2 = 10:4; middle), and anisotropic field with 22° orientation (V1:V2 = 4:10; right). Middle row: Corresponding single-particle potential energy landscapes. Bottom row: Representative experimental configurations for 2.34-μm silica colloids (N = 300) for each field.

RESULTS AND DISCUSSION

Anisotropic field shape actuation

To control electric field shape and field-mediated energy landscapes for quasi-2D colloidal dispersions, a gold film octupole electrode was fabricated on a glass microscope slide and connected to a function generator (Methods, Fig. 1A). A microscope and charge coupled device (CCD) camera was connected to a computer to monitor particles positions and sense the system state in real time, and by connecting the same computer to the function generator, it is possible to close the loop and perform feedback control on the system state. Key elements of the feedback loop involving sensing, actuation, and the control policy (Fig. 1B) are described in the following.

To actuate field-mediated dipolar colloidal interactions within the electrode, different voltages are applied to electrode pairs within the octupole (Fig. 1C). This relatively simple electrode configuration provides control of field anisotropy and orientation. By controlling ac voltage and frequency, it is also possible to control the magnitude of dipolar interactions and sign of dipole-field interactions. The kT-scale energy landscapes due to dipole-field interactions can be adjusted in shape and magnitude (see the Supplementary Materials). For high ac frequencies, particles are compressed toward the field minimum at the octupole center, where the assembled morphology depends on the energy landscape shape. We recently reported the equilibrium phase behavior and concentration profiles of colloids on these field-mediated energy landscapes (27). In brief, dipole field energy landscapes are given asu(x,y)=2πεma3fcmE2(x,y)(1)where εm is the medium dielectric constant, a is the particle radius, fcm is the Clausius-Mossotti factor that depends on particle and medium dielectric properties, and E(x,y) is the local electric field magnitude. The particle phase behavior and morphology is determined by the concentration profile, η(x,y), on the energy landscape asη0η(x,y)1ηd(ηZ)=1kT[u(x,y)u0](2)where η0 and u0 are a reference concentration and energy, and Z is the compressibility factor (effective hard disks in this work; see the Supplementary Materials). Equation 2 is obtained from integrating a differential force balance of local osmotic pressure and particle forces due to energy landscape gradients. These results allow us to specify electrode voltages for each field shape to ensure that all particles are in the crystal phase (fig. S1) with morphologies determined by the energy landscape shape.

Knowledge of equilibrium phase behavior in different shaped fields provides a basis to begin considering nonequilibrium structures and their dynamic evolution on changing landscapes. Static crystalline configurations within different field shapes show predicted phase behavior and morphology based on model predictions (Fig. 1C) (27), but with defects in the form of GBs between misoriented crystalline domains. With the present study’s objectives of creating defect-free crystalline microstructures with circular morphology, the first task is to understand how changing field shape influences GB removal and morphological dynamic responses.

Microstructure and morphology evolution

To quantify microstructure and morphology relaxation on different energy landscapes, here, we characterize the temporal evolution of 2D crystals following step changes in isotropic and anisotropic fields. We introduce reaction coordinates that track dynamic processes (Fig. 2A). Raw information includes the applied field shape and orientation from voltage settings and particle coordinates obtained either directly from simulations or from image analysis of microscopy experiments. To monitor GBs, we characterize global crystallinity via bond orientational order (ψ6), local crystallinity via sixfold connectivity for each particle (C6,i), and its ensemble average (〈C6〉), which are used together to identify GB orientation in laboratory coordinates (γ) and relative to the field (α). To monitor crystal morphology, we characterize circularity (c) as well as the shape major-axis orientation in laboratory coordinates (ϕ) and relative to the field (θ).

Fig. 2 Gain boundary and shape relaxation versus energy landscape shape and orientation.

(A) Particle coordinate analysis in experiments and simulations to obtain reaction coordinates (corner label) for static configurations (see quantitative definitions in the Supplementary Materials), which can be obtained in real-time trajectories (movie S1). Right: Rendered configurations and trajectories for sensors (reaction coordinates) and actuator settings (voltages) for different field shapes and orientations, including (B) anisotropic field nearly perpendicular to the GB (α = 87.5°) for 50 s, followed by isotropic field for 50 s, (C) anisotropic field nearly parallel to the GB (α = 2.5°) for 50 s, followed by isotropic field for 50 s, and (D) constant isotropic field for 100 s. Trajectories show global crystallinity (ψ6, red), circularity (c, green), GB orientation relative to field (α, yellow), morphology orientation relative to field (θ, cyan), and voltages (V1, black; V2, gray). The ψ6 and c trajectories of 19 additional experiments are shown by lighter colored lines.

Brownian dynamic (BD) simulations (Methods) are used to investigate how GB motion and crystal shape relaxation are affected by field shape and orientation. We first test whether field shape and orientation can favorably influence GB motion and shape relaxation and demonstrate use of the reaction coordinates for tracking relevant dynamic processes (as part of posing a problem for more extensive testing using RL). A circular crystal of N = 300 particles with a single GB is quenched for 50 s by anisotropic fields orientated perpendicular (Fig. 2B) and parallel (Fig. 2C) to the GB, which is then followed by 50-s quenches using isotropic fields. For comparison, a 100-s quench with an isotropic field is also shown (Fig. 2D). Results suggest how energy landscape shape determines directional stresses and collective dynamic processes to promote both GB migration and shape relaxation. As evident from the first 50 s of ψ6 trajectories, isotropic fields do not affect GB diffusion, whereas both anisotropic field orientations promote GB migration. Alignment of the anisotropic field long axis parallel to GBs (Fig. 2C) yields, on average, faster GB migration than the perpendicular orientation (Fig. 2B). It appears that compression of the crystal domains toward the GB coupled with elongation produces local anisotropic stresses; these appear to favor particle motion within the GB and a net drift of the GB to the crystal periphery.

While anisotropic fields enhance GB removal rates in bicrystals, they also sculpt cluster morphologies to match underlying landscapes (27). To make a fair comparison with feedback-controlled assembly of perfect crystals using isotropic fields (33), and to achieve the objective of creating circular defect-free crystals, it is necessary to understand relaxation of crystal morphologies toward circular states. The simulated trajectories reveal coupled microstructural and morphological dynamics during the 50-s isotropic quench (Fig. 2, B to D). An important aspect of this finding is that we observe that defect-free crystals do not relax easily to circular morphologies with only application of an isotropic field. As will be discussed in the following section on feedback control, fields can be shaped with moments orthogonal to the crystal shape anisotropy to provide stronger driving forces for shape relaxation. We defer further discussion of how field shape influences GB removal and shape evolution until after presenting these results.

Feedback-controlled microstructure and morphology

The initial experiments addressed scientific questions about how different shaped electric field–mediated energy landscapes influence GB motion and morphology relaxation (Fig. 2). These findings provide essential information to address the engineering challenge of designing a feedback approach to control crystal defects and shape. By using microscopy and particle tracking as sensors for crystallinity and shape, and reconfigurable electric field–mediated energy landscape shape as an actuator, the only remaining element (Fig. 1C) is a control policy to close the loop to achieve the objectives of assembling defect-free circular crystals in minimal time. The feedback control is designed with two objectives: to obtain (i) defect-free crystal microstructures and (ii) circular crystal morphologies. We describe the development of two control policies that address each objective by specifying how to actuate tunable energy landscapes.

To develop control policies, we use a combination of empirical testing and machine learning using BD simulations previously matched to experiments via rigorous thermodynamic and dynamic criteria (27, 32, 33, 36). For comparison, our previous work used a Markov decision process framework to develop control policies for assembly of perfect crystals in isotropic fields (33). This approach required knowledge of the transition probability between all states for each field based on a model; optimization in this approach was achieved by maximizing the probability of reaching a target state by choosing the correct actuator based on instantaneous observations of any system state. The present work avoids construction of a dynamic model for transition probability by using a combination of initial empirical parametric testing (i.e., human learning) with subsequent RL to obtain final policies.

To provide initial estimates of feedback control parameters, we first used empirical tests to determine characteristic relaxation time scales. Relaxation time was interrogated by testing anisotropic fields with fixed shape (Vmax:Vmin = 10:4) and random orientations (β = 23° or 113°), which were alternated with isotropic fields (fig. S6). For simplicity, the same time period was used for anisotropic and isotropic fields. Findings show that alternating anisotropic and isotropic fields every 20 s produced the fastest removal of GBs and morphology restoration. These characteristic relaxation times are consistent with previous measures of GB motion (32) and scaling arguments based on long-time diffusion of particles within GBs (33).

Next, using 20-s period of randomly oriented anisotropic fields alternated with 20-s isotropic fields, different field shapes were tested (i.e., voltage ratios, Vmax:Vmin = 10:3 to 10:10; fig. S7). Findings show that a 10:4 field aspect ratio produced the fastest GB removal. The most anisotropic field shape (Vmax:Vmin = 10:3) occasionally released particles from the octupole center, so it was not further considered. In short, a monotonic increase in GB removal rate was observed with increasing anisotropy in the underlying energy landscape. However, this result was observed with a fixed field shape for all control cycles. The optimal landscape shape is less obvious if both field shape and orientation can be dynamically selected.

Using the control update time and possible field shapes identified in initial testing, we next used RL to determine the optimal control policies for determining field orientation and shape in feedback control of GBs and crystal morphology (Methods). RL obtains optimal policies via algorithm-guided learning through iteratively simulated experiments (37); this avoids time-consuming dynamic model development (38). BD simulations are used to enable substantially more statistics than directly using experimental trajectories (and also overcome spatiotemporal sampling limitations). The two control objectives were considered separately during RL (see Methods). For both GB removal and crystal shape control, eight anisotropic field shapes and orientations were considered as candidate actuator settings (Fig. 3A). In both cases, RL is initialized with all actuator settings being equally likely so that optimal choices are identified in the course of evaluating statistical outcomes as learning proceeds.

Fig. 3 Feedback-controlled GB removal and circularity using morphing energy landscapes with RL control policy.

(A) Electric field shapes and orientations (actuator choices) for consideration in RL-based optimal control policies. RL control policies as look-up tables and representative states for (B) GB removal based on sensors for GB orientation (γ) and global crystallinity (ψ6) and (C) circular shape restoration based on sensors for crystal shape orientation (ϕ) and circularity (c). (D) Microscopy experiment including analyzed images and trajectories for (objectives) global crystallinity and circularity, (sensors) field orientation and shape major-axis orientation relative to the GB, and (actuators) applied voltages (movie S2). The gray region (74 s < t < 100 s) highlighted in the actuator panel represents the period for morphology control, while the other white region represents the period for GB removal (t < 74 s). (E) BD experiments including rendered configurations on underlying gray-scale energy landscapes and reaction coordinate trajectories with the same information in (B) (movie S3). In images, renderings, and plots, particles and trajectories are colored the same as in Fig. 2.

For GB removal, the control policy was discretized into a look-up table based on sensors for GB orientation (γ) and global crystallinity (ψ6), with representative states rendered for illustration (Fig. 3B). The resulting control policy from RL primarily chooses the most anisotropic field (10:4) oriented parallel to GBs, except for highly ordered configurations (ψ6 > 0.8), where a less anisotropic field (10:6) is chosen. The choice of field orientations parallel to GBs is consistent with the initial results in Fig. 2, but is now verified for many more starting configurations, control trajectories, and all eight combinations of field shape and orientation. The choice of a lower aspect ratio field for high values of global crystallinity is not obvious but indicates that less distortion of crystal morphology is necessary for GB removal. A single asymmetric feature is observed in the GB removal policy at (60° < γ < 75°, ψ6 < 0.2); this condition is for highly defective crystals with GBs oriented about halfway between both possible field orientations, where it appears that either field orientation is about equally effective. In short, RL identifies a clear trend for field orientation with a subtle shape change for highly crystalline states.

For crystal morphology control, the control policy was discretized into a look-up table based on sensors for crystal shape orientation (ϕ) and circularity (c), with representative states shown for illustration (Fig. 3C). The resulting policy always uses the most anisotropic field (10:4) oriented perpendicular to the crystal shape orientation. This finding is also consistent with preliminary results in Fig. 2 showing that defect-free anisotropic crystals (ψ6 ≈ 0.2, c < 1) did not relax to circular shapes in isotropic fields. Whereas GB motion is visibly part of morphology relaxation in polycrystals, microscopy and simulation videos (e.g., movies S2, S3, S8, and S9) show for defect-free crystals that a shearing motion along crystal planes is the primary morphology relaxation mechanism. In addition, GBs were not observed to form as part of morphology relaxation. The same movies also show that morphology relaxation does not involve particle migration around the crystal periphery (for either defect free or polycrystals), although such a mechanism is, in principle, possible. On the basis of these findings, when the mismatch between crystal and underlying energy landscape shape is small, the driving force for relaxation appears insufficient to overcome relevant elastic moduli of the perfect crystal interior. Practically, the RL policy reduces to a simple heuristic, where the most anisotropic field, with the least alignment to the crystal shape major axis, produces the fastest relaxation to circular crystals.

The resulting overall control policy (Table 1) then consists of the two subpolicies to achieve the two objectives of removing GBs (Table 2) and obtaining circular crystals (Table 3). The GB removal policy is called until a perfect crystal is obtained (i.e., ψ6 > 0.99), followed by the policy to restore morphology until a circular crystal is formed (i.e., c > 0.99). Endpoint detection within both subpolicies enables early termination as soon as objectives are met. To allow the possibility of introducing defects during the process for restoring circular morphology, the GB removal algorithm can be called at any control update time when defects are detected. In practice, we never observed GBs being reintroduced into crystals during morphology relaxation for the several hundred realizations conducted in the course of our study.

Table 1 Policy to obtain perfect, isotropic crystals.

View this table:
Table 2 Policy to obtain perfect crystal microstructure; remove grain boundaries (ψ6 < 0.99).

View this table:
Table 3 Policy to obtain circular crystal morphology; remove shape anisotropy (c < 0.99).

View this table:

The resulting optimized control policies were implemented in experiments to test their effectiveness for generating defect-free crystals with circular morphologies (Fig. 3 and movies S2 and S3). Individual realizations for feedback control of 300 particle ensembles are shown for microscopy (Fig. 3D) and simulated (Fig. 3E) experiments. Reaction coordinate trajectories now formally serve as sensors for microstructure and morphology. Global crystallinity (ψ6) and circularity (c) indicate progress toward control objectives of creating single-domain crystals and perfect circular morphology and are used for endpoint detection. The field orientation relative to the GB (α) and to the crystal shape major axis (θ) is used in control policies with feedback control to decide field orientation to promote GB removal and restoration of circular morphologies. Applied voltage to each electrode (V1, V2) shows how the electric field is actuated using the policy.

Simulated and experimental trajectories show how feedback control using the control policies works in practice. In both simulated and experimental trajectories, the GB removal policy is called during two control update periods and terminated early during the second period via endpoint detection. After GB removal, both defect-free crystal lattices have shapes corresponding to circularities of c = 0.7 to 0.9, which requires the circular shape restoration policy. In both cases, application of an anisotropic field orthogonal to the crystal long axis yields a circular morphology in ~5 to 10 s. Changing crystal morphology does not introduce any new internal defects; this appears to result from crystal shape relaxation occurring via shearing motion along crystal planes. On the basis of successful demonstrations in experiments and simulations, in the following, we obtain statistics for many more controlled trajectories, which we compare for different policies and system sizes.

Control performance and scaling

The feedback control performance can be quantified based on more statistics, comparison to other approaches, and as a function of system size. Control performance (Fig. 4) is reported for statistical comparisons of many individual stochastic trajectories including 102 microscopy experiments and 103 simulated experiments. Instantaneous and cumulative yields of perfect crystals versus time are reported for three cases: (i) the feedback-controlled alternating anisotropic/isotropic field developed in this work (Fig. 4A), (ii) a feedback-controlled isotropic field from our previous work (Fig. 4B) [details in the Supplementary Materials and (33)], and (iii) uncontrolled quenches to a constant isotropic field (Fig. 4C). For comparison, individual experimental and simulated realizations for the last two approaches are shown in figs. S8 and S9 and movies S4 and S7. All control experiments begin with quenched bicrystal configurations characterized by a single GB, high local order, and low global order (C6 > 0.9, ψ6 < 0.6). Each policy also has an identical objective of producing a defect-free single domain crystal (ψ6 > 0.99). The second objective of circular morphology is automatic for isotropic fields. Mean times (and SDs) to yield different percentages of defect-free circular crystals are compared for each case (Fig. 4D). The logarithmic fits capture cumulative yield versus time data up to 100% yields for controlled cases at finite times (and uncontrolled data at short times; long-time asymptotic behavior may differ).

Fig. 4 Defect-free circular crystal yield versus time for different control strategies.

Cumulative yield (Y, left axis) of defect-free circular crystals for 102 microscopy experiments (points) and 103 simulated experiments (lines) as well as instantaneous yield (ΔY, right axis) values from simulated experiments (bars). Results for three different control strategies include (A) the new feedback approach based on morphing energy landscapes with an RL-based policy, (B) a former feedback approach based on changing isotropic energy landscape amplitudes with an Markov decision process-based policy (33) (fig. S9 and movies S4 and S5), and (C) a constant uncontrolled quench using an isotropic field (26, 27) (fig. S8 and movies S6 and S7). (D) Cumulative defect-free circular crystal yield versus average time, 〈t〉, for each control policy in (A) to (C) (same coloring) using simulation results (points). 〈t〉 obtained for all t up to yield Y within the time limit. Left value error bar is SD for all t less than 〈t〉 given value of Y; right value error bar is SD for all t greater than 〈t〉 given value of Y. Empirical logarithmic fits to each case are given by anisotropic, Y = 288ln(0.025〈t〉), isotropic, Y = 49ln(0.009〈t〉), and quench, Y = 12ln(0.013〈t〉).

Feedback control using anisotropic fields displays an obvious superior capability for rapidly generating perfect circular crystals compared to benchmarks. To understand the advantages of the new approach, we first discuss results from benchmarks. For the case of a constant isotropic field quench, less than ~15% of trajectories relax into perfect crystals in 500 s (Fig. 4C). This shows that structures do not easily relax via GB diffusion without intervention, which is consistent to weak driving forces and slow dynamics on shallow-free energy landscape gradients (32). This uncontrolled case does not produce 100% perfect crystals on experimental time scales (microscopy or computer experiments). Because most of the trajectories never relax into perfect crystals, assembling perfect crystals on short times requires some type of control.

A previously reported method for optimal feedback control based on isotropic fields provides an important benchmark in terms of performance and concept. This approach that uses isotropic fields with varying inward radial compression (Fig. 4B), to control melting and recrystallization (33), has a ~50% success rate in 500 s. The optimal scheme using only isotropic fields represents a critical improvement in that it guarantees perfect crystals in 100% of trials compared to the small yield in the uncontrolled quench. The average crystallization time for the uncontrolled case is unbounded because many trajectories never crystallize, so in this respect, controlling isotropic compression represents an essentially infinite improvement compared to the uncontrolled case. This case provides the most relevant direct benchmark for comparing with the new feedback control method and policy. We are aware of other approaches to feedback-controlled assembly (34, 35), but not for field-based actuators and an objective of assembling perfect crystals.

The new approach developed in this work (Fig. 4A), using anisotropic fields to assist GB motion with the RL-optimized policy, has a 100% success rate at producing perfect circular crystals within ~100 s, which is ~10× faster than the best previous feedback control method using isotropic fields. Because our previous approach using isotropic fields produced circular crystals, we made this part of the current control objective. However, if we had only specified perfect crystal microstructures, then the performance would be even faster without the morphology correction step. In any case, the previous feedback control was optimal for isotropic fields and produced perfect crystals 100% of the time, but the new approach using anisotropic fields provides a nontrivial order of magnitude improvement.

A statistical analysis of many feedback-controlled trajectories (from Fig. 4A) also reveal trends about the number of cycles necessary to remove GBs and restore circular morphologies. In 103 simulated control experiments, the number of times the GB removal policy was called to remove defects was 1 (15%), 2 (50%), 3 (28%), 4 (4%), 5 (1%), and 6 (<1%). No controlled trajectories were observed where the GB removal policy was called more than six times. In microscopy experiments, results were similar except 2% of trajectories required six cycles to remove GBs, but this could be due to less statistics. For morphology control, >98% of simulated experiments yielded circular crystals after calling the morphology restoration policy one time. The other 2% of trajectories did not require morphology control because the final state after GB removal was already circular. Practically, for perfect crystals after GB removal, morphology control was essentially deterministic, which appears to be the reason for one cycle being sufficient for restoring circular morphologies.

Given the success of our new approach for creating perfect crystals of 300 particles, it is important to address scaling to larger system sizes [i.e., particle number (N)]. To investigate size effects, we first scaled the field to maintain the same thermodynamic conditions using Eqs. 1 and 2. The differential force balance underlying this thermodynamic model is based on dipole-dipole interactions averaging to zero for isotropically distributed particles, which has been shown in theory for macroscopic electrorheological fluids (39) and has been validated in experiments for 3D millimeter-scale electrode dimensions (40). However, the model could be updated to include dipole-dipole interactions if they were found to be important for different conditions. The model has also been validated in our own previous work on different system sizes (26) and dimensions (27, 41). Hence, the thermodynamic model appears to be appropriate for scaling to larger system dimensions. Practically, the electrode dimensions and voltages are sized for different N using Eqs. 1 and 2 to specify solid phases with the same osmotic pressure, morphology, and peripheral density profile for any field shape [i.e., arbitrary energy landscapes (27); see the Supplementary Materials for details]. Simplification of this model for specific cases in this work yields analytical expressions (eqs. S29 and S30), where the electrode gap scales as ~N1/2 and voltage scales as ~N3/4. The control update time and RL control policy are the same for all system sizes.

Given the excellent agreement between microscopy and simulated experiments in Figs. 2 to 4, we investigate system size effects using BD simulations for 300, 600, and 900 particles (Fig. 5, fig. S10, and movies S8 and S9). For each size, 103 controlled trajectories are summarized by instantaneous and cumulative yields of perfect circular crystals versus time (Fig. 5, A to C). Representative configurations in isotropic and anisotropic fields are also reported (Fig. 5D), as well as perfect crystal yield versus mean time for each system size (Fig. 5E) and system size dependence versus mean time for fixed perfect crystal yields (Fig. 5F). Average crystallization times, obtained by averaging cumulative crystallization trajectories up to 100%, are 72 s (N = 300), 110 s (N = 600), and 129 s (N = 900). In short, the current control method produces 100% perfect circular crystals on the order of ~100 s, which demonstrates its general effectiveness for different system sizes.

Fig. 5 Feedback-controlled morphing energy landscape control versus system size.

Instantaneous (Y, bars) and cumulative (ΔY, line) yields of defect-free circular crystals versus time for computer simulations of system sizes including (A) N = 300, (B) N = 600, and (C) N = 900 particles. Example controlled trajectories for 600 and 900 particles are shown in fig. S10 and movies S8 and S9. (D) Representative renderings of GBs and crystal morphology for different system sizes on (left) isotropic energy landscapes and (right) anisotropic energy landscapes for electrode gap dimensions of 100, 140, and 170 μm. (E) Cumulative defect-free circular crystal yield versus average time (same as Fig. 4D) for each system size in (A) to (C) (same coloring). Logarithmic fits to each case are given by Y = 288ln(0.025〈t〉) (N = 300), Y = 134ln(0.022〈t〉) (N = 600), and Y = 117ln(0.016〈t〉) (N = 900). (F) Average time to achieve cumulative defect-free circular crystal yields given by power law fits as N = 0.74〈t1.67 (25%, circle), N = 0.51〈t1.67 (50%, square), N = 0.38〈t1.67 (75%, diamond), and N = 0.27〈t1.67 (95%, triangle).

The time to obtain perfect crystals via GB removal with the new feedback control approach increases with increasing system size. The system size dependence on GB removal time scales (〈t〉 ∝ N3/5; inverting N ∝ 〈t5/3) is slightly greater than the N1/2 system size dependence of the electrode dimensions. However, GB dynamics should not obviously follow similar scaling to thermodynamic quantities (32). GB removal requires migration from crystal interiors to edges, where this distance is proportional to the crystal radius, which scales as N1/2 for circular morphologies. This scaling might account for similar GB removal time scales. The observed bias to a greater power likely results from additional effects including multiple GBs/domains and more complex noncircular shapes (26). The current control approach is partially based on orienting fields relative to GBs, which is less well defined for multiple domains. As we noted when investigating field orientations, all anisotropic field orientations promoted faster GB relaxation compared to isotropic fields. Despite additional complexity of multiple domains/GBs, GB removal times fit a consistent scaling up to the largest size investigated.

Our new feedback approach rapidly generates perfect circular crystals of up to ~103 particles, and nothing suggests that the trend will not extrapolate to larger systems. Given that GB removal time increases for larger system sizes, eventually these times may become impractically large. Further scaling-up beyond some threshold crystal size might be achieved through adaptation of the approach here to subdomains within larger crystals. For example, the approach here could be parallelized on an electrode array (1922) to work on local regions of larger crystals. Another approach could involve spatially translating fields through crystals in a manner analogous to zone refining. In general, the approach in this work can be scaled effectively in local microscopic systems of increasing size, but eventually creating very large crystals would require adapting the current approach to include some spatial addressability to operate on different regions within larger crystals. In any case, given the unprecedented nature of the feedback-controlled method developed here, it seems that there are opportunities for further development including scale-up to larger systems.

In conclusion, we developed a new feedback-controlled approach using morphing energy landscapes to remove GBs and produce circular shapes in colloidal crystals. We demonstrated this approach in microscopy and simulation experiments for colloidal particles in ac electric fields, although the reported approach is generalizable to any morphing energy landscape. We discovered how easily accessible energy landscape shapes and orientations together could enhance coupled GB and crystal morphology relaxation processes. On the basis of this finding, RL was used to develop an optimized control policy to close the loop between sensing states and actuating morphing energy landscapes to rapidly and reliably produce defect-free circular crystals.

The resulting optimized control enabled by morphing energy landscapes is superior to benchmarks and is scalable to different system sizes. Statistical comparisons of controlled stochastic trajectories quantify the speed and accuracy of producing defect-free circular crystals, where (i) feedback-controlled morphing energy landscapes produce perfect crystals an order of magnitude faster than feedback-controlled isotropic landscapes, and (2) both feedback-controlled methods are markedly faster than the nearly unbounded times required for uncontrolled relaxation of defective crystals into perfect structures. The key element provided by morphing energy landscapes that enables rapid creation of perfect crystals is the ability to exert anisotropic stresses to control crystal defects/shapes without melting, which is inherently faster than repeated melting/freezing (disassembly/assembly) processes. Last, this new control approach is scalable to different system sizes, with average times for creating defect-free circular crystals increasing as N0.5. In short, morphing energy landscapes in conjunction with optimized feedback control can assemble defect-free colloidal crystals with nontrivial improvements to speed and scaling compared to state-of-the-art methods.

Future work could extend the scalability of our approach, most likely without the need for higher fidelity control. Addressing multiple crystals on electrode arrays, or any periodic landscape, via parallel and serial combinations of our control approach, could be used to merge adjacent crystals into larger crystals or to create hierarchical patterned crystal structures. Given the success of elliptically shaped landscapes in this work, it is not obvious that more complex shaped landscapes would improve speed or reliability. For example, even in the limit of individually controlling every particle on highly complex landscapes, limitations due to cooperative rearrangement processes would not obviously allow faster net processes than coarse control of directional stresses within crystals via local field anisotropy. Another extension of this work could include controlling assembly of anisotropic colloidal particles, which have additional complexities due to additional orientational degrees of freedom important to equilibrium and nonequilibrium structures as well as defects. Last, other potential future directions include adapting our method to 3D target structures and using crystal optical responses (e.g., diffraction) as sensors for desired states. Ultimately, controlling locally repeating shapes on periodic landscapes could produce many defect-free local structures that collectively lead to a global defect-free hierarchical structure.

METHODS

Control cell

Coplanar octupole Au thin-film electrodes were patterned on glass microscope coverslips after cleaning with acetone (30 min), isopropanol (30 min), Nochromix (1 hour), and 0.1 M KOH (30 min). The coverslips were rinsed with deionized water and dried with N2 before use. The electrodes were fabricated by spin-coating photoresist (S1813, Shipley) onto microscope coverslips, ultraviolet exposure through a photomask, and physical vapor deposition of a 15-nm chromium adhesive layer and a 35-nm gold layer. The photoresist liftoff was accomplished with agitation in 1165 Remover (Shipley). To construct batch cells, polydimethylsiloxane O-rings (Corning, d = 10 mm) were coated with vacuum grease and sealed between the coverslip with the electrode and a glass coverslip. Nominal 2.34-μm-diameter silica colloids (100 μl) (Bangs Laboratories) were added to the batch cell before sealing it. The 22-gauge copper wires were attached to the electrode using conductive carbon tape. The electrode was then connected in series with a dual-channel function generator (Agilent 33220; Fig. 1A).

Microscopy

Microscopy was performed on an inverted optical microscope with a 63× Zeiss air objective lens (0.6 numerical aperture) at 1.25 magnification. A 12-bit CCD camera captured 336 pixel × 256 pixel (104 μm × 79 μm) digital images at a rate of 10 frames/s. Image capture and analysis were performed using MATLAB Image Processing and Image Acquisition Toolboxes. Image analysis algorithms coded in MATLAB were used to simultaneously locate and track particle centers, as well as compute reaction coordinates in real time.

Simulations

BD simulations, which previously matched to experiments (32), were used to compare with experiments and test additional conditions. Particle elevations above the substrate are fixed to their most probable value based on a balance of gravitational and electrostatic particle-wall interactions. This simulation time step is 0.1 ms, and the remainder of the parameters used in the simulations are reported in Table 4. To obtain initial configurations of self-assembly, the particles were randomly placed in the electric field, quenched under isotropic field using crystallization voltage (eqs. S29 and S30), and held for 50 s. The process is restarted if the configuration failed the starting configuration criterion (ψ6 < 0.6).

Table 4 Model parameters based on previous experiments (32).

View this table:

Control policies

RL, specifically Q-learning, was used to obtain the optimal anisotropic policy. The controlled system is characterized by a state space (S), and the control actions forms the action space (A). A score table, Q(s, a), is defined to reflect the preference of applying an action aA at a state sS. Optimal control policies for two objectives are learned separately with RL. For GB removal, the state space is defined by global crystallinity (ψ6) and grain orientation (γ), while for morphology relaxation, the state space is defined by circularity (c) and morphology orientation (ϕ). Crystallinity and circularity are discretized by 0.2, and GB and morphology orientations are discretized by 15°. In both policies, the action space consists of all combinations of anisotropic field shapes and orientations (Fig. 3A). Initially, actions are unbiased, i.e., Q(s, a) = 0. During learning, control actions are chosen at the beginning of each period (T) either randomly with 50% probability or by the highest Q score. A reward (R) is given at the end of every period to evaluate the outcome of the chosen control action. By observing a control period, the Q score is updated as (42) ΔQ(sT,aT)=[N(sT,aT)]0.51[R(sT+1)+0.5·maxiQ(sT+1,ai)Q(sT,aT)](3)where sT is the state at the beginning of control period, aT is the chosen control action, R(sT+1) is the reward calculated based on the period ending state, sT+1, and N(sT, aT) is the number of times state sT and action aT have been visited. Learning terminates when ΔQ(s, a) < 0.01 for all (s, a), indicating that further updates are negligible. For GB removal, reward is defined asR(sT+1)=1ψ6,T+12(4)where ψ6,T+1 is crystallinity at the period end. For morphology control, reward is defined asR(sT+1)=1cT+12(5)where cT+1 is morphology circularity at the period end. Rewards indicate effectiveness of the chosen control action toward achieving the control objective, and therefore, crystallinity and circularity are chosen as metrics. Nonlinear expressions are used to better distinguish crystals with intermediate to high crystallinity and circularity values, where the most challenging control situations arise. For GB removal, configurations with low crystallinity (e.g., ψ6 < 0.4) are rarely observed and rapidly resolved. Intermediate and high crystallinity values often have complex defect structures. The same is true for morphology control, where extremely anisotropic clusters (e.g., c < 0.5) are less common than nearly circular morphologies. The final optimized policies are plotted in Fig. 3A and listed in Table 1.

SUPPLEMENTARY MATERIALS

Supplementary material for this article is available at http://advances.sciencemag.org/cgi/content/full/6/48/eabd6716/DC1

https://creativecommons.org/licenses/by-nc/4.0/

This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial license, which permits use, distribution, and reproduction in any medium, so long as the resultant use is not for commercial advantage and provided the original work is properly cited.

REFERENCES AND NOTES

Acknowledgments: Funding: We acknowledge financial support by the Department of Energy BES DE-SC0017892. Author contributions: J.Z. and M.A.B. designed the experiments, simulations, and analysis; interpreted the data; and wrote the manuscript. J.Z. and J.Y. performed and analyzed the microscopy experiments. J.Z. performed and analyzed the computer simulation experiments. Y.Z. computed the electric field and energy landscapes. All authors have given approval to the final version of the manuscript. Competing interests: The authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. Additional data related to this paper may be requested from the authors.
View Abstract

Stay Connected to Science Advances

Navigate This Article