Finding novel materials for practical devices

Katie Luu: From MIT, this is the Energy Initiative. I’m Katie Luu. Today we’re talking with Heather Kulik, associate professor of chemical engineering.

Something that really struck me when I was reading the Energy Futures article, the companion piece to this podcast episode, is that you are a chemical engineer who is working with machine learning. Can you tell me how you went down that road?

Heather Kulik: My background is pretty diverse. My undergraduate degree is in chemical engineering. My PhD is actually from MIT in Materials Science and then I did postdoctoral studies in in chemistry so I’d broadly call myself a computational chemist. But that does not mean that we are trained to do machine learning. But it does mean that people in my field that have a strong affinity and our main day-to-day lab instrument is a computer. I came to machine learning from that background that we were already essentially using computers all the time and really the way it started was, like I think a lot of things happen in research these days, I had a student interested in discovering new materials and took a class in machine learning. He asked me what he should do for his homework for his project in the class, and next thing you know it’s what my group is doing for a large portion of full-time.

KL: What are the problems that you have been trying to solve with machine learning in your lab?

HK:: Broadly, my group works on inorganic materials especially transition metal complexes. Those are materials defined by having a transition metal which are elements in the periodic table ranging from iron to cobalt to manganese. These elements that are less common than carbon, nitrogen, and oxygen, maybe, from your undergraduate chemistry class. We’re interested in using these metals and tailoring their properties in order to make better catalysts or materials. A lot of interesting technology comes from being able to precisely design the environment around these metals, to make those metals in turn have interesting optical properties, catalytic properties, and so on.

KL: What kind of uses—end uses—are you aiming for?

HK:: A lot of what my group focuses on is very fundamental in the sense that we’re just trying to develop new ways. If you want to solve a problem in energy or materials or come up with a new technology, where do you start? We start on the very fundamental end of just answering, if the right answer is one in a billion, how are we going to be able to get an idea of what those billion compounds look like, so we can find that one in that space. Where do we need to go? We sit pretty far from the absolute end use of putting this into a new technology but long-term goals are thinking about catalysts that don’t deactivate, that can convert abundant feedstocks like methane to more desirable products like high-value chemicals, transportable liquids like methanol.

Also, in the materials space, there are what was in the Energy Futures article, which is thinking about switchable molecules that can have applications in quantum computing as sensors in functional materials that basically respond to their environment. But thinking about tailoring it on the smallest scale.

KL: Using machine learning, is the whole goal to speed up the discovery of these complexes?

HK:: I would say in the simplest, at the first step, the way we use machine learning is that once you train a machine learning model, it takes us far less time to evaluate what that machine learning model thinks of the next compound. So exactly what you said. In comparison to any kind of experiment in the lab, or what my group had been doing before, which was physics-based modeling. Trying to use laws of physics to solve the properties of the material. The other thing we are increasingly using machine learning for is new ways to think about the molecules themselves.

Once we have a machine learning model and its opinion of how a million molecules should behave, we start to think on a different scale than we normally would, as scientists, when we think about only a handful of molecules. Interpreting what the model thinks, working with the model, it doesn’t just speed up the exploration of the space but it also gives us sort of a map of the space.

KL: What would you say the biggest challenges have been thus far?

HK:: One for sure, that a student of mine and I are working on a little bit, is that machine learning itself is a really exponentially growing area of research. Basically, it went in the past couple of years from no one I know applying it, or very few people applying it, to almost everyone. What that means is there’s some amount of hype cycle that you need to educate your students about but then there’s also the case that an undergraduate education does not necessarily uniformly teach students all the techniques they need to know. My student and I are currently writing a book and have prepared a workshop on machine learning for chemists. I would say that’s one step. I hope to hand that book to my students in the future to simplify some of that.

But I think it’s the wild west right now in terms of how much in a published paper are people reporting the technical details of what they’ve done, you know. Do we all universally agree upon some syntax for explaining what was done? Do we all use the same language? I think there are some challenges there.

KL: When do you hope that your book and workshop will come out?

HK:: The workshop, my students and I, as part of a MISTI grant, put that together last year. The slides are somewhere on the internet. I can probably get a link to you. But the book we’re sending to the publishers for the first draft this month, so hopefully next year.

KL: In the article it talks a lot about neural networks. Could you go into detail about those?

HK:: Sure. Without using too much terminology, neural networks are actually a really old idea and I learned this while writing the book that I just mentioned. Neural networks, artificial neural networks, were something that first captivated people in the 1950s. People were really interested in how the human brain worked and they were interested in building computational models of what they thought a brain was doing. What it was in a neuron was doing. The idea is that you send a signal to a neuron. the neuron responds by turning on, and there are these networks of all of these interconnected neurons and some information about that signal turns it into a final output or result. What’s funny is this died off in the 60s and 70s because the early neural networks people came up with were not complex enough.

What differentiates modern 2010s neural networks from these early 1950s and 1960s neural networks is they have what’s called a “hidden layer”. We started with the 1950s neural network, which has an input and an output. Basically, I send a signal to it and my signal might be anything we think of as a sense we have, but transformed for a computer. Then on the other side we get an output. For instance, input is, “it feels cold out”, output is, “I think I will put on a jacket.” But hidden layers are additional transformations of that signal. Mixing of the inputs, more transformations. Neural network transformations are highly nonlinear so it means that they can make more complex mappings.

But both the theory and the development of how you would build these models lagged for 20 or 30 years until actually in the 1980s and 90s, when people start to develop better algorithms and computers became able to train those models. We’re actually in the second renaissance of neural networks. Which is something I learned recently and why I think it’s cool to mention. If you look if you look back into the scientific literature in the mid-1990s, people were just as excited about neural networks as they are today for exactly the same applications as they are today. The difference is, still then, the kind of things that you could think about applying a neural network to were much smaller in scale. Again, because computing power and complexity weren’t there yet. What I would say is, the reason machine learning and neural networks are ever-present now is largely thanks to advances in computing power. What differentiates a 2019 neural network from a 1960 neural network is how these different neurons are connected and how complex those mappings are.

KL: You gave the example of an input being “it’s cold outside” output being “I’ll put a jacket on.” What does it look like when you add more complexity to it?

HK:: There are, without getting too far into the weeds, there are a lot of ways, for instance, that the most complex models today take a whole series of possible inputs and within the hidden layers, make choices about which ones are the most important. For instance, this becomes very hard to generalize an analogy, but the inputs could be the temperature, how hungry you are, how tired you are, what color your co-worker’s shirt is. The output could be some much more complex processed combination of these inputs that ultimately lead to you deciding to go out to lunch that day. The sum of these models, the most complex ones, essentially learn what is the most important piece of information from those pieces to make the ultimate decision or prediction that they want to make. The hidden layers allow the model to essentially transform the input features much more. In early simple models, even thinking about things much simpler that still have a relationship to neural network like linear regression. A lot of people have fit a straight line through data before, if they’ve completed a math class or done an undergraduate lab. Neural networks are doing that but in a much more complex way. The more pieces there are in the architecture, the more complex that mapping is.

KL: Can you talk more about your process with your team for training and the various rounds of testing you may have done to get to your final outcomes?

HK:: In the piece we actually used a relatively old, what we call representation. That’s what I was calling inputs a few minutes ago. How do we decide to tell a computer what a molecule is? There a lot of ways to do that. That’s actually, as chemists, or as chemical engineers, that is where most of the effort really goes in. In machine learning, there’s the question of, how do you choose to represent the molecule to the computer? How complex is the model you decide to train? How much data do you have? These are all interdependent questions and answers. If you have a lot of data, you can choose a very brute force representation. If you use a very sophisticated model, it can overcome limitations in your representation.

The earliest pieces of that work came from the homework project I mentioned earlier. We started with representations of transition metal complexes based on what I thought I knew as a scientist for about a decade’s worth of training, myself, personal training. I would come up with a list of different possible inputs and then we’d look at how much the model liked them or not. That was through a pretty extended trial-and-error process to train the initial neural network model. We then moved on to a more systematic ways of doing this. We showed that the systematic ways later could reproduce what I thought. I was very proud of my 10 years of experience and what I knew, but we could we could let statistics tell us which of those pieces were right or wrong. I’d say that was one of them more significant pieces of effort in that project.

KL: When you were envisioning this idea and how to develop it over time and get more systemic, did you have some sort of sketching process that you did for this?

HK:: No, I was a relatively new professor. I was just at a workshop where we commented, people who use computers don’t tend to draw things out very much. I was meeting my student every week and literally every week I’d say, “Well, why don’t you try this? This should really work.” Then he’d need to come back and say, “No, that doesn’t really help anything.” When we were moving towards a more systematic approach, we had been reading papers and found that there was a nice representation of molecules people had actually been using since the 80s that that looked like it had a lot of relationship to what we had already been working with. It contained everything we’d been already working with but could extend it. It was a natural fit.

KL: I’d like to go a little bit into your background, if that’s okay with you. You had mentioned earlier that you have materials science training and now you’re in chemical engineering. Can you talk more about your journey and what led you back to MIT?

HK:: I was a postdoc at Stanford University for the most part, working in a computational chemistry group. I never, when I was a student, expected that I would come back here to be a professor. It was very shocking at first. I had to make rules for myself about what I was going to do that was going to be different about being a professor at MIT than a student. For me that rule is you never walk in the tunnels. Tunnels are for students, not for professors.

KL: You mentioned earlier that there is a hype regarding machine learning, artificial intelligence, in the community right now. Do you think people are too excited about its potential or not as excited as they should be?

HK:: There was an article in Chemical Engineering News last year called, “Is machine learning overhyped?” Half of the community answered yes and half of the community answered, basically, it’s not hyped enough. I was one of the people who answered yes. I think I think it’s overhyped. The minute you have to write an article like that the answer is probably yes. But, you know, we found that, I guess, going back even a few years, you know, I’ve seen work that we’ve done in the traditional way get maybe one-tenth of the attention than work we’ve done with machine learning. I think both are equally valuable and so to a certain extent the machine learning work does stand to drown out some traditional work that I think is extremely important.

That being said, going back into where I started as a faculty member. We were really interested at the time in this field called cheminformatics. Cheminformatics was a mature field of the therapeutic drug discovery community basically taking organic structures and knowing based on structure how you could predict a property. Will that be a good drug molecule? Will it get into the bloodstream? We were interested in extending that to transition metal chemistry. Calling it cheminformatics is a lot less trendy these days than machine learning. Cheminformatics itself was trendy for a time. I think for me, I just get a little nervous about, is the community swinging so far one way that it’s going to have a sort of boomerang effect? What I used to tell myself is, every time we apply machine learning in in my group’s research, do we think we’re doing something fundamentally better then we could do through a traditional or more established technique? If I woke up tomorrow and machine learning was a bad word in the community, like saying it in my research meant that people would roll their eyes a little, would I still do what I’m doing? That’s something that I try to ask every time we apply machine learning in our research. To keep us anticipating that at least some applications of machine learning right now are overhyped. There are cases where people are essentially doing linear regression that’s been around forever and calling it machine learning. People are going to get tired of hearing words that are not descriptive enough for the research that’s being done.

KL: Could you provide a few examples of the traditional methods that you’re, you feel are being a little overlooked?

HK:: What’s interesting, or what I would probably describe without too much jargon about machine learning, is that machine learning excels where the relationship you are trying to build or learn is extremely nonlinear. That if you try to write down laws for why A goes to B, you would really struggle. But there are lots of fields. There things like, we know how fast an object will drop if we drop it. There are a lot of things where there are physics based laws where we can predict the outcome deterministically and with a rigorous relationship or rule, will exactly predict the outcome. As a computational chemist, a lot of my community is, what they were doing before they were doing machine learning is developing those rules. Coming up with ways to solve those rules on a computer, developing new rules and insights. That is what I would call the traditional way. Building a relationship that seems rooted in the laws of physics and chemistry that deterministically and analytically gives you why A goes to B. There are fields of science where it’s hard to figure that out, or where the problem is too complex. There are fields of science where it’s easier to figure that out. Discriminating between the two is, I think, important.

KL: What are the key takeaways that you hope folks will have from your machine learning research, in terms of the transition metal compounds?

HK:: I would say for us some of the key things have been, like I mentioned earlier, coming up with a new way to understand structure property relationships. If a complex looks a certain way, what are its properties going to be? One of my, I think, favorite figures to come out of this, which may not have been in the article, is we imagined the properties of a few thousand transition metal complexes. We mapped them onto a picture of a very common transition metal complex known as a porphyrin. A porphyrin is also known heme and it’s responsible for how we breathe. Its binding of oxygen and release of oxygen is how oxygen circulates in our body. It’s a pretty important molecule. With our machine learning models and representations, we’re able to look at which parts of the heme light up to predict different properties of the heme. One is a kind of esoteric property that we’ve spent a lot of time on known as the quantum mechanical ground-state spin. This is important because different spin states of molecules have different macroscopic properties. If our hemes in our body were in a different spin state, then we would not be able to breathe oxygen properly, this type of thing. But then also something else like redox potential which is relevant for energy storage. By building these maps, we’re able to see that the parts of the molecule that light up for redox potential are different from the parts of the molecule that light up for spin. These days we build these maps for everything, for catalysis. We’re able to think, both through the machine learning model and through these property maps, are these two properties that I can design independently to meet an overall design objective, or are they two that are too similar? Once I tune one I’m going to tune the other. That’s really where my group is going in terms of applying these models. We are trying to push machine learning to the point where we can satisfy multiple design objectives in one molecule. In a way that traditional trial-and-error would not work.

KL: Could you define spin for me?

HK:: Spin is a quantum mechanical property of electrons. Electrons are a fundamental particle, the best part of the atom and electrons carry spin. When I say spin of a molecule, what I’m referring to is whether those electrons pair up or whether those electrons like to be unpaired. Traditionally knowing if they want to be paired or not is something that is a really hard property to predict with any with any certainty. Normally the only way to accomplish that is to use methods on a computer that model the electronic structure of these systems. Those calculations take a very long time and so machine learning models that predict spin are advantageous. But here I mean it to say, how many electrons are paired? An electron can have spin up and spin down and if you have two of them and there’s spin up and spin down then they’re paired and there’s no net spin.

KL: It seems like a battery almost? Like spin up is positive and spin down is negative?

HK:: Another analogy to maybe bring it closer to things people are used to thinking about is, basically, all magnets you’ve ever worked with have a permanently aligned net spin of their electrons. If you have a permanent magnet, that means all of the spins always stay and all pointed in one direction with a net spin. If you have something that will magnetize, when you bring it next to, when you bring it next to a magnet but then it loses it, that’s actually all of the electrons aligning in one direction temporarily. Then they’re relaxing their materials properties, meaning that they’re going back and they can be paired up again. If you’ve ever taken like a pin and magnetized it with a with a fridge magnet or something like that. There are interesting materials properties that relate to electronics and energy storage as well, where it may be advantageous to have the spins paired up or it may be advantageous to have the spins aligned. In terms of practical applications, being able to control that spin or switch it and hold it in one direction is really where the community is going for the smallest units of electronics. Where the quantum computing community is headed, for instance, that would be the most obvious practical application beyond fridge magnets.

KL: In terms of energy storage, can you give some examples of where spin would be helpful there?

HK:: I think that will take things down all a little bit of a trickier road. I think the most complex thing to say there is, basically, controlling in which spin channel electrons conduct can be helpful for designing pretty complex electronics. In terms of energy storage, a lot of the most promising redox flow batteries are transition metal complex catholyte and anolytes, with their own net spin and electronic properties. Changing the spin states of those molecules changes their redox capacity. Spin is something that’s always present, it’s just not something that is covered a lot. It’s sort of always in the background when you’re learning some of these concepts early on.

KL: Why do you think that is?

HK:: You’d need to take a year of quantum mechanics to understand, to begin to understand where the spin quantum number comes from. There are things that I think people learn in undergraduate chemistry, such as the Pauli Exclusion Principle, telling them how to pair up electrons or that there are certain rules. We tend to learn things the first time in chemistry and materials as heuristics. Spin is behind a lot of those heuristics but they tend to get written down by, okay, this wants to bond this way or these atoms are going to behave like this. For instance, oxygen, which I mentioned, is one of the most well-known examples. Oxygen has a net spin. The oxygen molecule O2, in its ground state, is a triplet. Which means it has two unpaired electrons. Singlet oxygen would be if we paired up all of the electrons. It can be toxic and damaging to cells and we would not want to breathe singlet oxygen, so spin is there, even in the most fundamental building blocks of life. It’s just not something most people spend a lot of time thinking about.

KL: How does using machine learning affect energy efficiency of computers?

HK:: It’s funny, I think we think about this all the time or it’s always in the background of what we think about as computational researchers. As computing gets more and more powerful, it becomes increasingly possible for me to have a supercomputer running full-time that draws down a lot of energy. What I would say about machine learning that’s interesting, or that we all have to admit as a community, is that we need training data from somewhere. Where my group started is to run those computational chemistry calculations, which are very time-intensive. Then model training, training the machine learning model, can take anywhere from minutes to maybe up to a day, depending on the complexity of the model. Those are both relatively energy-intensive processes. But once trained, the model costs nothing to evaluate. That’s where you start to see the payoff in efficiency. We’re now able to do one-tenth of the calculations we’d normally do.

An interesting example of this, that I hope I can explain clearly, is we actually have built machine learning models designed to take that a step further. We normally use the machine learning model to go out and discover new compounds. But the problem is, the model might take us to a place where the molecule isn’t valid. The molecule might fall apart. Or if a human were to set up that molecule, the human would look at it and say, no that’s not a good idea. I have my PhD. I know you should never simulate that model. In a recent paper for my group, we developed two different ways where we could decide, as a simulation is running, we should… this is a waste of computer time. Stop the simulation. The machine learning model now is not just predicting properties, but it’s predicting, will this simulation be a good use of computer time? We have two models like that. One actually looks at just the molecule, without doing any calculations at all, and says, should you run this calculation, yes or no? These calculations are time -intensive. What they essentially lead to is that we’re able to look at all of the practical molecules that we could characterizes as new leads and spend only a third of the time that we would normally spend on a computer to characterize them. That is that is one way I would say machine learning has led to efficiency directly in a practical way. Though it is a little esoteric.

KL: In the article, it refers to molSimplify. Can you tell us about that?

HK:: molSimplify is my group’s open source code for automating transition metal complex discovery. We put all of our machine learning models in there. It really goes back to that initial goal I mentioned to you, which was bringing cheminformatics to materials.

KL: You said it’s open source, right? Anyone can find it and check it out?

HK:: Yeah. It’s on our website. It’s available via Conda and Github. We’ve actually even used it to teach in my undergraduate/graduate computational chemistry class.

KL: It’s hard to talk about neural networks without mentioning Elon Musk’s Neuralink, which had its first public launch event in San Francisco last month. How similar are these neural networks to the ones that you’re working on?

HK:: My understanding is Neuralink is about bringing the interaction between people and machine learning models even closer. Right now, I think all of us are interacting more and more with machine learning models, but we don’t have electrodes directly implanted in our brains. I think there are, and I’m by no means an expert on it, I think there are interesting questions for us to ask as a society when machine learning models become capable of making decisions or aiding or influencing decisions, and how are we going to interact with them. I think that Neuralink will definitely force those conversations to happen sooner than later. But if those types of things can ultimately help in a lot of neurological diseases, there are a lot interesting questions I think in terms of, can you interpret and help people who are having, you know, things like Parkinson’s. Some of that already exists, where you need a direct feedback. There’s a clear need based on the health condition of need for feedback between what’s happening in the brain and being able to deliver medicine let’s say.

KL: For people who are interested in your area of research, what sorts of reading recommendations would you have for them or recommendations for other resources?

HK:: I think this is an interesting area, in the sense that domain knowledge is always key. I would not underestimate the value of domain knowledge. By which I mean, what is the physics or chemistry or materials problem you’re interested in. But then looking at the connection to machine learning, I think there is, unfortunately, I’m not going to remember the name of the title of the book, there is a book called like “Learn Machine Learning in One Hour” [“Mastering Machine Learning in One Day“] that’s particularly well-written but I don’t think that’s the precise title. I might have to go look it up. But I think there are a lot of great resources on the web, a lot of good online courses. I think I wouldn’t overestimate how expert you need to be to start applying techniques. I think on the low end, there are really useful things to learn about, that have application to even how people are doing traditional science that come out of machine learning or are more central in machine learning. These would be things like, learning to cross-validate. Learning to leave some of your predictions out. When you’re making a prediction, leave some of your test data out and see how well your model actually predicts something. Learning how to do feature selection or regression. Those things are well-covered in lots of basic texts and are a good place to start. You don’t have to start at neural networks, is my long answer. You could just cut to that.

KL: What’s next for you and your group?

HK:: I think the thing that we need to push for the most, is really showing now what we can do with machine learning models once they’re trained. We’ve spent a lot time showing we can train machine learning models. But really the exciting thing is to start to design materials with them in ways that we couldn’t before and for us that means a lot of multi-objective optimization. Which means, I want a molecule that has this perfect property but I also want it to be cheap or I want it to be earth-abundant or I want it to be stable and hold up for a lot of cycles or I want a redox couple that’s also soluble in the electrolyte. Satisfying many different design objectives at once, I think is where the machine learning models are going to shine.

KL: Where can we keep up with your research?

HK:: My group is on Twitter, where we update things a lot. Also, our group’s website. I’d say a lot of the computational chemistry and machine learning communities are on Twitter these days.

KL: Thank you so much for joining us today and talking about your research. It was such a pleasure.

HK:: Thanks for having me.