{"id":743248,"date":"2021-05-05T08:00:00","date_gmt":"2021-05-05T15:00:00","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?p=743248"},"modified":"2021-07-19T09:29:27","modified_gmt":"2021-07-19T16:29:27","slug":"advancing-excel-as-a-programming-language-with-andy-gordon-and-simon-peyton-jones","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/podcast\/advancing-excel-as-a-programming-language-with-andy-gordon-and-simon-peyton-jones\/","title":{"rendered":"Advancing Excel as a programming language with Andy Gordon and Simon Peyton Jones"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"577\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1024x577.jpg\" alt=\"Advancing Excel as a programming language with Andy Gordon and Simon Peyton Jones on the Microsoft Research Podcast\" class=\"wp-image-743689\" srcset=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1024x577.jpg 1024w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-300x169.jpg 300w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-768x432.jpg 768w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1536x865.jpg 1536w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-2048x1153.jpg 2048w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-16x9.jpg 16w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1066x600.jpg 1066w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-655x368.jpg 655w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-343x193.jpg 343w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-640x360.jpg 640w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-960x540.jpg 960w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1280x720.jpg 1280w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1920x1080.jpg 1920w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h3 id=\"episode-120-may-5-2021\">Episode 120&nbsp;|&nbsp;May 5, 2021&nbsp;<\/h3>\n\n\n\n<p>Today, people around the globe\u2014from teachers to small-business owners to finance executives\u2014use Microsoft Excel to make sense of the information that occupies their respective worlds, and whether they realize it or not, in doing so, they\u2019re taking on the role of programmer.&nbsp;<\/p>\n\n\n\n<p>In this episode, Senior Principal Research Manager&nbsp;<a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/people\/adg\/\" target=\"_blank\" rel=\"noreferrer noopener\">Andy Gordon<\/a>, who leads the&nbsp;<a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/project\/calc-intelligence\/\" target=\"_blank\" rel=\"noreferrer noopener\">Calc Intelligence team<\/a>&nbsp;at Microsoft Research, and Senior Principal Researcher&nbsp;<a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/people\/simonpj\/news-and-awards\/?facet%5Btax%5D%5Bnews-tab-types%5D%5B%5D=post\" target=\"_blank\" rel=\"noreferrer noopener\">Simon Peyton Jones<\/a>&nbsp;provide an inside account of the journey Excel has taken as a programming language, including the expansion of data types that has unlocked greater functionality and the release of the LAMBDA function, which makes the Excel formula language Turing-complete. They\u2019ll talk specifically about how research has influenced Excel and vice versa, programming as a human-computer interaction challenge, and a future in which Excel is the first language for budding programmers and a tool for incorporating probabilistic reasoning into our decision-making.&nbsp;&nbsp;<\/p>\n\n\n\n<h3 id=\"learn-more\">Learn more:&nbsp;<\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>Excel Blog:&nbsp;<a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/techcommunity.microsoft.com\/t5\/excel-blog\/announcing-lambda-turn-excel-formulas-into-custom-functions\/ba-p\/1925546\" target=\"_blank\" rel=\"noopener noreferrer\">\u201cAnnouncing LAMBDA: Turn Excel formulas into custom functions\u201d<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>&nbsp;<\/li><li>Microsoft Research Blog:&nbsp;<a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/lambda-the-ultimatae-excel-worksheet-function\/\" target=\"_blank\" rel=\"noreferrer noopener\">\u201cLAMBDA: The ultimate Excel worksheet function\u201d<\/a>&nbsp;<\/li><li>Research Collection:&nbsp;<a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/blog\/innovation-by-and-beyond-the-numbers-a-history-of-research-collaborations-in-excel\/\" target=\"_blank\" rel=\"noreferrer noopener\">\u201cInnovation by (and beyond) the numbers: A history of research collaborations in Excel\u201d<\/a>&nbsp;<\/li><\/ul>\n\n\n\n<h3 id=\"related\">Related:&nbsp;<\/h3>\n\n\n\n<p><strong>Subscribe to the<\/strong> <a href=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/podcast\"><strong>Microsoft Research Podcast<\/strong><\/a><strong>:<\/strong><br><a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/itunes.apple.com\/us\/podcast\/microsoft-research-a-podcast\/id1318021537?mt=2\" target=\"_blank\" rel=\"noopener noreferrer\">iTunes<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> | <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribebyemail.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noopener noreferrer\">Email<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> | <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/subscribeonandroid.com\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noopener noreferrer\">Android<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> | <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/open.spotify.com\/show\/6XNI6rGsLFYlWJxLItw4S0\" target=\"_blank\" rel=\"noopener noreferrer\">Spotify<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> | <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/www.blubrry.com\/feeds\/microsoftresearch.xml\" target=\"_blank\" rel=\"noopener noreferrer\">RSS feed<span class=\"sr-only\"> (opens in new tab)<\/span><\/a> | <a class=\"msr-external-link glyph-append glyph-append-open-in-new-tab glyph-append-xsmall\" href=\"https:\/\/note.microsoft.com\/ww-registration-microsoft-research-newsletter-s.html?wt.mc_id=S-webpage_podcast\" target=\"_blank\" rel=\"noopener noreferrer\">Microsoft Research Newsletter<span class=\"sr-only\"> (opens in new tab)<\/span><\/a>\u202f\u00a0<\/p>\n\n\n\n<hr class=\"wp-block-separator aligncenter\"\/>\n\n\n\n<h3 id=\"transcript\">Transcript&nbsp;<\/h3>\n\n\n\n<p><strong><em>TEASER (SIMON PEYTON JONES):<\/em><\/strong><em>&nbsp;I&nbsp;don\u2019t&nbsp;think I ever dreamt that we could offer something as exotic as LAMBDA.&nbsp;You could really write literally any program in Excel now.&nbsp;Certainly, it becomes computationally much more powerful.<\/em>&nbsp;<\/p>\n\n\n\n<p><strong>[MUSIC&nbsp;BREAK]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>ANDY GORDON:<\/strong>&nbsp;Welcome to the Microsoft Research Podcast.&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>[MUSIC&nbsp;BREAK]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:&nbsp;<\/strong>My name is Andy Gordon.&nbsp;The new LAMBDA function has been announced in Excel, and I\u2019m here to tell you about that with my colleague Simon Peyton Jones. Simon, would you like to introduce yourself?&nbsp;<\/p>\n\n\n\n<p><strong>SIMON PEYTON JONES:<\/strong>&nbsp;Yeah, so, I\u2019ve been here at Microsoft Research for 22 years, since 1998. I did my undergraduate&nbsp;degree at Cambridge. I worked for a few years in a small company, and then I worked at University College London and Glasgow University as a professor before moving to Microsoft. And my research interest has always been in functional programming, purely functional programming, as a radical and elegant attack on the entire enterprise of writing programs. But what about you, Andy? How did you get into functional programming and, indeed, Excel?&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Well, I was doing a PhD on lazy functional programming back in the late&nbsp;\u201980s. In fact, when you guys were starting Haskell, my PhD was on input-output for Haskell, and I was&nbsp;actually delighted&nbsp;you personally invited me to sit on the Haskell committee to help standardize input-output using monads. And I joined Microsoft&nbsp;actually in&nbsp;\u201997, so I was one of the first employees at&nbsp;Microsoft Research Cambridge.&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;That\u2019s right. You narrowly predated me.&nbsp;<strong>[LAUGHTER]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Not just that. I interviewed you.&nbsp;That was a tough decision.&nbsp;<strong>[LAUGHTER]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Well, you made a good call.&nbsp;<strong>[LAUGHTER]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;I think mostly. Yeah, so, about my research. Yeah, I started&nbsp;out&nbsp;in functional programming, and I\u2019ve done a bunch of other things in, like, security. And then about 10 years, I got into probabilistic programming for machine learning, which funny enough, led me to spreadsheets because we built a system to support writing probabilistic programs of data within spreadsheets. And at that point, I realized that to take things much further, it would be good to join forces with your work with the Excel team. And in fact, you were involved with the Excel team&nbsp;pretty early&nbsp;on in your time at Microsoft. Do you want to tell us about that?&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Yeah,&nbsp;pretty early, because when I first joined Microsoft, I thought to myself,&nbsp;\u201cWhat can I do that would advance the course of functional programming within Microsoft?\u201d&nbsp;And then I thought,&nbsp;\u201cWell, Excel is the world\u2019s most widely used functional programming language.\u201d&nbsp;It\u2019s not a very powerful one, perhaps, but when you write a formula in a spreadsheet, you&nbsp;<em>are<\/em>&nbsp;writing in a purely functional language. So, no side effects. You don\u2019t say, you know,&nbsp;\u201c=print3+7\u201d;&nbsp;it&nbsp;wouldn\u2019t make any sense.&nbsp;And&nbsp;moreover, it\u2019s extremely widely used, so there must be a hundred times as many users of formulae in Excel as there are professional programmers in the entire planet. And so, I was thinking maybe we should think about what&nbsp;would it&nbsp;take to start from Excel, but to grow it using the North Star of mainstream functional programming&nbsp;languages&nbsp;and try to see how much more powerful we could make it.&nbsp;Then I started talking to research colleagues, like Alan Blackwell here at Cambridge and Margaret&nbsp;Burnett in Oregon, and we started to come up with some ideas about defining functions of Excel. We had two&nbsp;principal&nbsp;things. The first was, Excel provides 600-odd&nbsp;<em>built-in<\/em>&nbsp;functions, but it doesn\u2019t provide you with a way to make a new function out of existing ones. So, in every other programming language, you can define a procedure, perhaps, by writing some code, wrapping it up, giving it a name and some parameters, and then you can call it repeatedly. Not so in Excel. If you want to do that, you&nbsp;have to&nbsp;write your procedure in VBA or JavaScript or something, and that\u2019s kind of, like, crazy. We\u2019d like you to be able to define&nbsp;<em>new<\/em>&nbsp;functions using the existing formula language. Every other programming language lets you do that. Why not Excel? So, we came up with a design for doing that, and we turned it into a research paper. It was published in ICFP in about 2002&nbsp;or&nbsp;\u201903. It was called&nbsp;\u201cUser-defined functions in Excel.\u201d&nbsp;So then, we started to realize that it wasn\u2019t just enough to have user-defined functions. They needed to be able to take&nbsp;<em>structured<\/em>&nbsp;data. It&nbsp;wasn\u2019t&nbsp;enough for them to take scalar data. And this is a part which you\u2019ve been much more involved in since the need for rich data in Excel, which sort of complements the need for user-defined functions. So, maybe you could tell us a bit about that.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah. I mean, you\u2019re right, Simon. I mean, until&nbsp;pretty recently, the only kind of data that you could have in Excel was text or numbers. That\u2019s all you could have sitting in cells, and so, if you\u2019re trying to turn the formula&nbsp;language&nbsp;into a proper functional programming language, you really need a lot more structures like that.&nbsp;There\u2019s only so many functions you can write that just take two scalars as input and produce another one as an output. There\u2019s not many of those. So, for example, you need a raise.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;You mean as&nbsp;arguments&nbsp;and results of functions?&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yes. You want to be able to process big pieces of data in one go, and&nbsp;it\u2019s&nbsp;also important to be able to store them in cells.&nbsp;You know, within a sheet-defined function, you might want to have a computation and have it sort of spread out within the grid and then return a whole array, say, as the result of a sheet-defined function. So, generally, we need&nbsp;first-class arrays in&nbsp;the&nbsp;language.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Yeah, we should pause a bit just to explain about what&nbsp;is a sheet-defined function. The idea that we put in this paper for defining new functions in Excel was to say&nbsp;imagine you could take a worksheet and nominate certain cells as the input and one cell as the output and give it a name. And then when you call a function with that name, when you call that function, it is as if you had created a fresh copy of that worksheet, filled in the input values, the arguments of the function in those input cells,&nbsp;calc\u2019d&nbsp;the worksheet, and taken the result out. That was our model for defining a new function. Now, if you want to define a function that works&nbsp;<em>over<\/em>&nbsp;arrays, for example,&nbsp;\u201cSort this array\u201d&nbsp;or&nbsp;\u201cPick the smallest element of it,\u201d&nbsp;then you&nbsp;can\u2019t&nbsp;just take an array of a&nbsp;<em>fixed<\/em>&nbsp;size, like a&nbsp;three-element array or a 30-element array. We\u2019d like to have an array of arbitrary size, which presumably would then land in a single cell in the worksheet that defines the function.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;This is a great vision, and you wrote a paper about that. So, how do things develop, Simon?&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Well, then, so having got the idea, I then would travel to Redmond, the Microsoft mothership, on a&nbsp;fairly regular&nbsp;basis, you know, two or three times a year.&nbsp;So&nbsp;I started to become very familiar with the Excel team, and I discovered that they were very warm towards the idea\u2014we&nbsp;had lots of interesting technical conversations about how Excel might work\u2014but Excel is a very&nbsp;<em>big<\/em>&nbsp;product, and it doesn\u2019t move quickly. It\u2019s like a sort of supertanker. So, for a long time, I felt a bit like a fly bashing against the&nbsp;bow&nbsp;of the supertanker, and there was always a good reason why now is not a good time to do it, and no criticism for that. There were other priorities, and there still are. But what&nbsp;actually happened&nbsp;in the end was that I got a bit discouraged,&nbsp;and I went away for about 10 years. I forget what provoked me to&nbsp;. Maybe it was our lab director, Andrew Blake, who said,&nbsp;\u201cYou should really have another go at this.\u201d&nbsp;But&nbsp;it&nbsp;turned out it matched up with their priorities, so we then started a much&nbsp;more active partnership between Microsoft Research Cambridge and Excel on this whole idea of sheet-defined functions and data. But things came out in a very different order than I&nbsp;anticipated. I had started on the sheet-defined function idea, but in fact, the first thing that happened was&nbsp;actually to&nbsp;do with data types and&nbsp;rich&nbsp;structured data involved in functions. In particular, the first thing that came out as a change to the product was called dynamic arrays, and you were quite involved in that, weren\u2019t you? Tell us a bit about it.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;So, dynamic arrays are&nbsp;really interesting&nbsp;because if you go way back, you discover that the formula language in Excel could in fact construct arrays.&nbsp;You could refer to bits of grid, a range, and then pull that into a formula and compute with it, but what you couldn\u2019t do was return the whole array in one go.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;You could do things like add two arrays together or add&nbsp;1&nbsp;to every element in an array.&nbsp;You could treat arrays as first-class values within a formula.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Right, and so you&nbsp;could lift formulas over whole arrays, so, like, plus&nbsp;1&nbsp;and so on, or you could lift over the whole&nbsp;array&nbsp;and you could get the result, but you could only have a scalar in a cell, and so, it was very awkward to return&nbsp;the&nbsp;whole array. There was a notion of a thing called&nbsp;<em>array formulas<\/em>, and if you were on Windows, the way you entered them was to do control-shift-enter, I think after you\u2019d&nbsp;<strong>[LAUGHTER]<\/strong>\u2014yeah, you\u2019d type the formula in, and then you&nbsp;had&nbsp;to remember to go control-shift-enter. It was very complicated. You would select the range, you know, multiple cells that were to hold the array,&nbsp;and&nbsp;you needed to know in advance how big it was, and then you would type in the formula and end it with control-shift-enter, and then that one formula would define the whole array. So, it was&nbsp;really cumbersome.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;<strong>[LAUGHTER]<\/strong>&nbsp;It\u2019s&nbsp;sort of hilarious, really, when you think about it. Curly&nbsp;braces&nbsp;would appear around the formula.&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;So, it was this advanced feature, and there were a few people who knew how to use it. In fact, there was a person who&nbsp;actually wrote&nbsp;a book called&nbsp;<em>Ctrl+Shift+Enter<\/em>, which was this sort of guidebook to how you could sort of use this feature, but it was very difficult to use. But&nbsp;now, I think since about 2018, we\u2019ve had dynamic arrays. And this is an amazing feature. You don\u2019t have this separate world of array formulas. Ordinary formulas can return arrays, and you don\u2019t have to decide in advance how big the array is going to be. Instead, you just write the formula, and if it computes an array, then we see it spills out from the cell.&nbsp;Maybe you write a formula that, you know, is a 3&nbsp;\u00d7&nbsp;3 array, and it will just spill out from\u2014you write the formula in the top left corner, and then the rest of the array will fill out. And&nbsp;then if you want to have another calculation somewhere else\u2014if the original array was in cell A1, then if you say&nbsp;\u201cA1 sharp\u201d&nbsp;then that expression, that formula, A1 sharp, refers to the whole array, and then you can compute to it. You can add that whole array to another one or take the sum of it or take the average of it.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;So&nbsp;there&nbsp;you don\u2019t&nbsp;need to&nbsp;know how big the array in A1 sharp is, and indeed, in different&nbsp;calc cycles, it might be different sizes.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Right.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;That\u2019s&nbsp;really important;&nbsp;the chain of formulae will work regardless of the sizes of the arrays involved, which may vary with the data.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;When dynamic arrays came out, we didn\u2019t have LAMBDAs, so I think for everyone in the team, we knew that LAMBDAs were the thing and combinators together with LAMBDAs like map and&nbsp;reduce and fold and scan\u2014these kinds of combinators that are familiar from functional languages like Haskell and others\u2014we knew that those would be really important for programming with arrays, but when they first came out, we&nbsp;didn\u2019t&nbsp;have those. So, we\u2019ve been struggling a little bit to really get the full power of dynamic arrays.&nbsp;There\u2019s&nbsp;various tricks you can use to construct dynamic arrays, but certain kind of common patterns are still quite hard to use, which changes with LAMBDAs coming in.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Dynamic arrays still produced a lot of excitement at the time because they really unlocked a huge amount of functionality. It was only previously available through this very arcane control-shift-enter mechanism, and then only if you knew the size of the array. So, they\u2019re dynamic. They really work&nbsp;over&nbsp;a variable size.&nbsp;It wasn\u2019t just arrays, either. So, arrays are a very natural data type for Excel, but we also talked to our colleagues in the product team a lot about records, and that came out at a similar kind of time, as well, didn\u2019t it?&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yes. So, these are known as&nbsp;<em>entities<\/em>&nbsp;in Excel, and you can get them from the data tab in the ribbon, and the funny thing is that they are records, but there isn\u2019t&nbsp;actually a&nbsp;function to create a record from scratch. Instead, the way it\u2019s been introduced is that Excel has got this intelligence feature that can recognize certain names of entities that occur in the workbook. So, if you refer to the name of a company or if you refer to a name\u2014say, Cambridge\u2014Excel can recognize Cambridge as a geographic entity, like the city of Cambridge in the UK, where Simon and I are at the moment, and what Excel will do is go out to Bing\u2019s database, fetch facts about the city of Cambridge, wrap them up in a record, and then drop that into the cell, and if it\u2019s the name of a company, then it would also grab facts like the current stock price or the name of&nbsp;the&nbsp;CEO and drop that into the grid.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;You&nbsp;have to&nbsp;do something to make it do that,&nbsp;don\u2019t&nbsp;you? It doesn\u2019t just spot\u2014anytime you say&nbsp;\u201cCambridge,\u201d&nbsp;it doesn\u2019t just go\u2014yeah, there\u2019s some gesture.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;You go into the data ribbon,&nbsp;and actually, one&nbsp;of the really cool things about those entities is they\u2019re known as&nbsp;<em>linked entities<\/em>.&nbsp;It\u2019s not just a record, but they also have this link to where they came from, sort of&nbsp;a&nbsp;pointer&nbsp;to the&nbsp;web,&nbsp;if&nbsp;you like. So, it means that if you&nbsp;open up&nbsp;a stock data type in a workbook, Excel will actually automatically refresh properties like the price, and then it would&nbsp;recalc&nbsp;anything that depends on that. So, these have become&nbsp;really popular&nbsp;as ways of building financial models or getting properties of geographies.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;But from a programming point of view, the interesting thing about this was the value in the cell was no longer a scalar. It really is a record, and you could programmatically say&nbsp;\u201cA1.price\u201d&nbsp;to extract the field. That\u2019s the first time that kind of rich, structured data of nested records was available.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah, I think that\u2019s the first time Excel had&nbsp;\u201cdot\u201d&nbsp;actually as&nbsp;an operator. So, we now have, like,&nbsp;\u201cdot,\u201d&nbsp;like in many other programming languages, with IntelliSense,&nbsp;so it&nbsp;kind of&nbsp;gives you an option for different fields you might want to pull out of the&nbsp;record.&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;It was a funny dynamic that, for a programming languages person,&nbsp;you\u2019d&nbsp;say,&nbsp;\u201cWell, we need an introduction form, which lets you create records, and then an elimination form that lets you take them apart\u201d\u2014that\u2018s the dot. But the way it came out in the product is we have the&nbsp;elim&nbsp;form but without the intro form. I think we all get an intro form, but it was just an interesting way in which working with a product group makes things come out in a different way, probably a way that\u2019s more useful for users in the first instance&nbsp;than&nbsp;we would have done as academics. I thought that dynamic was interesting.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah, actually, I mean, stepping back, Simon, I think this is one of the really big things that&nbsp;I\u2019ve&nbsp;learned from working with the Excel team\u2014I mean,&nbsp;we\u2019ve&nbsp;been working really intensely on&nbsp;this the last&nbsp;five years\u2014which is the kind of people who use Excel.&nbsp;I mean,&nbsp;it\u2019s&nbsp;a&nbsp;really broad&nbsp;population\u2014it\u2019s a super popular application\u2014but the kind of people who use it are generally not programmers, people who really care about writing code for its own sake. Like most people who are sort of functional programmers, say, Haskell programmers, are really passionate about programming, whereas people who are using Excel, not always, but generally, they are really keen to get some other job done, like they\u2019re an accountant,&nbsp;they\u2019re&nbsp;someone working in finance, maybe a&nbsp;teacher tracking statistics&nbsp;about how their students are doing, and&nbsp;they\u2019re&nbsp;what are known as end-user programmers.&nbsp;They are&nbsp;writing code\u2014they\u2019re&nbsp;writing programs\u2014but really for their own purposes. So, when the Excel team thinks about how best to serve end-user programmers, they often do things a little differently than if you were trying to produce a feature for a very sophisticated programmer.&nbsp;Entities are a good example of that, where the Excel team&nbsp;realized&nbsp;that end&nbsp;users who maybe cared&nbsp;about stocks would just want a very convenient way to go from the name of a company to an entity that would automatically update the stock price rather than having to have maybe a function called&nbsp;\u201crecord\u201d&nbsp;that they would call and have to put in arguments to determine which entity to fetch, and I think that\u2019s been quite successful, that end&nbsp;users have really been able to use these features, and they&nbsp;don\u2019t&nbsp;really know what a record is, but they can understand that particular entities matter for what they\u2019re trying to do.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;We also laid a lot of answers on&nbsp;compositionality. I think&nbsp;for&nbsp;the first time,&nbsp;we had&nbsp;not just&nbsp;data that wasn\u2019t&nbsp;a&nbsp;scalar, but data that was arbitrarily nested. So, a record for a country might contain a field that is the states of that country, and that\u2019s an array, an array of states. So, it\u2019s a record that contains an array, and that array contains entities themselves that are descriptors, records about the state. So, you get the sort of compositional, deeply nested form of structures that we\u2019re familiar with in programming&nbsp;languages, but which is&nbsp;really not&nbsp;in Excel at all until this point.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah, it\u2019s&nbsp;really rich.&nbsp;For example, Microsoft has done this amazing deal with Wolfram, the company that does the&nbsp;WolframAlpha&nbsp;knowledge base, and so Microsoft is giving access to a huge range of data types from Wolfram, like chemical compounds and stars and different food items with their calories and so forth, so there\u2019s a great repertoire of data and a lot of it is nested in the way that you\u2019re talking about.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;We started to go further than that, even.&nbsp;After we got arrays and records\u2014those are like two data types\u2014you might think,&nbsp;\u201cWell, what if we want more?\u201d&nbsp;So, we started to think,&nbsp;\u201cWhat would it mean to make Excel extensible with respect to&nbsp;its&nbsp;data types?\u201d&nbsp;Maybe you could walk up to it as a third-party developer\u2014now&nbsp;not as an end&nbsp;user, but as a real programmer\u2014and plug in something that adds another data type to Excel, let\u2019s say images, right? So, you might have an abstract data type of images with operations you provide that can overlay&nbsp;them&nbsp;or blur them or rotate them, and then you have a little functional programming language whose data types are images, and Excel serves as a sort of orchestration language that lets you write&nbsp;formally&nbsp;they\u2019re&nbsp;connected, but the new data&nbsp;type&nbsp;is provided as a third-party plug-in.&nbsp;That\u2019s not available yet. It\u2019s part of&nbsp;<em>our<\/em>\u2014you know,&nbsp;your&nbsp;and my vision\u2014and I think the team&nbsp;in&nbsp;Redmond takes&nbsp;it&nbsp;pretty seriously, and we\u2019ve designed the implementation to accommodate that, but we\u2019re starting with some fixed data types. But the idea that ultimately, we could turn Excel into an extensible platform for arbitrary data&nbsp;types&nbsp;that do&nbsp;really quite&nbsp;big, remarkable things in a financial contract,&nbsp;say,&nbsp;is quite an exciting one, I think.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah, that\u2019s&nbsp;gonna&nbsp;be&nbsp;really cool&nbsp;because we\u2019ve seen that customers often have used things like previous extensibility features like VBA to in a way sort of hack up some of these features for representing things like financial contracts or financial instruments and sort of simulate things like arrays and records using strings and various other tricks like that. But once this feature&nbsp;comes out, it\u2019ll be&nbsp;possible for third-party developers to sort of really enrich the product with new data types. It\u2019ll be extremely cool.&nbsp;<\/p>\n\n\n\n<p><strong>[MUSIC BREAK]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;So, we talked quite a bit about the way in which our partnership has influenced the product, but the reverse has also happened. In fact, we\u2019ve used the credibility with the lab of this very effective&nbsp;product&nbsp;group partnership to justify growing our research program into a group that we call the Calc Intelligence&nbsp;group, which you run now. Maybe you could just tell us a few examples of the broader&nbsp;research-based&nbsp;things&nbsp;we\u2019ve been doing on the back of all this.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah, we\u2019ve done a range of different things, and we\u2019ve got papers out on different topics. So, one example is what we call elastic SDFs. So, Simon, you\u2019d explained the idea of sheet-defined functions, that we can describe a calculation by a piece of grid with some of the cells being inputs and some of them being outputs, and I think one thing in your original paper that you didn\u2019t cover was what would happen if one of the inputs was&nbsp;actually a&nbsp;range,&nbsp;was like an array of items.&nbsp;And&nbsp;this is one of the cool things that we\u2019ve&nbsp;worked on together, is like&nbsp;what if the example that you gave was maybe a row of&nbsp;like three items, but you want to generalize to a sheet-defined function that wouldn\u2019t work just on a row of three items but maybe would work on a row of five items or any sort of array input size. And very often, we can do a sort of analysis of the formulas, and if they are processing those three inputs in a uniform way, typically it would be&nbsp;exactly the same&nbsp;formula as applied to&nbsp;each of&nbsp;them, a sort of parallel computation on the three inputs, and maybe with aggregates at the end where maybe you do some calculations and then you sum the results. Then that calculation can be sort of stretched\u2014it can&nbsp;be&nbsp;elasticized, is what we say\u2014to cover maybe an input of size five. So, we wrote this paper that I think is&nbsp;really cool&nbsp;that figured out a theory for when it is possible to generalize an example calculation of the grid that has fixed sizes to arbitrary sizes, and then even better, we worked with HCI experts who are in the Calc&nbsp;Intel team to do a user study to ask,&nbsp;\u201cWell, how do people actually find this in practice?\u201d&nbsp;And we did a study and found that they loved it, that it was a very natural way to program sheet-defined functions. So, that\u2019s a paper I\u2019m&nbsp;really proud&nbsp;of.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;It was a funny paper,&nbsp;though,&nbsp;wasn\u2019t it, because it straddles everything from user studies right the way through to a proof of most&nbsp;principal&nbsp;generalization, but we found it quite difficult to publish.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;We did. We shouldn\u2019t be bitter. Let\u2019s not be bitter, Simon.&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>[LAUGHTER]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;We\u2019re not bitter at all.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;We\u2019re not bitter.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:&nbsp;<\/strong>And&nbsp;in our field, there are kind of three major programming language conferences\u2014&nbsp;ICFP, POPL, and PLDI\u2014and on this one, we got the clean sweep. We were rejected by ICFP, rejected by POPL,&nbsp;<em>and<\/em>&nbsp;rejected by PLDI.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Bravo.&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>[LAUGHTER]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;That\u2019s right. It was a great paper. It is now published in the Journal of Functional Programming.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:&nbsp;<\/strong>We\u2019re laughing now.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong><strong>&nbsp;<\/strong>It\u2019s a funny story, but there\u2019s a thing behind it, which is that it is interdisciplinary work, right? It covers the stretch from programming languages&nbsp;theory&nbsp;through to HCI. That means it doesn\u2019t&nbsp;actually fit&nbsp;in any of the existing buckets very neatly. So, that can be a challenge when you\u2019re doing&nbsp;interdisciplinary research.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON<\/strong><strong>:<\/strong>&nbsp;I think it is a shame that these programming language conferences have a hard time, you know,&nbsp;accepting papers that use HCI methods.&nbsp;I\u2019d love it if we could start&nbsp;to&nbsp;have&nbsp;maybe&nbsp;tracks&nbsp;of HCI research at these conferences or co-locate with HCI conferences.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Yeah, but after all, what\u2019s the purpose of a programming language?&nbsp;<em>It is<\/em>&nbsp;the user interface for the computer, isn\u2019t it? The measure of whether a programming language is any good is whether it helps people get programs to work correctly, more quickly, so it\u2019s an HCI problem,&nbsp;programming.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;It totally is, and having worked a bit with HCI people, I\u2019m really&nbsp;humbled, actually. I mean, there are some great HCI folks in our team, and I learned such a lot from talking to them, and the thing is they really start from the users, talking to people, and asking them,&nbsp;\u201cWhat are the sort of problems you\u2019re having that might need some new features?\u201d&nbsp;And then they try and build features and then compare them.&nbsp;In the SDF paper that I\u2019ve just been talking about, we had two different ways of processing arrays, and then the study was&nbsp;actually a&nbsp;comparison of the two.&nbsp;And I\u2019ve really come to respect HCI people for the kind of methods they do to&nbsp;actually&nbsp;get&nbsp;rigorous results when working with people.&nbsp;I think it would be&nbsp;really great&nbsp;if we could have more co-location of HCI research conferences with more kind of technical PL conferences, because I think people would learn from each other in ways that just doesn\u2019t happen at the moment, when they\u2019re separated out at different conferences.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;So, Calc&nbsp;View is another example of such a project, isn\u2019t it, about sort of HCI aspects of programming? You want to just tell us a bit about Calc&nbsp;View? What is it?&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah. We&nbsp;actually published&nbsp;this at an HCI conference called VL\/HCC. So, it\u2019s just a radically different view of the grid.&nbsp;We have a spreadsheet grid, but then we have this second representation of it, a bit like a programming language. So, we use a textual notation, and we would literally have the text, like, maybe&nbsp;\u201cA1=2\u201d,&nbsp;then maybe&nbsp;\u201cA2=A1+1\u201d.&nbsp;And you&nbsp;would&nbsp;actually see&nbsp;formulas like that, equations like that,&nbsp;sort of visibly as if it was like a program.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;A textual program.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Like a textual program.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;You can&nbsp;edit&nbsp;with Emacs kind of thing.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Exactly. And so, the idea of Calc&nbsp;View is that you have the grid,&nbsp;and then to the right of it, you have this textual view, and certain things are&nbsp;really easy&nbsp;to do in the textual view. Like, for example, we have this nice notation for a formula that has been copied many times. You know, so if the same formula is copied from A1 to A100, you could literally say,&nbsp;\u201cA2:A100=A1+1\u201d.&nbsp;And so, that\u2019s a formula that would basically increment a counter.&nbsp;But&nbsp;textually, it\u2019s a very short program, a bit like a loop, and if you want to sort of change it, you can just change the one copy of that formula, and then every copy in the grid gets updated. And so, we did a user study, and again, we found that people were more effective at certain tasks using Calc&nbsp;View.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;And what I thought was interesting about it was that in programming, we\u2019re so used to the textual view of a program and we execute it later, and in Excel, we\u2019re so used to&nbsp;just seeing&nbsp;the data and having it execute sort of online all the time, you barely see the formulae, and with Calc&nbsp;View, we\u2019re trying to do it both at the same time and show you the data view continuously&nbsp;calc\u2019d&nbsp;on one side of the screen and the textual view on the other side of the screen, and you can edit in either and it affects the other. I thought that was&nbsp;a really interesting&nbsp;project.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;So, Simon, when you started working with the Excel team, it wasn\u2019t&nbsp;really clear, was it, what the formula language was? There wasn\u2019t really a sort of programming language style description of the Excel formulas.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Yeah, that\u2019s right. When I first started, I sort of looked at the formula language and thought,&nbsp;\u201cHey, how hard can this be? You know, it\u2019s just some formula. It\u2019s pretty simple syntax,\u201d&nbsp;and then the more I dug into what Excel actually does, it\u2019s really hard to find good descriptions about it. You\u2019ll find that, for example, references are first-class values. If you say&nbsp;\u201cF(A1:A3)\u201d,&nbsp;you\u2019re not just passing a little 3 vector of the values in A1 to A3;&nbsp;you\u2019re passing a&nbsp;<em>reference<\/em>&nbsp;to the&nbsp;<em>range<\/em>&nbsp;A1 to A3 as a first-class value&nbsp;to a function&nbsp;that can do sort of introspective things like, say, what is the row number of the last element. I didn\u2019t really realize that at all.&nbsp;So&nbsp;it turns out there are quite a lot of wrinkles like this. So, I started to write down a paper called&nbsp;\u201cThe Semantics of Excel.\u201d&nbsp;It was meant to be a document that says,&nbsp;\u201cNever mind the implementation. This is the semantics of a formula,\u201d&nbsp;and it was split into two in the end, the semantics of a single formula and then the semantics of&nbsp;calc, which is how is a whole spreadsheet full of formulas&nbsp;calc\u2019d&nbsp;and how do we make sure that it\u2019s always kept up to date and&nbsp;calc\u2019d&nbsp;in the right order with the right cells being made dirty?&nbsp;That led to all sorts of interesting conversations, sometimes with the Excel engineers who implemented&nbsp;particular features, because really, only they knew the honest-to-goodness ground truth about this. Again, it was an interesting dialogue between somebody who\u2019s coming at it completely from a semantics and programming languages point of view, and the engineers who\u2019d&nbsp;actually built&nbsp;it. But there was no independent standalone description in that kind of way. And that semantics then led to,&nbsp;\u201cOh, let\u2019s implement it. Let\u2019s implement a reference semantics for Excel,\u201d&nbsp;I think then as an F# program. So, that was meant to be, as it were, an Excel formula evaluator look-alike,&nbsp;a standalone,&nbsp;completely separate&nbsp;from the Excel code base,&nbsp;that implemented the reference model,&nbsp;because&nbsp;after all that, it could be a useful standalone entity. We thought that in an abstract way, and then you had the idea it wasn\u2019t just abstractly useful\u2014we might be able to concretely use it. That turned into&nbsp;Calc.ts, didn\u2019t it?&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yes. So, we did a hackathon about three years ago, mid-2017. So, we had this F# implementation that was&nbsp;actually getting&nbsp;pretty complete, at least the formula language. We didn\u2019t have all the worksheet functions, but we could evaluate a lot of formulas, and we were starting to wonder,&nbsp;\u201cMaybe we could take Excel formulas into other products.\u201d&nbsp;And so, Excel and&nbsp;Word&nbsp;have got an add-in model based on JavaScript. So, basically, we did this hackathon where we added Excel formulas to Word. It was&nbsp;sort of&nbsp;this grand hack. We had this code in F#, and we wanted to turn it into JavaScript, so we used\u2014oh, what was it called? Is it&nbsp;WebSharper? Yes,&nbsp;WebSharper&nbsp;is this sort of&nbsp;transpiler&nbsp;that compiles F# code to JavaScript. And so, we took this hunk of code and got it running in the hackathon, and then we made this silly video, and you were&nbsp;actually on&nbsp;the judging panel, but we won the&nbsp;MSR&nbsp;Cambridge&nbsp;hackathon&nbsp;that summer. It was great fun. Basically, we made a silly video that made people laugh, so that was great, but I mean, I think people liked the idea that we could take the formulas on a journey outside Excel, and then a really interesting thing happened, that this vice president of engineering, a gentleman called&nbsp;Ale\u0161&nbsp;Hole\u010dek, was visiting Cambridge, was looking at our work, and you know, he has a view about what\u2019s happening right across the different parts of Excel, and there is a web version of Excel, the web browser.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;This is Excel Online, right?&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Excel Online, yeah.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Every time you go to a&nbsp;web browser and wake up Excel, that\u2019s what you get.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah, and the way it&nbsp;works\u2014at least the way it&nbsp;worked until&nbsp;then\u2014was that it&nbsp;actually opened&nbsp;the Excel workbook in Azure inside a virtual machine, and then the client, the browser,&nbsp;was sort of like a dumb&nbsp;client, if you like;&nbsp;it didn\u2019t do any calculation there. It did the UI where you could enter formulas and you could enter data, but everything was calculated&nbsp;in Azure.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;If you entered a number, it would have to send a message back to a machine halfway across the planet to add 3 and 4 and send the result back to the web browser.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Exactly. So, it felt a bit sluggish.&nbsp;And there was a big push to make the product more usable, and so they decided they wanted to evaluate formulas in the browser. But&nbsp;unfortunately, the C++ code base that goes back to, I guess, 1985, when Excel started, it just wasn\u2019t practical to run that code base in the browser, and so&nbsp;Ale\u0161&nbsp;said to us,&nbsp;\u201cYou guys can help out. You guys need to go out to Israel because the Web Excel team is based in Herzliya in Israel, and they need a&nbsp;calc&nbsp;engine.\u201d&nbsp;So, we had this amazing work trip out to Israel,&nbsp;met the guys and started collaborating, and we realized&nbsp;pretty early&nbsp;on that we couldn\u2019t ship the F# code, so we did a&nbsp;big&nbsp;rewrite, and we rewrote it in TypeScript and then started a big push to write lots of the worksheet functions in TypeScript. And&nbsp;web&nbsp;Excel, because it\u2019s a website, could move&nbsp;really fast, so the amazing thing was that, like, six months after the hackathon, the initial version of&nbsp;Calc.ts&nbsp;went live,&nbsp;and it was starting to speed up people\u2019s calculations because it was doing the calculations in the browser.&nbsp;We had to do a bit of research to really make this viable, because not all the data in the cloud workbook ends up in the browser, so we had to have a&nbsp;calc&nbsp;engine that worked in the presence of partial data, and then we hit some precision problems because&nbsp;the original C++ code doesn\u2019t use IEEE arithmetic, believe it or not, because it was originally written\u2014&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES<\/strong>: Predated IEEE arithmetic. This is like code that has been around since forever.&nbsp;<strong>[LAUGHTER]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;That\u2019s right. But although it wasn\u2019t possible to do the following for&nbsp;all&nbsp;of&nbsp;the C++ code, we could use&nbsp;WebAssembly&nbsp;to take the core of the mathematics part of the&nbsp;calc engine that did 40-point arithmetic, and it was in C++, and we were able to compile that to JavaScript using&nbsp;WebAssembly&nbsp;or using that tool chain\u2014first to&nbsp;WebAssembly, then back to JavaScript. And so now, we\u2019ve got great&nbsp;performance numbers, so accuracy is, like, 99.95&nbsp;percent.&nbsp;We can evaluate more than 99&nbsp;percent&nbsp;of formulas, and&nbsp;a&nbsp;slightly goofy statistic is that there\u2019s so many customers of Excel for the web that&nbsp;Calc.ts&nbsp;saves them a total of seven years every single day&nbsp;because&nbsp;it speeds up the response you get without needing to wait for the server.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Also,&nbsp;it&nbsp;must be helping global warming, right? If you\u2019re&nbsp;gonna&nbsp;add 3 and 4, then that should be one machine instruction. If you\u2019ve&nbsp;gotta&nbsp;send it across the planet to a data center, that must be like bazillions of machine instructions, you know, that are just being wasted.&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>[MUSIC BREAK]<\/strong>&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:&nbsp;<\/strong>Going&nbsp;back to Excel as a product, that brings us sort of&nbsp;more or less up&nbsp;to date to the, um, the&nbsp;release of the LAMBDA function.&nbsp;Maybe you could say a little bit about what that is.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;I think we\u2019re all&nbsp;really super&nbsp;thrilled about having LAMBDAs in Excel. So, LAMBDA, it\u2019s the same LAMBDAs that you have in programming languages. It\u2019s an anonymous function.&nbsp;You can, say,&nbsp;double LAMBDA X\u2014right&nbsp;now, you\u2019d write in Excel as&nbsp;\u201cLAMBDA&nbsp;X, X+X\u201d&nbsp;and then that formula returns a function that if you give it an X, doubles it. Same as in any functional programming language. And this is going to be&nbsp;really great&nbsp;for users of Excel because it\u2019s going to make formulas more readable.&nbsp;Excel formulas are notorious because they tend to be really complicated if you\u2019re trying to do something clever, these notorious&nbsp;megaformulas&nbsp;that are really big things,&nbsp;that are&nbsp;really hard&nbsp;to read but are&nbsp;sort of&nbsp;squished down to a single cell, and LAMBDA will let you give names to parts of those, and we also have LET that&nbsp;let\u2019s&nbsp;you give names to subexpressions within a formula. So, overall, it\u2019s&nbsp;gonna&nbsp;make expressions much more readable&nbsp;and also&nbsp;much more reusable. You can define a computation once and then you can call it from many places, and just like in other programming languages, you get the great benefit that if there is something wrong with your initial definition or it was a preliminary definition for testing and you want to improve it and fix a bug, you just need to fix it in the one place and then that will propagate to all the uses.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;So, this is another funny thing, right? So, when Margaret and Alan and I were initially thinking about user-defined functions, we thought,&nbsp;\u201cWell, we have to define it as a worksheet.\u201d&nbsp;I don\u2019t think I ever dreamt that we could offer something as exotic as LAMBDA,&nbsp;which sounded very geeky and programming-language-y, and yet, it serves the same function. You can define named abstractions, but because it doesn\u2019t require much UI\u2014it\u2019s just another form of formula\u2014it\u2019s been much easier to introduce as the first form of the product. I still hope that we\u2019ll get sheet-defined functions, but it turns out that LAMBDA, and this really is the full-on lexically scoped LAMBDA\u2014you can define Church&nbsp;numerals, you can do the whole thing, you can define the Y combinator\u2014this is real LAMBDA\u2014it&nbsp;is now part of Excel, and that\u2019s amazing. That makes the language&nbsp;Turing&nbsp;complete. Because we can write the Y combinator,&nbsp;you could really write literally any program in Excel now.&nbsp;But that\u2019s a qualitative shift, right? So,&nbsp;certainly, it becomes computationally much more powerful, and because we can name these LAMBDAs, you can have functions that call functions that call functions.&nbsp;We can even write recursive functions. These named LAMBDAs can call each other.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yes. I mean, an example I love\u2014it\u2019s a very simple one, but&nbsp;just&nbsp;reversing a string. Believe it or not, although Excel has&nbsp;had&nbsp;lots of functions for processing strings, you can\u2019t write a function\u2014before LAMBDA,&nbsp;anyway\u2014within the formula language that can reverse a string. But with LAMBDAs, you can just write a nice, little recursive function that does that.&nbsp;There\u2019s real demand for a bit of functionality like that, but it wasn\u2019t possible to do within the formula language.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;And&nbsp;I love the way this is sort of empowering users, right?&nbsp;Now, users can write these new functions themselves and,&nbsp;in fact, can write functions specific to their&nbsp;particular domain&nbsp;or area of expertise or work group, and then call those functions just as easily as the built-in ones. I think that\u2019s a huge, huge change.&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;It\u2019s&nbsp;gonna&nbsp;be&nbsp;really interesting&nbsp;to see, you know,&nbsp;the kind of things people do and how we can share them. I\u2019m hoping there\u2019ll be a sort of open-source community&nbsp;build&nbsp;up&nbsp;around LAMBDAs.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Yeah, I really hope that there\u2019ll be a sort of third-party library and people who say,&nbsp;\u201cHere\u2019s my library for doing operations on&#8221;\u2014I don\u2019t know\u2014\u201ctensors\u201d&nbsp;or something. And we\u2019re&nbsp;gonna&nbsp;need to provide better support for libraries and richer data types, but I think that\u2019ll come. And then&nbsp;people can produce libraries themselves.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Yeah. It\u2019s&nbsp;really great. I mean,&nbsp;\u2019cause&nbsp;we\u2019ve been using LAMBDAs internally for a few years now, and it\u2019s just a different kind of experience. It\u2019s live programming. You\u2019ve got a&nbsp;grid,&nbsp;you\u2019ve got data just right in front of you. It\u2019s just&nbsp;really highly&nbsp;interpretive. You know, you can change one LAMBDA one place and instantly all the changes&nbsp;percolate.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;You\u2019re saying that even as a programming language person,&nbsp;doing functional programming in Excel with LAMBDAs and dynamic arrays feels quite different and more engaging than doing programming in, I don\u2019t know,&nbsp;Haskell or&nbsp;F#. Is that what you mean?&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:&nbsp;<\/strong>Yeah. Exactly.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;That&nbsp;actually leads&nbsp;me to just\u2014we\u2019re sort of,&nbsp;I guess,&nbsp;drawing towards the end. I was kind of reflecting on what have we learned from all this. One of the things that for me would be a very exciting outcome, if we can really make Excel into a language that doesn\u2019t have this sort of glass ceiling that prevents you getting beyond a certain point,&nbsp;that lets you write functions&nbsp;that can call functions that can call functions&nbsp;that can have data that has data&nbsp;that has data&nbsp;inside it, then we could maybe imagine introducing programming to children for the first time through the&nbsp;medium&nbsp;of Excel. So, the child\u2019s first programming language, maybe alongside something like Scratch, which is wildly popular, could be a spreadsheet, initially experienced in this very direct&nbsp;visceral way&nbsp;through just entering data and visualizing it with a chart, but then some simple data transformations using functions and then the ability to wrap those functions up into functions of your own. It would bring the pieces of my life together\u2014my functional programming life, my Microsoft life, and my education life. It could bring that all together into,&nbsp;\u201cWe teach children programming using Excel for the first time.\u201d&nbsp;&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:&nbsp;<\/strong>I think that\u2019s amazing. I think that\u2019s a fantastic, exciting vision, Simon. Those are things you\u2019ve worked on. I mean, it\u2019s&nbsp;really cool that those things are coming together, but I actually really believe that the spreadsheet environment&nbsp;<em>is<\/em>&nbsp;a great place to learn because it\u2019s so alive, you&nbsp;know?&nbsp;You make a little&nbsp;change&nbsp;and you can instantly see the results. It\u2019s a really sort of evocative way to program and to learn.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;Anything else that you feel you\u2019ve learned&nbsp;from&nbsp;this experience?&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;There\u2019s two things. I mean, I\u2019ve touched on HCI and programming languages, so I won\u2019t say so much about that except that I see a lot of excitement now in that interplay. There\u2019s a few, you know,&nbsp;professors\u2014I\u2019m thinking&nbsp;of&nbsp;people like&nbsp;Eleanor Glassman&nbsp;or&nbsp;Amy Coe\u2014professors who are on the intersection of programming languages in HCI and are doing&nbsp;really cool&nbsp;stuff, and there\u2019s summer schools on the topic. So, I think that\u2019s really growing as a research area, and I\u2019m really excited to be part of that direction.&nbsp;The other&nbsp;thing is, like I said near the start, that before getting into spreadsheets, I\u2019d spent quite a while looking at probabilistic programming, which is a way of expressing sort of Bayesian decision-making using code, and I sort of put that aside for a bit because I really, you know, I thought the way to achieve that, to sort of empower people to do Bayesian reasoning,&nbsp;to sort of use probabilistic reasoning to make good decisions,&nbsp;I think the way to get there is to get it into spreadsheets. And&nbsp;so,&nbsp;I think the kind of things that we\u2019ve&nbsp;been talking about are a step in that direction.&nbsp;When I was doing probabilistic programming, that was really aimed at quite sophisticated users, people who are like data scientists, but I think we are seeing in society generally a&nbsp;really big&nbsp;need for the general public to understand probabilities and to understand uncertainty.&nbsp;I\u2019m&nbsp;thinking about things like election forecast. I\u2019m&nbsp;thinking about the various uncertainties around COVID.&nbsp;I think to be sort of&nbsp;numerate&nbsp;about these uncertainties, well, you need to be&nbsp;numerate&nbsp;to deal with&nbsp;them&nbsp;properly,&nbsp;to make good decisions, and to be&nbsp;numerate, you need to take the uncertainty into account, which is usually using probabilities. So, I think&nbsp;that&nbsp;the kind of features that we\u2019re adding to Excel will make easier being able to visualize probabilities,&nbsp;to have charts that take probabilities into account,&nbsp;to sort of package up ways of visualizing probability distributions inside LAMBDAs so we can give textual descriptions of probabilities. People find, you know,&nbsp;a number like, you know, probability of 0.7 a little bit hard to understand;&nbsp;there\u2019s a big idea called natural frequencies, where you basically remove the points and you can say&nbsp;what 0.7 means, and studies have shown that people respond much better to those kinds&nbsp;of natural frequencies\u201470 out of 100\u2014than&nbsp;probabilities like&nbsp;0.7.&nbsp;So&nbsp;I think that&nbsp;the programming features we\u2019re&nbsp;adding to Excel will allow people to build libraries that make it easier to construct,&nbsp;to,&nbsp;say, represent distributions using natural&nbsp;frequencies,&nbsp;and eventually to make charts that represent probabilities nicely. And then a bit further out\u2014this is quite a long-term dream\u2014the idea of&nbsp;actually using&nbsp;Excel to build probabilistic models&nbsp;of situations will then be possible given the kind of work we\u2019re doing on adding LAMBDA to Excel. So, that\u2019s my long-term dream of where we\u2019re&nbsp;gonna&nbsp;go with LAMBDAs and these other features in Excel.&nbsp;<\/p>\n\n\n\n<p><strong>PEYTON JONES:<\/strong>&nbsp;More generally, I\u2019m quite excited about the possibilities of taking end-user programming seriously, by which I mean that because Excel starts from being an end-user tool\u2014that\u2019s what it&nbsp;has always been; by&nbsp;end-users, people&nbsp;who&nbsp;are trying to get some other job done\u2014that if we can bring that constituency of end users on a journey with us to empower them, to give them the tools to express more and do more, it would be quite exciting to see how far we can go. And I think&nbsp;that&nbsp;functional programming using Excel, there\u2019s a&nbsp;really high&nbsp;ceiling there that can go a long way. But I think just as we were talking about with Calc&nbsp;View and your sense that the programming using this just felt different, I think it may take us on a journey that isn\u2019t altogether where we expect.&nbsp;<strong>[MUSIC STARTS PLAYING UNDER DIALOGUE]<\/strong>&nbsp;We\u2019ve already seen that in the dynamics that we\u2019ve discussed between us and the product group, so I think the future\u2019s quite exciting there.&nbsp;<\/p>\n\n\n\n<p><strong>GORDON:<\/strong>&nbsp;Thanks, Simon. It\u2019s been great talking with you&nbsp;this afternoon. We\u2019re so excited about this work, and hope everyone listening is, too. Do check out the Microsoft Research website for more information. Thanks for joining us on the podcast.&nbsp;&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Today, people around the globe\u2014from teachers to small-business owners to finance executives\u2014use Microsoft Excel to make sense of the information that occupies their respective worlds, and whether they realize it or not, in doing so, they\u2019re taking on the role of programmer. In this episode, Senior Principal Research Manager Andy Gordon, who leads the Calc Intelligence team at Microsoft Research, and Senior Principal Researcher Simon Peyton Jones provide an inside account of the journey Excel has taken as a programming language, including the expansion of data types that has unlocked greater functionality and the release of the LAMBDA function, which makes the Excel formula language Turing-complete. They\u2019ll talk specifically about how research has influenced Excel and vice versa, programming as a human-computer interaction challenge, and a future in which Excel is the first language for budding programmers and a tool for incorporating probabilistic reasoning into our decision-making.<\/p>\n","protected":false},"author":39507,"featured_media":743689,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"https:\/\/player.blubrry.com\/id\/76856095\/","msr-podcast-episode":"120","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Andy Gordon","user_id":"30825"},{"type":"user_nicename","value":"Simon Peyton Jones","user_id":"33665"}],"msr_hide_image_in_river":0,"footnotes":""},"categories":[240054],"tags":[],"research-area":[13556,13554,13560],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243990],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-743248","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-msr-podcast","msr-research-area-artificial-intelligence","msr-research-area-human-computer-interaction","msr-research-area-programming-languages-software-engineering","msr-locale-en_us","msr-post-option-podcast-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"https:\/\/player.blubrry.com\/id\/76856095\/","podcast_episode":"120","msr_research_lab":[199561],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[511097],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-960x540.jpg\" class=\"img-object-cover\" alt=\"Advancing Excel as a programming language with Andy Gordon and Simon Peyton Jones on the Microsoft Research Podcast\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-960x540.jpg 960w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-300x169.jpg 300w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1024x577.jpg 1024w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-768x432.jpg 768w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1536x865.jpg 1536w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-2048x1153.jpg 2048w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-16x9.jpg 16w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1066x600.jpg 1066w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-655x368.jpg 655w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-343x193.jpg 343w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-640x360.jpg 640w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1280x720.jpg 1280w, https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-content\/uploads\/2021\/05\/1400x788_Simon_Andy__no_logos-1920x1080.jpg 1920w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"May 5, 2021","formattedExcerpt":"Today, people around the globe\u2014from teachers to small-business owners to finance executives\u2014use Microsoft Excel to make sense of the information that occupies their respective worlds, and whether they realize it or not, in doing so, they\u2019re taking on the role of programmer. In this episode,&hellip;","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/743248","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/39507"}],"replies":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/comments?post=743248"}],"version-history":[{"count":9,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/743248\/revisions"}],"predecessor-version":[{"id":761896,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/posts\/743248\/revisions\/761896"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/743689"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=743248"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/categories?post=743248"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/tags?post=743248"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=743248"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=743248"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=743248"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=743248"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=743248"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=743248"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=743248"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=743248"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}