Developing with React: GatsbyJS

Kyle Mathews, Creator of GatsbyJS • Wikia, Monday, Apr 18, 2016

React.js is surging in popularity as a modern build tool, and web developers are flocking to it to build sites and web apps.

Kyle Mathews is the creator of Gatsby, a React.js-based static site generator. He’s also the co-founder of RelateRocket.

Kyle presented at the most recent Static Web Tech meetup about building with React, the process of creating a static site builder, and why pairing the modern stack with git is increasingly the workflow of choice for frontend web devs.

Full Transcript is below

  • Hey everyone. Thanks for coming. I think this is, what, our sixth or seventh installment of the Static Web Tech meetup. And sixth, I’ve been told from over there.

I want to just start off today by saying a huge thank you to Wikia, who’s been sponsoring the chow and the drinks and keeping the beer flowing and providing accommodations, ever since I think, we only managed to do one in Netlify. In our HQ by our own Biilmann who started off talking more about the category of static tech in general and then we moved over here. And, very, very happy to stay. And now we’ve got a poster printed down in the lobby, not sure if you saw it, so now it’s officially, we are here to stay. Otherwise that poster was for nothing.

So most of these talks so far have been about static site generators including the one today, which is very exciting because it’s in React, sort of the framework of choice right now. Gatsby is new, but becoming popular very fast. And we have the creator, Kyle, to come in and talk about it which is great.

But yeah, wanted to talk a little bit about the category. So, these tools are very much in the center of this new way of static sites. But they’re also just a part of what we’re seeing, they’re part of a new category that’s been emerging over the last few years, but now is really getting legs to walk on. And we’re gonna do a Meetup on it really soon and really what it does is modern build tools, frameworks in a very git-centric workflow, takes advantage of HTML5 and modern browsers, to be able to create sites that can live and breath on a CDN as static sites and then communicate to the API via the browser. So anything that needs real time updating, or anything that will be traditionally referred to as dynamic, goes that way. And that of course leads to sites that are 10 times faster and have a time to first byte that’s almost immediate and way, way safer, because you’re doing away with this cumbersome legacy back end that you guys all know from WordPress and all the other stuff, right?

That’s also of course why we’re all here today. It’s really exciting to work with these tools but things have really opened up and we also think that there is some, there is a need to be able to talk about this category better. It’s really a new stack and at Netlify, where we do a platform that accommodates modern static site and then does everything around it, everything from continuous deployment, instant cache invalidation and CDN hosting, pre-rendering, all those things, it’s sort of a Heroku for this new kind of stack.

We often start off talking to clients where they spend 10, 15 minutes explaining a work flow that is around these tools that you guys are using, right? That is around React and modern built tools and git and all these things and we thought that we really needed some nomenclature. So some of the people that we’ve been talking to around this, both clients and advisors and industry leaders and the guys behind GitHub and Heroku and RackSpace and so on, and we were told how could we sort of present some terminology around this and we came up with JAMstack, which is JavaScript, APIs and Markup.

Our own Matt Biilman just presented that on the Smashing Magazine conference that was just in San Francisco; did any of you guys make it there? No? That was a great conference, by the way. Always is. And we’ll be publishing more about it in Smashing Magazine soon. He’ll be hosting a Next Speak here and talking about this stack and in generally will be widening it up so not just talk about static site generators but talk about the whole work flow, everything from the hosting side of CDNs to modern frameworks, what can you do with these modern JavaScripts to all these things that today you can actually do client side, everything from comment fields to web chats, to all kinds of dynamic plugins, right? And so we’re going to try and widen it out, so of course modern static site generators are very much at the heart of it, but static is sort of a bad name for it really, right? Because it’s very much not static.

At Netlify, more than 97% of the sites we host are not static at all. They’re actually connecting to all kinds of APIs. They’re eCommerce solutions, all kinds of stuff that’s super dynamic, right? But they can live on a CDN, these sites, which is why we call it static and so, yeah, it’s a great new world. We’d love to hear your comments about it afterwards if anyone wants to stick around to talk about this and now getting back to what you’re all here for today, which is Gatsby. And we’re really excited. We’ve been working with Kyle and, what’s that?

  • [Voiceover] React.

  • Yeah, it’s Gatsby. Done with React, right. Yeah. Static site generator Gatsby. And yeah, so, without further ado.

  • Thank you for the introduction And it’s nice to be here today and talk to you all about Gatsby. So first, we’ll do a little bit of audience questions just so I kind of get to know who my audience is tonight. So who first has heard of Gatsby before coming here tonight? Okay, decent percentage.

  • [Voiceover] We knew that you were coming, so–

  • Well, yeah, yeah, yeah, but I mean maybe before the meeting was announced, we’ll go with that.

  • [Voiceover] You were the draw.

  • What?

  • [Voiceover] You were the draw.

  • Thank you. Not too much praise; it’s embarrassing. Who’s used React, for realsies? Not just played around with it? Okay, about a third of you, okay, cool. So yeah, so, Gatsby, so, a bit about myself. Co-Founder of a start up, it’s cool. You can check us out if you want.

So a little bit about my history in building websites, kind of give a little backstory of Gatsby. So I started building websites back ten years ago, I guess, 2006 or so. And using Drupal, and actually got into Drupal for a while, went to some DrupalCons and spoke there and so forth and it was cool. I like Drupal a lot.

Then around 2011 I decided I really wanted to get into kind of JavaScript web apps because that was, that still is kind of the future. It’s that’s where most products are going. It’s just richer and richer experiences on the web. The web is a brilliant application platform and JavaScript’s the way to do it, and so in 2011, BackBone was kind of the thing to do that and so I got into BackBone and I joined a startup here in SF called Pantheon and I worked there for several years and built out their dashboard using BackBone and it was cool. It was real time single page app stuff. Fun times, learned a lot. And so I was mostly just doing JavaScript web apps around that time but then around 2014 or so, I quit my job at Pantheon and I was just exploring a ton of stuff and I decided to rebuild my website. And static sites were getting really, really popular around then, so I was like, “I’m going to do that.”

There’s the old adage that, you know, you don’t reinvent the wheel unless you want to understand how the wheel works, and so I decided to build my own little static generator which I thankfully never published or put anywhere because it’s really crappy. It was this little Gulp-driven thing and it’s actually pretty straightforward to put together and I was like, “Oh, this is cool,” I have a little template thing and I have my markdown files and I wrote it out and it spits out a bunch of HTML files “This is cool.” And super easy to maintain, super easy to, you know, use. I was like, “I really like this.”

I also started getting into React.JS and WebPack and ReactRouter and hot reloading around the same time and absolutely loved it, because it solved pretty much all of the big pain points I had with BackBone. So quickly became a huge fan of React and its kind of surrounding ecosystem. Yeah, and so also in 2014-ish or so, I started working on what became the startup I’m working on now and so at some point we needed to build a website and I was like, “Okay, I need a website. I don’t want to do it in Drupal anymore and I like static sites but the one I built was crappy, but I also want to use React because I fell in love with React and it’s just a really brilliant ecosystem and it was just super fast and super powerful to do all sorts of complicated stuff,” so I was like, “How do I marry this React.JS ecosystem I was developing that I had fallen in love with, with the world of static sites, where you just generate a bunch of stuff and you push it out somewhere and it just kind of does its thing, and you really don’t have to think about it any more?”

And so I was like, “with React, you can statically generate HTML.” What if I could somehow marry all these tools together in some sort of package that would make it really easy to build websites?” And so I kind of obsessed with this idea for a while and eventually in 2015, I decided “Okay this just has to happen.” And so I took a week off from startup work and I built version 0.1 of Gatsby and shockingly, it worked.

I had a little NPM installable thing and you could start a thing, and you had a little develop command and, you know, everything just kind of worked together. Yeah, and so I actually even kind of realized my goal even better than I hoped it was, Gatsby is essentially, it takes markdown or other static kind of data sources, and it turns it into a single page app and it also marries it with a pretty awesome kind of developer experience with HotReloading.

So on the left is my editor and then, you know, you save that, and then it reloads automatically. You can add some more content and also see how the title changed as well? So it makes for a really nice kind of developing, editing experience because as you’re modifying your theme, as you’re modifying content, you can see the changes live right in your browser, which just really, really speeds up experimenting with something and seeing if it works or not.

And so bringing it to today, we’re up to almost to 2600 stars in GitHub, so if three of you want to star it real quick then it can get us over the edge. But even more important, I’m really proud, there’s 33 contributors, so definitely developing, what do they call it? A bus factor for Gatsby. So not just me doing stuff anymore, which is nice because I don’t really have time to do a lot of it anyways, so, that’s great.

Yeah, so eight months old but quite a few sites have already been built with Gatsby. This was my blog. The very, very first Gatsby site. This is my startup and then another startup is using Gatsby, somebody’s blog, this is an open source tool, et cetera, okay.

So now I want to get into more of the meat of the presentation and I want to make a case for why Gatsby is the best choice, 2016, for building websites. Mostly static websites, but you know, if you’re creative and clever, maybe you could do all your websites. So first how do we choose tools? I assume everyone here is a developer or close to it so kind adapt that using tools and thinking about tools and then choosing tools.

There’s basically two strategies for picking something. You know, given the constraint that meets your requirements, you can either, A, choose the most popular tool in a category, which has a ton of advantages because probably all of the kinks have been worked out, you know, there’s tons of answers in StackOverflow, for whatever you need to do, there’s lots of blog post tutorials, et cetera, et cetera, and so it’s really nice to have just that kind of popularity to kind of surround you with its warm, comforting embrace. And so that’s what most people do.

There’s also kind of a secondary strategy, which appeals to kind of more hipster developers who A, want to do something a little off the beaten path and also it’s kind of a nice recruiting strategy–to pick a non-mainstream language because you pick the right one, it can have a really smart kind of crowd, you can easier to hire from because you’re the only company that’s using that kind of niche technology.

So that’s the kind of reason one I think why Gatsby’s a really excellent choice for adopting in 2016 is that kind of all much of the momentum in kind of the front end world is moving towards tools that Gatsby natively supports, you know, stuff like Postcss and Webpack and kind of component-driven web development, React.js and NPM, et cetera.

So if we’re looking at static site generators in a very broad sense and kind of comparing them to Gatsby, these are different pieces that make up a static site generator, you know. You have, almost every one supports Markdown out of the box and so everyone does pretty much the same thing. And so you have Markdown, and turns it down to HTML. There’s nothing special there. Same with HTML templates. At the end of the day, they all generate HTML. It’s kind of the same. CSS, most static site generators have pretty good support for CSS, so nothing special about Gatsby there.

JavaScript, though, is where Gatsby starts to get interesting.

Many static site generators, they’re kind of like this black box, you throw in Markdown and out comes HTML at the other end, but they don’t really have a good pipeline for kind of doing more sophisticated, interesting JavaScript things, so it’s kind of this awkward, either you have a side process which is handling your JavaScript, whatever, and so, for Gatsby on the other hand, at the end of the day it’s all JavaScript and so doing more sophisticated JavaScript stuff with React is a very natural thing to do with Gatsby. You can actually have React components which are like actual pages and so you can have pretty sophisticated, complex, single page app type stuff that’s happening inside your Gatsby site.

And the last piece is a component-driven website development. This is a huge trend. There’s a ton of advantages to it. I’m not going to go into a lot here, but yeah, Gatsby again has first class support for that using, again, through React.js. And the really nice thing about this, I think, is that one thing about component-driven development is that it allows you in an organization to share common components from website to website or even within the same website, you have your different pieces of the website, you can turn those into components and re-use them across the website and it adds a lot of consistency and speeds up your development a lot by having your own little tool chest of components.

Yeah, so a bit about the goals of Gatsby developer experience. A, batteries included. B … So there’s this … Yeah, so it kind of Gatbsy has this concept of boilerplates or starters that you can install, well, right now they’re called starters, we’re probably going to call boilerplate soon, but anyways, you kind of have partially built sites that you can then start from and then they’re very hackable, very easy to modify to kind of meet your needs and Gatsby has three official starters or boilerplates. A kind of a kitchen sink - I demonstrated a bunch of stuff, and then a blog and then a documentation site. And something that’s coming soon is a little more sophisticated version of that tentatively called Gatsby apps, which would be, say there was open source documentation site, which is a really common thing, you have some sort of open source tool and you have a little home page, splash page type thing, and then you have a little docs section and so forth, well Gatsby, you could abstract a lot of that functionality into an NPM package that has a template, has a theme, and then if you’re building a new site that’s a documentation site, you just NPM install that and then all you have to do is add some Markdown files for your documentation and then a home page and then everything would just be kind of otherwise be taken care of for you and it could be a community maintained thing and so forth, so there’s a lot of interesting possibilities there.

Because I think its tools that win are ones that allow a community to collaborate around common stuff. That was one of my big problems with BackBone is that it was very hard to share stuff, like components, BackBone components that you built with other people, but where React is getting a lot of its popularity from is actually very easy to share React.js components, push them to NPM and other people can use them without much modification and that makes React really, really powerful because now maybe 25, 30, 40 percent of your site can be those community-maintained components that percentage is just rising. And I think with Gatsby too, as it gets adoption, there can be more and more of these boilerplates and apps, or other pieces of sites that are really common from site to site to site that can be community maintained and that you can just get out of the box.

And then the hot reloading piece, again, is a huge boon to development. I’ve been using it for a year and a half now or so and I can’t really imagine not having that available anymore. Another nice thing about Gatsby is, if you’ve done any React development, it should feel very, very similar and the last is yeah, just that, Gatsby by itself is not that special. It’s basically a, it’s a glue project that glues things together. So most of its power is the fact that it makes easily available, kind of like I was talking about earlier, all of the little ecosystem pieces that kind of go into it. So kind of the goals for the Gatsby user experience is that it’s just very fast. Pages should, all static sites, it should have just the minimum amount of stuff that needs to load to get to the screen very quickly but more than that, Gatsby sites load all the content needed for subsequent pages so once you load the site, you might click from page to page without any page reloads. I can actually demo that real quick.

So if you go to my startup’s website. So you click on About, there’s no page transition. It’s almost instantaneous. And that’s because when the site loads, it loads a JavaScript bundle which has the code necessary to go from page to page plus the content, and so it’s a really good user experience and also, this isn’t built in yet, but service workers are super exciting. They offer really sophisticated kind of client caching plus they also make offline experiences actually really easily, which, if any of you ever tried to do that before, research that, it’s been pretty much impossible on the web, but service work actually makes it pretty straightforward and so I want to build in to Gatsby a way that’s a, you know one line configuration basically to enable service workers and offline support. So, okay, now’s a good time for questions if anyone has any quick questions and then we’re going to demo what it’s like building with Gatsby. Yeah?

  • [Voiceover] So how does it work with search engines?

  • Just fine. It’s a static site so every page is, so I mean when I was showing you, so I was showing you this there’s no transition but each of these pages also, so it has a single page app version but also each individual page is a separate HTML file so if I reload this, it reloads the About HTML file from the back end and then loads the same single page app in, so it works just fine for search engines. Yeah, in the back?

  • [Voiceover] Did you bake in Back and History support as well?

  • Yeah, so the question is, is there back button support? And yeah, it’s using, so under the hood it uses React router, which is a really popular React router, as the name suggests, but anyways, yeah, it has really, really good support for back button stuff. So yeah if you go back and forth, no problem. Other questions? Yeah.

  • [Voiceover] So can you go beyond markdown and describe something more complex in the markup? Do you have a way of handling that?

  • Yeah, I’m going to show a bit in the demo, kind of the kitchen sink which will show all the kind of out of the box supported file types. But it’s very possible to add more support and Gatsby’s soon going to move to a plugin architecture which will mean that, Gatsby core will have a few plugins for common file types supported, but you can create a plugin for ASCI doc, or something, or any other format that you can think of, that there’s an NPM module to support, but anyways, which is pretty much anything, but anyways, yeah, and you’ll be able to install that and have very easy support for other types of markdown-type languages. You had a question here? Or not. Any other questions? Yeah, in the back.

  • [Voiceover] Just a little unsure where the React part comes in, is that at build time or something? Or the React part in the Gatsby app. I know it uses it to build the actual end result, so is it just at that level, where it’s combining the Markdown and the data and spitting out HTML?

  • Yeah that’s a great question. So the basic architecture, so from a data flow perspective, what Gatsby’s doing is, its first step is to collect data and so that could be from a Markdown file, it could be from a JSON file, it could be from a web service, it could be anywhere, and it collects all that and then it basically has a system of matching different page data blobs to then React components, so Markdown files all get sent through the same React.js markdown wrapper. And so everything in Gatsby eventually gets run through a React component. Either it starts as a React component and it’s a direct conversion, or it starts as some sort of data source and then gets run through a React.js wrapper component.

So if we go to Gatsby as I was mentioning earlier, there’s several starters that you can use to start with and so I installed one of those and now I just run Gatsby develop and that sets up the development server with hot reloading and so forth. And so now we can just go to localhost:8000. Okay, so this is kind of the kitchen sink demo that shows all the kind of available options for file types and so, yeah, if we go here, kind of see, you know, bunch of markdown stuff is supported, code highlighting, anyways, kind of normal markdown stuff and then as I was mentioning earlier, it has native support for JavaScript components. You just drop JavaScript components inside your pages directory and it’ll get turned into a file and so this is generated static, I mean there’s a static representation, a static HTML that’s generated from this so if I turn off JavaScript, this’ll load like this but because you included support for the JavaScript bundle loading, now it has a fully working you know, React components, that gets loaded automatically. Same thing if CoffeeScript’s your flavor.

You can also just have raw HTML, and that’s supported. Then kind of fun, too, it has full support for JSON, YAML, and TOML. So this is what the raw JSON file looks like and it gets turned into this page. Same thing with YAML and TOML and it also has PostCSS and SASS and LESS support so … so forth. So okay, so some basic structure. So Gatsby sites, they have kind of a root directory and it has your, it has an HTML HTML page, which is, kind of wraps every page, and it’s just a JavaScript, it’s just a React.js component. And then there’s, just your package.json and a few other stuff that’s optional, oh and has a config tunnel, et cetera. But yeah, all the pages go in the pages directory so if we go there, here we can see, yeah here we can see the different pages that are converted into these different ones here. So we have this which turns into the Markdown file. You know we have the Coffee React and the React.js and then the TOML.TOML, YAML.YAML, et cetera, et cetera and all these are, either they’re raw data files…if we look at the JSON one for example, it’s just a JSON file and if we look at the YAML, it’s just a YAML file, and if we look at say the CoffeeScript file, you know, it’s just a normal coffee script React.js component. Yeah, and so to get at your question earlier about do you have raw power? Yeah, it’s just all React code so you can do whatever you could in a normal React component.

And also, Gatsby has a decent API now for doing other, kind of extending, other extension points, so if you want to, after the site is built, if you want to generate an RSS feed, for example, there’s an API hook for that and I’m going to be revamping the API soon, which I’ll be talking about in a bit and which will add a lot more extension points for doing custom modifications. Yeah and then the HTML page is just HTML and so forth. So I think it’d be interesting, well first, let me show the hot reloading. So this is the index.js and this is a React.js component, which generates this page here, so we have up here it says “Hi people”, so if we say “Hi static web meetup peeps” and then we save that, it’ll just hot reload and you say, “Welcome to your new” try it again, “awesome Gatsby site.” So that’s cool. Anyways, so that’s pretty straightforward and this works for CSS too, but also if we go to the markdown file, you know we go over here and then say “what do you think?” You know, same thing even though we’re editing markdown, same hot reloading. Cool. So, yes?

  • [Voiceover] Can I question what’s happening if you screw up?

  • Scroll up?

  • [Voiceover] Screw up, error.

  • Oh, oh, oh, yes. That’s actually a great question. It’s almost like you’re plants or something. So what if we have a syntax error in our JavaScript? See will this trigger it? Oh, there we go. Yeah, so it doesn’t happen for all errors, but for a lot of errors, there’s this really cool kind of error overlay that shows up while you’re developing your JavaScript code and it’ll have the whole error stack up there. Is that what you’re asking by the way?

  • [Voiceover] Yeah.

  • Okay, there you go. Anyways, and then over here “Oh darn, “we mistyped the import,” so we correct that and then back to normal. So yeah, and this is, none of this is what I’ve done this is really complicated stuff that really smart people spent a ton of time figuring out, and Gatsby, because it’s just being built on top of the React, Webpack, ecosystems, it just gets to take all these really, really nice developer tools and bring them in to this experience.

So it might be helpful to go through how Gatsby takes a raw Markdown file or raw any kind of file, really, and runs it up into you know, this final output. So if we go to this Markdown, so this is a very normal Markdown file with front matter and so forth and so, if any of you have built static sites, you’ve seen this a thousand times. So Gatsby uses Webpack loaders to convert Markdown into something a React.js component can handle. So a loader is basically a conversion script, it takes any sort of raw input and loads it into a JSON file so it turns something into JavaScript objects so if we look at the, see, certificate, yeah, so if we look here, the Markdown loader for the site, if we kind of skip down here, so we have this module.exports, this is kind of the meat of it, so this module.exports, it gets the content, every markdown file is passed into here and then it first extracts the front matter up here, and then it uses a tool called MarkdownIt to render the markdown stream and then all that’s kind of combined together, the front matter and now the HTML that’s been rendered from the markdown into this result and then it’s returned so it’s really, really simple, once you’re kind of used to this pattern, and it’s really powerful because with Webpack loaders you can take any file and basically, that’s almost anything, you can take an image, well, basically anything, and analyze it and turn it into data and then make it accessible and turn it into pages.

So yeah, so that’s the first step, take the markdown file and turn it into, and parse it, and turn it into a JavaScript object and then Gatsby internally, it associates every file type with a wrapper, which is a React.js component and they just go in this wrappers directory and … yeah, and it’s just a one to one association so if we open up the markdown wrapper you can see that this is a React.js component and in here, in the render function, so if you’re familiar with React, all components, when data is passed into it, it’s passing this props object and so Gatsby passes the data for each page on this and so we’re kind of taking that out onto this post object and then we just return this very simple structure where we have div classname is markdown and then we set the title from the front matter and then we just set the rendered post body here in this div and out it comes. So, really simple pattern, but it’s really powerful and kind of easy to tweak and modify to kind of fit whatever need.

For example, you know, if you had a much, say you had, you know, you had a whole bunch of different markdown files, you know, a lot of static site generators have a theme, a key that you put in your front matter, so in this, you know, you could just have different, you could switch over the different themes or something or render out different React.js components, depending on what type of markdown file was or something. So if we look at the JSON wrapper, it’s extremely similar. It’s also taking the data off of the and this it’s also setting the title and then it’s just stringifying the JSON so if we go out here, we kind of looked at this, where it takes the title off the JSON object that was on the file and then just has the raw view of the JSON, it just stringifies it out, so as you can see, it’s exactly the same as what was in the file.

Yeah, and so this is interesting because JSON support is interesting because there’s a lot of third party systems where you could just do a bunch of API calls and pull a bunch of information and dump it in, as pages in a Gatsby site and then automatically generate a site from there, which is kind of an interesting possibility. Yeah, cool. So questions on all this stuff? Yes, Matt?

  • [Voiceover] Yeah, so, one thing I’ve been wondering with the sort of generating it all as something that also turns into a single page is how well it scales to really large sites.

  • Yeah.

  • [Voiceover] 2000 pages is something that, how will we handle that?

  • Yeah, so … that’s a, it has a mixed performance right now. So there was somebody who, there is one community member I think has a 10,000 page Gatsby site and so the only way to do that right now is basically turn off the Gatsby, turn off generating the JavaScript bundle that has all your pages in it because I tested with like 5,000 pages and it generated a 2 megabyte gzipped JavaScript bundle, which is a bit much for most sites.

But there is a way, though, to make it quite scalable. Webpack supports something called codesplitting which is basically you can have yeah, basically you can split your JavaScript bundle into as many segments as you want and so Gatsby would be able, this isn’t built yet, but Gatsby would be able to say, only create JavaScript bundles that are maximum of 500 kilobytes, for example. And so when you land on a page, say you go to the home page, it would just load the JavaScript necessary for that page or maybe a few pages around it that’s necessary for that, but then you navigate to another part of the site, and React and React Router is built so that you can asynchronously load additional bundles, JavaScript bundles in the background and so we would be able to say “Okay you’re now going to the docs section of the site, which JavaScript we haven’t loaded yet” and then behind the scenes, you know, just put out a request for that JavaScript bundle and then it would pull everything in without, again, any page transitions, and it could be, it should be quite quick on most connections. Say there’s another 200 kilobytes that should only take, less than a second, so it should still be really, really quick, even loading stuff like that in the background.

Yeah, anyways, and with that technique Gatsby should easily scale to very, very large sites. It might make sense to even build in more…Oh also, one other thing that could be interesting too is that Webpack has support for parallel building, and so each of those chunks could be built in parallel to some extent, and then that would speed the build of very large sites. Yeah?

  • [Voiceover] Did you implement generation from a database?

  • Question was does Gatsby support generation of a database. Not natively. The plugin architecture that’s coming, you would be able to have a plugin that hooks directly into your database and loads stuff, that gets generated as a file, but right now, what you’d need to do is basically you’d have a script which would say, “Connect to the database, pull out all the information you need, and write out JSON files into your Gatsby pages directory.” And then it would work, but yeah, it’s definitely doable and it will be better supported soon. Okay, so Gatsby was, very much still kind of is, an experimental thing, I just did it because I was like, “Oh maybe this is “possible” and it turned out, it is possible, but whenever you’re, as a lot of you know, it’s like when you’re prototyping something, it’s oftentimes, almost all the time, not ideal in a lot of ways, and you learn a lot by prototyping something, building something out that then forms, if you have enough time and money, and don’t have other pressing things, to actually build it right the second time and so, so yeah, so Gatsby has some weaknesses.

It’s very much, it’s kind of monolithic right now, it’s a little bit hard to modify. Too much stuff is in the core, it’s a little too opinionated, which has its plus sides because out of the box, if you’re working within its constraints, it works, but it’s a little too opinionated. It pulls too much in and yeah, it doesn’t have as many extension points as I would like. So kind of been thinking about the last several months and with conversations with other people and seeing feature requests and so forth, how I would like Gatsby to go next is first, kind of revamp the data transformation pipeline. So, I have been describing it in this talk as you collect data, either from files or APIs and whatever else, and then pass it through this processing step where eventually it all gets turned into React.js components and then into HTML and a JavaScript bundle, and that’s the idealized version of things. It doesn’t actually, it’s not nearly that clean right now and so I want to rewrite it so it’s actually in code it’s that clean as well and then at each step in that process, there would be before and after hooks that you could slip in and do something, like “at this point in the data transformation step, I’m going to have this little function which goes through and does this modification or something.” This would be great both because it’d make it easier to extend your own site but also common tweaks could be turned into NPM modules, which you could install.

So say you wanted to have some sort of special, you know, URL structure for your files, for example, you want to do whatever but you wanted the end URLs to be cleaner. You could have a plugin which then you install which transforms the URLs for each page into what you want it to be or you know, RSS feed generation. Right now you kind of have to hand write that and it’s kind of a pain, but it’s a pretty common thing so anybody doing a Gatsby blog, there should be a RSS plugin which you install and it generates an RSS feed whenever you build the site.

So yeah, so that data transportation, the plugin API, have really clean support for pulling these plugins from NPM and installing them, and then also, I was talking about earlier the idea of having Gatsby apps, which is, you can have a blog that all you do, that the only thing inside your…the only source you’re controlling is the Markdown files, which I think would be ideal for a lot of sites that you can just install it, maybe have a few configuration options which you tweak, and then you just start adding content. I think that’s the ideal for a lot of things.

This is kind of how I’m thinking the plugin app API, whatever structure would work where you would have these apps which are basically bundling together plugins and templates and React.js components and wrappers and so forth and you could add those and then you could have plugins, you could have the Gatsby markdown plugin and the PostCSS plugin. And then these plugins then could take configuration, so for PostCSS, what if you wanted CSS modules? Well then you just turn that on. And then the last thing, postBuild, this is an example of how you’d extend your Gatsby site through the built-in API, so you’d have a postBuild hook, so that once everything was finished building, you’d be passed an array of pages, the data for all the pages, and then you’d be able to do whatever you want with that.

So that’s the presentation. So. Cool, so comments, questions, feedback?

So the question was, do I see Gatsby being used for very large content sites, and yeah, I, right now you’d probably top out at around 500 to 1000 pages is where Gatsby could easily do, but the pathway for making it work on sites of any number of pages is very straightforward. It’s just, somebody has to do it.

  • [Voiceover] What is that pathway?

  • Yeah, it’s basically set up code splitting. So the main bottleneck is Gatsby turns all your content, combines all your content into the JavaScript bundle so that you have kind of that single page app experience once it’s loaded, all the content’s there in your browser and you just click around and it’s all there. But at 500,000 pages, it’s starting, that bundle, that JavaScript bundle is starting to get pretty big. More than you’d normally want to load into your site and so but there’s a process where, with Webpack where you can split up the bundle and asynchronously load different bundles as you navigate around the site. You’d only need to load one of those bundles when you first open the site and then if you navigate to another section of the site it would then load in the background the JavaScript you need for that section of the site. So it’s very doable, it’s just this is a side project for me and I don’t need it so I’m waiting for that brave community member to step up to the plate and make it happen. Yeah?

  • [Voiceover] So what kind of person do you see being the user of Gatsby in the future? Kind of, is it converging towards non-programmers or developers? It can’t really satisfy everybody, right?

  • Yeah, right now it’s a little bit rough. And yeah, it takes decent amount of development skill to do something interesting. But I don’t see any reason why with the Gatsby app concept that I was talking about, that even people who know very little about programming should be able to build sites with Gatsby where you could almost have a WordPress type install experience, you know? Where you would install a blog or a little small business Gatsby app and then you would just go in there and edit the content, because you’d kind of have example stuff “Okay, here’s your home page” and “Here’s a splash image of whatever” and you just go in modify what’s already there and so even relatively unsophisticated people technically should be able to build stuff with Gatsby down the road. Just like with the WordPress ecosystem, has gazillions of people building websites with WordPress who aren’t developers by any stretch of the imagination I don’t see why not with, I don’t see why that can’t happen with Gatsby.

  • [Voiceover] Can you only build React sites with it or can you use it to build an Angular site or something else? I don’t know how you’d do that, but just curious.

  • Yeah, I don’t know how you’d do that. Yeah.

  • [Voiceover] So it’s mainly for React?

  • Yeah. I mean, you could probably, I mean, I don’t know if Angular has, so to do it with Angular, I don’t know if Angular has the concept of rendering to HTML, static HTML, outside of the browser environment. So I don’t know if you could do this with Angular. I know Ember has that capability now with their fast boot stuff, so you could build the Ember equivalent of Gatsby if you wanted to. But yeah, Gatsby’s for React, though.

  • [Voiceover]

  • I’m sorry, what?

  • [Voiceover] Not the actual tool, but the site you’re building, the actual end result site? You could have it be whatever, right?

  • Uh, no. It depends entirely on React. I think you can embed Angular inside React components, so if you really wanted to have Angular for some reason, you could make it work, but yeah it’s not supported.

  • [Voiceover] Thank you.

  • That was awesome. A static site that’s a single page app. All right, thank you so much to Kyle for coming and sharing his thoughts and ideas and work. His really, really great work. Yeah, as we mentioned, we run static site,, which is the most popular open source directory of static site generators and together with Hugo, which is a static site generator made in Go, Gatsby is definitely just flying up there. It’s really becoming very popular very fast. So yeah, thanks again for spending your evening with us. Thank you everyone for coming and hopefully we’ll see you next time.