Add new post about studio-frontend
This commit is contained in:
parent
802d162335
commit
9bd9499793
367
_posts/2018-04-26-studio-frontend.md
Normal file
367
_posts/2018-04-26-studio-frontend.md
Normal file
@ -0,0 +1,367 @@
|
|||||||
|
---
|
||||||
|
title: "Studio-Frontend: Developing Frontend Separate from edX Platform"
|
||||||
|
layout: post
|
||||||
|
---
|
||||||
|
|
||||||
|
*This is a blog post that I originally wrote for the [edX engineering
|
||||||
|
blog](https://engineering.edx.org/).*
|
||||||
|
|
||||||
|
At the core of edX is the [edx-platform](https://github.com/edx/edx-platform), a
|
||||||
|
monolithic Django code-base 2.7 times the size of Django itself.
|
||||||
|
<!--excerpt-->
|
||||||
|
|
||||||
|
```
|
||||||
|
edx-platform (master)> tokei
|
||||||
|
-------------------------------------------------------------------------------
|
||||||
|
Language Files Lines Code Comments Blanks
|
||||||
|
-------------------------------------------------------------------------------
|
||||||
|
ActionScript 1 118 74 23 21
|
||||||
|
CoffeeScript 25 2296 1492 545 259
|
||||||
|
CSS 55 17106 14636 1104 1366
|
||||||
|
HTML 667 72608 36892 30307 5409
|
||||||
|
JavaScript 1471 458791 349670 54805 54316
|
||||||
|
JSON 91 14421 14421 0 0
|
||||||
|
JSX 28 2104 1792 45 267
|
||||||
|
LESS 1 949 606 232 111
|
||||||
|
Makefile 1 34 21 8 5
|
||||||
|
Markdown 24 331 331 0 0
|
||||||
|
Mustache 1 1 1 0 0
|
||||||
|
Python 3246 553766 438165 29089 86512
|
||||||
|
ReStructuredText 48 4258 4258 0 0
|
||||||
|
Sass 423 75509 55536 4548 15425
|
||||||
|
Shell 13 830 453 251 126
|
||||||
|
SQL 4 6158 4971 1171 16
|
||||||
|
Plain Text 151 2982 2982 0 0
|
||||||
|
TypeScript 20 88506 76800 11381 325
|
||||||
|
XML 364 5283 4757 231 295
|
||||||
|
YAML 36 1643 1370 122 151
|
||||||
|
-------------------------------------------------------------------------------
|
||||||
|
Total 6670 1307694 1009228 133862 164604
|
||||||
|
-------------------------------------------------------------------------------
|
||||||
|
```
|
||||||
|
|
||||||
|
35% of the edx-platform is JavaScript. While it has served edX well since its
|
||||||
|
inception in 2012, reaching over 11 million learners in thousands of courses on
|
||||||
|
[edX.org](https://www.edx.org/) and many more millions on all of the [Open edX
|
||||||
|
instances across the
|
||||||
|
world](https://openedx.atlassian.net/wiki/spaces/COMM/pages/162245773/Sites+powered+by+Open+edX),
|
||||||
|
it is starting to show its age. Most of it comes in the form of [Backbone.js
|
||||||
|
apps](http://backbonejs.org/) loaded by [RequireJS](http://requirejs.org/) in
|
||||||
|
Django [Mako templates](http://www.makotemplates.org/), with
|
||||||
|
[jQuery](https://jquery.com/) peppered throughout (and — yes — some 2296 lines
|
||||||
|
of [CoffeeScript](http://coffeescript.org/)).
|
||||||
|
|
||||||
|
Many valiant efforts are underway to modernize the frontend of edx-platform
|
||||||
|
including replacing RequireJS with Webpack, Backbone.js with
|
||||||
|
[React](https://reactjs.org/), and ES5 JavaScript and CoffeeScript with ES6
|
||||||
|
JavaScript. Many of these efforts [were covered in detail at the last Open edX
|
||||||
|
conference](https://www.youtube.com/watch?v=xicBnbDX4AY) and in [Open edX
|
||||||
|
Proposal 11: Front End Technology
|
||||||
|
Standards](https://open-edx-proposals.readthedocs.io/en/latest/oep-0011-bp-FED-technology.html).
|
||||||
|
However, the size and complexity of the edx-platform means that these kind of
|
||||||
|
efforts are hard to prioritize, and, in the meantime, frontend developers are
|
||||||
|
forced to [wait over 10
|
||||||
|
minutes](https://openedx.atlassian.net/wiki/spaces/FEDX/pages/264700138/Asset+Compilation+Audit+2017-11-01)
|
||||||
|
for our home-grown asset pipeline to build before they can view changes.
|
||||||
|
|
||||||
|
There have also been efforts to incrementally modularize and extract parts of
|
||||||
|
the edx-platform into separate python packages that could be installed as
|
||||||
|
[Django apps](https://docs.djangoproject.com/en/2.0/ref/applications/), or even
|
||||||
|
as separately deployed
|
||||||
|
[microservices](https://en.wikipedia.org/wiki/Microservices). This allows
|
||||||
|
developers to work independently from the rest of the organization inside of a
|
||||||
|
repository that they own, manage, and is small enough that they could feasibly
|
||||||
|
understand it entirely.
|
||||||
|
|
||||||
|
When my team was tasked with improving the user experience of pages in
|
||||||
|
[Studio](https://studio.edx.org/), the tool that course authors use to create
|
||||||
|
course content, we opted to take a similar architectural approach with the
|
||||||
|
frontend and create a new repository where we could develop new pages in
|
||||||
|
isolation and then integrate them back into the edx-platform as a plugin. We
|
||||||
|
named this new independent repository
|
||||||
|
[studio-frontend](https://github.com/edx/studio-frontend). With this approach,
|
||||||
|
our team owns the entire studio-frontend code-base and can make the best
|
||||||
|
architectural changes required for its features without having to consult with
|
||||||
|
and contend with all of the other teams at edX that contribute to the
|
||||||
|
edx-platform. Developers of studio-frontend can also avoid the platform’s slow
|
||||||
|
asset pipeline by doing all development within the studio-frontend repository
|
||||||
|
and then later integrating the changes into platform.
|
||||||
|
|
||||||
|
## React and Paragon
|
||||||
|
|
||||||
|
When edX recently started to conform our platform to the [Web Content
|
||||||
|
Accessibility Guidelines 2.0 AA (WCAG 2.0
|
||||||
|
AA)](https://www.w3.org/WAI/intro/wcag), we faced many challenges in
|
||||||
|
retrofitting our existing frontend code to be accessible. Rebuilding Studio
|
||||||
|
pages from scratch in studio-frontend allows us to not only follow the latest
|
||||||
|
industry standards for building robust and performant frontend applications, but
|
||||||
|
to also build with accessibility in mind from the beginning.
|
||||||
|
|
||||||
|
The Javascript community has made great strides recently to [address
|
||||||
|
accessibility issues in modern web
|
||||||
|
apps](https://reactjs.org/docs/accessibility.html). However, we had trouble
|
||||||
|
finding an open-source React component library that fully conformed to WCAG 2.0
|
||||||
|
AA and met all of edX’s needs, so we decided to build our own:
|
||||||
|
[Paragon](https://github.com/edx/paragon).
|
||||||
|
|
||||||
|
Paragon is a library of building-block components like buttons, inputs, icons,
|
||||||
|
and tables which were built from scratch in React to be accessible. The
|
||||||
|
components are styled using the [Open edX theme of Bootstrap
|
||||||
|
v4](https://github.com/edx/edx-bootstrap) (edX’s decision to adopt Bootstrap is
|
||||||
|
covered in
|
||||||
|
[OEP-16](https://open-edx-proposals.readthedocs.io/en/latest/oep-0016-bp-adopt-bootstrap.html)).
|
||||||
|
Users of Paragon may also choose to use the
|
||||||
|
[themeable](https://github.com/edx/paragon#export-targets) unstyled target and
|
||||||
|
provide their own Bootstrap theme.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Studio-frontend composes together Paragon components into higher-level
|
||||||
|
components like [an accessibility
|
||||||
|
form](https://github.com/edx/studio-frontend/blob/master/src/accessibilityIndex.jsx)
|
||||||
|
or [a table for course assets with searching, filtering, sorting, pagination,
|
||||||
|
and upload](https://github.com/edx/studio-frontend/blob/master/src/index.jsx).
|
||||||
|
While we developed these components in studio-frontend, we were able to improve
|
||||||
|
the base Paragon components. Other teams at edX using the same components were
|
||||||
|
able to receive the same improvements with a single package update.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Integration with Studio
|
||||||
|
|
||||||
|
We were able to follow the typical best practices for developing a React/Redux
|
||||||
|
application inside studio-frontend, but at the end of the day, we still had to
|
||||||
|
somehow get our components inside of existing Studio pages and this is where
|
||||||
|
most of the challenges arose.
|
||||||
|
|
||||||
|
## Webpack
|
||||||
|
|
||||||
|
The aforementioned move from RequireJS to Webpack in the edx-platform made it
|
||||||
|
possible for us to build our studio-frontend components from source with Webpack
|
||||||
|
within edx-platform. However, this approach tied us to the edx-platform’s slow
|
||||||
|
asset pipeline. If we wanted rapid development, we had to duplicate the
|
||||||
|
necessary Webpack config between both studio-frontend and edx-platform.
|
||||||
|
|
||||||
|
Instead, studio-frontend handles building the development and production Webpack
|
||||||
|
builds itself. In development mode, the incremental rebuild that happens
|
||||||
|
automatically when a file is changed takes under a second. The production
|
||||||
|
JavaScript and CSS bundles, which take about 25 seconds to build, are published
|
||||||
|
with every new release to
|
||||||
|
[NPM](https://www.npmjs.com/package/@edx%2Fstudio-frontend). The edx-platform
|
||||||
|
`npm install`s studio-frontend and then copies the built production files from
|
||||||
|
`node_modules` into its Django static files directory where the rest of the
|
||||||
|
asset pipeline will pick it up.
|
||||||
|
|
||||||
|
To actually use the built JavaScript and CSS, edx-platform still needs to
|
||||||
|
include it in its Mako templates. We made a [Mako template
|
||||||
|
tag](https://github.com/edx/edx-platform/blob/master/common/djangoapps/pipeline_mako/templates/static_content.html#L93-L122)
|
||||||
|
that takes a Webpack entry point name in studio-frontend and generates script
|
||||||
|
tags that include the necessary files from the studio-frontend package. It also
|
||||||
|
dumps all of the initial context that studio-frontend needs from the
|
||||||
|
edx-platform Django app into [a JSON
|
||||||
|
object](https://github.com/edx/edx-platform/blob/master/cms/templates/asset_index.html#L36-L56)
|
||||||
|
in a script tag on the page that studio-frontend components can access via a
|
||||||
|
shared id. This is how studio-frontend components get initial data from Studio,
|
||||||
|
like which course it’s embedded in.
|
||||||
|
|
||||||
|
For performance, modules that are shared across all studio-frontend components
|
||||||
|
are extracted into `common.min.js` and `common.min.css` files that are included
|
||||||
|
on every Studio template that has a studio-frontend component. User's browsers
|
||||||
|
should cache these files so that they do not have to re-download libraries like
|
||||||
|
React and Redux every time they visit a new page that contains a studio-frontend
|
||||||
|
component.
|
||||||
|
|
||||||
|
## CSS Isolation
|
||||||
|
|
||||||
|
Since the move to Bootstrap had not yet reached the Studio part of the
|
||||||
|
edx-platform, most of the styling clashed with the Bootstrap CSS that
|
||||||
|
studio-frontend components introduced. And, the Bootstrap styles were also
|
||||||
|
leaking outside of the studio-frontend embedded component `div` and affecting
|
||||||
|
the rest of the Studio page around it.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
We were able to prevent styles leaking outside of the studio-frontend component
|
||||||
|
by scoping all CSS to only the `div` that wraps the component. Thanks to the
|
||||||
|
Webpack [postcss-loader](https://github.com/postcss/postcss-loader) and the
|
||||||
|
[postcss-prepend-selector](https://github.com/ledniy/postcss-prepend-selector)
|
||||||
|
we were able to automatically scope all of our CSS selectors to that `div` in
|
||||||
|
our build process.
|
||||||
|
|
||||||
|
Preventing the Studio styles from affecting our studio-frontend component was a
|
||||||
|
much harder problem because it means avoiding the inherently cascading nature of
|
||||||
|
CSS. A common solution to this issue is to place the 3rd-party component inside
|
||||||
|
of an
|
||||||
|
[`iframe`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe)
|
||||||
|
element, which essentially creates a completely separate sub-page where both CSS
|
||||||
|
and JavaScript are isolated from the containing page. Because `iframe`s
|
||||||
|
introduce many other performance and styling issues, we wanted to find a
|
||||||
|
different solution to isolating CSS.
|
||||||
|
|
||||||
|
The CSS style [`all:
|
||||||
|
initial`](https://developer.mozilla.org/en-US/docs/Web/CSS/all) allows
|
||||||
|
resetting all properties on an element to their initial values as defined in the
|
||||||
|
CSS spec. Placing this style under a wildcard selector in studio-frontend
|
||||||
|
allowed us to reset all inherited props from the legacy Studio styles without
|
||||||
|
having to enumerate them all by hand.
|
||||||
|
|
||||||
|
```css
|
||||||
|
* {
|
||||||
|
all: initial;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
While this CSS property doesn’t have broad browser support yet, we were able to
|
||||||
|
polyfill it thanks to postcss with the
|
||||||
|
[postcss-initial](https://github.com/maximkoretskiy/postcss-initial) plugin.
|
||||||
|
|
||||||
|
However, this resets the styles to *nothing*. For example, all `div`s are
|
||||||
|
displayed inline. To return the styles back to to some sane browser default we
|
||||||
|
had to re-apply a browser default stylesheet. You can read more about this
|
||||||
|
technique at
|
||||||
|
[default-stylesheet](https://github.com/thallada/default-stylesheet).
|
||||||
|
|
||||||
|
From there, Bootstrap’s
|
||||||
|
[reboot](https://getbootstrap.com/docs/4.0/content/reboot/) normalizes the
|
||||||
|
browser-specific styling to a common baseline and then applies the Bootstrap
|
||||||
|
styles conflict-free from the surrounding CSS cascade.
|
||||||
|
|
||||||
|
There's a candidate recommendation in CSS for a [`contains`
|
||||||
|
property](https://www.w3.org/TR/css-contain-1/), which will "allow strong,
|
||||||
|
predictable isolation of a subtree from the rest of the page". I hope that it
|
||||||
|
will provide a much more elegant solution to this problem once browsers support
|
||||||
|
it.
|
||||||
|
|
||||||
|
## Internationalization
|
||||||
|
|
||||||
|
Another major challenge with separating out the frontend from edx-platform was
|
||||||
|
that most of our internationalization tooling was instrumented inside the
|
||||||
|
edx-platform. So, in order to display text in studio-frontend components in the
|
||||||
|
correct language we either had to pass already-translated strings from the
|
||||||
|
edx-platform into studio-frontend, or set-up translations inside
|
||||||
|
studio-frontend.
|
||||||
|
|
||||||
|
We opted for the latter because it kept the content close to the code that used
|
||||||
|
it. Every display string in a component is stored in a
|
||||||
|
[displayMessages.jsx](https://github.com/edx/studio-frontend/blob/master/src/components/AssetsTable/displayMessages.jsx)
|
||||||
|
file and then imported and referenced by an id within the component. A periodic
|
||||||
|
job extracts these strings from the project, pushes them up to our translations
|
||||||
|
service [Transifex](https://www.transifex.com/), and pulls any new translations
|
||||||
|
to store them in our NPM package.
|
||||||
|
|
||||||
|
Because Transifex’s `KEYVALUEJSON` file format does not allow for including
|
||||||
|
comments in the strings for translation, [Eric](https://github.com/efischer19)
|
||||||
|
created a library called [reactifex](https://github.com/efischer19/reactifex)
|
||||||
|
that will send the comments in separate API calls.
|
||||||
|
|
||||||
|
Studio includes the user’s language in the context that it sends a
|
||||||
|
studio-frontend component for initialization. Using this, the component can
|
||||||
|
display the message for that language if it exists. If it does not, then it will
|
||||||
|
display the original message in English and [wrap it in a `span` with `lang="en"`
|
||||||
|
as an
|
||||||
|
attribute](https://github.com/edx/studio-frontend/blob/master/src/utils/i18n/formattedMessageWrapper.jsx)
|
||||||
|
so that screen-readers know to read it in English even if their default is some
|
||||||
|
other language.
|
||||||
|
|
||||||
|
Read more about studio-frontend’s internationalization process in [the
|
||||||
|
documentation that Eric
|
||||||
|
wrote](https://github.com/edx/studio-frontend/blob/master/src/data/i18n/README.md).
|
||||||
|
|
||||||
|
## Developing with Docker
|
||||||
|
|
||||||
|
To normalize the development environment across the whole studio-frontend team,
|
||||||
|
development is done in a Docker container. This is a minimal Ubuntu 16.04
|
||||||
|
container with specific version of Node 8 installed and its only purpose is to
|
||||||
|
run Webpack. This follows the pattern established in [OEP-5: Pre-built
|
||||||
|
Development
|
||||||
|
Environments](https://open-edx-proposals.readthedocs.io/en/latest/oep-0005-arch-containerize-devstack.html)
|
||||||
|
for running a single Docker container per process that developers can easily
|
||||||
|
start without installing dependencies.
|
||||||
|
|
||||||
|
Similar to edX’s [devstack](https://github.com/edx/devstack) there is a Makefile
|
||||||
|
with commands to start and stop the docker container. The docker container then
|
||||||
|
immediately runs [`npm run
|
||||||
|
start`](https://github.com/edx/studio-frontend/blob/master/package.json#L12),
|
||||||
|
which runs Webpack with the
|
||||||
|
[webpack-dev-server](https://github.com/webpack/webpack-dev-server). The
|
||||||
|
webpack-dev-server is a node server that serves assets built by Webpack.
|
||||||
|
[Studio-frontend's Webpack
|
||||||
|
config](https://github.com/edx/studio-frontend/blob/master/config/webpack.dev.config.js#L94)
|
||||||
|
makes this server available to the developer's host machine
|
||||||
|
at `http://localhost:18011`.
|
||||||
|
|
||||||
|
With [hot-reload](https://webpack.js.org/concepts/hot-module-replacement/)
|
||||||
|
enabled, developers can now visit that URL in their browser, edit source files
|
||||||
|
in studio-frontend, and then see changes reflected instantly in their browser
|
||||||
|
once Webpack finishes its incremental rebuild.
|
||||||
|
|
||||||
|
However, many studio-frontend components need to be able to talk to the
|
||||||
|
edx-platform Studio backend Django server. Using [docker’s network connect
|
||||||
|
feature](https://docs.docker.com/compose/networking/#use-a-pre-existing-network)
|
||||||
|
the studio-frontend container can join the developer’s existing docker devstack
|
||||||
|
network so that the studio-frontend container can make requests to the docker
|
||||||
|
devstack Studio container at `http://edx.devstack.studio:18010/` and Studio can
|
||||||
|
access studio-frontend at `http://dahlia.studio-fronend:18011/`.
|
||||||
|
|
||||||
|
The webpack-dev-server can now [proxy all
|
||||||
|
requests](https://github.com/edx/studio-frontend/blob/master/config/webpack.dev.config.js#L101)
|
||||||
|
to Studio API endpoints (like `http://localhost:18011/assets`)
|
||||||
|
to `http://edx.devstack.studio:18010/`.
|
||||||
|
|
||||||
|
## Developing within Docker Devstack Studio
|
||||||
|
|
||||||
|
Since studio-frontend components will be embedded inside of an existing Studio
|
||||||
|
page shell, it’s often useful to develop on studio-frontend containers inside of
|
||||||
|
this set-up. [This can be
|
||||||
|
done](https://github.com/edx/studio-frontend#development-inside-devstack-studio)
|
||||||
|
by setting a variable in the devstack's `cms/envs/private.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
STUDIO_FRONTEND_CONTAINER_URL = 'http://localhost:18011'
|
||||||
|
```
|
||||||
|
|
||||||
|
This setting is checked in the Studio Mako templates wherever studio-frontend
|
||||||
|
components are embedded. If it is set to a value other than `None`, then the
|
||||||
|
templates will request assets from that URL instead of the Studio's own static
|
||||||
|
assets directory. When a developer loads a Studio page with an embedded
|
||||||
|
studio-frontend component, their studio-frontend webpack-dev-server will be
|
||||||
|
requested at that URL. Similarly to developing on studio-frontend in isolation,
|
||||||
|
edits to source files will trigger a Webpack compilation and the Studio page
|
||||||
|
will be hot-reloaded or reloaded to reflect the changes automatically.
|
||||||
|
|
||||||
|
Since the studio-frontend JS loaded on `localhost:18010` is now requesting the
|
||||||
|
webpack-dev-server on `localhost:18011`,
|
||||||
|
an [`Access-Control-Allow-Origin` header](https://github.com/edx/studio-frontend/blob/master/config/webpack.dev.config.js#L98)
|
||||||
|
has to be configured on the webpack-dev-server to get around CORS violations.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Deploying to Production
|
||||||
|
|
||||||
|
[Each release of
|
||||||
|
studio-frontend](https://github.com/edx/studio-frontend#releases) will upload
|
||||||
|
the `/dist` files built by Webpack in production mode to
|
||||||
|
[NPM](https://www.npmjs.com/package/@edx/studio-frontend). edx-platform
|
||||||
|
requires a particular version of studio-frontend in its
|
||||||
|
[`package.json`](https://github.com/edx/edx-platform/blob/master/package.json#L7).
|
||||||
|
When a new release of edx-platform is made, `paver update_assets` will run
|
||||||
|
which will copy all of the files in the
|
||||||
|
`node_modules/@edx/studio-frontend/dist/` to the Studio static folder.
|
||||||
|
Because `STUDIO_FRONTEND_CONTAINER_URL` will be `None` in production, it will be
|
||||||
|
ignored, and Studio pages will request studio-frontend assets from that static
|
||||||
|
folder.
|
||||||
|
|
||||||
|
## Future
|
||||||
|
|
||||||
|
Instead of “bringing the new into the old”, we’d eventually like to move to a
|
||||||
|
model where we “work in the new and bring in the old if necessary”. We could
|
||||||
|
host studio-frontend statically on a completely separate server which talks to
|
||||||
|
Studio via a REST (or [GraphQL](https://graphql.org/)) API. This approach would
|
||||||
|
eliminate the complexity around CSS isolation and bring big performance wins for
|
||||||
|
our users, but it would require us to rewrite more of Studio.
|
||||||
BIN
img/blog/paragon-modal-storybook.png
Normal file
BIN
img/blog/paragon-modal-storybook.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 82 KiB |
BIN
img/blog/studio-frontend-assets-table.png
Normal file
BIN
img/blog/studio-frontend-assets-table.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 200 KiB |
BIN
img/blog/studio-frontend-docker-devstack.png
Normal file
BIN
img/blog/studio-frontend-docker-devstack.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
BIN
img/blog/studio-frontend-style-isolation.png
Normal file
BIN
img/blog/studio-frontend-style-isolation.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Loading…
Reference in New Issue
Block a user