[For Developers] Connecting Front-end Developers and Designers with the Token System

March 15

# Tech

# Front-End

# Token System

Intro

What is the Token System discusses what is the token system, includes the definition of tokens, and how to organize tokens to meet our customization needs.

A token system is an approach to abstracting, structuring, and organizing visual styles using tokens. The token system has three layers: reference tokens, system tokens, and component tokens, which are organized hierarchically to allow for customization at different levels. This system can be leveraged to safely and quickly modify any layers to meet design needs without any side effects.

The previous article introduced how designers can use tokens in their workflow. This article will focus on the developer's perspective, and explain how to integrate tokens into the development process.

Code Structure

First, let's take a look at the code structure. Github

1├── packages
2│ ├── react // a component lib demo that uses tokens
3│ │ ├── ...
4│ ├── token // token files and compilation logic
5│ │ ├── ...
6├── pnpm-lock.yaml
7├── pnpm-workspace.yaml
8├── package.json
9└── ...

This is a monorepo repository, and pnpm is used to manage dependencies.

There are two packages:

I named this demo using Ezreal because he is my favorite character in League of Legends. 😀

• token(@ezreal-ui/token): This package stores the token files and compilation logic.
• react(@ezreal-ui/react): A demo component library package used to showcase the use of tokens.

I will introduce the purpose of these two packages in the following sections.

Token Compilation

First, let's focus on the @ezreal-ui/token package (referred to as the style package below), which stores token files and compilation logic.

Let's review the workflow of the development phase:

DEV Stage CI&CD Workflow

We have two things to do:

• Build and publish the style package using CI/CD after UED pushes the changes to the token files to the remote.
• Update the style package after running git pull in the local repository when token files change.

This article will focus on token compilation, namely how to convert tokens into style files that can be used by components. The next article will discuss how to automate these steps and link them together through CI/CD.

Prerequisites

Style Dictionary

Before diving into the token compilation, let's take a look at a tool library for token compilation, Style Dictionary.

Style Dictionary is a build system that allows you to define styles once, in a way for any platform or language to consume. A single place to create and edit your styles, and a single command exports these rules to all the places you need them - iOS, Android, CSS, JS, HTML, sketch files, style documentation, or anything you can think of.

Using Style Dictionary, we can transform your design system's style and color definitions into code for multiple platforms and languages, speeding up development and maintaining consistency in the design system.

Style Dictionary provides many useful APIs to enhance the ability to manipulate tokens, like:

• extend(dictionary, extension): This method is used to extend a Style Dictionary object with additional properties.
• registerTransform(transform): This method is used to register a new transform function with Style Dictionary.
• registerFormat(format): This method is used to register a new format function with Style Dictionary.
• compileAllPlatforms(): This method is used to compile all platforms and formats registered with Style Dictionary.
• ...

Here's a simple example that shows how to use Style Dictionary to compile tokens into CSS.

Assuming we have a token file like this:

1const designTokens = {
2  color: {
3    primary: {
4      value: '#FF0000',
5    },
6    secondary: {
7      value: '#00FF00',
8    },
9  },
10  size: {
11    small: {
12      value: '12px',
13    },
14    medium: {
15      value: '16px',
16    },
17    large: {
18      value: '20px',
19    },
20  },
21};
22

We can extend it with API StyleDictionary.extend:

1// Extend design tokens with additional properties 
2const extendedDesignTokens = StyleDictionary.extend(designTokens, {
3  color: {
4    tertiary: { 
5      value: "#0000FF" 
6    } 
7  },
8});

We can transform the value of some tokens with API StyleDictionary.registerTransform:

1// Register a transform that converts color values to RGB
2function rgbTransform(prop) {
3  if (prop.attributes.category === "color") {
4    const hex = prop.original.value;
5    const r = parseInt(hex.substr(1, 2), 16);
6    const g = parseInt(hex.substr(3, 2), 16);
7    const b = parseInt(hex.substr(5, 2), 16);
8    return `rgb(${r}, ${g}, ${b})`;
9  }
10  return prop.original.value;
11}
12
13StyleDictionary.registerTransform({
14  name: "rgb",
15  type: "value",
16  transformer: rgbTransform,
17});

We can compile tokens to CSS with API StyleDictionary.registerFormat and StyleDictionary.compileAllPlatforms:

1// Register a format that outputs CSS
2function cssFormat(dictionary) {
3  return Object.keys(dictionary.properties)
4    .map((propertyName) => {
5      const prop = dictionary.properties[propertyName];
6      return `.${prop.name} { ${prop.attributes.cssProperty}: ${prop.value}; }`;
7    })
8    .join("\n");
9}
10
11StyleDictionary.registerFormat({
12  name: "css",
13  formatter: cssFormat,
14});
15
16// Compile all platforms and formats
17const compiled = StyleDictionary.compileAllPlatforms();
18
19// Output the compiled CSS 
20console.log(compiled.css);
21/**
22.primary { color: rgb(255, 0, 0); }
23.secondary { color: rgb(0, 255, 0); }
24.tertiary { color: rgb(0, 0, 255); }
25.small { font-size: 12px; }
26.medium { font-size: 16px; }
27.large { font-size: 20px; }
28*/

The API of Style Dictionary is mighty, and I won't list them one by one, you can refer to this document for more information. In general, we can use Style Dictionary to modify, format, and compile tokens as we want.

Build Style Files

In the previous article, the section For UED explains how tokens work in UEDs' workflow. and the section Sync Tokens mentions that after UED completes the design, the tokens will be pushed to the remote repository, like GitHub.

Setting of the Figma Plugin

There is a field called Branch, which refers to the concept of Git branches. The diagram uses a branch feat/alpha-publish. UED and FE should use the same branch for design and development to ensure that changes made by both parties are synchronized.

There is another field called file path, which specifies the directory where the files should be stored when using the plugin to push token files to Github.

Thus, when UED uses the plugin configured as above to push token files, the files will be pushed to the directory packages/token/src/tokens/cache/cache.json under the feat/alpha-publish branch.

The file path we entered is packages/token/src/tokens/cache/cache.json, so the plugin will store and push all tokens in a single JSON file named cache.json. Although the plugin supports exporting to multiple files, this feature is only available in the pro version.

Now that we have the token files, let's take a look at how to compile them with Style Dictionary when UED push changes.

Here is what we need to do, including:

1. Splitting files
2. Transforming values
3. Handling details
4. Generating style files

1. Splitting Files

Since we have exported all tokens to a single file named cache.json, it would be better to split them into a clear structure for easier token handling.

The token files will be split into two categories: component token files and core token files. The component token folder contains all component token files, while the core token folder contains all reference and system token files:

1token(@ezreal-ui/token)
2├── src
3│   └── style  // compilation result
4│       ├── components
5│       │   └── ...
6│       ├── core
7│       │   └── ...
8│       └── index.less
9│   └── tokens
10│       ├── cache  // from Figma
11│       │   └── cache.json
12│       ├── components  // auto generated
13│       │   └── button.json
14│       └── core  // auto generated
15│           ├── ref.json
16│           └── sys.json

Simply put, the splitting logic is as follows:

1. Read cache.json
2. Clear previous splitting results
3. Generate the core directory and generate the sys.json and ref.json token files based on the content read from cache.json
4. Generate the component directory and generate token files for each component based on the content read from cache.json.

The completed splitting logic can be viewed in the code.

2. Transforming Values

Although we have got and split the token files, some of the values in these files cannot be used directly. We need to use the StyleDictionary.registerTransform API we mentioned earlier to do some transforms, mainly including:

1. Adding units, such as px, to some numeric values
2. Standardizing color models, using RGB uniformly
3. Handling colors for dark and light modes.
4. ...

1// ...
2
3/**
4 * Helper: Transforms dimensions to px
5 */
6StyleDictionary.registerTransform({
7  type: "value",
8  transitive: true,
9  name: "transformDimension",
10  matcher: (token) => {
11    const { type, value } = token;
12    return dimensionTokenArr.includes(type) && numberRegex.test(value);
13  },
14  transformer: (token) => {
15    const value = token.original.value;
16    return value + SIZE_UNIT;
17  },
18});
19
20// ...

You can find the code here. The code uses many StyleDictionary.registerTransform API calls to perform various transform operations.

3. Handling Details

After transforming values, some details need to be handled, including:

1. Add headers to each automatically generated style file to indicate that these files should not be manually modified.
2. Split the final generated reference and system style files by token type, such as color and spacing.
3. Add prefixes to all the style variables generated based on tokens.

1// ...
2
3/**
4 * Helper: Append custom header
5 */
6StyleDictionary.registerFileHeader({
7  name: "appendCustomHeader",
8  fileHeader: () => {
9    return [WARNING_FILE_HEADER];
10  },
11});
12
13// ...

You can find the code here.

4. Generating Style Files

After completing all the preliminary work mentioned above, we can start generating the style files. We chose Less as the format for our final exported style files because Less provides powerful capabilities, including variable definitions and function calculations.

Additionally, the layering of the style files follows the token hierarchy structure we discussed earlier, namely reference, system, and component.

1// generate core files, including reference and system style variables
2coreFile.map(function (theme) {
3  const SD = StyleDictionary.extend(getThemeStyleDictionaryConfig(theme));
4  SD.buildAllPlatforms();
5});
6
7// generate component style variables
8componentTokenFiles.map(function (file) {
9  const SD = StyleDictionary.extend(getCompStyleDictionaryConfig(file));
10  SD.buildAllPlatforms();
11});

You can find the code here. packages/token/src/scripts/style-dictionary.ts is the entry point of our program.

At this point, we have completed the conversion from tokens to style files. Additionally, to facilitate the import of styles by users, I have added an index.less file and copy all these files to the dist directory. For more details, please refer to the code.

1//auto-generate index.less
2@import "./components/index.less";
3@import "./theme/index.less";
4
5@prefix: ezreal;
6@ezreal-vars-prefix: ~"--@{prefix}";

You can find all scripts in the package.json file.

1// ...
2"scripts": {
3    "build": "pnpm build:token",
4    "prebuild": "rm -rf ./dist ./src/style/*",
5    "postbuild": "mkdir dist && cp -R ./src/style/* ./dist",
6    "build:token": "ts-node --esm ./src/scripts/style-dictionary.ts",
7    "prebuild:figma": "ts-node --esm ./src/scripts/prebuild-for-figma.ts"
8  },
9//...

Token Usage

Now that we have obtained the style files converted from tokens, let's take a look at how to use them.

Let's check out the package.json files of these two packages.

1// @ezreal-ui/token
2"exports": {
3    ".": {
4      "import": "./dist/index.less"
5    }
6  },
7
8// @ezreal-ui/react
9"dependencies": {
10    "@ezreal-ui/token": "workspace:*",
11    //...
12  },
13

@ezreal-ui/token specifies the exported content through exports. Because we are using a monorepo, @ezreal-ui/react can reference another package, @ezreal-ui/token, in the same repository by using workspace:*.

Let's take a look at the structure of @ezreal-ui/react:

1.
2├── components
3│   ├── Button
4│   │   ├── index.less
5│   │   ├── index.tsx
6│   │   └── ...
7│   ├── ...
8│   └── index.ts
9└── package.json

Since we are mainly focused on styles, let's take a look at the contents of index.less under Button directory.

1@import "@ezreal-ui/token";
2
3@btn-cls-prefix: ~"@{prefix}-button";
4@btn-vars-prefix: ~"@{ezreal-vars-prefix}-button";
5@types: filled, outlined;
6@status: default, disabled;
7
8.@{btn-cls-prefix} {
9  cursor: pointer;
10  font-size: 16px;
11}
12
13.@{btn-cls-prefix}.@{btn-cls-prefix}-disabled {
14  cursor: not-allowed;
15}
16
17each(@types, .(@t) {
18  each(@status, .(@s) {
19    .@{btn-cls-prefix}.@{btn-cls-prefix}-@{t}.@{btn-cls-prefix}-@{s} {
20      background-color: rgb(var(~"@{btn-vars-prefix}-@{t}-color-bg-@{s}"));
21      color: rgb(var(~"@{btn-vars-prefix}-@{t}-color-text-@{s}"));
22      border: var(~"@{btn-vars-prefix}-@{t}-border-width") solid rgb(var(~"@{btn-vars-prefix}-@{t}-color-border-@{s}"));
23      border-radius: var(~"@{btn-vars-prefix}-@{t}-border-radius");
24      padding: var(~"@{btn-vars-prefix}-@{t}-padding-y") var(~"@{btn-vars-prefix}-@{t}-padding-x");
25    }
26  })
27});

We use the @import keyword provided by Less to import the @ezreal-ui/token. By doing this, we can use all the variables defined in @ezreal-ui/token. This includes some common variables, such as prefix and ezreal-vars-prefix, as well as variables that are generated from the tokens, such as --ezreal-button-outlined-color-bg-disabled and --ezreal-button-outlined-padding-y.

In addition, we used the each function provided by Less to loop through and generate style code, which helps improve our coding efficiency. You can refer to the documentation to become familiar with these functions.

In the End

In conclusion, we have discussed how to integrate tokens into the development process from the developer's perspective. We have also covered important details about how to use Style Dictionary to manipulate tokens, such as transforming values, adding headers, and generating style files. Finally, we briefly demonstrated how to use the generated style files in a component library.

By using tokens within a design system, developers can greatly improve their workflow and increase consistency across projects. Token systems can also help with maintaining and updating styles, as well as making it easier to collaborate with other developers and designers.

In the next article, we will focus on using automation to link and automate the compilation, testing, publishing, and updating of tokens.