XDS — Design System

XDSTokenizer@xds/core · Tokenizer

Usage

Tokenizer is a multi-select input that lets users search, select, and manage multiple items displayed as removable chips. Use it when users need to build a set of selections from a searchable data source, like adding team members, applying tags, or choosing filters.

Best practices

Guidance	Practices
Do	Write a placeholder that tells users what they can search for — "Search people…" or "Add tags…" — so the input is not a blank mystery.
Do	Set maxEntries when the number of selections should be bounded, like limiting a review to 5 approvers.
Do	Use hasCreate for free-form tagging where users need to enter values that do not exist in the search source.
Do	Show validation status with the status prop so users know immediately when a selection is missing or invalid.
Don't	Don’t use Tokenizer for single-item selection — use Typeahead instead. Tokenizer is for building sets of two or more items.
Don't	Avoid applying custom colors to individual tokens inside a Tokenizer — use the default token style for visual consistency across the set.
Don't	Don’t hide the label — every Tokenizer needs a visible label so users understand what they are selecting. Use isLabelHidden only when surrounding context makes the purpose obvious.

Anatomy

Element		Description
Label	required	The visible text above the input describing what the user is selecting. Also used as the accessible name.
Token chips		Removable chips representing each selected item. Each chip shows a label and a remove button.
Search input	required	The text input where users type to search the data source. Hides when maxEntries is reached.
Dropdown menu		The search results list that appears below the input as the user types.
End content		A trailing slot after the input for action buttons, counts, or other controls.
Clear button		A button that removes all selected tokens at once. Shown when hasClear is true and tokens are present.

Import

ts
import {XDSTokenizer} from '@xds/core/Tokenizer'

Props

Prop	Type	Description
labelrequired	string	Accessible label for the input.
searchSourcerequired	XDSSearchSource<T>	Data source providing search and bootstrap methods for populating the dropdown.
valuerequired	T[]	Array of currently selected items.
onChangerequired	(items: T[], change: XDSTokenizerChange<T>) => void	Called when selection changes. The change argument includes the affected item and type ('add' \| 'create' \| 'remove' \| 'reorder').
placeholder	string	Input placeholder text. Only shown when no tokens are selected.
maxEntries	number	Maximum number of selections allowed. Input is hidden when the limit is reached.
hasClear	boolean (default: false)	Show a clear-all button for bulk removal of all tokens.
renderToken	(item: T, onRemove: () => void) => ReactNode	Custom render function for selected tokens. Default renders XDSToken with label and onRemove.
renderItem	(item: T) => ReactNode	Custom render function for dropdown items. Default renders XDSTypeaheadItem.
isDisabled	boolean (default: false)	Disables the input and all token interactions.
status	XDSInputStatus	Validation status object with type and message for error/warning/success states.
isLabelHidden	boolean (default: false)	Visually hides the label while keeping it accessible.
description	string	Helper text displayed below the label.
isRequired	boolean (default: false)	Marks the field as required.
isOptional	boolean (default: false)	Shows an optional indicator on the label.
labelTooltip	string	Tooltip text shown on the label.
hasEntriesOnFocus	boolean (default: false)	Show bootstrap results on focus before typing.
maxMenuItems	number (default: 10)	Maximum number of dropdown items to display.
emptySearchResultsText	string (default: 'No results found')	Text shown when search returns no results.
hasAutoFocus	boolean (default: false)	Auto-focus the input on mount.
size	'sm' \| 'md' (default: 'md')	Input and token size.
debounceMs	number (default: 150)	Debounce delay in ms before triggering search. Set to 0 for synchronous sources.
hasCreate	boolean (default: false)	Allow users to create new tokens from free-text input. When true, a "Create" option appears in the dropdown for typed text that doesn't match existing results. The onChange change type is 'create' for these items.
onChangeQuery	(query: string) => void	Callback fired when the search query text changes.
endContent	ReactNode	Content to display at the end of the input row. Useful for buttons, result counts, or other controls.
xstyle	StyleXStyles	StyleX styles for layout customization (margins, positioning, sizing). Must be a stylex.create() value — not an inline style object like style={{}}.

Examples

Common configurations, variations, and states.

Tokenizer \u2014 ClearTokenizer with a built-in clear-all button for bulk removal of all selected tokens.

tsx
'use client';
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
const styles = stylex.create({
  fixed: {width: 400},
});
const users: XDSSearchableItem[] = [
  {id: '1', label: 'Alice Johnson'},
  {id: '2', label: 'Bob Smith'},
  {id: '3', label: 'Charlie Brown'},
  {id: '4', label: 'Diana Prince'},
  {id: '5', label: 'Eve Williams'},
];
const userSource: XDSSearchSource = {
  search: (query: string) =>
    users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
  bootstrap: () => users,
};
export default function TokenizerClear() {
  const [value, setValue] = useState<XDSSearchableItem[]>([users[0], users[1]]);
  return (
    <XDSStack direction="vertical" gap={2}>
      <XDSText type="supporting" color="secondary">
        Clear-all button appears when tokens are selected
      </XDSText>
      <XDSTokenizer
        label="Team Members"
        placeholder="Search people..."
        searchSource={userSource}
        value={value}
        onChange={items => setValue(items)}
        hasClear
        xstyle={styles.fixed}
      />
    </XDSStack>
  );
}

Tokenizer \u2014 CreatableFree-text tokenizer for creating custom tags and a combined create-or-search pattern. Use when users need to enter values that may not exist in a predefined list.

tsx
'use client';
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
const styles = stylex.create({
  fixed: {width: 400},
});
const emptySource: XDSSearchSource = {
  search: () => [],
  bootstrap: () => [],
};
const users: XDSSearchableItem[] = [
  {id: '1', label: 'Alice Johnson'},
  {id: '2', label: 'Bob Smith'},
  {id: '3', label: 'Charlie Brown'},
  {id: '4', label: 'Diana Prince'},
  {id: '5', label: 'Eve Williams'},
];
const userSource: XDSSearchSource = {
  search: (query: string) =>
    users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
  bootstrap: () => users,
};
export default function TokenizerCreatable() {
  const [tags, setTags] = useState<XDSSearchableItem[]>([]);
  const [members, setMembers] = useState<XDSSearchableItem[]>([]);
  return (
    <XDSStack direction="vertical" gap={4}>
      <XDSStack direction="vertical" gap={1}>
        <XDSText type="supporting" color="secondary">
          Free-text only
        </XDSText>
        <XDSTokenizer
          label="Tags"
          searchSource={emptySource}
          value={tags}
          onChange={items => setTags(items)}
          hasCreate
          placeholder="Type a tag and press Enter..."
          xstyle={styles.fixed}
        />
      </XDSStack>
      <XDSStack direction="vertical" gap={1}>
        <XDSText type="supporting" color="secondary">
          Create or search
        </XDSText>
        <XDSTokenizer
          label="Team Members"
          searchSource={userSource}
          value={members}
          onChange={items => setMembers(items)}
          hasCreate
          hasEntriesOnFocus
          placeholder="Search or type a new name..."
          xstyle={styles.fixed}
        />
      </XDSStack>
    </XDSStack>
  );
}

Tokenizer \u2014 End ContentTokenizer with an action button in the end slot. Use for inline actions like applying selections alongside the input.

tsx
'use client';
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSButton} from '@xds/core/Button';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
const styles = stylex.create({
  fixed: {width: 400},
});
const users: XDSSearchableItem[] = [
  {id: '1', label: 'Alice Johnson'},
  {id: '2', label: 'Bob Smith'},
  {id: '3', label: 'Charlie Brown'},
  {id: '4', label: 'Diana Prince'},
  {id: '5', label: 'Eve Williams'},
  {id: '6', label: 'Frank Miller'},
];
const userSource: XDSSearchSource = {
  search: (query: string) =>
    users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
  bootstrap: () => users,
};
export default function TokenizerEndContent() {
  const [value, setValue] = useState<XDSSearchableItem[]>([users[0], users[2]]);
  return (
    <XDSStack direction="vertical" gap={2}>
      <XDSText type="supporting" color="secondary">
        Action button in the end slot
      </XDSText>
      <XDSTokenizer
        label="Team Members"
        placeholder="Search people..."
        searchSource={userSource}
        value={value}
        onChange={items => setValue(items)}
        endContent={<XDSButton label="Apply" variant="primary" size="sm" />}
        xstyle={styles.fixed}
      />
    </XDSStack>
  );
}

Tokenizer \u2014 IconTokenizer with a leading search icon to visually reinforce the search behavior.

tsx
'use client';
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import {MagnifyingGlassIcon} from '@heroicons/react/24/outline';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
const styles = stylex.create({
  fixed: {width: 400},
});
const users: XDSSearchableItem[] = [
  {id: '1', label: 'Alice Johnson'},
  {id: '2', label: 'Bob Smith'},
  {id: '3', label: 'Charlie Brown'},
  {id: '4', label: 'Diana Prince'},
  {id: '5', label: 'Eve Williams'},
];
const userSource: XDSSearchSource = {
  search: (query: string) =>
    users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
  bootstrap: () => users,
};
export default function TokenizerIcon() {
  const [value, setValue] = useState<XDSSearchableItem[]>([users[0], users[2]]);
  return (
    <XDSStack direction="vertical" gap={2}>
      <XDSText type="supporting" color="secondary">
        Leading icon reinforces the search affordance
      </XDSText>
      <XDSTokenizer
        label="Team Members"
        placeholder="Search people..."
        searchSource={userSource}
        value={value}
        onChange={items => setValue(items)}
        startIcon={MagnifyingGlassIcon}
        xstyle={styles.fixed}
      />
    </XDSStack>
  );
}

Tokenizer \u2014 Max EntriesTokenizer with a maximum selection limit. The input hides automatically when the limit is reached, preventing further additions.

tsx
'use client';
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
const styles = stylex.create({
  fixed: {width: 400},
});
const skills: XDSSearchableItem[] = [
  {id: '1', label: 'React'},
  {id: '2', label: 'TypeScript'},
  {id: '3', label: 'GraphQL'},
  {id: '4', label: 'Node.js'},
  {id: '5', label: 'Python'},
  {id: '6', label: 'Rust'},
  {id: '7', label: 'Go'},
  {id: '8', label: 'Swift'},
];
const skillSource: XDSSearchSource = {
  search: (query: string) =>
    skills.filter(s => s.label.toLowerCase().includes(query.toLowerCase())),
  bootstrap: () => skills,
};
const MAX_SKILLS = 3;
export default function TokenizerMaxEntries() {
  const [value, setValue] = useState<XDSSearchableItem[]>([
    skills[0],
    skills[1],
  ]);
  return (
    <XDSStack direction="vertical" gap={2}>
      <XDSText type="supporting" color="secondary">
        Limited to {MAX_SKILLS} selections — {MAX_SKILLS - value.length}{' '}
        remaining
      </XDSText>
      <XDSTokenizer
        label="Top Skills"
        placeholder="Search skills..."
        description={`Choose up to ${MAX_SKILLS} skills`}
        searchSource={skillSource}
        value={value}
        onChange={items => setValue(items)}
        maxEntries={MAX_SKILLS}
        xstyle={styles.fixed}
      />
    </XDSStack>
  );
}

Tokenizer \u2014 OverflowTokenizer with overflow truncation when unfocused. Inline mode pushes content down on expand; layer mode overlays without shifting layout.

tsx
'use client';
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import {XDSText} from '@xds/core/Text';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
const styles = stylex.create({
  fixed: {width: 400, maxWidth: 400},
});
const users: XDSSearchableItem[] = [
  {id: '1', label: 'Alice Johnson'},
  {id: '2', label: 'Bob Smith'},
  {id: '3', label: 'Charlie Brown'},
  {id: '4', label: 'Diana Prince'},
  {id: '5', label: 'Eve Williams'},
  {id: '6', label: 'Frank Miller'},
];
const userSource: XDSSearchSource = {
  search: (query: string) =>
    users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
  bootstrap: () => users,
};
export default function TokenizerOverflow() {
  const [inlineValue, setInlineValue] = useState<XDSSearchableItem[]>(users);
  const [layerValue, setLayerValue] = useState<XDSSearchableItem[]>(users);
  return (
    <XDSStack direction="vertical" gap={4}>
      <XDSStack direction="vertical" gap={1}>
        <XDSText type="supporting" color="secondary">
          Inline overflow — content shifts down on expand
        </XDSText>
        <XDSTokenizer
          label="Inline Overflow"
          placeholder="Add more..."
          searchSource={userSource}
          value={inlineValue}
          onChange={items => setInlineValue(items)}
          tokenOverflowBehavior="unfocusedInline"
          xstyle={styles.fixed}
        />
      </XDSStack>
      <XDSStack direction="vertical" gap={1}>
        <XDSText type="supporting" color="secondary">
          Layer overflow — expands as overlay, no layout shift
        </XDSText>
        <XDSTokenizer
          label="Layer Overflow"
          placeholder="Add more..."
          searchSource={userSource}
          value={layerValue}
          onChange={items => setLayerValue(items)}
          tokenOverflowBehavior="unfocusedLayer"
          xstyle={styles.fixed}
        />
      </XDSStack>
    </XDSStack>
  );
}

Tokenizer \u2014 StatesTokenizer in disabled, error, warning, and success states. Use to communicate validation feedback or lock a selection from editing.

tsx
'use client';
import {useState} from 'react';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import {XDSStack} from '@xds/core/Layout';
import type {XDSSearchableItem, XDSSearchSource} from '@xds/core/Typeahead';
const styles = stylex.create({
  fixed: {width: 400},
});
const users: XDSSearchableItem[] = [
  {id: '1', label: 'Alice Johnson'},
  {id: '2', label: 'Bob Smith'},
  {id: '3', label: 'Charlie Brown'},
  {id: '4', label: 'Diana Prince'},
  {id: '5', label: 'Eve Williams'},
];
const userSource: XDSSearchSource = {
  search: (query: string) =>
    users.filter(u => u.label.toLowerCase().includes(query.toLowerCase())),
  bootstrap: () => users,
};
export default function TokenizerStates() {
  const [errorValue, setErrorValue] = useState<XDSSearchableItem[]>([]);
  const [warningValue, setWarningValue] = useState<XDSSearchableItem[]>([
    users[0],
  ]);
  const [successValue, setSuccessValue] = useState<XDSSearchableItem[]>([
    users[1],
    users[3],
  ]);
  return (
    <XDSStack direction="vertical" gap={4}>
      <XDSTokenizer
        label="Disabled field"
        searchSource={userSource}
        value={[users[0], users[2]]}
        onChange={() => {}}
        isDisabled
        xstyle={styles.fixed}
      />
      <XDSTokenizer
        label="Error message"
        placeholder="Search people..."
        searchSource={userSource}
        value={errorValue}
        onChange={items => setErrorValue(items)}
        isRequired
        status={{type: 'error', message: 'At least one reviewer is required'}}
        xstyle={styles.fixed}
      />
      <XDSTokenizer
        label="Warning message"
        placeholder="Search people..."
        searchSource={userSource}
        value={warningValue}
        onChange={items => setWarningValue(items)}
        status={{
          type: 'warning',
          message: 'Consider adding at least 2 approvers',
        }}
        xstyle={styles.fixed}
      />
      <XDSTokenizer
        label="Success message"
        placeholder="Search people..."
        searchSource={userSource}
        value={successValue}
        onChange={items => setSuccessValue(items)}
        status={{type: 'success', message: 'All required reviewers added'}}
        xstyle={styles.fixed}
      />
    </XDSStack>
  );
}

Showcase source

tsx
'use client';
import * as stylex from '@stylexjs/stylex';
import {XDSTokenizer} from '@xds/core/Tokenizer';
import type {XDSSearchSource} from '@xds/core/Typeahead';
const styles = stylex.create({
  fixed: {width: 400},
});
const source: XDSSearchSource = {
  search: () => [],
  bootstrap: () => [],
};
export default function TokenizerShowcase() {
  return (
    <XDSTokenizer
      label="Tags"
      placeholder="Search..."
      searchSource={source}
      value={[
        {id: '1', label: 'Design'},
        {id: '2', label: 'Engineering'},
      ]}
      onChange={() => {}}
      xstyle={styles.fixed}
    />
  );
}