Longest-Match Tokenizer with Unknown Token Merging

Problem: Longest-Match Tokenizer

Given a vocabulary and an input string, implement a tokenizer. Scan the string from left to right. At each position, choose the longest vocabulary token that match...