part of PR pkg/48447. BufferedTokenizer extracts token delimited entities from a sequence of arbitrary inputs.