Package net.aifusion.asn

Class Tokenizer

java.lang.Object
net.aifusion.asn.Tokenizer
public class Tokenizer extends Object
Class to break input into tokens
Author:
Sharad Singhal

  • Constructor Summary

    Constructors
    Constructor
    Description
    Tokenizer(File inputFile)
    Create a tokenizer and tokenize the input file
    Tokenizer(String inputString)
    Create a tokenizer and tokenize the input string

  • Method Summary

    Modifier and Type
    Method
    Description
    Token
    lookAhead()
    Get the lookAhead token
    Token
    lookAhead(int offset)
    Get the token at a given offset (positive or negative) from the current position
    boolean
    lookAheadIs(TokenType t)
    Check if the lookAhead token is of a given type
    boolean
    lookAheadIs(TokenType t, int offset)
    Check if the lookahead token at a given offset matches a given type
    boolean
    matches(TokenType[] expected)
    Check if a list of tokens matches the incoming tokens
    Token
    next()
    Get the next token from the tokenizer, and advance its position
    void
    pop()
    Pop the stack to reset the tokenizer position
    void
    push()
    Push the current position of the tokenizer onto a stack
    void
    reset()
    Reset the tokenizer
    Token
    scanFor(TokenType expected)
    Scan forward for a given token type (advancing the cursor to the token if found)
    Token
    scanFor(TokenType[] expected)
    Scan forward for a sequence of tokens
    Token
    skipOver(TokenType expected)
    Skip over a given token, and move past it

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • Tokenizer

      public Tokenizer(File inputFile)
      Create a tokenizer and tokenize the input file
      Parameters:
      inputFile - input file to use

    • Tokenizer

      public Tokenizer(String inputString)
      Create a tokenizer and tokenize the input string
      Parameters:
      inputString - - string to use

  • Method Details

    • push

      public void push()
      Push the current position of the tokenizer onto a stack

    • pop

      public void pop()
      Pop the stack to reset the tokenizer position

    • next

      public Token next()
      Get the next token from the tokenizer, and advance its position
      Returns:
      - next token.

    • lookAhead

      public Token lookAhead()
      Get the lookAhead token
      Returns:
      - token to be retrieved at the next call to next()

    • lookAhead

      public Token lookAhead(int offset)
      Get the token at a given offset (positive or negative) from the current position
      Parameters:
      offset - - offset from the current position
      Returns:
      - token at the location. EOF is returned if the location is past the end or the beginning

    • lookAheadIs

      public boolean lookAheadIs(TokenType t)
      Check if the lookAhead token is of a given type
      Parameters:
      t - - expected token type
      Returns:
      - true if the lookAheadToken matches the expected type, false otherwise

    • lookAheadIs

      public boolean lookAheadIs(TokenType t, int offset)
      Check if the lookahead token at a given offset matches a given type
      Parameters:
      t - - expected token type
      offset - - offset from the current position
      Returns:
      true if the token at the given offset matches the expected type, false otherwise

    • reset

      public void reset()
      Reset the tokenizer

    • skipOver

      public Token skipOver(TokenType expected)
      Skip over a given token, and move past it
      Parameters:
      expected - - expected token
      Returns:
      - token value of the expected token

    • matches

      public boolean matches(TokenType[] expected)
      Check if a list of tokens matches the incoming tokens
      Parameters:
      expected - - array containing list of tokens
      Returns:
      - true if the look-ahead matches the list, false otherwise

    • scanFor

      public Token scanFor(TokenType expected)
      Scan forward for a given token type (advancing the cursor to the token if found)
      Parameters:
      expected - - expected token type
      Returns:
      - expected token. Null if end is reached (position is not changed)

    • scanFor

      public Token scanFor(TokenType[] expected)
      Scan forward for a sequence of tokens
      Parameters:
      expected - - array containing the expected tokens
      Returns:
      - first token in the sequence. Null returned if not found (position is not changed)