org.antlr.runtime.TokenRewriteStream Class Reference

Inheritance diagram for org.antlr.runtime.TokenRewriteStream:

[legend]

Collaboration diagram for org.antlr.runtime.TokenRewriteStream:

[legend]

List of all members.

Public Member Functions
	TokenRewriteStream ()
	TokenRewriteStream (TokenSource tokenSource)
	TokenRewriteStream (TokenSource tokenSource, int channel)
void	rollback (int instructionIndex)
void	rollback (String programName, int instructionIndex)
void	deleteProgram ()
void	deleteProgram (String programName)
void	insertAfter (Token t, Object text)
void	insertAfter (int index, Object text)
void	insertAfter (String programName, Token t, Object text)
void	insertAfter (String programName, int index, Object text)
void	insertBefore (Token t, Object text)
void	insertBefore (int index, Object text)
void	insertBefore (String programName, Token t, Object text)
void	insertBefore (String programName, int index, Object text)
void	replace (int index, Object text)
void	replace (int from, int to, Object text)
void	replace (Token indexT, Object text)
void	replace (Token from, Token to, Object text)
void	replace (String programName, int from, int to, Object text)
void	replace (String programName, Token from, Token to, Object text)
void	delete (int index)
void	delete (int from, int to)
void	delete (Token indexT)
void	delete (Token from, Token to)
void	delete (String programName, int from, int to)
void	delete (String programName, Token from, Token to)
int	getLastRewriteTokenIndex ()
String	toOriginalString ()
String	toOriginalString (int start, int end)
String	toString ()
String	toString (String programName)
String	toString (int start, int end)
String	toString (String programName, int start, int end)
String	toDebugString ()
String	toDebugString (int start, int end)
Static Public Attributes
static final String	DEFAULT_PROGRAM_NAME = "default"
static final int	PROGRAM_INIT_SIZE = 100
static final int	MIN_TOKEN_INDEX = 0
Protected Member Functions
void	init ()
int	getLastRewriteTokenIndex (String programName)
void	setLastRewriteTokenIndex (String programName, int i)
List	getProgram (String name)
Map	reduceToSingleOperationPerIndex (List rewrites)
String	catOpText (Object a, Object b)
List	getKindOfOps (List rewrites, Class kind)
List	getKindOfOps (List rewrites, Class kind, int before)
Protected Attributes
Map	programs = null
Map	lastRewriteTokenIndexes = null
Private Member Functions
List	initializeProgram (String name)
Classes
class	DeleteOp
class	InsertBeforeOp
class	ReplaceOp
class	RewriteOperation

---

Detailed Description

Useful for dumping out the input stream after doing some augmentation or other manipulations.

You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :)

Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1.

Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example,

CharStream input = new ANTLRFileStream("input"); TLexer lex = new TLexer(input); TokenRewriteStream tokens = new TokenRewriteStream(lex); T parser = new T(tokens); parser.startRule();

Then in the rules, you can execute Token t,u; ... input.insertAfter(t, "text to put after t");} input.insertAfter(u, "text after u");} System.out.println(tokens.toString());

Actually, you have to cast the 'input' to a TokenRewriteStream. :(

You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer:

tokens.insertAfter("pass1", t, "text to put after t");} tokens.insertAfter("pass2", u, "text after u");} System.out.println(tokens.toString("pass1")); System.out.println(tokens.toString("pass2"));

If you don't use named rewrite streams, a "default" stream is used as the first example shows.

Definition at line 83 of file TokenRewriteStream.java.

---

Constructor & Destructor Documentation

org.antlr.runtime.TokenRewriteStream.TokenRewriteStream

(

)

Definition at line 163 of file TokenRewriteStream.java.

org.antlr.runtime.TokenRewriteStream.TokenRewriteStream

(

TokenSource

tokenSource

)

Definition at line 173 of file TokenRewriteStream.java.

org.antlr.runtime.TokenRewriteStream.TokenRewriteStream	(	TokenSource	tokenSource,
		int	channel
	)

Definition at line 178 of file TokenRewriteStream.java.

---

Member Function Documentation

void org.antlr.runtime.TokenRewriteStream.init

(

)

[protected]

Definition at line 167 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.rollback

(

int

instructionIndex

)

Definition at line 183 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.rollback	(	String	programName,
		int	instructionIndex
	)

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

Definition at line 191 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.deleteProgram

(

)

Definition at line 198 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.deleteProgram

(

String

programName

)

Reset the program so that no instructions exist

Definition at line 203 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertAfter	(	Token	t,
		Object	text
	)

Definition at line 207 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertAfter	(	int	index,
		Object	text
	)

Definition at line 211 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertAfter	(	String	programName,
		Token	t,
		Object	text
	)

Definition at line 215 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertAfter	(	String	programName,
		int	index,
		Object	text
	)

Definition at line 219 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertBefore	(	Token	t,
		Object	text
	)

Definition at line 225 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertBefore	(	int	index,
		Object	text
	)

Definition at line 229 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertBefore	(	String	programName,
		Token	t,
		Object	text
	)

Definition at line 233 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.insertBefore	(	String	programName,
		int	index,
		Object	text
	)

Definition at line 237 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	int	index,
		Object	text
	)

Definition at line 245 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	int	from,
		int	to,
		Object	text
	)

Definition at line 249 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	Token	indexT,
		Object	text
	)

Definition at line 253 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	Token	from,
		Token	to,
		Object	text
	)

Definition at line 257 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	String	programName,
		int	from,
		int	to,
		Object	text
	)

Definition at line 261 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.replace	(	String	programName,
		Token	from,
		Token	to,
		Object	text
	)

Definition at line 271 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete

(

int

index

)

Definition at line 278 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete	(	int	from,
		int	to
	)

Definition at line 282 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete

(

Token

indexT

)

Definition at line 286 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete	(	Token	from,
		Token	to
	)

Definition at line 290 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete	(	String	programName,
		int	from,
		int	to
	)

Definition at line 294 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.delete	(	String	programName,
		Token	from,
		Token	to
	)

Definition at line 298 of file TokenRewriteStream.java.

int org.antlr.runtime.TokenRewriteStream.getLastRewriteTokenIndex

(

)

Definition at line 302 of file TokenRewriteStream.java.

int org.antlr.runtime.TokenRewriteStream.getLastRewriteTokenIndex

(

String

programName

)

[protected]

Definition at line 306 of file TokenRewriteStream.java.

void org.antlr.runtime.TokenRewriteStream.setLastRewriteTokenIndex	(	String	programName,
		int	i
	)			[protected]

Definition at line 314 of file TokenRewriteStream.java.

List org.antlr.runtime.TokenRewriteStream.getProgram

(

String

name

)

[protected]

Definition at line 318 of file TokenRewriteStream.java.

List org.antlr.runtime.TokenRewriteStream.initializeProgram

(

String

name

)

[private]

Definition at line 326 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toOriginalString

(

)

Definition at line 332 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toOriginalString	(	int	start,
		int	end
	)

Definition at line 336 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toString

(

)

Reimplemented from org.antlr.runtime.CommonTokenStream.

Definition at line 344 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toString

(

String

programName

)

Definition at line 348 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toString	(	int	start,
		int	stop
	)

Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.

Reimplemented from org.antlr.runtime.CommonTokenStream.

Definition at line 352 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toString	(	String	programName,
		int	start,
		int	end
	)

Definition at line 356 of file TokenRewriteStream.java.

Map org.antlr.runtime.TokenRewriteStream.reduceToSingleOperationPerIndex

(

List

rewrites

)

[protected]

We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested). Inserts to same index need to be combined etc... Here are the cases:

I.i.u I.j.v leave alone, nonoverlapping I.i.u I.i.v combine: Iivu

R.i-j.u R.x-y.v | i-j in x-y delete first R R.i-j.u R.i-j.v delete first R R.i-j.u R.x-y.v | x-y in i-j ERROR R.i-j.u R.x-y.v | boundaries overlap ERROR

I.i.u R.x-y.v | i in x-y delete I I.i.u R.x-y.v | i not in x-y leave alone, nonoverlapping R.x-y.v I.i.u | i in x-y ERROR R.x-y.v I.x.u R.x-y.uv (combine, delete I) R.x-y.v I.i.u | i not in x-y leave alone, nonoverlapping

I.i.u = insert u before op @ index i R.x-y.u = replace x-y indexed tokens with u

First we need to examine replaces. For any replace op:

1. wipe out any insertions before op within that range. 2. Drop any replace op before that is contained completely within that range. 3. Throw exception upon boundary overlap with any previous replace.

Then we can deal with inserts:

1. for any inserts to same index, combine even if not adjacent. 2. for any prior replace with same left boundary, combine this insert with replace and delete this replace. 3. throw exception if index in same range as previous replace

Don't actually delete; make op null in list. Easier to walk list. Later we can throw as we add to index -> op map.

Note that I.2 R.2-2 will wipe out I.2 even though, technically, the inserted stuff would be before the replace range. But, if you add tokens in front of a method body '{' and then delete the method body, I think the stuff before the '{' you added should disappear too.

Return a map from token index to operation.

Definition at line 447 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.catOpText	(	Object	a,
		Object	b
	)			[protected]

Definition at line 533 of file TokenRewriteStream.java.

List org.antlr.runtime.TokenRewriteStream.getKindOfOps	(	List	rewrites,
		Class	kind
	)			[protected]

Definition at line 540 of file TokenRewriteStream.java.

List org.antlr.runtime.TokenRewriteStream.getKindOfOps	(	List	rewrites,
		Class	kind,
		int	before
	)			[protected]

Get all operations before an index of a particular kind

Definition at line 545 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toDebugString

(

)

Definition at line 555 of file TokenRewriteStream.java.

String org.antlr.runtime.TokenRewriteStream.toDebugString	(	int	start,
		int	end
	)

Definition at line 559 of file TokenRewriteStream.java.

---

Member Data Documentation

final String org.antlr.runtime.TokenRewriteStream.DEFAULT_PROGRAM_NAME = "default" [static]

Definition at line 84 of file TokenRewriteStream.java.

final int org.antlr.runtime.TokenRewriteStream.PROGRAM_INIT_SIZE = 100 [static]

Definition at line 85 of file TokenRewriteStream.java.

final int org.antlr.runtime.TokenRewriteStream.MIN_TOKEN_INDEX = 0 [static]

Definition at line 86 of file TokenRewriteStream.java.

Map org.antlr.runtime.TokenRewriteStream.programs = null [protected]

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)

Definition at line 158 of file TokenRewriteStream.java.

Map org.antlr.runtime.TokenRewriteStream.lastRewriteTokenIndexes = null [protected]

Map String (program name) -> Integer index

Definition at line 161 of file TokenRewriteStream.java.

---

The documentation for this class was generated from the following file:

• /Users/parrt/antlr/code/antlr/main/runtime/Java/src/org/antlr/runtime/TokenRewriteStream.java