Regular Expressions


This section provides an overview of how to create Regular Expressions for use with some CAST features that allow their use - for example the Text Replacement feature for .NET and J2EE. CAST currently uses the Boost (Perl) based Regular Expression "engine".

The text below is an edited version of the Regex++ Library's regular expression syntax documentation. The original text can be found at http://www.boost.org/doc/libs/1_41_0/libs/regex/doc/html/boost_regex/syntax/perl_syntax.html.

Perl Regular Expression Syntax

In Perl regular expressions, all characters match themselves, except for the following special characters: . , [, {, }, (, ) , \, *, +, ? , |, ^, $ .

The single character '.', when used outside of a character set, will match any single character (including the newline character) except the NUL character (i.e. '\x00').

Anchors

A '^' character shall match the start of a line.

A '$' character shall match the end of a line.

Marked sub-expressions

A section beginning '(' and ending ')' acts as a marked sub-expression, whatever the matched sub-expression is split out in a separate field by the matching algorithms. Marked sub-expressions can also be repeated, or referred to, by a back-reference.

Non-marking grouping

A marked sub-expression is useful to lexically group part of a regular expression, but has the side-effect of spitting out an extra field in the result. As an alternative, you can lexically group part of a regular expression, without generating a marked sub-expression, by using '(?:' and ')'.

For example "(?:ab)+" will repeat "ab" without splitting out any separate sub-expressions.

Repeats

Any atom (a single character, a marked sub-expression, or a character class) can be repeated with the '*', '+', '?', and '{'...'}' operators.
The '*' operator will match the preceding atom zero or more times.

For example the expression "a*b" will match any of the following:

  • b
  • ab
  • aaaaaaaab

The '+' operator will match the preceding atom one or more times.

For example the expression "a+b" will match any of the following:

  • ab
  • aaaaaaaab

...but will not match:

  • b

The '?' operator will match the preceding atom zero or one times.

For example the expression "ca?b" will match any of the following:

  • cb
  • cab

...but will not match:

  • caab

An atom can also be repeated with a bounded repeat:

  • 'a{n}' Matches a repeated exactly n times.
  • 'a{n,}' Matches a repeated n or more times.
  • 'a{n,m}' Matches a repeated between n and m times inclusive.

For example the expression "a{2,3}" will match any of the following:

  • aa
  • aaa

...but neither of the following:

  • a
  • aaaa

It is an error to use a repeat operator if the preceding construct cannot be repeated. For example "a(*y)" will raise an error as there is nothing for the '*' operator to be applied to.

Non greedy repeats

The normal repeat operators are "greedy", that is to say they will consume as much input as possible. There are non-greedy versions available that will consume as little input as possible while still producing a match.

  • '*?' Matches the previous atom zero or more times, while consuming as little input as possible.
  • '+?' Matches the previous atom one or more times, while consuming as little input as possible.
  • '??' Matches the previous atom zero or one times, while consuming as little input as possible.
  • '{n,}?' Matches the previous atom n or more times, while consuming as little input as possible.
  • '{n,m}?' Matches the previous atom between n and m times, while consuming as little input as possible.
Back references

An escape character '\' followed by a digit n, where n is in the range 1-9, matches the same string that was matched by the sub-expression n.

For example the expression "(a*).*\1" will match the string:

  • aaaxaaa

...but none of the following:

  • aaxa
  • aaaxaa
  • aaaaxxaa
Alternation

The '|' operator will match either of its arguments.

For example the expression "abc|def" will match any of the following:

  • abc
  • def

Parenthesis can be used to group alternations.

For example the expression "ab(d|ef)" will match any of the following:

  • abd
  • abef

Empty alternatives are not allowed, as for example "a|", "a|(?:)", "|a", "(?:)|a", and "(?:a)??".

Character sets

A character set is a bracket-expression starting with '[' and ending with ']'. It defines a set of characters, and matches any single character that is a member of that set.

A bracket-expression may contain any combination of the following:

Single characters

For example "[abc]" will match either of the characters 'a', 'b', or 'c'.

Character ranges

For example "[a-z]" will match any single character in the range 'a' to 'z'.

Warning: Contrary to the default behavior of Perl regular expressions, a character b isn't within the range a to c if it collates within that range, because this would result in locale specific behavior. Instead, whether a character appears within a range is determined by comparing the code points of the characters only. However, since character ranges are locale dependent because they match any character that collates between the endpoints of the range, on platforms that may be affected by collation rules we recommend trying out matching and mismatching sample expressions for making sure that the ranges operate as expected.

Negation

If the bracket-expression begins with the '^' character, then it matches the complement of the characters it contains.

For example "[^abc]" matches none of the characters 'a', 'b', 'c' , and "[^a-z]" matches any character that is not in the range 'a' to 'z'.

Character classes

An expression of the form '[:name:]' inside a character set declaration matches the named character class name.

For example "[[:lower:]]" matches any lower case character, depending upon the current locale.

The following character class names are supported:

Name

Description

Depend upon current locale

alnum

Any alpha-numeric character.

X

alpha

Any alphabetic character.

X

blank

Any whitespace character that is not a line separator.

 

cntrl

Any control character.

 

d

Any decimal digit.

 

digit

Any decimal digit.

 

graph

Any graphical character.

 

l

Any lower case character.

X

lower

Any lower case character.

X

print

Any printable character.

 

punct

Any punctuation character.

 

s

Any whitespace character.

 

space

Any whitespace character.

 

u

Any upper case character.

X

upper

Any upper case character.

X

w

Any word character, i.e. alphanumeric characters + the underscore.

X

word

Any word character, i.e. alphanumeric characters + the underscore.

X

xdigit

Any hexadecimal digit character.

 

Note: In the current version, '[:unicode:]' (which means "any extended character whose code point is >= 256") should never match since only single-byte character sets are supported; thus the "unicode" character class name isn't part of the above table.

Equivalence classes and collating elements are not supported

In Perl regular expressions, an expression of the form '[=col=]' inside a set declaration is called equivalence class, and matches any character or collating element whose primary sort key is the same as that for collating element col (a primary sort key is one that ignores case, accentuation, or locale-specific tailoring).

In Perl regular expressions, an expression of the form '[.col.]' inside a set declaration is called collating element, and matches the collating element col. A collating element is any single character, or any sequence of characters, that collates as a single unit.
Since implementation of primary sort key and collating elements are reliant on the platform's collation and localization support, neither equivalence classes (whatever of the form '[...[=col=]...]' or '[^...[=col=]...]') nor collating elements (whatever of the form '[...[.col. ]...]' or '[^...[.col.]...]') are supported, because this would result in locale specific behavior.

Escapes

All the escape sequences that match a single character or a single character class are permitted within a character class definition, except the negated character classes ('\D', '\L', '\S', '\U', and '\W').

Combinations

All of the above can be combined in one character set declaration, for example "[[:digit:]a-c\n]" which matches decimal digits, 'a', 'b', 'c', and the newline character.

Escapes

Any special character preceded by an escape shall match itself. The following escape sequences are also supported:

Escapes matching a specific character

The following escape sequences are all synonyms for single characters:

Escape

Character

Meaning

\a

\x07

\a (bell)

\e

\x1B

← (ASCII escape character)

\f

\x0C

\f (form feed)

\n

\x0A

\n (newline)

\r

\x0D

\r (carriage return)

\t

\x09

\t (horizontal tab)

\v

\x0B

\v (vertical tab)

\cA - \c_

\x01 - \x1F

Matches the single character whose code point is the one of the indicated character less 0x40 (note: the code point of the matched character is constrained to be less or equal to \x1F).
Ex.: '\cG' will match the character whose code point is \x7 because CodePoint('G')-0x40=0x7.

\xdd

\xdd

A hexadecimal escape sequence: matches the single character whose code point is 0xdd.

\x{dddd}

\x{ddd}

\x{dd}

\x{d}

\xdddd

\xddd

\xdd

\xd

A hexadecimal escape sequence: matches the single character whose code point is 0xdddd (respectively 0xddd, 0xdd, and 0xd).

\0ddd

 

An octal escape sequence: matches the single character whose code point is 0ddd.

\N{name}

 

Matches the single character which symbolic name is given.

Note: Since only single-byte character sets are supported in the current version, the values for dddd in '\x{dddd}' should be less or equal to 00FF.

The following symbolic names are recognized for "name" in '\N{name}':

Symbolic Name

Character

SOH

\x01

STX

\x02

ETX

\x03

EOT

\x04

ENQ

\x05

ACK

\x06

alert

\x07

backspace

\x08

tab

\t

newline

\n

vertical-tab

\v

form-feed

\f

carriage-return

\r

SO

\xE

SI

\xF

DLE

\x10

DC1

\x11

DC2

\x12

DC3

\x13

DC4

\x14

NAK

\x15

SYN

\x16

ETB

\x17

CAN

\x18

EM

\x19

SUB

\x1A

ESC

\x1B

IS4

\x1C

IS3

\x1D

IS2

\x1E

IS1

\x1F

space

\x20

exclamation-mark

!

quotation-mark

"

number-sign

#

dollar-sign

$

percent-sign

%

ampersand

&

apostrophe

'

left-parenthesis

(

right-parenthesis

)

asterisk

*

plus-sign

+

comma

,

hyphen

-

period

.

slash

/

zero

0

one

1

two

2

three

3

four

4

five

5

six

6

seven

7

eight

8

nine

9

colon

:

semicolon

;

less-than-sign

<

equals-sign

=

greater-than-sign

>

question-mark

?

commercial-at

@

left-square-bracket

[

backslash

\

right-square-bracket

]

circumflex

^

underscore

_

grave-accent

`

left-curly-bracket

{

vertical-line

|

right-curly-bracket

}

tilde

~

DEL

\x7F

Note: In the current version, '\N{NUL}' doesn't recognize the NUL character (aka '\x00').

"Single character" character classes

  • Any escaped character x, if x is the name of a character class, shall match any character that is a member of that class.
  • Any escaped character X, if x, lower case of X, is the name of a character class, shall match any character that isn't a member of that class.

The following are supported by default:

Escape sequence

Equivalent to

\d

[[:digit:]] or [[:d:]]

\l

[[:lower:]] or [[:l:]]

\s

[[:space:]] or [[:s:]]

\u

[[:upper:]] or [[:u:]]

\w

[[:word:]] or [[:w:]]

\D

[^[:digit:]] or [^[:d:]]

\L

[^[:lower:]] or [^[:l:]]

\S

[^[:space:]] or [^[:s:]]

\U

[^[:upper:]] or [^[:u:]]

\W

[^[:word:]] or [^[:w:]]

Character Properties

The character property names in the following table are all equivalent to the names used in character classes.

Escape sequence

Description

Equivalent to

\pX

Matches any character that has the property X.

[[:X:]]

\p{name}

Matches any character that has the property name.

[[:name:]]

\PX

Matches any character that does not have the property X.

[^[:X:]]

\P{name}

Matches any character that does not have the property name.

[^[:name:]]

Word Boundaries

The following escape sequences match the boundaries of words:

Escape sequence

Description

\<

Matches the start of a word.

\>

Matches the end of a word.

\b

Matches a word boundary (the start or end of a word).

\B

Matches only when not at a word boundary.

Quoting Escape

The escape sequence "\Q" begins a "quoted sequence", where all the subsequent characters are treated as literals, until either the end of the regular expression or "\E" is found.

For example the expression "x+\Q\*+\Ey+" will match any of the following:

  • x\*+y
  • xx\*+yyy

Unicode Escapes

Escape sequence

Description

\C

Matches a single code point, exactly as "." does.

\X

Matches a combining character sequence: that is any non-combining character followed by a sequence of zero or more combining characters.

Other escapes

Any other escape sequence matches the character that is escaped.
For example "\@" matches a literal '@'.

Perl Extended Patterns

All Perl-specific extensions to the regular expression syntax start with '(?'.

Comments

'(?#...)' is treated as a comment: its contents are ignored.

Modifiers

The following Perl modifiers are supported:

  • i to turn into case-insensitive
  • s to enter into "dot=all" mode, i.e. a dot (.) matches any character, including the line terminator.

...and they can be used in the two following forms, where letters before the '-' turn the corresponding Perl modifier on, whereas letters afterward turn it off:

'(?is-is)' alters which of the Perl modifiers are in effect within the pattern. Changes take effect from the point that the block is first seen, and extends to the nearest enclosing ')', or to the end of the regular expression if no ')' follows.

For example:

  • Even though matching is made in case-sensitive mode, the expression "x((?i)a)y" will match 'xAy', whereas "x((?i)a)Y" won't because its trailing "Y" is out of the scope to which extends the 'i' Perl modifier.
  • If matching is made in case-insensitive mode, the expression "x((?-i)a)y" will match 'XaY', but won't match 'XAY'.

'(?is-is:pattern)' applies the specified Perl modifiers to pattern only.

For example:

  • Even though matching is made in case-sensitive mode, the expression "x(?i:a)y" will match 'xAy'.
  • If matching is made in case-insensitive mode, the expression "x(?-i:a)y" will match 'XaY', but won't match 'XAY'.

Non-marking grouping

'(?:pattern)' lexically groups pattern, without generating an additional sub-expression.

Note: Empty alternatives are disallowed, as for example "a|(?:)".

Look ahead

Look ahead is typically used to create the logical AND of two regular expressions:

  • '(?=pattern)' consumes zero characters, only if pattern matches.
  • '(?!pattern)' consumes zero characters, only if pattern does not match.

For example, if a password must contain a lower case letter, an upper case letter, a punctuation symbol, and be at least 6 characters long, then the following expression could be used to validate the password:

"(?=.*[[:lower:]])(?=.*[[:upper:]])(?=.*[[:punct:]]).{6,}"

Look behind

  • '(?<=pattern)' consumes zero characters, only if pattern could be matched against the characters preceding the current position (pattern must be of fixed length).
  • '(?<!pattern)' consumes zero characters, only if pattern could not be matched against the characters preceding the current position (pattern must be of fixed length).

Independent sub-expressions

'(?>pattern)' pattern is matched independently of the surrounding patterns, the expression will never backtrack into pattern.

Independent sub-expressions are typically used to improve performance: only the best possible match for pattern will be considered. If this doesn't allow the expression as a whole to match, then no match is found at all.

Conditional Expressions

  • '(?(condition)yes-pattern|no-pattern)' attempts to match yes-pattern if the condition is true, otherwise attempts to match no-pattern.
  • '(?(condition)yes-pattern)' attempts to match yes-pattern if the condition is true, otherwise fails.

The above condition may be either a forward look ahead asserts, or the index of a marked sub-expression in which case the condition becomes true if the sub-expression has been matched.

Operator precedence

The order of precedence for operators is as shown in the following table:

Collation-related bracket symbols

Note: [==] and [..] are not supported because this would result in locale specific behavior.

[::] [==] [..]

Escaped characters

\

Character set (bracket expression)

[ ]

Grouping

( )

Quantification

* + ? {m,n}

Concatenation

 

Anchoring

^ $

Alternation

|

What gets matched

If you view the regular expression as a directed - possibly cyclic - graph, then the best match found while matching the input text is the first match found by a depth-first-search performed on that graph.

Alternatively: The best match found is the leftmost match, with individual elements matched as follows:

Construct

What gets matched

Atom1 Atom2

Locates the best match for Atom1 that has a following match for Atom2.

Expression1|Expression2

If Expresion1 can be matched then returns that match, otherwise attempts to match Expression2.

S{n}

Matches S repeated exactly n times.

S{n,m}

Matches S repeated between n and m times, and as many times as possible.

S{n,m}?

Matches S repeated between n and m times, and as few times as possible.

S?, S*, S+

The same as S{0,1}, S{0,UINT_MAX}, S{1,UINT_MAX}, respectively.

S??, S*?, S+?

The same as S{0,1}?, S{0,UINT_MAX}?, S{1,UINT_MAX}?, respectively.

(?>S)

Matches the best match for S, and only that.

(?=S), (?<=S)

Matches only the best match for S (this is only visible if there are capturing parenthesis within S).

(?!S), (?<!S)

Considers only whether a match for S exists or not.

(?(condition)yes-pattern|no-pattern)

If condition is true, then only yes-pattern is considered, otherwise only no-pattern is considered.

Perl smix Modifiers

The Perl smix modifiers can be applied using a "(?smix-smix)" prefix to the regular expression: see "Perl Extended Patterns" above.

Note: Among the Perl smix modifiers, only the 'i' and 's' modifiers are supported, whereas the 'm' and 'x' modifiers should not be used in regular expressions


CAST Website