AWK Scripting

July 11, 2016 by S4

Filed under AWS Script

Last modified July 15, 2016

AWK Scripting




This white paper explains about the AWK programming. AWK is a pattern-matching program for processing files, especially when they are databases. It also provides the command list and its usage.


Pre Requisite

The pre requisite to understand this document is to have the working knowledge of Unix and  Shell Programming.


Document Usage

This document helps to understand the AWK programming. This gives command syntax of most of the commands. This document helps in writing the programs using AWK.



Details of Document

1.    Overview

 awk is a pattern-matching program for processing files, especially when they are databases. The new version of awk, called nawk, provides additional capabilities. Every modern Unix system comes with a version of new awk, and its use is recommended over old awk.

Different systems vary in what the two versions are called. Some have oawk and awk, for the old and new versions, respectively. Others have awk and nawk. Still others only have awk, which is the new version.

This example shows what happens if your awk is the old one:

$ awk 1 /dev/null
awk: syntax error near line 1
awk: bailing out near line 1

awk exits silently if it is the new version.

With original awk, you can:

·         Think of a text file as made up of records and fields in a textual database.

·         Perform arithmetic and string operations.

·         Use programming constructs such as loops and conditionals.

·         Produce formatted reports.

With nawk, you can also:

·         Define your own functions.

·         Execute Unix commands from a script.

·         Process the results of Unix commands.

·         Process command-line arguments more gracefully.

·         Work more easily with multiple input streams.

·         Flush open output files and pipes (latest Bell Labs awk).

In addition, with GNU awk (gawk), you can:

·         Use regular expressions to separate records, as well as fields.

·         Skip to the start of the next file, not just the next record.

·         Perform more powerful string substitutions.

·         Retrieve and format system time values.

2.    Command-Line Syntax

The syntax for invoking awk has two forms:

awk  [options]  'script'  var=value  file(s)
awk  [options]  -f scriptfile  var=value  file(s)

You can specify a script directly on the command line, or you can store a script in a scriptfile and specify it with -f. nawk allows multiple -f scripts. Variables can be assigned a value on the command line. The value can be a literal, a shell variable ($name), or a command substitution (`cmd`), but the value is available only after the BEGIN statement is executed.

awk operates on one or more files. If none are specified (or if - is specified), awk reads from the standard input.

The recognized options are:


Set the field separator to fs. This is the same as setting the system variable FS. Original awk allows the field separator to be only a single character. nawk allows fs to be a regular expression. Each input line, or record, is divided into fields by whitespace (blanks or tabs) or by some other user-definable record separator. Fields are referred to by the variables $1, $2,…, $n. $0 refers to the entire record.

-v var=value

Assign a value to variable var. This allows assignment before the script begins execution (available in nawk only).

To print the first three (colon-separated) fields of each record on separate lines:

awk -F: '{ print $1; print $2; print $3 }' /etc/passwd

More examples are shown in the section “Simple Pattern-Procedure Examples.”

3.    Patterns and Procedures

awk scripts consist of patterns and procedures:

pattern  { procedure }

Both are optional. If pattern is missing, { procedure } is applied to all lines; if { procedure } is missing, the matched line is printed.

3.1 Patterns

A pattern can be any of the following:

/regular expression/
relational expression
pattern-matching expression

·         Expressions can be composed of quoted strings, numbers, operators, functions, defined variables, or any of the predefined variables described later in the section “Built-in Variables.”

·         Regular expressions use the extended set of metacharacters and are described in Chapter 6, Pattern Matching.

·         ^ and $ refer to the beginning and end of a string (such as the fields), respectively, rather than the beginning and end of a line. In particular, these metacharacters will not match at a newline embedded in the middle of a string.

·         Relational expressions use the relational operators listed in the section “Operators” later in this chapter. For example, $2 > $1 selects lines for which the second field is greater than the first. Comparisons can be either string or numeric. Thus, depending on the types of data in $1 and $2, awk does either a numeric or a string comparison. This can change from one record to the next.

·         Pattern-matching expressions use the operators ~ (match) and !~ (don’t match). See the section “Operators” later in this chapter.

·         The BEGIN pattern lets you specify procedures that take place before the first input line is processed. (Generally, you set global variables here.)

·         The END pattern lets you specify procedures that take place after the last input record is read.

·         In nawk, BEGIN and END patterns may appear multiple times. The procedures are merged as if there had been one large procedure.

Except for BEGIN and END, patterns can be combined with the Boolean operators || (or), && (and), and ! (not). A range of lines can also be specified using comma-separated patterns:


3.2 Procedures

Procedures consist of one or more commands, functions, or variable assignments, separated by newlines or semicolons, and contained within curly braces. Commands fall into five groups:

·         Variable or array assignments

·         Printing commands

·         Built-in functions

·         Control-flow commands

·         User-defined functions (nawk only)

3.3 Simple Pattern-Procedure Examples

·         Print first field of each line:

{ print $1 }

·         Print all lines that contain pattern:


·         Print first field of lines that contain pattern:

/pattern/ { print $1 }

·         Select records containing more than two fields:

NF > 2

·         Interpret input records as a group of lines up to a blank line. Each line is a single field:

BEGIN { FS = "\n"; RS = "" }

·         Print fields 2 and 3 in switched order, but only on lines whose first field matches the string “URGENT”:

$1 ~ /URGENT/ { print $3, $2 }

·         Count and print the number of pattern found:

·         /pattern/ { ++x }
END { print x }

·         Add numbers in second column and print total:

·         { total += $2 }
END { print "column total is", total}

·         Print lines that contain less than 20 characters:

length($0) < 20

·         Print each line that begins with Name: and that contains exactly seven fields:

NF == 7 && /^Name:/

·         Print the fields of each input record in reverse order, one per line:

·         {
·           for (i = NF; i >= 1; i--)
·                  print $i

4.    Built-in Variables






Current filename


Field separator (a space)


Number of fields in current record


Number of the current record


Output format for numbers ("%.6g") and for conversion to string


Output field separator (a space)


Output record separator (a newline)


Record separator (a newline)


Entire input record


nth field in current record; fields are separated by FS



Number of arguments on command line


An array containing the command-line arguments, indexed from 0 to ARGC - 1


String conversion format for numbers ("%.6g") (POSIX)


An associative array of environment variables


Like NR, but relative to the current file


Length of the string matched by match() function


First position in the string matched by match() function


Separator character for array subscripts ("\034")



Index in ARGV of current input file


A string indicating the error when a redirection fails for getline or if close() fails


A space-separated list of field widths to use for splitting up the record, instead of FS


When true, all regular expression matches, string comparisons, and calls to index()s ignore case


The text matched by RS, which can be a regular expression in gawk





5.    Operators

The following table lists the operators, in order of increasing precedence, that are available in awk. Note: while ** and **= are common extensions, they are not part of POSIX awk.



= += -= *= /= %= ^= **=



C conditional expression (nawk only)


Logical OR (short-circuit)


Logical AND (short-circuit)


Array membership (nawk only)

~ !~

Match regular expression and negation

< <= > >= != ==

Relational operators



+ -

Addition, subtraction

* / %

Multiplication, division, and modulus (remainder)

+ - !

Unary plus and minus, and logical negation

^ **


++ --

Increment and decrement, either prefix or postfix


Field reference

6.    Variables and Array Assignments

Variables can be assigned a value with an = sign. For example:

FS = ","

Expressions using the operators +, -, /, and % (modulo) can be assigned to variables.

Arrays can be created with the split() function (see below), or they can simply be named in an assignment statement. Array elements can be subscripted with numbers (array[1], …, array[n]) or with strings. Arrays subscripted by strings are called associative arrays.[2] For example, to count the number of widgets you have, you could use the following script:

[2] In fact, all arrays in awk are associative; numeric subscripts are converted to strings before using them as array subscripts. Associative arrays are one of awk‘s most powerful features.

/widget/ { count["widget"]++ }        Count widgets
END      { print count["widget"] }            Print the count

You can use the special for loop to read all the elements of an associative array:

for (item in array)
        process array[item]

The index of the array is available as item, while the value of an element of the array can be referenced as array[item].

You can use the operator in to see if an element exists by testing to see if its index exists (nawk only):

if (index in array)

This sequence tests that array[index] exists, but you cannot use it to test the value of the element referenced by array[index].

You can also delete individual elements of the array using the delete statement (nawk only).

6.1 Escape Sequences

Within string and regular expression constants, the following escape sequences may be used. Note: The \x escape sequence is a common extension; it is not part of POSIX awk.






Alert (bell)


Vertical tab




Literal backslash


Form feed


Octal value nnn




Hexadecimal value nn


Carriage return


Literal double quote (in strings)




Literal slash (in regular expressions)

7.    User-Defined Functions

nawk allows you to define your own functions. This makes it easy to encapsulate sequences of steps that need to be repeated into a single place, and reuse the code from anywhere in your program. Note: for user-defined functions, no space is allowed between the function name and the left parenthesis when the function is called.

The following function capitalizes each word in a string. It has one parameter, named input, and five local variables, which are written as extra parameters.

# capitalize each word in a string
function capitalize(input,    result, words, n, i, w)
        result = ""
        n = split(input, words, " ")
        for (i = 1; i <= n; i++) {
               w = words[i]
               w = toupper(substr(w, 1, 1)) substr(w, 2)
               if (i > 1)
                       result = result " "
               result = result w
        return result

# main program, for testing
{ print capitalize($0) }

With this input data:

A test line with words and numbers like 12 on it.

This program produces:

A Test Line With Words And Numbers Like 12 On It.

8.     Group Listing of awk Functions and Commands

The following table classifies awk functions and commands.





Control Flow
















































9.    Implementation Limits

Many versions of awk have various implementation limits, on things such as:

·         Number of fields per record

·         Number of characters per input record

·         Number of characters per output record

·         Number of characters per field

·         Number of characters per printf string

·         Number of characters in literal string

·         Number of characters in character class

·         Number of files open

·         Number of pipes open

·         The ability to handle 8-bit characters and characters that are all zero (ASCII NUL)

gawk does not have limits on any of these items, other than those imposed by the machine architecture and/or the operating system.

10. Alphabetical Summary of Functions and Commands

The following alphabetical list of keywords and functions includes all that are available in awk, nawk, and gawk. nawk includes all old awk functions and keywords, plus some additional ones (marked as {N}). gawk includes all nawk functions and keywords, plus some additional ones (marked as {G}). Items marked with {B} are available in the Bell Labs awk. Items that aren’t marked with a symbol are available in all versions.

1.        atan2

atan2(y, x)

Return the arctangent of y/x in radians. {N}

2.        break


Exit from a while, for, or do loop.

3.        close


In most implementations of awk, you can have only 10 files open simultaneously and one pipe. Therefore, nawk provides a close function that allows you to close a file or a pipe. It takes as an argument the same expression that opened the pipe or file. This expression must be identical, character by character, to the one that opened the file or pipe; even whitespace is significant. {N}

4.        continue


Begin next iteration of while, for, or do loop.

5.        cos


Return the cosine of x, an angle in radians. {N}

6.        delete

delete array[element]
delete array

Delete element from array. The brackets are typed literally. The second form is a common extension, which deletes all elements of the array at one shot. {N}

7.        do

while (expr)

Looping statement. Execute statement, then evaluate expr and, if true, execute statement again. A series of statements must be put within braces. {N}

8.        exit

exit [expr]

Exit from script, reading no new input. The END procedure, if it exists, will be executed. An optional expr becomes awk‘s return value.

9.        exp


Return exponential of x (ex).

10.      fflush


Flush any buffers associated with open output file or pipe output-expr. {B}

gawk extends this function. If no output-expr is supplied, it flushes standard output. If output-expr is the null string (""), it flushes all open files and pipes. {G}

11.      for

for (init-expr; test-expr; incr-expr)

C-style looping construct. init-expr assigns the initial value of a counter variable. test-expr is a relational expression that is evaluated each time before executing the statement. When test-expr is false, the loop is exited. incr-expr increments the counter variable after each pass. All the expressions are optional. A missing test-expr is considered to be true. A series of statements must be put within braces.

12.      for

for (item in array)

Special loop designed for reading associative arrays. For each element of the array, the statement is executed; the element can be referenced by array[item]. A series of statements must be put within braces.

13.      function

function name(parameter-list) {

Create name as a user-defined function consisting of awk statements that apply to the specified list of parameters. No space is allowed between name and the left paren when the function is called. {N}

14.      getline

getline [var] [< file]
command | getline [var]

Read next line of input. Original awk doesn’t support the syntax to open multiple input streams. The first form reads input from file; the second form reads the output of command. Both forms read one record at a time, and each time the statement is executed, it gets the next record of input. The record is assigned to $0 and is parsed into fields, setting NF, NR and FNR. If var is specified, the result is assigned to var, and $0 and NF aren’t changed. Thus, if the result is assigned to a variable, the current record doesn’t change. getline is actually a function and returns 1 if it reads a record successfully, 0 if end-of-file is encountered, and -1 if it’s otherwise unsuccessful. {N}

15.      gensub

gensub(r, s, h [, t])

General substitution function. Substitute s for matches of the regular expression r in the string t. If h is a number, replace the hth match. If it is "g" or "G", substitute globally. If t is not supplied, $0 is used. Return the new string value. The original t is not modified. (Compare gsub and sub.) {G}

16.      gsub

gsub(r, s [, t])

Globally substitute s for each match of the regular expression r in the string t. If t is not supplied, defaults to $0. Return the number of substitutions. {N}

17.      if

if (condition)

If condition is true, do statement(s); otherwise do statement in the optional else clause. The condition can be an expression using any of the relational operators <, <=, ==, !=, >=, or >, as well as the array membership operator in, and the pattern-matching operators ~ and !~ (e.g., if ($1 ~ /[Aa].*/)). A series of statements must be put within braces. Another if can directly follow an else in order to produce a chain of tests or decisions.

18.      index

index(str, substr)

Return the position (starting at 1) of substr in str, or zero if substr is not present in str.

19.      int


Return integer value of x by truncating any fractional part.

20.      length


Return length of arg, or the length of $0 if no argument.

21.      log


Return the natural logarithm (base e) of x.

22.      match

match(s, r)

Function that matches the pattern, specified by the regular expression r, in the string s, and returns either the position in s, where the match begins, or 0 if no occurrences are found. Sets the values of RSTART and RLENGTH to the start and length of the match, respectively. {N}

23.      next


Read next input line and start new cycle through pattern/procedures statements.

24.      nextfile


Stop processing the current input file and start new cycle through pattern/procedures statements, beginning with the first record of the next file. {B} {G}

25.      print

print [ output-expr[, ...]] [ dest-expr ]

Evaluate the output-expr and direct it to standard output, followed by the value of ORS. Each comma-separated output-expr is separated in the output by the value of OFS. With no output-expr, print $0.

26.      Output Redirections

dest-expr is an optional expression that directs the output to a file or pipe.

> file

Directs the output to a file, overwriting its previous contents.

>> file

Appends the output to a file, preserving its previous contents. In both cases, the file is created if it does not already exist.

| command

Directs the output as the input to a Unix command.

Be careful not to mix > and >> for the same file. Once a file has been opened with >, subsequent output statements continue to append to the file until it is closed.

Remember to call close() when you have finished with a file or pipe. If you don’t, eventually you will hit the system limit on the number of simultaneously open files.

27.      printf

printf(format [, expr-list ]) [ dest-expr ]

An alternative output statement borrowed from the C language. It can produce formatted output and also output data without automatically producing a newline. format is a string of format specifications and constants. expr-list is a list of arguments corresponding to format specifiers. See print for a description of dest-expr.

format follows the conventions of the C-language printf(3S) library function. Here are a few of the most common formats:


A string.


A decimal number.

A floating-point number; n = total number of digits. m = number of digits after decimal point.


n specifies minimum field length for format type c, while - left-justifies value in field; otherwise, value is right-justified.

Like any string, format can also contain embedded escape sequences: \n (newline) or \t (tab) being the most common. Spaces and literal text can be placed in the format argument by quoting the entire argument. If there are multiple expressions to be printed, there should be multiple formats specified.

28.      Example

Using the script:

{ printf("The sum on line %d is %.0f.\n", NR, $1+$2) }

The following input line:

5   5

produces this output, followed by a newline:

The sum on line 1 is 10.

29.      rand


Generate a random number between 0 and 1. This function returns the same series of numbers each time the script is executed, unless the random number generator is seeded using srand(). {N}

30.      return

return [expr]

Used within a user-defined function to exit the function, returning value of expr. The return value of a function is undefined if expr is not provided. {N}

31.      sin


Return the sine of x, an angle in radians. {N}

32.      split

split(string, array [, sep])

Split string into elements of array array[1],…,array[n]. The string is split at each occurrence of separator sep. If sep is not specified, FS is used. The number of array elements created is returned.

33.      sprintf

sprintf(format [, expressions])

Return the formatted value of one or more expressions, using the specified format (see printf). Data is formatted but not printed. {N}

34.      sqrt


Return square root of arg.

35.      srand


Use optional expr to set a new seed for the random number generator. Default is the time of day. Return value is the old seed. {N}

36.      strftime

strftime([format [,timestamp]])

Format timestamp according to format. Return the formatted string. The timestamp is a time-of-day value in seconds since midnight, January 1, 1970, UTC. The format string is similar to that of sprintf. (See the Example for systime.) If timestamp is omitted, it defaults to the current time. If format is omitted, it defaults to a value that produces output similar to that of date. {G}

37.      sub

sub(r, s [, t])

Substitute s for first match of the regular expression r in the string t. If t is not supplied, defaults to $0. Return 1 if successful; 0 otherwise. {N}

38.      substr

substr(string, beg [, len])

Return substring of string at beginning position beg and the characters that follow to maximum specified length len. If no length is given, use the rest of the string.

39.      system


Function that executes the specified command and returns its status. The status of the executed command typically indicates success or failure. A value of 0 means that the command executed successfully. A nonzero value indicates a failure of some sort. The documentation for the command you’re running will give you the details.

The output of the command is not available for processing within the awk script. Use command | getline to read the output of a command into the script. {N}

40.      systime


Return a time-of-day value in seconds since midnight, January 1, 1970, UTC. {G}

41.      Example

Log the start and end times of a data-processing program:

   now = systime()
   mesg = strftime("Started at %m/%d/%Y %H:%M:%S", now)
   print mesg
process data ...
   now = systime()
   mesg = strftime("Ended at %m/%d/%Y %H:%M:%S", now)
   print mesg

42.      tolower


Translate all uppercase characters in str to lowercase and return the new string.[6] {N}

[6] Very early versions of nawk don’t support tolower() and toupper(). However, they are now part of the POSIX specification for awk, and are included in the SVR4 nawk.

43.      toupper


Translate all lowercase characters in str to uppercase and return the new string. {N}

44.      while

while (condition)

Do statement while condition is true (see if for a description of allowable conditions). A series of statements must be put within braces.

11.     printf Formats

Format specifiers for printf and sprintf have the following form:


The control letter is required. The format conversion control letters are as follows.




ASCII character


Decimal integer


Decimal integer (added in POSIX)


Floating-point format ([-]d.precisione[+-]dd)


Floating-point format ([-]d.precisionE[+-]dd)


Floating-point format ([-]ddd.precision)


e or f conversion, whichever is shortest, with trailing zeros removed


E or f conversion, whichever is shortest, with trailing zeros removed


Unsigned octal value




Unsigned hexadecimal number; uses af for 10 to 15


Unsigned hexadecimal number; uses AF for 10 to 15


Literal %

The optional flag is one of the following.




Left-justify the formatted value within the field.


Prefix positive values with a space and negative values with a minus.


Always prefix numeric values with a sign, even if the value is positive.


Use an alternate form: %o has a preceding 0; %x and %X are prefixed with 0x and 0X, respectively; %e, %E, and %f always have a decimal point in the result; and %g and %G do not have trailing zeros removed.


Pad output with zeros, not spaces. This happens only when the field width is wider than the converted result.

The optional width is the minimum number of characters to output. The result will be padded to this size if it is smaller. The 0 flag causes padding with zeros; otherwise, padding is with spaces.

The precision is optional. Its meaning varies by control letter, as shown in this table.


Precision Means

%d, %i, %o

The minimum number of digits to print

%u, %x, %X


%e, %E, %f

The number of digits to the right of the decimal point

%g, %G

The maximum number of significant digits


The maximum number of characters to print

Leave a Comment