Copybook parser reference

Mainframe Connector uses an ANTLR4-based parser that supports standard COBOL copybooks.

This document lists the preprocessing tasks performed by the copybook parser, the supported data types and usage restrictions, and how to configure field modifiers using the transcoder configuration with the qsam decode and qsam encode commands.

Preprocessing

Before parsing a copybook, the copybook parser preprocesses the data and performs the following steps:

  • Removes comment lines (* in column 7, or inline *> and *>CE comments).
  • Resolves line continuation (- in column 7 followed by a quoted literal).
  • Blanks out sequence number areas (columns 1–6) and column 73 and beyond. Code must start at column 8 or later.
  • Preserves preprocessor-specific statements like EJECT, SKIP1, SKIP2, SKIP3, and TITLE. These statements are parsed, but ignored. Copybooks containing preprocessor parameters that can be used by COPY REPLACING are not supported by the copybook parser. In these copybooks, identifiers are surrounded by colons (:).

Supported data types and restrictions

The following are the data types supported by the copybook parser and the restrictions for their use:

  • Only a single top-level 01 record is allowed in a copybook, and the 01 record cannot have an OCCURS clause.
  • Level 66 (RENAMES / alias) and level 77 (standalone) fields are not supported. Level 88 (condition names) is parsed and ignored.
  • All elementary (non-group) fields must have a PICTURE clause, except for hexadecimal floating-point fields (COMP-1 and COMP-2). The following PICTURE fields are supported:
    • Alphabetic (PIC A), alphanumeric (PIC X), and alphanumeric-edited (A, X, 9, B, 0, /).
    • PIC G (DBCS; requires USAGE DISPLAY-1).
    • PIC N (national or DBCS) and national-edited (N, B, 0, /).
    • PIC U (UTF-8), including PIC U BYTE-LENGTH n.
    • Numeric (9, S, V) and numeric-edited ($, B, /, V, Z, 0, 9, ,, ., *, +, -, CR, DB). Note that PIC P (scaling position) is not supported.
    • Zoned decimal and packed decimal fields with precision up to 18 are decoded into 64-bit types (LONG or DECIMAL64), and higher-precision fields are decoded into arbitrary-precision types (BIG_INTEGER or BIG_DECIMAL). When schema_validation_mode is set to BIG_QUERY, the maximum supported precision is 76 (BIGNUMERIC) and the maximum scale is 38 (for POSTGRES, max precision is 147,455 and max scale is 16,383; for MYSQL, max precision is 65 and max scale is 30).
  • IBM Hexadecimal Floating-Point (HFP) is supported using COMP-1 / COMPUTATIONAL-1 (single precision, 4 bytes) and COMP-2 / COMPUTATIONAL-2 (double precision, 8 bytes), or by applying the hfp field modifier in the transcoder configuration.
  • REDEFINES clauses are supported on statically sized fields when all redefined variants have the same byte size. Redefined fields cannot be referenced as the length field of an OCCURS DEPENDING ON clause or as the target_field of a null_if modifier.
  • OCCURS clauses are supported for fixed-length arrays (OCCURS n TIMES), dynamic-length arrays (OCCURS min TO max TIMES DEPENDING ON field), and packed dynamic-length arrays (OCCURS UNBOUNDED DEPENDING ON field). The ASCENDING/DESCENDING KEY and INDEXED BY phrases in an OCCURS clause are not supported, and DEPENDING ON must reference a field name in scope.
  • Only the following USAGE clauses are supported:
    • DISPLAY (default for alphanumeric and zoned decimal fields)
    • DISPLAY-1 (DBCS)
    • NATIONAL (default for PIC N fields; encoded as UTF-16BE)
    • UTF-8 (default for PIC U fields)
    • BINARY, COMP, COMPUTATIONAL, COMP-4, COMPUTATIONAL-4, COMP-5, COMPUTATIONAL-5
    • COMP-3, COMPUTATIONAL-3, PACKED-DECIMAL
    • COMP-1, COMPUTATIONAL-1, COMP-2, COMPUTATIONAL-2
  • DATE and TIMESTAMP are supported using the transcoder configuration.
  • Null indicators are supported using the transcoder configuration.
  • Double-byte character set (DBCS) fields PIC G and PIC N are supported. By default, PIC N fields without a USAGE clause are assigned USAGE NATIONAL (UTF-16BE). To use a PIC N field as DBCS, either specify USAGE DISPLAY-1 in the copybook or override its character encoding in the transcoder configuration.
  • Variable-length character strings are supported using the transcoder configuration.
  • The SIGN clause (LEADING, TRAILING, SEPARATE CHARACTER) is supported.
  • Comments are supported.

Support for date and timestamp fields

Mainframe Connector supports decoding and encoding date and timestamp fields using the format_date and format_timestamp field modifiers in the transcoder configuration (--transcode-configuration). You can apply these modifiers across matching fields using field_suffixes or to specific fields using field_overrides:

{
  "field_suffixes": [
    {
      "suffix": "SUFFIX_STRING",
      "modifier": {
        "format_date": {
          "formats": [
            {
              "size": FIELD_SIZE,
              "pattern": "FORMAT_PATTERN"
            }
          ]
        }
      }
    }
  ]
}

The following list describes the configuration properties in more detail:

  • suffix: Matches any copybook field whose name ends with -SUFFIX_STRING or _SUFFIX_STRING (case-insensitive). Ensure that suffix doesn't contain a leading hyphen or underscore. Alternatively, use field_overrides with "field": "FIELD_NAME" to target an individual field.
  • format_date / format_timestamp: Defines the modifier used to parse or format the field:
    • format_date can be applied to sized string fields (PIC X) or integer numeric fields (such as PIC 9(n), COMP, or COMP-3 with scale 0 and precision up to 18, which are dynamically converted to their unsigned string representation before formatting).
    • format_timestamp can be applied to sized string fields (PIC X). Timestamps are interpreted in UTC unless a timezone or offset is included in the pattern.
  • formats: A list of DateTimeFormat objects, each specifying:
    • size: The size of the field that the pattern applies to (byte length for string fields, or precision for numeric fields).
    • pattern: A date or timestamp pattern. During decoding (qsam decode), formats matching the field's size are evaluated in order until one matches. During encoding (qsam encode), only the first matching format for the field's size is used. For more information on valid formats, see Supported date and timestamp formats.
  • Removing suffixes (Optional): To remove -SUFFIX_STRING or _SUFFIX_STRING from the output field name, add a rename transformation to the transformations array in your transcoder configuration.

Examples

  • To format 8-byte fields ending with -DT8 or _DT8 as DATE with pattern yyyyMMdd, and 10-byte fields ending with -DT or _DT with either MM-dd-yyyy or MM/dd/yyyy:

    {
      "field_suffixes": [
        {
          "suffix": "DT8",
          "modifier": {
            "format_date": {
              "formats": [
                {
                  "size": 8,
                  "pattern": "yyyyMMdd"
                }
              ]
            }
          }
        },
        {
          "suffix": "DT",
          "modifier": {
            "format_date": {
              "formats": [
                {
                  "size": 10,
                  "pattern": "MM-dd-yyyy"
                },
                {
                  "size": 10,
                  "pattern": "MM/dd/yyyy"
                }
              ]
            }
          }
        }
      ]
    }
    
  • To format 26-byte fields ending with -TIMESTAMP, _TIMESTAMP, -TS, or _TS as TIMESTAMP with pattern yyyy-MM-dd HH.mm.ss.SSSSSS and strip the suffix from the output column names:

    {
      "field_suffixes": [
        {
          "suffix": "TIMESTAMP",
          "modifier": {
            "format_timestamp": {
              "formats": [
                {
                  "size": 26,
                  "pattern": "yyyy-MM-dd HH.mm.ss.SSSSSS"
                }
              ]
            }
          }
        },
        {
          "suffix": "TS",
          "modifier": {
            "format_timestamp": {
              "formats": [
                {
                  "size": 26,
                  "pattern": "yyyy-MM-dd HH.mm.ss.SSSSSS"
                }
              ]
            }
          }
        }
      ],
      "transformations": [
        {
          "rename": {
            "find": "^(.*)[-_](TIMESTAMP|TS)$",
            "replace": "$1"
          }
        }
      ]
    }
    

Support for null indicators

Mainframe Connector supports null indicators through the null_if field modifier in the transcoder configuration (--transcode-configuration).

To configure a suffix-based null indicator (where a field such as COL1-NID acts as the null indicator for COL1 in the same record), define an entry in field_suffixes with "is_inverse": true and "target_field": "$self":

{
  "field_suffixes": [
    {
      "suffix": "NULL_INDICATOR_NAME",
      "is_inverse": true,
      "modifier": {
        "null_if": {
          "target_field": "$self",
          "null_value": "NULL_VALUE",
          "non_null_value": "NOT_NULL_VALUE"
        }
      }
    }
  ]
}

The following list describes the parameters you can use to configure null indicators:

  • suffix: Corresponds to the suffixes -NULL_INDICATORNAME or NULL_INDICATOR_NAME that identify a null-indicator field in a copybook.
  • is_inverse: When set to true on a FieldSuffix, the modifier is applied to the sibling field whose name matches the indicator field without the suffix (for example, COL1 when the indicator is COL1-NID1), and "$self" refers to the indicator field itself. You can also apply null_if directly on a specific field using field_overrides and set target_field to the name of the indicator field.
  • null_value or null_values: Specifies the single value (null_value) or list of values (null_values) in target_field that signal that the referenced field is null.
  • non_null_value or non_null_values: (Optional) Specifies the single value (non_null_value) or list of values (non_null_values) in target_field to signal that the referenced field is not null. If omitted, any value other than null_value or null_values is treated as not null. At least one of null_value(s) or non_null_value(s) must be specified.
  • Excluding or keeping the null-indicator field: By default, the null-indicator field is kept as a column in the output schema. To omit the null-indicator field from the output table while still using it during transcoding, add an exclude transformation for the indicator field in the transformations array (do not apply the filler modifier to a null-indicator field, as filler skips decoding the field altogether).
  • Overriding the indicator field type: (Optional) To force a null-indicator field to be decoded as bytes or binary, add a non-inverse field_suffixes (or field_overrides) rule for the indicator field with "modifier": { "bytes": {} } or "modifier": { "binary": {} }. For bytes, express the null_value and non_null_value as hexadecimal strings (for example, "FF" or "00").

Examples

In the following examples, COL1-NID1 and FIELD-NID2 are excluded from the output table after being evaluated as null indicators, while COL2-NULL is retained in the output schema.

Copybook snippet:

01 REC.
   10 COL1-NID1 PIC S9(4) USAGE COMP.
   10 COL1      PIC S9(6) USAGE COMP.
   10 FIELD     PIC X(10).
   10 FIELD-NID2 PIC X(1).
   10 COL2      PIC X(10).
   10 COL2-NULL PIC X(1).

Transcoder configuration (transcode-configuration.json):

{
  "field_suffixes": [
    {
      "suffix": "NID1",
      "is_inverse": true,
      "modifier": {
        "null_if": {
          "target_field": "$self",
          "null_value": "-1",
          "non_null_value": "0"
        }
      }
    },
    {
      "suffix": "NID2",
      "is_inverse": true,
      "modifier": {
        "null_if": {
          "target_field": "$self",
          "null_value": "?"
        }
      }
    },
    {
      "suffix": "NULL",
      "is_inverse": true,
      "modifier": {
        "null_if": {
          "target_field": "$self",
          "null_value": "?"
        }
      }
    }
  ],
  "transformations": [
    {
      "exclude": {
        "field": "COL1-NID1"
      }
    },
    {
      "exclude": {
        "field": "FIELD-NID2"
      }
    }
  ]
}

Support for DBCS fields

When using DBCS fields (PIC G or PIC N with USAGE DISPLAY-1), ensure the following:

  • PIC G fields must explicitly specify USAGE DISPLAY-1.
  • PIC N fields without a USAGE clause default to USAGE NATIONAL (UTF-16BE). To treat a PIC N field as a DBCS field, specify USAGE DISPLAY-1 in the copybook (or override its encoding in the transcoder configuration).
  • By default, USAGE DISPLAY-1 fields are decoded and encoded using the cp300 (x-IBM300) DBCS character set, which expects raw DBCS bytes without shift-out (0x0E) and shift-in (0x0F) control characters.
  • To use a different DBCS character set (such as Cp834) or a multi-byte character set (MBCS) that uses shift-out (0x0E) and shift-in (0x0F) bytes (such as x-IBM930, x-IBM933, x-IBM935, x-IBM937, x-IBM939, x-IBM942, x-IBM942C, x-IBM943, x-IBM943C, x-IBM949, x-IBM949C, x-IBM950, x-IBM964, x-IBM970, or x-IBM1364), configure the string field modifier in the transcoder configuration (--transcode-configuration).

You can set the DBCS encoding globally for all USAGE DISPLAY-1 fields using defaults.dbcs, or for specific fields using field_overrides or field_suffixes:

{
  "defaults": {
    "dbcs": {
      "string": {
        "encoding": "x-IBM930"
      }
    }
  }
}

Support for variable-length character strings

The copybook parser supports variable-length character string group fields using the varlen field modifier in the transcoder configuration:

10 VAR-GRP.
   15 VAR-LEN  PIC 9(4) USAGE COMP.
   15 VAR-TEXT PIC X(50).

When using the varlen field modifier, ensure the following:

  • Apply the varlen modifier to the group item (for example, VAR-GRP) using field_overrides or field_suffixes in the transcoder configuration:

    {
      "field_overrides": [
        {
          "field": "VAR-GRP",
          "modifier": {
            "varlen": {}
          }
        }
      ]
    }
    
  • The group item must contain exactly two subfields:

    1. The first subfield must decode to a 64-bit integer (LONG, such as a binary COMP or zoned decimal integer field) that specifies the byte length of the string data.
    2. The second subfield must be an unmodified string field (such as PIC X(n), PIC N(n), or PIC U(n)).
  • No specific subfield suffixes (such as -LEN or -TEXT) are required, and the resulting column in the output schema uses the name of the group item (VAR-GRP).

Unsupported fields and constructs

The following sections describe the fields and constructs that are not supported by the copybook parser.

Unsupported COBOL constructs

The copybook parser can parse the following COBOL constructs, even though these constructs are not supported. If you use these constructs in your copybook, Mainframe Connector shows an error:

  • ALIGNED
  • BLANK WHEN ZERO
  • COMMON, OWN, LOCAL
  • DYNAMIC LENGTH (such as PIC X DYNAMIC LENGTH or PIC U DYNAMIC LENGTH LIMIT n)
  • INTEGER, STRING
  • JUSTIFIED
  • ASCENDING/DESCENDING KEY within OCCURS
  • INDEXED BY within OCCURS
  • RECEIVED BY
  • RECORD AREA
  • RENAMES
  • SYNCHRONIZED / SYNC
  • THREAD-LOCAL
  • TYPE IS ...
  • TYPEDEF
  • USING LANGUAGE/CONVENTION

Unsupported data types

The following USAGE types can be parsed by the grammar, but are not supported by the copybook compiler:

  • BIT
  • CONTROL-POINT
  • DATE
  • DOUBLE
  • EVENT
  • FUNCTION-POINTER
  • INDEX
  • KANJI
  • LOCK
  • POINTER
  • PROCEDURE-POINTER
  • REAL
  • SQL
  • TASK