Parser examples

These are two example parsers built from the grammar presented in the language specification: one using the nearley.js parser generator, and one using ANTLR v4. Every file is listed below, both inline and as a download.

Both examples implement only the grammar. Their output is an abstract syntax tree, and they perform no semantic validation. Neither enforces the syntactic differences between proto2 and proto3 syntax. But they are reasonable starting points for building a more fully-featured parser.

For a more thorough example that implements all validation, as well as the other phases of a compiler up to and including descriptor production, see protocompile, the Go implementation that powers Buf. It’s described at the end of this page.

nearley.js

A small NPM package containing a lexer and parser for JavaScript. The lexer is built with moo.js, and the parser is built on top of it with the nearley.js parser generator.

To try it out, download the files below into a directory and use the included proto2ast.js script:

# First, make sure the package's dependencies are installed.
npm install

# This script parses proto source from stdin and prints the resulting
# parse tree to stdout.
./proto2ast.js < example.proto

The output is the default parse tree from nearley.js, which is an untyped tree of nested JavaScript arrays. Each array represents a production, and the leaves of the tree are tokens, each represented by a JavaScript object.

protobuf-grammar.js is generated from protobuf-grammar.ne. To regenerate it after editing the grammar, run npm run regen.

Files

• protobuf-grammar.ne - the nearley.js grammar
• protobuf-grammar.js - the parser generated from the grammar
• lexer.js - the moo.js lexer
• parser.js - constructs a parser from the generated grammar
• proto2ast.js - CLI entry point: reads proto from stdin, writes JSON to stdout
• example.proto - sample input exercising most of the grammar
• package.json and package-lock.json - NPM package definition

protobuf-grammar.ne

# A nearley.js grammar for the Protobuf IDL

@{%
const moolex = require("./lexer.js");
const lexer = moolex.lexer;

// discard whitespace and comment tokens
const ignore = [ "whitespace", "line_comment", "block_comment" ]
lexer.next = (next => () => {
  let token;
  while ((token = next.call(lexer)) && (
    ignore.includes(token.type)
  )) {}
  return token;
})(lexer.next);
%}

@lexer lexer

File -> %byte_order_mark:? SyntaxDecl:? FileElement:*

FileElement -> ImportDecl |
               PackageDecl |
               OptionDecl |
               MessageDecl |
               EnumDecl |
               ExtensionDecl |
               ServiceDecl |
               EmptyDecl

SyntaxDecl -> "syntax" "=" SyntaxLevel ";" |
              "edition" "=" Edition ";"

SyntaxLevel -> StringLiteral

Edition -> StringLiteral

StringLiteral -> %string_literal:+

EmptyDecl -> ";"

PackageDecl -> "package" PackageName ";"

PackageName -> QualifiedIdentifier

ImportDecl -> "import" ( "weak" | "public" | "option" ):? ImportedFileName ";"

ImportedFileName -> StringLiteral

TypeName -> ".":? QualifiedIdentifier

SymbolVisibility -> "export" | "local"

QualifiedIdentifier -> Identifier ( "." Identifier ):*

FieldDeclTypeName -> FieldDeclIdentifier ( "." QualifiedIdentifier ):? |
                     FullyQualifiedIdentifier

MessageFieldDeclTypeName -> MessageFieldDeclIdentifier ( "." QualifiedIdentifier ):? |
                            FullyQualifiedIdentifier

ExtensionFieldDeclTypeName -> ExtensionFieldDeclIdentifier ( "." QualifiedIdentifier ):? |
                              FullyQualifiedIdentifier

OneofFieldDeclTypeName -> OneofFieldDeclIdentifier ( "." QualifiedIdentifier ):? |
                          FullyQualifiedIdentifier

MethodDeclTypeName -> MethodDeclIdentifier ( "." QualifiedIdentifier ):? |
                      FullyQualifiedIdentifier

FieldDeclIdentifier -> %identifier | "message"    | "enum"     | "oneof"  |
                       "reserved"  | "extensions" | "extend"   | "option" |
                       "optional"  | "required"   | "repeated" | "stream"

MessageFieldDeclIdentifier -> %identifier | "stream"

ExtensionFieldDeclIdentifier -> %identifier | "message"  | "enum"       |
                                "oneof"     | "reserved" | "extensions" |
                                "extend"    | "option"   | "stream"

OneofFieldDeclIdentifier -> %identifier | "message"    | "enum"   | "oneof" |
                            "reserved"  | "extensions" | "extend" | "stream"

MethodDeclIdentifier -> %identifier | "message"    | "enum"     | "oneof"  |
                        "reserved"  | "extensions" | "extend"   | "option" |
                        "optional"  | "required"   | "repeated" | "group"

FullyQualifiedIdentifier -> "." QualifiedIdentifier

OptionDecl -> "option" OptionName "=" OptionValue ";"

CompactOptions -> "[" CompactOption ( "," CompactOption ):* "]"

CompactOption  -> OptionName "=" OptionValue

OptionName -> ( Identifier | "(" TypeName ")" ) ( "." OptionName ):*

OptionValue -> ScalarValue | MessageLiteralWithBraces

ScalarValue  -> StringLiteral | IntLiteral | FloatLiteral |
                SpecialFloatLiteral | Identifier

IntLiteral   -> "-":? %int_literal

FloatLiteral -> "-":? %float_literal

SpecialFloatLiteral -> "-"  "inf" | "-" "nan"

MessageLiteralWithBraces -> "{" MessageTextFormat "}"

MessageTextFormat -> ( MessageLiteralField ( "," | ";" ):? ):*

MessageLiteralField -> MessageLiteralFieldName ":" Value |
                       MessageLiteralFieldName MessageValue

MessageLiteralFieldName -> FieldName |
                           "[" SpecialFieldName "]"

SpecialFieldName        -> ExtensionFieldName | TypeURL

ExtensionFieldName      -> QualifiedIdentifier

TypeURL                 -> QualifiedIdentifier "/" QualifiedIdentifier

Value          -> ScalarValue | MessageLiteral | ListLiteral

MessageValue   -> MessageLiteral | ListOfMessagesLiteral

MessageLiteral -> MessageLiteralWithBraces |
                  "<" MessageTextFormat ">"

ListLiteral -> "[" ( ListElement ( "," ListElement ):* ):? "]"

ListElement -> ScalarValue | MessageLiteral

ListOfMessagesLiteral -> "[" ( MessageLiteral ( "," MessageLiteral ):* ):? "]"

MessageDecl -> SymbolVisibility:? "message" MessageName "{" MessageElement:* "}"

MessageName    -> Identifier

MessageElement -> MessageFieldDecl |
                  MapFieldDecl |
                  GroupDecl |
                  OneofDecl |
                  OptionDecl |
                  ExtensionRangeDecl |
                  MessageReservedDecl |
                  MessageDecl |
                  EnumDecl |
                  ExtensionDecl |
                  EmptyDecl

MessageFieldDecl -> FieldDeclWithCardinality |
                    MessageFieldDeclTypeName FieldName "=" FieldNumber
                       CompactOptions:? ";"

FieldDeclWithCardinality -> FieldCardinality FieldDeclTypeName FieldName
                            "=" FieldNumber CompactOptions:? ";"

FieldCardinality -> "required" | "optional" | "repeated"

FieldName        -> Identifier

FieldNumber      -> %int_literal

MapFieldDecl -> MapType FieldName "=" FieldNumber CompactOptions:? ";"

MapType    -> "map" "<" MapKeyType "," TypeName ">"

MapKeyType -> "int32"   | "int64"   | "uint32"   | "uint64"   | "sint32" | "sint64" |
              "fixed32" | "fixed64" | "sfixed32" | "sfixed64" | "bool"   | "string"

GroupDecl -> FieldCardinality:? "group" FieldName "=" FieldNumber
             CompactOptions:? "{" MessageElement:* "}"

OneofDecl -> "oneof" OneofName "{" OneofElement:* "}"

OneofName    -> Identifier

OneofElement -> OptionDecl |
                OneofFieldDecl |
                OneofGroupDecl

OneofFieldDecl -> OneofFieldDeclTypeName FieldName "=" FieldNumber
                  CompactOptions:? ";"

OneofGroupDecl -> "group" FieldName "=" FieldNumber
                  CompactOptions:? "{" MessageElement:* "}"

ExtensionRangeDecl -> "extensions" TagRanges CompactOptions:? ";"

TagRanges     -> TagRange ( "," TagRange ):*

TagRange      -> TagRangeStart ( "to" TagRangeEnd ):?

TagRangeStart -> FieldNumber

TagRangeEnd   -> FieldNumber | "max"

MessageReservedDecl -> "reserved" ( TagRanges | NameStrings | Names ) ";"

NameStrings -> StringLiteral ( "," StringLiteral ):*

Names -> Identifier ( "," Identifier ):*

EnumDecl -> SymbolVisibility:? "enum" EnumName "{" EnumElement:* "}"

EnumName    -> Identifier

EnumElement -> OptionDecl |
               EnumValueDecl |
               EnumReservedDecl |
               EmptyDecl

EnumValueDecl -> EnumValueName "=" EnumValueNumber CompactOptions:? ";"

EnumValueName   -> Identifier

EnumValueNumber -> "-":? %int_literal

EnumReservedDecl -> "reserved" ( EnumValueRanges | NameStrings | Names ) ";"

EnumValueRanges     -> EnumValueRange ( "," EnumValueRange ):*

EnumValueRange      -> EnumValueRangeStart ( "to" EnumValueRangeEnd ):?

EnumValueRangeStart -> EnumValueNumber

EnumValueRangeEnd   -> EnumValueNumber | "max"

ExtensionDecl -> "extend" ExtendedMessage "{" ExtensionElement:* "}"

ExtendedMessage  -> TypeName

ExtensionElement -> ExtensionFieldDecl |
                    GroupDecl

ExtensionFieldDecl -> FieldDeclWithCardinality |
                      ExtensionFieldDeclTypeName FieldName "=" FieldNumber
                         CompactOptions:? ";"

ServiceDecl -> "service" ServiceName "{" ServiceElement:* "}"

ServiceName    -> Identifier

ServiceElement -> OptionDecl |
                  MethodDecl |
                  EmptyDecl

MethodDecl -> "rpc" MethodName InputType "returns" OutputType ";" |
              "rpc" MethodName InputType "returns" OutputType "{" MethodElement:* "}"

MethodName    -> Identifier

InputType     -> MessageType

OutputType    -> MessageType

MethodElement -> OptionDecl |
                EmptyDecl

MessageType -> "(" "stream":? MethodDeclTypeName ")"

Identifier -> %identifier | %sometimes_identifier

lexer.js

const moo = require('moo');

exports.lexer = moo.compile({
    // discarded input
    whitespace: {
        match: /[ \n\r\t\f\v]+/,
        lineBreaks: true,
    },
    line_comment: /\/\/.*$/,
    block_comment: {
        match: /\/\*[^]*?\*\//,
        lineBreaks: true,
    },

    byte_order_mark: '\ufeff',

    // tokens
    identifier: {
        match: /[_A-Za-z][_A-Za-z0-9]*/,
        type: moo.keywords({
            sometimes_identifier: [
                "group", "message", "enum", "oneof", "reserved", "extensions",
                "extend", "option", "optional", "required", "repeated", "stream"
            ]
        })
    },
    numeric_literal: {
        match: /\.?[0-9](?:[.0-9a-dA-Df-zF-Z]|[eE][+-]?)*/,
        type: txt => {
            if (txt.match(/^0(?:[0-7]*|[xX][0-9a-zA-z]+)|[1-9][0-9]*$/)) {
                return 'int_literal';
            } else if (txt.match(/^(?:[0-9]+\.[0-9]*(?:[eE][+-]?[0-9]+)?|[0-9]+[eE][+-]?[0-9]+|\.[0-9]+(?:[eE][+-]?[0-9]+)?)$/)) {
                return 'float_literal';
            } else {
                // not used in grammar so will trigger parse failure
                return 'invalid numeric literal';
            }
        },
    },
    string_literal: /'(?:[^\n\\']|\\(?:[abfnrtv\\"'?]|x[A-Fa-f0-9]{2}|u[A-Fa-f0-9]{4}|U[A-Fa-f0-9]{8}|[0-7]{1,3}))*'|"(?:[^\n\\"]|\\(?:[abfnrtv\\"'?]|x[A-Fa-f0-9]{2}|u[A-Fa-f0-9]{4}|U[A-Fa-f0-9]{8}|[0-7]{1,3}))*"/,
    sym: /[;,./:=\-(){}\[\]<>]/
})

parser.js

const nearley = require("nearley");
const grammar = require("./protobuf-grammar.js");

exports.newParser = () => new nearley.Parser(nearley.Grammar.fromCompiled(grammar));

proto2ast.js

#! /usr/bin/env node

const process = require('process');
const parser = require('./parser.js');

async function read(stream) {
    const chunks = [];
    for await (const chunk of stream) chunks.push(chunk);
    return Buffer.concat(chunks).toString('utf8');
}

(async () => {
    const input = await read(process.stdin);

    let p = parser.newParser();

    p.feed(input);

    if (p.results.length === 0) {
        throw new Error("unexpected EOF");
    }

    if (p.results.length > 1) {
        console.log("Internal error! Grammar is ambiguous: " + p.results.length + " possible ASTs identified!");
    }

    let ast = JSON.stringify(p.results[0], null, 4);
    process.stdout.write(ast + '\n');
})();

package.json

{
  "scripts": {
    "regen": "nearleyc protobuf-grammar.ne -o protobuf-grammar.js",
    "start": "./proto2ast.js < example.proto > example-ast.json"
  },
  "dependencies": {
    "moo": "^0.5.1",
    "nearley": "^2.20.1"
  }
}

example.proto

edition = "2024";

package test.users.v1;

import option "buf/validate/validate.proto";

option features.field_presence = IMPLICIT;

message User {
  uint64 uid = 1;

  export message Name {
    string last_name = 1;
    string first_name = 2;
    string middle_initial = 3;
  }
  Name name = 2;

  export message Address {
    string street_line1 = 1;
    string street_line2 = 2;
    string city = 3;
    string state_province = 4;
    string postal_code = 5;
    string country = 6;
  }
  Address address = 3;
}

service UserService {
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
  rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse);
  rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse);
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
}

message CreateUserRequest {
  User.Name name = 1;
  User.Address address = 2;
}

message CreateUserResponse {
  uint64 uid = 1;
}

message DeleteUserRequest {
  uint64 uid = 1;
}

message DeleteUserResponse {
}

message UpdateUserRequest {
  User user = 1;
}

message UpdateUserResponse {
}

message GetUserRequest {
  uint64 uid = 1;
}

message GetUserResponse {
  User user = 1;
}

ANTLR

Configuration for building a lexer and parser for the Protobuf IDL using ANTLR v4. Use the ANTLR tool to generate a parser from the .g4 files.

ANTLR already ships grammars for the Protobuf IDL, with separate ones for proto2 and proto3 syntax. However, those are based on older grammars that were once published to Google’s official docs site. Not only do they require you to know or ascertain the file’s syntax level before parsing, they also contain inaccuracies that make them unsuitable for real use. Just as the language specification remedies the inaccuracies in the official documentation, the configuration here remedies the inaccuracies in the existing ANTLR grammars.

To try it out, download the files below into a directory and use the included build.sh and show_ast.sh scripts:

# First, build an ANTLR parser. This puts the parser (Java and class files)
# in a subdirectory named 'tmp'.
./build.sh

# This script parses proto source from stdin and shows a GUI with the
# resulting parse tree.
./show_ast.sh < example.proto

These scripts expect the ANTLR tool to be downloaded and installed at /usr/local/lib/antlr-4.10.1-complete.jar. If you downloaded the file to a different location, update your CLASSPATH environment variable to include that location before running either script.

Files

• ProtobufLexer.g4 - the lexer grammar
• ProtobufParser.g4 - the parser grammar
• build.sh - generates and compiles the parser into tmp/
• show_ast.sh - parses stdin and displays the parse tree in a GUI
• example.proto - sample input exercising most of the grammar

ProtobufLexer.g4

lexer grammar ProtobufLexer;

// discard whitespace and comment tokens
WS  :   [ \t\r\n\u000C]+ -> channel(HIDDEN);
LINE_COMMENT: '//' ~[\r\n]* -> channel(HIDDEN);
COMMENT: '/*' .*? '*/' -> channel(HIDDEN);

// character classes
fragment LETTER: [A-Za-z_];
fragment DECIMAL_DIGIT: [0-9];
fragment OCTAL_DIGIT: [0-7];
fragment HEX_DIGIT: [0-9A-Fa-f];

BYTE_ORDER_MARK: '\uFEFF';

// identifiers and keywords
SYNTAX: 'syntax';
EDITION: 'edition';
IMPORT: 'import';
WEAK: 'weak';
PUBLIC: 'public';
PACKAGE: 'package';
OPTION: 'option';
INF: 'inf';
NAN: 'nan';
REPEATED: 'repeated';
OPTIONAL: 'optional';
REQUIRED: 'required';
BOOL: 'bool';
STRING: 'string';
BYTES: 'bytes';
FLOAT: 'float';
DOUBLE: 'double';
INT32: 'int32';
INT64: 'int64';
UINT32: 'uint32';
UINT64: 'uint64';
SINT32: 'sint32';
SINT64: 'sint64';
FIXED32: 'fixed32';
FIXED64: 'fixed64';
SFIXED32: 'sfixed32';
SFIXED64: 'sfixed64';
GROUP: 'group';
ONEOF: 'oneof';
MAP: 'map';
EXTENSIONS: 'extensions';
TO: 'to';
MAX: 'max';
RESERVED: 'reserved';
ENUM: 'enum';
MESSAGE: 'message';
EXTEND: 'extend';
SERVICE: 'service';
RPC: 'rpc';
STREAM: 'stream';
RETURNS: 'returns';
EXPORT: 'export';
LOCAL: 'local';

IDENTIFIER: LETTER ( LETTER | DECIMAL_DIGIT )*;

// numeric literals
INT_LITERAL: DECIMAL_LITERAL | OCTAL_LITERAL | HEX_LITERAL;
fragment DECIMAL_LITERAL: [1-9] DECIMAL_DIGIT*;
fragment OCTAL_LITERAL: '0' OCTAL_DIGIT*;
fragment HEX_LITERAL: '0' ( 'x' | 'X' ) HEX_DIGIT+ ;

FLOAT_LITERAL: DECIMAL_DIGIT+ DOT DECIMAL_DIGIT* DECIMAL_EXPONENT? |
                DECIMAL_DIGIT+ DECIMAL_EXPONENT |
                DOT DECIMAL_DIGIT+ DECIMAL_EXPONENT?;
fragment DECIMAL_EXPONENT: ( 'e' | 'E' ) (PLUS | MINUS)? DECIMAL_DIGIT+;

// we can't do a two pass approach for identifying numeric literals, like the
// spec describes, but we can instead provide explicit tokens for *invalid*
// numeric literals, so we can still reject them (instead of incorrectly
// identifying input as a valid literal, no whitespace, and then another
// token).
INVALID_INT_LITERAL: INT_LITERAL ( LETTER | DOT );
INVALID_FLOAT_LITERAL: FLOAT_LITERAL ( LETTER | DOT );

// string literals
STRING_LITERAL: SINGLE_QUOTED_STRING_LITERAL | DOUBLE_QUOTED_STRING_LITERAL;

fragment SINGLE_QUOTED_STRING_LITERAL: '\'' ( ~[\n\u0000'\\] | RUNE_ESCAPE_SEQ )* '\'';
fragment DOUBLE_QUOTED_STRING_LITERAL: '"' ( ~[\n\u0000"\\] | RUNE_ESCAPE_SEQ )* '"';

fragment RUNE_ESCAPE_SEQ: SIMPLE_ESCAPE_SEQ | HEX_ESCAPE_SEQ | OCTAL_ESCAPE_SEQ | UNICODE_ESCAPE_SEQ;
fragment SIMPLE_ESCAPE_SEQ: '\\' ( 'a' | 'b' | 'f' | 'n' | 'r' | 't' | 'v' | '\\' | '\'' | '"' | '?' );
fragment HEX_ESCAPE_SEQ: '\\' ( 'x' | 'X' ) HEX_DIGIT HEX_DIGIT;
fragment OCTAL_ESCAPE_SEQ: '\\' OCTAL_DIGIT ( OCTAL_DIGIT OCTAL_DIGIT? )?;
fragment UNICODE_ESCAPE_SEQ: '\\' 'u' HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT |
                             '\\' 'U' HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT
                                      HEX_DIGIT HEX_DIGIT HEX_DIGIT HEX_DIGIT;

// punctuation and operators
SEMICOLON: ';';
COMMA: ',';
DOT: '.';
SLASH: '/';
COLON: ':';
EQUALS: '=';
MINUS: '-';
PLUS: '+';
L_PAREN: '(';
R_PAREN: ')';
L_BRACE: '{';
R_BRACE: '}';
L_BRACKET: '[';
R_BRACKET: ']';
L_ANGLE: '<';
R_ANGLE: '>';

ProtobufParser.g4

parser grammar ProtobufParser;

options {
    tokenVocab = ProtobufLexer;
}

file: BYTE_ORDER_MARK? syntaxDecl? fileElement* EOF;

fileElement: importDecl |
               packageDecl |
               optionDecl |
               messageDecl |
               enumDecl |
               extensionDecl |
               serviceDecl |
               emptyDecl;

syntaxDecl: SYNTAX EQUALS syntaxLevel SEMICOLON |
            EDITION EQUALS edition SEMICOLON;

syntaxLevel: stringLiteral;

edition: stringLiteral;

stringLiteral: STRING_LITERAL+;

emptyDecl: SEMICOLON;

packageDecl: PACKAGE packageName SEMICOLON;

packageName: qualifiedIdentifier;

importDecl: IMPORT ( WEAK | PUBLIC | OPTION )? importedFileName SEMICOLON;

importedFileName: stringLiteral;

typeName: DOT? qualifiedIdentifier;

symbolVisibility: EXPORT | LOCAL;

qualifiedIdentifier: identifier ( DOT identifier )*;

fieldDeclTypeName: fieldDeclIdentifier ( DOT qualifiedIdentifier )? |
                    fullyQualifiedIdentifier;

messageFieldDeclTypeName: messageFieldDeclIdentifier ( DOT qualifiedIdentifier )? |
                            fullyQualifiedIdentifier;

extensionFieldDeclTypeName: extensionFieldDeclIdentifier ( DOT qualifiedIdentifier )? |
                            fullyQualifiedIdentifier;

oneofFieldDeclTypeName: oneofFieldDeclIdentifier ( DOT qualifiedIdentifier )? |
                        fullyQualifiedIdentifier;

methodDeclTypeName: methodDeclIdentifier ( DOT qualifiedIdentifier )? |
                    fullyQualifiedIdentifier;

fieldDeclIdentifier: alwaysIdent  | MESSAGE    | ENUM     | ONEOF  |
                        RESERVED  | EXTENSIONS | EXTEND   | OPTION |
                        OPTIONAL  | REQUIRED   | REPEATED | STREAM;

messageFieldDeclIdentifier: alwaysIdent | STREAM;

extensionFieldDeclIdentifier: alwaysIdent | MESSAGE  | ENUM       |
                                ONEOF     | RESERVED | EXTENSIONS |
                                EXTEND    | OPTION   | STREAM;

oneofFieldDeclIdentifier: alwaysIdent | MESSAGE    | ENUM     | ONEOF  |
                            RESERVED  | EXTENSIONS | EXTEND   | OPTION |
                            OPTIONAL  | REQUIRED   | REPEATED | GROUP;

methodDeclIdentifier: alwaysIdent | MESSAGE    | ENUM     | ONEOF  |
                        RESERVED  | EXTENSIONS | EXTEND   | OPTION |
                        OPTIONAL  | REQUIRED   | REPEATED | GROUP;

fullyQualifiedIdentifier: DOT qualifiedIdentifier;

optionDecl: OPTION optionName EQUALS optionValue SEMICOLON;

compactOptions: L_BRACKET compactOption ( COMMA compactOption )* R_BRACKET;

compactOption : optionName EQUALS optionValue;

optionName: ( identifier | L_PAREN typeName R_PAREN ) ( DOT optionName )*;

optionValue: scalarValue | messageLiteralWithBraces;

scalarValue : stringLiteral | intLiteral | floatLiteral |
                specialFloatLiteral | identifier;

intLiteral  : MINUS? INT_LITERAL;

floatLiteral: MINUS? FLOAT_LITERAL;

specialFloatLiteral: MINUS INF | MINUS NAN;

messageLiteralWithBraces: L_BRACE messageTextFormat R_BRACE;

messageTextFormat: ( messageLiteralField ( COMMA | SEMICOLON )? )*;

messageLiteralField: messageLiteralFieldName COLON value |
                       messageLiteralFieldName messageValue;

messageLiteralFieldName: fieldName |
                           L_BRACKET specialFieldName R_BRACKET;

specialFieldName       : extensionFieldName | typeURL;

extensionFieldName     : qualifiedIdentifier;

typeURL                : qualifiedIdentifier SLASH qualifiedIdentifier;

value         : scalarValue | messageLiteral | listLiteral;

messageValue  : messageLiteral | listOfMessagesLiteral;

messageLiteral: messageLiteralWithBraces |
                  L_ANGLE messageTextFormat R_ANGLE;

listLiteral: L_BRACKET ( listElement ( COMMA listElement )* )? R_BRACKET;

listElement: scalarValue | messageLiteral;

listOfMessagesLiteral: L_BRACKET ( messageLiteral ( COMMA messageLiteral )* )? R_BRACKET;

messageDecl: symbolVisibility? MESSAGE messageName L_BRACE messageElement* R_BRACE;

messageName   : identifier;

messageElement: messageFieldDecl |
                  groupDecl |
                  oneofDecl |
                  optionDecl |
                  extensionRangeDecl |
                  messageReservedDecl |
                  messageDecl |
                  enumDecl |
                  extensionDecl |
                  mapFieldDecl |
                  emptyDecl;

messageFieldDecl: fieldDeclWithCardinality |
                  messageFieldDeclTypeName fieldName EQUALS fieldNumber
                       compactOptions? SEMICOLON;

fieldDeclWithCardinality: fieldCardinality fieldDeclTypeName fieldName
                          EQUALS fieldNumber compactOptions? SEMICOLON;

fieldCardinality: REQUIRED | OPTIONAL | REPEATED;

fieldName       : identifier;

fieldNumber     : INT_LITERAL;

mapFieldDecl: mapType fieldName EQUALS fieldNumber compactOptions? SEMICOLON;

mapType   : MAP L_ANGLE mapKeyType COMMA typeName R_ANGLE;

mapKeyType:   INT32   | INT64   | UINT32   | UINT64   | SINT32 | SINT64 |
              FIXED32 | FIXED64 | SFIXED32 | SFIXED64 | BOOL   | STRING;

groupDecl: fieldCardinality? GROUP fieldName EQUALS fieldNumber
             compactOptions? L_BRACE messageElement* R_BRACE;

oneofDecl: ONEOF oneofName L_BRACE oneofElement* R_BRACE;

oneofName   : identifier;

oneofElement: optionDecl |
                oneofFieldDecl |
                oneofGroupDecl;

oneofFieldDecl: oneofFieldDeclTypeName fieldName EQUALS fieldNumber
                  compactOptions? SEMICOLON;

oneofGroupDecl: GROUP fieldName EQUALS fieldNumber
                  compactOptions? L_BRACE messageElement* R_BRACE;

extensionRangeDecl: EXTENSIONS tagRanges compactOptions? SEMICOLON;

tagRanges    : tagRange ( COMMA tagRange )*;

tagRange     : tagRangeStart ( TO tagRangeEnd )?;

tagRangeStart: fieldNumber;

tagRangeEnd  : fieldNumber | MAX;

messageReservedDecl: RESERVED ( tagRanges | nameStrings | names ) SEMICOLON;

nameStrings: stringLiteral ( COMMA stringLiteral )*;

names: identifier ( COMMA identifier )*;

enumDecl: symbolVisibility? ENUM enumName L_BRACE enumElement* R_BRACE;

enumName   : identifier;

enumElement: optionDecl |
               enumValueDecl |
               enumReservedDecl |
               emptyDecl;

enumValueDecl: enumValueName EQUALS enumValueNumber compactOptions? SEMICOLON;

enumValueName  : identifier;

enumValueNumber: MINUS? INT_LITERAL;

enumReservedDecl: RESERVED ( enumValueRanges | nameStrings | names ) SEMICOLON;

enumValueRanges    : enumValueRange ( COMMA enumValueRange )*;

enumValueRange     : enumValueRangeStart ( TO enumValueRangeEnd )?;

enumValueRangeStart: enumValueNumber;

enumValueRangeEnd  : enumValueNumber | MAX;

extensionDecl: EXTEND extendedMessage L_BRACE extensionElement* R_BRACE;

extendedMessage : typeName;

extensionElement: extensionFieldDecl |
                    groupDecl;

extensionFieldDecl: fieldDeclWithCardinality |
                    extensionFieldDeclTypeName fieldName EQUALS fieldNumber
                       compactOptions? SEMICOLON;

serviceDecl: SERVICE serviceName L_BRACE serviceElement* R_BRACE;

serviceName   : identifier;

serviceElement: optionDecl |
                  methodDecl |
                  emptyDecl;

methodDecl: RPC methodName inputType RETURNS outputType SEMICOLON |
              RPC methodName inputType RETURNS outputType L_BRACE methodElement* R_BRACE;

methodName   : identifier;

inputType    : messageType;

outputType   : messageType;

methodElement: optionDecl |
                emptyDecl;

messageType: L_PAREN STREAM? methodDeclTypeName R_PAREN;

identifier: alwaysIdent | sometimesIdent;

alwaysIdent: IDENTIFIER
    | SYNTAX
    | EDITION
    | IMPORT
    | WEAK
    | PUBLIC
    | PACKAGE
    | INF
    | NAN
    | BOOL
    | STRING
    | BYTES
    | FLOAT
    | DOUBLE
    | INT32
    | INT64
    | UINT32
    | UINT64
    | SINT32
    | SINT64
    | FIXED32
    | FIXED64
    | SFIXED32
    | SFIXED64
    | MAP
    | TO
    | MAX
    | SERVICE
    | RPC
    | RETURNS
    | EXPORT
    | LOCAL;

sometimesIdent: MESSAGE
    | ENUM
    | ONEOF
    | RESERVED
    | EXTENSIONS
    | EXTEND
    | OPTION
    | OPTIONAL
    | REQUIRED
    | REPEATED
    | GROUP
    | STREAM;

build.sh

#!/bin/sh

set -e

mkdir -p tmp
cd tmp
cp ../Protobuf*.g4 .
java -Xmx500M -cp "/usr/local/lib/antlr-4.10.1-complete.jar:$CLASSPATH" org.antlr.v4.Tool Protobuf*.g4
javac *.java

show_ast.sh

#!/bin/sh

set -e

if [[ ! -d ./tmp ]]; then
  ./build.sh
fi
cd tmp
java -Xmx500M -cp "/usr/local/lib/antlr-4.10.1-complete.jar:$CLASSPATH" org.antlr.v4.gui.TestRig Protobuf file -gui

example.proto

edition = "2024";

package test.users.v1;

import option "buf/validate/validate.proto";

option features.field_presence = IMPLICIT;

message User {
  uint64 uid = 1;

  export message Name {
    string last_name = 1;
    string first_name = 2;
    string middle_initial = 3;
  }
  Name name = 2;

  export message Address {
    string street_line1 = 1;
    string street_line2 = 2;
    string city = 3;
    string state_province = 4;
    string postal_code = 5;
    string country = 6;
  }
  Address address = 3;
}

service UserService {
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
  rpc DeleteUser(DeleteUserRequest) returns (DeleteUserResponse);
  rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse);
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
}

message CreateUserRequest {
  User.Name name = 1;
  User.Address address = 2;
}

message CreateUserResponse {
  uint64 uid = 1;
}

message DeleteUserRequest {
  uint64 uid = 1;
}

message DeleteUserResponse {
}

message UpdateUserRequest {
  User user = 1;
}

message UpdateUserResponse {
}

message GetUserRequest {
  uint64 uid = 1;
}

message GetUserResponse {
  User user = 1;
}

The parser that powers Buf

Another example worth examining is the actual parser that powers Buf. It uses YACC for Go (aka goyacc), so it includes inlined Go code for producing an AST. That configuration is in parser/proto.y. Lexical analysis in Buf uses a hand-written tokenizer in Go.