Protobuf Tip #10: Choosing the right integer type

Use int64.

If you have been adding integer fields to your schemas by typing int64 and moving on with your day, you have been doing it right. You can stop reading here.

Protobuf does have ten different integer types, so it is reasonable to wonder whether picking the wrong one matters. Mostly, it doesn’t. But there are a few differences worth knowing about.

So many types

Protobuf’s integer types fall into three families, and the family determines how the value is encoded on the wire:

The 1-to-10 range mostly applies to the 64-bit varints. uint32 and sint32 cap out at 5 bytes; int32 is the odd one out because negative values can still take 10.

int32, int64, uint32, and uint64 all use varints. Seven bits of each byte hold the value and the remaining continuation bit says whether another byte follows. Values under 128 fit in a single byte. Larger values need more bytes and a few more trips through the decoding loop. The uint variants use the same wire encoding, but give you an unsigned range instead.

Plain varints get ugly with negative numbers. A negative int64 always takes 10 bytes, even if the value is -1. A negative int32 also takes 10 bytes, even though the type only holds 4 bytes, because the encoder widens the value to 64 bits first and fills all the added bits with ones.

sint32 and sint64 avoid this strange behavior with ZigZag encoding, which maps signed integers to unsigned values with the sign encoded in the low bit. Small negative numbers then stay small on the wire, so -1 takes one byte instead of ten. I tried to save you from this wire-format Protobuf trivia, but you insisted on reading further.

Fixed-size integers skip the continuation-bit nonsense. The value is always four or eight little-endian bytes. No continuation bits, no varint loop.

Varints do a little more work to save bytes. Fixed-width integers use more space so the parser can do less work. But how much CPU are we actually talking about?

But it doesn’t matter

I wanted to put some numbers on that “little more work,” so I benchmarked fields containing 1,000 integers using the standard google.golang.org/protobuf runtime. I benchmarked both marshal and unmarshal with small and large positive values, plus negative values for the signed types. The chart below sticks to the 64-bit types; the 32-bit variants use the same encodings over a smaller range.

A thousand packed integers in one field is already a pretty extreme case. A message with three scalar fields would show even less difference.

I also ran the same 1,000 integers through encoding/json and encoding/json/v2. JSON isn’t really the competition here. It just gives the chart a familiar reference point for how small these numbers are.

Below is unmarshal time, with all three groups drawn on one shared scale. Marshal numbers are in the appendix and tell the same story.

The slowest case is under 6 microseconds for 1,000 values. sfixed64 is almost five times faster than negative int64, but we’re talking about 1.2 microseconds versus 5.8 microseconds. Even the JSON cases top out at 32 microseconds. These are very small numbers.

One exception is a 64-bit value that’s basically random, like a hash or generated ID. Varints don’t buy you much there because nearly every value is large. fixed64 uses eight bytes every time, so it can actually be both smaller and cheaper to decode.

Just use int64 and move on with your life

Outside of that, my default answer is still int64. It handles negatives, keeps small values small on the wire, and is very unlikely to be the reason your service is slow.

If your schemas are already full of int64, leave them alone. When you add the next integer field, int64 is still where I’d start.

syntax = "proto3";
 
package bench.v1;
 
message Int64List {
  repeated int64 values = 1;
}
 
message Sint64List {
  repeated sint64 values = 1;
}
 
message Sfixed64List {
  repeated sfixed64 values = 1;
}

func benchmarkUnmarshal(b *testing.B, msg proto.Message) {
    payload, err := proto.Marshal(msg)
    if err != nil {
        b.Fatal(err)
    }
 
    dst := msg.ProtoReflect().Type().New().Interface()
    b.SetBytes(int64(len(payload)))
    b.ReportAllocs()
    b.ResetTimer()
 
    for b.Loop() {
        proto.Reset(dst)
        if err := proto.Unmarshal(payload, dst); err != nil {
            b.Fatal(err)
        }
    }
}

type jsonList struct {
    Values []int64 `json:"values"`
}

GOEXPERIMENT=jsonv2 go test -run='^$' -bench=. -benchmem -benchtime=5s -count=5