Controller unable to start: pgid (######) above high water mark
Scenario
An error seen in the controller logs and blocks it starting up:
2022-01-03 03:06:33.953508 E | etcdserver: cannot monitor file descriptor usage (cannot get FDUsage on darwin)
2022-01-03 03:06:33.953688 I | etcdserver: a51704efaf54bd03 as single-node; fast-forwarding 9 ticks (election ticks 10)
panic: pgid (8388923) above high water mark (3340)
goroutine 132 [running]:
github.com/coreos/bbolt.(*node).spill(0xc001240d90, 0xc002b57ba0, 0x5)
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/bbolt/node.go:375 +0x6a5
github.com/coreos/bbolt.(*Bucket).spill(0xc00126a558, 0xe1a29b8, 0x3f1a9a0)
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/bbolt/bucket.go:568 +0x473
github.com/coreos/bbolt.(*Tx).Commit(0xc00126a540, 0xe19d8e0, 0x3f1a9a0)
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/bbolt/tx.go:160 +0xec
github.com/coreos/etcd/mvcc/backend.(*batchTx).commit(0xc001421230, 0x0)
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/etcd/mvcc/backend/batch_tx.go:167 +0x7c
github.com/coreos/etcd/mvcc/backend.(*batchTxBuffered).unsafeCommit(0xc001421230, 0xc001314700)
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/etcd/mvcc/backend/batch_tx.go:239 +0x49
github.com/coreos/etcd/mvcc/backend.(*batchTxBuffered).commit(0xc001421230, 0xc001314700)
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/etcd/mvcc/backend/batch_tx.go:227 +0x4c
github.com/coreos/etcd/mvcc/backend.(*batchTxBuffered).Commit(0xc001421230)
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/etcd/mvcc/backend/batch_tx.go:214 +0x42
github.com/coreos/etcd/mvcc/backend.(*backend).run(0xc001442150)
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/etcd/mvcc/backend/backend.go:273 +0xb9
created by github.com/coreos/etcd/mvcc/backend.newBackend
/Users/anka/workspace/cloud-build_release_v1.18.0/src/github.com/coreos/etcd/mvcc/backend/backend.go:161 +0x267
Common Causes
- Disk exhaustion caused corruption
- An etcd bug
Solution
You’ll want to delete the entire etcd data directory. This directory is under /Library/Application\ Support/Veertu/Anka/anka-controller for the native macOS package, or in the location you specify in your docker configuration.