396 lines
14 KiB
Markdown
396 lines
14 KiB
Markdown
# gRPC Migration Plan
|
||
|
||
File: GRPC-MIGRATION-PLAN.md
|
||
Author: (Generated plan)
|
||
Status: Draft for review
|
||
Target Release: Next major version (breaking change – no mixed compatibility)
|
||
|
||
---
|
||
|
||
## 1. Overview
|
||
|
||
This document describes the full migration of the current custom TCP frame-based protocol (both the cart mutation/state channel on port `1337` and the control plane on port `1338`) to gRPC. We will remove all legacy packet framing (`FrameWithPayload`, `RemoteGrain`, `GenericListener` handlers for these two ports) and replace them with two gRPC services:
|
||
|
||
1. Cart Actor Service (mutations + state retrieval)
|
||
2. Control Plane Service (cluster membership, negotiation, ownership change, lifecycle)
|
||
|
||
We intentionally keep:
|
||
- Internal `CartGrain` logic, message storage format, disk persistence, and JSON cart serialization.
|
||
- Existing message type numeric mapping for backward compatibility with persisted event logs.
|
||
- HTTP/REST API layer unchanged (it still consumes JSON state from the local/remote grain pipeline).
|
||
|
||
We do NOT implement mixed-version compatibility; migration occurs atomically (cluster restart with new image).
|
||
|
||
---
|
||
|
||
## 2. Goals
|
||
|
||
- Remove custom binary frame protocol & simplify maintenance.
|
||
- Provide clearer, strongly defined interfaces via `.proto` schemas.
|
||
- Improve observability via gRPC interceptors (metrics & tracing hooks).
|
||
- Reduce per-call overhead compared with the current manual connection pooling + handwritten framing (HTTP/2 multiplexing + connection reuse).
|
||
- Prepare groundwork for future enhancements (streaming, typed state, event streaming) without rewriting again.
|
||
|
||
---
|
||
|
||
## 3. Non-Goals (Phase 1)
|
||
|
||
- Converting the cart state payload from JSON to a strongly typed proto.
|
||
- Introducing authentication / mTLS (may be added later).
|
||
- Changing persistence or replay format.
|
||
- Changing the HTTP API contract.
|
||
- Implementing streaming watchers or push updates.
|
||
|
||
---
|
||
|
||
## 4. Architecture After Migration
|
||
|
||
Ports:
|
||
- `:1337` → gRPC CartActor service.
|
||
- `:1338` → gRPC ControlPlane service.
|
||
|
||
Each node:
|
||
- Runs one gRPC server with both services (can use a single listener bound to two services or keep two separate listeners; we will keep two ports initially to minimize operational surprise, but they could be merged later).
|
||
- Maintains a connection pool of `*grpc.ClientConn` objects keyed by remote hostname (one per remote host, reused for both services).
|
||
|
||
Call Flow (Mutation):
|
||
1. HTTP request hits `PoolServer`.
|
||
2. `SyncedPool.getGrain(cartId)`:
|
||
- Local: direct invocation.
|
||
- Remote: uses `RemoteGrainGRPC` (new) which invokes `CartActor.Mutate`.
|
||
3. Response JSON returned unchanged.
|
||
|
||
Control Plane Flow:
|
||
- Discovery (K8s watch) still triggers `AddRemote(host)`.
|
||
- Instead of custom `Ping`, `Negotiate`, etc. via frames, call gRPC methods on `ControlPlane` service.
|
||
- Ownership changes use `ConfirmOwner` RPC.
|
||
|
||
---
|
||
|
||
## 5. Proto Design
|
||
|
||
### 5.1 Cart Actor Proto (Envelope Pattern)
|
||
|
||
We keep an envelope with `bytes payload` holding the serialized underlying cart mutation proto (existing types in `messages.proto`). This minimizes churn.
|
||
|
||
Indented code block (proto sketch):
|
||
|
||
syntax = "proto3";
|
||
package cart;
|
||
option go_package = "git.tornberg.me/go-cart-actor/proto;proto";
|
||
|
||
enum MutationType {
|
||
MUTATION_TYPE_UNSPECIFIED = 0;
|
||
MUTATION_ADD_REQUEST = 1;
|
||
MUTATION_ADD_ITEM = 2;
|
||
MUTATION_REMOVE_ITEM = 4;
|
||
MUTATION_REMOVE_DELIVERY = 5;
|
||
MUTATION_CHANGE_QUANTITY = 6;
|
||
MUTATION_SET_DELIVERY = 7;
|
||
MUTATION_SET_PICKUP_POINT = 8;
|
||
MUTATION_CREATE_CHECKOUT_ORDER = 9;
|
||
MUTATION_SET_CART_ITEMS = 10;
|
||
MUTATION_ORDER_COMPLETED = 11;
|
||
}
|
||
|
||
message MutationRequest {
|
||
string cart_id = 1;
|
||
MutationType type = 2;
|
||
bytes payload = 3; // Serialized specific mutation proto
|
||
int64 client_timestamp = 4; // Optional; server fills if zero
|
||
}
|
||
|
||
message MutationReply {
|
||
int32 status_code = 1;
|
||
bytes payload = 2; // JSON cart state or error string
|
||
}
|
||
|
||
message StateRequest {
|
||
string cart_id = 1;
|
||
}
|
||
|
||
message StateReply {
|
||
int32 status_code = 1;
|
||
bytes payload = 2; // JSON cart state
|
||
}
|
||
|
||
service CartActor {
|
||
rpc Mutate(MutationRequest) returns (MutationReply);
|
||
rpc GetState(StateRequest) returns (StateReply);
|
||
}
|
||
|
||
### 5.2 Control Plane Proto
|
||
|
||
syntax = "proto3";
|
||
package control;
|
||
option go_package = "git.tornberg.me/go-cart-actor/proto;proto";
|
||
|
||
message Empty {}
|
||
|
||
message PingReply {
|
||
string host = 1;
|
||
int64 unix_time = 2;
|
||
}
|
||
|
||
message NegotiateRequest {
|
||
repeated string known_hosts = 1;
|
||
}
|
||
message NegotiateReply {
|
||
repeated string hosts = 1; // Healthy hosts returned
|
||
}
|
||
|
||
message CartIdsReply {
|
||
repeated string cart_ids = 1;
|
||
}
|
||
|
||
message OwnerChangeRequest {
|
||
string cart_id = 1;
|
||
string new_host = 2;
|
||
}
|
||
message OwnerChangeAck {
|
||
bool accepted = 1;
|
||
string message = 2;
|
||
}
|
||
|
||
message ClosingNotice {
|
||
string host = 1;
|
||
}
|
||
|
||
service ControlPlane {
|
||
rpc Ping(Empty) returns (PingReply);
|
||
rpc Negotiate(NegotiateRequest) returns (NegotiateReply);
|
||
rpc GetCartIds(Empty) returns (CartIdsReply);
|
||
rpc ConfirmOwner(OwnerChangeRequest) returns (OwnerChangeAck);
|
||
rpc Closing(ClosingNotice) returns (OwnerChangeAck);
|
||
}
|
||
|
||
---
|
||
|
||
## 6. Message Type Mapping
|
||
|
||
| Legacy Constant | Numeric | New Enum Value |
|
||
|-----------------|---------|-----------------------------|
|
||
| AddRequestType | 1 | MUTATION_ADD_REQUEST |
|
||
| AddItemType | 2 | MUTATION_ADD_ITEM |
|
||
| RemoveItemType | 4 | MUTATION_REMOVE_ITEM |
|
||
| RemoveDeliveryType | 5 | MUTATION_REMOVE_DELIVERY |
|
||
| ChangeQuantityType | 6 | MUTATION_CHANGE_QUANTITY |
|
||
| SetDeliveryType | 7 | MUTATION_SET_DELIVERY |
|
||
| SetPickupPointType | 8 | MUTATION_SET_PICKUP_POINT |
|
||
| CreateCheckoutOrderType | 9 | MUTATION_CREATE_CHECKOUT_ORDER |
|
||
| SetCartItemsType | 10 | MUTATION_SET_CART_ITEMS |
|
||
| OrderCompletedType | 11 | MUTATION_ORDER_COMPLETED |
|
||
|
||
Persisted events keep original numeric codes; reconstruction simply casts to `MutationType`.
|
||
|
||
---
|
||
|
||
## 7. Components To Remove / Replace
|
||
|
||
Remove (after migration complete):
|
||
- `remote-grain.go`
|
||
- `rpc-server.go`
|
||
- Any packet/frame-specific types solely used by the above (search: `FrameWithPayload`, `RemoteHandleMutation`, `RemoteGetState` where not reused by disk or internal logic).
|
||
- The constants representing network frame types in `synced-pool.go` (RemoteNegotiate, AckChange, etc.) replaced by gRPC calls.
|
||
- netpool usage for remote cart channel (control plane also no longer needs `Connection` abstraction).
|
||
|
||
Retain (until reworked or optionally cleaned later):
|
||
- `message.go` (for persistence)
|
||
- `message-handler.go`
|
||
- `cart-grain.go`
|
||
- `messages.proto` (underlying mutation messages)
|
||
- HTTP API server and REST handlers.
|
||
|
||
---
|
||
|
||
## 8. New / Modified Components
|
||
|
||
New files (planned):
|
||
- `proto/cart_actor.proto`
|
||
- `proto/control_plane.proto`
|
||
- `grpc/cart_actor_server.go` (server impl)
|
||
- `grpc/cart_actor_client.go` (client adapter implementing `Grain`)
|
||
- `grpc/control_plane_server.go`
|
||
- `grpc/control_plane_client.go`
|
||
- `grpc/interceptors.go` (metrics, logging, optional tracing hooks)
|
||
- `remote_grain_grpc.go` (adapter bridging existing interfaces)
|
||
- `control_plane_adapter.go` (replaces frame handlers in `SyncedPool`)
|
||
|
||
Modified:
|
||
- `synced-pool.go` (remote host management now uses gRPC clients; negotiation logic updated)
|
||
- `main.go` (initialize both gRPC services on startup)
|
||
- `go.mod` (add `google.golang.org/grpc`)
|
||
|
||
---
|
||
|
||
## 9. Step-by-Step Migration Plan
|
||
|
||
1. Add proto files and generate Go code (`protoc --go_out --go-grpc_out`).
|
||
2. Implement `CartActorServer`:
|
||
- Translate `MutationRequest` to `Message`.
|
||
- Use existing handler registry for payload encode/decode.
|
||
- Return JSON cart state.
|
||
3. Implement `CartActorClient` wrapper (`RemoteGrainGRPC`) implementing:
|
||
- `HandleMessage`: Build envelope, call `Mutate`.
|
||
- `GetCurrentState`: Call `GetState`.
|
||
4. Implement `ControlPlaneServer` with methods:
|
||
- `Ping`: returns host + time.
|
||
- `Negotiate`: merge host lists; emulate old logic.
|
||
- `GetCartIds`: iterate local grains.
|
||
- `ConfirmOwner`: replicate quorum flow (accept always; error path for future).
|
||
- `Closing`: schedule remote removal.
|
||
5. Implement `ControlPlaneClient` used inside `SyncedPool.AddRemote`.
|
||
6. Refactor `SyncedPool`:
|
||
- Replace frame handlers registration with gRPC client calls.
|
||
- Replace `Server.AddHandler(...)` start-up with launching gRPC server.
|
||
- Implement periodic health checks using `Ping`.
|
||
7. Remove old connection constructs for 1337/1338.
|
||
8. Metrics:
|
||
- Add unary interceptor capturing duration and status.
|
||
- Replace packet counters with `cart_grpc_mutate_calls_total`, `cart_grpc_control_calls_total`, histograms for latency.
|
||
9. Update `main.go` to start:
|
||
- gRPC server(s).
|
||
- HTTP server as before.
|
||
10. Delete legacy files & update README build instructions.
|
||
11. Load testing & profiling on Raspberry Pi hardware (or ARM emulation).
|
||
12. Final cleanup & dead code removal (search for now-unused constants & structs).
|
||
13. Tag release.
|
||
|
||
---
|
||
|
||
## 10. Performance Considerations (Raspberry Pi Focus)
|
||
|
||
- Single `*grpc.ClientConn` per remote host (HTTP/2 multiplexing) to reduce file descriptor and handshake overhead.
|
||
- Use small keepalive pings (optional) only if connections drop; default may suffice.
|
||
- Avoid reflection / dynamic dispatch in hot path: pre-build a mapping from `MutationType` to handler function.
|
||
- Reuse byte buffers:
|
||
- Implement a `sync.Pool` for mutation serialization to reduce GC pressure.
|
||
- Enforce per-RPC deadlines (e.g. 300–400ms) to avoid pile-ups.
|
||
- Backpressure:
|
||
- Before dispatch: if local grain pool at capacity and target grain is remote, abort early with 503 to caller (optional).
|
||
- Disable gRPC compression for small payloads (mutation messages are small). Condition compression if payload > threshold (e.g. 8KB).
|
||
- Compile with `-ldflags="-s -w"` in production to reduce binary size (optional).
|
||
- Enable `GOMAXPROCS` tuned to CPU cores; Pi often benefits from leaving default but monitor.
|
||
- Use histograms with limited buckets to reduce Prometheus cardinality.
|
||
|
||
---
|
||
|
||
## 11. Testing Strategy
|
||
|
||
Unit:
|
||
- Message type mapping tests (legacy -> enum).
|
||
- Envelope roundtrip: Original proto -> payload -> gRPC -> server decode -> internal Message.
|
||
|
||
Integration:
|
||
- Two-node cluster simulation:
|
||
- Mutate cart on Node A, ownership moves, verify remote access from Node B.
|
||
- Quorum failure simulation (temporarily reject `ConfirmOwner`).
|
||
- Control plane negotiation: start nodes in staggered order, assert final membership.
|
||
|
||
Load/Perf:
|
||
- Benchmark local mutation vs remote mutation latency.
|
||
- High concurrency test (N goroutines each performing X mutations).
|
||
- Memory profiling (ensure no large buffer retention).
|
||
|
||
Failure Injection:
|
||
- Kill a node mid-mutation; client call should timeout and not corrupt local state.
|
||
- Simulated network partition: drop `Ping` replies; ensure host removal path triggers.
|
||
|
||
---
|
||
|
||
## 12. Rollback Strategy
|
||
|
||
Because no mixed-version compatibility is provided, rollback = redeploy previous version containing legacy protocol:
|
||
1. Stop all new-version pods.
|
||
2. Deploy old version cluster-wide.
|
||
3. No data migration needed (event persistence unaffected).
|
||
|
||
Note: Avoid partial upgrades; perform full rolling restart quickly to prevent split-brain (new nodes won’t talk to old nodes).
|
||
|
||
---
|
||
|
||
## 13. Risks & Mitigations
|
||
|
||
| Risk | Description | Mitigation |
|
||
|------|-------------|------------|
|
||
| Full-cluster restart required | No mixed compatibility | Schedule maintenance window |
|
||
| gRPC adds CPU overhead | Envelope + marshaling cost | Buffer reuse, keep small messages uncompressed |
|
||
| Ownership race | Timing differences after refactor | Add explicit logs + tests around `RequestOwnership` path |
|
||
| Hidden dependency on frame-level status codes | Some code may assume `FrameWithPayload` fields | Wrap gRPC responses into minimal compatibility structs until fully removed |
|
||
| Memory growth | Connection reuse & pooled buffers not implemented initially | Add `sync.Pool` & track memory via pprof early |
|
||
|
||
---
|
||
|
||
## 14. Logging & Observability
|
||
|
||
- Structured log entries for:
|
||
- Ownership changes
|
||
- Negotiation rounds
|
||
- Remote spawn events
|
||
- Mutation failures (with cart id, mutation type)
|
||
- Metrics:
|
||
- `cart_grpc_mutate_duration_seconds` (histogram)
|
||
- `cart_grpc_mutate_errors_total`
|
||
- `cart_grpc_control_duration_seconds`
|
||
- `cart_remote_hosts` (gauge)
|
||
- Retain existing grain counts.
|
||
- Optional future: OpenTelemetry tracing (span per remote mutation).
|
||
|
||
---
|
||
|
||
## 15. Future Enhancements (Post-Migration)
|
||
|
||
- Replace JSON state with `CartState` proto and provide streaming watch API.
|
||
- mTLS between nodes (certificate rotation via K8s Secret or SPIRE).
|
||
- Distributed tracing integration.
|
||
- Ownership leasing with TTL and optimistic renewal.
|
||
- Delta replication or CRDT-based conflict resolution for experimentation.
|
||
|
||
---
|
||
|
||
## 16. Task Breakdown & Estimates
|
||
|
||
| Task | Estimate |
|
||
|------|----------|
|
||
| Proto definitions & generation | 0.5d |
|
||
| CartActor server/client | 1.0d |
|
||
| ControlPlane server/client | 1.0d |
|
||
| SyncedPool refactor | 1.0d |
|
||
| Metrics & interceptors | 0.5d |
|
||
| Remove legacy code & cleanup | 0.5d |
|
||
| Tests (unit + integration) | 1.5d |
|
||
| Benchmark & tuning | 0.5–1.0d |
|
||
| Total | ~6–7d |
|
||
|
||
---
|
||
|
||
## 17. Open Questions (Confirm Before Implementation)
|
||
|
||
1. Combine both services on a single port (simplify ops) or keep dual-port first? (Default here: keep dual, but easy to merge.)
|
||
2. Minimum Go version remains 1.24.x—acceptable to add `google.golang.org/grpc` latest?
|
||
3. Accept adding `sync.Pool` micro-optimizations in first pass or postpone?
|
||
|
||
---
|
||
|
||
## 18. Acceptance Criteria
|
||
|
||
- All previous integration tests (adjusted to gRPC) pass.
|
||
- Cart operations (add, remove, delivery, checkout) function across at least a 2‑node cluster.
|
||
- Control plane negotiation forms consistent host list.
|
||
- Latency for a remote mutation does not degrade beyond an acceptable threshold (define baseline before merge).
|
||
- Legacy networking code fully removed.
|
||
|
||
---
|
||
|
||
## 19. Next Steps (If Approved)
|
||
|
||
1. Implement proto files and commit.
|
||
2. Scaffold server & client code.
|
||
3. Refactor `SyncedPool` and `main.go`.
|
||
4. Add metrics and tests.
|
||
5. Run benchmark on target Pi hardware.
|
||
6. Review & merge.
|
||
|
||
---
|
||
|
||
End of Plan. |