This commit is contained in:
@@ -17,7 +17,7 @@ go test -run TestRouteLookup ./server/ # Run a single test
|
|||||||
docker build -t mc-router . # Build Docker image
|
docker build -t mc-router . # Build Docker image
|
||||||
```
|
```
|
||||||
|
|
||||||
Go version: 1.25. Testing uses `testify` (assert/require). Tests are table-driven.
|
Go version: 1.26.2. Testing uses `testify` (assert/require). Tests are table-driven with subtests. Mock pattern: embed `mock.Mock` and call `m.MethodCalled()` (see `k8s_test.go`). Protocol packet tests use hex fixture files in `testdata/` (e.g., `handshake-status.hex`). Test setup for route tests calls `NewRoutes()` and restores the global `Routes` singleton with defer.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
@@ -39,7 +39,8 @@ Go version: 1.25. Testing uses `testify` (assert/require). Tests are table-drive
|
|||||||
- `routes.go` — In-memory route table mapping server addresses to backends; supports default route fallback
|
- `routes.go` — In-memory route table mapping server addresses to backends; supports default route fallback
|
||||||
- `routes_config_loader.go` — Loads/watches JSON routes config file (with fsnotify)
|
- `routes_config_loader.go` — Loads/watches JSON routes config file (with fsnotify)
|
||||||
- `k8s.go` — Kubernetes service discovery via annotation `mc-router.itzg.me/externalServerName`
|
- `k8s.go` — Kubernetes service discovery via annotation `mc-router.itzg.me/externalServerName`
|
||||||
- `docker.go` / `docker_swarm.go` — Docker/Swarm container discovery via label `mc-router.host`
|
- `docker.go` — Docker container discovery via label `mc-router.host`; event-driven via Docker Events API (`client.Events`). Each event handled incrementally by `applyEvent` → `containersForID` (single `ContainerInspect`) → `applyContainerRoutesLocked` (touches only that container's routes). Full `monitorContainers` re-list runs at startup and on event-stream reconnect (exponential backoff)
|
||||||
|
- `docker_swarm.go` — Docker Swarm service discovery via label `mc-router.host`; event-driven, but each service event triggers a full `reconcileServices` re-list (services churn rarely, swarm has no autoscaling)
|
||||||
- `down_scaler.go` — Auto-scale down after idle period
|
- `down_scaler.go` — Auto-scale down after idle period
|
||||||
- `api_server.go` — REST API (`GET/POST /routes`, `POST /defaultRoute`, `DELETE /routes/{serverAddress}`)
|
- `api_server.go` — REST API (`GET/POST /routes`, `POST /defaultRoute`, `DELETE /routes/{serverAddress}`)
|
||||||
- `metrics.go` — Pluggable metrics backends (Prometheus, InfluxDB, expvar, discard)
|
- `metrics.go` — Pluggable metrics backends (Prometheus, InfluxDB, expvar, discard)
|
||||||
@@ -60,7 +61,7 @@ CLI flags are the primary config mechanism, with environment variable support vi
|
|||||||
Routes are populated from three sources that can be combined:
|
Routes are populated from three sources that can be combined:
|
||||||
1. Static `--mapping` flags or JSON config file
|
1. Static `--mapping` flags or JSON config file
|
||||||
2. Kubernetes: watches Services with `mc-router.itzg.me/externalServerName` annotation
|
2. Kubernetes: watches Services with `mc-router.itzg.me/externalServerName` annotation
|
||||||
3. Docker/Swarm: watches containers with `mc-router.host` label
|
3. Docker/Swarm: watches containers/services via the Docker Events API, filtered to lifecycle events (label `mc-router.host`)
|
||||||
|
|
||||||
### Key Dependencies
|
### Key Dependencies
|
||||||
|
|
||||||
@@ -72,6 +73,21 @@ Routes are populated from three sources that can be combined:
|
|||||||
- `golang.ngrok.com/ngrok` — ngrok tunnel integration
|
- `golang.ngrok.com/ngrok` — ngrok tunnel integration
|
||||||
- `github.com/stretchr/testify` — Test assertions
|
- `github.com/stretchr/testify` — Test assertions
|
||||||
|
|
||||||
|
### Concurrency Model
|
||||||
|
|
||||||
|
- **`routes.go`**: global singleton `var Routes = NewRoutes()`. `sync.RWMutex` protects `mappings` and `defaultRoute` — `RLock` for all reads, `Lock` for mutations.
|
||||||
|
- **`connector.go`**: `ActiveConnections` map guarded by `sync.RWMutex`; `totalActiveConnections` counter uses `atomic.AddInt32`. Shutdown drain uses `sync.Cond` in `WaitForConnections()`.
|
||||||
|
- **Bidirectional proxy**: two goroutines per connection (client→backend, backend→client) communicate via a buffered `chan error` (size 2) — first error triggers mutual close.
|
||||||
|
- All goroutines respect context cancellation via `select { case <-ctx.Done() }`.
|
||||||
|
|
||||||
|
### Error Handling
|
||||||
|
|
||||||
|
- Wrap with context: `fmt.Errorf("message: %w", err)`
|
||||||
|
- Check specific sentinels: `errors.Is(err, io.EOF)`
|
||||||
|
- Log with fields: `logrus.WithError(err).WithField("key", val).Error("msg")`
|
||||||
|
|
||||||
### Protocol Notes
|
### Protocol Notes
|
||||||
|
|
||||||
The `mcproto` package handles Minecraft Java protocol quirks: Forge mod identifiers appended to server addresses (separated by `\x00`), DNS root zone trailing dots, legacy server list ping format, and VarInt encoding. Server address matching in routes strips these suffixes before lookup.
|
The `mcproto` package handles Minecraft Java protocol quirks: Forge mod identifiers appended to server addresses (separated by `\x00`), DNS root zone trailing dots, legacy server list ping format, and VarInt encoding. Server address matching in routes also strips TCP Shield patterns (`///...`) and lowercases before lookup.
|
||||||
|
|
||||||
|
Auto-scale MOTD fallback: waking servers display `LoadingMOTD`; sleeping servers display `AsleepMOTD`. Per-route MOTDs take precedence over global ones.
|
||||||
|
|||||||
@@ -50,8 +50,6 @@ Some other features included:
|
|||||||
host:port of a default Minecraft server to use when mapping not found (env DEFAULT)
|
host:port of a default Minecraft server to use when mapping not found (env DEFAULT)
|
||||||
-docker-api-version string
|
-docker-api-version string
|
||||||
Instead of auto-negotiating, use specific Docker API version (env DOCKER_API_VERSION)
|
Instead of auto-negotiating, use specific Docker API version (env DOCKER_API_VERSION)
|
||||||
-docker-refresh-interval int
|
|
||||||
Refresh interval in seconds for the Docker integrations (env DOCKER_REFRESH_INTERVAL) (default 15)
|
|
||||||
-docker-socket string
|
-docker-socket string
|
||||||
Path to Docker socket to use (env DOCKER_SOCKET) (default "unix:///var/run/docker.sock")
|
Path to Docker socket to use (env DOCKER_SOCKET) (default "unix:///var/run/docker.sock")
|
||||||
-docker-timeout int
|
-docker-timeout int
|
||||||
@@ -171,7 +169,7 @@ To test out this example, add these two entries to my "hosts" file:
|
|||||||
|
|
||||||
### Using Docker auto-discovery
|
### Using Docker auto-discovery
|
||||||
|
|
||||||
When running `mc-router` in a Docker environment you can pass the `--in-docker` or `--in-docker-swarm` command-line argument or set the environment variables `IN_DOCKER` or `IN_DOCKER_SWARM` to "true". With that, it will poll the Docker API periodically to find all the running containers/services for Minecraft instances. To enable discovery, you have to set the `mc-router.host` label on the container.
|
When running `mc-router` in a Docker environment you can pass the `--in-docker` or `--in-docker-swarm` command-line argument or set the environment variables `IN_DOCKER` or `IN_DOCKER_SWARM` to "true". With that, it will subscribe to the Docker event stream to react to container/service lifecycle changes (start, stop, pause, unpause, rename, network connect/disconnect for containers; create, update, remove for swarm services) and update routes immediately. An initial listing is performed on startup, and the stream is reconnected with exponential backoff on errors (e.g. daemon restart). To enable discovery, you have to set the `mc-router.host` label on the container.
|
||||||
|
|
||||||
When using in Docker, make sure to volume mount the Docker socket into the container, such as
|
When using in Docker, make sure to volume mount the Docker socket into the container, such as
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -40,7 +40,7 @@ type Config struct {
|
|||||||
InDockerSwarm bool `usage:"Use Docker Swarm service discovery"`
|
InDockerSwarm bool `usage:"Use Docker Swarm service discovery"`
|
||||||
DockerSocket string `usage:"Path to Docker socket to use"`
|
DockerSocket string `usage:"Path to Docker socket to use"`
|
||||||
DockerTimeout time.Duration `usage:"Timeout (as duration) for the Docker integrations"`
|
DockerTimeout time.Duration `usage:"Timeout (as duration) for the Docker integrations"`
|
||||||
DockerRefreshInterval time.Duration `default:"15s" usage:"Refresh interval (as duration) for the Docker integrations"`
|
DockerRefreshInterval time.Duration `usage:"Deprecated and ignored: Docker discovery is now event-driven"`
|
||||||
DockerApiVersion string `usage:"Instead of auto-negotiating, use specific Docker API version"`
|
DockerApiVersion string `usage:"Instead of auto-negotiating, use specific Docker API version"`
|
||||||
MetricsBackend string `default:"discard" usage:"Backend to use for metrics exposure/publishing: discard,expvar,influxdb,prometheus"`
|
MetricsBackend string `default:"discard" usage:"Backend to use for metrics exposure/publishing: discard,expvar,influxdb,prometheus"`
|
||||||
MetricsBackendConfig MetricsBackendConfig
|
MetricsBackendConfig MetricsBackendConfig
|
||||||
|
|||||||
+240
-71
@@ -9,7 +9,10 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
cerrdefs "github.com/containerd/errdefs"
|
||||||
"github.com/docker/docker/api/types/container"
|
"github.com/docker/docker/api/types/container"
|
||||||
|
"github.com/docker/docker/api/types/events"
|
||||||
|
"github.com/docker/docker/api/types/filters"
|
||||||
"github.com/docker/docker/client"
|
"github.com/docker/docker/client"
|
||||||
"github.com/sirupsen/logrus"
|
"github.com/sirupsen/logrus"
|
||||||
)
|
)
|
||||||
@@ -34,7 +37,6 @@ type dockerWatcherConfig struct {
|
|||||||
autoScaleDown bool
|
autoScaleDown bool
|
||||||
socket string
|
socket string
|
||||||
timeout time.Duration
|
timeout time.Duration
|
||||||
refreshInterval time.Duration
|
|
||||||
apiVersion string
|
apiVersion string
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -48,12 +50,11 @@ func (c *dockerWatcherConfig) apiVersionOpt() client.Opt {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewDockerWatcher(socket string, timeout time.Duration, refreshInterval time.Duration, autoScaleUp bool, autoScaleDown bool, dockerApiVersion string) IDockerWatcher {
|
func NewDockerWatcher(socket string, timeout time.Duration, autoScaleUp bool, autoScaleDown bool, dockerApiVersion string) IDockerWatcher {
|
||||||
return &dockerWatcherImpl{
|
return &dockerWatcherImpl{
|
||||||
config: dockerWatcherConfig{
|
config: dockerWatcherConfig{
|
||||||
socket: socket,
|
socket: socket,
|
||||||
timeout: timeout,
|
timeout: timeout,
|
||||||
refreshInterval: refreshInterval,
|
|
||||||
autoScaleUp: autoScaleUp,
|
autoScaleUp: autoScaleUp,
|
||||||
autoScaleDown: autoScaleDown,
|
autoScaleDown: autoScaleDown,
|
||||||
apiVersion: dockerApiVersion,
|
apiVersion: dockerApiVersion,
|
||||||
@@ -111,12 +112,7 @@ func (w *dockerWatcherImpl) makeWakerFunc(rc *routableContainer) WakerFunc {
|
|||||||
}
|
}
|
||||||
endpoint := net.JoinHostPort(data.ip, strconv.Itoa(int(data.port)))
|
endpoint := net.JoinHostPort(data.ip, strconv.Itoa(int(data.port)))
|
||||||
|
|
||||||
// Update the route mappings
|
// Route table updates via Docker `start`/`network connect` events.
|
||||||
err = w.monitorContainers(ctx)
|
|
||||||
if err != nil {
|
|
||||||
logrus.WithError(err).Error("Docker monitoring failed")
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Wait until the container is reachable
|
// Wait until the container is reachable
|
||||||
deadline := time.Now().Add(60 * time.Second)
|
deadline := time.Now().Add(60 * time.Second)
|
||||||
@@ -164,15 +160,15 @@ func (w *dockerWatcherImpl) makeSleeperFunc(rc *routableContainer) SleeperFunc {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
err = w.monitorContainers(ctx)
|
// Route table updates via Docker `die`/`stop`/`network disconnect` events.
|
||||||
if err != nil {
|
|
||||||
logrus.WithError(err).Error("Docker monitoring failed")
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// monitorContainers does a full re-list of Docker containers and reconciles
|
||||||
|
// the route table against it. Used for initial sync at startup and for
|
||||||
|
// resync after the event stream reconnects (to catch any events missed
|
||||||
|
// during disconnect).
|
||||||
func (w *dockerWatcherImpl) monitorContainers(ctx context.Context) error {
|
func (w *dockerWatcherImpl) monitorContainers(ctx context.Context) error {
|
||||||
w.monitorLock.Lock()
|
w.monitorLock.Lock()
|
||||||
defer w.monitorLock.Unlock()
|
defer w.monitorLock.Unlock()
|
||||||
@@ -184,11 +180,147 @@ func (w *dockerWatcherImpl) monitorContainers(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
visited := map[string]struct{}{}
|
byID := map[string][]*routableContainer{}
|
||||||
for _, rs := range containers {
|
for _, rc := range containers {
|
||||||
if oldRs, ok := w.containerMap[rs.externalContainerName]; !ok {
|
byID[rc.containerID] = append(byID[rc.containerID], rc)
|
||||||
|
}
|
||||||
|
|
||||||
|
for id, desired := range byID {
|
||||||
|
w.applyContainerRoutesLocked(id, desired)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Remove entries whose container is no longer present at all
|
||||||
|
for name, rc := range w.containerMap {
|
||||||
|
if _, present := byID[rc.containerID]; present {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
delete(w.containerMap, name)
|
||||||
|
if name != "" {
|
||||||
|
Routes.DeleteMapping(name)
|
||||||
|
} else {
|
||||||
|
Routes.SetDefaultRoute("", "", nil, nil, "", "")
|
||||||
|
}
|
||||||
|
logrus.WithField("routableContainer", rc).Debug("DELETE")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// applyEvent reacts to a single Docker event by reconciling only the routes
|
||||||
|
// belonging to the affected container — no full re-list.
|
||||||
|
func (w *dockerWatcherImpl) applyEvent(ctx context.Context, ev events.Message) error {
|
||||||
|
containerID := ev.Actor.ID
|
||||||
|
if ev.Type == events.NetworkEventType {
|
||||||
|
containerID = ev.Actor.Attributes["container"]
|
||||||
|
}
|
||||||
|
if containerID == "" {
|
||||||
|
logrus.WithField("event", ev).Warn("network event missing container attribute, skipping")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var desired []*routableContainer
|
||||||
|
if !(ev.Type == events.ContainerEventType && ev.Action == events.ActionDestroy) {
|
||||||
|
got, err := w.containersForID(ctx, containerID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
desired = got
|
||||||
|
}
|
||||||
|
|
||||||
|
w.monitorLock.Lock()
|
||||||
|
defer w.monitorLock.Unlock()
|
||||||
|
|
||||||
|
// Only trace events that affect a routed container — either one we already
|
||||||
|
// track or one becoming routable now. Filters out unrelated daemon noise.
|
||||||
|
relevant := len(desired) > 0
|
||||||
|
if !relevant {
|
||||||
|
for _, rc := range w.containerMap {
|
||||||
|
if rc.containerID == containerID {
|
||||||
|
relevant = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if relevant {
|
||||||
|
logrus.WithFields(logrus.Fields{"type": ev.Type, "action": ev.Action, "id": containerID}).Trace("Docker event")
|
||||||
|
}
|
||||||
|
|
||||||
|
w.applyContainerRoutesLocked(containerID, desired)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// containersForID inspects a single container and returns the routableContainers
|
||||||
|
// it should produce. Returns nil if the container is gone or not routable.
|
||||||
|
func (w *dockerWatcherImpl) containersForID(ctx context.Context, containerID string) ([]*routableContainer, error) {
|
||||||
|
inspect, err := w.client.ContainerInspect(ctx, containerID)
|
||||||
|
if err != nil {
|
||||||
|
if cerrdefs.IsNotFound(err) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
data, ok := w.parseContainerData(&inspect)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
endpoint := ""
|
||||||
|
if !data.notRunning {
|
||||||
|
endpoint = fmt.Sprintf("%s:%d", data.ip, data.port)
|
||||||
|
}
|
||||||
|
var result []*routableContainer
|
||||||
|
for _, host := range data.hosts {
|
||||||
|
result = append(result, &routableContainer{
|
||||||
|
containerEndpoint: endpoint,
|
||||||
|
externalContainerName: host,
|
||||||
|
containerID: containerID,
|
||||||
|
autoScaleUp: data.autoScaleUp,
|
||||||
|
autoScaleDown: data.autoScaleDown,
|
||||||
|
autoScaleAsleepMOTD: data.autoScaleAsleepMOTD,
|
||||||
|
autoScaleLoadingMOTD: data.autoScaleLoadingMOTD,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if data.def != nil && *data.def {
|
||||||
|
result = append(result, &routableContainer{
|
||||||
|
containerEndpoint: endpoint,
|
||||||
|
externalContainerName: "",
|
||||||
|
containerID: containerID,
|
||||||
|
autoScaleUp: data.autoScaleUp,
|
||||||
|
autoScaleDown: data.autoScaleDown,
|
||||||
|
autoScaleAsleepMOTD: data.autoScaleAsleepMOTD,
|
||||||
|
autoScaleLoadingMOTD: data.autoScaleLoadingMOTD,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// applyContainerRoutesLocked reconciles the routes for a single containerID
|
||||||
|
// against the desired set. Caller must hold monitorLock.
|
||||||
|
func (w *dockerWatcherImpl) applyContainerRoutesLocked(containerID string, desired []*routableContainer) {
|
||||||
|
desiredByName := map[string]*routableContainer{}
|
||||||
|
for _, rc := range desired {
|
||||||
|
desiredByName[rc.externalContainerName] = rc
|
||||||
|
}
|
||||||
|
|
||||||
|
// Drop entries previously owned by this container that are no longer desired
|
||||||
|
for name, rc := range w.containerMap {
|
||||||
|
if rc.containerID != containerID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, keep := desiredByName[name]; keep {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
delete(w.containerMap, name)
|
||||||
|
if name != "" {
|
||||||
|
Routes.DeleteMapping(name)
|
||||||
|
} else {
|
||||||
|
Routes.SetDefaultRoute("", "", nil, nil, "", "")
|
||||||
|
}
|
||||||
|
logrus.WithField("routableContainer", rc).Debug("DELETE")
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, rs := range desired {
|
||||||
|
oldRs, exists := w.containerMap[rs.externalContainerName]
|
||||||
|
if !exists {
|
||||||
w.containerMap[rs.externalContainerName] = rs
|
w.containerMap[rs.externalContainerName] = rs
|
||||||
logrus.WithField("routableContainer", rs).Debug("ADD")
|
|
||||||
wakerFunc := w.makeWakerFunc(rs)
|
wakerFunc := w.makeWakerFunc(rs)
|
||||||
sleeperFunc := w.makeSleeperFunc(rs)
|
sleeperFunc := w.makeSleeperFunc(rs)
|
||||||
if rs.externalContainerName != "" {
|
if rs.externalContainerName != "" {
|
||||||
@@ -196,12 +328,17 @@ func (w *dockerWatcherImpl) monitorContainers(ctx context.Context) error {
|
|||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute(rs.containerEndpoint, "", wakerFunc, sleeperFunc, rs.autoScaleAsleepMOTD, rs.autoScaleLoadingMOTD)
|
Routes.SetDefaultRoute(rs.containerEndpoint, "", wakerFunc, sleeperFunc, rs.autoScaleAsleepMOTD, rs.autoScaleLoadingMOTD)
|
||||||
}
|
}
|
||||||
} else if oldRs.containerEndpoint != rs.containerEndpoint ||
|
logrus.WithField("routableContainer", rs).Debug("ADD")
|
||||||
oldRs.containerID != rs.containerID ||
|
continue
|
||||||
oldRs.autoScaleUp != rs.autoScaleUp ||
|
}
|
||||||
oldRs.autoScaleDown != rs.autoScaleDown ||
|
if oldRs.containerEndpoint == rs.containerEndpoint &&
|
||||||
oldRs.autoScaleAsleepMOTD != rs.autoScaleAsleepMOTD ||
|
oldRs.containerID == rs.containerID &&
|
||||||
oldRs.autoScaleLoadingMOTD != rs.autoScaleLoadingMOTD {
|
oldRs.autoScaleUp == rs.autoScaleUp &&
|
||||||
|
oldRs.autoScaleDown == rs.autoScaleDown &&
|
||||||
|
oldRs.autoScaleAsleepMOTD == rs.autoScaleAsleepMOTD &&
|
||||||
|
oldRs.autoScaleLoadingMOTD == rs.autoScaleLoadingMOTD {
|
||||||
|
continue
|
||||||
|
}
|
||||||
w.containerMap[rs.externalContainerName] = rs
|
w.containerMap[rs.externalContainerName] = rs
|
||||||
wakerFunc := w.makeWakerFunc(rs)
|
wakerFunc := w.makeWakerFunc(rs)
|
||||||
sleeperFunc := w.makeSleeperFunc(rs)
|
sleeperFunc := w.makeSleeperFunc(rs)
|
||||||
@@ -213,20 +350,6 @@ func (w *dockerWatcherImpl) monitorContainers(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
logrus.WithFields(logrus.Fields{"old": oldRs, "new": rs}).Debug("UPDATE")
|
logrus.WithFields(logrus.Fields{"old": oldRs, "new": rs}).Debug("UPDATE")
|
||||||
}
|
}
|
||||||
visited[rs.externalContainerName] = struct{}{}
|
|
||||||
}
|
|
||||||
for _, rs := range w.containerMap {
|
|
||||||
if _, ok := visited[rs.externalContainerName]; !ok {
|
|
||||||
delete(w.containerMap, rs.externalContainerName)
|
|
||||||
if rs.externalContainerName != "" {
|
|
||||||
Routes.DeleteMapping(rs.externalContainerName)
|
|
||||||
} else {
|
|
||||||
Routes.SetDefaultRoute("", "", nil, nil, "", "")
|
|
||||||
}
|
|
||||||
logrus.WithField("routableContainer", rs).Debug("DELETE")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerWatcherImpl) Start(ctx context.Context) error {
|
func (w *dockerWatcherImpl) Start(ctx context.Context) error {
|
||||||
@@ -248,49 +371,95 @@ func (w *dockerWatcherImpl) Start(ctx context.Context) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
w.containerMap = map[string]*routableContainer{}
|
||||||
// TODO: replace all this with events listening
|
|
||||||
ticker := time.NewTicker(w.config.refreshInterval)
|
|
||||||
|
|
||||||
logrus.Trace("Performing initial listing of Docker containers")
|
logrus.Trace("Performing initial listing of Docker containers")
|
||||||
initialContainers, err := w.listContainers(ctx)
|
if err := w.monitorContainers(ctx); err != nil {
|
||||||
if err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
w.containerMap = map[string]*routableContainer{}
|
// streamEvents will resync on (re)connect and otherwise apply incremental
|
||||||
for _, c := range initialContainers {
|
// updates from the Docker event stream — no periodic polling.
|
||||||
w.containerMap[c.externalContainerName] = c
|
go w.streamEvents(ctx)
|
||||||
wakerFunc := w.makeWakerFunc(c)
|
|
||||||
sleeperFunc := w.makeSleeperFunc(c)
|
|
||||||
if c.externalContainerName != "" {
|
|
||||||
Routes.CreateMapping(c.externalContainerName, c.containerEndpoint, "", wakerFunc, sleeperFunc, c.autoScaleAsleepMOTD, c.autoScaleLoadingMOTD)
|
|
||||||
} else {
|
|
||||||
Routes.SetDefaultRoute(c.containerEndpoint, "", wakerFunc, sleeperFunc, c.autoScaleAsleepMOTD, c.autoScaleLoadingMOTD)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
for {
|
|
||||||
select {
|
|
||||||
case <-ticker.C:
|
|
||||||
err := w.monitorContainers(ctx)
|
|
||||||
if err != nil {
|
|
||||||
logrus.WithError(err).Error("Docker monitoring failed")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
case <-ctx.Done():
|
|
||||||
logrus.Debug("Stopping Docker monitoring")
|
|
||||||
ticker.Stop()
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
logrus.Info("Monitoring Docker for Minecraft containers")
|
logrus.Info("Monitoring Docker for Minecraft containers")
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// streamEvents subscribes to the Docker event stream and triggers reconciliation
|
||||||
|
// of routes whenever container or network events relevant to routing occur.
|
||||||
|
// Reconnects with backoff on stream errors (e.g. daemon restart).
|
||||||
|
func (w *dockerWatcherImpl) streamEvents(ctx context.Context) {
|
||||||
|
backoff := time.Second
|
||||||
|
const maxBackoff = 30 * time.Second
|
||||||
|
|
||||||
|
for {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
logrus.Debug("Stopping Docker monitoring")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
eventFilters := filters.NewArgs(
|
||||||
|
filters.Arg("type", string(events.ContainerEventType)),
|
||||||
|
filters.Arg("type", string(events.NetworkEventType)),
|
||||||
|
filters.Arg("event", string(events.ActionStart)),
|
||||||
|
filters.Arg("event", string(events.ActionUnPause)),
|
||||||
|
filters.Arg("event", string(events.ActionStop)),
|
||||||
|
filters.Arg("event", string(events.ActionDie)),
|
||||||
|
filters.Arg("event", string(events.ActionPause)),
|
||||||
|
filters.Arg("event", string(events.ActionDestroy)),
|
||||||
|
filters.Arg("event", string(events.ActionRename)),
|
||||||
|
filters.Arg("event", string(events.ActionConnect)),
|
||||||
|
filters.Arg("event", string(events.ActionDisconnect)),
|
||||||
|
)
|
||||||
|
|
||||||
|
eventCh, errCh := w.client.Events(ctx, events.ListOptions{Filters: eventFilters})
|
||||||
|
|
||||||
|
// Resync after (re)connecting in case we missed events while disconnected
|
||||||
|
if err := w.monitorContainers(ctx); err != nil {
|
||||||
|
logrus.WithError(err).Error("Docker resync failed")
|
||||||
|
} else {
|
||||||
|
backoff = time.Second
|
||||||
|
}
|
||||||
|
|
||||||
|
loop:
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case ev, ok := <-eventCh:
|
||||||
|
if !ok {
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
if err := w.applyEvent(ctx, ev); err != nil {
|
||||||
|
logrus.WithError(err).Error("Docker event handling failed")
|
||||||
|
}
|
||||||
|
case err, ok := <-errCh:
|
||||||
|
if !ok {
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
logrus.WithError(err).Warn("Docker event stream error, reconnecting")
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-time.After(backoff):
|
||||||
|
}
|
||||||
|
if backoff < maxBackoff {
|
||||||
|
backoff *= 2
|
||||||
|
if backoff > maxBackoff {
|
||||||
|
backoff = maxBackoff
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (w *dockerWatcherImpl) listContainers(ctx context.Context) ([]*routableContainer, error) {
|
func (w *dockerWatcherImpl) listContainers(ctx context.Context) ([]*routableContainer, error) {
|
||||||
containers, err := w.client.ContainerList(ctx, container.ListOptions{All: true})
|
containers, err := w.client.ContainerList(ctx, container.ListOptions{All: true})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
+82
-32
@@ -10,6 +10,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
dockertypes "github.com/docker/docker/api/types"
|
dockertypes "github.com/docker/docker/api/types"
|
||||||
|
"github.com/docker/docker/api/types/events"
|
||||||
"github.com/docker/docker/api/types/filters"
|
"github.com/docker/docker/api/types/filters"
|
||||||
"github.com/docker/docker/api/types/network"
|
"github.com/docker/docker/api/types/network"
|
||||||
"github.com/docker/docker/api/types/swarm"
|
"github.com/docker/docker/api/types/swarm"
|
||||||
@@ -19,12 +20,11 @@ import (
|
|||||||
"github.com/sirupsen/logrus"
|
"github.com/sirupsen/logrus"
|
||||||
)
|
)
|
||||||
|
|
||||||
func NewDockerSwarmWatcher(socket string, timeout time.Duration, refreshInterval time.Duration, autoScaleUp bool, autoScaleDown bool, dockerApiVersion string) IDockerWatcher {
|
func NewDockerSwarmWatcher(socket string, timeout time.Duration, autoScaleUp bool, autoScaleDown bool, dockerApiVersion string) IDockerWatcher {
|
||||||
return &dockerSwarmWatcherImpl{
|
return &dockerSwarmWatcherImpl{
|
||||||
config: dockerWatcherConfig{
|
config: dockerWatcherConfig{
|
||||||
socket: socket,
|
socket: socket,
|
||||||
timeout: timeout,
|
timeout: timeout,
|
||||||
refreshInterval: refreshInterval,
|
|
||||||
autoScaleUp: autoScaleUp,
|
autoScaleUp: autoScaleUp,
|
||||||
autoScaleDown: autoScaleDown,
|
autoScaleDown: autoScaleDown,
|
||||||
apiVersion: dockerApiVersion,
|
apiVersion: dockerApiVersion,
|
||||||
@@ -36,6 +36,8 @@ type dockerSwarmWatcherImpl struct {
|
|||||||
sync.RWMutex
|
sync.RWMutex
|
||||||
config dockerWatcherConfig
|
config dockerWatcherConfig
|
||||||
client *client.Client
|
client *client.Client
|
||||||
|
serviceMap map[string]*routableService
|
||||||
|
monitorLock sync.Mutex
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerSwarmWatcherImpl) makeWakerFunc(_ *routableService) WakerFunc {
|
func (w *dockerSwarmWatcherImpl) makeWakerFunc(_ *routableService) WakerFunc {
|
||||||
@@ -75,40 +77,33 @@ func (w *dockerSwarmWatcherImpl) Start(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
ticker := time.NewTicker(w.config.refreshInterval)
|
w.serviceMap = map[string]*routableService{}
|
||||||
serviceMap := map[string]*routableService{}
|
|
||||||
|
|
||||||
logrus.Trace("Performing initial listing of Docker containers")
|
logrus.Trace("Performing initial listing of Docker swarm services")
|
||||||
initialServices, err := w.listServices(ctx)
|
if err := w.reconcileServices(ctx); err != nil {
|
||||||
if err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, s := range initialServices {
|
go w.streamEvents(ctx)
|
||||||
serviceMap[s.externalServiceName] = s
|
|
||||||
wakerFunc := w.makeWakerFunc(s)
|
logrus.Info("Monitoring Docker Swarm for Minecraft services")
|
||||||
sleeperFunc := w.makeSleeperFunc(s)
|
return nil
|
||||||
if s.externalServiceName != "" {
|
|
||||||
Routes.CreateMapping(s.externalServiceName, s.containerEndpoint, "", wakerFunc, sleeperFunc, "", "")
|
|
||||||
} else {
|
|
||||||
Routes.SetDefaultRoute(s.containerEndpoint, "", wakerFunc, sleeperFunc, "", "")
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
go func() {
|
func (w *dockerSwarmWatcherImpl) reconcileServices(ctx context.Context) error {
|
||||||
for {
|
w.monitorLock.Lock()
|
||||||
select {
|
defer w.monitorLock.Unlock()
|
||||||
case <-ticker.C:
|
|
||||||
services, err := w.listServices(ctx)
|
services, err := w.listServices(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logrus.WithError(err).Error("Docker failed to list services")
|
logrus.WithError(err).Error("Docker failed to list services")
|
||||||
continue
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
visited := map[string]struct{}{}
|
visited := map[string]struct{}{}
|
||||||
for _, rs := range services {
|
for _, rs := range services {
|
||||||
if oldRs, ok := serviceMap[rs.externalServiceName]; !ok {
|
if oldRs, ok := w.serviceMap[rs.externalServiceName]; !ok {
|
||||||
serviceMap[rs.externalServiceName] = rs
|
w.serviceMap[rs.externalServiceName] = rs
|
||||||
logrus.WithField("routableService", rs).Debug("ADD")
|
logrus.WithField("routableService", rs).Debug("ADD")
|
||||||
wakerFunc := w.makeWakerFunc(rs)
|
wakerFunc := w.makeWakerFunc(rs)
|
||||||
sleeperFunc := w.makeSleeperFunc(rs)
|
sleeperFunc := w.makeSleeperFunc(rs)
|
||||||
@@ -118,7 +113,7 @@ func (w *dockerSwarmWatcherImpl) Start(ctx context.Context) error {
|
|||||||
Routes.SetDefaultRoute(rs.containerEndpoint, "", wakerFunc, sleeperFunc, "", "")
|
Routes.SetDefaultRoute(rs.containerEndpoint, "", wakerFunc, sleeperFunc, "", "")
|
||||||
}
|
}
|
||||||
} else if oldRs.containerEndpoint != rs.containerEndpoint {
|
} else if oldRs.containerEndpoint != rs.containerEndpoint {
|
||||||
serviceMap[rs.externalServiceName] = rs
|
w.serviceMap[rs.externalServiceName] = rs
|
||||||
wakerFunc := w.makeWakerFunc(rs)
|
wakerFunc := w.makeWakerFunc(rs)
|
||||||
sleeperFunc := w.makeSleeperFunc(rs)
|
sleeperFunc := w.makeSleeperFunc(rs)
|
||||||
if rs.externalServiceName != "" {
|
if rs.externalServiceName != "" {
|
||||||
@@ -131,9 +126,9 @@ func (w *dockerSwarmWatcherImpl) Start(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
visited[rs.externalServiceName] = struct{}{}
|
visited[rs.externalServiceName] = struct{}{}
|
||||||
}
|
}
|
||||||
for _, rs := range serviceMap {
|
for _, rs := range w.serviceMap {
|
||||||
if _, ok := visited[rs.externalServiceName]; !ok {
|
if _, ok := visited[rs.externalServiceName]; !ok {
|
||||||
delete(serviceMap, rs.externalServiceName)
|
delete(w.serviceMap, rs.externalServiceName)
|
||||||
if rs.externalServiceName != "" {
|
if rs.externalServiceName != "" {
|
||||||
Routes.DeleteMapping(rs.externalServiceName)
|
Routes.DeleteMapping(rs.externalServiceName)
|
||||||
} else {
|
} else {
|
||||||
@@ -142,16 +137,71 @@ func (w *dockerSwarmWatcherImpl) Start(ctx context.Context) error {
|
|||||||
logrus.WithField("routableService", rs).Debug("DELETE")
|
logrus.WithField("routableService", rs).Debug("DELETE")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
case <-ctx.Done():
|
func (w *dockerSwarmWatcherImpl) streamEvents(ctx context.Context) {
|
||||||
ticker.Stop()
|
backoff := time.Second
|
||||||
|
const maxBackoff = 30 * time.Second
|
||||||
|
|
||||||
|
for {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
logrus.Debug("Stopping Docker Swarm monitoring")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
logrus.Info("Monitoring Docker Swarm for Minecraft services")
|
eventFilters := filters.NewArgs(
|
||||||
return nil
|
filters.Arg("type", string(events.ServiceEventType)),
|
||||||
|
filters.Arg("event", string(events.ActionCreate)),
|
||||||
|
filters.Arg("event", string(events.ActionUpdate)),
|
||||||
|
filters.Arg("event", string(events.ActionRemove)),
|
||||||
|
)
|
||||||
|
|
||||||
|
eventCh, errCh := w.client.Events(ctx, events.ListOptions{Filters: eventFilters})
|
||||||
|
|
||||||
|
if err := w.reconcileServices(ctx); err != nil {
|
||||||
|
logrus.WithError(err).Error("Docker Swarm resync failed")
|
||||||
|
} else {
|
||||||
|
backoff = time.Second
|
||||||
|
}
|
||||||
|
|
||||||
|
loop:
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case ev, ok := <-eventCh:
|
||||||
|
if !ok {
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
logrus.WithFields(logrus.Fields{"type": ev.Type, "action": ev.Action, "id": ev.Actor.ID}).Trace("Docker Swarm event")
|
||||||
|
if err := w.reconcileServices(ctx); err != nil {
|
||||||
|
logrus.WithError(err).Error("Docker Swarm reconciliation failed")
|
||||||
|
}
|
||||||
|
case err, ok := <-errCh:
|
||||||
|
if !ok {
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
logrus.WithError(err).Warn("Docker Swarm event stream error, reconnecting")
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-time.After(backoff):
|
||||||
|
}
|
||||||
|
if backoff < maxBackoff {
|
||||||
|
backoff *= 2
|
||||||
|
if backoff > maxBackoff {
|
||||||
|
backoff = maxBackoff
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerSwarmWatcherImpl) listServices(ctx context.Context) ([]*routableService, error) {
|
func (w *dockerSwarmWatcherImpl) listServices(ctx context.Context) ([]*routableService, error) {
|
||||||
|
|||||||
+7
-2
@@ -140,9 +140,14 @@ func NewServer(ctx context.Context, config *Config) (*Server, error) {
|
|||||||
routeWatchers = append(routeWatchers, k8sWatcher)
|
routeWatchers = append(routeWatchers, k8sWatcher)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if config.DockerRefreshInterval != 0 {
|
||||||
|
logrus.WithField("value", config.DockerRefreshInterval).
|
||||||
|
Warn("--docker-refresh-interval is deprecated and ignored; Docker discovery is now event-driven")
|
||||||
|
}
|
||||||
|
|
||||||
// TODO convert to RouteFinder
|
// TODO convert to RouteFinder
|
||||||
if config.InDocker {
|
if config.InDocker {
|
||||||
watcher := NewDockerWatcher(config.DockerSocket, config.DockerTimeout, config.DockerRefreshInterval, config.AutoScale.Up, config.AutoScale.Down, config.DockerApiVersion)
|
watcher := NewDockerWatcher(config.DockerSocket, config.DockerTimeout, config.AutoScale.Up, config.AutoScale.Down, config.DockerApiVersion)
|
||||||
err = watcher.Start(ctx)
|
err = watcher.Start(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not start docker integration: %w", err)
|
return nil, fmt.Errorf("could not start docker integration: %w", err)
|
||||||
@@ -151,7 +156,7 @@ func NewServer(ctx context.Context, config *Config) (*Server, error) {
|
|||||||
|
|
||||||
// TODO convert to RouteFinder
|
// TODO convert to RouteFinder
|
||||||
if config.InDockerSwarm {
|
if config.InDockerSwarm {
|
||||||
watcher := NewDockerSwarmWatcher(config.DockerSocket, config.DockerTimeout, config.DockerRefreshInterval, config.AutoScale.Up, config.AutoScale.Down, config.DockerApiVersion)
|
watcher := NewDockerSwarmWatcher(config.DockerSocket, config.DockerTimeout, config.AutoScale.Up, config.AutoScale.Down, config.DockerApiVersion)
|
||||||
err = watcher.Start(ctx)
|
err = watcher.Start(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not start docker swarm integration: %w", err)
|
return nil, fmt.Errorf("could not start docker swarm integration: %w", err)
|
||||||
|
|||||||
Reference in New Issue
Block a user