Docker auto-scale and asleep motd status (#488)
This commit is contained in:
@@ -14,7 +14,8 @@ Some other features included:
|
|||||||
- Rate limits incoming connections to reduce DDoS attacks.
|
- Rate limits incoming connections to reduce DDoS attacks.
|
||||||
- Can be configured to allow/deny IP addresses or ranges
|
- Can be configured to allow/deny IP addresses or ranges
|
||||||
- Includes a webhook integration for notifying other systems when a player connects and disconnects from a server.
|
- Includes a webhook integration for notifying other systems when a player connects and disconnects from a server.
|
||||||
- Can auto-scale (between zero and one) backend servers deployed as Kubernetes StatefulSets.
|
- Can auto-scale (between zero and one) backend servers deployed as Kubernetes StatefulSets
|
||||||
|
- or start and stop backend servers running as docker containers.
|
||||||
- Built-in ngrok integration where mc-router acts as an agent
|
- Built-in ngrok integration where mc-router acts as an agent
|
||||||
- Exports/exposes metrics for various Prometheus and InfluxDB. If enabled, includes player login metrics.
|
- Exports/exposes metrics for various Prometheus and InfluxDB. If enabled, includes player login metrics.
|
||||||
|
|
||||||
@@ -25,12 +26,14 @@ Some other features included:
|
|||||||
The host:port bound for servicing API requests (env API_BINDING)
|
The host:port bound for servicing API requests (env API_BINDING)
|
||||||
-auto-scale-allow-deny string
|
-auto-scale-allow-deny string
|
||||||
Path to config for server allowlists and denylists. If a global/server entry is specified, only players allowed to connect to the server will be able to trigger a scale up when -auto-scale-up is enabled or cancel active down scalers when -auto-scale-down is enabled (env AUTO_SCALE_ALLOW_DENY)
|
Path to config for server allowlists and denylists. If a global/server entry is specified, only players allowed to connect to the server will be able to trigger a scale up when -auto-scale-up is enabled or cancel active down scalers when -auto-scale-down is enabled (env AUTO_SCALE_ALLOW_DENY)
|
||||||
|
-auto-scale-asleep-motd string
|
||||||
|
MOTD to display when auto-scaled down servers are accessed; if empty, no status will be served (env AUTO_SCALE_ASLEEP_MOTD)
|
||||||
-auto-scale-down
|
-auto-scale-down
|
||||||
Decrease Kubernetes StatefulSet Replicas (only) from 1 to 0 on respective backend servers after there are no connections (env AUTO_SCALE_DOWN)
|
Scale to zero after idle. For Kubernetes, decreases StatefulSet replicas from 1 to 0. For Docker, gracefully stops the container when there are no connections (env AUTO_SCALE_DOWN)
|
||||||
-auto-scale-down-after string
|
-auto-scale-down-after string
|
||||||
Server scale down delay after there are no connections (env AUTO_SCALE_DOWN_AFTER) (default "10m")
|
Server scale down delay after there are no connections (env AUTO_SCALE_DOWN_AFTER) (default "10m")
|
||||||
-auto-scale-up
|
-auto-scale-up
|
||||||
Increase Kubernetes StatefulSet Replicas (only) from 0 to 1 on respective backend servers when accessed (env AUTO_SCALE_UP)
|
Scale from zero on access. For Kubernetes, increases StatefulSet replicas from 0 to 1. For Docker, starts or unpauses the container when accessed (env AUTO_SCALE_UP)
|
||||||
-clients-to-allow value
|
-clients-to-allow value
|
||||||
Zero or more client IP addresses or CIDRs to allow. Takes precedence over deny. (env CLIENTS_TO_ALLOW)
|
Zero or more client IP addresses or CIDRs to allow. Takes precedence over deny. (env CLIENTS_TO_ALLOW)
|
||||||
-clients-to-deny value
|
-clients-to-deny value
|
||||||
@@ -169,6 +172,35 @@ These are the labels scanned:
|
|||||||
- `mc-router.port`: This value must be set to the port the Minecraft server is listening on. The default value is 25565.
|
- `mc-router.port`: This value must be set to the port the Minecraft server is listening on. The default value is 25565.
|
||||||
- `mc-router.default`: Set this to a truthy value to make this server the default backend. Please note that `mc-router.host` is still required to be set.
|
- `mc-router.default`: Set this to a truthy value to make this server the default backend. Please note that `mc-router.host` is still required to be set.
|
||||||
- `mc-router.network`: Specify the network you are using for the router if multiple are present in the container/service. You can either use the network ID, it's full name or an alias.
|
- `mc-router.network`: Specify the network you are using for the router if multiple are present in the container/service. You can either use the network ID, it's full name or an alias.
|
||||||
|
- `mc-router.auto-scale-up`: Per-container override to enable/disable auto scale up for Docker. When true (or left unspecified and the global `-auto-scale-up` flag is enabled), mc-router will start or unpause this container when a client connects to the declared hostname(s).
|
||||||
|
- `mc-router.auto-scale-down`: Per-container override to enable/disable auto scale down for Docker. When true (or left unspecified and the global `-auto-scale-down` flag is enabled), mc-router will stop this container after it has been idle for the configured `-auto-scale-down-after` duration.
|
||||||
|
- `mc-router.auto-scale-asleep-motd`: Per-container override for MOTD to show when container is scaled to zero. If empty or not set the host will
|
||||||
|
appear unresponsive.
|
||||||
|
|
||||||
|
#### Docker Auto Scale Up/Down
|
||||||
|
|
||||||
|
To use scale-to-zero with Docker containers:
|
||||||
|
|
||||||
|
- Start mc-router with Docker discovery and scaling enabled, for example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run --rm \
|
||||||
|
-p 25565:25565 \
|
||||||
|
-v /var/run/docker.sock:/var/run/docker.sock:ro \
|
||||||
|
itzg/mc-router \
|
||||||
|
-in-docker -auto-scale-up -auto-scale-down -auto-scale-down-after=10m
|
||||||
|
```
|
||||||
|
|
||||||
|
- Label each Minecraft container with at least `mc-router.host`. You can also set per-container autoscale overrides using `mc-router.auto-scale-up` and `mc-router.auto-scale-down` labels.
|
||||||
|
|
||||||
|
For usage with docker compose refer to the [examples/docker-autoscale/compose.yml](examples/docker-autoscale/compose.yml) or [examples/docker-autoscale/compose-minimal.yml](examples/docker-autoscale/compose-minimal.yml) examples.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- When a client connects to a labeled hostname and the container is stopped or paused, mc-router will start/unpause it and wait until it becomes reachable (up to ~60s).
|
||||||
|
- When no clients remain connected and the idle timer elapses (`-auto-scale-down-after`), mc-router gracefully stops the container.
|
||||||
|
|
||||||
|
Note: Docker Swarm discovery is supported; however, auto scale up/down is not yet supported for Swarm services.
|
||||||
|
|
||||||
#### Example Docker deployment
|
#### Example Docker deployment
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,27 @@
|
|||||||
|
services:
|
||||||
|
router:
|
||||||
|
image: itzg/mc-router
|
||||||
|
environment:
|
||||||
|
IN_DOCKER: true
|
||||||
|
AUTO_SCALE_DOWN: true
|
||||||
|
AUTO_SCALE_UP: true
|
||||||
|
AUTO_SCALE_DOWN_AFTER: 2h
|
||||||
|
AUTO_SCALE_ASLEEP_MOTD: "Server is asleep. Join again to wake it up!"
|
||||||
|
ports:
|
||||||
|
- "25565:25565"
|
||||||
|
volumes:
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
vanilla:
|
||||||
|
image: itzg/minecraft-server
|
||||||
|
environment:
|
||||||
|
EULA: "TRUE"
|
||||||
|
labels:
|
||||||
|
mc-router.host: "vanilla.example.com"
|
||||||
|
paper:
|
||||||
|
image: itzg/minecraft-server
|
||||||
|
environment:
|
||||||
|
EULA: "TRUE"
|
||||||
|
TYPE: PAPER
|
||||||
|
labels:
|
||||||
|
mc-router.host: "paper.example.com"
|
||||||
|
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# This is a verbose example with comments and explanations for configuring auto-scaling behavior
|
||||||
|
# for Docker backend servers. See compose-minimal.yml for a simple minimal example.
|
||||||
|
services:
|
||||||
|
router:
|
||||||
|
image: itzg/mc-router
|
||||||
|
environment:
|
||||||
|
IN_DOCKER: true
|
||||||
|
# Global auto-scaling settings for all docker-backend servers
|
||||||
|
# Settings can be overridden per-backend using labels
|
||||||
|
# as shown in the backend services below (except for AUTO_SCALE_DOWN_AFTER which is global only)
|
||||||
|
# Enable auto-scaling down after inactivity for all backends by default
|
||||||
|
AUTO_SCALE_DOWN: true
|
||||||
|
# Enable auto-scaling up after player join for all backends by default
|
||||||
|
AUTO_SCALE_UP: true
|
||||||
|
# Time of inactivity after which to scale down (default: 10m) - Global only setting
|
||||||
|
AUTO_SCALE_DOWN_AFTER: 2h
|
||||||
|
# MOTD to show when server is asleep (default: empty string - don't show MOTD, show server offline instead)
|
||||||
|
AUTO_SCALE_ASLEEP_MOTD: "Server is asleep. Join again to wake it up!"
|
||||||
|
ports:
|
||||||
|
- "25565:25565"
|
||||||
|
volumes:
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
vanilla:
|
||||||
|
image: itzg/minecraft-server
|
||||||
|
environment:
|
||||||
|
EULA: "TRUE"
|
||||||
|
labels:
|
||||||
|
# If global auto scaling settings are enabled, this backend will
|
||||||
|
# auto-scale without any additional auto-scale related configuration
|
||||||
|
mc-router.host: "vanilla.example.com"
|
||||||
|
fabric:
|
||||||
|
image: itzg/minecraft-server
|
||||||
|
environment:
|
||||||
|
EULA: "TRUE"
|
||||||
|
TYPE: FABRIC
|
||||||
|
labels:
|
||||||
|
mc-router.host: "fabric.example.com"
|
||||||
|
# Disable auto-scaling for this backend specifically
|
||||||
|
mc-router.auto-scale-up: false
|
||||||
|
mc-router.auto-scale-down: false
|
||||||
|
paper:
|
||||||
|
image: itzg/minecraft-server
|
||||||
|
environment:
|
||||||
|
EULA: "TRUE"
|
||||||
|
TYPE: PAPER
|
||||||
|
labels:
|
||||||
|
mc-router.host: "paper.example.com"
|
||||||
|
# Override asleep MOTD for this backend
|
||||||
|
mc-router.auto-scale-asleep-motd: "Paper is folded. Join to unfold!"
|
||||||
|
|
||||||
+23
-1
@@ -58,7 +58,29 @@ func ReadPacket(reader *bufio.Reader, addr net.Addr, state State) (*Packet, erro
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
packet.Data = remainder.Bytes()
|
// For status state, decode based on packet ID:
|
||||||
|
// - 0x00 Status Request: no payload
|
||||||
|
// - 0x01 Ping: 8-byte long payload
|
||||||
|
if state == StateStatus {
|
||||||
|
switch packet.PacketID {
|
||||||
|
case PacketIdStatusRequest:
|
||||||
|
// no payload
|
||||||
|
packet.Data = nil
|
||||||
|
case PacketIdPingRequest:
|
||||||
|
timestamp, err := ReadLong(remainder)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
packet.Data = &PingPayload{
|
||||||
|
Timestamp: timestamp,
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
// unknown in status state; keep raw
|
||||||
|
packet.Data = remainder.Bytes()
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
packet.Data = remainder.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
logrus.
|
logrus.
|
||||||
WithField("client", addr).
|
WithField("client", addr).
|
||||||
|
|||||||
@@ -79,6 +79,13 @@ const (
|
|||||||
PacketIdHandshake = 0x00
|
PacketIdHandshake = 0x00
|
||||||
PacketIdLogin = 0x00 // during StateLogin
|
PacketIdLogin = 0x00 // during StateLogin
|
||||||
PacketIdLegacyServerListPing = 0xFE
|
PacketIdLegacyServerListPing = 0xFE
|
||||||
|
PacketIdStatusRequest = 0x00
|
||||||
|
PacketIdPingRequest = 0x01
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
PacketIdStatusResponse = 0x00
|
||||||
|
PackedIdPongResponse = 0x01
|
||||||
)
|
)
|
||||||
|
|
||||||
type Handshake struct {
|
type Handshake struct {
|
||||||
@@ -106,6 +113,30 @@ type LegacyServerListPing struct {
|
|||||||
ServerPort uint16
|
ServerPort uint16
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// StatusResponse is a minimal structure for the status JSON
|
||||||
|
type StatusResponse struct {
|
||||||
|
Version struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Protocol int `json:"protocol"`
|
||||||
|
} `json:"version"`
|
||||||
|
Players struct {
|
||||||
|
Max int `json:"max"`
|
||||||
|
Online int `json:"online"`
|
||||||
|
Sample []struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
ID string `json:"id"`
|
||||||
|
} `json:"sample,omitempty"`
|
||||||
|
} `json:"players"`
|
||||||
|
Description map[string]interface{} `json:"description"`
|
||||||
|
Favicon string `json:"favicon,omitempty"`
|
||||||
|
EnforcesSecureChat *bool `json:"enforcesSecureChat,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// PingPayload represents the status ping payload (packet 0x01)
|
||||||
|
type PingPayload struct {
|
||||||
|
Timestamp int64
|
||||||
|
}
|
||||||
|
|
||||||
type ByteReader interface {
|
type ByteReader interface {
|
||||||
ReadByte() (byte, error)
|
ReadByte() (byte, error)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,148 @@
|
|||||||
|
package mcproto
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"encoding/binary"
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"unicode/utf16"
|
||||||
|
)
|
||||||
|
|
||||||
|
// WriteVarInt writes a VarInt (Minecraft format) to w
|
||||||
|
func WriteVarInt(w io.Writer, value int32) error {
|
||||||
|
var buf [5]byte
|
||||||
|
i := 0
|
||||||
|
v := uint32(value)
|
||||||
|
for {
|
||||||
|
temp := byte(v & 0x7F)
|
||||||
|
v >>= 7
|
||||||
|
if v != 0 {
|
||||||
|
temp |= 0x80
|
||||||
|
}
|
||||||
|
buf[i] = temp
|
||||||
|
i++
|
||||||
|
if v == 0 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_, err := w.Write(buf[:i])
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// WriteString writes a Minecraft length-prefixed string
|
||||||
|
func WriteString(w io.Writer, s string) error {
|
||||||
|
if err := WriteVarInt(w, int32(len(s))); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_, err := io.WriteString(w, s)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildPacket builds a framed packet: [length VarInt][packetId VarInt][payload]
|
||||||
|
func buildPacket(packetID int32, payload []byte) []byte {
|
||||||
|
var b bytes.Buffer
|
||||||
|
_ = WriteVarInt(&b, packetID)
|
||||||
|
b.Write(payload)
|
||||||
|
|
||||||
|
var framed bytes.Buffer
|
||||||
|
_ = WriteVarInt(&framed, int32(b.Len()))
|
||||||
|
framed.Write(b.Bytes())
|
||||||
|
return framed.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
// WriteStatusJSONPacket writes a Status Response (packet 0x00) with the provided JSON string
|
||||||
|
func WriteStatusJSONPacket(w io.Writer, jsonString string) error {
|
||||||
|
// payload is the JSON as a Minecraft string
|
||||||
|
var payload bytes.Buffer
|
||||||
|
if err := WriteString(&payload, jsonString); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
pkt := buildPacket(PacketIdStatusResponse, payload.Bytes())
|
||||||
|
_, err := w.Write(pkt)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// WriteStatusFromStruct writes a Status Response from a struct
|
||||||
|
func WriteStatusFromStruct(w io.Writer, status StatusResponse) error {
|
||||||
|
b, err := json.Marshal(status)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return WriteStatusJSONPacket(w, string(b))
|
||||||
|
}
|
||||||
|
|
||||||
|
// WritePongPacket writes Pong (packet 0x01) with the same payload
|
||||||
|
func WritePongPacket(w io.Writer, timestamp int64) error {
|
||||||
|
var pl bytes.Buffer
|
||||||
|
// payload is a signed long (64-bit)
|
||||||
|
var buf [8]byte
|
||||||
|
binary.BigEndian.PutUint64(buf[:], uint64(timestamp))
|
||||||
|
pl.Write(buf[:])
|
||||||
|
pkt := buildPacket(PackedIdPongResponse, pl.Bytes())
|
||||||
|
_, err := w.Write(pkt)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// WriteLegacySLPResponse writes the 1.6-compatible legacy response packet (0xFF)
|
||||||
|
// Format: FF, [length short], UTF16BE string beginning with "\u00A7\u0031\u0000" then null-delimited fields
|
||||||
|
// fields: protocol, version, motd, online, max
|
||||||
|
func WriteLegacySLPResponse(w io.Writer, protocol int, version string, motd string, online int, max int) error {
|
||||||
|
// Build the string with null separators
|
||||||
|
s := "\u00A7\u0031\u0000" +
|
||||||
|
intToString(protocol) + "\u0000" +
|
||||||
|
version + "\u0000" +
|
||||||
|
motd + "\u0000" +
|
||||||
|
intToString(online) + "\u0000" +
|
||||||
|
intToString(max)
|
||||||
|
|
||||||
|
// Encode UTF-16BE
|
||||||
|
runes := []rune(s)
|
||||||
|
encoded := utf16.Encode(runes)
|
||||||
|
var be bytes.Buffer
|
||||||
|
for _, v := range encoded {
|
||||||
|
var tmp [2]byte
|
||||||
|
binary.BigEndian.PutUint16(tmp[:], v)
|
||||||
|
be.Write(tmp[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
bw := bufio.NewWriter(w)
|
||||||
|
// 0xFF
|
||||||
|
if _, err := bw.Write([]byte{0xFF}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// length short in code units
|
||||||
|
var lenBuf [2]byte
|
||||||
|
binary.BigEndian.PutUint16(lenBuf[:], uint16(len(encoded)))
|
||||||
|
if _, err := bw.Write(lenBuf[:]); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, err := bw.Write(be.Bytes()); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return bw.Flush()
|
||||||
|
}
|
||||||
|
|
||||||
|
// helpers
|
||||||
|
func intToString(i int) string {
|
||||||
|
if i == 0 {
|
||||||
|
return "0"
|
||||||
|
}
|
||||||
|
neg := false
|
||||||
|
if i < 0 {
|
||||||
|
neg = true
|
||||||
|
i = -i
|
||||||
|
}
|
||||||
|
var buf [20]byte
|
||||||
|
pos := len(buf)
|
||||||
|
for i > 0 {
|
||||||
|
pos--
|
||||||
|
buf[pos] = byte('0' + (i % 10))
|
||||||
|
i /= 10
|
||||||
|
}
|
||||||
|
if neg {
|
||||||
|
pos--
|
||||||
|
buf[pos] = '-'
|
||||||
|
}
|
||||||
|
return string(buf[pos:])
|
||||||
|
}
|
||||||
@@ -81,7 +81,7 @@ func routesCreateHandler(writer http.ResponseWriter, request *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
Routes.CreateMapping(definition.ServerAddress, definition.Backend, EmptyScalerFunc, EmptyScalerFunc)
|
Routes.CreateMapping(definition.ServerAddress, definition.Backend, nil, nil, "")
|
||||||
RoutesConfigLoader.SaveRoutes()
|
RoutesConfigLoader.SaveRoutes()
|
||||||
writer.WriteHeader(http.StatusCreated)
|
writer.WriteHeader(http.StatusCreated)
|
||||||
}
|
}
|
||||||
@@ -102,7 +102,7 @@ func routesSetDefault(writer http.ResponseWriter, request *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
Routes.SetDefaultRoute(body.Backend)
|
Routes.SetDefaultRoute(body.Backend, nil, nil, "")
|
||||||
RoutesConfigLoader.SaveRoutes()
|
RoutesConfigLoader.SaveRoutes()
|
||||||
writer.WriteHeader(http.StatusOK)
|
writer.WriteHeader(http.StatusOK)
|
||||||
}
|
}
|
||||||
|
|||||||
+5
-4
@@ -6,10 +6,11 @@ type WebhookConfig struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type AutoScale struct {
|
type AutoScale struct {
|
||||||
Up bool `usage:"Increase Kubernetes StatefulSet Replicas (only) from 0 to 1 on respective backend servers when accessed"`
|
Up bool `usage:"Scale from zero on access. For Kubernetes, increases StatefulSet replicas from 0 to 1. For Docker, starts or unpauses the container when accessed"`
|
||||||
Down bool `default:"false" usage:"Decrease Kubernetes StatefulSet Replicas (only) from 1 to 0 on respective backend servers after there are no connections"`
|
Down bool `default:"false" usage:"Scale to zero after idle. For Kubernetes, decreases StatefulSet replicas from 1 to 0. For Docker, gracefully stops the container when there are no connections"`
|
||||||
DownAfter string `default:"10m" usage:"Server scale down delay after there are no connections"`
|
DownAfter string `default:"10m" usage:"Server scale down delay after there are no connections"`
|
||||||
AllowDeny string `usage:"Path to config for server allowlists and denylists. If a global/server entry is specified, only players allowed to connect to the server will be able to trigger a scale up when -auto-scale-up is enabled or cancel active down scalers when -auto-scale-down is enabled"`
|
AllowDeny string `usage:"Path to config for server allowlists and denylists. If a global/server entry is specified, only players allowed to connect to the server will be able to trigger a scale up when -auto-scale-up is enabled or cancel active down scalers when -auto-scale-down is enabled"`
|
||||||
|
AsleepMOTD string `usage:"MOTD to display when auto-scaled down servers are accessed; if empty, no status will be served"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type RoutesConfig struct {
|
type RoutesConfig struct {
|
||||||
|
|||||||
+184
-21
@@ -38,30 +38,30 @@ func NewActiveConnections() *ActiveConnections {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (sm *ActiveConnections) Increment(serverAddress string) {
|
func (sm *ActiveConnections) Increment(backendAddress string) {
|
||||||
sm.Lock()
|
sm.Lock()
|
||||||
defer sm.Unlock()
|
defer sm.Unlock()
|
||||||
if _, ok := sm.activeConnections[serverAddress]; !ok {
|
if _, ok := sm.activeConnections[backendAddress]; !ok {
|
||||||
sm.activeConnections[serverAddress] = 1
|
sm.activeConnections[backendAddress] = 1
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
sm.activeConnections[serverAddress] += 1
|
sm.activeConnections[backendAddress] += 1
|
||||||
}
|
}
|
||||||
|
|
||||||
func (sm *ActiveConnections) Decrement(serverAddress string) {
|
func (sm *ActiveConnections) Decrement(backendAddress string) {
|
||||||
sm.Lock()
|
sm.Lock()
|
||||||
defer sm.Unlock()
|
defer sm.Unlock()
|
||||||
if activeConnections, ok := sm.activeConnections[serverAddress]; ok && activeConnections <= 0 {
|
if activeConnections, ok := sm.activeConnections[backendAddress]; ok && activeConnections <= 0 {
|
||||||
sm.activeConnections[serverAddress] = 0
|
sm.activeConnections[backendAddress] = 0
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
sm.activeConnections[serverAddress] -= 1
|
sm.activeConnections[backendAddress] -= 1
|
||||||
}
|
}
|
||||||
|
|
||||||
func (sm *ActiveConnections) GetCount(serverAddress string) int {
|
func (sm *ActiveConnections) GetCount(backendAddress string) int {
|
||||||
sm.Lock()
|
sm.Lock()
|
||||||
defer sm.Unlock()
|
defer sm.Unlock()
|
||||||
if activeConnections, ok := sm.activeConnections[serverAddress]; ok {
|
if activeConnections, ok := sm.activeConnections[backendAddress]; ok {
|
||||||
return activeConnections
|
return activeConnections
|
||||||
}
|
}
|
||||||
return 0
|
return 0
|
||||||
@@ -100,6 +100,7 @@ type Connector struct {
|
|||||||
clientFilter *ClientFilter
|
clientFilter *ClientFilter
|
||||||
autoScaleUpAllowDenyConfig *AllowDenyConfig
|
autoScaleUpAllowDenyConfig *AllowDenyConfig
|
||||||
connectionNotifier ConnectionNotifier
|
connectionNotifier ConnectionNotifier
|
||||||
|
asleepMOTD string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Connector) UseConnectionNotifier(notifier ConnectionNotifier) {
|
func (c *Connector) UseConnectionNotifier(notifier ConnectionNotifier) {
|
||||||
@@ -312,7 +313,7 @@ func (c *Connector) HandleConnection(frontendConn net.Conn) {
|
|||||||
Debug("Got user info")
|
Debug("Got user info")
|
||||||
}
|
}
|
||||||
|
|
||||||
c.findAndConnectBackend(frontendConn, clientAddr, inspectionBuffer, handshake.ServerAddress, playerInfo, handshake.NextState)
|
c.findAndConnectBackend(frontendConn, clientAddr, inspectionBuffer, handshake.ServerAddress, playerInfo, handshake.NextState, false, int(handshake.ProtocolVersion))
|
||||||
|
|
||||||
} else if packet.PacketID == mcproto.PacketIdLegacyServerListPing {
|
} else if packet.PacketID == mcproto.PacketIdLegacyServerListPing {
|
||||||
handshake, ok := packet.Data.(*mcproto.LegacyServerListPing)
|
handshake, ok := packet.Data.(*mcproto.LegacyServerListPing)
|
||||||
@@ -332,7 +333,7 @@ func (c *Connector) HandleConnection(frontendConn net.Conn) {
|
|||||||
|
|
||||||
serverAddress := handshake.ServerAddress
|
serverAddress := handshake.ServerAddress
|
||||||
|
|
||||||
c.findAndConnectBackend(frontendConn, clientAddr, inspectionBuffer, serverAddress, nil, mcproto.StateStatus)
|
c.findAndConnectBackend(frontendConn, clientAddr, inspectionBuffer, serverAddress, nil, mcproto.StateStatus, true, 0)
|
||||||
} else {
|
} else {
|
||||||
logrus.
|
logrus.
|
||||||
WithField("client", clientAddr).
|
WithField("client", clientAddr).
|
||||||
@@ -343,6 +344,110 @@ func (c *Connector) HandleConnection(frontendConn net.Conn) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// serveStatus writes a predefined status JSON and optionally handles ping/pong
|
||||||
|
func (c *Connector) serveStatus(frontendConn net.Conn, reader *bufio.Reader, serverAddress string, clientProtocol int) {
|
||||||
|
motd := Routes.GetAsleepMOTD(serverAddress)
|
||||||
|
if motd == "" {
|
||||||
|
motd = c.asleepMOTD
|
||||||
|
}
|
||||||
|
if motd == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Consume Status Request (0x00) if present; some clients may send Ping (0x01) directly
|
||||||
|
_ = frontendConn.SetReadDeadline(time.Now().Add(3 * time.Second))
|
||||||
|
firstPkt, err := mcproto.ReadPacket(reader, frontendConn.RemoteAddr(), mcproto.StateStatus)
|
||||||
|
var pingPending bool
|
||||||
|
var pingVal int64
|
||||||
|
if err == nil && firstPkt != nil {
|
||||||
|
if firstPkt.PacketID == mcproto.PacketIdPingRequest {
|
||||||
|
if payload, ok := firstPkt.Data.(mcproto.PingPayload); ok {
|
||||||
|
pingPending = true
|
||||||
|
pingVal = payload.Timestamp
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"client": frontendConn.RemoteAddr(),
|
||||||
|
"ping_val": pingVal,
|
||||||
|
}).Debug("Predefined status: received immediate ping")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// else 0x00 is the normal status request; proceed to write response
|
||||||
|
} else if err != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"client": frontendConn.RemoteAddr(),
|
||||||
|
"error": err,
|
||||||
|
}).Warn("Predefined status: error reading initial status packet")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build and write Status Response
|
||||||
|
viName, viProto := c.getVersionInfo(serverAddress, clientProtocol)
|
||||||
|
var status mcproto.StatusResponse
|
||||||
|
status.Version.Name = viName
|
||||||
|
status.Version.Protocol = viProto
|
||||||
|
status.Players.Max = 1
|
||||||
|
status.Players.Online = 0
|
||||||
|
status.Description = map[string]interface{}{"text": motd}
|
||||||
|
|
||||||
|
// Write Status Response
|
||||||
|
_ = frontendConn.SetWriteDeadline(time.Now().Add(handshakeTimeout))
|
||||||
|
if err := mcproto.WriteStatusFromStruct(frontendConn, status); err != nil {
|
||||||
|
logrus.WithError(err).Warn("Failed to write predefined status response")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// If we didn't already get a ping, briefly wait for one
|
||||||
|
if !pingPending {
|
||||||
|
_ = frontendConn.SetReadDeadline(time.Now().Add(2 * time.Second))
|
||||||
|
if nextPkt, err2 := mcproto.ReadPacket(reader, frontendConn.RemoteAddr(), mcproto.StateStatus); err2 == nil && nextPkt != nil {
|
||||||
|
if nextPkt.PacketID == mcproto.PacketIdPingRequest {
|
||||||
|
if payload, ok := nextPkt.Data.(mcproto.PingPayload); ok {
|
||||||
|
pingPending = true
|
||||||
|
pingVal = payload.Timestamp
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"client": frontendConn.RemoteAddr(),
|
||||||
|
"ping_val": pingVal,
|
||||||
|
}).Debug("Predefined status: received ping after status")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if err2 != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"client": frontendConn.RemoteAddr(),
|
||||||
|
"error": err2,
|
||||||
|
}).Debug("Predefined status: error/timeout reading ping after status")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if pingPending {
|
||||||
|
if err := mcproto.WritePongPacket(frontendConn, pingVal); err != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"client": frontendConn.RemoteAddr(),
|
||||||
|
"error": err,
|
||||||
|
}).Warn("Predefined status: failed to write pong")
|
||||||
|
} else {
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"client": frontendConn.RemoteAddr(),
|
||||||
|
"ping_val": pingVal,
|
||||||
|
}).Debug("Predefined status: wrote pong")
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"client": frontendConn.RemoteAddr(),
|
||||||
|
}).Debug("Predefined status: no ping received, closing")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveLegacyStatus writes a simple legacy SLP response and closes the connection
|
||||||
|
func (c *Connector) serveLegacyStatus(frontendConn net.Conn) {
|
||||||
|
motd := c.asleepMOTD
|
||||||
|
if motd == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_ = frontendConn.SetWriteDeadline(time.Now().Add(handshakeTimeout))
|
||||||
|
// 127 protocol for legacy response per spec; version name and motd from predefined JSON if available
|
||||||
|
// write a basic response: protocol=127, version="1.7+", motd, online=0, max=1
|
||||||
|
if err := mcproto.WriteLegacySLPResponse(frontendConn, 127, "1.7+", motd, 0, 1); err != nil {
|
||||||
|
logrus.WithError(err).Warn("Failed to write legacy SLP response")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (c *Connector) readPlayerInfo(protocolVersion mcproto.ProtocolVersion, bufferedReader *bufio.Reader, clientAddr net.Addr, state mcproto.State) (*PlayerInfo, error) {
|
func (c *Connector) readPlayerInfo(protocolVersion mcproto.ProtocolVersion, bufferedReader *bufio.Reader, clientAddr net.Addr, state mcproto.State) (*PlayerInfo, error) {
|
||||||
loginPacket, err := mcproto.ReadPacket(bufferedReader, clientAddr, state)
|
loginPacket, err := mcproto.ReadPacket(bufferedReader, clientAddr, state)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -375,10 +480,10 @@ func (c *Connector) cleanupBackendConnection(clientAddr net.Addr, serverAddress
|
|||||||
c.metrics.ActiveConnections.Set(float64(
|
c.metrics.ActiveConnections.Set(float64(
|
||||||
atomic.AddInt32(&c.totalActiveConnections, -1)))
|
atomic.AddInt32(&c.totalActiveConnections, -1)))
|
||||||
|
|
||||||
c.activeConnections.Decrement(serverAddress)
|
c.activeConnections.Decrement(backendHostPort)
|
||||||
c.metrics.ServerActiveConnections.
|
c.metrics.ServerActiveConnections.
|
||||||
With("server_address", serverAddress).
|
With("server_address", serverAddress).
|
||||||
Set(float64(c.activeConnections.GetCount(serverAddress)))
|
Set(float64(c.activeConnections.GetCount(backendHostPort)))
|
||||||
|
|
||||||
if c.recordLogins && playerInfo != nil {
|
if c.recordLogins && playerInfo != nil {
|
||||||
c.metrics.ServerActivePlayer.
|
c.metrics.ServerActivePlayer.
|
||||||
@@ -388,14 +493,19 @@ func (c *Connector) cleanupBackendConnection(clientAddr net.Addr, serverAddress
|
|||||||
Set(0)
|
Set(0)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if checkScaleDown && c.activeConnections.GetCount(serverAddress) <= 0 {
|
logrus.
|
||||||
DownScaler.Begin(serverAddress)
|
WithField("client", clientAddr).
|
||||||
|
WithField("backendHostPort", backendHostPort).
|
||||||
|
WithField("connectionCount", c.activeConnections.GetCount(backendHostPort)).
|
||||||
|
Info("Closed connection to backend")
|
||||||
|
if checkScaleDown && c.activeConnections.GetCount(backendHostPort) <= 0 {
|
||||||
|
DownScaler.Begin(backendHostPort)
|
||||||
}
|
}
|
||||||
c.connectionsCond.Signal()
|
c.connectionsCond.Signal()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Connector) findAndConnectBackend(frontendConn net.Conn,
|
func (c *Connector) findAndConnectBackend(frontendConn net.Conn,
|
||||||
clientAddr net.Addr, preReadContent io.Reader, serverAddress string, playerInfo *PlayerInfo, nextState mcproto.State) {
|
clientAddr net.Addr, preReadContent io.Reader, serverAddress string, playerInfo *PlayerInfo, nextState mcproto.State, isLegacy bool, clientProtocol int) {
|
||||||
|
|
||||||
backendHostPort, resolvedHost, waker, _ := Routes.FindBackendForServerAddress(c.ctx, serverAddress)
|
backendHostPort, resolvedHost, waker, _ := Routes.FindBackendForServerAddress(c.ctx, serverAddress)
|
||||||
cleanupMetrics := false
|
cleanupMetrics := false
|
||||||
@@ -415,13 +525,29 @@ func (c *Connector) findAndConnectBackend(frontendConn net.Conn,
|
|||||||
Debug("checked if player is allowed to wake up the server")
|
Debug("checked if player is allowed to wake up the server")
|
||||||
if serverAllowsPlayer {
|
if serverAllowsPlayer {
|
||||||
// Cancel down scaler if active before scale up
|
// Cancel down scaler if active before scale up
|
||||||
DownScaler.Cancel(serverAddress)
|
if backendHostPort != "" {
|
||||||
|
DownScaler.Cancel(backendHostPort)
|
||||||
|
}
|
||||||
cleanupCheckScaleDown = true
|
cleanupCheckScaleDown = true
|
||||||
if err := waker(c.ctx); err != nil {
|
logrus.WithField("serverAddress", serverAddress).Info("Waking up backend server")
|
||||||
|
newBackendHostPort, err := waker(c.ctx)
|
||||||
|
if err != nil {
|
||||||
logrus.WithFields(logrus.Fields{"serverAddress": serverAddress}).WithError(err).Error("failed to wake up backend")
|
logrus.WithFields(logrus.Fields{"serverAddress": serverAddress}).WithError(err).Error("failed to wake up backend")
|
||||||
c.metrics.Errors.With("type", "wakeup_failed").Add(1)
|
c.metrics.Errors.With("type", "wakeup_failed").Add(1)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if newBackendHostPort == "" {
|
||||||
|
logrus.WithFields(logrus.Fields{"serverAddress": serverAddress}).Warn("waker did not return a backend address")
|
||||||
|
c.metrics.Errors.With("type", "wakeup_no_address").Add(1)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Cancel again in case any routes were changed during wake up
|
||||||
|
DownScaler.Cancel(newBackendHostPort)
|
||||||
|
backendHostPort = newBackendHostPort
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"serverAddress": serverAddress,
|
||||||
|
"backendHostPort": backendHostPort,
|
||||||
|
}).Info("Woke up backend server")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -440,6 +566,22 @@ func (c *Connector) findAndConnectBackend(frontendConn net.Conn,
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// If status request and configured, serve predefined response
|
||||||
|
if nextState == mcproto.StateStatus && Routes.HasRoute(serverAddress) {
|
||||||
|
logrus.WithFields(logrus.Fields{
|
||||||
|
"client": clientAddr,
|
||||||
|
"server": serverAddress,
|
||||||
|
"isLegacy": isLegacy,
|
||||||
|
}).Debug("Missing backend: serving predefined status response")
|
||||||
|
|
||||||
|
// Read Status Request and Ping directly from the client connection
|
||||||
|
br := bufio.NewReader(frontendConn)
|
||||||
|
if isLegacy {
|
||||||
|
c.serveLegacyStatus(frontendConn)
|
||||||
|
} else {
|
||||||
|
c.serveStatus(frontendConn, br, serverAddress, clientProtocol)
|
||||||
|
}
|
||||||
|
}
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -483,10 +625,10 @@ func (c *Connector) findAndConnectBackend(frontendConn net.Conn,
|
|||||||
c.metrics.ActiveConnections.Set(float64(
|
c.metrics.ActiveConnections.Set(float64(
|
||||||
atomic.AddInt32(&c.totalActiveConnections, 1)))
|
atomic.AddInt32(&c.totalActiveConnections, 1)))
|
||||||
|
|
||||||
c.activeConnections.Increment(serverAddress)
|
c.activeConnections.Increment(backendHostPort)
|
||||||
c.metrics.ServerActiveConnections.
|
c.metrics.ServerActiveConnections.
|
||||||
With("server_address", serverAddress).
|
With("server_address", serverAddress).
|
||||||
Set(float64(c.activeConnections.GetCount(serverAddress)))
|
Set(float64(c.activeConnections.GetCount(backendHostPort)))
|
||||||
|
|
||||||
if c.recordLogins && playerInfo != nil {
|
if c.recordLogins && playerInfo != nil {
|
||||||
logrus.
|
logrus.
|
||||||
@@ -624,3 +766,24 @@ func (c *Connector) UseReceiveProxyProto(trustedProxyNets []*net.IPNet) {
|
|||||||
c.trustedProxyNets = trustedProxyNets
|
c.trustedProxyNets = trustedProxyNets
|
||||||
c.receiveProxyProto = true
|
c.receiveProxyProto = true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UseAsleepMOTD configures a predefined MOTD to serve when backends are asleep
|
||||||
|
func (c *Connector) UseAsleepMOTD(motd string) {
|
||||||
|
c.asleepMOTD = motd
|
||||||
|
}
|
||||||
|
|
||||||
|
// getVersionInfo falls back to client protocol and a derived name but in future
|
||||||
|
// could be extended to cache server-reported versions
|
||||||
|
func (c *Connector) getVersionInfo(_ string, clientProtocol int) (string, int) {
|
||||||
|
// no cache; use client protocol
|
||||||
|
return protocolToName(clientProtocol), clientProtocol
|
||||||
|
}
|
||||||
|
|
||||||
|
// protocolToName maps protocol numbers to a friendly name; falls back to "1.7+"
|
||||||
|
func protocolToName(proto int) string {
|
||||||
|
switch proto {
|
||||||
|
// TODO: expand this mapping as needed
|
||||||
|
default:
|
||||||
|
return "1.7+"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+195
-46
@@ -3,12 +3,12 @@ package server
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
dockertypes "github.com/docker/docker/api/types"
|
|
||||||
"github.com/docker/docker/api/types/container"
|
"github.com/docker/docker/api/types/container"
|
||||||
"github.com/docker/docker/client"
|
"github.com/docker/docker/client"
|
||||||
"github.com/sirupsen/logrus"
|
"github.com/sirupsen/logrus"
|
||||||
@@ -19,10 +19,13 @@ type IDockerWatcher interface {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const (
|
const (
|
||||||
DockerRouterLabelHost = "mc-router.host"
|
DockerRouterLabelHost = "mc-router.host"
|
||||||
DockerRouterLabelPort = "mc-router.port"
|
DockerRouterLabelPort = "mc-router.port"
|
||||||
DockerRouterLabelDefault = "mc-router.default"
|
DockerRouterLabelDefault = "mc-router.default"
|
||||||
DockerRouterLabelNetwork = "mc-router.network"
|
DockerRouterLabelNetwork = "mc-router.network"
|
||||||
|
DockerRouterLabelAutoScaleUp = "mc-router.auto-scale-up"
|
||||||
|
DockerRouterLabelAutoScaleDown = "mc-router.auto-scale-down"
|
||||||
|
DockerRouterLabelAutoScaleAsleepMOTD = "mc-router.auto-scale-asleep-motd"
|
||||||
)
|
)
|
||||||
|
|
||||||
type dockerWatcherConfig struct {
|
type dockerWatcherConfig struct {
|
||||||
@@ -63,22 +66,94 @@ type dockerWatcherImpl struct {
|
|||||||
client *client.Client
|
client *client.Client
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerWatcherImpl) makeWakerFunc(_ *routableContainer) ScalerFunc {
|
func (w *dockerWatcherImpl) makeWakerFunc(rc *routableContainer) WakerFunc {
|
||||||
if !w.config.autoScaleUp {
|
if rc == nil || !rc.autoScaleUp {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return func(ctx context.Context) error {
|
return func(ctx context.Context) (string, error) {
|
||||||
logrus.Fatal("Auto scale up is not yet supported for docker")
|
containerID := rc.containerID
|
||||||
return nil
|
if containerID == "" {
|
||||||
|
return "", fmt.Errorf("missing container id for wake")
|
||||||
|
}
|
||||||
|
inspect, err := w.client.ContainerInspect(ctx, containerID)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if inspect.State == nil {
|
||||||
|
return "", fmt.Errorf("unable to determine container state")
|
||||||
|
}
|
||||||
|
// If paused, unpause; if not running, start; otherwise no-op
|
||||||
|
if inspect.State.Paused {
|
||||||
|
logrus.WithFields(logrus.Fields{"containerID": containerID}).Debug("Unpausing container for wake")
|
||||||
|
if err := w.client.ContainerUnpause(ctx, containerID); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
} else if !inspect.State.Running {
|
||||||
|
logrus.WithFields(logrus.Fields{"containerID": containerID}).Debug("Starting container for wake")
|
||||||
|
if err := w.client.ContainerStart(ctx, containerID, container.StartOptions{}); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
inspect, err = w.client.ContainerInspect(ctx, containerID)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
data, ok := w.parseContainerData(&inspect)
|
||||||
|
if !ok {
|
||||||
|
return "", fmt.Errorf("failed to parse container data after starting")
|
||||||
|
}
|
||||||
|
if data.ip == "" {
|
||||||
|
return "", fmt.Errorf("container has no accessible IP after starting")
|
||||||
|
}
|
||||||
|
endpoint := net.JoinHostPort(data.ip, strconv.Itoa(int(data.port)))
|
||||||
|
|
||||||
|
// Wait until the container is reachable
|
||||||
|
deadline := time.Now().Add(60 * time.Second)
|
||||||
|
for {
|
||||||
|
conn, err := net.DialTimeout("tcp", endpoint, 1*time.Second)
|
||||||
|
if err == nil {
|
||||||
|
_ = conn.Close()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return endpoint, ctx.Err()
|
||||||
|
}
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
return endpoint, fmt.Errorf("timeout waiting for container to become reachable at %s", endpoint)
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return endpoint, ctx.Err()
|
||||||
|
case <-time.After(500 * time.Millisecond):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return endpoint, nil
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerWatcherImpl) makeSleeperFunc(_ *routableContainer) ScalerFunc {
|
func (w *dockerWatcherImpl) makeSleeperFunc(rc *routableContainer) SleeperFunc {
|
||||||
if !w.config.autoScaleDown {
|
if rc == nil || !rc.autoScaleDown {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return func(ctx context.Context) error {
|
return func(ctx context.Context) error {
|
||||||
logrus.Fatal("Auto scale down is not yet supported for docker")
|
containerID := rc.containerID
|
||||||
|
if containerID == "" {
|
||||||
|
return fmt.Errorf("missing container id for sleep")
|
||||||
|
}
|
||||||
|
inspect, err := w.client.ContainerInspect(ctx, containerID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if inspect.State != nil && inspect.State.Running {
|
||||||
|
// Graceful stop with 60s timeout
|
||||||
|
timeout := 60
|
||||||
|
logrus.WithFields(logrus.Fields{"containerID": containerID}).Debug("Stopping container for sleep")
|
||||||
|
if err := w.client.ContainerStop(ctx, containerID, container.StopOptions{Timeout: &timeout}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -104,7 +179,6 @@ func (w *dockerWatcherImpl) Start(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
ticker := time.NewTicker(refreshInterval)
|
ticker := time.NewTicker(refreshInterval)
|
||||||
containerMap := map[string]*routableContainer{}
|
|
||||||
|
|
||||||
logrus.Trace("Performing initial listing of Docker containers")
|
logrus.Trace("Performing initial listing of Docker containers")
|
||||||
initialContainers, err := w.listContainers(ctx)
|
initialContainers, err := w.listContainers(ctx)
|
||||||
@@ -112,12 +186,15 @@ func (w *dockerWatcherImpl) Start(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
containerMap := map[string]*routableContainer{}
|
||||||
for _, c := range initialContainers {
|
for _, c := range initialContainers {
|
||||||
containerMap[c.externalContainerName] = c
|
containerMap[c.externalContainerName] = c
|
||||||
|
wakerFunc := w.makeWakerFunc(c)
|
||||||
|
sleeperFunc := w.makeSleeperFunc(c)
|
||||||
if c.externalContainerName != "" {
|
if c.externalContainerName != "" {
|
||||||
Routes.CreateMapping(c.externalContainerName, c.containerEndpoint, w.makeWakerFunc(c), w.makeSleeperFunc(c))
|
Routes.CreateMapping(c.externalContainerName, c.containerEndpoint, wakerFunc, sleeperFunc, c.autoScaleAsleepMOTD)
|
||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute(c.containerEndpoint)
|
Routes.SetDefaultRoute(c.containerEndpoint, wakerFunc, sleeperFunc, c.autoScaleAsleepMOTD)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -137,18 +214,26 @@ func (w *dockerWatcherImpl) Start(ctx context.Context) error {
|
|||||||
if oldRs, ok := containerMap[rs.externalContainerName]; !ok {
|
if oldRs, ok := containerMap[rs.externalContainerName]; !ok {
|
||||||
containerMap[rs.externalContainerName] = rs
|
containerMap[rs.externalContainerName] = rs
|
||||||
logrus.WithField("routableContainer", rs).Debug("ADD")
|
logrus.WithField("routableContainer", rs).Debug("ADD")
|
||||||
|
wakerFunc := w.makeWakerFunc(rs)
|
||||||
|
sleeperFunc := w.makeSleeperFunc(rs)
|
||||||
if rs.externalContainerName != "" {
|
if rs.externalContainerName != "" {
|
||||||
Routes.CreateMapping(rs.externalContainerName, rs.containerEndpoint, w.makeWakerFunc(rs), w.makeSleeperFunc(rs))
|
Routes.CreateMapping(rs.externalContainerName, rs.containerEndpoint, wakerFunc, sleeperFunc, rs.autoScaleAsleepMOTD)
|
||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute(rs.containerEndpoint)
|
Routes.SetDefaultRoute(rs.containerEndpoint, wakerFunc, sleeperFunc, rs.autoScaleAsleepMOTD)
|
||||||
}
|
}
|
||||||
} else if oldRs.containerEndpoint != rs.containerEndpoint {
|
} else if oldRs.containerEndpoint != rs.containerEndpoint ||
|
||||||
|
oldRs.containerID != rs.containerID ||
|
||||||
|
oldRs.autoScaleUp != rs.autoScaleUp ||
|
||||||
|
oldRs.autoScaleDown != rs.autoScaleDown ||
|
||||||
|
oldRs.autoScaleAsleepMOTD != rs.autoScaleAsleepMOTD {
|
||||||
containerMap[rs.externalContainerName] = rs
|
containerMap[rs.externalContainerName] = rs
|
||||||
|
wakerFunc := w.makeWakerFunc(rs)
|
||||||
|
sleeperFunc := w.makeSleeperFunc(rs)
|
||||||
if rs.externalContainerName != "" {
|
if rs.externalContainerName != "" {
|
||||||
Routes.DeleteMapping(rs.externalContainerName)
|
Routes.DeleteMapping(rs.externalContainerName)
|
||||||
Routes.CreateMapping(rs.externalContainerName, rs.containerEndpoint, w.makeWakerFunc(rs), w.makeSleeperFunc(rs))
|
Routes.CreateMapping(rs.externalContainerName, rs.containerEndpoint, wakerFunc, sleeperFunc, rs.autoScaleAsleepMOTD)
|
||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute(rs.containerEndpoint)
|
Routes.SetDefaultRoute(rs.containerEndpoint, wakerFunc, sleeperFunc, rs.autoScaleAsleepMOTD)
|
||||||
}
|
}
|
||||||
logrus.WithFields(logrus.Fields{"old": oldRs, "new": rs}).Debug("UPDATE")
|
logrus.WithFields(logrus.Fields{"old": oldRs, "new": rs}).Debug("UPDATE")
|
||||||
}
|
}
|
||||||
@@ -160,7 +245,7 @@ func (w *dockerWatcherImpl) Start(ctx context.Context) error {
|
|||||||
if rs.externalContainerName != "" {
|
if rs.externalContainerName != "" {
|
||||||
Routes.DeleteMapping(rs.externalContainerName)
|
Routes.DeleteMapping(rs.externalContainerName)
|
||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute("")
|
Routes.SetDefaultRoute("", nil, nil, "")
|
||||||
}
|
}
|
||||||
logrus.WithField("routableContainer", rs).Debug("DELETE")
|
logrus.WithField("routableContainer", rs).Debug("DELETE")
|
||||||
}
|
}
|
||||||
@@ -179,28 +264,46 @@ func (w *dockerWatcherImpl) Start(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerWatcherImpl) listContainers(ctx context.Context) ([]*routableContainer, error) {
|
func (w *dockerWatcherImpl) listContainers(ctx context.Context) ([]*routableContainer, error) {
|
||||||
containers, err := w.client.ContainerList(ctx, container.ListOptions{})
|
containers, err := w.client.ContainerList(ctx, container.ListOptions{All: true})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
var result []*routableContainer
|
var result []*routableContainer
|
||||||
for _, container := range containers {
|
for _, container := range containers {
|
||||||
data, ok := w.parseContainerData(&container)
|
inspect, err := w.client.ContainerInspect(ctx, container.ID)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{"containerID": container.ID}).WithError(err).Error("Failed to inspect Docker container")
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
data, ok := w.parseContainerData(&inspect)
|
||||||
if !ok {
|
if !ok {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
endpoint := ""
|
||||||
|
if !data.notRunning {
|
||||||
|
endpoint = fmt.Sprintf("%s:%d", data.ip, data.port)
|
||||||
|
}
|
||||||
|
|
||||||
for _, host := range data.hosts {
|
for _, host := range data.hosts {
|
||||||
result = append(result, &routableContainer{
|
result = append(result, &routableContainer{
|
||||||
containerEndpoint: fmt.Sprintf("%s:%d", data.ip, data.port),
|
containerEndpoint: endpoint,
|
||||||
externalContainerName: host,
|
externalContainerName: host,
|
||||||
|
containerID: container.ID,
|
||||||
|
autoScaleUp: data.autoScaleUp,
|
||||||
|
autoScaleDown: data.autoScaleDown,
|
||||||
|
autoScaleAsleepMOTD: data.autoScaleAsleepMOTD,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
if data.def != nil && *data.def {
|
if data.def != nil && *data.def {
|
||||||
result = append(result, &routableContainer{
|
result = append(result, &routableContainer{
|
||||||
containerEndpoint: fmt.Sprintf("%s:%d", data.ip, data.port),
|
containerEndpoint: endpoint,
|
||||||
externalContainerName: "",
|
externalContainerName: "",
|
||||||
|
containerID: container.ID,
|
||||||
|
autoScaleUp: data.autoScaleUp,
|
||||||
|
autoScaleDown: data.autoScaleDown,
|
||||||
|
autoScaleAsleepMOTD: data.autoScaleAsleepMOTD,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -209,18 +312,24 @@ func (w *dockerWatcherImpl) listContainers(ctx context.Context) ([]*routableCont
|
|||||||
}
|
}
|
||||||
|
|
||||||
type parsedDockerContainerData struct {
|
type parsedDockerContainerData struct {
|
||||||
hosts []string
|
hosts []string
|
||||||
port uint64
|
port uint64
|
||||||
def *bool
|
def *bool
|
||||||
network *string
|
network *string
|
||||||
ip string
|
ip string
|
||||||
|
autoScaleDown bool
|
||||||
|
autoScaleUp bool
|
||||||
|
autoScaleAsleepMOTD string
|
||||||
|
notRunning bool
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerWatcherImpl) parseContainerData(container *dockertypes.Container) (data parsedDockerContainerData, ok bool) {
|
func (w *dockerWatcherImpl) parseContainerData(container *container.InspectResponse) (data parsedDockerContainerData, ok bool) {
|
||||||
for key, value := range container.Labels {
|
data.autoScaleUp = w.config.autoScaleUp
|
||||||
|
data.autoScaleDown = w.config.autoScaleDown
|
||||||
|
for key, value := range container.Config.Labels {
|
||||||
if key == DockerRouterLabelHost {
|
if key == DockerRouterLabelHost {
|
||||||
if data.hosts != nil {
|
if data.hosts != nil {
|
||||||
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Names}).
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
Warnf("ignoring container with duplicate %s label", DockerRouterLabelHost)
|
Warnf("ignoring container with duplicate %s label", DockerRouterLabelHost)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -229,14 +338,14 @@ func (w *dockerWatcherImpl) parseContainerData(container *dockertypes.Container)
|
|||||||
|
|
||||||
if key == DockerRouterLabelPort {
|
if key == DockerRouterLabelPort {
|
||||||
if data.port != 0 {
|
if data.port != 0 {
|
||||||
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Names}).
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
Warnf("ignoring container with duplicate %s label", DockerRouterLabelPort)
|
Warnf("ignoring container with duplicate %s label", DockerRouterLabelPort)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
var err error
|
var err error
|
||||||
data.port, err = strconv.ParseUint(value, 10, 32)
|
data.port, err = strconv.ParseUint(value, 10, 32)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Names}).
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
WithError(err).
|
WithError(err).
|
||||||
Warnf("ignoring container with invalid %s label", DockerRouterLabelPort)
|
Warnf("ignoring container with invalid %s label", DockerRouterLabelPort)
|
||||||
return
|
return
|
||||||
@@ -244,24 +353,51 @@ func (w *dockerWatcherImpl) parseContainerData(container *dockertypes.Container)
|
|||||||
}
|
}
|
||||||
if key == DockerRouterLabelDefault {
|
if key == DockerRouterLabelDefault {
|
||||||
if data.def != nil {
|
if data.def != nil {
|
||||||
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Names}).
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
Warnf("ignoring container with duplicate %s label", DockerRouterLabelDefault)
|
Warnf("ignoring container with duplicate %s label", DockerRouterLabelDefault)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
data.def = new(bool)
|
defaultValue, err := strconv.ParseBool(strings.TrimSpace(value))
|
||||||
|
if err != nil {
|
||||||
lowerValue := strings.TrimSpace(strings.ToLower(value))
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
*data.def = lowerValue != "" && lowerValue != "0" && lowerValue != "false" && lowerValue != "no"
|
WithError(err).
|
||||||
|
Warnf("ignoring container with invalid value for %s label", DockerRouterLabelDefault)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
data.def = &defaultValue
|
||||||
}
|
}
|
||||||
if key == DockerRouterLabelNetwork {
|
if key == DockerRouterLabelNetwork {
|
||||||
if data.network != nil {
|
if data.network != nil {
|
||||||
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Names}).
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
Warnf("ignoring container with duplicate %s label", DockerRouterLabelNetwork)
|
Warnf("ignoring container with duplicate %s label", DockerRouterLabelNetwork)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
data.network = new(string)
|
data.network = new(string)
|
||||||
*data.network = value
|
*data.network = value
|
||||||
}
|
}
|
||||||
|
if key == DockerRouterLabelAutoScaleUp {
|
||||||
|
autoScaleUp, err := strconv.ParseBool(strings.TrimSpace(value))
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
|
WithError(err).
|
||||||
|
Warnf("ignoring container with invalid value for %s label", DockerRouterLabelAutoScaleUp)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
data.autoScaleUp = autoScaleUp
|
||||||
|
}
|
||||||
|
if key == DockerRouterLabelAutoScaleDown {
|
||||||
|
autoScaleDown, err := strconv.ParseBool(strings.TrimSpace(value))
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
|
WithError(err).
|
||||||
|
Warnf("ignoring container with invalid value for %s label", DockerRouterLabelAutoScaleDown)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
data.autoScaleDown = autoScaleDown
|
||||||
|
}
|
||||||
|
if key == DockerRouterLabelAutoScaleAsleepMOTD {
|
||||||
|
data.autoScaleAsleepMOTD = value
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// probably not minecraft related
|
// probably not minecraft related
|
||||||
@@ -270,7 +406,7 @@ func (w *dockerWatcherImpl) parseContainerData(container *dockertypes.Container)
|
|||||||
}
|
}
|
||||||
|
|
||||||
if len(container.NetworkSettings.Networks) == 0 {
|
if len(container.NetworkSettings.Networks) == 0 {
|
||||||
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Names}).
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
Warnf("ignoring container, no networks found")
|
Warnf("ignoring container, no networks found")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -304,7 +440,7 @@ func (w *dockerWatcherImpl) parseContainerData(container *dockertypes.Container)
|
|||||||
// if there's more than one network on this container, we should require that the user specifies a network to avoid
|
// if there's more than one network on this container, we should require that the user specifies a network to avoid
|
||||||
// weird problems.
|
// weird problems.
|
||||||
if len(container.NetworkSettings.Networks) > 1 {
|
if len(container.NetworkSettings.Networks) > 1 {
|
||||||
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Names}).
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
Warnf("ignoring container, multiple networks found and none specified using label %s", DockerRouterLabelNetwork)
|
Warnf("ignoring container, multiple networks found and none specified using label %s", DockerRouterLabelNetwork)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -315,12 +451,21 @@ func (w *dockerWatcherImpl) parseContainerData(container *dockertypes.Container)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if data.ip == "" {
|
if data.ip == "" && container.State != nil && container.State.Running {
|
||||||
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Names}).
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
Warnf("ignoring container, unable to find accessible ip address")
|
Warnf("ignoring container, unable to find accessible ip address")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if container.State != nil && !container.State.Running {
|
||||||
|
if !w.config.autoScaleUp {
|
||||||
|
logrus.WithFields(logrus.Fields{"containerId": container.ID, "containerNames": container.Name}).
|
||||||
|
Warnf("ignoring container, not running and auto scale up is disabled")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
data.notRunning = true
|
||||||
|
}
|
||||||
|
|
||||||
ok = true
|
ok = true
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -329,4 +474,8 @@ func (w *dockerWatcherImpl) parseContainerData(container *dockertypes.Container)
|
|||||||
type routableContainer struct {
|
type routableContainer struct {
|
||||||
externalContainerName string
|
externalContainerName string
|
||||||
containerEndpoint string
|
containerEndpoint string
|
||||||
|
containerID string
|
||||||
|
autoScaleUp bool
|
||||||
|
autoScaleDown bool
|
||||||
|
autoScaleAsleepMOTD string
|
||||||
}
|
}
|
||||||
|
|||||||
+17
-11
@@ -38,17 +38,17 @@ type dockerSwarmWatcherImpl struct {
|
|||||||
client *client.Client
|
client *client.Client
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerSwarmWatcherImpl) makeWakerFunc(_ *routableService) ScalerFunc {
|
func (w *dockerSwarmWatcherImpl) makeWakerFunc(_ *routableService) WakerFunc {
|
||||||
if !w.config.autoScaleUp {
|
if !w.config.autoScaleUp {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return func(ctx context.Context) error {
|
return func(ctx context.Context) (string, error) {
|
||||||
logrus.Fatal("Auto scale up is not yet supported for docker swarm")
|
logrus.Fatal("Auto scale up is not yet supported for docker swarm")
|
||||||
return nil
|
return "", nil
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *dockerSwarmWatcherImpl) makeSleeperFunc(_ *routableService) ScalerFunc {
|
func (w *dockerSwarmWatcherImpl) makeSleeperFunc(_ *routableService) SleeperFunc {
|
||||||
if !w.config.autoScaleDown {
|
if !w.config.autoScaleDown {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -89,10 +89,12 @@ func (w *dockerSwarmWatcherImpl) Start(ctx context.Context) error {
|
|||||||
|
|
||||||
for _, s := range initialServices {
|
for _, s := range initialServices {
|
||||||
serviceMap[s.externalServiceName] = s
|
serviceMap[s.externalServiceName] = s
|
||||||
|
wakerFunc := w.makeWakerFunc(s)
|
||||||
|
sleeperFunc := w.makeSleeperFunc(s)
|
||||||
if s.externalServiceName != "" {
|
if s.externalServiceName != "" {
|
||||||
Routes.CreateMapping(s.externalServiceName, s.containerEndpoint, w.makeWakerFunc(s), w.makeSleeperFunc(s))
|
Routes.CreateMapping(s.externalServiceName, s.containerEndpoint, wakerFunc, sleeperFunc, "")
|
||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute(s.containerEndpoint)
|
Routes.SetDefaultRoute(s.containerEndpoint, wakerFunc, sleeperFunc, "")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -111,18 +113,22 @@ func (w *dockerSwarmWatcherImpl) Start(ctx context.Context) error {
|
|||||||
if oldRs, ok := serviceMap[rs.externalServiceName]; !ok {
|
if oldRs, ok := serviceMap[rs.externalServiceName]; !ok {
|
||||||
serviceMap[rs.externalServiceName] = rs
|
serviceMap[rs.externalServiceName] = rs
|
||||||
logrus.WithField("routableService", rs).Debug("ADD")
|
logrus.WithField("routableService", rs).Debug("ADD")
|
||||||
|
wakerFunc := w.makeWakerFunc(rs)
|
||||||
|
sleeperFunc := w.makeSleeperFunc(rs)
|
||||||
if rs.externalServiceName != "" {
|
if rs.externalServiceName != "" {
|
||||||
Routes.CreateMapping(rs.externalServiceName, rs.containerEndpoint, w.makeWakerFunc(rs), w.makeSleeperFunc(rs))
|
Routes.CreateMapping(rs.externalServiceName, rs.containerEndpoint, wakerFunc, sleeperFunc, "")
|
||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute(rs.containerEndpoint)
|
Routes.SetDefaultRoute(rs.containerEndpoint, wakerFunc, sleeperFunc, "")
|
||||||
}
|
}
|
||||||
} else if oldRs.containerEndpoint != rs.containerEndpoint {
|
} else if oldRs.containerEndpoint != rs.containerEndpoint {
|
||||||
serviceMap[rs.externalServiceName] = rs
|
serviceMap[rs.externalServiceName] = rs
|
||||||
|
wakerFunc := w.makeWakerFunc(rs)
|
||||||
|
sleeperFunc := w.makeSleeperFunc(rs)
|
||||||
if rs.externalServiceName != "" {
|
if rs.externalServiceName != "" {
|
||||||
Routes.DeleteMapping(rs.externalServiceName)
|
Routes.DeleteMapping(rs.externalServiceName)
|
||||||
Routes.CreateMapping(rs.externalServiceName, rs.containerEndpoint, w.makeWakerFunc(rs), w.makeSleeperFunc(rs))
|
Routes.CreateMapping(rs.externalServiceName, rs.containerEndpoint, wakerFunc, sleeperFunc, "")
|
||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute(rs.containerEndpoint)
|
Routes.SetDefaultRoute(rs.containerEndpoint, wakerFunc, sleeperFunc, "")
|
||||||
}
|
}
|
||||||
logrus.WithFields(logrus.Fields{"old": oldRs, "new": rs}).Debug("UPDATE")
|
logrus.WithFields(logrus.Fields{"old": oldRs, "new": rs}).Debug("UPDATE")
|
||||||
}
|
}
|
||||||
@@ -134,7 +140,7 @@ func (w *dockerSwarmWatcherImpl) Start(ctx context.Context) error {
|
|||||||
if rs.externalServiceName != "" {
|
if rs.externalServiceName != "" {
|
||||||
Routes.DeleteMapping(rs.externalServiceName)
|
Routes.DeleteMapping(rs.externalServiceName)
|
||||||
} else {
|
} else {
|
||||||
Routes.SetDefaultRoute("")
|
Routes.SetDefaultRoute("", nil, nil, "")
|
||||||
}
|
}
|
||||||
logrus.WithField("routableService", rs).Debug("DELETE")
|
logrus.WithField("routableService", rs).Debug("DELETE")
|
||||||
}
|
}
|
||||||
|
|||||||
+33
-26
@@ -10,17 +10,17 @@ import (
|
|||||||
|
|
||||||
type IDownScaler interface {
|
type IDownScaler interface {
|
||||||
Reset()
|
Reset()
|
||||||
Begin(serverAddress string)
|
Begin(backendEndpoint string)
|
||||||
Cancel(serverAddress string)
|
Cancel(backendEndpoint string)
|
||||||
}
|
}
|
||||||
|
|
||||||
var DownScaler IDownScaler
|
var DownScaler IDownScaler
|
||||||
|
|
||||||
func NewDownScaler(ctx context.Context, enabled bool, delay time.Duration) IDownScaler {
|
func NewDownScaler(ctx context.Context, enabled bool, delay time.Duration) IDownScaler {
|
||||||
ds := &downScalerImpl{
|
ds := &downScalerImpl{
|
||||||
enabled: enabled,
|
enabled: enabled,
|
||||||
delay: delay,
|
delay: delay,
|
||||||
parentContext: ctx,
|
parentContext: ctx,
|
||||||
contextCancellations: make(map[string]context.CancelFunc),
|
contextCancellations: make(map[string]context.CancelFunc),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -43,7 +43,7 @@ func (ds *downScalerImpl) Reset() {
|
|||||||
ds.contextCancellations = make(map[string]context.CancelFunc)
|
ds.contextCancellations = make(map[string]context.CancelFunc)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (ds *downScalerImpl) Begin(serverAddress string) {
|
func (ds *downScalerImpl) Begin(backendEndpoint string) {
|
||||||
ds.Lock()
|
ds.Lock()
|
||||||
defer ds.Unlock()
|
defer ds.Unlock()
|
||||||
|
|
||||||
@@ -52,17 +52,17 @@ func (ds *downScalerImpl) Begin(serverAddress string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// If an existing scale down routine exists, cancel it
|
// If an existing scale down routine exists, cancel it
|
||||||
if scaleDownCancel, ok := ds.contextCancellations[serverAddress]; ok {
|
if scaleDownCancel, ok := ds.contextCancellations[backendEndpoint]; ok {
|
||||||
scaleDownCancel()
|
scaleDownCancel()
|
||||||
}
|
}
|
||||||
|
|
||||||
logrus.WithField("serverAddress", serverAddress).Debug("Beginning scale down")
|
logrus.WithField("backendEndpoint", backendEndpoint).Debug("Beginning scale down")
|
||||||
scaleDownContext, scaleDownContextCancellation := context.WithCancel(ds.parentContext)
|
scaleDownContext, scaleDownContextCancellation := context.WithCancel(ds.parentContext)
|
||||||
ds.contextCancellations[serverAddress] = scaleDownContextCancellation
|
ds.contextCancellations[backendEndpoint] = scaleDownContextCancellation
|
||||||
go ds.scaleDown(scaleDownContext, serverAddress)
|
go ds.scaleDown(scaleDownContext, backendEndpoint)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (ds *downScalerImpl) Cancel(serverAddress string) {
|
func (ds *downScalerImpl) Cancel(backendEndpoint string) {
|
||||||
ds.Lock()
|
ds.Lock()
|
||||||
defer ds.Unlock()
|
defer ds.Unlock()
|
||||||
|
|
||||||
@@ -70,27 +70,34 @@ func (ds *downScalerImpl) Cancel(serverAddress string) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
if scaleDownContextCancellation, ok := ds.contextCancellations[serverAddress]; ok {
|
if scaleDownContextCancellation, ok := ds.contextCancellations[backendEndpoint]; ok {
|
||||||
logrus.WithField("serverAddress", serverAddress).Debug("Canceling scale down")
|
logrus.WithField("backendEndpoint", backendEndpoint).Debug("Canceling scale down")
|
||||||
scaleDownContextCancellation()
|
scaleDownContextCancellation()
|
||||||
delete(ds.contextCancellations, serverAddress)
|
delete(ds.contextCancellations, backendEndpoint)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (ds *downScalerImpl) scaleDown(ctx context.Context, serverAddress string) {
|
func (ds *downScalerImpl) scaleDown(ctx context.Context, backendEndpoint string) {
|
||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return
|
return
|
||||||
case <-time.After(ds.delay):
|
case <-time.After(ds.delay):
|
||||||
_, _, _, sleeper := Routes.FindBackendForServerAddress(ctx, serverAddress)
|
sleepers := Routes.GetSleepers(backendEndpoint)
|
||||||
if sleeper == nil {
|
if len(sleepers) == 0 {
|
||||||
return
|
|
||||||
}
|
|
||||||
if err := sleeper(ctx); err != nil {
|
|
||||||
logrus.WithField("serverAddress", serverAddress).WithError(err).Error("failed to scale down backend")
|
|
||||||
}
|
|
||||||
return
|
return
|
||||||
|
}
|
||||||
|
for _, sleeper := range sleepers {
|
||||||
|
go func(s SleeperFunc) {
|
||||||
|
err := s(ctx)
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithError(err).
|
||||||
|
WithField("backendEndpoint", backendEndpoint).
|
||||||
|
Error("Error while executing sleeper function")
|
||||||
|
}
|
||||||
|
}(sleeper)
|
||||||
|
}
|
||||||
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+38
-14
@@ -5,6 +5,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"net"
|
"net"
|
||||||
"strconv"
|
"strconv"
|
||||||
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
@@ -183,9 +184,9 @@ func (w *K8sWatcher) handleUpdate(oldObj interface{}, newObj interface{}) {
|
|||||||
"new": newRoutableService,
|
"new": newRoutableService,
|
||||||
}).Debug("UPDATE")
|
}).Debug("UPDATE")
|
||||||
if newRoutableService.externalServiceName != "" {
|
if newRoutableService.externalServiceName != "" {
|
||||||
w.routesHandler.CreateMapping(newRoutableService.externalServiceName, newRoutableService.containerEndpoint, newRoutableService.autoScaleUp, newRoutableService.autoScaleDown)
|
w.routesHandler.CreateMapping(newRoutableService.externalServiceName, newRoutableService.containerEndpoint, newRoutableService.autoScaleUp, newRoutableService.autoScaleDown, "")
|
||||||
} else {
|
} else {
|
||||||
w.routesHandler.SetDefaultRoute(newRoutableService.containerEndpoint)
|
w.routesHandler.SetDefaultRoute(newRoutableService.containerEndpoint, newRoutableService.autoScaleUp, newRoutableService.autoScaleDown, "")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -200,7 +201,7 @@ func (w *K8sWatcher) handleDelete(obj interface{}) {
|
|||||||
if routableService.externalServiceName != "" {
|
if routableService.externalServiceName != "" {
|
||||||
w.routesHandler.DeleteMapping(routableService.externalServiceName)
|
w.routesHandler.DeleteMapping(routableService.externalServiceName)
|
||||||
} else {
|
} else {
|
||||||
w.routesHandler.SetDefaultRoute("")
|
w.routesHandler.SetDefaultRoute("", nil, nil, "")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -214,9 +215,9 @@ func (w *K8sWatcher) handleAdd(obj interface{}) {
|
|||||||
logrus.WithField("routableService", routableService).Debug("ADD")
|
logrus.WithField("routableService", routableService).Debug("ADD")
|
||||||
|
|
||||||
if routableService.externalServiceName != "" {
|
if routableService.externalServiceName != "" {
|
||||||
w.routesHandler.CreateMapping(routableService.externalServiceName, routableService.containerEndpoint, routableService.autoScaleUp, routableService.autoScaleDown)
|
w.routesHandler.CreateMapping(routableService.externalServiceName, routableService.containerEndpoint, routableService.autoScaleUp, routableService.autoScaleDown, "")
|
||||||
} else {
|
} else {
|
||||||
w.routesHandler.SetDefaultRoute(routableService.containerEndpoint)
|
w.routesHandler.SetDefaultRoute(routableService.containerEndpoint, routableService.autoScaleUp, routableService.autoScaleDown, "")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -225,8 +226,8 @@ func (w *K8sWatcher) handleAdd(obj interface{}) {
|
|||||||
type routableService struct {
|
type routableService struct {
|
||||||
externalServiceName string
|
externalServiceName string
|
||||||
containerEndpoint string
|
containerEndpoint string
|
||||||
autoScaleUp ScalerFunc
|
autoScaleUp WakerFunc
|
||||||
autoScaleDown ScalerFunc
|
autoScaleDown SleeperFunc
|
||||||
}
|
}
|
||||||
|
|
||||||
// obj is expected to be a *v1.Service
|
// obj is expected to be a *v1.Service
|
||||||
@@ -271,22 +272,37 @@ func (w *K8sWatcher) buildDetails(service *core.Service, externalServiceName str
|
|||||||
} else if len(mcPort) > 0 {
|
} else if len(mcPort) > 0 {
|
||||||
port = mcPort
|
port = mcPort
|
||||||
}
|
}
|
||||||
|
endpoint := net.JoinHostPort(clusterIp, port)
|
||||||
|
wakerFunc := w.buildScaleFunction(service, 0, 1)
|
||||||
rs := &routableService{
|
rs := &routableService{
|
||||||
externalServiceName: externalServiceName,
|
externalServiceName: externalServiceName,
|
||||||
containerEndpoint: net.JoinHostPort(clusterIp, port),
|
containerEndpoint: endpoint,
|
||||||
autoScaleUp: w.buildScaleFunction(service, 0, 1),
|
autoScaleUp: func(ctx context.Context) (string, error) {
|
||||||
autoScaleDown: w.buildScaleFunction(service, 1, 0),
|
if err := wakerFunc(ctx); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return endpoint, nil
|
||||||
|
},
|
||||||
|
autoScaleDown: w.buildScaleFunction(service, 1, 0),
|
||||||
}
|
}
|
||||||
return rs
|
return rs
|
||||||
}
|
}
|
||||||
|
|
||||||
func (w *K8sWatcher) buildScaleFunction(service *core.Service, from int32, to int32) ScalerFunc {
|
func (w *K8sWatcher) buildScaleFunction(service *core.Service, from int32, to int32) SleeperFunc {
|
||||||
// Currently, annotations can only be used to opt-out of auto-scaling.
|
// Currently, annotations can only be used to opt-out of auto-scaling.
|
||||||
// However, this logic is prepared also for opt-in, as it returns a `ScalerFunc` when flags are false but annotations are set to `enabled`.
|
// However, this logic is prepared also for opt-in, as it returns a `SleeperFunc` when flags are false but annotations are set to `enabled`.
|
||||||
if from <= to {
|
if from <= to {
|
||||||
enabled, exists := service.Annotations[AnnotationAutoScaleUp]
|
enabled, exists := service.Annotations[AnnotationAutoScaleUp]
|
||||||
if exists {
|
if exists {
|
||||||
if enabled == "false" {
|
enabledBool, err := strconv.ParseBool(strings.TrimSpace(enabled))
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{"service": service.Name}).
|
||||||
|
WithError(err).
|
||||||
|
Warnf("invalid value for %s annotation - disabling service auto-scale-up", AnnotationAutoScaleUp)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if !enabledBool {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
@@ -298,7 +314,15 @@ func (w *K8sWatcher) buildScaleFunction(service *core.Service, from int32, to in
|
|||||||
if from >= to {
|
if from >= to {
|
||||||
enabled, exists := service.Annotations[AnnotationAutoScaleDown]
|
enabled, exists := service.Annotations[AnnotationAutoScaleDown]
|
||||||
if exists {
|
if exists {
|
||||||
if enabled == "false" {
|
enabledBool, err := strconv.ParseBool(strings.TrimSpace(enabled))
|
||||||
|
if err != nil {
|
||||||
|
logrus.WithFields(logrus.Fields{"service": service.Name}).
|
||||||
|
WithError(err).
|
||||||
|
Warnf("invalid value for %s annotation - disabling service auto-scale-down", AnnotationAutoScaleDown)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if !enabledBool {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
+20
-8
@@ -3,10 +3,11 @@ package server
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"github.com/stretchr/testify/mock"
|
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/mock"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
v1 "k8s.io/api/core/v1"
|
v1 "k8s.io/api/core/v1"
|
||||||
@@ -27,22 +28,27 @@ func (m *MockedRoutesHandler) GetBackendForServer(server string) string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m *MockedRoutesHandler) CreateMapping(serverAddress string, backend string, waker ScalerFunc, sleeper ScalerFunc) {
|
func (m *MockedRoutesHandler) CreateMapping(serverAddress string, backend string, waker WakerFunc, sleeper SleeperFunc, asleepMOTD string) {
|
||||||
m.MethodCalled("CreateMapping", serverAddress, backend, waker, sleeper)
|
m.MethodCalled("CreateMapping", serverAddress, backend, waker, sleeper, asleepMOTD)
|
||||||
if m.routes == nil {
|
if m.routes == nil {
|
||||||
m.routes = make(map[string]string)
|
m.routes = make(map[string]string)
|
||||||
}
|
}
|
||||||
m.routes[serverAddress] = backend
|
m.routes[serverAddress] = backend
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m *MockedRoutesHandler) SetDefaultRoute(backend string) {
|
func (m *MockedRoutesHandler) SetDefaultRoute(backend string, waker WakerFunc, sleeper SleeperFunc, asleepMOTD string) {
|
||||||
m.MethodCalled("SetDefaultRoute", backend)
|
m.MethodCalled("SetDefaultRoute", backend, waker, sleeper, asleepMOTD)
|
||||||
if m.routes == nil {
|
if m.routes == nil {
|
||||||
m.routes = make(map[string]string)
|
m.routes = make(map[string]string)
|
||||||
}
|
}
|
||||||
m.defaultBackend = backend
|
m.defaultBackend = backend
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (m *MockedRoutesHandler) GetAsleepMOTD(serverAddress string) string {
|
||||||
|
args := m.MethodCalled("GetAsleepMOTD", serverAddress)
|
||||||
|
return args.String(0)
|
||||||
|
}
|
||||||
|
|
||||||
func (m *MockedRoutesHandler) DeleteMapping(serverAddress string) bool {
|
func (m *MockedRoutesHandler) DeleteMapping(serverAddress string) bool {
|
||||||
args := m.MethodCalled("DeleteMapping", serverAddress)
|
args := m.MethodCalled("DeleteMapping", serverAddress)
|
||||||
if m.routes == nil {
|
if m.routes == nil {
|
||||||
@@ -177,7 +183,9 @@ func TestK8sWatcherImpl_handleAddThenUpdate(t *testing.T) {
|
|||||||
DownScaler = NewDownScaler(context.Background(), false, 1*time.Second)
|
DownScaler = NewDownScaler(context.Background(), false, 1*time.Second)
|
||||||
|
|
||||||
routesHandler := new(MockedRoutesHandler)
|
routesHandler := new(MockedRoutesHandler)
|
||||||
routesHandler.On("CreateMapping", mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
routesHandler.On("CreateMapping", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
||||||
|
routesHandler.On("SetDefaultRoute", mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
||||||
|
routesHandler.On("GetAsleepMOTD", mock.Anything).Return("")
|
||||||
routesHandler.On("DeleteMapping", mock.Anything).Return(true)
|
routesHandler.On("DeleteMapping", mock.Anything).Return(true)
|
||||||
|
|
||||||
watcher := &K8sWatcher{
|
watcher := &K8sWatcher{
|
||||||
@@ -256,7 +264,9 @@ func TestK8sWatcherImpl_handleAddThenDelete(t *testing.T) {
|
|||||||
DownScaler = NewDownScaler(context.Background(), false, 1*time.Second)
|
DownScaler = NewDownScaler(context.Background(), false, 1*time.Second)
|
||||||
|
|
||||||
routesHandler := new(MockedRoutesHandler)
|
routesHandler := new(MockedRoutesHandler)
|
||||||
routesHandler.On("CreateMapping", mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
routesHandler.On("CreateMapping", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
||||||
|
routesHandler.On("SetDefaultRoute", mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
||||||
|
routesHandler.On("GetAsleepMOTD", mock.Anything).Return("")
|
||||||
routesHandler.On("DeleteMapping", mock.Anything).Return(true)
|
routesHandler.On("DeleteMapping", mock.Anything).Return(true)
|
||||||
|
|
||||||
watcher := &K8sWatcher{
|
watcher := &K8sWatcher{
|
||||||
@@ -353,7 +363,9 @@ func TestK8s_externalName(t *testing.T) {
|
|||||||
DownScaler = NewDownScaler(context.Background(), false, 1*time.Second)
|
DownScaler = NewDownScaler(context.Background(), false, 1*time.Second)
|
||||||
|
|
||||||
routesHandler := new(MockedRoutesHandler)
|
routesHandler := new(MockedRoutesHandler)
|
||||||
routesHandler.On("CreateMapping", mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
routesHandler.On("CreateMapping", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
||||||
|
routesHandler.On("SetDefaultRoute", mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return()
|
||||||
|
routesHandler.On("GetAsleepMOTD", mock.Anything).Return("")
|
||||||
routesHandler.On("DeleteMapping", mock.Anything).Return(true)
|
routesHandler.On("DeleteMapping", mock.Anything).Return(true)
|
||||||
|
|
||||||
watcher := &K8sWatcher{
|
watcher := &K8sWatcher{
|
||||||
|
|||||||
+68
-23
@@ -9,9 +9,11 @@ import (
|
|||||||
"github.com/sirupsen/logrus"
|
"github.com/sirupsen/logrus"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ScalerFunc func(ctx context.Context) error
|
// WakerFunc is a function that wakes up a server and returns its address.
|
||||||
|
type WakerFunc func(ctx context.Context) (string, error)
|
||||||
|
|
||||||
var EmptyScalerFunc = func(ctx context.Context) error { return nil }
|
// SleeperFunc is a function that puts a server to sleep.
|
||||||
|
type SleeperFunc func(ctx context.Context) error
|
||||||
|
|
||||||
var tcpShieldPattern = regexp.MustCompile("///.*")
|
var tcpShieldPattern = regexp.MustCompile("///.*")
|
||||||
|
|
||||||
@@ -22,8 +24,8 @@ type RouteFinder interface {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type RoutesHandler interface {
|
type RoutesHandler interface {
|
||||||
CreateMapping(serverAddress string, backend string, waker ScalerFunc, sleeper ScalerFunc)
|
CreateMapping(serverAddress string, backend string, waker WakerFunc, sleeper SleeperFunc, asleepMOTD string)
|
||||||
SetDefaultRoute(backend string)
|
SetDefaultRoute(backend string, waker WakerFunc, sleeper SleeperFunc, asleepMOTD string)
|
||||||
// DeleteMapping requests that the serverAddress be removed from routes.
|
// DeleteMapping requests that the serverAddress be removed from routes.
|
||||||
// Returns true if the route existed.
|
// Returns true if the route existed.
|
||||||
DeleteMapping(serverAddress string) bool
|
DeleteMapping(serverAddress string) bool
|
||||||
@@ -38,9 +40,12 @@ type IRoutes interface {
|
|||||||
// Otherwise, an empty string is returned. Also returns the normalized version of the given serverAddress.
|
// Otherwise, an empty string is returned. Also returns the normalized version of the given serverAddress.
|
||||||
// The 3rd value returned is an (optional) "waker" function which a caller must invoke to wake up serverAddress.
|
// The 3rd value returned is an (optional) "waker" function which a caller must invoke to wake up serverAddress.
|
||||||
// The 4th value returned is an (optional) "sleeper" function which a caller must invoke to shut down serverAddress.
|
// The 4th value returned is an (optional) "sleeper" function which a caller must invoke to shut down serverAddress.
|
||||||
FindBackendForServerAddress(ctx context.Context, serverAddress string) (string, string, ScalerFunc, ScalerFunc)
|
HasRoute(serverAddress string) bool
|
||||||
|
FindBackendForServerAddress(ctx context.Context, serverAddress string) (string, string, WakerFunc, SleeperFunc)
|
||||||
|
GetSleepers(backend string) []SleeperFunc
|
||||||
GetMappings() map[string]string
|
GetMappings() map[string]string
|
||||||
GetDefaultRoute() string
|
GetDefaultRoute() (string, WakerFunc, SleeperFunc)
|
||||||
|
GetAsleepMOTD(serverAddress string) string
|
||||||
SimplifySRV(srvEnabled bool)
|
SimplifySRV(srvEnabled bool)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -56,20 +61,21 @@ func NewRoutes() IRoutes {
|
|||||||
|
|
||||||
func (r *routesImpl) RegisterAll(mappings map[string]string) {
|
func (r *routesImpl) RegisterAll(mappings map[string]string) {
|
||||||
for k, v := range mappings {
|
for k, v := range mappings {
|
||||||
r.CreateMapping(k, v, EmptyScalerFunc, EmptyScalerFunc)
|
r.CreateMapping(k, v, nil, nil, "")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
type mapping struct {
|
type mapping struct {
|
||||||
backend string
|
backend string
|
||||||
waker ScalerFunc
|
waker WakerFunc
|
||||||
sleeper ScalerFunc
|
sleeper SleeperFunc
|
||||||
|
asleepMOTD string
|
||||||
}
|
}
|
||||||
|
|
||||||
type routesImpl struct {
|
type routesImpl struct {
|
||||||
sync.RWMutex
|
sync.RWMutex
|
||||||
mappings map[string]mapping
|
mappings map[string]mapping
|
||||||
defaultRoute string
|
defaultRoute mapping
|
||||||
simplifySRV bool
|
simplifySRV bool
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -78,23 +84,45 @@ func (r *routesImpl) Reset() {
|
|||||||
DownScaler.Reset()
|
DownScaler.Reset()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *routesImpl) SetDefaultRoute(backend string) {
|
func (r *routesImpl) SetDefaultRoute(backend string, waker WakerFunc, sleeper SleeperFunc, asleepMOTD string) {
|
||||||
r.defaultRoute = backend
|
r.defaultRoute = mapping{backend: backend, waker: waker, sleeper: sleeper, asleepMOTD: asleepMOTD}
|
||||||
|
|
||||||
logrus.WithFields(logrus.Fields{
|
logrus.WithFields(logrus.Fields{
|
||||||
"backend": backend,
|
"backend": backend,
|
||||||
}).Info("Using default route")
|
}).Info("Using default route")
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *routesImpl) GetDefaultRoute() string {
|
func (r *routesImpl) GetDefaultRoute() (string, WakerFunc, SleeperFunc) {
|
||||||
return r.defaultRoute
|
return r.defaultRoute.backend, r.defaultRoute.waker, r.defaultRoute.sleeper
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *routesImpl) GetAsleepMOTD(serverAddress string) string {
|
||||||
|
r.RLock()
|
||||||
|
defer r.RUnlock()
|
||||||
|
|
||||||
|
if serverAddress == "" {
|
||||||
|
return r.defaultRoute.asleepMOTD
|
||||||
|
}
|
||||||
|
|
||||||
|
if m, ok := r.mappings[serverAddress]; ok {
|
||||||
|
return m.asleepMOTD
|
||||||
|
}
|
||||||
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *routesImpl) SimplifySRV(srvEnabled bool) {
|
func (r *routesImpl) SimplifySRV(srvEnabled bool) {
|
||||||
r.simplifySRV = srvEnabled
|
r.simplifySRV = srvEnabled
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *routesImpl) FindBackendForServerAddress(_ context.Context, serverAddress string) (string, string, ScalerFunc, ScalerFunc) {
|
func (r *routesImpl) HasRoute(serverAddress string) bool {
|
||||||
|
r.RLock()
|
||||||
|
defer r.RUnlock()
|
||||||
|
|
||||||
|
_, exists := r.mappings[serverAddress]
|
||||||
|
return exists
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *routesImpl) FindBackendForServerAddress(_ context.Context, serverAddress string) (string, string, WakerFunc, SleeperFunc) {
|
||||||
r.RLock()
|
r.RLock()
|
||||||
defer r.RUnlock()
|
defer r.RUnlock()
|
||||||
|
|
||||||
@@ -136,7 +164,23 @@ func (r *routesImpl) FindBackendForServerAddress(_ context.Context, serverAddres
|
|||||||
return mapping.backend, serverAddress, mapping.waker, mapping.sleeper
|
return mapping.backend, serverAddress, mapping.waker, mapping.sleeper
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return r.defaultRoute, serverAddress, nil, nil
|
return r.defaultRoute.backend, serverAddress, r.defaultRoute.waker, r.defaultRoute.sleeper
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *routesImpl) GetSleepers(backend string) []SleeperFunc {
|
||||||
|
r.RLock()
|
||||||
|
defer r.RUnlock()
|
||||||
|
|
||||||
|
var sleepers []SleeperFunc
|
||||||
|
for _, m := range r.mappings {
|
||||||
|
if m.backend == backend && m.sleeper != nil {
|
||||||
|
sleepers = append(sleepers, m.sleeper)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if r.defaultRoute.backend == backend && r.defaultRoute.sleeper != nil {
|
||||||
|
sleepers = append(sleepers, r.defaultRoute.sleeper)
|
||||||
|
}
|
||||||
|
return sleepers
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *routesImpl) GetMappings() map[string]string {
|
func (r *routesImpl) GetMappings() map[string]string {
|
||||||
@@ -155,9 +199,8 @@ func (r *routesImpl) DeleteMapping(serverAddress string) bool {
|
|||||||
defer r.Unlock()
|
defer r.Unlock()
|
||||||
logrus.WithField("serverAddress", serverAddress).Info("Deleting route")
|
logrus.WithField("serverAddress", serverAddress).Info("Deleting route")
|
||||||
|
|
||||||
DownScaler.Cancel(serverAddress)
|
if m, ok := r.mappings[serverAddress]; ok {
|
||||||
|
DownScaler.Cancel(m.backend)
|
||||||
if _, ok := r.mappings[serverAddress]; ok {
|
|
||||||
delete(r.mappings, serverAddress)
|
delete(r.mappings, serverAddress)
|
||||||
return true
|
return true
|
||||||
} else {
|
} else {
|
||||||
@@ -165,7 +208,7 @@ func (r *routesImpl) DeleteMapping(serverAddress string) bool {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *routesImpl) CreateMapping(serverAddress string, backend string, waker ScalerFunc, sleeper ScalerFunc) {
|
func (r *routesImpl) CreateMapping(serverAddress string, backend string, waker WakerFunc, sleeper SleeperFunc, asleepMOTD string) {
|
||||||
r.Lock()
|
r.Lock()
|
||||||
defer r.Unlock()
|
defer r.Unlock()
|
||||||
|
|
||||||
@@ -175,8 +218,10 @@ func (r *routesImpl) CreateMapping(serverAddress string, backend string, waker S
|
|||||||
"serverAddress": serverAddress,
|
"serverAddress": serverAddress,
|
||||||
"backend": backend,
|
"backend": backend,
|
||||||
}).Info("Created route mapping")
|
}).Info("Created route mapping")
|
||||||
r.mappings[serverAddress] = mapping{backend: backend, waker: waker, sleeper: sleeper}
|
r.mappings[serverAddress] = mapping{backend: backend, waker: waker, sleeper: sleeper, asleepMOTD: asleepMOTD}
|
||||||
|
|
||||||
// Trigger auto scale down when mapping is created to ensure servers are shut down if router restarts
|
// Trigger auto scale down when mapping is created to ensure servers are shut down if router restarts
|
||||||
DownScaler.Begin(serverAddress)
|
if DownScaler != nil && backend != "" {
|
||||||
|
DownScaler.Begin(backend)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ func (r *routesConfigLoader) Load(routesConfigFileName string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
Routes.RegisterAll(config.Mappings)
|
Routes.RegisterAll(config.Mappings)
|
||||||
Routes.SetDefaultRoute(config.DefaultServer)
|
Routes.SetDefaultRoute(config.DefaultServer, nil, nil, "")
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -62,7 +62,7 @@ func (r *routesConfigLoader) Reload() error {
|
|||||||
logrus.WithField("routesConfig", r.fileName).Info("Re-loading routes config file")
|
logrus.WithField("routesConfig", r.fileName).Info("Re-loading routes config file")
|
||||||
Routes.Reset()
|
Routes.Reset()
|
||||||
Routes.RegisterAll(config.Mappings)
|
Routes.RegisterAll(config.Mappings)
|
||||||
Routes.SetDefaultRoute(config.DefaultServer)
|
Routes.SetDefaultRoute(config.DefaultServer, nil, nil, "")
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -135,8 +135,9 @@ func (r *routesConfigLoader) SaveRoutes() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
server, _, _ := Routes.GetDefaultRoute()
|
||||||
err := r.writeFile(&RoutesConfigSchema{
|
err := r.writeFile(&RoutesConfigSchema{
|
||||||
DefaultServer: Routes.GetDefaultRoute(),
|
DefaultServer: server,
|
||||||
Mappings: Routes.GetMappings(),
|
Mappings: Routes.GetMappings(),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -66,7 +66,7 @@ func Test_routesImpl_FindBackendForServerAddress(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
r := NewRoutes()
|
r := NewRoutes()
|
||||||
|
|
||||||
r.CreateMapping(tt.mapping.serverAddress, tt.mapping.backend, EmptyScalerFunc, EmptyScalerFunc)
|
r.CreateMapping(tt.mapping.serverAddress, tt.mapping.backend, nil, nil, "")
|
||||||
|
|
||||||
if got, server, _, _ := r.FindBackendForServerAddress(context.Background(), tt.args.serverAddress); got != tt.want {
|
if got, server, _, _ := r.FindBackendForServerAddress(context.Background(), tt.args.serverAddress); got != tt.want {
|
||||||
t.Errorf("routesImpl.FindBackendForServerAddress() = %v, want %v", got, tt.want)
|
t.Errorf("routesImpl.FindBackendForServerAddress() = %v, want %v", got, tt.want)
|
||||||
|
|||||||
+4
-2
@@ -49,7 +49,7 @@ func NewServer(ctx context.Context, config *Config) (*Server, error) {
|
|||||||
|
|
||||||
metricsBuilder := NewMetricsBuilder(config.MetricsBackend, &config.MetricsBackendConfig)
|
metricsBuilder := NewMetricsBuilder(config.MetricsBackend, &config.MetricsBackendConfig)
|
||||||
|
|
||||||
downScalerEnabled := config.AutoScale.Down && (config.InKubeCluster || config.KubeConfig != "")
|
downScalerEnabled := config.AutoScale.Down && (config.InKubeCluster || config.KubeConfig != "" || config.InDocker)
|
||||||
downScalerDelay, err := time.ParseDuration(config.AutoScale.DownAfter)
|
downScalerDelay, err := time.ParseDuration(config.AutoScale.DownAfter)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not parse auto-scale-down-after duration: %w", err)
|
return nil, fmt.Errorf("could not parse auto-scale-down-after duration: %w", err)
|
||||||
@@ -73,7 +73,7 @@ func NewServer(ctx context.Context, config *Config) (*Server, error) {
|
|||||||
|
|
||||||
Routes.RegisterAll(config.Mapping)
|
Routes.RegisterAll(config.Mapping)
|
||||||
if config.Default != "" {
|
if config.Default != "" {
|
||||||
Routes.SetDefaultRoute(config.Default)
|
Routes.SetDefaultRoute(config.Default, nil, nil, "")
|
||||||
}
|
}
|
||||||
|
|
||||||
if config.ConnectionRateLimit < 1 {
|
if config.ConnectionRateLimit < 1 {
|
||||||
@@ -86,6 +86,8 @@ func NewServer(ctx context.Context, config *Config) (*Server, error) {
|
|||||||
config.RecordLogins,
|
config.RecordLogins,
|
||||||
autoScaleAllowDenyConfig)
|
autoScaleAllowDenyConfig)
|
||||||
|
|
||||||
|
connector.UseAsleepMOTD(config.AutoScale.AsleepMOTD)
|
||||||
|
|
||||||
clientFilter, err := NewClientFilter(config.ClientsToAllow, config.ClientsToDeny)
|
clientFilter, err := NewClientFilter(config.ClientsToAllow, config.ClientsToDeny)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not create client filter: %w", err)
|
return nil, fmt.Errorf("could not create client filter: %w", err)
|
||||||
|
|||||||
Reference in New Issue
Block a user