集群模式需要有关注这块，如果是all-in-one模式，你部署多个节点，这不叫集群模式，它会有多种状态信息不一致，会有多种问题，如果希望将网关tw,平台acp(admin,consumer,publisher)，密钥管理km，限流管理tm分开部署多份，需要使用Distributed Deployment 模式(https://apim.docs.wso2.com/en/latest/install-and-setup/setup/kubernetes-deployment/kubernetes/am-pattern-3-acp-tm-gw/)。

• https://www.yenlo.com/blogs/custom-rate-limiting-with-wso2-apim/
• https://apim.docs.wso2.com/en/4.1.0/design/rate-limiting/advanced-topics/configuring-rate-limiting-api-gateway-cluster/
• https://apim.docs.wso2.com/en/latest/manage-apis/design/rate-limiting/advanced-topics/configuring-rate-limiting-api-gateway-cluster/
• https://github.com/wso2/api-manager/issues/3961
• https://github.com/wso2/api-manager/issues/359

helm-distribute分布式部署

部署顺序

1. control-plane
2. gateway
3. traffic-manager
4. key-manager[未在私有仓库找到景象，目前它集成到control-plane里]

各子服务Dockerfile添加jdbc

FROM wso2/wso2am-tm:4.5.0

ARG USER=wso2carbon
ARG USER_HOME=/home/${USER}
ARG WSO2_SERVER_NAME=wso2am-tm
ARG WSO2_SERVER_VERSION=4.5.0
ARG WSO2_SERVER=${WSO2_SERVER_NAME}-${WSO2_SERVER_VERSION}
ARG WSO2_SERVER_HOME=${USER_HOME}/${WSO2_SERVER}

# Copy JDBC MySQL driver
ADD --chown=wso2carbon:wso2 https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar ${WSO2_SERVER_HOME}/repository/components/lib

gateway中配置acp的服务名和实例名

wso2:
 apim:
   configurations
      throttling:
        # -- Traffic manager loadbalancer service url for throttling.
        # If the traffic manager is separated from control-plane, all urls should be pointed to traffic-manager ones.
        serviceUrl: "wso2am-acp-service" # 对应control-plane的主服务名
      eventhub:
        # -- Event hub (control plane) enabled.
        # This should be enabled if the traffic manager is separated from the control-plane.
        enabled: false
        # -- Event hub (control plane) loadbalancer service url
        serviceUrl: "wso2am-acp-service"

多节点集群

1. cp(control-plane) 支持多副本，多个deployment
2. gateway 无状态，直接使用deployment多副本功能
3. traffic-manager 支持多副本，多个deployment

redis throttle相关

相关动态

Update 02/07/2025

• Until now, Redis was only tested in an all-in-one deployment. Since the actual requirement is to support a multi-Traffic Manager (TM) node deployment, set up a distributed APIM deployment—with support from @kavindasr —consisting of 2 ACP nodes, 2 GW nodes, 2 TM nodes, and a Redis Cluster hosted in AWS.

• To support the Redis Cluster, I refactored the implementation to use JedisCluster instead of JedisPooled, and verified the functionality in this distributed setup.

• Initially, a configuration mismatch caused the Redis count to be incremented multiple times per API request—once by each Traffic Manager. After identifying the issue, adjusted the configuration to ensure correct and consistent counter updates.

redis配置

/deployment.toml

[apim.redis_config]
host = "localhost"
port = "6379"
gateway_id="my_gateway_1234"

[throttle_properties]
'throttling.distributed.counter.type' = "redis"
'throttling.sync-async_hybrid_mode.enable' = true
'throttling.local_quota_buffer_percentage' = 30

限流throttling的redis的pub/sub

通过使用redis的pub/sub功能，对某个频道wso2_sync_mode_init_channel进行订阅，订阅中会统计数量

查看当前活跃的频道

C:UsersUser>redis-cli PUBSUB CHANNELS
1) "wso2_sync_mode_init_channel"

查看某个频道订阅的数量

C:UsersUser>redis-cli PUBSUB NUMSUB  wso2_sync_mode_init_channel
1) "wso2_sync_mode_init_channel"
2) (integer) 1