基于docker搭建mongodb分片集群

什么是mognodb分片集群

官方描述

Sharding is a method for distributing data across multiple machines. MongoDB uses sharding to support deployments with very large data sets and high throughput operations.

自己的理解

横向拓展，解决单台服务器磁盘不够用的问题
将写入压力分散到各个分片

为什么要使用分片集群

复制所有的写入操作到主节点
延迟的敏感数据会在主节点查询
单个副本集限制在12个节点
当请求量巨大时会出现内存不足
本地磁盘不足
垂直扩展价格昂贵

基于docker搭建最简单的分片集群

再此之前应该先了解mongodb副本集搭建的原理，详情参考这篇文章

基本思路是这个

mongodb分片集群

以下全部代码都在这里

集群中各个角色的说明

配置服务器
- 是一个独立的mongod进程，保存集群和分片的元数据，即各分片包含了哪些数据的信息。最先开始建立，启用日志功能。像启动普通的mongod一样启动配置服务器，指定configsvr选项。不需要太多的空间和资源，配置服务器的1KB空间相当于真是数据的200MB。保存的只是数据的分布表。当服务不可用，则变成只读，无法分块、迁移数据。
- 这里使用一个由三个节点（一个PRIMARY，两个SECONDARY）组成的config server副本集
路由服务器
- 即mongos，起到一个路由的功能，供程序连接。本身不保存数据，在启动时从配置服务器加载集群信息，开启mongos进程需要知道配置服务器的地址，指定configdb选项。
- 这里用了三个独立的mongos，避免出现mongos挂掉之后无法操作数据库的情况
分片服务器
- 一个独立普通的mongod进程，保存数据信息。可以是一个副本集也可以是单独的一台服务器
- 这里用了3个分片，每一个分片由一个三个节点（一个PRIMARY，两个SECONDARY）组成的mongodb副本集构成，防止单点故障

环境说明

docker version

Client:
	Version:      1.12.0
	API version:  1.24
	Go version:   go1.6.3
	Git commit:   8eab29e
	Built:        Thu Jul 28 21:15:28 2016
	OS/Arch:      darwin/amd64

Server:
	Version:      1.12.0
	API version:  1.24
	Go version:   go1.6.3
	Git commit:   8eab29e
	Built:        Thu Jul 28 23:54:00 2016
	OS/Arch:      linux/amd64

docker-compose version

docker-compose version 1.8.0, build f3628c7
docker-py version: 1.9.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.2h  3 May 2016

mongo, 用的版本较新，稳定版是3.2

1 2	MongoDB shell version: 3.3.10 MongoDB server version: 3.3.10

编写执行脚本的`Dockerfile`

集群搭建的过程中需要进入mongo客户端执行命令，个人比较喜欢run一个container去做这些事情

搭建副本集的Dockerfile

FROM mongo:3.3
MAINTAINER "kivi"

RUN mkdir -p /opt/mongo-replset-script
COPY ./replset.sh /opt/mongo-replset-script
WORKDIR /opt/mongo-replset-script
RUN chmod +x replset.sh

搭建副本集的脚本replset.sh

#!/bin/bash

# -m PRIMARY节点的ip:port
# -s SECONDARY节点的ip:port
# -a ARBITER节点的ip:port
# -r replset名称

MASTER_ADDRESS=
SLAVE_ADDRESS=
ARBITER_ADDRESS=
SETNAME=

while getopts ":qvm:s:a:r:" OPTION 
do 
    case "$OPTION" in 
    "q") 
        echo "quiet" 
        ;; 
    "v") 
        echo "verbose" 
        ;; 
    "m") 
        # echo "Option $OPTION has value $OPTARG"
        MASTER_ADDRESS=$OPTARG
        ;; 
    "s") 
        # echo "Option $OPTION has value $OPTARG" 
        SLAVE_ADDRESS=$OPTARG
        ;; 
    "a") 
        # echo "Option $OPTION has value $OPTARG" 
        ARBITER_ADDRESS=$OPTARG
        ;; 
    "r") 
        # echo "Option $OPTION has value $OPTARG" 
        SETNAME=$OPTARG
        ;;
    "?") 
        echo "Unknown option $OPTARG" 
        ;; 
    ":") 
        echo "No argument value for option $OPTARG" 
        ;; 
    *) 
        # Should not occur 
        echo "Unknown error while processing options" 
        ;; 
    esac 
done

mongo $MASTER_ADDRESS/admin --eval "rs.initiate({\"_id\": \"$SETNAME\", \"members\": [{\"_id\": 0, \"host\": \"$MASTER_ADDRESS\"}]});"

# secondary
hosts=$(echo $SLAVE_ADDRESS | tr "," "\n")

for hst in $hosts
do
  mongo $MASTER_ADDRESS/admin --eval "rs.add(\"$hst\")";
done

# arbiter
arbiters=$(echo $ARBITER_ADDRESS | tr "," "\n")

for arbiter in $arbiters
do
  mongo $MASTER_ADDRESS/admin --eval "rs.addArb(\"$arbiter\")";
done

给mongos添加分片的Dockerfile

FROM mongo:3.3
MAINTAINER "kivi"

RUN apt-get update && apt-get install -y wget
ENV DOCKERIZE_VERSION v0.2.0
RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
    && tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz

RUN mkdir -p /opt/mongo-shard-script
COPY ./shard.sh /opt/mongo-shard-script
WORKDIR /opt/mongo-shard-script
RUN chmod +x shard.sh

给mongos添加分片的脚本shard.sh

#!/bin/bash

# -a mongos的ip:port
# -r 所有分片的信息
# -d 需要分片的数据库(暂时没用)
# -t 需要分片的collection(暂时没用)
# -k 片键(暂时没用)

ADDRESS=
REPLSET=
DATABASE=
TABLE=
KEY=


while getopts ":qva:r:d:t:k:" OPTION 
do 
    case "$OPTION" in 
    "q") 
        echo "quiet" 
        ;; 
    "v") 
        echo "verbose" 
        ;; 
    "a") 
        # echo "Option $OPTION has value $OPTARG"
        ADDRESS=$OPTARG
        ;; 
    "r") 
        # echo "Option $OPTION has value $OPTARG" 
        REPLSET=$OPTARG
        ;; 
    "d") 
        # echo "Option $OPTION has value $OPTARG" 
        DATABASE=$OPTARG
        ;; 
    "t") 
        # echo "Option $OPTION has value $OPTARG" 
        TABLE=$OPTARG
        ;;
    "k") 
        # echo "Option $OPTION has value $OPTARG" 
        KEY=$OPTARG
        ;;
    "?") 
        echo "Unknown option $OPTARG" 
        ;; 
    ":") 
        echo "No argument value for option $OPTARG" 
        ;; 
    *) 
        # Should not occur 
        echo "Unknown error while processing options" 
        ;; 
    esac 
done 

dockerize -wait tcp://$ADDRESS

# secondary
replsets=$(echo $REPLSET | tr "|" "\n")

for replset in $replsets
do
  mongo $ADDRESS/admin --eval "db.runCommand({\"addshard\": \"$replset\"});"
done

编写一键启动的`docker-compose.yml`，然后`docker-compose up -d`就OK了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
version: "2"
services:
  # 基于replset构建3个shard
  mongo1-1:
    image: mongo:3.3
    container_name: "mongo1-1"
    network_mode: "mongo-cluster"
    ports:
      - "30010:27017"
    command: mongod --replSet shard1
  mongo1-2:
    image: mongo:3.3
    container_name: "mongo1-2"
    network_mode: "mongo-cluster"
    ports:
      - "30011:27017"
    command: mongod --replSet shard1
  mongo1-3:
    image: mongo:3.3
    container_name: "mongo1-3"
    network_mode: "mongo-cluster"
    ports:
      - "30012:27017"
    command: mongod --replSet shard1
  mongo-repl1:
    image: mongo-replset
    container_name: mongo-repl1
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongo1-1"
      - "mongo1-2"
      - "mongo1-3"
    command: ./replset.sh -m mongo1-1:27017 -s mongo1-2:27017 -a mongo1-3:27017 -r shard1
  mongo2-1:
    image: mongo:3.3
    container_name: "mongo2-1"
    network_mode: "mongo-cluster"
    ports:
      - "30013:27017"
    command: mongod --replSet shard2
  mongo2-2:
    image: mongo:3.3
    container_name: "mongo2-2"
    network_mode: "mongo-cluster"
    ports:
      - "30014:27017"
    command: mongod --replSet shard2
  mongo2-3:
    image: mongo:3.3
    container_name: "mongo2-3"
    network_mode: "mongo-cluster"
    ports:
      - "30015:27017"
    command: mongod --replSet shard2
  mongo-repl2:
    image: mongo-replset
    container_name: mongo-repl2
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongo2-1"
      - "mongo2-2"
      - "mongo2-3"
    command: ./replset.sh -m mongo2-1:27017 -s mongo2-2:27017 -a mongo2-3:27017 -r shard2
  mongo3-1:
    image: mongo:3.3
    container_name: "mongo3-1"
    network_mode: "mongo-cluster"
    ports:
      - "30016:27017"
    command: mongod --replSet shard3
  mongo3-2:
    image: mongo:3.3
    container_name: "mongo3-2"
    network_mode: "mongo-cluster"
    ports:
      - "30017:27017"
    command: mongod --replSet shard3
  mongo3-3:
    image: mongo:3.3
    container_name: "mongo3-3"
    network_mode: "mongo-cluster"
    ports:
      - "30018:27017"
    command: mongod --replSet shard3
  mongo-repl3:
    image: mongo-replset
    container_name: mongo-repl3
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongo3-1"
      - "mongo3-2"
      - "mongo3-3"
    command: ./replset.sh -m mongo3-1:27017 -s mongo3-2:27017 -a mongo3-3:27017 -r shard3
  # 构建config server副本集
  mongo-config1:
    image: mongo:3.3
    container_name: "mongo-config1"
    network_mode: "mongo-cluster"
    ports:
      - "30001:27019"
    command: mongod --configsvr --replSet config
  mongo-config2:
    image: mongo:3.3
    container_name: "mongo-config2"
    network_mode: "mongo-cluster"
    ports:
      - "30002:27019"
    command: mongod --configsvr --replSet config
  mongo-config3:
    image: mongo:3.3
    container_name: "mongo-config3"
    network_mode: "mongo-cluster"
    ports:
      - "30003:27019"
    command: mongod --configsvr --replSet config
  mongo-repl-config:
    image: mongo-replset
    container_name: mongo-repl-config
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongo-config1"
      - "mongo-config2"
      - "mongo-config3"
    command: ./replset.sh -m mongo-config1:27019 -s mongo-config2:27019,mongo-config3:27019 -r config
  # 启动3个mongos1
  mongos1:
    image: mongo:3.3
    container_name: "mongos1"
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongo-repl-config"
      - "mongo-repl1"
      - "mongo-repl2"
      - "mongo-repl3"
    ports:
      - "30004:27017"
    command: mongos --configdb config/mongo-config1:27019,mongo-config2:27019,mongo-config3:27019
  mongo-shard-config1:
    image: mongo-shard
    container_name: mongo-shard-config1
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongos1"
    command: ./shard.sh -a mongos1:27017 -r shard1/mongo1-1:27017,mongo1-2:27017,mongo1-3:27017|shard2/mongo2-1:27017,mongo2-2:27017,mongo2-3:27017|shard3/mongo3-1:27017,mongo3-2:27017,mongo3-3:27017 -d local -t user -k _id
  mongos2:
    image: mongo:3.3
    container_name: "mongos2"
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongo-repl-config"
      - "mongo-repl1"
      - "mongo-repl2"
      - "mongo-repl3"
    ports:
      - "30005:27017"
    command: mongos --configdb config/mongo-config1:27019,mongo-config2:27019,mongo-config3:27019
  mongo-shard-config2:
    image: mongo-shard
    container_name: mongo-shard-config2
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongos2"
      - "mongo-shard-config1"
    command: ./shard.sh -a mongos2:27017 -r shard1/mongo1-1:27017,mongo1-2:27017,mongo1-3:27017|shard2/mongo2-1:27017,mongo2-2:27017,mongo2-3:27017|shard3/mongo3-1:27017,mongo3-2:27017,mongo3-3:27017 -d local -t user -k _id
  mongos3:
    image: mongo:3.3
    container_name: "mongos3"
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongo-repl-config"
      - "mongo-repl1"
      - "mongo-repl2"
      - "mongo-repl3"
    ports:
      - "30006:27017"
    command: mongos --configdb config/mongo-config1:27019,mongo-config2:27019,mongo-config3:27019
  mongo-shard-config3:
    image: mongo-shard
    container_name: mongo-shard-config3
    network_mode: "mongo-cluster"
    depends_on: 
      - "mongos3"
      - "mongo-shard-config1"
      - "mongo-shard-config2"
    command: ./shard.sh -a mongos3:27017 -r shard1/mongo1-1:27017,mongo1-2:27017,mongo1-3:27017|shard2/mongo2-1:27017,mongo2-2:27017,mongo2-3:27017|shard3/mongo3-1:27017,mongo3-2:27017,mongo3-3:27017 -d local -t user -k _id

配置需要分片的`db`和`collection`

这里暂时还没有使用docker container去执行，执行下面的命令就可以了

连接数据库

1	docker run -it --network mongo-cluster --rm mongo sh -c 'exec mongo mongos3:27017/admin'

添加分片数据库

1	db.runCommand({enablesharding: "test"});

添加分片collection

1	db.runCommand({shardcollection: "test.users",key: {_id: 1}});

插入100000条数据

1	for (var i = 1; i <= 100000; i++) { db.users.save({name: "kivi", "sex": "male"}); }

查看数据分片情况

1	db.users.stats();

参考链接

mongodb官网
 MongoDB 分片
 MongoDB 分片的原理、搭建、应用

什么是mognodb分片集群

为什么要使用分片集群

基于docker搭建最简单的分片集群

基本思路是这个

集群中各个角色的说明

环境说明

编写执行脚本的Dockerfile

编写一键启动的docker-compose.yml，然后docker-compose up -d就OK了

配置需要分片的db和collection

参考链接

编写执行脚本的`Dockerfile`

编写一键启动的`docker-compose.yml`，然后`docker-compose up -d`就OK了

配置需要分片的`db`和`collection`