Skip to main content

Background

There are some s3 object key and s3 metadata changes after s3's preview version(from pre 0.2.0 version to 0.2.0 version). And we provide some tools to migrate from old version. Only use this if you are using old version of byconity and store data in s3.

There are 4 steps in total.

Usage

1. Stop bycontiy cluster

stop the cluster.

2. Migrate s3 data with provided tool

If you know where the clickhouse binary is(If you are running byconity with docker you can find it by login into server/worker pod and using ps command to find the clickhouse binary path)

./clickhouse s3-rename --s3_ak_id ${S3_AK_ID} --s3_ak_secret ${S3_AK_SECRET} --from_bucket ${BYCONITY_S3_BUCKET} --s3_endpoint ${S3_ENDPOINT} --s3_region ${S3_REGION} --root_prefix ${BYCONTIY_DATA_PREFIX}

Or if you are running byconity with docker wrapper

docker run -it --rm --network host --name byconity-s3-sanitizer byconity/byconity:0.2.0 clickhouse s3-rename --s3_ak_id ${S3_AK_ID} --s3_ak_secret ${S3_AK_SECRET} --from_bucket ${BYCONITY_S3_BUCKET} --s3_endpoint ${S3_ENDPOINT} --s3_region ${S3_REGION} --root_prefix ${BYCONTIY_DATA_PREFIX}
tool argumentsexplanation
s3_ak_ids3 access key id which byconity is using
s3_ak_secrets3 secret access key which byconity is using
s3_regions3 region which byconity is using
s3_endpoins3 endpoint which byconity is using
from_buckets3 bucket name which byconity is using
root_prefixs3 object prefix which byconity is using
thread_numberusing how many threads, default to 1 thread
need_deletewhether delete origin file, default true
uuid_checkwhether check uuid is valid or not, default true
enable_loggingwhether enable logging output
logging_levellogging level, default is debug output

3. Migrate s3 metadata with provided tool

If you know where the clickhouse binary is

./clickhouse s3-meta-sanitizer -C cnch_config.xml

If you are running byconity with docker wrapper

docker run -it --rm --network host --name byconity-s3-sanitizer byconity/byconity:0.2.0 clickhouse s3-meta-sanitizer --config-file /root/app/config/cnch_config.xml
tool argumentsexplanation
Cspecify config file from cnch_config configuration field in your bycontiy server configuration file

4. Start byconity cluster

start the cluster.