python脚本备份数据到OSS

需求背景

客户需要将数据备份到对象存储（windows环境）
最初使用的是osscmd工具(此工具目前已下线)将压缩的数据上传到oss，此方法是先利用bat脚本对目标路径进行压缩打包，然后使用osscmd上传，再进行上传检查告警通知。
目前改良后使用python压缩以及调用api上传数据，根据时间创建对应oss目录。

一些步骤

• 安装环境（windows）

1. 安装python3
https://www.python.org/ftp/python/3.8.1/python-3.8.1-amd64.exe
配置系统环境变量
2. 安装相关模块
pip install oss2
pip install pyyaml
3. 安装压缩软件
此处使用的winara，在配置文件config.yaml中设置winara路径。

• 修改配置文件

  根据自身环境情况设置

• 配置任务计划

  根据自身需求配置任务计划

测试

相关代码

# config.yaml
# 配置文件
# 客户名称
name: liyk
# 压缩命令路径
rar_path: D:\Program Files (x86)\WinRAR\WinRAR.exe
# 压缩目标目录
compressdir: D:\\liyk
# 压缩目的路径
compress_dst: D:\\
# 压缩名
#compressname:

# oss公网url
ossurl: http://oss-cn-xxx.aliyuncs.com
# bucket名
ossbucket: xxzx-test
# 阿里云api密钥
keyid: LTAIhf******6tW
keysecret: GwfAMvR********gRfAso

# oss_oper.py
#! /usr/bin/env python
# -*- coding:utf-8 -*-
# Auther: liyk time:2018/12/11
# File : oss_oper.py
import oss2
import yaml
import os
import datetime,time
import logging

# 日志配置
LOG_FORMAT = "%(asctime)s - %(levelname)s - %(message)s"
DATE_FORMAT = "%m/%d/%Y %H:%M:%S %p"
logging.basicConfig(filename='oss_upload.log', level=logging.DEBUG, format=LOG_FORMAT, datefmt=DATE_FORMAT)

# 获取配置文件
curpath = os.path.dirname(os.path.realpath(__file__))
yamlpath = os.path.join(curpath, "config.yaml")
f = open(yamlpath, 'r', encoding='utf-8')
cfg = f.read()
d = yaml.load(cfg, Loader=yaml.FullLoader)
f.close()

# 定义上传对象名和上传文件名
i = datetime.datetime.now()
now_date = "{}/{}".format(i.year,i.month)
name = d.get('name')
ObjName = f'{name}/{now_date}/{name}-{i.month}-{i.day}.rar'
LocalFile = f'{name}.rar'
print(LocalFile)
# 压缩目的目录和压缩名
target_dir = d.get('compress_dst')
target = target_dir + d.get('name') + time.strftime('%m-%d') + '.rar'

def Auth(id, key):
    # 认证
    Auth = oss2.Auth(id, key)
    bucket = oss2.Bucket(Auth, d.get('ossurl'), d.get('ossbucket'))
    return bucket

def upload(bucket, ObjName, target):
    # 端口续传 上传文件
    logging.info("Start uploading...")
    oss2.resumable_upload(bucket, '%s' %ObjName, "%s" %target,
                          store=oss2.ResumableStore(root='D:\\'),
                          multipart_threshold=20000*1024,
                          part_size=10000*1024,
                          num_threads=4)
    logging.info("Backup upload completed")

def list_object(bucket):
    # 列出文件
    obj_list = []
    for obj in oss2.ObjectIterator(bucket, delimiter='/'):
        if obj.is_prefix():
            print('dir:' + obj.key)
        else:
            obj_list.append(obj.key)
    return obj_list

def delete_obj(bucket,obj_list):
    # 删除文件
    for i in obj_list:
        bucket.delete_object('%s' %i)

def compress_data():
    # 压缩
    source = d.get('compressdir')
    rar_path = d.get('rar_path')
    rar_command = f'"{rar_path}" a {target} {source}'
    # 判断压缩文件是否存在
    if os.path.exists(target):
        logging.info("file exists!")
        pass
    else:
        os.system(rar_command)
        logging.info("Data compression successful")

if __name__ == "__main__":
    id = d.get('keyid')
    key = d.get('keysecret')
    compress_data()
    Bucket = Auth(id, key)
    upload(Bucket, ObjName, target)

    # obj_list = list_object(Bucket)
    # delete_obj(Bucket, obj_list)