通过redis扩展分布式存储fastdfs的数据对应及方案

简介:

前言:

   周天被朋友拖住了,他说最近要实现一个文件分享的网站,一个是图片,一个是特定的文件,比如压缩包、pdf这类的。 他是做idc的,手里的资源还是不少的, 想通过文件分享网站来导入点所谓的流量。哥年轻时跟他搞过一个php的淘宝图片站,结果有个卖衣服的站点,流量很大,成本太高了,再说也没钱再次投入了,也就放弃 ~   没赚多少钱,倒是赔给那几个淘宝卖家100多块钱。  想起往事,还算有趣的 ~

现在的我以前不是当初菜鸟的我了,google了半天,从别地方,拔下来一个html模板,而且还是很高端的样子,后端是用pyton的tornado框架改了改,跑了几个实例都是没有问题的 ~


实现的方法简单的

上传

建立表单,文件上传,然后写如到tmp目录的下,调用自己写的python fastdfs模块上传文件。也可以放到目录后,用inotify 来做文件的监控,触发文件后,就上传到fastdfs分布式存储里面。

下载

可以通过fastdfs本身的web做下载连接,但是为了更高的性能,推荐使用nginx fastdfs模块来实现的下载,给朋友也是这么做的。


(还没有正式的上线,嘿嘿)

subprocess 调用系统的fastdfs客户端

1
上传文件:/usr/local/bin/fdfs_upload_file  <config_file> <local_filename>


tornado的文件上传api

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def post(self):
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
     upload_path=os.path.join(os.path.dirname(__file__), 'files' )  #文件的暂存路径
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
     file_metas=self.request.files[ 'file' ]    #提取表单中‘name’为‘file’的文件元数据
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
     for  meta  in  file_metas:
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
         filename=meta[ 'filename' ]
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
         filepath=os.path.join(upload_path,filename)
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
         with  open(filepath, 'wb' as  up:      #有些文件需要已二进制的形式存储,实际中可以更改
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
             up.write(meta[ 'body' ])
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
         self.write( 'finished!' )
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
app=tornado.web.Application([
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
     (r '/file' ,UploadFileHandler),
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
])


wKiom1LJmfKi7TBIAAE4agppxMA452.jpg


一般来说,上传后得到的文件名字,是一串hash,和以前的名字比起来不太一样。


     举个例子,我明明上传的文件时 xiaorui.jpg,结果fastdfs接口给我返回的是 一堆的hash ,这是在让人有点小郁闷。

    我这里还只是图片的应用,后期文件功能出来后,让人看到的都是hash,你都不能通过hash想象出他的文件。

wKioL1LJogihA8GBAAA1TDtMvdE176.jpg


查看了uc那边的技术博客,扩展了一套简单的路子。


这边用到的是   nginx tornado  jquery bootstrap


1.   应用系统在上传文件到FastDFS成功时将原始文件名和“文件索引(FID)”保存下来,我这里会记录在redis里面。

2.  在浏览器下载的时用Nginx的域名和FID拼出url,在url后面增加一个参数,指定原始文件名。例如: xxx.xx.xx.xx/group2/M00/00/89/eQ6h3FKJf_PRl8p4AUz4wO8tqaA688.apk?attname=xiaorui.cc


3. 在Nginx上进行如下配置,这样Nginx就会截获url中的参数attname,在Http响应头里面加上字段 Content-Disposition “attachment;filename=$arg_attname”。



1
2
3
4
5
6
7
8
9
10
11
12
13
14
location /group1/M00 {
root /data/store/data1;
if  ($arg_attname ~  "^(.*).cc" ) {
     add_header Content-Disposition  "attachment;filename=$arg_attname" ;
}
ngx_fastdfs_module;
}
location /group2/M00 {
root /data/store/data2;
if  ($arg_attname ~  "^(.*).cc" ) {
     add_header Content-Disposition  "attachment;filename=$arg_attname" ;
}
ngx_fastdfs_module;
}


4.   浏览器发现响应头里面有Content-Disposition “attachment;filename=$arg_attname”时,就会把文件名显示成filename指定的名称。


大家可以测试下,原来就是给用户的url多出一个get参数,用nginx识别到之后,插入到header头信息Content-Disposition 协议。


到半夜为止,和哥们一块已经实现了该项目的大部门网络框架,当然前端的nginx


是用kvm虚拟机做的,毕竟7层对io需求不是很大。针对nginx做的轮训是在代码


层面实现的。


wKiom1LJq5KzMMsmAAC8ES5PABE643.jpg



好了,就这样吧。 这fastdfs真是个好东西 ! 特别适合以文件为载体的在线服


务,如图片站,文件分享等等 !






 本文转自 rfyiamcool 51CTO博客,原文链接:http://blog.51cto.com/rfyiamcool/1348839,如需转载请自行联系原作者




相关文章
|
11月前
|
存储 负载均衡 NoSQL
【赵渝强老师】Redis Cluster分布式集群
Redis Cluster是Redis的分布式存储解决方案,通过哈希槽(slot)实现数据分片,支持水平扩展,具备高可用性和负载均衡能力,适用于大规模数据场景。
752 2
|
11月前
|
存储 缓存 NoSQL
【📕分布式锁通关指南 12】源码剖析redisson如何利用Redis数据结构实现Semaphore和CountDownLatch
本文解析 Redisson 如何通过 Redis 实现分布式信号量(RSemaphore)与倒数闩(RCountDownLatch),利用 Lua 脚本与原子操作保障分布式环境下的同步控制,帮助开发者更好地理解其原理与应用。
808 6
|
10月前
|
NoSQL Java 调度
分布式锁与分布式锁使用 Redis 和 Spring Boot 进行调度锁(不带 ShedLock)
分布式锁是分布式系统中用于同步多节点访问共享资源的机制,防止并发操作带来的冲突。本文介绍了基于Spring Boot和Redis实现分布式锁的技术方案,涵盖锁的获取与释放、Redis配置、服务调度及多实例运行等内容,通过Docker Compose搭建环境,验证了锁的有效性与互斥特性。
854 0
分布式锁与分布式锁使用 Redis 和 Spring Boot 进行调度锁(不带 ShedLock)
|
9月前
|
存储 监控 算法
117_LLM训练的高效分布式策略:从数据并行到ZeRO优化
在2025年,大型语言模型(LLM)的规模已经达到了数千亿甚至数万亿参数,训练这样的庞然大物需要先进的分布式训练技术支持。本文将深入探讨LLM训练中的高效分布式策略,从基础的数据并行到最先进的ZeRO优化技术,为读者提供全面且实用的技术指南。
885 2
|
10月前
|
缓存 NoSQL 关系型数据库
Redis缓存和分布式锁
Redis 是一种高性能的键值存储系统,广泛用于缓存、消息队列和内存数据库。其典型应用包括缓解关系型数据库压力,通过缓存热点数据提高查询效率,支持高并发访问。此外,Redis 还可用于实现分布式锁,解决分布式系统中的资源竞争问题。文章还探讨了缓存的更新策略、缓存穿透与雪崩的解决方案,以及 Redlock 算法等关键技术。
|
缓存 NoSQL 关系型数据库
美团面试:MySQL有1000w数据,redis只存20w的数据,如何做 缓存 设计?
美团面试:MySQL有1000w数据,redis只存20w的数据,如何做 缓存 设计?
美团面试:MySQL有1000w数据,redis只存20w的数据,如何做 缓存 设计?
|
缓存 NoSQL Java
Redis+Caffeine构建高性能二级缓存
大家好,我是摘星。今天为大家带来的是Redis+Caffeine构建高性能二级缓存,废话不多说直接开始~
1703 0
|
9月前
|
缓存 负载均衡 监控
135_负载均衡:Redis缓存 - 提高缓存命中率的配置与最佳实践
在现代大型语言模型(LLM)部署架构中,缓存系统扮演着至关重要的角色。随着LLM应用规模的不断扩大和用户需求的持续增长,如何构建高效、可靠的缓存架构成为系统性能优化的核心挑战。Redis作为业界领先的内存数据库,因其高性能、丰富的数据结构和灵活的配置选项,已成为LLM部署中首选的缓存解决方案。
883 25
|
10月前
|
存储 缓存 NoSQL
Redis专题-实战篇二-商户查询缓存
本文介绍了缓存的基本概念、应用场景及实现方式,涵盖Redis缓存设计、缓存更新策略、缓存穿透问题及其解决方案。重点讲解了缓存空对象与布隆过滤器的使用,并通过代码示例演示了商铺查询的缓存优化实践。
387 1
Redis专题-实战篇二-商户查询缓存
|
9月前
|
缓存 运维 监控
Redis 7.0 高性能缓存架构设计与优化
🌟蒋星熠Jaxonic,技术宇宙中的星际旅人。深耕Redis 7.0高性能缓存架构,探索函数化编程、多层缓存、集群优化与分片消息系统,用代码在二进制星河中谱写极客诗篇。
1714 3