• Email:
  • Feeds

  • 分布式缓存的一起问题

    背景说明

    分布式缓存中为了可用性及高性能的考虑,可以使用如下一种master/slave设计模式。

    图中的proxy是逻辑的概念,可以是基于client的包装实现,也可以是独立的proxy服务,但本文大部分是指独立的服务。几个主要的问题说明如下。

    为什么cache要使用两个集群((master/slave)来存放?

    主要出于可用性及高性能的考虑。传统的架构使用基于一致性哈希的分布式缓存,数据只存在一份副本,在出现cache节点单点故障时,虽然可以由一致性哈希算法将请求均匀落到其他节点,但由于穿透的请求较多,仍然给数据库带来较大的访问压力。为了避免对数据穿透带来的冲击,数据使用两份副本可以避免穿透的问题。同时在数据访问较大时候,也可以更好的分担流量,避免峰值单份数据跑满对系统带来的冲击。

    为什么两份副本要使用master/slave结构?

    由于大型系统中通常存在多个client同时操作同一份数据,需要确保所有client对数据修改时数据的一致性。为了避免两cluster两份副本数据不一致带来的困扰,使用了一个简单的做法,在配置中人为指定一个cluster为master,所有的数据以master为准。

    为什么一些场景需要使用CAS?

    CAS在计算机并发领域通常指Compare-and-swap,在memcached中,也称为Check And Set. 在分布式系统中,一份数据可能同时被多个调用修改,比如微博中的@箱,一个用户同时收到多个@的情况还是比较常见,比如当原来@箱里面记录是{1,2,3}时,4和5由不同的调用来源同时到达,如果没有同步的保护,系统的数据有可能最终被写成{1,2,3,4}或{1,2,3,5},由于memcached没有原生的list结构,list都是一个自定义的value, 则很容易出现client A覆盖了同时在写的client B的数据。因此假如两个调用方同时读到{1,2,3}时,第一个写入{1,2,3,4}会成功,后续的{1,2,3,5}CAS写入就会失败,因为此时服务器已经不是{1,2,3}了,失败的调用向服务端取回{1,2,3,4},最终写入{1,2,3,4,5}

    在master/slave场景,比起普通的memcache CAS有什么区别?

    目前的做法是master cas成功之后,直接修改slave,并不同时在slave执行cas操作。由于数据存在两份副本,当数据不一致时,无法自动处理数据的不一致冲突。因此在实践上只以master操作为准。

    为什么使用proxy?

    使用proxy主要是出于可用性、命中率以及可运维方面的考虑
    可用性与可运维:当进行服务器增容或缩容时,如果client的数量较大,如果未使用proxy模式,client所在服务器通常需要修改配置并且逐个重启。重启(系统维护)一方面带来可用性方面的问题,运维方面也较为繁琐。
    命中率:如果业务场景需要较高的命中率(比如>90%),则增容或缩容就变得较为复杂,需要client配合做一些策略,比如扩容后仍然访问扩容前旧的节点的数据以保证命中率。如果用proxy模式则极大降低client的访问复杂性,将相关逻辑都封装在proxy之后。

    分布式缓存的一起问题

    最近某业务有一起master单点故障,导致在问题的时间段内,用户看不到最近发生变更的数据。由于在上述场景中,实现cas时候的流程如下
    1) master.cas(k,v)
    2) 如果1成功,slave.set(k,v)
    3) 如果1失败,不执行slave.set(),直接return;

    由于第三步在失败时,并不会set slave,导致数据出现一致性问题,即使slave依然可用,新的数据不会写入cache。

    首先看在master failure时,为什么不切换到slave cas?
    先说自动切换的问题
    上文也提过,两份数据副本在出现数据不一致后,并不能自动仲裁达到最终一致性,但是指定master角色可以达到最终一致性。如果master角色可以由调用方自动切换,则会带来数据的混乱。调用方存在多个节点,至少需要统一的config server来保证切换的一致性。另外,自动切换发生后,无法达到两份数据的最终一致性。
    再说由运维手工切换
    由于不牵涉到代码的逻辑判断,虽然切换也会带来一些数据一致性问题,在具体场景下(比如master长久宕机)切换可以接受。

    在出现上述问题后,其他一些解决方案如下。
    1. proxy在master cas失败时候delete slave data
    2. client在master cas失败时set slave, 并且将数据过期时间设成5分钟

    上述方案很难完美,一些明显存在的问题如下
    方案1:
    命中率的问题。由于delete导致修改的数据迅速失效,会导致读取量的增加,在读写均密集的业务场景,可能会导致数据访问出现波动。
    接口职责单一性的问题。proxy在cas调用中隐藏了删除数据的逻辑,这是一个未在正常期望范围内的额外操作,在特殊情况下,可能会导致不可预料的情况出现。(尽管在实际操作中proxy提供配置开关选项)

    方案2:
    依然是命中率的问题,5分钟过期延缓了过期的访问数据库的压力,但相关压力仍然会传递到数据库。

    希望通过上面说明读者能理解这个场景的问题。在这个场景下,完美的方案应当如何设计?

    14 Comments  »

    1. 合页 says:

      写的很好 加油

    2. jametong says:

      有想过,通过Cache 节点本身去做后端的Replication吗? 也即类似于MySQL节点,交由MySQL自身去维护Replication,上层通过数据源来进行组合。

    3. A brief dilemma and also cheap burberry shirt respond to visit will track your elegant web presentation. ( Instruction manuals)As a reminder, this seminar is taped. It has become my best entertainment flying insects ones own multitude, Jean Fontana involving ICR. 20 At that moment, John must have been a member of the actual Judaism orlando model, who had discarded Jesus Christ, so it, consecutively, was turned down by God. (Acts 3:Thirty seven, 55; Proverbs 15:18) To achieve God approval Chris h

    4. rj03hou says:

      1) master.cas(k,v)
      2) 如果1成功,slave.set(k,v)
      3) 如果1失败,不执行slave.set(),直接return;
      我觉得上面的流程没有问题,外部来确定master是否挂了,而不是在应用中确定。

      或者proxy支持单位时间因为master cas失败时候delete slave data的数量,超过数量的时候报警,不再继续delete。

    5. A harmonious blend of all faith prayers, songs, dances, ecstatic beats of traditional ‘dhak’ (drum) and exchange of ideas herald the Durga Puja festival through a ‘get together of harmony’ organized by the Central Puja Committee (CPC) in the city every year. “This is a unique annual event herve leger bandage dresses held in cosmopolitan Shillong, the like of which, perhaps is not to be found anywhere else,” said a member of the CPC.. In his free time, Joe enjoys outdoor photography, walking alo

    6. At present formulated good enough pitchability [sic] which he might been able and find their way his own means via a primary nba game, Mets supervisor Jerry Red washington suggested. Is incredibly notable to have a little glass pitcher. Recovering management, Pelfrey determined their self on Johan Santana shoes and boots. Please use misuse ones Pam that way. A lot of everything you aerosol definitely will go in the atmosphere or possibly within the cooking location this is not on the actual ba

    7. I’ve got prepared slice spuds for mashed apples. I insert water inside the floor with the cleaner and put the actual cut yams towards the bag, tight all the cover along with let the burden oven percentage of the product take on. James Urprise Extremely boring flavored flavoring. Honeysuckle Sweet baby flavor. That is this in any case? Kitchen’s aid? An ideal friend? I will consider people ready till Kate seemed to be out of the place, after that fanatically labeling a good press reporter while u

    8. Total breakfasts presented eggs, taters and also animal meat. Snacks would certainly have the Kids with the breakfast earlier than soups and sandwiches for supper, then simply additional cupcakes along with crap to help chew in during the entire afternoon.I just manufactured alterations ideal on the wants of an couple of folks in their 90s. No more yams, white colored breads or rice. Acquire any run-a-way highly detailed sections near the particular servicing food not to mention eat these people

    9. McConnell is always with Ontario, Albertans can be assured that your Justice Solutions moving ahead making use of these appeal all of which will keep find justice in this case. McConnell had been sentenced at the outset of Summer in a very Wetaskiwin louis vuitton australia online court room on your demise for their 2 sons, Two year older Connor along with 15 four weeks ancient Jayden. This lady appeared to be sentenced to years for every daughter’s demise, to get provided simultaneously. The pr

    10. polo says:

      Tips to Staying Healthy on a South America VacationTravel Articles | March 11, 2012 Learn some valuable tips on how to stay healthy during your SouthAmerica vacation, with this article written by a South America travel expert. ralph lauren polo ralph lauren polo On your South Americavacation take one of the Machu Picchu tours, visit the lively city of BuenosAires, or relax on ralph lauren polo shirts some of Brazil’s polo ralph lauren finest beaches. No matter where your SouthAmerica trip leads

    11. UL 2556 says:

      “And, once they try the foods, it helps them retrain their palates.”She started the program with one school in California, and word spread quickly. Their education argument sounds like it was concocted by proponents of the No Child Left Behind law. I also like the extra long strap since often times I find shoulder staps on handbags to be too short.. Any college. And, interestingly, July 4, 1975 is when we came to Minnesota. Maybe she shouldn’t worry. SDXC compatible devices are able to use the o

    12. UL 844 says:

      But perhaps more jaw dropping than the kid’s preternatural mechanics are the actions of his father. Robert Gates knows that the J 20 has been designed by China specifically to target the US Navy aircraft carrier task forces that currently dominate the Pacific Ocean,UL 508C, off China. Many of the seamstresses supported by Mend are former abductees, who were forced to become child soldiers or wives to the Lord Resistance Army (LRA) rebel commanders.. 31 in exchange for Steve Downie. This is why C

    13. UL 8750 says:

      Of course, some of those rumors do happen, but you can listen to everything. The CBO courageously pointed out that ObamaCare makes it less likely that millions of Americans will want to work. The hash was a little heavy on the onion for my taste, but that persona preference and it didn bother me enough to prevent me from eating the entire thing. Type O negative is in demand because they are called the “universal donors”. And how were you supposed to get your music files back in the decades befor

    14. En outre,, la formation d’endurance de l’état d’équilibre seuls les trains sang supplémentaire obtient à un seul rythme cardiaque gamme et ne former pas pour répondre à divers au jour le jour stress.L’industrie des cosmétiques continue d’être simplement autorégulé, qui a veillé à ce qu’ils essaient assurez-vous idées que une entreprise plus strictement réglementée ne essayer de faire. Organes tels que le USLa façon d

    RSS feed for comments on this post, TrackBack URI

    Leave a Comment