Redis Dead Redemption: Redis Cache Timeouts with Sitecore Azure PaaS

Our production CD environments (v9.0 Initial Release) began experiencing significant performance issues to the point where the application kept restarting our site would hang for several minutes at a time.  When the site finally came to, a few minutes later, it'd go right back down.

With the site having been stable for months, it was odd that this issue surfaced out of the blue.

Our logs revealed the following error (one of many):

What is Redis? 

In a nutshell, Azure Redis cache is the default, OOB session provider Sitecore 9.0+ for PaaS instances.
I think George Tucker's StackOverflow answer summed it up best:

Sitecore uses this session provider as a means of managing out of process (ie. distributed) session state to share contact data across browser sessions and devices. This is required to support base functionality in Sitecore XP (Analytics, XDB etc). Even if you are using a custom session provider for other purposes, you will likely still need the Sitecore sessions configured to get full value out of the solution.  

Redis is designed as a means for fast, distributed access to a key/value store. If you're digging into the redis instance directly, you're likely to see all of the keys of your current sessions (private and shared) with non-readable values (usually stored as binary vals). These values are usually managed and accessed via the Sitecore Session API abstractions. It's rare you would need to access Redis directly.

 Initial Troubleshooting

We attempted a few things in attempts to re-stabilize the environment.

1. Standard procedures to restart applications: generally no effect
2. Ran 'flushdb' command against Redis Console: errors cleared initially, but the issue resurfaced soon after.
3. Reboot Redis: Site restores, goes down soon after
4. Scaled up Redis: Site restores - crashes soon after with the same Redis errors
5. Applied Support fix outlined in KB464570: Redis driver timeout issues to both CM and CD environments.

To no avail, we decided to take it up with the Sitecore Support team. 

Sitecore Support's Take

Based on the reported errors, Sitecore support identified the following

• - The "in: 7923" parameter means that there is an amount of data in the Redis queue that sill need to be processed
• - The "WORKER: (Busy=85,Free=32682,Min=150,Max=32767)" shows that there is some pressure on the Redis side (even if it is not visible as the CPU is not stressed)

You can read more into what your specific Redis errors may mean from this handy Microsoft blog bost documentation.

Support sanctioned our use of KB464570: Redis driver timeout issues and also recommended applying solutions outlined in a separate KB article: KB858026: Excessive load on ASP.NET Session State store.

Additional suggestions included:

- Scale up the Redis instance.  "You may need extra resources to process all the items  "in: 7923")"

- Increase the timeout until you could see the application running.

Among several solutions offered in KB858026: Excessive load on ASP.NET Session State store. (mostly configuration changes to the Web.config and Sitecore.Analytics.Tracking.config),  one included splitting the Sitecore.Sessions databases into dedicated instances - one for private and another for shared.

Splitting the two databases meant creating a new Redis instance in Azure and use different connections strings for the Shared and Private session definitions.

The documentation articles for adjusting the configuration can be found in two separate documents:

- Shared Session

- Private Session

Diagnosis

Since the site was stable for months before this began occurring, Support concluded that the most possible cause is that the amount of data in the Redis' queue slowly built up until the CDs could not keep up with the timeouts.

Splitting the sessions into Private and Shared should allow more control over which Redis instance needs more or less processing power and avoid obscure locks.  Also, increasing the timeout values to higher values allows the queue more time to process.

The Split

After creating a new Redis instance in Azure:

1) Create a new redis.sessions.shared entry in the ConnectionString.config under the existing redis.sessions entry.
2) Leave the original (Private) <sessionState> Redis provider as-is in the Web.config
3) Replace the existing redis.sessions connectionString parameter to redis.sessions.shared.

A Note on Timeout Values

Regarding the timeout parameters (connectionTimeoutInmilliseconds, operationTimeoutInMilliseconds, retryTimeoutInMilliseconds) there is no exact value to set.  Like Sitecore caches, it must be tuned until your instances do not experience timeouts.

Sitecore's rule of thumb here is to:
- double the timeout until the environment stabilizes
- set a high value (30 minutes) and then adjust it if it needs further tuning

You can find out what each timeout is designed for here: Redis Provider Settings 

After making these changes, the site has since appeared to have stabilized. Of course, we anticipate to further monitor the logs for Redis errors, but for now, it seems we're in the clear.  

Happy Redis-ing!

Read the full post →

Implementing Sitecore Experience Commerce™ 9.0.2 (300) Training Course Overview

Last week, I attended the Implementing Sitecore Experience Commerce™ 9.0.2 (Sitecore Commerce 300) training course with several of my fellow Rightpoint team members led by experienced Sitecore trainer Michael Mavrofrides (always highly recommended).  It was the first official on-site Sitecore training course I've taken in over 5 years and first training course I've taken for Sitecore's Commerce offering.

If you're planning to take the course this year, this quick overview may help clarify what to expect and how to prepare in order to get the most out of your training sessions.

Course Objectives

1. Discuss the technical features Sitecore Experience Commerce (Shops, Pricing, Catalog, Promotions, Inventory, Fulfillment, Payments, Orders, Shopping Cart, Checkout, Entitlements, BizFX Merchandiser Tool).
   
2. Build a storefront using SXA components
   
3. Explain the benefits of SXA
   
4. Understand plugin architecture and how to extend Sitecore Experience Commerce
   
5. Pass a 'quiz' to obtain a Certificate of Training.  (This is not a certification for Commerce, but rather just an overview with ~32 questions covering material from the course)

Course Structure

The course offers 25 hours (4 days) of material which really ties together concepts covered in the Sitecore® Experience Commerce™ 9 100: Overview and Sitecore® Experience Commerce™ 9 200: Essentials eLearning courses.  The training is held onsite - in many cases -a Microsoft training center. 

The material is a combination of presentation slides and student labs. Student Labs consist of material covered during the slides and are completed on a VM provided by the trainer so you won't have to worry about installing anything locally.  Simply make sure you're able to access remote machines using Windows Remote Desktop Manager.

Getting the Most Out of Training

Complete the Prerequisite Courses

Definitely come with the minimum prerequisite courses.  It's important to come into class with a baseline understanding of what Sitecore Commerce has to offer.  If you have any questions from those lower level courses, this class gives you the opportunity to ask a certified Sitecore trainer for any clarification.

Have Exposure (If Possible)

One of our co-workers had a few weeks of real life Sitecore Commerce development exposure and he certainly benefited where the course was able to clarify certain aspects prior. If you have the opportunity to be exposed to commerce development  (specifically Commerce Plugin development and SXA integration), it's a great way to get ahead of the course material and discover things in-flight during training that you may have come across during development.

In any case, you'll still find the course material useful from a conceptual perspective.

Engage in Meaningful Discussion

One of the best ways to learn is to engage in meaningful discussions. Talking through architectural and development concepts that may not make sense right away in plain text will facilitate a better understanding of those concepts.

Our course was filled with Rightpoint developers and architects, so we had an extra advantage to openly discussing any specific questions regarding course material.  During the class, ask questions or clarification at any point to make an effort to understand the material fully. Even specific questions regarding specific scenarios not explicitly covered in the course are not off limits.  This is the time to ask!

Understand the Coding Exercises. 

The student labs have a technical side to them as the course is gear towards "advanced-level course for ASP.NET MVC web developers and solution architects with advanced C# skills". There's an expectation that you're comfortable working within various Windows tools including Visual Studio, IIS, machine certificates, system services, etc. 

Many of the labs require you to create a series of classes to build Commerce Plugin components. Luckily, the labs come with all the code snippets you need to complete each exercise. The downside there is because there's a whole lot of copying pasting, it may be difficult to understand what you're actually doing. 

Take the time to read all the notes within the student lab guide, as well as adding breakpoints/debugging through the code to build a proper understanding.  Ultimately, your Sitecore Commerce development skills will improve with real-world scenarios and repetition, but following the general code structure in the labs will help facilitate those basic building blocks.

Provide Feedback

There were some issues throughout the student labs - whether it was a lapse in steps or terminology confusion.  That's okay! The Sitecore training team is constantly trying to improve the course for a better experience for the next group.  Issues we ran into may no longer be problematic in future courses due to our feedback. 

Pass any feedback along to make sure future attendees and trainers can handle issues more swiftly.  After the quiz, you'll have the opportunity to provide an assessment of the course and trainer.  This feedback is imperative to improving the course for future students, so be specific and honest!

After the Course

After completing the quiz, you'll get a nice little certification and a code to take the Sitecore Commerce Certification exam (which expires 1 year after the course ends). 

Take the knowledge you learned from the courses and apply them to your Sitecore Commerce projects, and prepare to take the Certification exam by studying/reviewing all the material. If you have any questions after the course, you can always email your trainer.

Goooo Sitecore Commerce, Go!

Read the full post →

Quick and Easy Azure PaaS PublishSettings Import to FileZilla with PowerShell

With the advancement of Sitecore on Azure Platform as a Service (PaaS), gone are the days where you remote into a physical or virtual machine. If you need to access the application files, you'll need to download a Publish Profile settings file from the Azure Portal. Just click the button.

The Publish Profile setting file simply an XML file (surprise, surprise) with connection information about the resource - which can be directly imported to Visual Studio to deploy files.   You can manually copy those values from the FTP host endpoint (publishUrl), username (userName), and password (userPWD) nodes into an FTP client and connect.

A standard Sitecore 9 Experience Platform comes packaged with at least 9 Application Services when installed on Azure - each with unique FTP endpoints and credentials.

Our team uses the FileZilla FTP client - in which you can create a set of accessible FTP sites and share them in a locked down repository where other developers who need access can get it without downloading the Publish Profile settings themselves.

As you can imagine, working through 9 unique Publish Profile settings is rather tedious when setting up these 'racks' in FileZilla - copying and pasting via the FileZilla GUI

via GIPHY

To streamline this process when setting up a new instance of Sitecore on PaaS - or onboarding an existing Sitecore PaaS, I've written a Windows PowerShell script that eliminates the need to copy and paste anything.



Simply download each App Service's FTP Publish Profile setting file to a folder on your local machine.

Run the script and select that same folder

The script processes each .PublishSettings file and generates an AzurePaaS-FileZillaSites.xml file in the same directory.  This file is compatible with FileZilla's import feature.

In FileZilla, go to File > Import

Select the script-generated AzurePaaS-FileZilla.xml file.

Click through the Import Settings (Site Manager Entries will already be selected)

You'll get a success message when the import is successful

The entries will be available in FileZilla and ready to connect!

The benefit of this approach allows you to quickly and easily create an XML file that can be shared amongst the team of developers who would need to access those files.

Once you've completed an import, use FileZilla's Site Manager to organize these entries into folders then export it again from FileZilla (this will encrypt the passwords using your FileZilla's master password).

Read the full post →