Caching¶

A fundamental trade-off in dynamic websites like Nautobot is that, well, they’re dynamic. Each time a user requests a page, the Web server makes all sorts of calculations – from database queries to template rendering to business logic – to create the page that your site’s visitor sees. This is a lot more expensive, from a processing-overhead perspective, than your standard read-a-file-off-the-filesystem server arrangement.

That’s where caching comes in.

To cache something is to save the result of an expensive calculation so that you don’t have to perform the calculation next time.

Nautobot makes extensive use of caching; this is not a simple topic but it's a useful one for a Nautobot administrator to understand, so read on if you please.

How it Works¶

Nautobot supports database query caching using django-cacheops and Redis. When a query is made, the results are cached in Redis for a short period of time, as defined by the CACHEOPS_DEFAULTS parameter (15 minutes by default). Within that time, all recurrences of that specific query will return the pre-fetched results from the cache. Caching can be completely disabled by toggling CACHEOPS_ENABLED to False (it is True by default).

If a change is made to any of the objects returned by the cached query within that time, or if the timeout expires, the cached results are automatically invalidated and the next request for those results will be sent to the database.

Caching is a complex topic and there are some important details to clarify with how caching is implemented and configured in Nautobot.

Caching in Django¶

Django includes with its own cache framework that works for common cases, but does not work well for the wide array of use-cases within Nautobot. For that reason, Django's built-in caching is not used for the caching of web UI views, API results, and underlying database queries. Instead, we use django-cacheops. Please see below for more on this.

`CACHES` and `django-redis`¶

The CACHES setting is used to, among other things, configure Django's built-in caching. You'll observe that, even though we aren't using Django's built-in caching, we still have this as a required setting. Here's why:

Nautobot uses the django-redis Django plugin which allows it to use Redis as a backend for caching and session storage. This is used to provide a concurrent write lock for preventing race conditions when allocating IP address objects, and also to define centralized Redis connection settings that will be used by RQ.

django-redis also uses the CACHES setting, in its case to simplify the configuration for establishing concurrent write locks, and also for referencing the correct Redis connection information when defining RQ task queues using the RQ_QUEUES setting.

Again: CACHES is not used for Django's built-in caching at this time, but it is still a required setting for django-redis to function properly.

Django Cacheops¶

Cacheops (aka django-cacheops) is a Django plugin that does some very advanced caching, but does not leverage the built-in cache framework. Instead it uses a technique called "monkey patching". By monkey patching, a library can inject its own functionality into the core code behind the scenes.

This technique allows Cacheops to do more advanced caching operations that are not provided by the Django built-in cache framework without requiring Nautobot to also include some elaborate code of its own. This is accomplished by intercepting calls to the underlying queryset methods that get and set cached results in Redis.

For this purpose, Cacheops has its own CACHEOPS_* settings required to configure it that are not related to the CACHES setting.

For more information on the required settings needed to configure Cacheops, please see the Caching section of the required settings documentation.

The optional settings include:

CACHEOPS_DEFAULTS: To define the cache timeout value (Defaults to 15 minutes)
CACHEOPS_ENABLED : To turn on/off caching (Defaults to True)

Invalidating Cached Data¶

Although caching is performed automatically and rarely requires administrative intervention, Nautobot provides the invalidate management command to force invalidation of cached results. This command can reference a specific object my its type and UUID:

$ nautobot-server invalidate dcim.Device.84ae706d-c189-4d13-a898-9737648e34b3

Alternatively, it can also delete all cached results for an object type:

$ nautobot-server invalidate dcim.Device

Finally, calling it with the all argument will force invalidation of the entire cache database:

$ nautobot-server invalidate all

High Availability Caching¶

Redis provides two different methods to achieve high availability: The first is Redis Sentinel and the second is the newer Redis Clustering feature. Unfortunately, due to an known issue with django-cacheops (last updated November 2021) Nautobot is unable to support Redis Clustering at this time. Therefore, Nautobot only supports Redis Sentinel for high availability.

Using Redis Sentinel¶

The installation/configuration of the Redis Sentinel cluster itself is outside the scope of this document, this section is intended to provide the steps necessary to configure Nautobot to connect to a Sentinel cluster.

We need to configure django-redis, django-cacheops, and celery to use Sentinel. Each library is configured differently, so please pay close attention to the details.

`django-redis` Sentinel Configuration¶

Notable settings:

SENTINELS: List of tuples or tuple of tuples with each inner tuple containing the name or IP address of the Redis server and port for each sentinel instance to connect to
LOCATION: Similar to a redis URL, however, the hostname in the URL is the master/service name in redis sentinel
SENTINEL_KWARGS: Options which will be passed directly to Redis Sentinel
PASSWORD: The redis password (if set), the SENTINEL_KWARGS["password"] setting is the password for Sentinel

Example:

DJANGO_REDIS_CONNECTION_FACTORY = "django_redis.pool.SentinelConnectionFactory"
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://nautobot/0",  # in this context 'nautobot' is the redis master/service name
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.SentinelClient",
            "CONNECTION_POOL_CLASS": "redis.sentinel.SentinelConnectionPool",
            "PASSWORD": "",
            "SENTINEL_KWARGS": {
                "password": "",  # likely the same password from above
            },
            "SENTINELS": [
                ("mysentinel.redis.example.com", 26379),
                ("othersentinel.redis.example.com", 26379),
                ("thirdsentinel.redis.example.com", 26379)
            ],
        },
    },
}

Note

It is permissible to use Sentinel for only one database and not the other, see RQ_QUEUES for details.

For more details on configuring django-redis with Redis Sentinel, please see the documentation for Django Redis.

`django-cacheops` Sentinel Configuration¶

Notable settings:

locations: List of tuples or tuple of tuples with each inner tuple containing the name or IP address of the Redis server and port for each sentinel instance to connect to
service_name: the master/service name in redis sentinel
Additional parameters may be specified in the CACHEOPS_SENTINEL dictionary which are passed directly to Sentinel

Note

locations for django-cacheops has a different meaning than the LOCATION value for django-redis

Warning

CACHEOPS_REDIS and CACHEOPS_SENTINEL are mutually exclusive and will result in an error if both are set.

Example:

CACHEOPS_REDIS = False
CACHEOPS_SENTINEL = {
    "db": 1,
    "locations": [
        ("mysentinel.redis.example.com", 26379),
        ("othersentinel.redis.example.com", 26379),
        ("thirdsentinel.redis.example.com", 26379)
    ],
    "service_name": "nautobot",
    "socket_timeout": 10,
    "sentinel_kwargs": {
        "password": ""
    },
    "password": "",
    # Everything else is passed to `Sentinel()`
}

For more details on how to configure Cacheops to use Redis Sentinel see the documentation for Cacheops setup.

`celery` Sentinel Configuration¶

Note

Celery is not directly related caching but it does utilize Redis, therefore in more advanced deployments if Redis Sentinel is required for caching, Celery must also be configured to use Redis Sentinel to high availability.

Celery Sentinel configuration is controlled by four settings within your nautobot_config.py:

By default Nautobot configures the celery broker and results backend with the same settings, so this pattern is mirrored here.

redis_password = ""
sentinel_password = ""

CELERY_BROKER_URL = (
    f"sentinel://:{redis_password}@mysentinel.redis.example.com:26379;"
    f"sentinel://:{redis_password}@othersentinel.redis.example.com:26379;"
    # The final entry must not have the `;` delimiter
    f"sentinel://:{redis_password}@thirdsentinel.redis.example.com:26379"
)
CELERY_BROKER_TRANSPORT_OPTIONS = {
    "master_name": "nautobot",
    "sentinel_kwargs": {"password": sentinel_password},
}

CELERY_RESULT_BACKEND = CELERY_BROKER_URL
CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = CELERY_BROKER_TRANSPORT_OPTIONS

Please see the official Celery documentation for more information on how to configure Celery to use Redis Sentinel.

Please also see the Nautobot documentation on required settings for Celery for additional information.

Caching¶

How it Works¶

Caching in Django¶

CACHES and django-redis¶

Django Cacheops¶

Invalidating Cached Data¶

High Availability Caching¶

Using Redis Sentinel¶

django-redis Sentinel Configuration¶

django-cacheops Sentinel Configuration¶

celery Sentinel Configuration¶

`CACHES` and `django-redis`¶

`django-redis` Sentinel Configuration¶

`django-cacheops` Sentinel Configuration¶

`celery` Sentinel Configuration¶