Table of Contents

KEFCore: use cases

Entity Framework Core provider for Apache Kafka can be used in some operative conditions. Here a possible, non exhaustive list, of use cases.

Before read following chapters it is important to understand how it works.

Apache Kafka as Database

The first use case can be coupled to a standard usage of Entity Framework Core, the same when it is used with database providers. In getting started is proposed a simple example following the online documentation. In the example the data within the model are stored in multiple Apache Kafka topics, each topic is correlated to the DbSet described from the DbContext.

The constraints are managed using OnModelCreating of DbContext.

A different way to define data within Apache Kafka topics

Changing the mind about model, another use case can be coupled on how an Entity Framework Core model can be used. Starting from the model proposed in getting started, the data within the model are stored in multiple Apache Kafka topics. If the model is written in a way it describes the data to be stored within the topics it is possible to define an uncorrelated model containing the data of interest:

public class ModelContext : KafkaDbContext
{
    public DbSet<FirstData> FirstDatas { get; set; }
    public DbSet<SecondData> SecondDatas { get; set; }
}

public class FirstData
{
    public int ItemId { get; set; }
    public string Data { get; set; }
}

public class SecondData
{
    public int ItemId { get; set; }
    public string Data { get; set; }
}

Then using standard APIs of Entity Framework Core, an user interacting with Entity Framework Core can stores, or retrieves, data without any, or limited, knowledge of Apache Kafka.

Apache Kafka as distributed cache

Changing the mind a model is written for, it is possible to define a set of classes which acts as storage for data we want to use as a cache. It is possible to build a new model like:

public class CachingContext : KafkaDbContext
{
    public DbSet<Item> Items { get; set; }
}

public class Item
{
    public int ItemId { get; set; }
    public string Data { get; set; }
}

Sharing it between multiple applications and allocating the CachingContext in each application, the cache is shared and the same data are available.

Apache Kafka as a triggered distributed cache

Continuing from the previous use case, using the events reported from Entity Framework Core provider for Apache Kafka it is possible to write a reactive application. When a change event is triggered the application can react to it and take an action.

Alt text

SignalR

The triggered distributed cache can be used side-by-side with SignalR: combining Entity Framework Core provider for Apache Kafka and SignalR in an application, subscribing to the change events, it is possible to feed the connected applications to SignalR.

Redis

The triggered distributed cache can be seen as a Redis backend.

Data processing out-side Entity Framework Core application

The schema used to write the information in the topics are available, or can be defined from the user, so an external application can use the data in many mode:

  • Using the feature to extract the entities stored in the topics outside the application based on Entity Framework Core
  • Use some features of Apache Kafka like Apache Kafka Streams or Apache Kafka Connect.

External application

An application, not based on Entity Framework Core, can subscribe to the topics to:

  • store all change events to another medium
  • analyze the data or the changes
  • and so on

Apache Kafka Streams

Apache Kafka comes with the powerful Streams feature. An application based on Streams can analyze streams of data to extract some information or converts the data into something else. It is possible to build an application, based on Apache Kafka Streams, which hear on change events and produce something else or just sores them in another topic containing all events not only the latest (e.g. just like the transaction log of SQL Server does it).

Apache Kafka Connect

Apache Kafka comes with another powerful feature called Connect: it comes with some ready-made connector which connect Apache Kafka with other systems (database, storage, etc). There are sink or source connectors, each connector has its own specificity:

  • Database: the data in the topics can be converted and stored in a database
  • File: the data in the topics can be converted and stored in one, or more, files
  • Other: there are many ready-made connectors or a connector can be built using a Connect SDK

NOTE: While Apache Kafka Streams is an application running alone, Apache Kafka Connect can allocate the connectors using the distributed feature which load-balance the load and automatically restarts operation if something is going wrong.

Edit this page