Hello everybody,

also I have written about Acumatica scaling horizontally, but it seems, like additional explanations are needed. 

So, take a look on the picture below:

How request flows, in case of single server ( without load balancing )

• Per-Request Object Lifecycle: Each time a user request hits an Acumatica server, objects are spun up as needed for that request and then disposed of once the request is fully processed. This keeps the server's memory footprint small since nothing unnecessary is hanging around. All state data for that session is stored in a session object that gets serialized when necessary.

• Minimal Session Serialization: Acumatica keeps serialization overhead low by only storing the data that has actually changed—new, deleted, modified, or held records. This means the session only contains the essentials. The smaller the serialized data, the faster the read/write operations, so this approach improves both performance and scalability.

• On-Demand Data Fetching: Anything not stored in the session is pulled from the database, but Acumatica doesn't just grab everything. A custom algorithm ensures that only the required data for the specific request is fetched from the database, keeping database calls efficient and reducing unnecessary overhead.

• Custom Serialization Logic: The built-in .NET serialization works fine in general use cases, but Acumatica goes a step further. It uses a custom serialization mechanism that's optimized to store only what’s absolutely relevant. This cuts down on the extra metadata and service information that you’d typically get with standard serialization, making the process much leaner.

• Lazy Creation of Runtime Structures: Key runtime structures like hash tables, constraints, relations, and indexes are only built when the business logic actually needs them. Acumatica avoids wasting CPU cycles and memory by not creating these structures for every request if they’re not necessary. This technique significantly reduces the load on the system, particularly under high demand.

How a Request Travels in Acumatica's Multi-Server Setup:

When a user sends a request to an Acumatica ERP server (via a load balancer like NGINX), the load balancer directs the request to one of the available servers (Node1, Node2, etc.). Here’s what happens:

• Step 1: A fresh request hits one of the nodes (say, Node1). Objects necessary for that request are created on the server.

• Step 2: The session state is checked. If relevant data is already serialized and stored in a remote cache like Redis or MS SQL, the server will deserialize it to rebuild the session.

• Step 3: The system only serializes the data that has changed (e.g., new sales orders, updated inventory records) to avoid bloating the session object. Any unmodified data is fetched directly from the database as needed, thanks to an efficient query mechanism.

• Step 4: As the server processes the request, it only creates runtime objects (like hash tables, indexes, etc.) if the specific request requires them. This way, CPU and memory usage remain optimized.

• Step 5: Once the request is completed, all objects created during the request lifecycle are disposed of, and the session state is either stored in a remote session store (like Redis or MS SQL) or kept on the server, depending on the configuration.

This approach ensures that Acumatica remains responsive and scalable, even when spread across multiple servers.

Besides that, want to describe some dll's in Acumatica, which play the role of foundation of horizontal scalability.

Key DLLs Supporting Scalability in Acumatica

1. StackExchange.Redis.dll

   • Purpose: This library is the Redis client used for distributed caching. Redis is an in-memory data store, often used in high-performance systems to cache frequently used data across different nodes.
   • Scalability Contribution:
     • With Redis, session data or frequently accessed information is cached, reducing the load on databases and improving response times.
     • Redis enables session sharing across multiple application nodes, allowing Acumatica to scale horizontally (across multiple servers) while ensuring user session continuity.

2. Microsoft.Extensions.Caching.Memory.dll

   • Purpose: Provides support for in-memory caching on each server. This caching system stores data locally on the application server, making retrieval faster by avoiding external database calls for frequently requested data.
   • Scalability Contribution:
     • Reduces load on the database by storing temporary data locally.
     • Useful for applications that don't require distributed caching but still need to improve performance by reducing the need to repeatedly query the database for static or slow-changing data.

3. Microsoft.Extensions.Caching.Redis.dll

   • Purpose: Part of Microsoft's caching suite that integrates Redis as a distributed cache. This helps ensure that different nodes (servers) in a distributed system can share the same session data and cache.
   • Scalability Contribution:
     • Critical in environments where Acumatica is scaled across multiple instances or servers. Using Redis for caching allows the same data to be shared across servers, ensuring consistency and reducing latency caused by repeatedly fetching data from the database.

4. Serilog.Extensions.Logging.dll / Serilog.Sinks.File.dll

   • Purpose: Serilog is used for logging and monitoring. It logs application events and performance metrics to a variety of sinks (such as files, databases, or cloud platforms).
   • Scalability Contribution:
     • When scaling applications, monitoring becomes crucial to detect bottlenecks, performance issues, or system failures. By using Serilog, Acumatica can track these events across multiple nodes, allowing developers to address performance bottlenecks quickly and improve scalability.
     • Serilog can handle a massive number of logs in a distributed environment, storing logs centrally or across different storage mediums.

5. Microsoft.Extensions.Logging / Microsoft.Extensions.Diagnostics.HealthChecks.dll

   • Purpose: These libraries are part of the .NET extensions that handle structured logging, monitoring, and health checks of applications.
   • Scalability Contribution:
     • HealthChecks allow Acumatica to monitor the status of each service or server in a multi-node environment. If any node fails or experiences performance degradation, the system can detect it and reroute traffic, ensuring that the application remains scalable and resilient.
     • Logging helps track application performance, identify any bottlenecks, and monitor resource usage across scaled instances.

6. Autofac.dll

   • Purpose: Autofac is a dependency injection framework that enables flexible component registration and management.
   • Scalability Contribution:
     • By decoupling services and using dependency injection, Autofac allows Acumatica to scale services independently. For instance, different service layers (API, business logic, etc.) can be scaled separately, optimizing resource use based on demand.
     • It simplifies the architecture, making it easier to add more nodes or instances when scaling horizontally.

7. Swashbuckle.AspNetCore.Swagger.dll / Swashbuckle.AspNetCore.SwaggerGen.dll

   • Purpose: These DLLs provide Swagger integration for API documentation.
   • Scalability Contribution:
     • As Acumatica scales and APIs are consumed more frequently, having well-documented APIs (via Swagger) allows for smoother integration with external systems and clients. This improves API usability, reduces friction for developers, and supports high traffic from external systems.
     • Swagger also allows easier scaling of API gateways and microservices by providing clear documentation for load balancers or external services interacting with the API.

8. Rebus.dll / Rebus.Handlers.dll

   • Purpose: Rebus is a service bus implementation used for managing distributed messaging between application services.
   • Scalability Contribution:
     • It allows Acumatica to handle messaging between different services in a distributed system. By implementing a message bus, Acumatica can scale services independently, enabling each service to communicate asynchronously and handle its own load without being blocked by other services.
     • This decouples different parts of the system, improving overall resilience and scalability.

9. OpenTelemetry.dll / OpenTelemetry.Exporter.Zipkin.dll

   • Purpose: OpenTelemetry provides libraries for distributed tracing and monitoring across multiple services.
   • Scalability Contribution:
     • OpenTelemetry helps trace requests as they travel through different services and nodes in a scaled environment. By using distributed tracing, Acumatica can monitor how requests are processed across multiple servers, identifying slow points, bottlenecks, or failures.
     • This is essential for debugging and optimizing large-scale, distributed applications.

10. System.Net.Http.dll

    • Purpose: Handles HTTP communications for sending and receiving data across different nodes, databases, and services.
    • Scalability Contribution:
      • In a scaled environment, especially one involving microservices or distributed servers, HTTP requests are common for inter-node communication. System.Net.Http helps efficiently handle these requests, allowing Acumatica to scale by communicating seamlessly between its components.

11. PX.Caching.dll

    • Purpose: Likely part of Acumatica’s internal caching mechanism for optimizing data retrieval and improving response times.
    • Scalability Contribution:
      • Efficient caching is key to reducing database loads, especially when scaling horizontally across multiple servers. By caching frequently accessed data, PX.Caching.dll helps Acumatica reduce database queries, keeping the application responsive even under heavy load.

12. PX.MessageBus.dll

    • Purpose: This is likely used for handling messaging between different services or components in Acumatica.
    • Scalability Contribution:
      • A message bus is crucial for enabling distributed, asynchronous communication between different application components. This allows for better fault tolerance and helps balance loads across various parts of the system, especially when Acumatica is scaled horizontally.

Summary: How These DLLs Support Scalability

From the provided DLLs, it's clear that Acumatica is leveraging key technologies like Redis for distributed caching, OpenTelemetry for distributed tracing, and Rebus for messaging across a distributed system. These allow Acumatica to scale horizontally, manage session state across multiple servers, balance load efficiently, and ensure that performance is not compromised even under heavy user traffic.

The caching-related libraries like StackExchange.Redis, Microsoft.Extensions.Caching.Memory, and PX.Caching.dll ensure that frequently accessed data is stored in memory or distributed caches, reducing the load on the database and speeding up response times.

Libraries like OpenTelemetry and Serilog play a key role in monitoring and diagnostics, making it easier to detect and resolve issues in a scaled system, ensuring smooth operation as the system grows.

In short, the combination of these libraries enables Acumatica to scale efficiently across multiple servers, handle large numbers of concurrent users, and maintain performance while keeping the system responsive.