Evolution of Servers

Dedicated Servers

• Definition: Physical machines which were actual computers which run your hosted application. Still used today and could be used with virtualization to run multiple applications on the same machine.
• Operation: They are always on, handle traffic in parallel, and need to be scaled manually.
• Model: Operate on a centralized model where the server can be a single point of failure. This can be seen as good as you have more control, and data management is streamlined.
• Benefits: This provides consistent performance, security, and reliability since you know exactly how the data is being handled and where it is being stored.
• Management: Uptime, power supply, connection speed, and similar factors depend on your management.
• Limitations: Optimizing performance beyond the hardware’s capabilities requires manual scaling (upgrading or adding more hardware). Complex deployment, scaling, and management can potentially delay the time-to-market for critical applications.
• Cost: High upfront costs, maintenance, and management overhead. You are paying for the hardware and the time it takes to manage it.

Edge Computing and CDNs

To tackle the problem of latency, which occurs when the server is geographically far from the client (a common issue as the internet grew), Content Delivery Networks (CDNs) & Edge Computing emerged. These involve a network of servers or a distributed computing paradigm spread across the globe. This allows content to be cached closer to the user, significantly reducing latency.

Traffic patterns are also something to keep in mind here. If you have high traffic during the day from one region but not another, managing server uptime efficiently becomes complicated with things like timezones, especially when trying to cut costs or scale with a centralized dedicated server.

This distributed approach started to make applications feel more responsive and faster since data is served from a geographically nearby server. Being decentralized with multiple servers also simplifies scaling and reduces bandwidth consumed by transferring data to distant centralized servers. Cloudflare is a good example of a catalyst for this change, making edge deployment easier with offerings like Cloudflare Pages and Cloudflare Workers.

Serverless Computing

The term “Serverless” can be misleading; it’s perhaps better described as “server-less” management or even “lazy servers”.

• Event-Driven & Scalable: Serverless functions are event-driven (e.g., triggered by HTTP requests) and automatically scale based on demand.
• Managed Infrastructure: No need to manage underlying servers or infrastructure, allowing developers to focus solely on writing code.
• Statelessness: Functions are typically stateless; persistent state needs to be stored externally in a database or other storage solution.
• Lifecycle: Instances “die” or spin down when not actively used, hence the “lazy” aspect.
• Concurrency Model:
  • Each instance typically serves one request at a time. A new request often gets a new compute node/dedicated Virtual Machine (VM) for its duration.
  • How is concurrency handled? While dedicated servers handle concurrency using threads within a single machine, traditional serverless functions often achieve concurrency by spinning up multiple VMs—one for each concurrent request.
  • You aren’t blocking and waiting for a response from one instance; instead, a new VM is spun up for each concurrent request, even if an existing VM is about to finish.
  • To mitigate issues like cold starts (the delay in spinning up a new VM) and improve user experience, developers implemented strategies like caching and techniques to keep function instances “warm”.
• Ideal Use Cases: This model excels at short-lived tasks requiring low latency and quick response times.
• Limitations: Long-lived requests aren’t ideal, as they can lead to inefficient resource use (keeping potentially many VMs active). The “cold start” problem can also impact the performance of infrequently used functions.

Fluid Computing

Fluid computing aims to blend the benefits of dedicated servers (resource utilization) and serverless functions (scalability, managed infrastructure) into a hybrid model. The key idea is optimizing concurrency—making sure resources are fully utilized rather than spinning up a new VM for every single request unless absolutely necessary. This involves enabling “in-function concurrency” or multithreading within a single function instance.

• Efficiency: There are moments in asynchronous code execution (like waiting for network requests) where a server isn’t actively processing. Fluid computing takes advantage of this downtime within a single instance to handle other requests, making it more cost-effective.

• Optimized Concurrency: This allows a single lambda/function instance to handle multiple concurrent invocations. This is a departure from the traditional serverless model where each invocation often gets its own isolated instance. Optimized concurrency leads to more efficient use of compute resources and can reduce idle time, especially for I/O-bound tasks (uses concurrecy to improve efficiency since its usually waiting during input/output or connecting with db and stuff) (common in AI applications waiting for external data).

• Complex State Management: Unfortunately allowing multiple invocations within the same process requires careful consideration of potential issues related to shared global state and request isolation.

“Traditional Serverless wastes idle time. It fails to efficiently utilize available resources during periods of inactivity.”

Fluid compute models often aim to bill based on actual CPU time used, rather than billing for the entire time an instance is active, including idle time. Techniques like pre-warmed instances (in rust btw 🦀) and bytecode caching further reduce cold starts, making these models suitable for demanding applications like AI. Fluid compute models often support dense global compute and multi-region failover.

Amazing Example of Fluid Compute in Action

Server Architecture Comparison

Further Reading