Topic 1: Introduction to Distributed Systems

- readings: CDK, Distributed Systems: Concepts and Designs, Chapter 1

Lecture 1

AWOL from first lecture

Lecture 2

Went over the course handout. Gave course overview through a series of pictures.

Distributed System: a system in which hardware or software components located at networked computers communicate and coordinate their actions only by passing messages.

Port: a resource managed by the nucleus of the OS that can be requested by and granted to a (server) process. To implement operations with TCP and UDP on the Internet Protocol, certain well-known ports are assigned to server processes at system start up. For example, port 21 is assigned to ftp, 22 to ssh, 23 to telnet, and 80 to http. There are TCP and UDP numbers, but typically the ftp server requests both TCP port number 21 and UDP port number 21.

Lecture 3

Three significant characteristics of a Distributed System:

1. Concurrent execution

· concurrency: several processes are in between starting and finishing at the same time

· parallelism: several processes are executing at the same time

2. No global clock

· limits to how closely computers can synchronize their clocks

3. Independent failures

· any component of the network can fail and software must deal with this

Prime Motivation for Distributed Systems: resource sharing

· hardware components: printers, disks

· software components: files, databases, other data objects

· transient components: stream of video frames from digital camera, audio connection to a mobile phone

Other Motivations:

· economic (cheap) – not mentioned in CDK

· fast: concurrency and scalability

· reliable: secure and fault-tolerant

· extendible: heterogeneity and openness

· convenient: transparency

Challenges for Distributed Systems

1. Heterogeneity: variety and differences in networks, computer hardware, operating systems, programming languages, and implementations by different developers

• Middleware: software layer that provides a programming abstraction as well as masking the heterogeneity of the underlying systems (networks, hardware, operating systems, and programming languages): e.g., CORBA and Java RMI.
• Mobile code: code that can be sent from one computer to another and run at the destination; e.g., Java applets.

2. Openness: system made extensible by publishing the specification and documentation for key software interfaces, i.e. made available to software developers.

3. Security: confidentiality (protection against disclosure to unauthorized individuals), integrity (protection against alteration or corruption), and availability (protection against interference with the means to access the resource).

• Denial of service attacks: bombard a service with such a large number of pointless requests that serious users are unable to use it.

4. Scalability: system will remain effective if there is a significant increase in the number of resources and users

• Cost of physical resources should O(n) for n users, performance on lookup tasks should be no worse than O(log n) for data of size n, software resources shouldn’t run out, and performance bottlenecks should be avoided.
• Techniques for avoiding bottlenecks: distributed algorithms; caching, replication, and other forms of distributed data; e.g., multiple servers.

5. Failure Handling: failure may be partial in a distributed system because some components may fail while others continue processing.

• Techniques are needed for failure detection, failure masking, failure tolerance (by software and users), and recovery from failure.
• Use of redundant components is crucial to failure handling: e.g., multiple routes, Domain Name Servers, and databases.

6. Concurrency: allowing requests from several clients to be processed such that they are in between starting and finishing at the same time

• Operations on any object in a distributed environment must be synchronized such that the object’s data remains consistent

7. Transparency: concealment from the user and application programmer of the separation of components in a distributed system , so that the system is perceived as a whole rather than as a collection of independent components.

Tanenbaum’s Types of Transparency

1. Location Transparency

· users can not tell where resources such as CPUs, printers, files, and database are located

· also cannot tell where other users are located

2. Migration Transparency (replaced by Mobility Transparency in CDK)

· resources can be moved from one location to another without their names changing

· e.g., files or printers moving between servers

3. Replication Transparency

· users don’t need to know how many copies of resources are available

· e.g., files

4. Concurrency Transparency

· system ensures resources are accessed in a sequential manner (at some level of detail)

· users do not have to consider other users

· a more elaborate scheme than simple file locking

5. Parallelism Transparency (absorbed into Replication Transparency)

· computer, runtime system and OS make use of all processors available without explicit coding by user

· e.g., same executable creates more parallel processes when more processors are available

CDK Types of Transparency

1. Access Transparency: enables local and remote resources to be accessed using identical operations

2. Location Transparency: enables resources to be accessed without knowledge of their (physical) location

a. Tanenbaum's number 1

b. Access transparency and location transparency are together referred to as network transparency.

3. Concurrency Transparency: enables several processes to operate concurrently using shared resources without interference between them

a. Tanenbaum's number 4

4. Replication Transparency: enables multiple instances of resources to be used to increase reliability and performance without knowledge of the replicas by users or application programmers

a. Tanenbaum's number 3 and 5

5. Failure Transparency: enables the concealment of faults, allowing users and application programs to complete their tasks despite the failure of hardware or software components

6. Mobility Transparency: allows the movement of resources and clients within a system without affecting the operation of users or programs

a. Tanenbaum's number 2

7. Performance Transparency: allows the system to be reconfigured to improve performance as loads change

8. Scaling Transparency: Transparency: allows the system and applications to expand in scale without change to the system structure or the application algorithms

Advantages of Distributed Systems

a) Compared to centralized systems:

• better price/performance than mainframe
• more total computing power
• sum of the computing power of the processors in the distributed system may be greater than any single processor available (parallel processing)
• some applications are inherently distibuted
• improved reliability because system can survive crash of one processor
• incremental growth can be achieved by adding one processor at a time
• shared ownership facilitated

b) Compared to isolated PCs:

• shared management of system
• backups
• maintenance: download new software
• data sharing: many users can access a shared database
• device sharing: share expensive peripherals
• communication between users easier (e.g., electronic mail)
• flexibility: spread workload over machines (the jobs are not necessarily run on the owner’s machine)

Disadvantages of Distributed Systems

• software may not be available
• networks can become saturated (may need additional wiring)
• security problems protecting data (only an isolated machine is safe)
• incremental growth is difficult in practice because of changing hardware and software

Scalability of Distributed Systems

• think about very large systems
• avoid centralized components, tables, and algorithms
• e.g., for a mail server:
• centralized mail server is no good
• it’s better to have a hierarchy of servers with a “committee on top”
• much mail is local or semilocal
• most mail from cs goes to cs
• most mail from uregina goes to uregina
• most of rest to Canada
• e.g., phone book
• centralized phone book requires all updates to same place
• will be overload
• better to have local phone books
• decentralized algorithms have the following characteristics:
• no machine has complete info
• decisions based on local information
• failure of one machine does not ruin algorithm
• no implicit assumption of a global clock
• if there are many machines operating at the same time, synchronization CANNOT be achieved

Reliability of a Distributed System

• three components of reliability are availability, security, and fault tolerance

1. availability

• the fraction of the time the system is usable
• try to reduce dependence between servers because this increases the chance that the user can still do something
• to improve availability, key components should be replicated

2. security

• must authenticate every message in a distributed OS
• can’t trust id field in a message
• should also encrypt contents to prevent snooping on the network
• Secure Sockets Layer (SSL) provides this type of security

3. fault tolerance

• refers to the general ability of the system to tolerate errors
• most often achieved by having multiple instances of a resource
• server should be able to crash and recover without affecting service

- may want to reject the request that caused crash

• should be able to vary number by choice

- performance degrades when one crashes or is removed from service

• perhaps if one crashes, others should take over its work queue

REMAINING NOTES WERE NOT COVERED IN CLASS

MAY BE USED IN LATER LECTURES

Hardware for Distributed Systems

a) distributed systems are MIMD, i.e., have multiple instruction streams and multiple data streams

b) matrix:

type | shared memory? |

of | Yes = Multiprocessor | No = Multicomputer |

connection | (usually tightly coupled) | (usually loosely coupled) |

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

bus | Sequent , Encore | Workstation on a LAN |

| SGI (Grendel) | (Linux boxes) |

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

switched | Ultracomputer, RP3 | Hypercube, Transputer |

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

c) tightly coupled hardware

• there is a small delay when sending messages
• fast rate of data transfer
• usually used in parallel systems
• parallel algorithm: processors work on the same problem

d) loosely coupled hardware

• there is a longer delay when sending messages
• slower rate of data transfer

e) multiprocessor

• a distributed system in which the processors share memory, i.e., they all share a single virtual address space
• symmetric multiprocessor (SMP): all processors are of the same type

f) multicomputer

• a distributed system in which the processors do NOT share memory
• an example: a collection of personal computers connected by a network

g) bus system

• a distributed system with a single network, backplane, bus, cable, or other medium which connects all processors

h) switched system

• a distributed system which has individual wires from machine to machine
• many different wiring patterns
• messages move along wires
• switching (routing) decisions are made step by step along the route

i) bus-based multiprocessors

• one memory for several processors
• performance drops when bus gets overloaded
• (when several CPU’s are attached to the bus)
• adding cache memory may allow the addition of more CPU’s before the bus becomes overloaded

Software for Distributed Systems

• determines user view of system
• loosely connected software: gives view that there are many machines
• tightly coupled software: organizes system for single task

Types of Software for Distributed Systems

a) Network Operating System

• loosely connected software on loosely connected hardware (multicomputer)
• network of workstations or PC’s
• each machine running an OS; the OSes may even differ
• e.g., our lab of Linux machines

b) Integrated Distributed System

• tightly coupled software on loosely coupled hardware
• runs on a collection of shared machines that do not have shared memory yet work like one computer
• we don’t have one
• we had a package called Xpress that gave many of these features, but it silently disappeared because no one used it

c) Multiprocessor Timesharing System

• tightly coupled software on tightly coupled hardware
• e.g., UNIX machine with several processors
• shared memory
• one short-term scheduler queue (kept in shared memory)
• e.g., Grendel (24 processor SGI super computer)

d) Doesn’t make sense:

• loosely coupled software on tightly coupled hardware

Performance of a Distributed System

• measures:

• response time
• number of jobs per hour
• system utilization
• fraction of network capacity

• strongly affected by messaging delays

• transmission time
• protocol handling

• must distribute inherently distributed computations. i.e. info sharing among users
• perhaps best addressed by grain size of computation

o distribute computations with course-grained parallelism (large computations, low interaction rates, little data)

o don’t distribute computations with fine-grained parallelism (small computations, interact with one another)

Multiprocessor Timesharing Systems

• not an example of a distributed system for this course
• tightly coupled software on tightly coupled hardware
• most common: UNIX machine with multiple CPU’s, e.g., Grendel or Zeus
• Example:

P1 P2 P2 shared

cache cache cache memory

| | | |

================================== bus

In (shared) main memory:

• OS code, including scheduler
• OS data structures, including run queue
• protected by semaphores to allow mutual exclusion

Short-term scheduler’s queue (run queue)

• contains runnable processes: A, B, C, D, E

Processors: P1, P2, P3

• A is running on P1
• B is running on P2
• C is running on P3

Process B blocks (e.g., requires I/O)

• P2 runs OS code
• P2 obtains exclusive access to run queue (acquires semaphore)
• P2 quickly removes first entry (D) from the run queue
• P2 releases semaphore for run queue
• P2 runs D

Features:

• each processor has its own cache
• given several available CPUs, a process should be run on the CPU where it last executed, because some useful items may still be in the cache
• single shared file system: OS has a single block cache, used by all processors
• with appropriate mutual exclusion, this method avoids missing updates to files done by other processors