Report

A.E. Csallner Department of Applied Informatics University of Szeged Hungary Algorithm: Finite sequence of finite steps Provides the solution to a given problem Properties: Finiteness Definiteness Executability About algorithms Communication: Input Output Algorithms and Data Structures I 2 Design strategies: Bottom-up: synthesize smaller algorithmic parts into bigger ones Top-down: formulate the problem and repeatedly break it up into smaller and smaller parts About algorithms Algorithms and Data Structures I 3 Example : Shoe a horse shoe a horse a horse has four hooves shoe a hoof need a horseshoe hammer a horseshoe need to fasten the horseshoe to the hoof drive a cog into a hoof need cogs hammer a cog Structured programming Algorithms and Data Structures I 4 Basic elements of structured programming Sequence: series of actions Selection: branching on a decision Iteration: conditional repetition All structured algorithms can be defined using only these three elements (E.W. DIJKSTRA 1960s) Structured programming Algorithms and Data Structures I 5 An algorithm description method defines an algorithm so that the description code should be unambiguous; programming language independent; still easy to implement; state-of-the-art Algorithm description Algorithms and Data Structures I 6 Some possible types of classification: Age (when the description method was invented) Purpose (e.g. structural or object-oriented) Formulation (graphical or text code, etc.) ... Algorithm description Algorithms and Data Structures I 7 Most popular and useful description methods Flow diagram old not definitely structured(!) graphical very intuitive and easy to use Algorithm description Algorithms and Data Structures I 8 A possible notation of flow diagrams Circle: START STOP STOP Algorithm description Algorithms and Data Structures I 9 A possible notation of flow diagrams Rectangle: Any action execution can be given here Algorithm description Algorithms and Data Structures I 10 A possible notation of flow diagrams Diamond: Any yes/no question yes Algorithm description no yes Algorithms and Data Structures I 11 A possible notation of flow diagrams An example: START Iteration Selection Sequence Need more horseshoes? yes Hammer a horseshoe no Shoe a hoof STOP Algorithm description Algorithms and Data Structures I 12 Most popular and useful description methods Pseudocode old definitely structured text based very easy to implement Algorithm description Algorithms and Data Structures I 13 Properties of a possible pseudocode Assignment instruction: Looping constructs as in Pascal: for-do instruction (counting loop) for variable initial value to/downto final value do body of the loop Algorithm description Algorithms and Data Structures I 14 Properties of a possible pseudocode while-do instruction (pre-test loop) while stay-in test do body of the loop repeat-until instruction (post-test loop) repeat body of the loop until exit test Algorithm description Algorithms and Data Structures I 15 Properties of a possible pseudocode Conditional constructs as in Pascal: if-then-else instruction (else clause is optional) if test then test passed clause else test failed clause Blocks are denoted by indentation Algorithm description Algorithms and Data Structures I 16 Properties of a possible pseudocode Object identifiers are references Field of an object separator is a dot: object.field object.method object.method(formal parameter list) Empty reference is NIL Algorithm description Algorithms and Data Structures I 17 Properties of a possible pseudocode Arrays are objects Parameters are passed by value Algorithm description Algorithms and Data Structures I 18 Properties of a possible pseudocode An example: ShoeAHorse(Hooves) hoof 1 Iteration while hoof ≤ Hooves.Count do horseshoe HammerAHorseshoe Hooves[hoof] horseshoe hoof hoof + 1 Algorithm description Algorithms and Data Structures I Sequence 19 Algorithm classification on the I/O structure Sequence → Value Sequence → Sequence More sequences → Sequence Sequence → More sequences Type algorithms Algorithms and Data Structures I 20 Sequence → Value sequence calculations (e.g. summation, product of a series, linking elements together, etc.), decision (e.g. checking whether a sequence contains any element with a given property), selection (e.g. determining the first element in a sequence with a given property provided we know that there exists at least one), Type algorithms Algorithms and Data Structures I 21 Sequence → Value (continued) search (e.g. finding a given element), counting (e.g. counting the elements having a given property), minimum or maximum search (e.g. finding the least or the largest element). Type algorithms Algorithms and Data Structures I 22 Sequence → Sequence selection (e.g. collect the elements with a given property of a sequence), copying (e.g. copy the elements of a sequence to create a second sequence), sorting (e.g. arrange elements into an increasing order). Type algorithms Algorithms and Data Structures I 23 More sequences → Sequence union (e.g. set union of sequences), intersection (e.g. set intersection of sequences), difference (e.g. set difference of sequences), uniting sorted sequences (merging / combing two ordered sequences). Type algorithms Algorithms and Data Structures I 24 Sequence → More sequences filtering (e.g. filtering out elements of a sequence having given properties). Type algorithms Algorithms and Data Structures I 25 Iterative algorithm Consists of two parts: Initialization (usually initializing data) Iteration (repeated part) Special algorithms Algorithms and Data Structures I 26 Recursive algorithms Basic types: direct (self-reference) indirect (mutual references) Two alternative parts depending on the base criterion: Base case (if the problem is small enough) Recurrences (direct or indirect self-reference) Special algorithms Algorithms and Data Structures I 27 An example of recursive algorithms: Towers of Hanoi Aim: Move n disks from a rod to another, using a third one Rules: One disk moved at a time No disk on top of a smaller one Special algorithms Algorithms and Data Structures I 28 Recursive solution of the problem 1st step: move n–1 disks 2nd step: move 1 disk Special algorithms 3rd step: move n–1 disks Algorithms and Data Structures I 29 Pseudocode of the recursive solution TowersOfHanoi(n,FirstRod,SecondRod,ThirdRod) 1 if n > 0 2 then TowersOfHanoi(n – 1,FirstRod,ThirdRod,SecondRod) 3 write “Move a disk from ” FirstRod “ to ” SecondRod 4 TowersOfHanoi(n – 1, ThirdRod,SecondRod,FirstRod) line 2 line 3 Special algorithms line 4 Algorithms and Data Structures I 30 Backtracking algorithms Backtracking algorithm: Sequence of systematic trials Builds a tree of decision branches Steps back (backtracking) in the tree if no branch at a point is effective Special algorithms Algorithms and Data Structures I 31 An example of the backtracking algorithms Eight Queens Puzzle: eight chess queens to be placed on a chessboard so that no two queens attack each other Special algorithms Algorithms and Data Structures I 32 Pseudocode of the iterative solution EightQueens 1 column 1 2 RowInColumn[column] 0 3 repeat 4 repeat inc(RowInColumn[column]) 5 until IsSafe(column, RowInColumn) 6 if RowInColumn[column] > 8 7 then column column – 1 8 else if column < 8 9 then column column + 1 10 RowInColumn[column] 0 11 else draw chessboard 12 until column = 0 Special algorithms Algorithms and Data Structures I 33 Questions regarding an algorithm: Does it solve the problem? How fast does it solve the problem? How much storage place does it occupy to solve the problem? Complexity issues of the algorithm Analysis of algorithms Algorithms and Data Structures I 34 Elementary storage or time: independent from the size of the input. Example 1 If an algorithm needs 500 kilobytes to store some internal data, this can be considered as elementary. Example 2 If an algorithm contains a loop whose body is executed 1000 times, it counts as an elementary algorithmic step. Analysis of algorithms Algorithms and Data Structures I 35 Hence a block of instructions count as a single elementary step if none of the particular instructions depends on the size of the input. A looping construct counts as a single elementary step if the number of iterations it executes does not depend on the size of the input and its body is an elementary step. ⇒ to shoe a horse can be considered as an elementary step ⇔ it takes constant time (one step) to shoe a horse Analysis of algorithms Algorithms and Data Structures I 36 The time complexity of an algorithm is a function depending on the size of the input. Notation: T(n) where n is the size of the input Function T can depend on more than one variable, e.g. T(n,m) if the input of the algorithm is an n⨯m matrix. Analysis of algorithms Algorithms and Data Structures I 37 Example: Find the minimum of an array. Minimum(A) 1 min A[1] 1 2 i1 3 repeat 4 ii+1 5 if A[i] < min 6 then min A[i] 7 until i A.Length 8 return min Analysis of algorithms Algorithms and Data Structures I 1 n−1 38 Hence T(n) = n (where n = A.Length) Does this change if line 8 (return min) is considered as an extra step? ? In other words: n ≈ n + 1 this counts as a single elementary step It does not change! Proof: n + 1 = (n − 1) + 2 ≈ (n − 1) + 1 = n Analysis of algorithms Algorithms and Data Structures I 39 This so-called asymptotic behavior can be formulated rigorously in the following way: We say that f (x) = O(g(x)) (big O notation) if (∃C, x0 > 0) (∀x ≥ x0) 0 ≤ f (x) ≤ C∙g(x) means that g is an asymptotic upper bound of f Analysis of algorithms Algorithms and Data Structures I 40 C∙g(x) f (x) g(x) x0 Analysis of algorithms Algorithms and Data Structures I 41 The O notation denotes an upper bound. If g is also a lower bound of f then we say that f (x) = θ (g(x)) if (∃c, C, x0 > 0) (∀x ≥ x0) 0 ≤ c∙g(x) ≤ f (x) ≤ C∙g(x) means that f asymptotically equals g Analysis of algorithms Algorithms and Data Structures I 42 f (x) C∙g(x) g(x) c∙g(x) x0C Analysis of algorithms x0c =x0 Algorithms and Data Structures I 43 What does the asymptotic notation show us? We have seen: T(n) = θ (n) for the procedure Minimum(A) where n = A.Length However, due to the definition of the θ function T(n) = θ (n), T(2n) = θ (n), T(3n) = θ (n) ... ? Minimum does not run slower on more data? Analysis of algorithms Algorithms and Data Structures I 44 What does the asymptotic notation show us? Asymtotic notation shows us the tendency: T(n) = θ (n (n)2)linear quadratic tendency tendency n data → a certain amount of time t 2n data → time ≈ 22t2t = 4t 3n data → time ≈ 33t2t = 9t Analysis of algorithms Algorithms and Data Structures I 45 Recursive algorithm – recursive function T Example: Towers of Hanoi TowersOfHanoi(n,FirstRod,SecondRod,ThirdRod) 1 if n > 0 2 then TowersOfHanoi(n – 1,FirstRod,ThirdRod,SecondRod) 3 write “Move a disk from ” FirstRod “ to ” SecondRod 4 TowersOfHanoi(n – 1, ThirdRod,SecondRod,FirstRod) T(n)= Analysis of algorithms T(n−1) +T(n−1) +1 Algorithms and Data Structures I =2T(n−1)+1 46 T(n) = 2T(n−1) + 1 is a recursive function In general it is very difficult (sometimes insoluble) to determine the explicit form of an implicit (recursive) formula If the algorithm is recursive, the solution can be achieved using recursion trees. T(n)= Analysis of algorithms =2T(n−1)+1 Algorithms and Data Structures I 47 Recursion tree of TowersOfHanoi: n n−1 n−2 1 1 1 1 n−1 1 n−2 n−2 1 1 1 1 1 1 n−2 2 1 4 2n−1 2n−1 Analysis of algorithms Algorithms and Data Structures I 48 Time complexity: T(n) = 2n − 1 = θ (2n) − exponential time (very slow) Example: n = 64 (from the original legend) T(n) = 2n − 1 = 264 − 1 = (assuming one disk move per second) ≈ 1.8∙1019 seconds = ≈ 3∙1017 minutes = ≈ 5.1∙1015 hours = ≈ 2.1∙1014 days = ≈ 5.8∙1011 years > half a trillion years Analysis of algorithms Algorithms and Data Structures I 49 Problem (example): search a given element in a sequence (array). LinearSearch(A,w) 1 i0 2 repeat i i + 1 3 until A[i] = w or i = A.Length 4 if A[i] = w then return i 5 else return NIL Analysis of algorithms Algorithms and Data Structures I 50 Array: 8 1 3 9 5 6 2 Best case Element wanted: 8 Time complexity: T(n) = 1 = θ (1) Worst case Element wanted: 2 Time complexity: T(n) = n = θ (n) Average case? Analysis of algorithms Algorithms and Data Structures I 51 Array: 8 1 3 9 5 6 2 The mean value of the time complexities on all possible inputs: T(n) = ( 1 + 2+ 3+ 4+ ... + n ) / n = = n∙(n + 1) / 2n = (n + 1) / 2 = θ (n) (The same as in the worst case) Average case? Analysis of algorithms Algorithms and Data Structures I 52 To store a set of data of the same type in a linear structure, two basic solutions exist: Arrays: physical sequence in the memory 18 29 22 Linked lists: the particular elements are linked together using links (pointers or indices) head 18 key Arrays and linked lists 29 22 link Algorithms and Data Structures I 53 Arrays vs. linked lists Time complexity of some operations on arrays and linked lists in the worst case Search Insert Delete Minimum Maximum Successor Predecessor Array O(n) O(n) O(n) O(n) O(n) O(n) O(n) Linked list O(n) O(1) O(1) O(n) O(n) O(n) O(n) Arrays and linked lists Algorithms and Data Structures I 54 Doubly linked lists: 18 head 22 Dummy head lists: dummy head 29 X 18 29 22 pointer Indirection (indirect reference): pointer.key Double indirection: pointer.link.key to be continued... Arrays and linked lists Algorithms and Data Structures I 55 dummy head Array representation of linked lists X 18 1 key link dummy head Arrays and linked lists 3 2 3 29 4 5 6 22 7 22 X 18 29 0 7 2 5 8 Problem: a lot of garbage Algorithms and Data Structures I 56 Garbage collection for array-represented lists The empty cells are linked to a separate garbage list using the link array: 1 dummy head Arrays and linked lists 3 4 22 X key link 2 8 3 0 5 5 6 18 0 garbage 7 8 29 7 1 2 4 6 Algorithms and Data Structures I 57 To allocate place for a new key and use it: the first element of the garbage list is linked out from the garbage and linked into the proper list with a new key (33 here) if necessary. 1 dummy head Arrays and linked lists 3 4 22 X key link 2 8 3 0 5 6 5 6 7 8 18 33 29 0 garbage 7 1 5 6 1 Algorithms and Data Structures I 2 4 new 6 58 Pseudocode for garbage management Allocate(link) 1 if link.garbage = 0 2 then return 0 3 else new link.garbage 4 link.garbage link[link.garbage] 5 return new Free(index,link) 1 link[index] link.garbage 2 link.garbage index Arrays and linked lists Algorithms and Data Structures I 59 Dummy head linked lists (...continued) FindAndDelete for simple linked lists FindAndDelete(toFind,key,link) 1 if key[link.head] = toFind extra case: 2 then toDelete link.head the first element is to be deleted 3 link.head link[link.head] 4 Free(toDelete,link) 5 else toDelete link[link.head] an additional pointer is needed 6 pointer link.head to step forward 7 while toDelete 0 and key[toDelete] toFind 8 do pointer toDelete 9 toDelete link[toDelete] 10 if toDelete 0 11 then link[pointer] link[toDelete] 12 Free(toDelete,link) Arrays and linked lists Algorithms and Data Structures I 60 Dummy head linked lists (...continued) FindAndDelete for dummy head linked lists FindAndDeleteDummy(toFind,key,link) 1 pointer link.dummyhead 2 while link[pointer] 0 and key[link[pointer]] toFind 3 do pointer link[pointer] 4 if link[pointer] 0 5 then toDelete link[pointer] 6 link[pointer] link[toDelete] 7 Free(toDelete,link) Arrays and linked lists Algorithms and Data Structures I 61 Common properties: only two operations are defined: store a new key (called push and enqueue, resp.) extract a key (called pop and dequeue, resp.) all (both) operations work in constant time Different properties: stacks are LIFO structures queues are FIFO (or pipeline) structures Stacks and queues Algorithms and Data Structures I 62 Two erroneous cases: an empty data structure is intended to be extracted from: underflow no more space but insertion attempted: overflow Stacks and queues Algorithms and Data Structures I 63 Stack management using arrays push(8) Stack: push(1) 3 push(3) 1 push(9) Stack overflow pop 8 top pop pop pop Stack underflow Stacks and queues Algorithms and Data Structures I 64 Stack management using arrays Push(key,Stack) 1 if Stack.top = Stack.Length stack overflow 2 then return Overflow error 3 else Stack.top Stack.top + 1 4 Stack[Stack.top] key Stacks and queues Algorithms and Data Structures I 65 Stack management using arrays Pop(Stack) 1 if Stack.top = 0 stack underflow 2 then return Underflow error 3 else Stack.top Stack.top − 1 4 return Stack[Stack.top + 1] Stacks and queues Algorithms and Data Structures I 66 Queue management using arrays end ↓ Queue: 8 3 1 6 4 7 Empty ?queue: •beginning = n 2 9 5 •end = 0 ← beginning Stacks and queues Algorithms and Data Structures I 67 Queue management using arrays Enqueue(key,Queue) 1 if Queue.beginning = Queue.end queue overflow 2 then return Overflow error 3 else if Queue.end = Queue.Length 4 then Queue.end 1 5 else Queue.end Queue.end + 1 6 Queue[Queue.end] key Stacks and queues Algorithms and Data Structures I 68 Queue management using arrays Dequeue(Queue) 1 if Queue.end = 0 queue underflow 2 then return Underflow error 3 else if Queue.beginning = Queue.Length 4 then Queue.beginning 1 5 else inc(Queue.beginning) 6 key Queue[Queue.beginning] 7 if Queue.beginning = Queue.end 8 then Queue.beginning Queue.Length 9 Queue.end 0 10 return key Stacks and queues Algorithms and Data Structures I 69 Linear data structures cannot provide better time complexity than n in some cases Idea: let us use another kind of structure Solution: rooted trees (especially binary trees) special order of keys (‘search trees’) Binary search trees Algorithms and Data Structures I 70 A binary tree: Notions: depth (height) levels root vertex (node) edge parent - child twins (siblings) Binary search trees leaf Algorithms and Data Structures I 71 A binary tree: search 28 all keys in the left subtree are smaller 12 7 30 21 14 Binary search trees for all vertices all keys in the right subtree are greater 49 26 Algorithms and Data Structures I 50 72 Implementation of binary search trees: 28 12 link to the parent 30 key and other data 7 21 14 Binary search trees 49 to the link left child 26 Algorithms and Data Structures I link to the right child 50 73 increasing order Binary search tree operations: tree walk 28 inorder: 1. left 2. root 3. right 12 7 21 14 Binary search trees 30 49 26 Algorithms and Data Structures I 7 12 14 21 26 28 30 49 50 50 74 InorderWalk(Tree) 1 if Tree NIL 2 then InorderWalk(Tree.Left) 3 visit Tree, e.g. check it or list it 4 InorderWalk(Tree.Right) The so-called preorder and postorder tree walks only differ by the order of lines 2-4: preorder: root → left → right postorder: left → right → root Binary search trees Algorithms and Data Structures I 75 Binary search tree operations: tree search 28 < < 12 < 21 49 < 14 Binary search trees TreeSearch(45) 30 < 7 TreeSearch(14) < 26 Algorithms and Data Structures I 50 76 TreeSearch(toFind,Tree) 1 while Tree NIL and Tree.key toFind 2 do if toFind < Tree.key 3 then Tree Tree.Left 4 else Tree Tree.Right 5 return Tree Binary search trees Algorithms and Data Structures I 77 Binary search tree operations: insert 28 < 12 TreeInsert(14) 30 < 7 21 49 < new vertices are always inserted 14 as leaves Binary search trees 26 Algorithms and Data Structures I 50 78 Binary search tree operations: tree minimum 28 tree maximum 12 7 21 14 Binary search trees 30 49 26 Algorithms and Data Structures I 50 79 TreeMinimum(Tree) 1 while Tree.Left NIL 2 do Tree Tree.Left 3 return Tree TreeMaximum(Tree) 1 while Tree.Right NIL 2 do Tree Tree.Right 3 return Tree Binary search trees Algorithms and Data Structures I 80 Binary search tree operations: successor of an 28 element parent-left child relation 12 7 30 21 tree 14minimum 26 Binary search trees Algorithms and Data Structures I TreeSuccessor(12) if the element has no right child: TreeSuccessor(26) 49 50 81 TreeSuccessor(Element) 1 if Element.Right NIL 2 then return TreeMinimum(Element.Right) 3 else Above Element.Parent 4 while Above NIL and Element = Above.Right 5 do Element Above 6 Above Above.Parent 7 return Above Finding the predecessor is similar. Binary search trees Algorithms and Data Structures I 82 Binary search tree operations: delete 1. if the element has 28 no children: TreeDelete(26) 12 7 21 14 Binary search trees 30 49 26 Algorithms and Data Structures I 50 83 Binary search tree operations: delete 2. if the element has 28 only one child: TreeDelete(30) 12 7 21 14 Binary search trees 30 49 26 Algorithms and Data Structures I 50 84 Binary search tree operations: delete 3. if the element has 28 12 is substituted for a close key, e.g. the successor, 14 12 7 two children: TreeDelete(12) 30 21 the successor, found in the right tree subtree has at 14minimum 26 most one child Binary search trees Algorithms and Data Structures I 49 50 85 The case if Element has no children: TreeDelete(Element,Tree) 1 if Element.Left = NIL and Element.Right = NIL 2 then if Element.Parent = NIL 3 then Tree NIL 4 else if Element = (Element.Parent).Left 5 then (Element.Parent).Left NIL 6 else (Element.Parent).Right NIL 7 Free(Element) 8 return Tree 9- next page Binary search trees Algorithms and Data Structures I 86 The case if Element has only a right child: -8 previous page 9 if Element.Left = NIL and Element.Right NIL 10 then if Element.Parent = NIL 11 then Tree Element.Right 12 (Element.Right).Parent NIL 13 else (Element.Right).Parent Element.Parent 14 if Element = (Element.Parent).Left 15 then (Element.Parent).Left Element.Right 16 else (Element.Parent).Right Element.Right 17 Free(Element) 18 return Tree 19- next page Binary search trees Algorithms and Data Structures I 87 The case if Element has only a left child: -18 previous page 19 if Element.Left NIL and Element.Right = NIL 20 then if Element.Parent = NIL 21 then Tree Element.Left 22 (Element.Left).Parent NIL 23 else (Element.Left).Parent Element.Parent 24 if Element = (Element.Parent).Left 25 then (Element.Parent).Left Element.Left 26 else (Element.Parent).Right Element.Left 27 Free(Element) 28 return Tree 29-next page Binary search trees Algorithms and Data Structures I 88 The case if Element has two children: -28 previous page 29 if Element.Left NIL and Element.Right NIL 30 then Substitute TreeSuccessor(Element) Substitute is linked out 31 if Substitute.Right NIL from its place 32 then (Substitute.Right).Parent Substitute.Parent 33 if Substitute = (Substitute.Parent).Left 34 then (Substitute.Parent).Left Substitute.Right 35 else (Substitute.Parent).Right Substitute.Right 36 Substitute.Parent Element.Parent 37 if Element.Parent = NIL Substitute is linked into 38 then Tree Substitute Elements place 39 else if Element = (Element.Parent).Left 40 then (Element.Parent).Left Substitute 41 else (Element.Parent).Right Substitute 42 Substitute.Left Element.Left 43 (Substitute.Left).Parent Substitute 44 Substitute.Right Element.Right 45 (Substitute. Right).Parent Substitute 27 Free(Element) 28 return Tree Binary search trees Algorithms and Data Structures I 89 Time complexity of binary search tree operations T(n) = O(d) for all operations (except for the walk), where d denotes the depth of the tree The depth of any randomly built binary search tree is d = O(log n) Hence the time complexity of the search tree operations in the average case is T(n) = O(log n) Stacks and queues Algorithms and Data Structures I 90 If insert and delete is used rarely then it is more convenient and faster to use an oredered array instead of a binary search tree. Faster: the following operations have T(n) = O(1) constant time complexity: minimum, maximum, successor, predecessor. Search has the same T(n) = O(log n) time complexity as on binary search trees: Binary search Algorithms and Data Structures I 91 Search has the same T(n) = O(log n) time complexity as on binary search trees: Let us search key 29 in the ordered array below: central element 2 3 < 7 12 29 31 45 search here Binary search Algorithms and Data Structures I 92 Search has the same T(n) = O(log n) time complexity as on binary search trees: Let us search key 29 in the ordered array below: < 2 3 central element 7 12 29 31 45 search here Binary search Algorithms and Data Structures I 93 Search has the same T(n) = O(log n) time complexity as on binary search trees: Let us search key 29 in the ordered array below: central element = 2 3 found! 7 12 29 31 45 search here Binary search Algorithms and Data Structures I 94 Search has the same T(n) = O(log n) time complexity as on binary search trees: O(log n) 2 3 7 12 29 31 45 This result can also be derived from: if we halve n elements k times, we get 1 ⇔ n / 2k = 1 ⇔ k = log2 n = O(log n) Binary search Algorithms and Data Structures I 95 Problem There is a set of data from a base set with a given order over it (e.g. numbers, texts). Arrange them according to the order of the base set. Example 12 2 Sorting 7 3 Algorithms and Data Structures I sorting 96 Sorting sequences We sort sequences in a lexicographical order: from two sequences the sequence is ‘smaller’ which has a smaller value at the first position where they differ. Example (texts) g o n e <? g o o d n < o in the alphabet Sorting Algorithms and Data Structures I 97 Principle 14 8 69 22 75 Insertion sort Algorithms and Data Structures I 98 Implementation of insertion sort with arrays insertion step: 22 69 75 38 14 sorted part Insertion sort unsorted part Algorithms and Data Structures I 99 InsertionSort(A) 1 for i 2 to A.Length 2 do ins A[i] 3 ji–1 4 while j > 0 and ins < A[j] 5 do A[j + 1] A[j] 6 jj–1 7 A[j + 1] ins Insertion sort Algorithms and Data Structures I 100 Time complexity of insertion sort Best case In each step the new element is inserted to the end of the sorted part: T(n) = 1 + 1 + 1 +...+ 1 = n − 1 = θ (n) Worst case In each step the new element is inserted to the beginning of the sorted part: T(n) = 2 + 3 + 4 +...+ n = n(n + 1)/2 − 1 = θ (n2) Insertion sort Algorithms and Data Structures I 101 Time complexity of insertion sort Average case In each step the new element is inserted somewhere in the middle of the sorted part: T(n) = 2/2 + 3/2 + 4/2 +...+ n/2 = = (n(n + 1)/2 − 1) / 2 = θ (n2) The same as in the worst case Insertion sort Algorithms and Data Structures I 102 Another implementation of insertion sort The input is providing elements continually (e.g. file, net) The sorted part is a linked list where the elements are inserted one by one The time complexity is the same in every case. Insertion sort Algorithms and Data Structures I 103 Another implementation of insertion sort The linked list implementation delivers an on-line algorithm: after each step the subproblem is completely solved the algorithm does not need the whole input to partially solve the problem Cf. off-line algorithm: the whole input has to be known prior to the substantive procedure Insertion sort Algorithms and Data Structures I 104 Principle 8 14 14 69 69 8 75 75 25 2 2 22 22 25 36 36 sort the parts recursively Merge sort Algorithms and Data Structures I 105 8 14 69 75 2 22 25 36 merge (comb) readythe parts Merge sort Algorithms and Data Structures I 106 Time complexity of merge sort Merge sort is a recursive algorithm, and so is its time complexity function T(n) What it does: First it halves the actual (sub)array: O(1) Then calls itself for the two halves: 2T(n/2) Last it merges the two ordered parts: O(n) Hence T(n) = 2T(n/2) + O(n) = ? Merge sort Algorithms and Data Structures I 107 Recursion tree of merge sort: n n n/2 n/4 1 n/2 n/4 1 n/4 1 2(n/2) n/4 1 4(n/4) n n∙log n Merge sort Algorithms and Data Structures I 108 Time complexity of merge sort is T(n) = θ (n∙logn) This worst case time complexity is optimal among comparison sorts (using only pair comparisons) ⇒ fast but unfortunately merge sort does not sort in-place, i.e. it uses auxiliary storage of a size comparable with the input Merge sort Algorithms and Data Structures I 109 An array A is called heap if for all its elements A[i] ≥ A[2i] and A[i] ≥ A[2i + 1] 45 27 34 20 23 31 18 19 3 14 This property is called heap property It is easier to understand if a binary tree is built from the elements filling the levels row by row Heapsort Algorithms and Data Structures I 110 45 27 34 20 23 31 18 19 3 14 Heapsort Algorithms and Data Structures I 111 1 2 4 8 20 19 Heapsort 3 27 5 9 3 45 10 14 23 6 31 34 7 18 The heap property turns into a simple parent-child relation in the tree representation Algorithms and Data Structures I 112 An important application of heaps is realizing priority queues: A data structure supporting the operations insert maximum (or minimum) extract maximum (or extract minimum) Heapsort Algorithms and Data Structures I 113 First we have to build a heap from an array. Let us suppose that only the kth element infringes the heap property. In this case it is sunk level by level to a place where it fits. In the example k = 1 (the root): Heapsort Algorithms and Data Structures I 114 1 2 4 8 20 19 Heapsort 3 37 5 9 3 15 10 14 23 6 31 34 7 18 k=1 •The key and its children are compared •It is exchanged for the greater child Algorithms and Data Structures I 115 1 2 4 8 20 19 Heapsort 3 15 5 9 3 37 10 14 23 6 31 34 7 18 k=2 •The key and its children are compared •It is exchanged for the greater child Algorithms and Data Structures I 116 1 2 4 8 20 19 Heapsort 3 23 5 9 3 37 10 14 15 6 31 34 7 18 k=5 •The key and its children are compared •It is the greatest ⇒ ready Algorithms and Data Structures I 117 Sink(k,A) 1 if 2*k ≤ A.HeapSize and A[2*k] > A[k] 2 then greatest 2*k 3 else greatest k 4 if 2*k + 1 ≤ A.HeapSize and A[2*k + 1] > A[greatest] 5 then greatest 2*k + 1 6 if greatest k 7 then Exchange(A[greatest],A[k]) 8 Sink(greatest,A) Heapsort Algorithms and Data Structures I 118 To build a heap from an arbitrary array, all elements are mended by sinking them: this is the array’s last element that has any children BuildHeap(A) 1 A.HeapSize A.Length 2 for k A.Length / 2 downto 1 3 do Sink(k,A) we are stepping backwards; this way every visited element has only ancestors which fulfill the heap property Heapsort Algorithms and Data Structures I 119 Time complexity of building a heap To sink an element costs O(logn) in the worst case Since n/2 elements have to be sunk, an upper bound for the BuildHeap procedure is T(n) = O(n∙logn) It can be proven that the sharp bound is T(n) = θ (n) Heapsort Algorithms and Data Structures I 120 Time complexity of the priority queue operations if the queue is realized using heaps insert append the new element to the array O(1) exchange it for the root O(1) sink the root O(logn) The time complexity is T(n) = O(logn) Heapsort Algorithms and Data Structures I 121 Time complexity of the priority queue operations if the queue is realized using heaps maximum read out the key of the root O(1) The time complexity is T(n) = O(1) Heapsort Algorithms and Data Structures I 122 Time complexity of the priority queue operations if the queue is realized using heaps extract maximum exchange the root for the array’s last element O(1) extract the last element O(1) sink the root O(logn) The time complexity is T(n) = O(logn) Heapsort Algorithms and Data Structures I 123 The heapsort algorithm build a heap θ (n) iterate the following (n−1)∙O(logn) = O(n∙logn): exchange the root for the array’s last element O(1) exclude the heap’s last element from the heap O(1) sink the root O(logn) The time complexity is T(n) = O(n∙logn) Heapsort Algorithms and Data Structures I 124 HeapSort(A) 1 BuildHeap(A) 2 for k A.Length downto 2 3 do Exchange(A[1],A[A.HeapSize]) 4 A.HeapSize A.HeapSize – 1 5 Sink(1,A) Heapsort Algorithms and Data Structures I 125 Principle 22 69 8 75 25 12 14 36 Rearrange and part the elements so that every key in the first part is smaller than any in the second part. Quicksort Algorithms and Data Structures I 126 Principle 14 12 8 75 25 69 22 36 Rearrange and part the elements so that every key in the first part is smaller than any in the second part. Quicksort Algorithms and Data Structures I 127 Principle 148 12 148 75 22 25 69 36 22 69 36 75 Sort each part recursively, this will result in the whole array being sorted. Quicksort Algorithms and Data Structures I 128 The partition algorithm choose any of the keys stored in the array; this will be the so-called pivot key exchange the large elements at the beginning of the array to the small ones at the end of it not lesspivot thankey the pivot key 22 Quicksort 69 8 not greater than the pivot key 75 25 Algorithms and Data Structures I 12 14 36 129 Partition(A,first,last) 1 left first – 1 2 right last + 1 3 pivotKey A[RandomInteger(first,last)] 4 repeat 5 repeat left left + 1 6 until A[left] ≥ pivotKey 7 repeat right right – 1 8 until A[right] ≤ pivotKey 9 if left < right 10 then Exchange(A[left],A[right]) 11 else return right 12 until false Quicksort Algorithms and Data Structures I 130 The time complexity of the partition algorithm is T(n) = θ (n) because each element is visited exactly once. The sorting is then: QuickSort(A,first,last) 1 if first < last 2 then border Partition(A,first,last) 3 QuickSort(A,first,border) 4 QuickSort(A,border+1,last) Quicksort Algorithms and Data Structures I 131 Quicksort is a divide and conquer algorithm like merge sort, however, the partition is unbalanced (merge sort always halves the subarray). The time complexity of a divide and conquer algorithm highly depends on the balance of the partition. In the best case the quicksort algorithm halves the subarrays at every step ⇒ T(n) = θ (n∙logn) Quicksort Algorithms and Data Structures I 132 Recursion tree of the worst case n n n−1 1 n−2 1 1 n−1 1 n−2 0 n∙(n + 1) / 2 Quicksort Algorithms and Data Structures I 133 Thus, the worst case time complexity of quicksort is T(n) = θ (n2) The average case time complexity is T(n) = θ (n∙logn) the same as in the best case! The proof is difficult but let’s see a special case to understand quicksort better. Quicksort Algorithms and Data Structures I 134 Let λ be a positive number smaller than 1: 0<λ<1 Assumption: the partition algorithm never provides a worse partition ratio than (1− λ) : λ Example 1: Let λ := 0.99 The assumption demands that the partition algorithm does not leave less than 1% as the smaller part. Quicksort Algorithms and Data Structures I 135 Example 2: Let λ := 0.999 999 999 Due to the assumption, if we have at most one billion(!) elements then the assumption is fulfilled for any functioning of the partition algorithm. (Even if it always cuts off only one element from the others). In the following it is assumed for the sake of simplicity that λ ≥ 0.5, i.e. always the λ part is bigger. Quicksort Algorithms and Data Structures I 136 Recursion tree of the λ ratio case n n (1 − λ)n λn (1 − λ)λn n λ2n λdn ≤n ≤n ≤ n∙logn Quicksort Algorithms and Data Structures I 137 In the special case if none of the parts arising at the partitions are bigger than a given λ ratio (0.5 ≤ λ < 1), the time complexity of quicksort is T(n) = O(n∙logn) The time complexity of quicksort is practically optimal because the number of elements to be sorted is always bounded by a number N (finite storage). Using the value λ = 1 − 1/N it can be proven that quicksort finishes in O(n∙logn) time in every possible case. Quicksort Algorithms and Data Structures I 138 Problem Optimization problem: Let a function f(x) be given. Find an x where f is optimal (minimal or maximal) ‘under given circumstances’ ‘Given circumstances’: An optimization problem is constrained if functional constraints have to be fulfilled such as g(x) ≤ 0 Greedy algorithms Algorithms and Data Structures I 139 Feasible set: the set of those x values where the given constraints are fulfilled Constrained optimization problem: minimize f(x) subject to g(x) ≤ 0 Greedy algorithms Algorithms and Data Structures I 140 Example Problem: There is a city A and other cities B1,B2,...,Bn which can be reached from A by bus directly. Find the farthest of these cities where you can travel so that your money suffices. A B1 Greedy algorithms B2 ... Algorithms and Data Structures I Bn 141 Model: Let x denote any of the cities: x ∊ {B1,B2,...,Bn}, f(x) the distance between A and x, t(x) the price of the bus ticket from A to x, m the money you have, and g(x) = t(x) − m the constraint function. The constrained optimization problem to solve: minimize (− f(x)) s.t. g(x) ≤ 0 Greedy algorithms Algorithms and Data Structures I 142 In general, optimization problems are much more difficult! However, there is a class of optimization problems which can be solved using a step-bystep simple straightforward principle: greedy algorithms: at each step the same kind of decision is made, striving for a local optimum, and decisions of the past are never revisited. Greedy algorithms Algorithms and Data Structures I 143 Question:Which problems can be solved using greedy algorithms? Answer: Problems which obey the following two rules: Greedy choice property: If a greedy choice is made first, it can always be completed to achieve an optimal solution to the problem. Optimal substructure property: Any substructure of an optimal solution provides an optimal solution to the adequate subproblem. Greedy algorithms Algorithms and Data Structures I 144 Counter example Find the shortest route from Szeged to Budapest. The greedy choice property is infringed: You cannot simply choose the closest town first Greedy algorithms Algorithms and Data Structures I 145 Budapest Szeged Deszk Deszk is the closest to Szeged but situated in the opposite direction Greedy algorithms Algorithms and Data Structures I 146 Proper example Activity-selection problem: Let’s spend a day watching TV. Aim: Watch as many programs (on the wole) as you can. Greedy strategy: Watch the program ending first, then the next you can watch on the whole ending first, etc. Activity-selection problem Algorithms and Data Structures I 147 Let’smore Include Exclude No sortthe those programs thefirst programs which oneleft: have by ready already their ending begun times The optimum is 4 (TV programs) Activity-selection problem Algorithms and Data Structures I 148 Check the greedy choice property: The first choice of any optimal solution can be exchanged for the greedy one Activity-selection problem Algorithms and Data Structures I 149 Check the optimal substructure property: The part of an optimal solution is optimal also for the subproblem If this was not optimal for the subproblem, the whole solution could be improved by improving the subproblem’s solution Activity-selection problem Algorithms and Data Structures I 150 Notions C is an alphabet if it is a set of symbols F is a file over C if it is a text built up of the characters of C Huffman codes Algorithms and Data Structures I 151 Assume we have the following alphabet C = {a, b, c, d, e} Code it with binary codewords of equal length How many bits per codeword do we need at least? 2 are not enough (only four codewords: 00, 01, 10, 11) Build codewords using 3 bit coding a = 000 b = 001 c = 010 d = 011 e = 100 Huffman codes Algorithms and Data Structures I 152 a = 000 b = 001 c = 010 d = 011 e = 100 Build the T binary tree of the coding 0 0 1 0 1 a b Huffman codes 1 0 c 0 1 d 0 e Algorithms and Data Structures I 153 Further notation For each cC character its frequency in the file is denoted by f(c) For each cC character its length is defined by its depth in the T tree of coding, d T (c) Hence the length of the file (in bits) equals B(T)= cC f(c)d T (c) Huffman codes Algorithms and Data Structures I 154 Problem Let a C alphabet and a file over it given. Find a T coding of the alphabet with minimal B(T) Huffman codes Algorithms and Data Structures I 155 Example Consider an F file of 20,000 characters over the alphabet C = {a, b, c, d, e} Assume the frequencies of the particular characters in the file are f(a) = 5,000 f(b) = 2,000 f(c) = 6,000 f(d) = 3,000 f(e) = 4,000 Huffman codes Algorithms and Data Structures I 156 Using the 3 bit coding defined previously, the bitlength of the file equals B(T)= cC f(c)d T (c)= 5,0003+2,0003+6,0003+3,0003+4,0003= (5,000+2,000+6,000+3,000+4,000)3= 20,0003=60,000 This is a so-called fixed-length code since for all x,yC d T (x)=d T (y) holds Huffman codes Algorithms and Data Structures I 157 The fixed-length code is not always optimal 0 0 1 0 1 a b Huffman codes 1 0 c 0 1 d 0 e Algorithms and Data Structures I B(T’)= B(T)−f(e)1= 60,000−4,0001 = 56,000 158 Idea Construct a variable-length code, i.e., where the code-lengths for different characters can differ from each other We expect that if more frequent characters get shorter codewords then the resulting file will become shorter Huffman codes Algorithms and Data Structures I 159 Problem: How do we recognize when a codeword ends and a new begins. Using delimiters is too “expensive” Solution: Use prefix codes, i.e., codewords none of which is also a prefix of some other codeword Result: The codewords can be decoded without using delimiters Huffman codes Algorithms and Data Structures I 160 For instance if a = 10 b = 010 c = 00 then the following codes’ meaning is 1000010000010010 = a c b c c a b However, what if a variable-length code was not prefix-free: Huffman codes Algorithms and Data Structures I 161 Then if a = 10 b = 100 c=0 then 100= b or 1 0 0 = a c ? An extra delimiter would be needed Huffman codes Algorithms and Data Structures I 162 Realize the original idea with prefix codes f(a) = 5,000 f(b) = 2,000 f(c) = 6,000 f(d) = 3,000 f(e) = 4,000 rare frequent Frequent codewords should be shorter, e.g., a = 00, c = 01, e = 10 Rare codewords can be longer, e.g., b = 110, d = 111 Huffman codes Algorithms and Data Structures I 163 Question: How can such a coding be done algorithmically? Answer: The Huffman codes provide exactly this solution Huffman codes Algorithms and Data Structures I 164 The bitlength of the file using this K prefix code is B(K)= cC f(c)d K (c)= 5,0002+2,0003+6,0002+3,0003+4,0002= (5,000+6,000+4,000)2+(2,000+3,000 )3= 30,000+15,000=45,000 (cf. the fix-length codes gave 60,000, the improved one 56,000) Huffman codes Algorithms and Data Structures I 165 The greedy method producing Huffman codes 1. Sort the characters of the C alphabet in increasing order according to their frequency in the file and link them to an empty list 2. Delete the two leading characters, some x and y from the list and connect them with a common parent z node. Let f(z)=f(x)+f(y), insert z into the list and repeat step 2 until the the list runs empty. Huffman codes Algorithms and Data Structures I 166 Example character List: Huffman codes a:5 frequency (thousands) b:2 c:6 d:3 Algorithms and Data Structures I e:4 167 Example 1. Sort List: Huffman codes a:5 b:2 c:6 d:3 Algorithms and Data Structures I e:4 168 Example 2. Merge and rearrange 5 List: Huffman codes b:2 d:3 e:4 a:5 Algorithms and Data Structures I c:6 169 Example 2. Merge and rearrange 9 List: 5 e:4 b:2 Huffman codes a:5 c:6 d:3 Algorithms and Data Structures I 170 Example 2. Merge and rearrange 11 List: a:5 c:6 9 5 e:4 b:2 Huffman codes Algorithms and Data Structures I d:3 171 Example 2. Merge and rearrange 20 0 List: Huffman codes 1 9 11 0 1 0 1 e:4 5 a:5 c:6 0 1 b:2 d:3 Algorithms and Data Structures I 172 Example Ready 20 0 a = 10 11 b = 010 c1 = 11 0 d = 011 a:5 c e: 6= 00 9 Huffman codes 0 1 e:4 5 1 0 1 b:2 d:3 Algorithms and Data Structures I 173 Example Length of file in bits f(a) = 5,000 f(b) = 2,000 f(c) = 6,000 f(d) = 3,000 f(e) = 4,000 a = 10 B(H)= cC f(c)d H (c)= b = 010 c = 11 5,0002+2,0003+6,0002+3,0003+4,0002= d = 011 (5,000+6,000+4,000)2+(2,000+3,000 e = 00 )3= 30,000+15,000=45,000 Huffman codes Algorithms and Data Structures I 174 Optimality of the Huffman codes Assertion 1. There exists an optimal solution where the two rarest characters are deepest twins in the tree of the coding Assertion 2. Merging two (twin) characters leads to a problem similar to the original one Corollary. The Huffman codes provide an optimal character coding Huffman codes Algorithms and Data Structures I 175 Proof of Assertion 1 (There exists an optimal solution where the two rarest characters are deepest twins in the tree of the coding). Two rarest characters Huffman codes Changing nodes this way the total lenght does not increase Algorithms and Data Structures I 176 Proof of Assertion 2 (Merging two (twin) characters leads to a problem similar to the original one). Twin characters Huffman codes The new problem is smaller than the original one but similar to it Algorithms and Data Structures I 177 Graphs can represent different structures, connections and relations 1 Weighted graphs can represent capacities or actual flow rates 7 4 2 4 2 5 3 Graphs Algorithms and Data Structures I 178 1 7 1 2 3 4 1 0 1 2 0 71 2 1 2 0 0 1 4 3 0 0 0 1 5 4 1 7 1 4 51 0 4 2 4 2 Adjacency-matrix 5 3 Drawback 1: there is an 1: edge redundant leading elements from ‘row’ to ‘column’ 0: there is no Drawback 2: superfluous such edge elements Graphs Algorithms and Data Structures I 179 1 4 2 3 Graphs Adjacency-list 1 2 4 2 4 1 3 4 4 1 3 2 Optimal storage usage Drawback: slow search operations Algorithms and Data Structures I 180 Graphs Problem: find the shortest path between two vertices in a graph Source: the starting point (vertex) Single-source shortest path method: algorithm to find the shortest path to all vertices in a graph running out Algorithms and Data Structures I 181 Walk a graph: choose an initial vertex as the source visit all vertices starting from the source Graph walk methods: depth-first search breadth-first search Graph walk Algorithms and Data Structures I 182 Depth-first search Backtrack algorithm It goes as far as it can without revisiting any vertex, then backtracks source Graph walk Algorithms and Data Structures I 183 Breadth-first search Like an explosion in a mine The shockwave reaches the adjacent vertices first, and starts over from them Graph walk Algorithms and Data Structures I 184 The breadth-first search is not only simpler to implement but it is also the basis for several important graph algorithms (e.g. Dijkstra) Notation in the following pseudocode: A is the adjacency-matrix of the graph s is the source D is an array containing the distances from the source P is an array containing the predecessor along a path Q is the queue containing the unprocessed vertices already reached Graph walk Algorithms and Data Structures I 185 BreadthFirstSearch(A,s,D,P) 1 for i 1 to A.CountRows 2 do P[i] 0 3 D[i] ∞ 4 D[s] 0 5 Q.Enqueue(s) 6 repeat 7 v Q.Dequeue 8 for j 1 to A.CountColumns 9 do if A[v,j] > 0 and D[j] = ∞ 10 then D[j] D[v] + 1 11 P[j] v 12 Q.Enqueue(j) 13 until Q.IsEmpty Graph walk Algorithms and Data Structures I 186 The D,P pairs are displayed in the figure. 1,4 1,4 2,6 6 1 9 0,0 4 3,9 1,4 2 1,4 3,9 5 3 8 2,6 2,6 10 7 D is the shortest distance from the source The shortest paths can be reconstructed using P Graph walk Algorithms and Data Structures I 187 Problem: find the shortest path between two vertices in a weighted graph Idea: extend the breadth-first search for graphs having integer weights: unweighted edges (total weight = 3∙1 = 3) 3 virtual vertices Dijkstra’s algorithm Algorithms and Data Structures I 188 Dijkstra(A,s,D,P) 1 for i 1 to A.CountRows 2 do P[i] 0 minimum priority queue 3 D[i] ∞ 4 D[s] 0 5 for i 1 to A.CountRows 6 do M.Enqueue(i) 7 repeat 8 v M.ExtractMinimum 9 for j 1 to A.CountColumns 10 do if A[v,j] > 0 11 then if D[j] > D[v] + A[v,j] 12 then D[j] D[v] + A[v,j] 13 P[j] v 14 until M.IsEmpty Dijkstra’s algorithm Algorithms and Data Structures I 189 Time complexity of Dikstra’s algorithm Initialization of D and P: O(n) Building a heap for the priority queue: O(n) Search: n∙O(logn + n) = O(n(logn + n)) = O(n2) Grand total: T(n) = O(n2) extracting the minimum checking all neighbors number of loop executions Dijkstra’s algorithm Algorithms and Data Structures I 190