### F(n)

```CS420 lecture four
Recurrence Relations
wim bohm, cs.colostate
Complexity
• Divide a program in basic blocks and count the
order of magnitude number of basic block
executions
– basic block: code fragment that takes constant
time to execute
– careful: some language constructs may take more
than constant time
• contains in Java
• + in APL / C++
• pattern matching in SNOBOL / Python
Recurrence Relations
• The complexity of an algorithm can often be
expressed in a recurrence relation with the
input size as parameter. Eg (merge sort)
T(n) = 2T(n/2)+n
T(1) = 1
Repeated substitution
• Simple recurrence relations (one recurrent
term in the rhs) can sometimes be solved
using repeated substitution
• Two types: Linear and DivCo
– Linear F(n) = aF(n-d)+g(n), base: F(1)=v1
– Divco F(n)= aF(n/d)+g(n), base: F(1)=v1
• Two question:
– what is the pattern
– how often is it applied until we hit the base case
Linear Example
Hanoi: M(n)=2M(n-1)+1, M(1)=1
M(n) = 2M(n-1)+1 = 2(2M(n-2)+1)+1 = 4M(n-2)+2+1 =
4(2M(n-3)+1)+2+1= 8M(n-3)+4+2+1= ...
2kM(n-k)+2k-1+2k-2+...+2+1=
hit base for k = n-1:
= 2n-1M(1)+2n-1+2n-2+...+2+1 = 2n-1
DivCo example
Merge sort:
T(n) = 2T(n/2) + n, T(1)=1
n = 2k
T(n)=2(2(T(n/4)+n/2)+n =
4T(n/4) + 2n = 8T(n/8) + 3n ... = 2kT(n/2k)+kn
hit base for k = logn
= 2kT(n/2k)+kn = n+kn = O(nlogn)
Another one: binary search
Let n = 2k and f(1)=1 and f(n) = f(n/2)+c
f(n)=f(n/2)+c = f(n/4)+2c =
f(n/8)+3c = f(n/2k)+kc =
hit base for k=lg n:
f(1)+lgn.c = O(lg n)
DivCo repeated substitution pattern
n = bk
f(n)= af(n/b)+g(n) = a(af(n/b2)+g(n/b))+g(n) =
a2f(n/b2)+ag(n)+g(n) = a2(af(n/b2)+g(n)+ag(n)+g(n) =
a3f(n/b3)+a2g(n/b2)+ag(n/b)+g(n) = ...
k 1
f ( n )  a f ( n / b )   a g( n / b )
k
k
i
i
i 0
n  b , so
k
k 1
f ( n )  a f (1)   a g( n / b )
k
i
i 0
i
Linear repeated substitution pattern
n = kb+1
f(n)= af(n-b)+g(n) = a(f(n-2b)+g(n-b)+g(n)=
a2f(n-2b)+ag(n-b)+g(n) =a2(af(n-3b)+g(n))+ag(nb)+g(n)=
a3f(n-3b)+a2g(n-2b)+ag(n-b)+g(n)
= ...
k 1
f ( n )  a f ( n  kb )   a g( n  ib )
k
i
i 0
k 1
f ( n )  a f (1)   a g( n  ib )
k
i
i 0
Master Method
Cookbook approach to solution, recognizing the
form. Cormen et.al.'s method is more complex
than Rosen:
An = C An/d+knp
– An = O(np)
if C < dp eg An = 3 An/2+n2
– An = O(nplog(n)) if C = dp eg An = 2An/2+n
– An = O(nlogdc)
if C > dp eg An = 3 An/2+n
Proof based on repeated substitution plus induction
Let's prove a simpler version
Let f(n)=af(n/b)+c, n=bk, then
proof:
f (n)  O(n
log b a
f ( n )  O (log n )
) if
a 1
if
a 1
k 1
f ( n )  a f (1) 
k
a
j
c
from
repeated
subst
i 0
case
k 1

f ( n )  f (1)  c  1
a 1:
i 0
k  lg
b
f ( n )  f (1)  c log
n so
b
n
k 1
case
a  1 : geometric
progression
a
a 1
k
i

a 1
i 0
a 1
k
f ( n )  a f (1)  c
k
because
a  a
k
a 1
log b n
 n
 a ( f (1) 
k
log b a
c
a 1
: f ( n )  C1n
)
c
a 1
log b a
 C2
 C1a  C 2
k
App: multiplying two n digit numbers
A x B = an-1an-2..a1a0 X bn-1bn-2..b1b0
• high school method: O(n2)
• divide and conquer approach
A = A110n/2+A2
B = B110n/2+B2
AxB = A1B110n+(A1B2+A2B1)10n/2+ A2B2
A1B2+A2B1 = (A1+A2)(B1+B2)-A1B1-A2B2
so we only need 3 n/2 digit multiplications:
A1B2, A2B1 and (A1+A2)(B1+B2)
Complexity of DivCo n digit multiply
Tn=3Tn/2+O(n)
Look up in master table:
Complexity of DivCo n digit multiply
Tn=3Tn/2+O(n)
Look up in master table:
Tn=O(nlg3)~O(n1.6)
Similar trick for matrix multiply
n
C i, j   A i,k * B k , j
C = AxB,
k 1
3
simple O(n ) code
for i=1 to n
for j =1 to n
C[i,j]=0
for k = 1 to n C[i,j]+=A[i,k]*B[k,j]
Block matrix multiply
A1,1 A1,2
B1,1 B1,2
*
A2,1 A2,2
=
B2,1 B2,2
A1,1 B1,1 + A1,2 B2,1
A1,1 B1,2 + A1,2 B2,2
A2,1 B1,1 + A2,2 B2,1
A2,1 B1,2 + A2,2 B2,2
naive block matmult
T(n) = 8T(n/2) + O(n2)
master table:
naive block matrix multiply
T(n) = 8T(n/2) + O(n2)
master table:
T(n) = O(nlog8)=O(n3)
Strassen
Strassen shows that we only need 7 block multiplies
(plus O(n2) work: matrix adds/subtracts )
see notes
T(n) = 7T(n/2) + O(n2)
master table: T(n) = O(nlg7)~O(n2.81)
Linear homogeneous recurrence
relations (source: Rosen chapter 7)
• an = c1.an-1 + c2.an-2 + .. + cr.an-r
a0 = v0, .. , ar-1 = vr-1 : initial values
• Guess individual solutions of form n
– give rise to characteristic equation
– solve to obtain characteristic roots
• General solution
– linear combination of roots
• Specific solution
– find coefficients of general solution using initial values
Example: Fibonacci
Fn=Fn-1+Fn-2, F0=F1=1
1, 1, 2, 3, 5, 8, 13, 21, 34, 55, ...
Exponential growth, guess n and substitute
n =n-1 +n-2
2 = +1 characteristic eq: 2 -  - 1 = 0
solution:
x1,2=(-b±sqrt(b2-4ac))/2a
General Solution
• Roots of 2 -  - 1 = 0
1,2= (1±sqrt(1+4))/2 = (1±sqrt(5))/2
general solution: linear combination of the roots
F n  A1 (
1
5
2
)  A2 (
n
1
5
2
)
n
A1 and A2 for specific Solution
• Specific solution uses the initial values F0=F1=1
F0  1 :
F1  1 :
A1  A 2  1
A1 (
1
5
2
)  A2 (
1
5
2
) 1
• two equations two unknowns, solve to find As
A1 
1
A2  
(
1
2
5
1
5
5
(
1
)
5
2
)
Specific Solution
Fn 
1
(
1
5 2

1
2
5)
n 1

1
(
1
5 2

1
2
5)
n 1
Does it always work?
Characteristic equation  roots r1 and r2.
The general solution: A1r1n+A2r2n
Initial values: f0=C0 and f1=C1  specific solution:
f0= C0 = A1+A2
 A2=C0-A1
f1= C1 = A1r1+A2r2  C1=A1r1+(C0-A1)r2= A1(r1-r2)+C0r2
A1

A1 
C 1  C 0 r2
r1  r2
A 2  C 0  A1  C 0 
C 1  C 0 r2
r1  r2

C 0 r1  C 1
r1  r2
only works if
r1 ≠ r 2
Is it a problem?
Not really.
Equal roots r come from characteristic equation:
(x-r)(x-r)=x2-2rx+r2=0
come from recurrence relation: fn=2rfn-1 - r2fn-2
If our recurrence relation has only positive coefficients we
cannot get equal roots.
For Analysis of Algorithms we can restrict the recurrences to
having only positive coefficients. We ADD the complexity of
sub solutions together
First order Linear Homogeneous
Recurrence Relations
Simple form Fn=cFn-1 F0=a
Use repeated substitution to solve
First order Linear Homogeneous
Recurrence Relations
Simple form Fn=cFn-1 F0=a
Use repeated substitution to solve
Fn=cFn-1 =c2Fn-2=c3Fn-3= ... = cnF0=cna
20% yearly interest on \$1000
What is the value in year n
Vn=1.2Vn-1 V0=1000
20% yearly interest on \$1000
What is the value in year n
Vn=1.2Vn-1 V0=1000
Vn=(1.2)n.1000
Exercise
• Determine the general and specific solution
of
Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Exercise 2
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Characteristic equation?
Exercise 3
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Characteristic equation:
x2-x-2=0
Exercise 4
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Characteristic equation:
x2-x-2=0
Roots?
x1,2=(1±sqrt(1-(-8)))/2
Exercise 5
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Characteristic equation:
x2-x-2=0
Roots:
r1=2 r2=-1
Exercise 6
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
General Solution?
Exercise 7
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
General Solution:
Fn=A12n+A2(-1)n
Exercise 8
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Specific Solution?
Find A1 and A2 using initial values
Exercise 9
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Specific Solution:
F0=2=A1+A2
F1=7=A1.2+A2.(-1)
A1=3 A2=-1
Fn=3.2n-(-1)n
Exercise 10
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Specific Solution:
F0=2=A1+A2
F1=7=A1.2+A2.(-1)
Fn=3.2n-(-1)n
A1=3 A2=-1
DONE???
Exercise 11
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Specific Solution:
F0=2=A1+A2
F1=7=A1.2+A2.(-1)
Fn=3.2n-(-1)n
A1=3 A2=-1
Exercise 11
• Fn= Fn-1+2Fn-2 with F0=2 and F1=7
Specific Solution:
F0=2=A1+A2
F1=7=A1.2+A2.(-1)
Fn=3.2n-(-1)n
A1=3 A2=-1
Check F0 F1 F2 F3
Linear INhomogeneous recurrences
Find the homogeneous solution
Suppose pn is a particular solution
Then the general solution is
constant*homogeneous solution + particular
solution
Linear Inhomogeneous
Simple case an = c1.an-1 + f(n) happens most often
–
Very simple, special case: c1 = 1
•
–
an = an-1 + f(n) = ... = a0 + f(1) +..+ f(n)
General case:
•
•
Solve homogeneous part: hn = A.c1n
if *n is a particular solution then
an = A.c1n + *n is the general solution
• Particular solution “mimics” f(n)
f(n): c, d.n , d.n2
, dn
*n : b, a.n + b, a.n2+b.n+c, b.dn or b.n.dn
(dc1) or (d=c1)
Some particular solutions for
An = C An-1 + f(n)
F(n)
Particular(n)
a (constant)
b (constant)
an
bn+c
a n2
b n2 + c n + d
an
b an
Let’s try it on Mn = 2 Mn-1 +1, M1 = 1
Example: Hanoi
Mn=2Mn-1+1 M1=1
Homogeneous solution: A.2n
Particular solution:
B
B obeys the recurrence relation so
B=2B+1 -> B=-1
General solution: Mn=A.2n-1
M1=1 so A.2-1=1 -> A=1
Specific solution: Mn=2n-1
Another example
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 ...
1. homogeneous solution?
Another example 1
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
1. homogeneous solution:
An=2An-1 An=2n
Another example 2
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
1. homogeneous solution:
An=2An-1 An=2n
2. particular solution?
Another example 3
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
1. homogeneous solution:
An=2An-1 An=2n
2. particular solution .... has form
a.n+b
and fits the recurrence
Another example 4
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
2. particular solution has form a.n+b and fits the recurrence
a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n
Another example 5
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
2. particular solution has form a.n+b and fits the recurrence
a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n
0 = a.n-2a+b+n
Another example 6
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
2. particular solution has form a.n+b and fits the recurrence
a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n
0 = a.n-2a+b+n
(a+1).n=2a-b
Another example 7
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
2. particular solution has form a.n+b and fits the recurrence
a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n
0 = a.n-2a+b+n
(a+1).n=2a-b
n=0:
0=2a-b -> 2a=b
n=1: a+1=0
-> a=-1 -> b=-2
particular solution: -n-2
Another example 8
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
2. particular solution has form a.n+b and fits the recurrence
a.n+b=2(a.(n-1)+b)+n = 2a.n-2a+2b+n
0 = a.n-2a+b+n
(a+1).n=2a-b
n=0:
0=2a-b -> 2a=b
n=1: a+1=0
-> a=-1 -> b=-2
particular solution: -n-2
CHECK IT!
Another example 9
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
2. particular solution has form a.n+b and fits the recurrence
particular solution: -n-2
CHECKING whether –n-2 fits:
An=2An-1+n
-n-2 = 2(-(n-1)-2) +n = -2n+2-4+n = -n-2 CHECK
Another example 10
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
2. particular solution has form a.n+b and fits the recurrence
particular solution: -n-2
CHECKING whether –n-2 fits:
An=2An-1+n
-n-2 = 2(-(n-1)-2) +n = -2n+2-4+n = -n-2 CHECK
3. General Solution?
Another example 11
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26
2. particular solution has form a.n+b and fits the recurrence
particular solution: -n-2
CHECKING whether –n-2 fits:
An=2An-1+n
-n-2 = 2(-(n-1)-2) +n = -2n+2-4+n = -n-2 CHECK
3. General Solution:
constant*homogeneous + particular
Another example 12
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26 ... looks exponential
2. particular solution: -n-2
3. general Solution: constant*homogeneous+particular
An=C.2n+(-n-2)
4.
specific solution?
Another example 13
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26 ...
looks exponential
2. particular solution: -n-2
3. general Solution: An=C.2n+(-n-2)
4. specific solution: A0=0 -> C-2=0
An=2n+1-n-2
CHECK IT!!
-> C=2
Another example 14
An=2An-1+n
A0=0
A1=1 A2=4 A3=11 A4=26 ...
2. particular solution: -n-2
3. general Solution: An=C.2n+(-n-2)
looks exponential
4. specific solution: A0=0 -> C-2=0 -> C=2
An=2n+1-n-2
CHECKING:
A0=2-0-2=0 A1=4-1-2=1 A2=8-2-2=4 A3=16-3-2=11 check
Assignment1: Ackermann
Ackermann invented a recurrence relation that:
• has only increments and decrements,
• grows insanely fast, and
• makes even more recursive calls when
implemented as a recursive function
Ackermann
int ack (int m, int n) {
c = c+1;
if
(m==0) return(n+1);
else if (n==0) return(ack(m-1,1));
else
return(ack(m-1,ack(m,n-1)));
}
c counts the number of calls
Ackermann
Play with it, run it for m=1,2,3,4
(careful with m=4)
Analyse it:
1. What are
ack(1,n), ack(2,n), ack(3,n)?
2. How many calls for
ack(1,n), ack(2,n), ack(3,n)?
Solving Ackermann
• Don't infer solution by fitting the answer
• Create recurrences and solve them, eg
A(1,n)=A(0,A(1,n-1))=A(1,n-1)+1
A(1,0)=A(0,1)=2
Solve it.
Then go on to A(2,n), then A(3,n)
Solving Ackermann
• part deux: derive the number of calls for
m=1,2,3
Create recurrences for the number of
function calls. They are all inhomogeneous
recurrences of the kind we discussed.
Solve them. Case m=3 is complex.
You can leave the answer in terms of sums
another exercise
fn=2fn-1+3fn-2 f0=f1=1
characteristic eq?
another exercise 1
fn=2fn-1+3fn-2 f0=f1=1
characteristic eq:
x2-2x-3=0
solve it
another exercise 2
fn=2fn-1+3fn-2 f0=f1=1
characteristic eq:
x2-2x-3=0
x 1,2 
2
x 1  1
general solution?

4  12
2
x2  3
another exercise 3
fn=2fn-1+3fn-2 f0=f1=1
x1=-1 x2=3
general solution:
A13n+A2(-1)n
specific solution?
another exercise 4
fn=2fn-1+3fn-2 f0=f1=1
general solution: fn = A13n+A2(-1)n
specific solution:
A1+A2=1
3A1-A2=1
A1=1/2 A2=1/2
another exercise 5
fn=2fn-1+3fn-2 f0=f1=1
general solution: fn = A13n+A2(-1)n
specific solution: fn = (1/2)3n+(1/2)(-1)n
A1+A2=1
3A1-A2=1
A1=1/2 A2=1/2
CHECK IT
another exercise 6
fn=2fn-1+3fn-2 f0=f1=1
general solution: fn = A13n+A2(-1)n
specific solution: fn = (1/2)3n+(1/2)(-1)n
1/2+1/2=1 3/2-1/2=1
9/2+1/2=5 27/2-1/2=13 check
inhomogeneous
fn=2fn-1+3fn-2+5 f0=f1=0
homogeneous solution: hn = A13n+A2(-1)n
particular solution?
inhomogeneous 1
fn=2fn-1+3fn-2+5 f0=f1=0
homogeneous solution: hn = A13n+A2(-1)n
particular solution: B
B=2B+3B+5
4B=-5
B=-5/4
CHECK IT
inhomogeneous 2
fn=2fn-1+3fn-2+5 f0=f1=0
homogeneous solution: hn = A13n+A2(-1)n
particular solution: B
B=2B+3B+5
4B=-5
B=-5/4
-5/4=-10/4-15/4+20/4
check
inhomogeneous 3
fn=2fn-1+3fn-2+5 f0=f1=0
homogeneous solution: hn = A13n+A2(-1)n
particular solution: -5/4
general solution: fn= A13n+A2(-1)n-5/4
specific solution?
inhomogeneous 4
fn=2fn-1+3fn-2+5 f0=f1=0
general solution: fn= A13n+A2(-1)n-5/4
specific solution:
f0=0= A1+A2-5/4
f1=0= 3A1-A2-5/4
A1=A2=5/8
inhomogeneous 5
fn=2fn-1+3fn-2+5 f0=f1=0
general solution: fn= A13n+A2(-1)n-5/4
specific solution: fn= (5/8)3n+(5/8)(-1)n-5/4
check it
inhomogeneous 6
fn=2fn-1+3fn-2+5 f0=f1=0
general solution: fn= A13n+A2(-1)n-5/4
specific solution: fn= (5/8)3n+(5/8)(-1)n-5/4
f0=0= 5/8+5/8-5/4
f1=0= 3(5/8)-5/8-5/4
f2=5= 9(5/8)+5/8-5/4=50/8 – 10/8 = 40/8
f3=15= 27(5/8)-5/8-5/4=120/8 check
one more
fn=3fn-1+n f0=1
homogeneous hn= A13n
particular solution?
one more 1
fn=3fn-1+n f0=1
homogeneous hn= A13n
particular solution: shape an+b
an+b = 3(a(n-1)+b)+n = 3an-3a+3b+n = (3a+1)n-3a+3b
0 = (2a+1)n-3a+2b
n=0 3a=2b a=(2/3)b
n=1 2a+1 – 3a+2b=0  a=2b+1  (2/3)b=2b+1  b=-3/4
a=(2/3)b= -1/2
one more 2
fn=3fn-1+n f0=1
homogeneous hn= A13n
particular solution: Pn=-(1/2)n-3/4
check it!
one more 3
fn=3fn-1+n f0=1
homogeneous hn= A13n
particular solution: Pn=-(1/2)n-3/4
-(1/2)n-3/4 = 3((-1/2)(n-1)-3/4)+n = -(3/2)n+3/2+9/4+n
= -1/2n-3/4 check
general solution?
one more 4
fn=3fn-1+n f0=1
homogeneous hn= A13n
particular solution: Pn=-(1/2)n-3/4
general solution: A3n-(1/2)n-3/4
specific solution?
one more 5
fn=3fn-1+n f0=1
homogeneous hn= A13n
particular solution: Pn=-(1/2)n-3/4
general solution: A3n-(1/2)n-3/4
specific solution: A-3/4 = 1  A=7/4
fn=(7/4)3n-(1/2)n-3/4
one more 6
fn=3fn-1+n f0=1
specific solution: A-3/4 = 1  A=7/4
fn=(7/4)3n-(1/2)n-3/4
one more 7
fn=3fn-1+n f0=1
specific solution: A-3/4 = 1  A=7/4
fn=(7/4)3n-(1/2)n-3/4
7/4-3/4 = 1 21/4-1/2-3/4=16/4=4
(7/4)9-1-3/4=63/4-7/4=56/4=14 check
```