The Strassen’s method of matrix multiplication is a typical divide and conquer algorithm. We’ve seen so far some divide and conquer algorithms like merge sort and the Karatsuba’s fast multiplication of large numbers. However let’s get again on what’s behind the divide and conquer approach.

Unlike the dynamic programming where we “expand” the solutions of sub-problems in order to get the final solution, here we are talking more on joining sub-solutions together. These solutions of some sub-problems of the general problem are equal and their merge is somehow well defined.

A typical example is the merge sort algorithm. In merge sort we have two sorted arrays and all we want is to get the array representing their union again sorted. Of course, the tricky part in merge sort is the merging itself. That’s because we’ve to pass through the two arrays, A and B, and we’ve to compare each “pair” of items representing an item from A and from B. A bit off topic, but this is the weak point of merge sort and although its worst-case time complexity is O(n.log(n)), quicksort is often preferred in practice because there’s no “merge”. Quicksort just concatenates the two sub-arrays. Note that in quicksort the sub-arrays aren’t with an equal length in general and although its worst-case time complexity is O(n^2) it often outperforms merge sort.

This simple example from the paragraph above shows us how sometimes merging the solutions of two sub-problems actually isn’t a trivial task to do. Thus we must be careful when applying any divide and conquer approach.

History

Volker Strassen is a German mathematician born in 1936. He is well known for his works on probability, but in the computer science and algorithms he’s mostly recognized because of his algorithm for matrix multiplication that’s still one of the main methods that outperforms the general matrix multiplication algorithm.

Strassen firstly published this algorithm in 1969 and proved that the n^3 algorithm isn’t the optimal one. Actually the given solution by Strassen is slightly better, but his contribution is enormous because this resulted in many more researches about matrix multiplication that led to some faster approaches, i.e. the Coppersmith-Winograd algorithm with O(n^2,3737).

Overview

The general algorithm on multiplying two matrices A[NxN] and B[NxN] is fairly simple. Although it’s more difficult than multiplying two numbers and also it is not commutative it’s still very simple – but slow.

Let’s first define what’s a matrix A[NxN]. As we speak about matrices NxN we usually think of a square grid with N rows and N columns. In each row and column A[i][j] we’ve a value.

Of course, as developers, we can think of a matrix as a two-dimensional array.

// PHP two-dimensional array
$a = array(
    0 => array($v1, $v2, $v3, $v4),
    1 => array($v5, $v6, $v7, $v8),
    2 => array($v9, $v10, $v11, $v12),
); 

Don’t forget that a NxN matrix is just a private case for a matrix. We can equally likely have any other size of a matrix NxM (N <> M).

However the size of a matrix is crucial in order to multiply it with another matrix. Why is that?

As I said above multiplying matrices isn’t the same as multiplying numbers. First of all this operation isn’t commutative.

And the second problem is the way you multiply two matrices A with B.

Just because this works with NxN matrices we can see the problem with multiplying rectangular matrices. Indeed, this wouldn’t be possible unless the second dimension of A isn’t exactly equal to the first dimension of B.

Hopefully we are now talking about square matrices with exactly the same dimensions.

OK, so now we know how to multiply two square matrices (with the same dimensions NxN) and now let’s evaluate the time complexity for the general purpose algorithm.

As we know A.B = C only when:

C[i][j] = sum(A[i][k] * B[k][j]) for k = 0 .. n

Thus we have n^3 operations. Let’s try to find out a divide and conquer approach.

Indeed this isn’t difficult in case of matrices because as we know we can divide in matrix in smaller sub-matrices.

Now what do we have?

Again – the same complexity – we have 8 products and 4 sums. Where’s the catch?

Of course in order to get faster solution we’ve to be looking as Strassen did in 1969. He defined P1, P2, P3, P4, P5, P6 and P7 as defined on the image below.

Complexity

As I mentioned above the Strassen’s algorithm is slightly faster than the general matrix multiplication algorithm. The general algorithm’s time complexity is O(n^3), while the Strassen’s algorithm is O(n^2.80).

You can see on the chart below how slightly faster is this even for large n.

Application

Although this algorithm seems to be more close to pure mathematics than to computer practically everywhere we use NxN arrays we can benefit from matrix multiplication.

In the other hand the algorithm of Strassen is not much faster than the general n^3 matrix multiplication algorithm. That’s very important because for small n (usually n < 45) the general algorithm is practically a better choice. However as you can see from the chart above for n > 100 the difference can be very big.

In the same time typically NxN arrays are used always when we talk about adjacency matrix of graphs |V| = n and some graph algorithms practically depend on matrix multiplication.

Typically multiplying two n-digit numbers require n² multiplications. That is actually how we, humans, multiply numbers. Let’s take a look of an example in case we’ve to multiply two 2-digit numbers.

12 x 15 = ?

OK, we know that the answer is 180 and there are lots of intuitive methods that help us get the right answer. Indeed 12 x 15 it’s just a bit more difficult to calculate than 10 x 15, because multiplying by 10 it really easy – we just add one 0 at the end of the number. Thus 15 x 10 equals 150. But now again on 12 x 15 – we know that this equals 10 x 15 (which is 150) and 2 x 15, which is also very easy to calculate and it is 30. The result of 12×15 will be 150 + 30, which fortunately isn’t difficult to get and equals to 180.

That was easy but in some cases the calculations are a bit more difficult and we need a structured algorithm to get the right answer. What about 65 x 97? That is not so easy as 12 x 15, right?

The algorithm we know from the primary school, described on the diagram below, is well structured and help us multiply two numbers.

We see that even for two-digit numbers this is quite difficult – we have 4 multiplications and some additions.

However so far we know how to multiply numbers, the only problem is that our task becomes very difficult as the numbers grow. If multiplying 65 by 97 was somehow easy, what about

374773294776321
x
222384759707982

It seems almost impossible.

History

Andrey Kolmogorov is one of the brightest russian mathematicians of the 20th century. In 1960, during a seminar, Kolmogorov stated that two n-digit numbers can’t be multiplied with less than n² multiplications!
Only a week later a 23-year young student called Anatolii Alexeevitch Karatsuba proved that the multiplication of two n-digit numbers can be computed with n ^ lg(3) multiplications with an ingenious divide and conquer approach.

Overview

Basically Karatsuba stated that if we have to multiply two n-digit numbers x and y, this can be done with the following operations, assuming that B is the base of and m < n.

First both numbers x and y can be represented as x1,x2 and y1,y2 with the following formula.

x = x1 * B^m + x2 
y = y1 * B^m + y2 

Obviously now xy will become as the following product.

xy = (x1 * B^m + x2)(y1 * B^m + y2) =>

a = x1 * y1
b = x1 * y2 + x2 * y1
c = x2 * y2

Finally xy will become:

xy = a * B^2m + b * B^m + c

However a, b and c can be computed at least with four multiplication, which isn’t a big optimization. That is why Karatsuba came up with the brilliant idea to calculate b with the following formula:

b = (x1 + x2)(y1 + y2) - a - c

That make use of only three multiplications to get xy.

Let’s see this formula by example.

47 x 78

x = 47
x = 4 * 10 + 7

x1 = 4
x2 = 7

y = 78
y = 7 * 10 + 8

y1 = 7
y2 = 8

a = x1 * y1 = 4 * 7 = 28
c = x2 * y2 = 7 * 8 = 56
b = (x1 + x2)(y1 + y2) - a - c = 11 * 15 - 28 - 56

Now the thing is that 11 * 15 it’s again a multiplication between 2-digit numbers, but fortunately we can apply the same rules two them. This makes the algorithm of Karatsuba a perfect example of the “divide and conquer” algorithm.

Implementation

Standard Multiplication

Typically the standard implementation of multiplication of n-digit numbers require n² multiplications as you can see from the following PHP implementation.

$x = array(1,2,3,4);
$y = array(5,6,7,8);

function multiply($x, $y)
{	
	$len_x = count($x);
	$len_y = count($y);
	$half_x = ceil($len_x / 2);
	$half_y = ceil($len_y / 2);
	$base = 10;

	// bottom of the recursion
	if ($len_x == 1 && $len_y == 1) {
		return $x[0] * $y[0];
	}
	
	$x_chunks = array_chunk($x, $half_x);
	$y_chunks = array_chunk($y, $half_y);
	
	// predefine aliases
	$x1 = $x_chunks[0];
	$x2 = $x_chunks[1];
	$y1 = $y_chunks[0];
	$y2 = $y_chunks[1];
	
	return  multiply($x1, $y1) * pow($base, $half_x * 2) 					// a
		 	+ (multiply($x1, $y2) + multiply($x2, $y1)) * pow($base, $half_x) 	// b
		 	+ multiply($x2, $y2);							// c
}

// 7 006 652
echo multiply($x, $y);

Karatsuba Multiplication

Karatsuba replaces two of the multiplications – this of x1 * y2 + x2 * y1 with only one – (x1 + x2)(y1 + y2) and this makes the algorithm faster.

$x = array(1,2,3,4);
$y = array(5,6,7,8);

function karatsuba($x, $y) 
{
	$len_x = count($x);
	$len_y = count($y);
	
	// bottom of the recursion
	if ($len_x == 1 && $len_y == 1) {
		return $x[0] * $y[0];
	} 
	if ($len_x == 1 || $len_y == 1) {
		$t1 = implode('', $x);
		$t2 = implode('', $y);
		return (int)$t1 * $t2;
	}
	
	$a = array_chunk($x, ceil($len_x/2));
	$b = array_chunk($y, ceil($len_y/2));
	
	$deg = floor($len_x/2);
	
	$x1 = $a[0];	// 1
	$x2 = $a[1];	// 2
	$y1 = $b[0];	// 1
	$y2 = $b[1];	// 2

	return  ($a = karatsuba($x1, $y1)) * pow(10, 2 * $deg)
			+ ($c = karatsuba($x2, $y2))
			+ (karatsuba(sum($x1, $x2), sum($y1, $y2)) - $a - $c) * pow(10, $deg);
}

// 7 006 652
echo karatsuba($x, $y);

Complexity

Assuming that we replace two of the multiplications with only one makes the program faster. The question is how fast. Karatsuba improves the multiplication process by replacing the initial complexity of O(n²) by O(n^lg3), which as you can see on the diagram below is much faster for big n.

Application

It’s obvious where the Karatsuba algorithm can be used. It is very efficient when it comes to integer multiplication, but that isn’t its only advantage. It is often used for polynomial multiplications.