Sorting Techniques with Algorithm analysis

Heap Sort

Algorithm Analysis

The heap sort is the slowest of the O(n log n) sorting algorithms, but unlike the merge and quick sorts it doesn't require massive recursion or multiple arrays to work. This makes it the most attractive option for very large data sets of millions of items.

The heap sort works as it name suggests - it begins by building a heap out of the data set, and then removing the largest item and placing it at the end of the sorted array. After removing the largest item, it reconstructs the heap and removes the largest remaining item and places it in the next open position from the end of the sorted array. This is repeated until there are no items left in the heap and the sorted array is full. Elementary implementations require two arrays - one to hold the heap and the other to hold the sorted elements.

To do an in-place sort and save the space the second array would require, the algorithm below "cheats" by using the same array to store both the heap and the sorted array. Whenever an item is removed from the heap, it frees up a space at the end of the array that the removed item can be placed in.

Pros: In-place and non-recursive, making it a good choice for extremely large data sets.
Cons: Slower than the merge and quick sorts.

As mentioned above, the heap sort is slower than the merge and quick sorts but doesn't use multiple arrays or massive recursion like they do. This makes it a good choice for really large sets, but most modern computers have enough memory and processing power to handle the faster sorts unless over a million items are being sorted.

The "million item rule" is just a rule of thumb for common applications - high-end servers and workstations can probably safely handle sorting tens of millions of items with the quick or merge sorts. But if you're working on a common user-level application, there's always going to be some yahoo who tries to run it on junk machine older than the programmer who wrote it, so better safe than sorry.

Source Code
Below is the basic heap sort algorithm. The siftDown() function builds and reconstructs the heap.

void heapSort(int numbers[], int array_size) { int i, temp; for (i = (array_size / 2)-1; i >= 0; i--) siftDown(numbers, i, array_size); for (i = array_size-1; i >= 1; i--) { temp = numbers[0]; numbers[0] = numbers[i]; numbers[i] = temp; siftDown(numbers, 0, i-1); } } void siftDown(int numbers[], int root, int bottom) { int done, maxChild, temp; done = 0; while ((root*2 <= bottom) && (!done)) { if (root*2 == bottom) maxChild = root * 2; else if (numbers[root * 2] > numbers[root * 2 + 1]) maxChild = root * 2; else maxChild = root * 2 + 1; if (numbers[root] <> { temp = numbers[root]; numbers[root] = numbers[maxChild]; numbers[maxChild] = temp; root = maxChild; } else done = 1; } }

Quick Sort

Algorithm Analysis

The quick sort is an in-place, divide-and-conquer, massively recursive sort. As a normal person would say, it's essentially a faster in-place version of the merge sort. The quick sort algorithm is simple in theory, but very difficult to put into code (computer scientists tied themselves into knots for years trying to write a practical implementation of the algorithm, and it still has that effect on university students).

The recursive algorithm consists of four steps (which closely resemble the merge sort):

1. If there are one or less elements in the array to be sorted, return immediately.
2. Pick an element in the array to serve as a "pivot" point. (Usually the left-most element in the array is used.)
3. Split the array into two parts - one with elements larger than the pivot and the other with elements smaller than the pivot.
4. Recursively repeat the algorithm for both halves of the original array.

The efficiency of the algorithm is majorly impacted by which element is choosen as the pivot point. The worst-case efficiency of the quick sort, O(n²), occurs when the list is sorted and the left-most element is chosen. Randomly choosing a pivot point rather than using the left-most element is recommended if the data to be sorted isn't random. As long as the pivot point is chosen randomly, the quick sort has an algorithmic complexity of O(n log n).

Pros: Extremely fast.
Cons: Very complex algorithm, massively recursive.

The quick sort is by far the fastest of the common sorting algorithms. It's possible to write a special-purpose sorting algorithm that can beat the quick sort for some data sets, but for general-case sorting there isn't anything faster.

As soon as students figure this out, their immediate implulse is to use the quick sort for everything - after all, faster is better, right? It's important to resist this urge - the quick sort isn't always the best choice. As mentioned earlier, it's massively recursive (which means that for very large sorts, you can run the system out of stack space pretty easily). It's also a complex algorithm - a little too complex to make it practical for a one-time sort of 25 items, for example.

With that said, in most cases the quick sort is the best choice if speed is important (and it almost always is). Use it for repetitive sorting, sorting of medium to large lists, and as a default choice when you're not really sure which sorting algorithm to use. Ironically, the quick sort has horrible efficiency when operating on lists that are mostly sorted in either forward or reverse order - avoid it in those situations.

Source Code
Below is the basic quick sort algorithm.

void quickSort(int numbers[], int array_size) { q_sort(numbers, 0, array_size - 1); } void q_sort(int numbers[], int left, int right) { int pivot, l_hold, r_hold; l_hold = left; r_hold = right; pivot = numbers[left]; while (left <> { while ((numbers[right] >= pivot) && (left <> right--; if (left != right) { numbers[left] = numbers[right]; left++; } while ((numbers[left] <= pivot) && (left <> left++; if (left != right) { numbers[right] = numbers[left]; right--; } } numbers[left] = pivot; pivot = left; left = l_hold; right = r_hold; if (left <> q_sort(numbers, left, pivot-1); if (right > pivot) q_sort(numbers, pivot+1, right); }

Merge Sort

Algorithm Analysis

The merge sort splits the list to be sorted into two equal halves, and places them in separate arrays. Each array is recursively sorted, and then merged back together to form the final sorted list. Like most recursive sorts, the merge sort has an algorithmic complexity of O(n log n).

Elementary implementations of the merge sort make use of three arrays - one for each half of the data set and one to store the sorted list in. The below algorithm merges the arrays in-place, so only two arrays are required. There are non-recursive versions of the merge sort, but they don't yield any significant performance enhancement over the recursive algorithm on most machines.

Pros: Marginally faster than the heap sort for larger sets.
Cons: At least twice the memory requirements of the other sorts; recursive.

The merge sort is slightly faster than the heap sort for larger sets, but it requires twice the memory of the heap sort because of the second array. This additional memory requirement makes it unattractive for most purposes - the quick sort is a better choice most of the time and the heap sort is a better choice for very large sets.

Like the quick sort, the merge sort is recursive which can make it a bad choice for applications that run on machines with limited memory.

Source Code
Below is the basic merge sort algorithm.

void mergeSort(int numbers[], int temp[], int array_size) { m_sort(numbers, temp, 0, array_size - 1); } void m_sort(int numbers[], int temp[], int left, int right) { int mid; if (right > left) { mid = (right + left) / 2; m_sort(numbers, temp, left, mid); m_sort(numbers, temp, mid+1, right); merge(numbers, temp, left, mid+1, right); } } void merge(int numbers[], int temp[], int left, int mid, int right) { int i, left_end, num_elements, tmp_pos; left_end = mid - 1; tmp_pos = left; num_elements = right - left + 1; while ((left <= left_end) && (mid <= right)) { if (numbers[left] <= numbers[mid]) { temp[tmp_pos] = numbers[left]; tmp_pos = tmp_pos + 1; left = left +1; } else { temp[tmp_pos] = numbers[mid]; tmp_pos = tmp_pos + 1; mid = mid + 1; } } while (left <= left_end) { temp[tmp_pos] = numbers[left]; left = left + 1; tmp_pos = tmp_pos + 1; } while (mid <= right) { temp[tmp_pos] = numbers[mid]; mid = mid + 1; tmp_pos = tmp_pos + 1; } for (i=0; i <= num_elements; i++) { numbers[right] = temp[right]; right = right - 1; } }

Quicksort

Although the shell sort algorithm is significantly better than insertion sort, there is still room for

improvement. One of the most popular sorting algorithms is quicksort. Quicksort executes in

O(n lg n) on average, and O(n2) in the worst-case. However, with proper precautions, worst-case

behavior is very unlikely. Quicksort is a non-stable sort. It is not an in-place sort as stack space

is required. For further reading, consult Cormen [1990].

Theory

The quicksort algorithm works by partitioning the array to be sorted, then recursively sorting

each partition. In Partition (Figure 2-3), one of the array elements is selected as a pivot value.

Values smaller than the pivot value are placed to the left of the pivot, while larger values are

placed to the right.

- 12 -

Figure 2-3: Quicksort Algorithm

In Figure 2-4(a), the pivot selected is 3. Indices are run starting at both ends of the array.

One index starts on the left and selects an element that is larger than the pivot, while another

index starts on the right and selects an element that is smaller than the pivot. In this case,

numbers 4 and 1 are selected. These elements are then exchanged, as is shown in Figure 2-4(b).

This process repeats until all elements to the left of the pivot are £ the pivot, and all items to the

right of the pivot are ³ the pivot. QuickSort recursively sorts the two sub-arrays, resulting in the

array shown in Figure 2-4(c).

4 2 3 5 1

1 2 3 5 4

1 2 3 4 5

_D_

_E_

_F_

Lb Ub

SLYRW

Lb M Lb

Figure 2-4: Quicksort Example

As the process proceeds, it may be necessary to move the pivot so that correct ordering is

maintained. In this manner, QuickSort succeeds in sorting the array. If we’re lucky the pivot

selected will be the median of all values, equally dividing the array. For a moment, let’s assume

int function Partition (Array A, int Lb, int Ub);

begin

select a pivot from A[Lb]…A[Ub];

reorder A[Lb]…A[Ub] such that:

all values to the left of the pivot are £ pivot

all values to the right of the pivot are ³ pivot

return pivot position;

end;

procedure QuickSort (Array A, int Lb, int Ub);

begin

if Lb < Ub then

M = Partition (A, Lb, Ub);

QuickSort (A, Lb, M – 1);

QuickSort (A, M + 1, Ub);

end;

- 13 -

that this is the case. Since the array is split in half at each step, and Partition must eventually

examine all n elements, the run time is O(n lg n).

To find a pivot value, Partition could simply select the first element (A[Lb]). All other

values would be compared to the pivot value, and placed either to the left or right of the pivot as

appropriate. However, there is one case that fails miserably. Suppose the array was originally in

order. Partition would always select the lowest value as a pivot and split the array with one

element in the left partition, and Ub – Lb elements in the other. Each recursive call to quicksort

would only diminish the size of the array to be sorted by one. Therefore n recursive calls would

be required to do the sort, resulting in a O(n2) run time. One solution to this problem is to

randomly select an item as a pivot. This would make it extremely unlikely that worst-case

behavior would occur.

Implementation

The source for the quicksort algorithm may be found in file qui.c. Typedef T and comparison

operator compGT should be altered to reflect the data stored in the array. Several enhancements

have been made to the basic quicksort algorithm:

· The center element is selected as a pivot in partition. If the list is partially ordered,

this will be a good choice. Worst-case behavior occurs when the center element happens

to be the largest or smallest element each time partition is invoked.

· For short arrays, insertSort is called. Due to recursion and other overhead, quicksort

is not an efficient algorithm to use on small arrays. Consequently, any array with fewer

than 12 elements is sorted using an insertion sort. The optimal cutoff value is not critical

and varies based on the quality of generated code.

· Tail recursion occurs when the last statement in a function is a call to the function itself.

Tail recursion may be replaced by iteration, resulting in a better utilization of stack space.

This has been done with the second call to QuickSort in Figure 2-3.

· After an array is partitioned, the smallest partition is sorted first. This results in a better

utilization of stack space, as short partitions are quickly sorted and dispensed with.

Included in file qsort.c is the source for qsort, an ANSI-C standard library function usually

implemented with quicksort. Recursive calls were replaced by explicit stack operations. Table

2-1 shows timing statistics and stack utilization before and after the enhancements were applied.

count before after before after

16 103 51 540 28

256 1,630 911 912 112

4,096 34,183 20,016 1,908 168

65,536 658,003 470,737 2,436 252

time ( m s) stacksize

Table 2-1: Effect of Enhancements on Speed and Stack Utilization