Too much to digest? Slice it up in chunks!

Posted by : Michael Mortensen Sunday, June 9, 2013

From time to time you will find yourself in the need of slicing up large chunks of bites into smaller chunks of consumable bites.

Since I have been unable to find such a method in the .NET framework, I decided to include one on the EnumerableUtility class found in the Cuemon.Collections.Generic namespace. The name for the method was carefully chosen to be Chunk, and consist of two overloads; both accepting an IEnumerable<T> where the chunk size is defaulted to 128 elements and the other with a custom size.

To see the Chunk method in action, have a look at Figure 1. For simplicity i set a chunk size to 255 and a workload count to 4096. This means that our workload consists of 4096 elements which we will process 255 elements at time. This leaves us with a workload of 17 runs in total.

[TestClass]

public class ChunkExample

{

    [TestMethod]

    public void TestChunk()

    {

        int chunkSize = 255;

        int workload = 4096;

        int expectedWorkloadIterations = (workload + chunkSize - 1 ) / chunkSize;

        IEnumerable<int> simulatedWorkload = new List<int>(EnumerableUtility.Range(0, workload));

        Debug.WriteLine("Work to process: {0} items.", workload);

        Debug.WriteLine("Expected iterations for work: {0} runs.", expectedWorkloadIterations);

        int actualWorkloadIterations = 0;

        while (((List<int>)simulatedWorkload).Count > 0)

        {

            IEnumerable<int> chunkedWorkload = EnumerableUtility.Chunk(ref simulatedWorkload, chunkSize);

            IEnumerator<int> chunkedWorkloadEnumerator = chunkedWorkload.GetEnumerator();

            int countOfChunkedElements = EnumerableUtility.Count(chunkedWorkload);

            Assert.IsTrue(countOfChunkedElements <= chunkSize);

            int enumeratorCount = 0;

            while (chunkedWorkloadEnumerator.MoveNext())

            {

                enumeratorCount++;

            }

            actualWorkloadIterations++;

            Assert.IsTrue(enumeratorCount <= chunkSize);

            Debug.WriteLine("Remaining work to process: {0} items.", ((List<int>)simulatedWorkload).Count);

        }

        Assert.IsTrue(expectedWorkloadIterations == actualWorkloadIterations);

        Debug.WriteLine("Actual iterations for work: {0} runs.", actualWorkloadIterations);

    }

}

Figure 1: A simple test method that utilizes the Chunk method found on the EnumerableUtility class.

When executed, the test method should produce this debug output:

Debug Trace:

Work to process: 4096 items.

Expected iterations for work: 17 runs.

Remaining work to 
process: 3841 items.

Remaining work to process: 3586 
items.

Remaining work to process: 3331 items.

Remaining work to process: 3076 items.

Remaining work to 
process: 2821 items.

Remaining work to process: 2566 
items.

Remaining work to process: 2311 items.

Remaining work to process: 2056 items.

Remaining work to 
process: 1801 items.

Remaining work to process: 1546 
items.

Remaining work to process: 1291 items.

Remaining work to process: 1036 items.

Remaining work to 
process: 781 items.

Remaining work to process: 526 items.

Remaining work to process: 271 items.

Remaining work to 
process: 16 items.

Remaining work to process: 0 items.

Actual iterations for work: 17 runs.

Figure 2: The output of the test in Figure 1.

I hope you liked this little peek into one of the many day-to-day helper methods found through out the Cuemon assembly family. Happy coding!