@prefix foaf: . @prefix ns1: . ns1:this foaf:made . @prefix atom: . atom:contains ; atom:entry . @prefix sioc: . sioc:container_of . @prefix rdfs: . rdfs:label "page 1" . sioc:has_container . @prefix dt: . @prefix dcterms: . dcterms:created "2014-09-30T16:33:29.998391-04:00"^^dt:dateTime ; foaf:maker ns1:this ; rdfs:seeAlso ; dcterms:modified "2015-06-10T12:05:50.674672-04:00"^^dt:dateTime ; sioc:link ; sioc:id "37a936055a77f48c9536de5db8bc11ad" ; sioc:content "

Scalability, specifically linear scalability, means that twice the data takes twice as long to process, or that double the gear processes the same data in half the time. This is only literally true for "embarrassingly parallel" workloads.

\n\n

There are parts of TPC-H which have an embarrassingly parallel nature, like Q1 and Q7. There are parts that are almost as easy, like Q14, Q17, Q19, and Q21, where there is a big scan and a selective hash join with a hash table small enough to replicate everywhere. The scan scales linearly; building the hash does not, since it is done at single-server speed (once in each process). Some queries like Q9 and Q13 end up doing a big cross-partition join which runs into communication overheads.

\n\n

This is our first look at how performance behaves with bigger data and a larger platform. The results shown here are interesting but are not final. I bet I can do better; by how much is what we'll find out soon enough.

\n\n

We will here compare a 1000G setup on my desktop, and a 3000G setup at the CWI's Scilens cluster. The former is 2 boxes of dual Xeon E5 2630, and the latter is 8 boxes of dual Xeon E5 2650v2. All things run from memory and both have QDR IB interconnect. Counting cores and clock, the CWI cluster is 6x larger.

\n\n

As a rough approximation, for the worst queries, 6x the gear runs 3x the data in the same amount of real time. The 1000G setup has near full platform utilization and the 3000G setup has about half platform utilization. In both cases, running two instances of the same query at the same time takes twice as long.

\n\n

We use Q9 for this study. The plan makes a hash table of part with 1/14 of all parts, replicating to all processes. Then there is a hash table of partsupp with a key of ps_partkey, ps_suppkey, and a dependent of ps_supplycost. This is much larger than the part hash table and is therefore partitioned on ps_partkey. The build is for 1/14th of partsupp. Then there is a scan of lineitem filtered by the part hash table; then a cross-partition join to the partsupp hash table; then a cross partition join to orders, this time by index; then a hash join on a replicated hash table of supplier; then nation; then aggregation. The aggregation is done in each slice; then the slices are added up at the end.

\n\n

The plan could be made better by one fewer partition crossing. Now there is a crossing from l_orderkey to l_partkey and back to o_orderkey. This would not be so if the cost model knew that the partsupp always hits. The cost model thinks it hits 1/14 of the time, because it does not know that the selection on the build is exactly the same as on the probe.

\n\n

For the present purposes, the extra crossing just serves to make the matter of interest more visible.

\n\n

So, for the 1000G setup, we have 43.6 seconds (s) and

\n\n

Cluster 4 nodes, 44 s. 459 m/s 119788 KB/s  3120% cpu 0%  read 19% clw threads 1r 0w 0i buffers 17622126 68 d 0 w 0 pfs\n

\n\n

For the 3000G setup, we have 49.9 s and

\n\n

Cluster 16 nodes, 50 s. 49389 m/s 1801815 KB/s  7283% cpu 0%  read 18% clw threads 1r 0w 0i buffers 135122893 15895255 d 0 w 17 pfs\n

\n\n

The platform utilization on the small system is better, at 31/48 (running/total threads); the large one has 73/256.

\n\n

The large case is clearly network bound. If this were for CPU only, it should be done in half the time it takes the small system to do 1000G.

\n\n

We confirm this by looking at write wait: 3940 seconds of thread time blocked on write over 50s of real time. The figures on the small one are 3.9s of thread time blocked for 39s of real time. The data transfer on the large one is 93 GB.

\n\n

How to block less? One idea would be to write less. So we try compression; there is a Google snappy-based message compression option in Virtuoso.

\n\n

We now get 39.6 s and

\n\n

Cluster 16 nodes, 40 s. 65161 m/s 1239922 KB/s  10201% cpu 0%  read 21% clw threads 1r 0w 0i buffers 52828440 172 d 0 w 0 pfs\n

\n\n

The write block time is 397 s of thread time over 39 s of real time, 10x better. The data transfer is 50.9 GB after compression. Snappy is somewhat effective for compression and very fast; in CPU profile, it is under 3% of Q9 on the small system. Gains on the small system are less, though, since blocking is not a big issue to start with.

\n\n

This is still not full platform. But if the data transfer is further cut in half by a better plan, the situation will be quite good. Now we have 102/256 threads running, meaning that there could be another 40-50% of throughput to be added. The last 128 threads are second threads of a core, so count for roughly 30% of a real core.

\n\n

The main cluster-specific operation is a send from one to many. This is now done by formulating the message to each recipient in a chain of string buffers; then, after all the messages are prepared, these are optionally compressed and sent to their recipient. This is needlessly simple: Compressing can proceed if ever there is a would-block situation on writing. If all the compression is done, then a blocked write should switch to another recipient, and only after all recipients have a would-block situation, then the thread can call-select with all descriptors and block on them collectively. There is a piece of code to this effect, but is not now being used. It has been seen to add no value in small cases, but could be useful here.

\n\n

The IB fabric has been seen to do 1.8 GB/s bidirectionally on multiple independent point-to-point TCP links. This is about half the nominal 4 GB/s (40 Gbit/s with 10/8 encoding). So the aggregate throughputs that we see here are nowhere near the nominal spec of the network. Lower level interfaces and the occasional busy wait on the reading end could be tried to some advantage. We have not tried 10GbE either; but if that works at nominal speed, then 10GbE should also be good enough. We will try this at Amazon in due time.

\n\n

In the meantime, there is a 3000G test made at the CWI cluster without message compression. The score is about 4x that of the single server at 300G using the same hardware. The run is with approximately half platform utilization. There are three runs of power plus throughput, the first run being cold.

\n\n\n\n \n \n \n \n\n\n \n \n \n \n\n\n \n \n \n \n\n\n \n \n \n \n\n

Run	Power	Throughput	Composite
Run 1	305,881.5	1,072,411.9	572,739.8
Run 2	1,292,085.1	1,179,391.6	1,234,453.1
Run 3	1,178,534.1	1,092,936.2	1,134,928.4

\n\n

The numerical quantities summaries follow. One problem of the run is a high peak of query memory consumption leading to slowdown. Some parts should probably be done in multiple passes to keep the peak lower and not run into swapping. The details will have to be sorted out. This is a demonstration of capability; the perfected accomplishment is to follow.

\n\n\n\n

3000G Run 1

\n\n

Virt-H Executive Summary

\n\n\n\n \n \n\n\n\n \n \n\n\n\n\n \n \n\n\n\n \n \n\n\n \n \n\n\n \n \n\n\n\n \n \n\n

Report Date	September 29, 2014
Database Scale Factor	3000
Query Streams for Throughput Test	8
Virt-H Power	305,881.5
Virt-H Throughput	1,072,411.9
Virt-H Composite Query-per-Hour Metric (Qph@100GB)	572,739.8
Measurement Interval in Throughput Test (Ts)	1,772.554000 seconds

\n\n

Duration of stream execution

\n\n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

	Start Date/Time	End Date/Time	Duration
Stream 0	09/29/2014 12:54:52	09/29/2014 13:31:17	0:36:25
Stream 1	09/29/2014 13:31:24	09/29/2014 13:59:24	0:28:00
Stream 2	09/29/2014 13:31:24	09/29/2014 13:58:59	0:27:35
Stream 3	09/29/2014 13:31:24	09/29/2014 13:58:29	0:27:05
Stream 4	09/29/2014 13:31:24	09/29/2014 13:58:52	0:27:28
Stream 5	09/29/2014 13:31:24	09/29/2014 14:00:06	0:28:42
Stream 6	09/29/2014 13:31:24	09/29/2014 13:58:18	0:26:54
Stream 7	09/29/2014 13:31:24	09/29/2014 13:59:25	0:28:01
Stream 8	09/29/2014 13:31:24	09/29/2014 13:58:50	0:27:26
Refresh 0	09/29/2014 12:54:52	09/29/2014 12:56:59	0:02:07
	09/29/2014 13:31:17	09/29/2014 13:31:23	0:00:06
Refresh 1	09/29/2014 14:00:38	09/29/2014 14:01:11	0:00:33
Refresh 2	09/29/2014 13:31:25	09/29/2014 13:36:57	0:05:32
Refresh 3	09/29/2014 13:36:56	09/29/2014 13:47:02	0:10:06
Refresh 4	09/29/2014 13:47:03	09/29/2014 13:51:40	0:04:37
Refresh 5	09/29/2014 13:51:42	09/29/2014 13:56:40	0:04:58
Refresh 6	09/29/2014 13:56:40	09/29/2014 13:59:25	0:02:45
Refresh 7	09/29/2014 13:59:25	09/29/2014 14:00:10	0:00:45
Refresh 8	09/29/2014 14:00:11	09/29/2014 14:00:37	0:00:26

\n\n

Numerical Quantities Summary Timing Intervals in Seconds:

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8
Stream 0	601.576975	90.803782	108.725110	177.112667	171.995572	2.098138	15.768311	152.511444
Stream 1	13.310341	32.722946	125.551415	1.912836	46.041675	13.294214	85.345068	165.424288
Stream 2	19.425885	9.248670	150.855556	7.085737	88.445566	10.490432	49.318554	322.500839
Stream 3	30.534391	14.273478	100.987791	59.341763	46.442443	9.613795	64.186196	146.324186
Stream 4	28.211213	37.134522	64.189335	10.931513	100.610673	9.929866	112.270530	108.489951
Stream 5	29.226411	18.132589	95.245160	63.100068	115.663908	6.151231	46.251309	127.742471
Stream 6	30.750930	20.888658	108.894177	55.168565	82.016828	69.451493	65.161517	103.697733
Stream 7	13.462570	18.033847	32.065492	78.910373	202.998301	10.688279	47.167022	139.601948
Stream 8	24.354314	16.711503	112.008551	8.307098	126.849630	7.127605	51.083118	98.648077
Min Qi	13.310341	9.248670	32.065492	1.912836	46.041675	6.151231	46.251309	98.648077
Max Qi	30.750930	37.134522	150.855556	78.910373	202.998301	69.451493	112.270530	322.500839
Avg Qi	23.659507	20.893277	98.724685	35.594744	101.133628	17.093364	65.097914	151.553687

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q9	Q10	Q11	Q12	Q13	Q14	Q15	Q16
Stream 0	92.991259	5.175922	42.238393	29.239879	367.805534	3.604910	15.557396	11.650267
Stream 1	149.502128	30.197806	50.786184	217.190836	283.545905	11.653171	73.321150	116.860455
Stream 2	245.783668	22.278841	50.578731	36.301810	181.405269	32.236754	57.631764	61.540533
Stream 3	377.782738	24.129319	84.097657	10.959661	171.698669	8.973519	54.532180	45.527142
Stream 4	341.148908	74.358770	85.782399	43.116347	151.146233	22.870727	74.439693	51.871535
Stream 5	72.259919	11.424035	79.310504	9.833135	562.871920	14.961209	127.861874	55.377721
Stream 6	373.301225	41.379753	81.983260	9.373200	95.039317	19.071346	76.159452	48.324504
Stream 7	449.871952	16.099152	48.047940	8.559784	211.094730	10.569071	26.710228	72.571454
Stream 8	395.771006	33.537585	54.850876	141.526389	153.763316	12.997092	127.961975	57.100346
Min Qi	72.259919	11.424035	48.047940	8.559784	95.039317	8.973519	26.710228	45.527142
Max Qi	449.871952	74.358770	85.782399	217.190836	562.871920	32.236754	127.961975	116.860455
Avg Qi	300.677693	31.675658	66.929694	59.607645	226.320670	16.666611	77.327289	63.646711

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q17	Q18	Q19	Q20	Q21	Q22	RF1	RF2
Stream 0	12.230334	70.991261	33.092797	17.517230	15.798438	19.743562	127.494687	5.893471
Stream 1	27.550293	14.970857	16.442806	111.138612	68.214095	7.884782	27.109441	6.087067
Stream 2	43.277918	12.748690	22.681844	92.835566	84.416610	14.661934	151.094498	153.285076
Stream 3	129.696125	13.435663	14.674499	129.179966	39.176513	6.286296	181.596838	416.052710
Stream 4	110.348816	7.080225	21.051910	85.758973	65.130356	7.292999	123.386514	151.000786
Stream 5	43.365006	9.847612	32.881770	94.752284	67.788314	9.035439	72.539334	223.967821
Stream 6	34.534280	36.347298	27.849276	122.736244	51.447492	25.051058	80.452175	84.519426
Stream 7	48.021860	30.594474	22.522426	99.245893	73.076698	7.260729	38.585852	5.697277
Stream 8	29.484201	12.368769	40.344043	84.137820	30.813313	4.856991	22.196547	4.600057
Min Qi	27.550293	7.080225	14.674499	84.137820	30.813313	4.856991	22.196547	4.600057
Max Qi	129.696125	36.347298	40.344043	129.179966	84.416610	25.051058	181.596838	416.052710
Avg Qi	58.284812	17.174198	24.806072	102.473170	60.007924	10.291279	87.120150	130.651277

\n\n\n

3000G Run 2

\n\n

Virt-H Executive Summary

\n\n\n\n \n \n\n\n\n \n \n\n\n\n \n \n\n\n\n \n \n\n\n \n \n\n\n \n \n\n\n\n \n \n\n

Report Date	September 29, 2014
Database Scale Factor	3000
Query Streams for Throughput Test	8
Virt-H Power	1292085.1
Virt-H Throughput	1179391.6
Virt-H Composite Query-per-Hour Metric (Qph@100GB)	1234453.1
Measurement Interval in Throughput Test (Ts)	1611.779000 seconds

\n\n

Duration of stream execution

\n\n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

	Start Date/Time	End Date/Time	Duration
Stream 0	09/29/2014 14:01:15	09/29/2014 14:06:48	0:05:33
Stream 1	09/29/2014 14:06:53	09/29/2014 14:30:22	0:23:29
Stream 2	09/29/2014 14:06:53	09/29/2014 14:32:30	0:25:37
Stream 3	09/29/2014 14:06:53	09/29/2014 14:31:23	0:24:30
Stream 4	09/29/2014 14:06:53	09/29/2014 14:31:34	0:24:41
Stream 5	09/29/2014 14:06:53	09/29/2014 14:32:53	0:26:00
Stream 6	09/29/2014 14:06:53	09/29/2014 14:29:51	0:22:58
Stream 7	09/29/2014 14:06:53	09/29/2014 14:31:34	0:24:41
Stream 8	09/29/2014 14:06:53	09/29/2014 14:30:35	0:23:42
Refresh 0	09/29/2014 14:01:15	09/29/2014 14:01:35	0:00:20
	09/29/2014 14:06:49	09/29/2014 14:06:53	0:00:04
Refresh 1	09/29/2014 14:33:16	09/29/2014 14:33:45	0:00:29
Refresh 2	09/29/2014 14:06:55	09/29/2014 14:12:28	0:05:33
Refresh 3	09/29/2014 14:12:29	09/29/2014 14:21:55	0:09:26
Refresh 4	09/29/2014 14:21:55	09/29/2014 14:27:40	0:05:45
Refresh 5	09/29/2014 14:27:43	09/29/2014 14:31:14	0:03:31
Refresh 6	09/29/2014 14:31:14	09/29/2014 14:31:51	0:00:37
Refresh 7	09/29/2014 14:31:51	09/29/2014 14:32:52	0:01:01
Refresh 8	09/29/2014 14:32:52	09/29/2014 14:33:16	0:00:24

\n\n

Numerical Quantities Summary Timing Intervals in Seconds:

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8
Stream 0	9.451169	3.644118	18.419151	1.404395	15.740525	2.085038	15.171847	25.400834
Stream 1	19.558041	6.607300	85.774410	4.503525	81.448472	11.976129	92.140470	145.743853
Stream 2	31.042019	7.877299	71.958033	8.862111	142.452144	18.489193	81.003310	85.856529
Stream 3	38.833612	12.440326	86.063103	7.165120	84.707025	16.931531	100.442710	122.411252
Stream 4	15.751913	33.026762	50.457193	7.064220	114.130257	5.992556	66.035959	84.596973
Stream 5	18.462884	28.047942	110.690543	16.566547	104.403789	5.303453	72.552640	402.383383
Stream 6	17.858339	33.988800	110.431091	7.238431	72.229953	16.850955	68.231546	180.601000
Stream 7	23.055572	17.044813	96.105520	8.941132	171.130879	8.423100	70.634541	147.261648
Stream 8	19.840798	13.860740	74.961175	16.171566	56.165875	5.904921	47.646217	125.991819
Min Qi	15.751913	6.607300	50.457193	4.503525	56.165875	5.303453	47.646217	84.596973
Max Qi	38.833612	33.988800	110.690543	16.566547	171.130879	18.489193	100.442710	402.383383
Avg Qi	23.050397	19.111748	85.805134	9.564082	103.333549	11.233980	74.835924	161.855807

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q9	Q10	Q11	Q12	Q13	Q14	Q15	Q16
Stream 0	54.766945	5.551163	29.216632	3.035008	52.816902	3.346243	15.767022	10.066112
Stream 1	130.666380	9.658277	49.332720	103.036705	194.520370	12.166344	65.144599	97.158571
Stream 2	254.754936	22.605298	38.102466	21.121168	300.467330	12.262318	108.203491	50.696657
Stream 3	283.761567	19.327164	73.414574	7.431651	183.121904	12.573854	73.814766	46.802493
Stream 4	290.341947	57.452026	58.354221	13.066162	189.263163	18.998781	121.269774	54.831406
Stream 5	81.787025	8.410538	79.822552	16.005077	190.730342	21.697136	100.456487	46.744884
Stream 6	202.558515	39.360009	74.519981	15.960756	137.321631	26.583824	57.537668	60.758997
Stream 7	226.790801	44.175536	73.992368	7.561897	182.853851	17.597471	31.128055	44.389893
Stream 8	275.423934	21.980040	60.538239	39.736622	173.574795	58.786316	95.124912	25.564108
Min Qi	81.787025	8.410538	38.102466	7.431651	137.321631	12.166344	31.128055	25.564108
Max Qi	290.341947	57.452026	79.822552	103.036705	300.467330	58.786316	121.269774	97.158571
Avg Qi	218.260638	27.871111	63.509640	27.990005	193.981673	22.583255	81.584969	53.368376

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q17	Q18	Q19	Q20	Q21	Q22	RF1	RF2
Stream 0	13.620157	2.288504	4.166807	16.468447	9.991810	1.101775	20.152227	4.294680
Stream 1	44.026143	31.720525	25.684461	134.254716	30.797008	9.568594	24.328205	4.319533
Stream 2	40.283148	9.970277	29.731019	133.083785	29.322194	8.859556	73.251098	249.850045
Stream 3	44.288244	18.914661	38.162762	144.458624	22.556235	6.184842	117.267234	445.700238
Stream 4	67.147744	6.649451	27.876825	59.226248	69.373248	44.478703	61.381724	282.608075
Stream 5	36.403227	12.226129	21.997683	95.912670	44.219799	21.117974	106.473817	97.896971
Stream 6	42.114038	30.805969	25.929027	51.658733	26.475662	34.816500	31.309953	5.608395
Stream 7	48.601889	18.708127	18.893532	132.558026	50.476383	12.309402	22.661371	37.610815
Stream 8	34.413417	34.709883	37.058335	121.710608	44.676485	9.449332	19.311945	4.420232
Min Qi	34.413417	6.649451	18.893532	51.658733	22.556235	6.184842	19.311945	4.319533
Max Qi	67.147744	34.709883	38.162762	144.458624	69.373248	44.478703	117.267234	445.700238
Avg Qi	44.659731	20.463128	28.166705	109.107926	39.737127	18.348113	56.998168	141.001788

\n\n\n

3000G Run 3

\n\n

Virt-H Executive Summary

\n\n\n\n \n \n\n\n\n \n \n\n\n\n\n \n \n\n\n\n \n \n\n\n \n \n\n\n \n \n\n\n\n \n \n\n

Report Date	September 29, 2014
Database Scale Factor	3000
Query Streams for Throughput Test	8
Virt-H Power	1178534.1
Virt-H Throughput	1092936.2
Virt-H Composite Query-per-Hour Metric (Qph@100GB)	1134928.4
Measurement Interval in Throughput Test (Ts)	1739.269000 seconds

\n\n

Duration of stream execution

\n\n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n

	Start Date/Time	End Date/Time	Duration
Stream 0	09/29/2014 14:33:48	09/29/2014 14:40:59	0:07:11
Stream 1	09/29/2014 14:41:04	09/29/2014 15:10:02	0:28:58
Stream 2	09/29/2014 14:41:04	09/29/2014 15:09:07	0:28:03
Stream 3	09/29/2014 14:41:04	09/29/2014 15:09:17	0:28:13
Stream 4	09/29/2014 14:41:04	09/29/2014 15:09:55	0:28:51
Stream 5	09/29/2014 14:41:04	09/29/2014 15:09:39	0:28:35
Stream 6	09/29/2014 14:41:04	09/29/2014 15:09:46	0:28:42
Stream 7	09/29/2014 14:41:04	09/29/2014 15:09:58	0:28:54
Stream 8	09/29/2014 14:41:04	09/29/2014 15:08:58	0:27:54
Refresh 0	09/29/2014 14:33:48	09/29/2014 14:34:07	0:00:19
	09/29/2014 14:40:59	09/29/2014 14:41:04	0:00:05
Refresh 1	09/29/2014 15:06:57	09/29/2014 15:09:49	0:02:52
Refresh 2	09/29/2014 14:41:05	09/29/2014 14:47:39	0:06:34
Refresh 3	09/29/2014 14:47:40	09/29/2014 14:56:46	0:09:06
Refresh 4	09/29/2014 14:56:49	09/29/2014 15:03:19	0:06:30
Refresh 5	09/29/2014 15:03:24	09/29/2014 15:06:45	0:03:21
Refresh 6	09/29/2014 15:06:46	09/29/2014 15:06:49	0:00:03
Refresh 7	09/29/2014 15:06:50	09/29/2014 15:06:53	0:00:03
Refresh 8	09/29/2014 15:06:53	09/29/2014 15:10:04	0:03:11

\n\n

Numerical Quantities Summary Timing Intervals in Seconds:

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8
Stream 0	9.393632	5.001910	17.053567	1.427500	17.813839	2.230451	13.884490	25.610995
Stream 1	12.971454	9.383520	94.257760	1.603106	127.940946	20.791892	78.869819	138.521273
Stream 2	21.428177	31.431513	96.366083	5.611843	58.394596	11.279502	47.114473	407.135077
Stream 3	23.377920	37.474814	83.640621	9.152178	71.186158	11.001543	46.763758	110.015662
Stream 4	49.580860	31.979940	87.662950	8.983661	68.052295	14.367631	59.266063	301.788652
Stream 5	13.483836	20.203772	391.980128	12.505446	77.966993	10.487869	52.989448	226.837637
Stream 6	38.104903	21.271630	84.689348	8.626460	86.620802	11.981171	69.182098	111.810485
Stream 7	20.243617	12.298692	99.547203	6.020951	151.584400	17.528287	62.037348	101.023802
Stream 8	22.808294	17.583072	59.180595	5.618565	123.108771	11.477376	42.485363	92.035709
Min Qi	12.971454	9.383520	59.180595	1.603106	58.394596	10.487869	42.485363	92.035709
Max Qi	49.580860	37.474814	391.980128	12.505446	151.584400	20.791892	78.869819	407.135077
Avg Qi	25.249883	22.703369	124.665586	7.265276	95.606870	13.614409	57.338546	186.146037

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q9	Q10	Q11	Q12	Q13	Q14	Q15	Q16
Stream 0	146.487681	6.798942	29.834475	3.177879	55.067866	4.503738	17.215591	9.333281
Stream 1	177.581204	44.178095	69.746005	12.306166	215.602727	30.443709	64.276384	45.266949
Stream 2	211.311651	27.403143	61.412478	12.173058	216.879170	18.272234	96.753886	35.587072
Stream 3	482.581456	68.663026	60.354163	13.408513	187.921639	17.469237	62.337222	31.706120
Stream 4	178.297373	23.711312	67.129677	15.216904	328.149575	20.258853	78.891201	84.852368
Stream 5	209.496498	28.346366	55.584081	9.644075	131.622351	24.171156	80.046801	43.625932
Stream 6	521.691639	24.126176	72.964805	15.311409	146.152570	34.748843	71.957130	58.470644
Stream 7	580.320149	17.054563	56.172396	7.530832	200.100326	12.444021	25.910599	75.653693
Stream 8	472.231674	15.064398	89.875570	42.394675	166.589234	12.831209	81.697881	73.821769
Min Qi	177.581204	15.064398	55.584081	7.530832	131.622351	12.444021	25.910599	31.706120
Max Qi	580.320149	68.663026	89.875570	42.394675	328.149575	34.748843	96.753886	84.852368
Avg Qi	354.188955	31.068385	66.654897	15.998204	199.127199	21.329908	70.233888	56.123068

\n\n\n\n \n \n \n \n \n \n \n \n \n\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n

Query	Q17	Q18	Q19	Q20	Q21	Q22	RF1	RF2
Stream 0	12.252670	2.593733	4.115862	16.895672	10.183350	1.240096	18.679685	4.876067
Stream 1	356.740980	21.197870	30.422216	81.779038	65.468650	3.947503	63.933750	107.563796
Stream 2	54.087768	10.152604	34.940701	113.510640	70.908809	12.316233	109.091578	283.076004
Stream 3	52.807104	18.525982	13.740089	212.364908	16.413964	17.998809	58.653503	483.718271
Stream 4	42.389062	36.157809	28.909260	86.427025	21.605419	7.608729	54.910853	331.074114
Stream 5	48.214794	15.778893	20.681799	130.560005	43.846752	33.905533	54.536966	139.563667
Stream 6	84.061840	26.224851	16.546432	117.265210	34.766856	39.037423	0.710642	1.645351
Stream 7	63.034890	15.966686	31.666488	112.689765	28.661943	12.828171	1.274731	1.780452
Stream 8	43.879104	8.596666	32.585746	177.928730	26.763334	6.112333	1.187693	0.533668
Min Qi	42.389062	8.596666	13.740089	81.779038	16.413964	3.947503	0.710642	0.533668
Max Qi	356.740980	36.157809	34.940701	212.364908	70.908809	39.037423	109.091578	483.718271
Avg Qi	93.151943	19.075170	26.186591	129.065665	38.554466	16.719342	43.037465	168.619415

\n\n\n

To be continued...

\n\n

\nIn Hoc Signo Vinces (TPC-H) Series

\n In Hoc Signo Vinces (part 1): Virtuoso meets TPC-H\n
\n In Hoc Signo Vinces (part 2): TPC-H Schema Choices\n
\n In Hoc Signo Vinces (part 3): Benchmark Configuration Settings\n
\n In Hoc Signo Vinces (part 4): Bulk Load and Refresh\n
\n In Hoc Signo Vinces (part 5): The Return of SQL Federation\n
\n In Hoc Signo Vinces (part 6): TPC-H Q1 and Q3: An Introduction to Query Plans\n
\n In Hoc Signo Vinces (part 7): TPC-H Q13: The Good and the Bad Plans\n
\n In Hoc Signo Vinces (part 8): TPC-H: INs, Expressions, ORs\n
\n In Hoc Signo Vinces (part 9): TPC-H Q18, Ordered Aggregation, and Top K\n
\n In Hoc Signo Vinces (part 10): TPC-H Q9, Q17, Q20 - Predicate Games \n
\n In Hoc Signo Vinces (part 11): TPC-H Q2, Q10 - Late Projection \n
\n In Hoc Signo Vinces (part 12): TPC-H: Result Preview \n
\n In Hoc Signo Vinces (part 13): Virtuoso TPC-H Kit Now on V7 Fast Track \n
\n In Hoc Signo Vinces (part 14): Virtuoso TPC-H Implementation Analysis \n
\n In Hoc Signo Vinces (part 15): TPC-H and the Science of Hash \n
\n In Hoc Signo Vinces (part 16): Introduction to Scale-Out \n
\n In Hoc Signo Vinces (part 17): 100G and 300G Runs on Dual Xeon E5 2650v2 \n
\n In Hoc Signo Vinces (part 18): Cluster Dynamics \n
\nIn Hoc Signo Vinces (part 19): Scalability, 1000G, and 3000G (this post)\n
\n In Hoc Signo Vinces (part 20): 100G and 1000G With Cluster; When is Cluster Worthwhile; Effects of I/O \n
\n In Hoc Signo Vinces (part 21): Running TPC-H on Virtuoso Cluster on Amazon EC2 \n

\n" . @prefix dc: . dc:title "In Hoc Signo Vinces (part 19 of n): Scalability, 1000G, and 3000G" . @prefix opl: . opl:isDescribedUsing ; atom:source ; atom:updated "2015-06-10T16:05:50Z" ; atom:title "In Hoc Signo Vinces (part 19 of n): Scalability, 1000G, and 3000G" ; sioc:links_to , , , , , , , , , , , , , , , , , , , , , , ; atom:author ns1:this ; rdfs:label "In Hoc Signo Vinces (part 19 of n): Scalability, 1000G, and 3000G" ; atom:published "2014-09-30T20:33:29Z" . @prefix ns9: . @prefix ns10: . ns9:has_services ns10:item . @prefix rdf: . rdf:type atom:Entry . @prefix schema: . rdf:type schema:BlogPosting . @prefix sioct: . rdf:type sioct:BlogPost . ns10:item ns9:services_of .