6.3 Comparative Experiments of System Performance
6.3.4 Experiments
We will evaluate each testbed separately, and then compare their results at last.
6.3.4.1 Experimental Results from Testbed-MySQL
The experimental results shown inFigure 6.13make it easy for us to reach the following conclusions:
1) The best performance of inserting state data is gotten from using prepared state-ments. For update and read operations in both experiential environments, the use of stored procedures helps more. We will then use the best result for each operation in this testbed to compare with that from testbed-Cassandra.
2) The experiential results have proven that both prepared statement and stored pro-cedure can help for enhancing the system performance.
3) Stored procedure is more suitable for our application scenario. Some complex database operations, like update, insert, query and delete data from multiple ta-bles can be packaged together as a stored procedure, and manage as a transaction in the database.
4) The read performance of MySQL Cluster is high. The reason is that MySQL Cluster partitions and distributes data by hashing on keys. It uses then a hash index to query data, rather than scanning all rows.
5) The performance of update is higher than insert in our experiment. That is because by using theCopyUpdated strategy less data have been modified during the updating.
6) The data volume in the database does not affect on the results significantly. The reason could be that in this experiment, it is not a challenge to use a five-node MySQL Cluster to manage records of one million characters.
6.3.4.2 Experimental Results from Testbed-Cassandra
Figure 6.14 shows the experimental results of testbed-Cassandra. Accordingly, we get the following conclusions:
1) In contrast with the results from testbed-MySQL, the write performance of Cassan-dra is higher than read. The reason has been discussed inSection 3.4. Especially in this experiment, the read consistency level is ALL, which is higher than the write consistency level (ONE).
2) The data volume does not affect the experimental results. A five-node Cassandra cluster can also deal with the records of one million characters.
6.3. Comparative Experiments of System Performance 91
Insert Update Read
Basic 2983157 2034964 257146
Prepared Statement 1948707 1710949 251526
Stored Procedure 2195305 1555513 190943
0 500000 1000000 1500000 2000000 2500000 3000000 3500000
Average Running Time for 10000 Operations (ms)
(a) Comparison of the Performance of Insert, Update and Read Using Three Methods in the Experimental Environment I (No Character)
Update Read
Basic 1905043 297146
Prepared Statement 1709935 291786
Stored Procedure 1653074 177632
0 500000 1000000 1500000 2000000 2500000
Average Running Time for 10000 Operations (ms)
(b) Comparison of the Performance of Update and Read Using Three Methods in the Experimental Environment II (One Million Characters)
Figure 6.13: Comparison of the Performance (Average Running Time for 10000 Op-erations) of Different Operations of Testbed-MySQL Using Three Methods in Two Experimental Environments
Insert Update Read Average Running Time 108776,9 110803,4 113359,9
106000 107000 108000 109000 110000 111000 112000 113000 114000
Average Running Time for 10000 Operations (ms)
(a) The Performance of Insert, Update and Read in the Experimental Environ-ment I (No Character)
Update Read
Average Running Time 101119,2 122918
0 20000 40000 60000 80000 100000 120000 140000
Average Running Time for 10000 Operations (ms)
(b) The Performance of Update and Read in the Experimental Environment II (One Million Characters)
Figure 6.14: Performance (Average Running Time for 10000 Operations) of Different Operations of Testbed-Cassandra in Two Experimental Environments
6.3. Comparative Experiments of System Performance 93
6.3.4.3 Comparison of Experimental Results from Two Testbeds
We have integrated the best results from two testbeds into two diagrams (seeFigure 6.15 and Figure 6.16). The comparative results are obvious and convincing. In both two experimental environments, use of Cassandra can significantly improve the efficiency of data processing, even though we have not used any optimization technique. For example, inFigure 6.15the running time for inserting, updating and reading state data has been reduced respectively by 94.4%, 92.9% and 40.6% through using Cassandra.
Besides the different ways of data processing between RDBMS and NoSQL DBMS, there are following specific reasons:
Database Schema : in testbed-MySQL, the information of one character are as-signed to nine tables. Moreover, each bridge table holds a maximum of 20 records of a character. Thus, each data operation could involve at most 161 rows, which are distributed in different tables and even nodes in the cluster. The communica-tion and I/O costs are consequently high. In contrast, in testbed-Cassandra, the entire information of one character is kept together in one row. Therefore, the overhead for data access is minimized.
Consistency mechanism : the MySQL Cluster used in testbed-MySQL hosts two replicas for all data. By using the default two-phase commit mechanism, data are synchronized to both replicas during executing a write operation (see Sec-tion 2.2). Furthermore, distributed locks are required here to guarantee ACID transactions. However, in testbed-Cassandra, as long as one replica has been updated successfully, the write operation is considered to be complete.
Additionally, we can also reach some other conclusions:
1) The read performance has not been enhanced as much as the write performance.
There are two reasons: MySQL Cluster caches all data in memory. Cassandra only caches the latest or the most frequently accessed data. Therefore, disk I/O is inevitable; the read consistency level in Cassandra is specified to ALL. In contrast, MySQL Cluster fetched data only from one replica. In the next section, we will focus on improving the read performance of Cassandra.
2) The running time of checkpointing has been reduced obviously by using Cassandra.
The benefit is that the frequency of checkpointing could be increased, which reduces the loss caused by a game server failure. Or we can checkpoint more characters’
state data at the original frequency.
Overall, these experimental results have shown that the use of an RDBMS for data checkpointing and recovery in our game scenario is not efficient. The performance could be easily improved by applying Cassandra instead.
Testbed-MySQL
Testbed-Cassandra Insert 1948707 108776,9 Average Running Time for 10000 Operations (ms)
Insert
(a) Comparison of Insert Performance
Testbed-MySQL
Testbed-Cassandra Update 1555513 110803,4 Average Running Time for 10000 Operations (ms)
Update
(b) Comparison of Update Performance
Testbed-MySQL
Testbed-Cassandra Read 190943 113359,9 Average Running Time for 10000 Operations (ms)
Read
(c) Comparison of Read Performance
Figure 6.15: Comparison of the Performance (Average Running Time for 10000 Oper-ations) of Two Testbeds in the Experimental Environment I (No Character)
6.4. Experimental Proof of the Timestamp-based Model 95
Testbed-MySQL
Testbed-Cassandra Update 1653074 101119,2 Average Running Time for 10000 Operations (ms)
Update
(a) Comparison of Update Performance
Testbed-MySQL
Testbed-Cassandra
Read 177632 122918
Average Running Time for 10000 Operations (ms)
Read
(b) Comparison of Read Performance
Figure 6.16: Comparison of the Performance (Average Running Time for 10000 Oper-ations) of Two Testbeds in the Experimental Environment II (One Million Characters)
6.4 Experimental Proof of the Timestamp-based Model
In Chapter 5, we have pointed out Cassandra’ problem in guaranteeing read-your-writes Consistency. Moreover, in Section 6.3, experimental results have shown that using Cassandra cannot enhance the read performance as much as write performance.
Therefore, we have proposed the TSModel to address this problem. In this section, we want to verify it by experiments.