First Steps Into Big Data With Apache Cassandra

I’ve got a monitoring application at work that I wrote and maintain which currently uses MySQL as a storage back end. With the amount of data it holds and the activity in the system, MySQL has gone from “probably not the optimal solution” to “really stupid”. The system is comprised of many storage servers, most of which are completely I/O bottle-necked because of MySQL’s write overhead. It’s a typical “big data” kind of problem, and I need a big data solution.

Over the past couple of weeks, I’ve been experimenting with Apache Cassandra. We recently started using it in another context, and it seems pretty damned slick. Based on what I read, it seems like a great fit for my needs. The data model is consistent with what I do in MySQL, and the built-in redundancy and replication is awesome.

Most of the stuff I’ve tried so far has Just Worked (TM). Setting up a basic test cluster was easy, and once I found a suitable PHP client library for accessing Cassandra, I was able to make my development setup store collected data in under 30 minutes. I started off using a data model that pretty much mirrored what I was doing in MySQL, but as I learned more, I was able to strip away a few of the MySQL-specific “optimizations” (read: hacks) in favor of a more streamlined setup.

However, there are a few things that just make me scratch my head. From what I can tell, updating rows in Cassandra is “strange”. In my testing so far, inserting new data works flawlessly. Both adding new rows and adding columns onto an existing row work as expected. However, I notice lots of weirdness when updating pre-existing columns in pre-existing rows. It seems as though Cassandra is only updating the values associated with columns if the value is “larger” than the previous. See the following for an example.

# ./casstest
truncating column family for cleanliness...
========================================================
What we're storing...
Array
(
    [timestamp] => 1339529068
    [value] => 0.01
)
storing ...
sleeping a second for consistency...
What is retrieved from a get()...
Array
(
    [timestamp] => 1339529068
    [value] => 0.01
)
========================================================
What we're storing...
Array
(
    [timestamp] => 1339529071
    [value] => 1.01
)
storing ...
sleeping a second for consistency...
What is retrieved from a get()...
Array
(
    [timestamp] => 1339529071
    [value] => 1.01
)
========================================================
What we're storing...
Array
(
    [timestamp] => 1339529074
    [value] => 2.01
)
storing ...
sleeping a second for consistency...
What is retrieved from a get()...
Array
(
    [timestamp] => 1339529074
    [value] => 2.01
)
========================================================
What we're storing...
Array
(
    [timestamp] => 1339529077
    [value] => 1.01
)
storing ...
sleeping a second for consistency...
What is retrieved from a get()...
Array
(
    [timestamp] => 1339529077
    [value] => 2.01
)
========================================================
What we're storing...
Array
(
    [timestamp] => 1339529080
    [value] => 0.05
)
storing ...
sleeping a second for consistency...
What is retrieved from a get()...
Array
(
    [timestamp] => 1339529080
    [value] => 2.01
)
========================================================

In the example above, the timestamp column is just the result of a call to time(), so it will always increment over time. The values for the value column are just a few static entries pulled from a pre-populated array I used for testing. They increment three times, then decrement twice. I’m just making a simple array out of the two pieces of data, and then doing a set operation to write the data into Cassandra. As you can see, the timestamp fields show the proper values each time the key is retrieved, but the value column only shows the proper values when the value being written is larger than the last. WTF? I don’t know whether to blame Cassandra or the PHP client library I’m using (CPCL), but it’s really cramping my style at this point. I’ve gone as far as watching the contents of the TCP connections between client and server with tcpdump/wireshark to see if the client is making the same set requests for all values, and it seems to be. I’ve also tried changing the write consistency level, with no change.

It is also worth noting that when using the cassandra-cli utility to do inserts sets/gets manually, things work as I would expect.

[default@keyspace] assume TestCF VALIDATOR as utf8; 
Assumption for column family 'TestCF' added successfully.
[default@keyspace] assume TestCF SUB_COMPARATOR as utf8; 
Assumption for column family 'TestCF' added successfully.
[default@keyspace] assume TestCF keys as utf8; 
Assumption for column family 'TestCF' added successfully.
[default@keyspace] assume TestCF COMPARATOR as utf8; 
Assumption for column family 'TestCF' added successfully.
[default@keyspace] get TestCF['TestKey'];
=> (column=timestamp, value=1339532764, timestamp=172800)
=> (column=value, value=2.01, timestamp=172800)
Returned 2 results.
Elapsed time: 2 msec(s).
[default@keyspace] set TestCF['TestKey']['value'] = utf8('0.0');
Value inserted.
Elapsed time: 1 msec(s).
[default@keyspace] get TestCF['TestKey'];
=> (column=timestamp, value=1339532764, timestamp=172800)
=> (column=value, value=0.0, timestamp=1339532783904000)
Returned 2 results.
Elapsed time: 2 msec(s).
[default@keyspace] set TestCF['TestKey']['value'] = utf8('2.0');
Value inserted.
Elapsed time: 2 msec(s).
[default@keyspace] get TestCF['TestKey'];
=> (column=timestamp, value=1339532764, timestamp=172800)
=> (column=value, value=2.0, timestamp=1339532783913000)
Returned 2 results.
Elapsed time: 2 msec(s).
[default@keyspace] set TestCF['TestKey']['value'] = utf8('1.5');
Value inserted.
Elapsed time: 1 msec(s).
[default@keyspace] get TestCF['TestKey'];
=> (column=timestamp, value=1339532764, timestamp=172800)
=> (column=value, value=1.5, timestamp=1339532783923000)
Returned 2 results.
Elapsed time: 2 msec(s).
[default@keyspace] set TestCF['TestKey']['value'] = utf8('0.2');
Value inserted.
Elapsed time: 0 msec(s).
[default@keyspace] get TestCF['TestKey'];
=> (column=timestamp, value=1339532764, timestamp=172800)
=> (column=value, value=0.2, timestamp=1339532783933000)
Returned 2 results.
Elapsed time: 2 msec(s).

Another thing that isn’t acting as I would expect is row deletions. In my testing, it seems that once a row has been deleted, subsequent attempts to write to that row will just silently fail. I suspect that it has to do with the fact that Cassandra’s distributed nature makes deletes a bit counter-intuitive, which is outlined here in the Cassandra documentation. It would be nice to know for sure, though.

EDIT: I was doing it wrong. Sigh. Deletes are still weird to me though.

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>