How to manipulate data on memcached level and rewrite to db

Hi,

On all examples I could find of using memcached it is used as query
result cache. This means that all data manipulations and updates are
done on the db level and not in memory.

On applications with many updates, it seems to be much more efficient
to update the memcache object and write it to the db periodically with
a background process.

I couldn't find any information on how to do this on PHP/Mysql
configuration.

In pseudo code:

is user data in cache?
if not check if user data in database. if so fetch it.
if user data is not in the database/cache create PHP object
representing user data row with default values.
update user data

once in 10 minutes write all updated users data to database.

I tried to implement it on a rather complex user data and found its
complicated to create the object and write it to the db.
Has anyone done it automatically?
Some of the issues that have to be tackeled:
1.Creating empty user data object from mysql .
2.Writing the data back to the database from the data object.

Any help is very much appreciated.

On Sep 20, 12:35 pm, Nbd <nir...@gmail.com> wrote:

> On all examples I could find of using memcached it is used as query
> result cache. This means that all data manipulations and updates are
> done on the db level and not in memory.

This is not a recommended usage.

Stuff comes out of the database in result set form. I have trouble
imagining a case where you don't convert that data into something else
(an object or list of objects (for which multiple queries may be
required), or in the worst cases, a direct web page). That "something
else" is what you should be caching.

> once in 10 minutes write all updated users data to database.

From where? Hopefully not the cache. Caches are never guaranteed
to have data. And you'd still need to know which records, exactly you
need updated.

It sounds like what you're looking for is a job queue. I'd
recommend gearman or beanstalkd depending on your needs.

What you want is an object-relational mapper, but I have no idea if there
are any good ones for PHP. We implemented our own in C# after having looked
at Django which we liked alot. As always, google is your friend:

http://en.wikipedia.org/wiki/List_of_object-relational_mapping_software


/Henrik

On Sun, Sep 20, 2009 at 21:35, Nbd <[email protected]> wrote:

>
> Hi,
>
> On all examples I could find of using memcached it is used as query
> result cache. This means that all data manipulations and updates are
> done on the db level and not in memory.
>
> On applications with many updates, it seems to be much more efficient
> to update the memcache object and write it to the db periodically with
> a background process.
>
> I couldn't find any information on how to do this on PHP/Mysql
> configuration.
>
> In pseudo code:
>
> is user data in cache?
> if not check if user data in database. if so fetch it.
> if user data is not in the database/cache create PHP object
> representing user data row with default values.
> update user data
>
> once in 10 minutes write all updated users data to database.
>
> I tried to implement it on a rather complex user data and found its
> complicated to create the object and write it to the db.
> Has anyone done it automatically?
> Some of the issues that have to be tackeled:
> 1.Creating empty user data object from mysql .
> 2.Writing the data back to the database from the data object.
>
> Any help is very much appreciated.
>
>
>

OBject relational mapper looks like an overkill. All I need is a way
to create 'empty' database object just like mysql would return from a
row query and a function to update the data back to mysql on the
background process. seems simple and streightforward, and gives
memcached a very powerful use case.



On Sep 21, 1:09 pm, Henrik Schröder <skro...@gmail.com> wrote:
> What you want is an object-relational mapper, but I have no idea if there
> are any good ones for PHP. We implemented our own in C# after having looked
> at Django which we liked alot. As always, google is your friend:
>
> http://en.wikipedia.org/wiki/List_of_object-relational_mapping_software
>
> /Henrik
>
> On Sun, Sep 20, 2009 at 21:35, Nbd <nir...@gmail.com> wrote:
>
> > Hi,
>
> > On all examples I could find of using memcached it is used as query
> > result cache. This means that all data manipulations and updates are
> > done on the db level and not in memory.
>
> > On applications with many updates, it seems to be much more efficient
> > to update the memcache object and write it to the db periodically with
> > a background process.
>
> > I couldn't find any information on how to do this on PHP/Mysql
> > configuration.
>
> > In pseudo code:
>
> > is user data in cache?
> > if not check if user data in database. if so fetch it.
> > if user data is not in the database/cache create PHP object
> > representing user data row with default values.
> > update user data
>
> > once in 10 minutes write all updated users data to database.
>
> > I tried to implement it on a rather complex user data and found its
> > complicated to create the object and write it to the db.
> > Has anyone done it automatically?
> > Some of the issues that have to be tackeled:
> > 1.Creating empty user data object from mysql .
> > 2.Writing the data back to the database from the data object.
>
> > Any help is very much appreciated.

As Dustin said, it is strongly recommended that you do not cache recordset
objects, but that you instead only cache intermediary objects, and have ways
to save and load these to your database. This is what a bare-bones ORM does,
it lets you work with objects that can be persisted easily in a database,
and adding caching to such a framework is pretty easy.

I have no idea what the quality of the PHP frameworks are, but maybe you can
look at them and take only the bare minimum that you need?


/Henrik

On Mon, Sep 21, 2009 at 17:42, Nbd <[email protected]> wrote:

>
> OBject relational mapper looks like an overkill. All I need is a way
> to create 'empty' database object just like mysql would return from a
> row query and a function to update the data back to mysql on the
> background process. seems simple and streightforward, and gives
> memcached a very powerful use case.
>
>
>
> On Sep 21, 1:09 pm, Henrik Schröder <skro...@gmail.com> wrote:
> > What you want is an object-relational mapper, but I have no idea if there
> > are any good ones for PHP. We implemented our own in C# after having
> looked
> > at Django which we liked alot. As always, google is your friend:
> >
> > http://en.wikipedia.org/wiki/List_of_object-relational_mapping_software
> >
> > /Henrik
> >
> > On Sun, Sep 20, 2009 at 21:35, Nbd <nir...@gmail.com> wrote:
> >
> > > Hi,
> >
> > > On all examples I could find of using memcached it is used as query
> > > result cache. This means that all data manipulations and updates are
> > > done on the db level and not in memory.
> >
> > > On applications with many updates, it seems to be much more efficient
> > > to update the memcache object and write it to the db periodically with
> > > a background process.
> >
> > > I couldn't find any information on how to do this on PHP/Mysql
> > > configuration.
> >
> > > In pseudo code:
> >
> > > is user data in cache?
> > > if not check if user data in database. if so fetch it.
> > > if user data is not in the database/cache create PHP object
> > > representing user data row with default values.
> > > update user data
> >
> > > once in 10 minutes write all updated users data to database.
> >
> > > I tried to implement it on a rather complex user data and found its
> > > complicated to create the object and write it to the db.
> > > Has anyone done it automatically?
> > > Some of the issues that have to be tackeled:
> > > 1.Creating empty user data object from mysql .
> > > 2.Writing the data back to the database from the data object.
> >
> > > Any help is very much appreciated.
>

I'm going to +1 on this recommendation. ORMs greatly simplify your
life, especially when it comes to caching. I'm also going to
recommend that you don't try to write anything that prevents updates
to the business objects from going directly to the DB unless you are
really OK with the idea of losing data. Sure, you can do multiple
changes to an object in memory before writing it to disk, but you
definitely don't want to do all your updates to objects on copies that
are in the cache and then rely on a background thread to sync those
objects to DB-- that sounds like a recipe for losing data, letting
inconsistency creep in, etc...

--
awl